last day (24 days later) » 

4:11 PM
@QHarr when trying the link with this CSS ".rgt-col" I got single results not as the image you posted on the thread .. How can I get the same results as in your image?
 
To the last question of yours I answered?
To get the same query results you need to be using the same HTML and same query. So verify both of those are the same.
 
I see options in jsoup link. What is the best option?
as for Parser: I select 'HTML'
 
Yes. That is all you need.
 
Will I check any of the three checkboxes?
 
Pretty print
 
4:16 PM
and what about cleaner?
 
No
 
you mean "Not Enabled" option
 
yes. Sorry should have been more specific.
 
and as for the html content will I use Fetch URL or it is better to open the url and view the page source and copy paste to jsoup
 
It uses XMLHTTP in the background so you can use that if you plan to use XMLHTTP otherwise you need to copy paste the page HTML in.
 
4:21 PM
After a while I got this
not as in your image in explanation.
 
3 messages moved from Dawg's Waffle Haus
15 messages moved from Dawg's Waffle Haus
hello
 
@QHarr Hello my tutor
 
ok.... so this is someone else's question I answered?
 
Sorry if I am disturbing you
 
It's fine
 
4:27 PM
Yes .. I am trying to learn from you
You are really a legend
 
So let's talk through this..
 
and I like a lot your solutions
 
that is kind
 
Thanks a lot for your patience to teach me
 
We do it for each other on SO.
I get help, I give help.
 
4:28 PM
But you are the best in my opinion
 
Lol
 
what "Cleaner" should I select?
 
I don't use one. I leave as not enabled.
 
Is there an addon that enables me to deal with css queries?
 
probably but you know you can do CSS queries in the browser itself, right?
 
4:30 PM
I am not expert at these stuff in fact. I am still found some difficulties to get most problems
 
No worries.
 
That's may be because of the language
as you know English is not my native language and I found documentations difficult for me
 
No worries. We can talk it through.
 
so you find me ask a lot of questions .. although I am doing a lot of searches
 
I ask lots of questions too. I think it is a good thing.
 
4:33 PM
That's great.
 
So, what is the problem in the above please?
 
These are the results I got after a while ..not as in your image in explanation
 
The CSS query is not the same
 
you got all the text of the column on one line
.rgt-col
Is that right?
 
No. Look at the images in the answer.
Or just copy the queries from the variables
 
4:36 PM
copy the queries from the variables? I didn't get that
 
Set teamsVisitors = .querySelectorAll(".lineup__team.is-visit")
Set teamsHomies = .querySelectorAll(".lineup__team.is-home")
Set nickNamesVisitors = .querySelectorAll(".lineup__mteam.is-visit")
Set nickNamesHomies = .querySelectorAll(".lineup__mteam.is-home")
Set visitors = .querySelectorAll(".lineup__list.is-visit") ' then by li
Set homies = .querySelectorAll(".lineup__list.is-home") ' then by li
the CSS query is the bit between ""
querySelectorAll applies that query (well selector) to the DOM document
 
What about this selector ".rgt-col" .. what is the correct?
 
depends what you mean by correct. What are you trying to retrieve? Do you understand what type of selector you are using there?
 
I am trying to debug this line
Set iColumns = .document.querySelectorAll(".rgt-col")
How can I debug iColumns ..? When trying to use local the excel hangs and I had to restart
In your image I see you used this
.rgt-col
This is your image ...
 
I think I am looking at a different question. What is the answer link please?
oh...found it! Oops
 
4:44 PM
0
Q: I'm trying to scrape website data with Excel VBA

WessSo I will start by saying that I am very new to VBA. I am trying to extract data from the table on this page. I haven't acomplished much as far as the code goes so take it easy on me. I am looking for some direction on how to approach it and if it can be done, which I believe it can. If anyone is...

As I told you I am trying to learn so I may ask you about your answers to other members
As for my question was for the today's post about the school report (local html) .. how can I recognize the start and end row of the table
Your code is great .. but I need to know this point so as to use it again for other reports
 
Ok...
So, I can found out what is going on
Remember I said jsp uses xmlhttp in the background?
 
That's OK
 
Well.... it is and it isn't.
The html returned by xmlhttp is unlikely to be the same as via browser
 
it is and it isn't. Is that a puzzle :)
So what?
 
Meaning the same selector can produce different results depending on your access method,
 
4:49 PM
Can you guide me the steps you do exactly to get the same image as explained?
 
XMLHTTP doesn't carry all the info the browser rendering does. Particularly it doesn't have the javascript modifications to the page.
 
Navigate first to the link through ie or chrome?
 
Simply go open this in the browser: rotogrinders.com/team-stats/nba-earned?site=draftkings
 
then right-click and view source
 
No. Right-click > inspect
 
4:51 PM
IE or Chrome which is better?
 
Which browser are you using btw?
I am using Chrome
 
OK .. That's fine
I right-click then Inspect
What's next?
 
yup, then go the opening <html
highlight it by selecting
right-click > copy element
 
OK
That's great
 
paste that into jsp
observe the difference for your query
 
4:53 PM
The jsp took some time for you or this just happening for me?
 
it can take a second or two
it relies on internet connection
 
As for me took more than 30 seconds .. May be because of the internet
That's perfect. i got results now
from 0 to 17
 
yes
 
Thank you very very much for this great tip
 
No worries.
 
4:55 PM
That what I was searching for
You are awesome
As for the second point in "LOCAL HTML" ...
 
ok
 
how can I determine the start row and end row of the table
 
Decide first - what is the access method?
 
what do you mean with access method?
 
XMLHTTP or browser ?
 
4:57 PM
3
Q: Scraping table from local HTML with unicode characters

YasserKhalilI have tried the following code to scrape a table from local HTML file stored on my PC Sub Test() Dim mtbl As Object Dim tableData As Object Dim tRow As Object Dim tcell As Object Dim trowNum As Integer Dim tcellNum As Integer Dim webpage ...

 
you have seen that they differ
 
The method you used in your code is browser
 
Ah... in fact we don't use a browser at all.
 
I am talking about these lines
--------------------------------
startTableNumber = 43
endTableNumber = 330
numColumns = 9
 
We are reading HTML in from a file.
Simply copy your HTML from the file into jsp
 
4:58 PM
oh no I remembered
so how did you know about these exact numbers?
 
copy the html into jsp...
 
I have tried different approaches but with no use
again to the same site
 
yup
 
OK I did
what's next?
 
now put the CSS query as table
and you will see why it is so difficult to scrape
 
5:00 PM
Should I type only 'table'
 
yes
 
without dot before table
??
 
ok... so what do you think the "." means?
 
I don't know in fact
I have no great idea about css
 
5:01 PM
oops all these results from my file
I see the documentation but as I told you I can't get most of all these stuff
 
I will try to watch videos on that
 
try the last link I gave
 
Thanks for the last link. it seems well
 
the instructions/teaching is on the right
 
5:03 PM
OK I will it later. just I need to solve the current point
 
Ok.... so "." is a class selector. It selects by class name.
No "." is an element selector
 
That's fine
 
so table is an element selector (it selects by tag name)
And what you will discover is that your HTML is a nightmare of nested tables.
 
I know this is a bad government site that we deal with
It is very bad and when we are trying to print we have to use IE v. 8 or v. 9 only
 
Or it was designed to be difficult to scrape. I think, however, it is just badly designed.
 
5:05 PM
No way to use any other browsers
 
really?
 
I think it is just badly designed
Yes and that's why I know it is bad .. You discovered that from the report you see
 
So, you can see how I looked to find where the content I was after was?
 
I have searched in jsp for "16093241001" and I found 13 ... and I expect to find that for just once
 
nope!
 
5:09 PM
how did you find the content?
 
I familiarized myself with the html structure
Working out where data sat in the structure
I looked for a pattern that would isolate information that I could recognise as what I wanted in table cells
Then I constructed a table from those isolated elements
 
that's beyond my little experience
 
If you examine the query results for table
 
there are a lot of table results in this query
 
Then, pardon my arabic..
أوافق (I agree?) would be column 1
cell 1
 
5:14 PM
No as for Arabic the direction is from right to left so the column is the last column not the first
 
yes... I mean for writing out
Sorry!
 
and as for results I got reversed columns but I can solve that using arrays
 
Aha
Ok... well I am thinking about how I write out from right to left, top to bottom
So my first result I want أوافق and last I want 33087609002
and then I want everything in between those.
 
OK
I see 43
for the أوافق
 
yup
and do you see the pattern there?
All the items you want appear repeated twice
 
5:17 PM
yes and I don't know why repeating the data
 
because of their nested structure
but we can easily deal with that by making our loop use Step 2
So we only pick up one out of every two
 
That's great .. Now I could get why you step 2 :)
 
Yeah... I should have explained that.
I was in a rush doing other stuff when I answered the question. Sorry
 
Never mind my tutor
God bless you
 
So, you can see you have a list that is everything you want in a table
You then just use some maths to write it out in a table format
 
5:20 PM
But it seems no way to detect those by code
 
what do you mean?
 
I have many other reports so I have to hard-coded the code each time??!!! that's pain
 
maybe.... it maybe possible that there is a common bit of text you can search for/ HTML that will indicate the start of the info you want
 
I mean is there a way using code to automate the start and end row
 
Without seeing the reports hard to say.
 
5:22 PM
Can I give you the password of the school to have a look
 
You want to look for any unique feature that will help you consistently identify start/end
@YasserKhalil No. You shouldn't.
I am happy to help but don't feel comfortable with having someone else's password.
 
I don't know how to do this point in fact .. and I have no experience
I trust in you
I really trust in you
You are teaching me and that's enough for me
 
I am happy to teach but you may to provide files without potentially violating any existing trust agreements for that site.
Can these documents even be shared? Are they public record?
 
Don't worry about the site... Many people in my school uses it
not public for everyone .. just for our school members
 
Who agrees for the school who can access?
 
5:26 PM
It is me who is responsible
 
Is the info in those reports confidential?
 
No
they are related to students and the data is public
 
Well.... it could be a project to sift through enough to really understand the scale of the problem. How many different report formats are there? Are there actually consistent formats across reports?
 
Will I send you the credentials to have a look
 
Is it legal to do so?
 
5:29 PM
I think there are nearly two different formats
yes no problem
It is for the benefit of our work as this will save us time and effort
 
Is there a way you can put them in a drop box or some such and share that link so I am not touching the site?
 
In jsp in the first result I notice all data are there .. Is there a way to scrape the first resul of the css selector result?
 
Yes but that first element is basically the entire thing... so you would have to then sort through that. Given your arabic is better than mine that might not be a problem for you
the first result there would be .querySelector("table")
.querySelector returns one match. The first on the page.
 
Can you provide me with a code that enables me to extract the first table ?? and how can I debug to see the results
this is the link .. you can remove after clicking on it
@QHarr are you still there .. ? please remove the link after clicking on it
 
5:47 PM
I have flagged for a mod to remove.
 
OK thanls
 
only you can remove and you have 2 minutes to do that.
 
thanks a lot
 
@jonclements - thanks
 
It is removed now
I tried this line to be specific
Set hTables = html.querySelector("<table width=""100%"" align=""center"" id=""all"" border=""0"" cellspacing=""0"" cellpadding=""0"">")
But I got error 'Invalid argument'
 
5:51 PM
You need to do some homework on CSS selectors first
That is not a selector you have there.
You have put HTML into the place where selector(s) should go
 
I tried first using .querySelector("table") but the result are too much
 
too much?
 
I tried this code
https://pastebin.com/Naf0Juim
Is it possible now to parse the data from the output file?
 
I am going to have a look into this later.
 
OK my tutor. Take your time
See you later
 
6:00 PM
ok
 
 
4 hours later…
10:22 PM
Hello again .. Please have a look at this code .. I have tried to do it but I can't adjust the results
@QHarr
Thanks a lot for great help
 
No worries.
 

  last day (24 days later) »