last day (15 days later) » 

9:06 PM
1
A: Beautiful Soup trouble getting nested HTML

Bitto Bennichanselect returns a list of all tags matching that tag.You can use an index to this list to get your required tag and then use .text to get the text inside. from bs4 import BeautifulSoup html=""" <div class="css-rs2cuv"> <button class="css-rzdbbc" type="button"> <svg viewBox="0 95 57 95...

 
I used lxml parser. Would this make a difference?
 
@Liondancer No. I just tested it.
 
I tried soup.select('.css-812ha7') but my code returns []
 
@Liondancer I hope you got the output.
 
With your example I did but I think my python code it giving me different HTML for some reason compared to my browser
 
9:06 PM
@Liondancer Sometimes setting a user agent similar to your browser will solve it.
 
Hello
 
Hi
Are you using requests library?
 
nope
aiohttp
pretty new to this
trying to do some web scraping
but the HTML differs than my browser
 
requests is a lot easier
 
I want to learn more async things =]
 
9:14 PM
I think setting a user agent will solve your problem. This happened to me many times.
 
mmm
it didnt =[
i see in my browser
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36
`async with aiohttp.ClientSession(
loop=loop,
connector=aiohttp.TCPConnector(ssl=False),
headers={
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
}) as session:`
 
Can you share the url you are trying to scrap?
 
gimme 5 mins i'll check
 
kk thank you
appreciate it
 
9:21 PM
Hi
I think i found your problem
Are you using the "inspect" option in chrome to find out this?
 
yes
i am
 
I was also able to get it through the inspect option. But if you go to "view page source" and check it, it is not present there.
 
oooooo
HMMM
 
The inspect method is very unreliable.
 
do you know why ?
well
actually
shoot yea im not sure how else to find the element
i am trying to find the number of pages for that website by the pagination URLs
not sure how to go about finding a fix for this =/
 
9:26 PM
I don't know exactly to be frank. But only through experience. I faced this problem before.
 
were you able to solve it?
ooo i see
do you think if i used selenium this problem could be avoided?
 
Yeah i found the real class name form the source and used that.
That is how i fixed it then.
 
hmm i see
okay ill look more into this
thank you for your help
i really appreciate it
 
Okay thanks for the link. Cheers.
 

last day (15 days later) »