last day (14 days later) » 

3:57 PM
1
A: Parse the html code for a whole webpage scrolled down

alecxeI would still insist on using the Twitter API. Alternatively, here is how you can approach the problem with selenium: use Explicit Waits and define a custom Expected Condition to wait for tweets to load on scroll perform the scroll to a last loaded tweet via scrollIntoView() Implementation: ...

 
Great! Thanks so much. I wished to convert the html_code read using driver.page_source to soup using soup = BeautifulSoup(driver.page_source). But soup doesn't contain the full page source. Can you please advise where I am going wrong.
please check the above doubt.
 
@AbhishekBhatia sure, are you calling the soup = BeautifulSoup(driver.page_source) after the while loop finished the job?
 
Yes, please check edit above.
 
@AbhishekBhatia yeah, just get the page source before closing the browser - before calling driver.close().
 
Thanks again for the help! Your suggestion worked but I still face a weird issue while parsing through the html code in soup. Please check the above code. Ideally it should return me all tweets but it isn't right now. text_tweet contains all the tweets, I use simple web scrapping.
 
3:57 PM
@AbhishekBhatia btw, you don't really need BeautifulSoup here. You can approach it with selenium itself - it is quite powerful in locating elements. Would you be okay with selenium-only option?
 
@Thanks for the continued support. I would prefer to BeautifulSoup as most of my code is in it. Why do think this happens? I tried your advice by replacing alltweets = soup.find_all(attrs={'data-item-type' : 'tweet'}) with alltweets=driver.find_elements_by_xpath("//li[@data-item-type='tweet']"). But it returns me with <class 'selenium.webdriver.remote.webelement.WebElement'> objects instead of list of html code of each elements founds.
 
@AbhishekBhatia okay, I've added an HTML parsing code snippet that extracts the texts of tweets.
 
Thanks for the help, but this code doesn't return all of them. If possible I wanted just code in the following form import requests from bs4 import BeautifulSoup url='https://twitter.com/thecoolstacks' req = requests.get(url) soup = BeautifulSoup(req.content) alltweets = soup.find_all(attrs={'data-item-type' : 'tweet'}) print alltweets[0] I could parse the rest.
hey!
 
hey there
let's stick to the code I've posted
quick question
what do you mean by "the code doesn't return all of them"?
 
I mean it returns not all tweets obtained when the page is scrolled to the bottom.
 
4:03 PM
how many it returns?
 
it returns
ommericals Banned For Being Too Sexy
http://coolstacks.com/commericals-banned-for-being-too-sexy/ ...
http://coolstacks.com is on sale. If interested http://www.ebay.com/itm/151019960015?ssPageName=STRK:MESELX:IT&_trksid=p3984.m1555.l2649 ...
Sofia Vergara Nip Slip - One of Sofia Vergara's breasts popped out of her sexy strapless number as she fell to the g... http://ow.ly/2udmBo
Bieber Paparazzo Dies - Justin Bieber has expressed his shock and sorrow after a man was killed while taking photogr... http://ow.ly/2u8JnM
instead of 89 present
 
okay, let's try the following
install lxml module
pip install lxml
 
okay done.
 
then, once installed replace BeautifulSoup(page_source) with BeautifulSoup(page_source, "lxml")
and check how many does it return now
 
doesn't help
Commericals Banned For Being Too Sexy
http://coolstacks.com/commericals-banned-for-being-too-sexy/ ...
http://coolstacks.com is on sale. If interested http://www.ebay.com/itm/151019960015?ssPageName=STRK:MESELX:IT&_trksid=p3984.m1555.l2649 ...
Sofia Vergara Nip Slip - One of Sofia Vergara's breasts popped out of her sexy strapless number as she fell to the g... http://ow.ly/2udmBo
Bieber Paparazzo Dies - Justin Bieber has expressed his shock and sorrow after a man was killed while taking photogr... http://ow.ly/2u8JnM
 
4:10 PM
okay, let me debug it a little bit
hm, I'm getting 86 tweets
can you post the complete code you have at the moment
(better create a gist at gist.github.com)
 
okay wait i will create one.
 
thanks
 
I am using python 2.7 by the way.
 
4:25 PM
interesting
I tried your code as is
it returned 89 for me
what's the result on your end?
Don't worry, we'll solve it
 
Thanks. But I get this:
20
40
60
82
89
Commericals Banned For Being Too Sexy
http://coolstacks.com/commericals-banned-for-being-too-sexy/ …
http://coolstacks.com  is on sale. If interested http://www.ebay.com/itm/151019960015?ssPageName=STRK:MESELX:IT&_trksid=p3984.m1555.l2649 …
Sofia Vergara Nip Slip - One of Sofia Vergara’s breasts popped out of her sexy strapless number as she fell to the g... http://ow.ly/2udmBo
Bieber Paparazzo Dies - Justin Bieber has expressed his shock and sorrow after a man was killed while taking photogr... http://ow.ly/2u8JnM
 
same here, yes
well, not exactly
okay, at least we know now that the selenium part worked well and there were 89 tweets loaded
this is good news
the problem is definitely in the HTML parsing part
 
Yeah, it retrives all 89 but doesn't parse them accordingly
 
good
let's do the following
pip install --upgrade beautifulsoup4 lxml
and rerun the code
 
done they seem to be already update and rerun returns the same.
done they seem to be already up to date and rerun returns the same.
 
4:37 PM
okay, how about using BeautifulSoup(page_source, "html.parser")?
 
okay running it. Please wait.
 
4:56 PM
Thanks!
It works like a charm!
 
wow! Glad to hear that!
 
Also, I integrated it with BeautifulSoup code. It works very well.
You don't know how much of a tension you have relieved.
 
:)
okay, see you on SO! Happy web-scraping :)
 
Just to mention, I accepted your answer. Your explantation of approach also helped me understand it.
Thanks! bye.
 
good
bye
 

last day (14 days later) »