last day (23 days later) » 

1:05 AM
2
A: How to efficiently parse large HTML div-class and span data on Python BeautifulSoup?

Jack FleetingEDIT - At the request of @Life is complex, edited to add date headings. Try this using lxml: import requests from lxml import html url = 'https://finance.yahoo.com/quote/AAPL/balance-sheet?p=AAPL' url2 = 'https://finance.yahoo.com/quote/AAPL/financials?p=AAPL' page = requests.get(url) page2 = ...

 
I would have never considered this approach, so thanks for posting this answer. This answer definitely gets an up-vote from me.
 
@Lifeiscomplex - Thanks for the kind words!
 
Zac
@JackFleeting Thanks. It looks like I need to read the lxml documentation. How does this work? 2. How can I get several years' worth of data as opposed to just the last 3 in that row. This company, if you scrolled horizontally has data all the way for several years. Let's say there is another company which has only 1 years' data... Is there a way to get a 'num_cols' variable and set it to the number of columns there are in that page? How to achieve that? (length of the arrays total_assets etc.. should be the same as number of columns there. Thanks again!
 
@Zac - You probably don't need a num_cols variable; the code would grab whatever columns are there, be it one column or 10. The link in your question shows only 3 columns - looks like the rest require premium access. If you have that access, the code should get you the data from available columns.
 
Zac
@JackFleeting How to do the premium login via your code? This doesn't seem to have a request POST with username/password as I know in SOUP.
 
1:05 AM
@Zac - Not sure; this assumes you are logged in (or site is open). If you have access, try logging in - the url you get may be different form the one we common folks use :), and may include some kind of token you could use in the url in the code.
 
Zac
@JackFleeting Which browser should I be logged in? I changed code to this, but no change... (also the URL for Premium is exactly the same, just that more columns are visible.) page = requests.get(url, auth=requests.auth.HTTPBasicAuth( 'username@yahoo.com', 'password')) page2 = requests.get(url2, auth=requests.auth.HTTPBasicAuth( 'username@yahoo.com', 'password')) **********Does this look right?*****
 
@Zac - That's a whole topic in itself, take a look here: stackoverflow.com/questions/11892729/…
 
Zac
@JackFleeting my confusion is they are trying to use POST requests in the answers whereas your answer has a GET request. Shall I follow the answer in that post and will that work?
 
@Zac - It should work, but unfortunately I can't test it myself.
 
Zac
@JackFleeting Hi, if I remove just the one URL and run your code sequentially with many different URLs, several times the code fails with this error, " for d in entry[0]: IndexError: list index out of range"... Is there anything else I need to do while rerunning the code? I am running your code within a FOR loop which iterates through a list of URLs...
 
1:05 AM
For the authorization, a tricky way is just to find your cookies in Chrome inspector after you login, then parse it to key-value dict. After that, you can use requests.get(url='your_url', cookies='your_parsed_cookies_dict)' to get response.
 
@JackFleeting I'm not an expert in lxml, so I was wondering how do parse the dates associated with the columns being collected. I have attempted to figure out the path, but I still haven't gotten it correct.
 
@Lifeiscomplex - Are you referring to (in the balance sheet page) the 3 column headers containing 9/30/2018 9/30/2017 9/30/2016?
 
@JackFleeting yes, I was trying to obtain those dates using your answer, but I haven't been able to get the path correct.
 
@Lifeiscomplex - Got it. Give me a couple of hours to get to my "real" desk and I'll try to tackle it.
 
@JackFleeting ok. Thanks.
 
1:05 AM
@Lifeiscomplex See edited answer.
 
Thanks for taking the time to show me how to obtain this date information. Your answer is extremely clean compared to mine and the accepted one. XPATH always gives me problems, so I lean toward Beautiful Soup or Selenium to harvest data from websites. The next time that I need to harvest from a URL, I will consider using LXML first.
 

last day (23 days later) »