last day (14 days later) » 

1:19 AM
2
Q: Scrapy: scraping a site that uses sessions to paginate

padIn scrapy I observed that some of my post requests to paginated pages don't work and some do. I am debugging one of these cases. I am inside pdb inspecting one such request Inspecting the XHR on the browser, the following is included in the request body Which I have checked is exactly the sam...

 
Can you share the link to the web-site and the scrapy spider code you have at the moment? Would really help with debugging. Thanks.
 
pad
@alecxe the spider code is just passing the formrequest like here: dpaste.com/1YYVMHT thanks for considering.
@alecxe the referring link in this case is hhgregg.com/furniture-home/kitchentables (going from page 1 to page 2).
 
Thanks for the information. May be this is about endpoint you use, or I am missing something. See this snippet. It doesn't matter how often I execute it - it gives me the product names..
Other ideas you might give a try: use FormRequest.from_response(). Provide headers when you yield a FormRequest: Referrer and User-Agent. Increase DOWNLOAD_DELAY setting. Well, there are lots of things to try. Could you provide me with a complete code of your spider for me to experiment with? Thanks.
 
pad
@alecxe ok, I will make the code runnable and post. I am rerunning the crawler with the trick you suggested (i.e stripping out all query params from the endpoint). It worked for a single case so I want to see if it corrects my problems.
hi
 
hey there
 
pad
1:20 AM
posting code..
scrapy runspider <filename> -o test.json
 
good, thanks. What are the symptoms right now?
 
pad
data = json.load(open('test.json'))
urls = [d['url'] for d in data]
len(set(urls))
this should be around 3000, so many items are not getting scraped
 
due to errors, right?
 
pad
with earlier code I was getting around 2500 items
 
cause I see errors from time to time
 
pad
1:25 AM
most of the missing items are deeper in pagination
 
well, I see unicode_to_str must receive a unicode or str object, got int often
 
pad
i.e page 2, page 3, etc.
 
might be a reason for it?
 
pad
I put a breakpoint in line 136
although status code was 200
and for some of the cases the response.body was empty
 
I see one problem here
 
pad
1:30 AM
For a check I use the following urls
test_urls = [
# Page-5 /appliances-home/refrigerators
'http://www.hhgregg.com/whirlpool-22-cu-ft-stainless-steel-side-by-side-refrigerator/item/WRS322FDAM',
# Page-7 /appliances-home/refrigerators
'http://www.hhgregg.com/westinghouse-1-6-cu-ft-compact-refrigerator/item/WCR16W',
# Page-2 /appliances-home/cooktops-hoods
'http://www.hhgregg.com/bosch-30-stainless-steel-gas-cooktop/item/NGM8055UC',
oh..
yeah, that bit I changed recently. I was trying to pass a url into Forrequest.from_response
1 min
 
for starters, cast page to str
form_data = {
u"contentBeginIndex": u"0",
u"productBeginIndex": str(page),
u"beginIndex": str(page),
u"orderBy": u"6",
u"isHistory": u"false",
u"pageView": u"image",
u"resultType": u"products",
u"orderByContent": u"",
u"searchTerm": u"",
u"facet": u"",
u"minPrice": u"",
u"maxPrice": u"",
u"resultsPerPage": u"12",
u"storeId": str(query_s['storeId']),
u"catalogId": str(query_s['catalogId']),
u"langId": u"-1",
u"NUMITEMSINCART": u" item(s)",
u"objectId": u"",
u"requesttype": u"ajax",
}
 
pad
yes, sorry got disconnected. That was how it was, was experimenting with some changes. Here's the correct version. dpaste.com/1D0W9Y0
 
1:46 AM
experimenting..
so
one problem I see
you are sending FormRequest to an endpoint, not to endpoint_stripped
after fixing it I'm getting 5067 scrapped items
 
pad
@alecxe yes, but the set is only about 971
even lesser than with just "endpoint"
which is why I changed it back
*set = uniques
since this is a single-script spider without pipelines there are lots of dupes
data = json.load(open('test.json'))
urls = [d['url'] for d in data]
len(set(urls))
 
ah, I see, let me check
 
pad
paste above script in shell to quickly check
 
yup
so
you are saying that response.body is sometimes empty inside parse_paginated_listing() method?
 
pad
yes
 
1:58 AM
hm
 
pad
and I think the very first one would be empty
you can put import pdb; pdb.set_trace() on line 137
that was where I put the data in my question
 
cannot reproduce it..
with endpoint_stripped used
 
pad
with endpoint it gives the empty values
 
let me check with endpoint, yeah
why not use endpoint_stripped?..
 
pad
even lesser than with endpoint
with endpoint_stripped it worked for a couple of times that I pressed c for, but then the total number of unique items is too less
 
2:03 AM
btw, I'm using inspect_response(response, self) for debugging
pretty convenient
 
pad
cool
didnt know this
 
pad
2:17 AM
so I just finished a scrape (with endpoint) this time, and there are 2171 uniques
now, with inspect response, I am getting resposne.body as empty string
retrying with endpoint_stripped to see if I can find out why it gives so few uniques
 
endpoint_stripped should be used, I'm pretty sure
debugging too
I think I see the problem
 
pad
ok..
it is returning the same response over and over in parse_paginated_listings, for some reason
that's what I think. let me investiagate.
 
2:36 AM
yup, same response, and only items from the first run
I think they are using cookies to detect the page number in the sub-category in the category
a bit tricky
 
pad
yeah, I thought so too
but requests seems to handle it
with Session
 
nope, requests would not work too
since there is a js logic involved
that sets the cookies
 
pad
hmm
they've made this change recently
because a week ago I was scraping 3000 items without trouble
 
2:55 AM
it is probably an anti-web-scraping technique
well, switching to selenium would makes things a lot easier
and slower at the same time
so, let's dig a bit more to see where and what cookies are set
 
pad
ok
so I need to manually set three parameters, right - sat_category, sat_sub_category, sat_sub_sub_category
let me try
 
yup, but I see more things set in cookies
 
pad
3:13 AM
you're right
so I changed the code to include cookies,
'http://www.hhgregg.com/appliances-home/stackables'
(Pdb) response.request.headers['Cookie']
'sat_category=Appliances; sat_sub_category=Laundry; sat_sub_sub_category=Stackable Washers & Dryers'
for the above url, I've including the cookies I sent along
and you're saying that the other cryptic fields are somehow generated by javascript
 
they are doing a good job obfuscating stuff and making it difficult to scrape
yup
inspect the XHR request for the pagination
 
pad
I'm probably the culprit :)
 
you'll see things like s_sess
 
pad
ok
 
may be :)
things like
s_sess=%20s_ria%3Dflash%252016%257Csilverlight%2520not%2520detected%3B%20s_cpc%3‌​D0%3B%20s_ppv%3D11%3B%20s_v2%3Dnon-internal%2520campaign%3B%20s_stv%3Dnon-search%‌​3B%20s_cc%3Dtrue%3B%20s_prop_17...
and if you unquote the s_sess cookie value multiple times
you'll see that there is a js code partially inside
containing pageNumber
may be this pageNumber actually makes the difference..
it's fucked up anyway :)
 
pad
3:16 AM
you mean this ..
 
are you okay with selenium-based approach? :)
 
pad
I guess.. but it is very slow, although there'd be no other option it seems
without cookies enabled in firefox, pagination does not work
so even if I get the page number from that string
I don't know if I can do anything much
selenium seems to be the only option now
 
well
I'm currently being distracted like every minute and a kid is hanging on my shoulders - so, I'll give it an another try probably later
 
pad
haha. np. thanks a lot for looking into this
i learnt somethings from you today :)
 
selenium - yup, what is good about - it is that you don't need to worry about the technical things like cookies or headers or etc
high-level approach
and you can go with a headless approach (phantomjs)
also
there is scrapyjs project
try it out - who knows, may be it would handle your case
I've learned a lot today too, thank you
 
pad
3:25 AM
ok, Ill look into them.
@alecxe if you come back to this, would have a suggestion for why some pagination was working (with endpoint) but not all. Thanks a lot for the help.
 
@pad ok, I'll throw something as a comment, or answer if there will be any progress. Feel free to reach me out also (see linkedin link in the profile). Thanks.
see you on SO anyway
 

last day (14 days later) »