last day (14 days later) » 

8:37 AM
1
A: How to include current page in Scrapy?

LawrenceTry renaming your parse_item() function to parse_start_url(), and rename the callback in the rule accordingly. This is an undocumented function that is called for the start urls in the CrawlSpider, so overriding it gives you the desired functionality. Alternatively, you can use start_requests() ...

 
Thanks Lawrence. I tried both option. First one ie renaming to parse_start_url() worked as expected. But another one ie. start_requests doesn't worked. I tried exactly same as code you provided (edited) but its still returns results from next page.
 
Sorry, I forgot to add the callback parameter. Try it now.
 
its throw error self._scrape(response, request, spider).chainDeferred(deferred) after adding callback parameter
 
Sorry, I didn't test it, writing from my head. Use Request instead of FormRequest. Check the code now.
 
Now its fetch only first page ie. http://www.example.com/category results.
BTW I am accepting answer as First option works perfectly fine, Thanks. Please update second option if you find solution.
 
8:37 AM
You need to update the callback on your rule. Are you talking about the first approach? Or the second?
 
First approach is working ie renaming parse_item() function to parse_start_url()
 
Is the second approach throwing an exception now?
 
Now in second approach, Its returns scrap results from first page only. ie. example.com/category.
 
The issue here is most likely that the results are being filtered out. Add this line to your settings.py file and try running the spider: DUPEFILTER_CLASS = 'scrapy.dupefilter.BaseDupeFilter'. It will disable filtering globally.
 
I modified settings.py, but still returns only first page results.
 
8:43 AM
Could you paste your code for me to run it, you can paste it here: gist.github.com
It would be helpful if you can give the correct URL's also, so I can check it here myself.
 
spider code is at pastebin.com/bnjYrxyw
title = scrapy.Field()
price = scrapy.Field()
discount_price = scrapy.Field()
original_price = scrapy.Field()
url = scrapy.Field()
designer = scrapy.Field()
designer_url = scrapy.Field()
these are items
 
ok, I got it working
I'll update my answer.
 
great
 
9:01 AM
It seems that the second approach work the same way. To run the response trough your Rule, you need to allow the request to callback the 'parse()' instead of a custom callback. So the same answer stands for the second approach, overriding the parse_start_url with a custom function. This is achieved by just adding parse_start_url = parse_item.
So you can use start_urls or start_requests, whichever you like. Ill simplify my answer.
 
OK got it. parse_start_url = parse_item
Its working now.
 

last day (14 days later) »