I'm looking for tools that help me to do this like RabbitMQ or Redis, but I'm not sure of the functionality of this tools and how to make the connections so to speak.
For example, I want to send 1 or 200 amazon URL's products to some database, or some system, and the scraper should read those url's and start extracting information
Yes, but not all the websites have API's. I understand what you say, but that is an ideal world, I would like to be like that... But unfortunately is not.
Somebody in here as production experience with scrapers? I've done multiple scrapers but I'm wondering what is the optimal process for this kind of things.... In a serious company, how this process normally runs? How is stored?
I think I find some throubles with Multiprocessing, mostly because is not so good with handling multiple network connections, the crawler is builded with Requests library.
I get it. If that's the solution I can use it, but this is for an interview and I suposse that they need to see something more "pro" (?), like something that handle queues.