Victor Martinez

Python

Room rules: sopython.com/chatroom Code formatting guide: tinyu...
Jul 4, 2019 21:25
Thanks!
Jul 4, 2019 21:25
right now yes, I'm reading some articles, hoping for something in my mind can make the connection.
Jul 4, 2019 21:24
I'm going to read more about that.
Jul 4, 2019 21:24
Pub-sub. I didn't knew that term.
Jul 4, 2019 20:52
And then when some URL is stored there, the scraper should take it and start extracting info.
Jul 4, 2019 20:52
Or a good way to store the URL's to scrape
Jul 4, 2019 20:52
What I really need is something like queues or something like that.
Jul 4, 2019 20:51
I get it, but that is not my problem. The scraper it's done.
Jul 4, 2019 20:31
I'm looking for tools that help me to do this like RabbitMQ or Redis, but I'm not sure of the functionality of this tools and how to make the connections so to speak.
Jul 4, 2019 20:30
For example, I want to send 1 or 200 amazon URL's products to some database, or some system, and the scraper should read those url's and start extracting information
Jul 4, 2019 20:30
I want to have the scrapper and lets say... Send or store the URLS (Always same structure website URL) and then the scraper should start
Jul 4, 2019 20:29
Currently I'm bulding an scraper
Jul 4, 2019 20:28
Could somebody give me and advice please?
Jul 4, 2019 20:28
Hey!
Jun 25, 2019 16:11
@Kevin Yes, and in my experience there are more sites without a formal API. An API is not so normal in most companies.
Jun 25, 2019 15:59
haha
Jun 25, 2019 15:59
Yes, but not all the websites have API's. I understand what you say, but that is an ideal world, I would like to be like that... But unfortunately is not.
Jun 25, 2019 15:55
Yeah
Jun 25, 2019 15:54
Currently I'm doing some projects where I extract - Trying not to harm - some websites
Jun 25, 2019 15:54
@Kevin Yeah, I understand that. But there a serious companies that must obtain the information in one way or another.
Jun 25, 2019 15:53
@ParitoshSingh I get it... That's a good point : )
Jun 25, 2019 15:52
@roganjosh Yes, you are right, what I want to know is the whole process.
Jun 25, 2019 15:50
But, I don't know how is the optimal or best practices in companies dedicated to this.
Jun 25, 2019 15:50
After I while I started saving this CSV's into a couple of tables in a PotsgreeSQL
Jun 25, 2019 15:49
Normally I download the data and save it into a CSV
Jun 25, 2019 15:49
@ParitoshSingh Yes
Jun 25, 2019 15:49
Yes
Jun 25, 2019 15:48
I don't know if I'm being clear with what I want to say.
Jun 25, 2019 15:47
Somebody in here as production experience with scrapers? I've done multiple scrapers but I'm wondering what is the optimal process for this kind of things.... In a serious company, how this process normally runs? How is stored?
Jun 25, 2019 15:45
Hey, good morning.
Jun 6, 2019 21:28
I think I find some throubles with Multiprocessing, mostly because is not so good with handling multiple network connections, the crawler is builded with Requests library.
Jun 6, 2019 21:23
@AndrasDeak Do you think that the best solution to scalability is Multiprocessing?
Jun 6, 2019 21:22
Just to see how it works.
Jun 6, 2019 21:22
I tried Multiproccessing, but I had some problems with it. I wanted to give a try to Docker
Jun 6, 2019 21:21
Jaaja oh great, you remember : )
Jun 6, 2019 21:20
@AndrasDeak What do you mean with the first round?
Jun 6, 2019 21:20
My idea is, if I have 20 sites to crawl, I want to launch 20 Dockers. But I don't know yet how to connect that to docker. I mean, some kind of queues.
Jun 6, 2019 21:19
I have a crawler... But I want to make it scalable.
Jun 6, 2019 21:19
I have a problem, I would like a little of help if somebody can.
Jun 6, 2019 21:18
Good afternoon...
Jun 3, 2019 19:06
The principal problem is that I learned English by myself, I lack a lot of things that people learn at school.
Jun 3, 2019 19:05
Spanish
Jun 3, 2019 19:03
@PaulMcG Oh yes, my native language is not english... And is hard as hell to use that kind of words.
Jun 3, 2019 18:59
thank you very much!
Jun 3, 2019 18:59
I am going to read more and to try to solve it
Jun 3, 2019 18:58
I have used threading before in other projects
Jun 3, 2019 18:58
I'm going to read more about that.
Jun 3, 2019 18:58
I get it
Jun 3, 2019 18:58
All right
Jun 3, 2019 18:50
I get it. If that's the solution I can use it, but this is for an interview and I suposse that they need to see something more "pro" (?), like something that handle queues.