« first day (2000 days earlier)      last day (1356 days later) » 

11:31 AM
while True:
try:
table = driver.find_element_by_xpath("//table[contains(@class,'el-table__body')]")[0]
for row in table.find_elements_by_xpath("//*[contains(@class,'el-table__row')]"):
sku = driver.find_elements_by_xpath("//span[@title='Product Details']")[0].get_attribute('textContent')
Lines.append(sku)
quantity = driver.find_element_by_xpath("//span[@title='Product Details']/ancestor::tr/td[3]//div[1]//div").get_attribute('textContent')
Lines.append(quantity)
logging.info(Lines)
break

Hi everyone, does any one know why my loop keep looping forever ?
 
1 message moved from Python
 
 
3 hours later…
user11867287
2:52 PM
but only once indexed on mainpage
 
user11867287
which is great works great push notif within 1 second
 
user11867287
BUT, now I need to auto-buy with Paypal API
 
I have no idea what's going on tbh
 
user11867287
and to do so, first step I simply need to scan the webserver for newly uploaded pages that are not indexed
 
If the question is "how do I determine when website.com creates a new page with a name I don't know ahead of time, for example website.com/12345678"?, I don't think there's any easy way to do that besides crawling through every link
And even that won't work if there isn't any way to get to the new page from the front page
 
user11867287
2:54 PM
You can directly go to the newly uploaded pages
 
user11867287
before indexed
 
user11867287
eg: website.com/11223344
 
but do you have the link?
 
user11867287
Exact link no
 
user11867287
but a fairly precise idea
 
user11867287
2:55 PM
of the range
 
I feel like we're quickly moving from web scraping to an unintentional DOS attack here
 
user11867287
e.g.
 
user11867287
@Aran-Fey if I want DDoS
 
user11867287
I no need halp
 
well, he said unintentional :D
 
user11867287
2:55 PM
I need a credit card!
 
user11867287
Good point though
 
user11867287
I did fear it would cause problems with my initial script
 
user11867287
but absolutely didnt, probably not the only bot connected to their servers either
 
user11867287
@Kevin Say I knew, the first 3 digits
 
user11867287
of the 6 digits page
 
2:57 PM
that leaves exactly 729 combinations
 
user11867287
Which should take how long if the request is simply waiting for 201 code
 
user11867287
then onto the next one
 
user11867287
with say, 10 concurrent connections
 
user11867287
Like, it can have delays, as this is just the precursor
 
user11867287
Once uploaded, it's not yet available to buy anyways for a couple of hours sometimes
 
2:59 PM
It sounds like you already have a design in mind. Go ahead and get started writing it, and let us know if you run into any specific problems.
 
user11867287
So even if its slow and takes 5 minutes per mini-scan, good by me
 
(Oh, and of course verify that the site's TOS allows webcrawlers first)
 
user11867287
@Kevin I barely know python, so I was wondering if you ever heard of something similar. Also, can you vouch for the principle?
 
user11867287
Yes, robots.txt, yes sir.
 
The principle sounds feasible to me.
 
3:01 PM
it's pretty simple to implement on your own. If you know node.js, why not use that instead?
 
user11867287
Ok here's a specific question for you.

How do I craft the main part, the "10 connections, the ruleset (123XXX), and the skip to next asap (201)"?
 
user11867287
Because there's no room if I get into issues
 
user11867287
and I'm making Andras lose his hair
 
user11867287
with all my node questions in pyroom
 
Oh, is this a JavaScript question? I must have missed that part. Try the JS room.
 
user11867287
3:03 PM
No it wasn't
 
user11867287
But your prompt answer vouches my point.
 
user11867287
@Kevin Is there some sort of ultra-fast crawler that you know of?
 
user11867287
Any leads other than "great idea, now go make it!"
 
user13415013
Hi @AndrasDeak, my problem is solved. Actually, tqdm didnt shows the progress if any exception happens, that was reason, which look like download is freezing. Thanks
 
@nerd that's a bit surprising to me; are you sure it wasn't your except block that hid the exception?
 
3:05 PM
I doubt you need a crawler if you just need to guess the url
 
Can't say that I do. I just write my own crawler from requests and beautifulsoup when I need one. If anything I aim to make them as slow as I can tolerate so I don't trigger the web server's rate limiter.
 
@nerd oh nevermind, I misread. It's possible that it stops being a progress bar, yes.
 
user11867287
@khajvah Yeah I know, but I have no other leads.
 
user11867287
Plus delay wouldnt be a BIG issue
 
user11867287
and if I could fetch an element or two from 201 URLs, that'd also be great
 
3:07 PM
write a function that generates all the guesses. write another function that takes those guesses and tries one by one
 
user11867287
My man.
 
user13415013
yes, actually problem was my url, crawling is easily detected by sites.
 
user11867287
@nerd out of curiosity, what was the website and how aggressive were you crawling>
 
@nerd you don't have to answer that question
 
user13415013
yes
 
3:09 PM
you can if you want to, but you have no obligation to sate Gary's curiosity
 
Since bandwidth tends to be the bottleneck when it comes to internet-facing applications, try using requests.head instead of requests.get. If all you need to know is "does this url return "200 OK" when I poke it?", then you may as well tell the server to skip the body
 
user11867287
Ahhhh thats what im talking about.
 
user11867287
chewy, meaty info
 
user13415013
@GaryOak, It is quite illegal to say here, but i am making dataset of girls faces.
 
one upvote for Kevin
 
3:10 PM
@nerd If it's illegal to say here, don't say it here
 
user11867287
@nerd very interesting, any girls?
 
... At least I assume that's what requests.head does. The documentation is a little light on details.
 
user11867287
I'll try it out
 
user11867287
@nerd still pretty curious about your rendering/training.

I have mad computing power for free if you need
 
user11867287
Tesla GPUs n sh*t
 
user13415013
3:14 PM
@GaryOak , Yes, bro yes. colab isnt handle too much, I have about 11 laks and 10 laks links, I'll dm you.
 
user11867287
dm me ill send u email
 
2 messages moved from Python
 
user11867287
and if you know people who pay for cloud rendering, switch em to me and we split 50/50
 
user13415013
@GaryOak, I am nerd, just looking for jobs, I dont know any people except codes.
 
4 messages moved from Python
 
user11867287
3:16 PM
@Andras lol.
 
user13415013
@GaryOak, send your email
 
2 messages moved from Python
 
user11867287
Mods, can you create a room temp?
 
@GaryOak you're on your own, do whatever you want just elsewhere
 
user11867287
Mods (excluding Andras unless directly addressed, thank you)
 
3:18 PM
3 messages moved from Python
 
user11867287
3:29 PM
Amazing.
 
user11867287
Thank you.
 
user11867287
I have another decently specific question
 
@GaryOak no
 
user11867287
@khajvah you said something about 700ish possibilities
 
user11867287
Isn't it more?
 
user11867287
3:32 PM
like a 1000
 
user11867287
and how would you approach the auto-gen?
Print to file and then use file line by line?
 
Maybe he's assuming ids won't have leading zeroes, and so would have only 9^3 possibilities
... Although I suspect that formula is a little off
 
user11867287
Ah.
 
user11867287
and do you think I should auto generate a url-possibilities file?
 
user11867287
then use that file to check for requests.head?
 
user11867287
3:35 PM
line per line?
 
@GaryOak in case Andras's "no" wasn't clear, he was telling you not to ask more questions at this time.
 
gtg for a bit
 
user11867287
Thanks a bunch.
 
user11867287
@davidism Do you have an idea?
 
18 messages moved from Python
 
user11867287
3:55 PM
Someone got a good page to learn about python permutations?
 
1 message moved from Python
 
71 messages moved from Python
 
user13415013
4:51 PM
@JonClements, what do you think would be better urllib.requests.urlretrieve or requests.get
 
user13415013
@AndrasDeak, requests.head returns content of None but requests.get returns bytes . Is there any problem with these
 
@nerd please don't ping people directly for random questions. Additionally, given the moderation that was applied to your previous posts, I think it's time for you to take a break from this room for a while.
 
user13415013
ok, please forgot my mistakes, Andras was right, i think he was liar , he just didnt response after sending my links,.
 
user13415013
ill be back after sometimes. good day
 
5 messages moved from Python
 

« first day (2000 days earlier)      last day (1356 days later) »