last day (15 days later) » 

8:14 AM
good morning
I have this link which we fix the other day but now is not pulling the price, I think they change the selectors for price
we used Set finalPrices = .Document.querySelectorAll(".card.js-product-card span.final-price")
I tried we used Set finalPrices = .Document.querySelectorAll(".card.js-product-card a[href]") but it does pull the price
does not :)
 
8:33 AM
@MariaGeorgali I will have a look
 
I think is not pulling anything not only the price
 
[data-type='final_price']
that pulls the final price
 
based on that I want to ask you that you know better, how to simulate this more like a human? applying more delays?
So I change .Document.querySelectorAll(".card.js-product-card span.final-price") to...
 
why to be more human?
 
not to be blocked
 
8:35 AM
Also, you can restrict in ways like:
#prices .card.js-product-card [data-type='final_price']
 
every 5 min they block it
 
You can add a delay every x times during a loop
 
I tried behind vpn and again the same
 
e.g. For i = 1 to 100 : If i mod 10 = 0 then : Application.Wait Now + Timeserial(0,0,2)
There are many patterns that can be looked for not just IP. You will need to play around and see what may work.
That can include VPN, changing user-agent/ IP spoofing/ adding waits/ incognito browser
etc
 
why you put a scrolling ie.Document.parentWindow.execScript "window.scrollBy(0, window.innerHeight);", "javascript"
 
8:40 AM
that page generates items as you scroll down. If you don't scroll you only get the immediately visible items. If you want all you will need to scroll until no more items are added.
This is a case of where an API, if available, would be helpful.
 
I tried the Set finalPrices = .Document.querySelectorAll("#prices .card.js-product-card [data-type='final_price']") but nothing
 
Give a moment.
 
they had and API but they turned it off
 
So I get 6 items without scrolling and just applying that selector
Option Explicit

Public Sub GetInfo()
    Dim ie As New InternetExplorer
    With ie
        .Visible = True
        .navigate "https://www.skroutz.gr/s/10202627/OEM-T95z-Plus-16GB.html"

        While .Busy Or .readyState < 4: DoEvents: Wend

        Dim prices As Object
        Set prices = .Document.querySelectorAll("#prices .card.js-product-card [data-type='final_price']")

        Stop
    End With
End Sub
What do you see when browser opens?
 
I see the page and product it scrolls down I clear the cookies it closes the IE I go back to worksheet and I have empty column
 
8:46 AM
If you run my script and when you get to the Stop line then open immediate window and type ?prices.length .... what do you get?
 
With ie
Set wks = wb.Sheets("TESTINGS")

For j = 2 To 2
wks.Range(Cells(j, 1), Cells(j, 5)).Interior.ColorIndex = 38
counter = counter + 1
'Do
' RndNumber = Int((MaxNumber - MinNumber + 1) * Rnd + MinNumber)
'Loop While UsedNumber.exists(RndNumber)
'UsedNumber.Add RndNumber, RndNumber

'Debug.Print RndNumber

mylink1 = wks.Cells(j, 2).Value


.Visible = True
.Navigate mylink1

While .Busy Or .ReadyState < 4: DoEvents: Wend


Set products = .Document.querySelectorAll(".card.js-product-card")
t = Timer
 
Run my script first please
 
wait :) to run the script please
 
:-)
 
how I open immediate window?
 
8:48 AM
ctrl + g
 
i type ?prices.length and enter
 
yes
 
and I get an error 462
the remote machine does not exist
 
is the code on the STOP line?
 
or is unavailable
yes
 
8:50 AM
? Is IE still open?
 
My script should leave IE open - run again and then open immediate window and type the above without closing IE
 
got it
it says 4
 
Ok... so we know it can match items for you.
When you say you are blocked....
is this happening in a loop over webpages?
 
It displays a msg Are you sure you are not a robot
 
8:52 AM
aha
 
after about 5 min aand about 20 webpages
Additionally I have made my internet explorer images not to be displayed to be faster if this means something
 
You may need to experiment with waits. Maybe even process pages in batches with long gaps in between. Depends how important this is. If not against terms and conditions and urgent you could run off a few friends machines and collate results.
Where does it display robots message?
on the page?
 
in the middle of the page is a recaptcha message
I then close the module
close ie
change IP
run again from i=.. and continues
 
Can you test for presence of message with Instr and if present via code apply a wait before retrying to continue loop? May also be possible to apply the IP change through code etc.....
The means for detecting bots can be numerous and varied.
 
I have bought the cyber ghost as application so I dont know how to call it from inside vba
 
8:58 AM
Or the recaptcha element yoou could test for
@MariaGeorgali Don't know this.
ah... vpn
 
now I do the script without vpn
just one page
so you think is a delay problem and not server matter?
 
I have no idea other than the site suspects you of being a bot by your web traffic/actions on the page.
 
ok sorry for the morning trouble
thank youuu
 
I suggest you play around with the different measures you can find detailed on SO regarding avoiding bot detection (but familiarize yourself with the sites terms and conditions first as you don't want more serious ramifications)
 
have a nice day :)
 
9:03 AM
Not a problem. Happy to help if I can.
You can usually find me in dawghaus
 
what is dawhaus? :)
aaa a chat room :)
ok
thank youuu
bye
 
bye
 
 
2 hours later…
user8444404
11:41 AM
hi @ADyson
 
user8444404
sorry to disturb again.....
 
user8444404
i need to implement as this site
 
user8444404
how to get all the code from that site ?
 
user8444404
frontend code only
 

last day (15 days later) »