« first day (3819 days earlier)      last day (1119 days later) » 

12:10 AM
nvm... i finally found the issue
it's this line in the ajax function lol
`xhttp.setRequestHeader("Content-type", "application/json")`
iam sending a formdata with ajax headers.
 
12:29 AM
@LoopingDev that would do it. Is there a reason you are sending a FormData instead of a JSON object?
oh, probably because a is a file...
 
12:48 AM
@Code-Apprentice guess what.
when i removed that line .. i was able to see the value of the
q = JSON.stringify(jsonList));
but the file is still none...
i tried to add setRequestHeader("Content-type", "multipart/form-data") as everyone suggested it
it made both none again.
 
1:04 AM
nvm i knocked it down ;) , they said its not possible with vanilla js, well I've just proved them wrong ^^
 
 
7 hours later…
7:46 AM
I want to assgin e.g. m,v,p = 'a/b/c'.split('/'). But there are values like 'a/b/' and I got not enough value to pack error.
If there's no enough value, I want to set the rest of m,v,p to empty string ''. If there's a pythonic way to do that?
 
Not really. Only thing I can think of is m, v, p, *_ = 'a/b/c'.split('/') + ['']*3, which is pretty ugly
 
ok
problem values I mean values like 'a' or 'a/b' , 'a/b/' is fine.
 
m, v, p = ('a/b/'.split('/') + ['']*3) [:3]
This does not have the memory problem of using *_
 
8:01 AM
"Memory problem"?
 
Well I would just go for 3 if else
: )
 
Would 'a/b/c/ex' be valid input? What should m, v, p be then?
 
a,b,c then
 
Right, so no need for split('/', 3) , then.
 
I have 3 variables need to be assigned, dict['m'], dict['v'], dict['p'] .
 
8:04 AM
Or should that be 2?
 
@holdenweb If we use *_ then all the additional data will also be stored in the memory. I think
 
I would recommend you stop focusing on such trivia, particularly when it isn't based on certain knowledge.
Can you give an estimate of its impact on a 10kB and a 10MB string (say) in terms of what additional resources would be required? If not, what is the basis for your conceern?
 
8:21 AM
I meant to say that if any additional data is provided, like 'a/b/c/ex' which need not be stored, this is going to store it as a variable and those extra empty strings at the end.
As you mentioned it is trivial. We can ignore it.
 
And should ;-). Simpler to design the algorithm to do exactly what you want, then optimise it only when performance becomes an issue.
Unless you are fairly well-versed in Python's data representation it's hard to make accurate guesses about emory utilisation anyway.
 
 
2 hours later…
10:35 AM
I want to download multiple files from a website: https://www1.nseindia.com/products/content/equities/equities/eq_security.htm
Basically, I want to download all historical data of all the stocks.
I have made something using selenium.
https://colab.research.google.com/drive/1fyU1O_Fzm94Xs3O0sDgBO7egVEKW19aK?usp=sharing
But the code is too slow.
Is there any way to run my code parallelly and without opening the browser tab?
 
@PoornaChandraR use the requests pypi package to download and take a look at the threading module to execute in parallel
 
user13752012
hello
 
user13752012
i want to ask about adb
 
user13752012
is there a way to unlock my phone with adb shell without any input of text?
 
user13752012
with no input
 
user13752012
10:40 AM
command
 
user13752012
(adb shell input)
 
I want to click on the download link in the web page after submitting a form.
Can I do that using requests?
 
@PoornaChandraR why click on a download link when you can just keep changing the URL params and download data for multiple things?
 
Full data is getting loaded to the webpage.
Only 100 lines are appearing in the web page.
But full data is getting downloaded if I click the link
 
the download link - find it - copy it. it has paramaters which you can change to download that and other things
 
10:48 AM
:51895588
$.get('/products/dynaContent/common/productsSymbolMapping.jsp',{symbol:symbol,segmentLink:segmentLink,symbolCount:symbolCount,
series:series,dateRange:dateRange,fromDate:fromDate,toDate:toDate,dataType:dataType},
function(html){
html=$.trim(html);
$("#historicalData").html(html);
$("#historicalData tr:even").addClass('alt');
$("#historicalData").show();

$(".download-data-link > a").click(function (event) {
exportDivToCSV.apply( $(this), [$("#csvContentDiv").html(),$(csvFileName).val()]);
This is the source code. Can you suggest me what to do?
 
Forget the source code, hit Ctrl+K to open your browser's dev console, switch to the "Network" tab, click a download link, and the URL will pop up in the console.
 
11:04 AM
I tried that. But for some reason, it is limiting the output to 100 rows.
 
I don't really follow, but the process remains the same: You click the buttons that do what you want, and the dev console tells you the corresponding URLs.
 
Is StackOverflow the right place to make a post about why I can't get Cuda to recognize my GPU?
 
https://www1.nseindia.com/products/content/equities/equities/eq_security.htm
This is the link. Please try once.
Put the symbol as 'SBIN' and period as for '24months'.
 
Ok? And what would selenium do now to load more results?
 
It just clicks on 'Download file in csv format'
 
11:18 AM
So you can either 1) Click the download link and see the URL in the dev console, or 2) extract the download url from the HTTP request that created it
When you press "Get Data", your browser sends a GET request to productsSymbolMapping.jsp?symbol=SBIN&segmentLink=3&... and the server sends back some HTML code that contains the download link
 
11:53 AM
What is the 'bytes' type in python?
 
@PoornaChandraR OK, I know you've been following youtube videos for a year. I think it's time that you read the official tutorial docs.python.org/3/tutorial
at least I can only hope that this question would be answered there :P
googling through the tutorial makes me less optimistic
 
yeah, the tutorial has awful SEO. It does cover most topics, but finding them is hard
 
BTW, I did not see any YouTube videos. I used 'CS50's Web Programming with Python and JavaScript'.
did not complete it though\
 
Ah, OK, I misunderstood your profile then
Although you should definitely learn solid fundamentals before building frameworks on top
 
12:09 PM
@AndrasDeak That was written 6 months ago.
@AndrasDeak 👍
 
What do you even use bytes for? That really shouldn't be something you have to interact with in a webscraping project
 
I tried this:
soup = BeautifulSoup(response.text, 'html.parser')
type(soup.select('div')[1].encode_contents())
It returned bytes
 
well, encoding str will do that usually
 
Well, don't use encode_contents then. I'm sure there's a corresponding method that returns text.
 
@Aran-Fey It turns out that GET request is getting all the data and storing in a div which has style="display:none;"
 
12:18 PM
Hahaha, wow
 
hack the world
 
12:32 PM
I've got a little web scraping project that had been working perfectly for about six months, but the semi-documented endpoint I was GETting suddenly started returning HTML instead of json. I am sad.
If it's returning the same data in a different format, that's not so bad. If they overhauled the whole API and I have to find the correct semi-documented endpoint again, that will be a pain in the butt.
 
morning cabbages folks
 
Just recently I wasted an hour re-reverse-engineering an API only to discover it was still the same as before, only on a different domain
 
ugh
 
I guess I'll open Schrodinger's Cat's Box and see what's in store for me
 
12:52 PM
Result: I had to find the new endpoint, it was a pain in the butt, but functionality is restored. Thus ends this dramatic tale.
 
That was a pretty short pain in the butt though
 
Like a swift kick in the rear yeah
The page I'm scraping is a visual punch to the face which is part of the reason I'm scraping it
The other reason is that it uses an additional 10% of my CPU every time I click "see more results". A table of mostly-static data should not do this.
 
it's JS cryptomining in the background
 
sadly plausible
 
plot twist: the numbers you get when you click on "see more results" are performance metrics from the cryptominer and have nothing to do with the actual data you're after. The whole thing is conspiracy to get you to stay on the page longer, so that they can cryptomine off you for more time
 
1:14 PM
"... Why does it say that tomorrow's humidity is 0.0166 USD/watt?"
 
that's the electricity price you'll need to face, to run your dehumidifier to bring the humidity to your normal level
 
Oh cool, I suspect nothing in that case
 
2:00 PM
I wanna make one more joke about the Suez Canal... but that ship has sailed
 
Anyone know of any literature on mixing concurrency mechanisms? Like gevent and threading in the same program? All I can find via the Google are articles that are comparisons, not case studies of combining them.
 
Any of these help?
https://stackoverflow.com/q/20199242/198633
https://stackoverflow.com/q/26638345/198633
Apparently, gipc combines gevent + multiprocessing (missed it by that much)
 
2:32 PM
@PaulMcG I used to work with a codebase that used greenlets underneath async on 3.5. It sucked and was a horrible mess. I suspect the reason there is no literature is because most people are lucky enough and/or sensible enough to avoid situations that require it.
 
At work, we have a situation where a vendor library uses threading to buffer up network messages and uses a thread to send the buffered contents every so often. This library gets used by other microservices in our environment, some of which use gevent. It turns out that gevent has a monkeypatching method gevent.patch_all() for a number of modules (such as threading and requests) to make them play nice with the gevent event loop - if you import and patch in the correct order.
@inspectorG4dget Thanks, IG
We have isolated the vendor library call behind our own insulating call. I've decorated that call with a guard to ensure that, if called with gevent enabled, that the threading module has been monkeypatched, and if not, raises a WhatWereYouThinkingException
 
safeguard.ProtectionLevel.IKnowWhatImDoingTrustMe = True
This is actually a trap to detect users that use PascalCase outside of class names, and apply even more safeguards to them
 
2:50 PM
@PaulMcG Great defensive programming practice!
 
3:04 PM
@Kevin Sounds very similar to my @im_the_human_i_know_what_im_doing() decorator
def im_the_human_i_know_what_im_doing(fn):
    def _inner(*args, **kwargs):
        try:
            return fn(*args, **kwargs)
        except:
            print("I'm the human, I know what I'm doing")
    return _inner

@im_the_human_i_know_what_im_doing
def do_something_dumb():
    1/0

do_something_dumb()
 
attitude like that will put you first on their list when they rise
 
I thought about doing "except: pass", but then the print statement seemed better (gives the illusion of actually handling the exception).
 
oh God! this reminds me of when I learned decorators and made them yam around with globals(). That was fun
 
has anybody ever started working somewhere and the code WASN'T a huge mess of spaghetti code? Like why does nobody read these basic style guides and rules. They are so simple, why don't people go huh, maybe my design has some flaws, if 80% of the lines are bright yellow in Pycharm with warnings.
 
3:22 PM
when I started at current g4dgetCorp, the codebase had been maintained by a dev that was an English lit major in a previous life. Most modular and well-documented code I'd ever seen
 
<3
 
What kinds of warnings does PyCharm display? I'm curious what it can detect.
I'm guessing stylistic things like "use PascalCase for class names" are pretty easy
 
I'd imagine PascalCase for classnames would be unable to catch "Classname" errors. Please tell me I'm wrong - I'd love to know this problem has been solved (how?)
 
Classname is fine depending on the name.
just like no computer can tell you if it's PenIsland vs PenisLand :P
 
simply train a neural network on all english words, and apply it to the class name ;-)
 
3:36 PM
btw, does pycharm still use its own linter instead of wrapping flake8, which makes it impossible to preconfigure it nicely for new developers and also results in different results in CI (with flake8) and the editor?
 
Ok, revision: I'm guessing things like "class names should start with a capital letter" are pretty easy
"does pycharm still use its own linter" -- that's what I'm trying to find out right now, so I can browse through the linter's docs
 
I use IntelliJ with the Python plugin so it's basically the same as PyCharm. IDK what it does for linting. It seems to catch all PEP8 stuff.
 
I know from my PyCharm (pre my flake8 config/setup) use that it catches most PEP8 stuff. There's a config option in settings for which of these you could turn on/off. Meanwhile, it also does some interesting things like catch typos
 
If it catches all PEP8 stuff but only PEP8 stuff, that answers @hakaishin's question. You can have perfect style and still write spaghetti code, because there's no rule against spaghetti code in Python's style guide.
[citation needed]
 
PEP 8.1 No spaghetti code
@inspectorG4dget typos? as in "spell checking"? I think that is separate from the linter.
 
3:50 PM
Discovered a nice feature in JS, sprinkled it on my code, works in Firefox, not supported in IE. Why did I let myself get hurt again ;_;
 
@Code-Apprentice yes and yes
 
@Kevin PyCharm calls it an "inspector". I find PyCharm's "Settings" menu structure kind of wonky - look for it under "PyCharm/Preferences/Editor/Inspections". Here is a screenshot of what you can inspect for.
And each option (like the PEP8 ones) can have a sublist of inspections to enable/disable, or options to configure (like line length)
Project settings are also in this tree, as "PyCharm/Preferences/Project: your-project-name" This is the part that is weird to me. I would have expected to be able to get to the project settings by right clicking on the project in the Project view. But once you get used to it, you learn it. But I don't have to like it.
The spell checking is supposed to help you find variable name typos, but I find it kind of intrusive when it complains when I name a scratch file name variable fname.
 
AAB
Hi all,
Is using sqlite3 as a queue a bad idea?
I have process that will insert an id and another process that will consume them, my loads are low/non-existent now
but what is an approach that will be good
 
4:05 PM
@AAB sounds good to me
 
AAB
@Code-Apprentice I am confused if I should use something like zeromq because all I need is a temporary storage/cache and then I will delete them after cosnuming them
I am not sure which will be lighter and safer :/
 
I'm not familiar with zeromq, so I can't comment there
 
AAB
@Code-Apprentice any alternatives or others way you would suggest to do the same?
 
Using a database table as a queue is not out of the norm, but sqlite may fall down quickly, since it doesn't (to my knowledge) do any optimized read/write to its file persistence. When you update a table by adding or removing an item, how does sqlite flush it out to its file? It has been a while, but the last time I used sqlite, it just rewrote the whole thing - hence the "lite"-ness.
 
If you need a queue that multiple processes can access, consider multiprocessing.Queue
 
4:12 PM
^^ what @Kevin said
 
AAB
@PaulMcG sqlite3 is in-memory only when commit is called it writes to file right?
but if I call commit after every write it will be slow :| hmm I guess sqlite3 may not be good then
@Kevin thanks will look into it.
 
@Kevin - whenever I share resources across processes using multiprocessing (which I will now refer to as 'mp'), I need to fire up the process as mp.Processes, and point them at the Queue. Can a separate Python program "find" an mp.Queue and interact with it somehow?
 
AAB
@Kevin so my scrapy code creates the queue and puts data in it, a seaprate python script is running
or as in the example I should create a function that does the job and start as part of scrapy?
I was planning like when the cron job starts the scrapy crawl I will start this python script as well
The doc says the communications happen through a pipe so this will only work for parent-child processes right?
 
4:43 PM
I would just create a tiny REST service (falcon? bottle?) with PUT and GET methods to push and pop from a deque. (Yes, using GET for "pop" violates idempotency of REST GETs, shoot me.) The benefit here is that enque-ers and deque-ers can just find your little server by host and port number, so they can even be on separate hosts. Then decide if you want to persist the queue or not. (Hmm, I'm learning docker, might make for a nice little docker thingy.)
 
4:53 PM
Unless you are dealing with tons of unhandled messages (>1k) just dumping each message as a separate file is a lightweight, portable means for IPC.
 
5:17 PM
I haven't tried putting a queue between non-parent-child processes, so I'm not sure if it's possible
 
AAB
@PaulMcG curious but how are you learning docker? :P
@PaulMcG @Kevin I am looking to give zeromq a go just will read up a bit about it all I need is a queue seems like this may be lite and work.
what do you think? or is it overkill for what I want?
 
Go for it
 
AAB
5:32 PM
@Kevin thanks
 
6:05 PM
Hmm, should I implement a fun quadtree algorithm, or should I actually measure my code's performance first to see if it needs a quadtree algorithm
Surely converting an O(N) process to an O(log N) one will reap hefty benefits for my use case of N = 100, constant multiplier = 4e-2 milliseconds
 
Use a dedicated quadtree package to see if it makes it faster, and then implement it from scratch?
 
\o/ Hey! I found a foot-gun!
from dataclasses import dataclass

@dataclass
class Node:
    name: str
    left: int
    right: int
    __call__: callable
but alas, Node('f', 1, 2, lambda x: x)(3) doesn't work
 
Hmm, there is a quadtree lib available in my target language's package ecosystem...
 
just don't spend more time wiring in the third-party lib than you would writing your own :P
@piRSquared if it doesn't work then where's the footgun? What does work?
 
Node('f', 1, 2, lambda x: x).__call__(3) /-:
Not! pretty
 
6:16 PM
heh
 
Hi guys
Google colab is not returning anything for this request.
response = requests.get('https://www1.nseindia.com/products/dynaContent/common/productsSymbolMapping.jsp', headers=headers, params=params)

Any idea why?
Other links are working fine
 
What does "not returning anything" mean?
 
It just keeps running
 
Is it a possibility that the page just doesn't load...?
 
This doesn't solve your problem, but consider providing a timeout argument to get() so it crashes instead of freezing docs.python-requests.org/en/master/user/quickstart/#timeouts
Useful for debugging since you won't have to guess what statement is causing the hang
 
6:38 PM
@AndrasDeak It is loading when I run it locally.
 
Is it a possibility that the page just doesn't load from google's cloud?
because, for instance, someone else had scraped them and now the whole IP range is blacklisted
 
IDK, Is there anyway to check that?
 
You could try making a request and checking if you get a response ;)
 
You could email the website and ask them
 
If you email from google's cloud there's two independent signals
 
6:45 PM
@AAB IDK much about queue libraries. On my current project, we use celery and redis, but I don't know much about either one.
 
@Kevin found this on their terms of use page:"You may not conduct any systematic or automated data collection activities (including scraping, data mining, data extraction and data harvesting) on or in relation to our website without our express written consent."
 
you'll have to rephrase your email then
 
Well, that's easy then, if you can get their written consent then they'll probably put an exception in their blacklist for you
I wonder how not-so-scrupulous people get around IP blacklists. The only anti-anti-bot measure I have practical experience with is changing my user agent string, but obviously that won't help here.
 
proxies
 
I only use proxies when I'm playing Magic: The Gathering at my kitchen table ;-)
No point dropping 200+ dollars on a Tabernacle of Pendrell Vale if I can just write that on a one cent Plains and pretend it's the same thing
And Hasbro is none the wiser >:-) please don't tell Hasbro.
 
6:54 PM
you don't just print your own cards?
 
Oops, I missed a digit, Tabernacle is 2000+ dollars
@Code-Apprentice Nah, I don't have a printer. That's more of a scam than the MTG secondary market.
 
yah, me, either.
 
@AAB Mostly watching YouTube videos, TechWorld with Nana. I had to reproduce a sharded mongo db to repro a work issue, found this article which was really interesting, and ran that in a Ubuntu VM on my Windows machine. I've had one docker project in mind (creating an image with my plusminus package running in a bottle server - or maybe FastAPI), but creating that little queue server would just generally be a handy thing.
 
I have two printers, and one of them is cheaper than a new set of ink cartridges that go in it
I guess that would also hold for colour laser printers
 
talking about printers...I had to get a new driver's license recently. Part of the requirements is documenting your mailing address. Since I don't do snail mail for any bills, I had to print out the electronic versions to take to the DMV. Then they just turned around and scanned it into their systems. Why can't I just upload the PDFs that I already have and save some trees?
 
6:58 PM
@Code-Apprentice because that would open the way to forgery, duh! D:
 
I mean I could just forge a "bill" in Word easily enough
 
One of the government's secret objectives is to produce pointless busywork for people who would otherwise be unemployed
 
@Code-Apprentice yes but printing it makes it real
@Kevin good guy government
 
Anyone in a position to change it will notice that greater efficiency will reduce the required size of the department they rule over, which is the opposite of what they want
 
this is very true, and is often the entry-point to my minimum basic wage argument -
> you're creating work that doesn't need to be done to give someone money. Why not just cut out the middleman (or middlework, in this case)?
 
7:00 PM
@AndrasDeak ok, I get the sarcasm here. I was just really annoyed by the whole thing since I had to find a printer to use.
@inspectorG4dget UBI!
 
@inspectorG4dget but that's communism
 
could give it a soft start, just to trial it. Could call it... UBIsoft
 
I recall reading about a country that 100% guarantees employment for anyone to ask. They put you in a warehouse and pay you minimum wage to thread beads onto string for eight hours.
 
@inspectorG4dget does it include microtransactions?
 
I fear you've out-referenced me
 
7:03 PM
in that you don't get my reference or you don't have a follow up?
 
There is some talk that the warehouse next door employs people to unthread those same beads, but I forget if there is any firm evidence.
 
@Kevin that's a socialist trope
 
Oops, "employment for anyone to ask" should be "employment for anyone who asks"
 
I don't get your reference. What is "microtransactions", aside from the literal interpretation of "transaction of small value"?
 
Let's have a citation race, I'll look for bead-threading welfare in real life, you look for it in fiction
 
7:05 PM
Unemployment is "a hazard to society", so everyone is given a job, which might just be a box assembly factory situated right next to a box dismantling factory. We have a satirical movie from the eighties with that very premise.
 
Reminds me of George Marshall's revival plan after the Great Depression: you there, dig a long ditch; and you! fill in the ditch he just dug
 
@Kevin I just closed the wiki page, although it's not very detailed in English (the movie itself is a bit NSFW) en.wikipedia.org/wiki/Sound_Eroticism
 
@AndrasDeak was that movie featured in an Apple release at some point? The name escapes me at the moment and that bugs me
 
There's no plot summary in English, but I believe you. I award you one IOU for a uranium kewpie doll.
If I somehow gain possession of the kewpie doll that you already own, you can cash this in to get it back
 
@inspectorG4dget no idea
@Kevin sounds an awful lot like shorting
 
7:09 PM
@AndrasDeak absolutely not. I saw no references to monkeys or bananas :P
SO Main is down?!
 
@inspectorG4dget it's when you pay a small amount of money for something in an app...often in games
 
@inspectorG4dget nope
 
must have been a glitch then. I definitely got the "we're down" page
 
It's had some hiccups these past few days
 
@Code-Apprentice ahh I see. Very well played - clearly beyond the reach of my meatbag brain
 
7:12 PM
@AndrasDeak similar yes, but there's no SEC equivalent to punish me if I can't deliver the goods at the promised time
The SOPython economy runs mostly on a combination of honor and vigilante justice
 
can't wait for the quatloo bubble
 
Quatloo value is rock solid, it's backed by the value of uranium kewpie dolls
 
AAB
7:33 PM
@Code-Apprentice thanks
 
@inspectorG4dget That's one of the first things I think of when I hear "Ubisoft"
 
AAB
@PaulMcG nice i subscribed to the youtube channel but have not gotten around watching them I just lack dedication :/
 
@Code-Apprentice I'm not as much of a gamer. Do they have many mobile games, rich in in-game purchases?
 
@inspectorG4dget IDK about their mobile games, but even their triple-A titles get lambasted for all the in-game purchases. Case in point: Assassin's Creed Valhalah.
 
oh really?! I've only ever played the original Far Cry (which was free-in-game), so I had no idea
 
7:47 PM
AC is one of my most favorite franchises
 
@AAB If it lowers your activation energy any, watching at 1.25 speed cuts your time investment without significant reduction in knowledge transfer.
 
8:01 PM
Also, take hand-written notes in a notebook that you use specifically for learning new things. There is just something about writing down words vs hearing them or even typing them into a document. Just writing will really help in your retention - also drawing diagrams (well, preferably diagrams related to the content).
 
AAB
@PaulMcG yea will keep this in mind
 

« first day (3819 days earlier)      last day (1119 days later) »