« first day (3583 days earlier)      last day (1351 days later) » 

12:48 AM
@Code-Apprentice huh, never seen that one - there is the "You've made this webpage in Word. Should I take a screenshot and fax it back to you one" though
 
 
3 hours later…
3:29 AM
cbg, will it make sense to convert a single app.py flask project to a ipynb file?
I have to submit it for college work and I am not sure as ipynb are only for ML related works, is it worth the hassle to put different routes in cells and document or the docsctrings are enough?
 
 
2 hours later…
5:51 AM
@python_learner why don't you just submit the program, i.e. .py files, as is?
ipynb is atrocious for actual applications.
 
 
2 hours later…
7:28 AM
I will go with .py then, I tried converting and it was not good tbh
 
notebooks are only for interactive work and mainly data visualization
 
.ipynb is really only good if you are primarily shipping a document for humans, which just happens to consist also of code. If your goal is to ship code, which just happens to consist also of text for humans, use .py and liberally apply docstrings/comments.
 
Ugh, humans
 
Every task is easier without humans. :/
If a bug occurs in a forrest and no-one is there to open a ticket, does it affect QOS?
TIL: My understanding of iowait is one, big, ugly lie. I'll probably be pondering the world on a remote mountain today, i.e. room 6.
 
 
1 hour later…
9:10 AM
@roganjosh Sorry, was just playing around a bit 😅
Thought it might be interesting
 
9:51 AM
For everyone who thinks "I have this JSON" is a warning flag on questions – you should see "I have this JSON containing CSV" instead.
 
Hi guys, could i have some help with selenium here ?
 
Just ask the question into the void. People that can answer will.
 
10:13 AM
oh , nevermind, i stucked 2 days for it but somehow it's worked now :|
thanks
 
10:28 AM
@DuyĐặng all's well that ends well :)
 
As a test recently I constructed a JSON string some of whose contents were JSON strings, ... to 100 levels, and confirmed that the original data could be retrieved. I wouldn't recommend putting gratuitous line feeds in the JSON representations, but they do roundtrip.
 
Can someone help. I'm getting a RuntimeError: threads can only be started once error when I'm trying to thread. I'm using thread.join() with no success This is part of the code in question
Sorry it's a bit long (and %s part isn't my code)
count = 0
global_count = 0
thread_list = []

for line in range(number_lines):

    #prepare URL
    url = "https://%s.website.com/.json" % lines[line].rstrip()

    thread_list.append(threading.Thread(target=get_Request, args=(url, user_agents, RESPONSE_THRESHOLD,)))

    if(count == MAX_THREADS):
        for thread in thread_list:
            thread.start()
        count = 0
        for thread in thread_list:
            thread.join()

    elif (number_lines - global_count) < MAX_THREADS:
        for thread in thread_list:
 
Well, clearly you iterate over thread_list multiple times and start each thread in that list every time you do
The whole thing seems needlessly complicated; can't you just create a list of threads, then start them all, and then join them all? What do you need nested loops for?
 
I have 4000 get_Request() I need to run, and I only want to run a MAX_THREAD (maximum number of threads) at 50. If I try to just run 4000, I'm pretty sure my PC will crash
This was the best solution to the problem I could come up with.
 
Hmm, that code will run the threads in batches of 50, though. You'd probably want to start a new thread as soon as one finishes, right?
I've never used it before, but I suspect you'll want a ThreadPoolExecutor
 
10:44 AM
I want to run batches of 50 threads at a time (as far as I know that is possible). Then I need to run a new batch of 50 thereafter (I hope this is clear)
I will look into it thank you
 
If you really want to wait for the whole batch to complete before starting the next one, that's easy. Pseudocode:
for line_group in chunked(lines, 50):
    threads = [Thread() for line in line_group]

    for thread in threads:
        thread.start()

    for thread in threads:
        thread.join()
 
11:00 AM
Thank you. Looks like the pressure is off anyway. Turns out the URL that my client wanted me to use is not working anyway, so until they figure out why, my script doesn't need to be finished. I can take my time and firgure out this mess.
 
@Simon famous last words ;)
 
It's OK, I always do stuff like this. And end up panicking/regretting it last minute
 
I created a simple script for myself to handle some frontend npm dependencies in Python. I have some output being printed out to stdout, should I use a logger instead? Would it be considered best practice in general to use a logger instead of print statements? Or don't bother? This is not an opens source script so I don't envision it being used by anybody else.
 
11:21 AM
I ended up figuring it out. It was much easier than that
I just need to clear the list, after the thread had finished
 
you should still consider taking a look at concurrent.futures.ThreadPoolExecutor. It does practically the entire dispatch/chunk/batch stuff for you automatically.
 
I will. I also have to use concurrent if I want to get return values from the function I want to thread. That alone makes it worthwhile
 
 
1 hour later…
12:26 PM
Hi, can someone help me with
quantity1 = driver.find_elements_by_xpath("//tr[1]//td[3]//div[1]//div[1]")[0].text
quantity2 = driver.find_elements_by_xpath("//tr[1]//td[3]//div[1]//div[1]")[1].text

I got list index out of range
tried to put time.sleep(10) between them but no help
 
recbg
 
^^ i can only run 1 of them , if i try to run both it would return that error
 
Dev
12:51 PM
Hello Everyone, looking for some guidance around the issue with decompress .gz files in S3 and upload them back with multiprocessing.
Posted code and details here, https://stackoverflow.com/questions/63293669/issue-with-decompress-gz-files-in-s3-and-upload-them-back-with-multiprocessing
I will appreciate any insights into it.
 
1:17 PM
Sounds like you're getting race conditions. Perhaps you can narrow down which methods might be causing problems by putting Locks around likely culprits. Anything involving writing or uploading data is a good place to start.
 
Dev
Thanks Kevin. I narrowed it down and getting error during upload
client.upload_fileobj(Fileobj=Fileobj,Bucket=bucket_name,Key=uncomp_keyString)
 
OSError: Not a gzipped file (b'\x05i') is raised if the first two bytes of the gzip file are anything other than b'\x1f\x8b. It makes me suspicious that the two gzip files are overwriting one another at some point
 
Dev
@Kevin I suspect that too because if I run in serially, I have no issues.
Wondering how can I solve this problem with multiprocessing
 
stackoverflow.com/questions/52820971/… suggests that some parts of boto3 are not thread safe. Having never used the library, I can't give much practical advice beyond "try the code in these answers"
The third answer seems fairly simple... Create an individual client for each thread, inside a lock so they don't trip over one another
The file uploading itself will be outside the lock so hopefully you'll still reap the benefits of parallelism
 
Dev
Thanks @Kevin This is helpful. Will give it a try
 
 
6 hours later…
7:07 PM
(Hope memes (every once in a blue moon) are safe here? 👀)
 
@Lapys no
@Lapys Especially lame ones. From non-regulars who don't seem to work with python. Who have already "had a bit of fun" and got chided for it.
@Lapys Posted to multiple rooms at once.
 
@Lapys yeah... let's not do that please - they're not welcome
 
@AndrasDeak Lol
Understood
 
had a little tidy up - let's not do that again thanks :)
 
Sure sure
👋
 
7:23 PM
A new low on "why not use Pandas?!". There's a missing comma in some code for SQL and now they're suggesting to import Pandas to then run SQL through. Ugh.
stackoverflow.com/q/63307795/4799172 typo. Alternatively, is there canon for this? I've seen it plenty of times, but my search results are piddly threads - I assume somewhere there is a central answer
I'll just answer it before I become too much of a curmudgeon
 
Does anyone know where i need to put the include for std::string in SWIG .i file?
 
Rob
@eminem Is this a python question?
 
Kind of. Im trying to use SWIG to wrap c++ code for python
@Rob Any change you have ideas for what i can do?
*chance
 
Rob
7:42 PM
I don't know any python sorry, just saw you posted that same question here and in the c++ channel, wanted to check you were posting in the right place
 
Yeah, I don't think I've seen SWIG mentioned here.
Which is to say, eminem probably needs C++ experts more than python experts
 
 
3 hours later…
10:39 PM
I'm curious whether there is a tangible application for Selenium that doesn't involve web scrapping?
(Even as the keys are pressed for "scrapping" I have to quell my inner self-hatred but it is apt, I think)
 
Playing web-based clicker games? Not sure if selenium can actually do that in general
 
I think it'll be the first tag I block. I'm so tired of those questions
 

« first day (3583 days earlier)      last day (1351 days later) »