« first day (3787 days earlier)      last day (1144 days later) » 

7:34 AM
Heh, those two quotes nearly back-to-back from me was something like a This is Your Life (an old UK show that puts people's life in review) moment for me @AndrasDeak :P Outgrowing Anaconda was fun
 
8:05 AM
I'm building a package for all things vehicle routing related. I now want to build an endpoint that will translate a postcode to a lat/long. I already have the data to do this in a CSV that's ~13MB compressed, 179MB uncompressed (I'm confused by that level of compression, but anyway). Say I wanted to make the package with that feature as optional if you want a "slim" install; what part of setuptools am I targeting to do this?
I can find a few examples where additional dependencies can be optional, but this would basically be a wget call on install or something
So it feels like an argparse job on pip install?
Or, of course, it might not be possible at all. It's mostly out of curiosity because none of us are really restricted on space
 
I don't think there's an way to optionally include a file. So you can either create a separate module that's responsible for postcode translation, or you can just download the file on demand at runtime
 
is what roganjosh asking something like how aiohttp offers addons?
 
Pretty much, but that mechanism can only install additional dependencies, not download a regular file for you
 
ahh I see, thanks for clarifying
 
8:20 AM
Yeah, I think what Aran-Fey has suggested is the only way to do it. Thanks for confirming :)
 
9:09 AM
The more I think about it, I wonder if there's a reason there's not a pip install -do_arbitrary_work flag. I mean, it's probably quite niche, but pretty basic and probably useful in a broader range of circumstances than my example. Maybe this has to be balanced against the security risk of people squatting on library name typos
In fact, I think I've answered my own question, though eval exists, as does the security hole on pickle. A rm -rf job on install could be fun, though. You'd never typo again :P
 
9:27 AM
@AndrewLiu Belated welcome to the room :) I don't use repl.it. Anecdotal stuff; a number of people have brought problems to this room using repl.it and by the time someone opens it up to try help (case in point, some replies can come hours later) the whole thing is wiped or entirely different to what it was at the time of the question. This alone really puts me off using it
If you want to collaborate on code, VSCode handles this in a not-awful way. Not the highest accolade for a feature, but there's always going to be a headache when multiple people are editing code
 
Hi. Sorry I don't really know where I am. I'm stocked failing to deploy my flask app on heroku for days now. Just looking for any sort of help
 
Hello :) If you mean you don't know where you are as-in this chat room, we have a community website (powered by flask btw) to explain. If you mean in terms of your code, have you been following the Flask Mega Tutorial?
 
9:42 AM
Yes I meant on this chat room and no I'm not following the that tutorial
 
I highly recommend that tutorial, though the part I linked is quite late-stage so it might be a bit of a pain to adapt your app to it
 
I'd just like to add that my "problem" is currently on satckoverflow and it's more about connections with aws s3 than flask itself
 
I have seen your question, and it's confusing tbh. I suggest removing the flask context entirely, in that case
 
@roganjosh sounds like a job for namespace packages
 
@roganjosh you mean remove the flask tag?
 
9:48 AM
At the company I work at, we have a wrapper library that just handles all the boto3 stuff for us. I once suggested I needed something that wasn't handled by that library and the maintainers came down on me and suggested a hacky way to make it work just to avoid having to deal with boto3 so... anecdotally, it's not a fun library to work with
 
@AlinoIkonicBuddy yes. If you're having issues with S3 access, it's tangential that it stops your app from launching. You'd be having the same problems even if it wasn't part of a flask app
I'd go further than just removing the flask tag, I'd remove the flask component from your code (probably). This is part of making a MCVE
 
I just to able to upload a file, read it and do some ntlk stuff. I thought aws s3 was the best-easy way
 
@MisterMiyagi That preempted the question I'll probably have later today :)
@AlinoIkonicBuddy depends how big the file is, I guess. You might just be able to process it in memory from an upload
 
I have a datetime column in a dataframe. In this column, which format is Y:M:D H:M:S, the second is equal to zero for all records.

for examply, 2017:09:01 04:24:00.

According to the number of repetition of minute it can be guessed the time interval for example:
4:24:00
4:24:00
4:24:00
So, as the minute 24 repeated 3 times we can guess each 20s data are gathered. Although, this 20s time interval is not equal and for some minutes it is 6s or 5s.
So, how we can I add this time interval (s) to secondÉ
 
10:00 AM
@AlinoIkonicBuddy That edit has made the question totally unanswerable. Showing server logs with no code that's actually crashing is not helpful
 
Omg ok
I should let the same code their?
 
If the gunicorn worker fails to boot, you want to be showing the code that is crashing, not the logs
 
@asadyarahmadi I think you're looking for resample with interpolate?
 
@roganjosh I checked the resample methods, even I raised the question but unfortunately, I found no solution.
 
10:05 AM
There are a number of steps you need to take to understand this problem:
1) Remove `gunicorn` from your setup and launch the app on the flask development server
2) Set `debug=True`
3) Follow the traceback to find the problem code that's crashing
4) Isolate the process that this code is doing and test it outside of a Flask app
5) Post the result of that analysis as the body of your question so that other people could repeat it (even if only in their head)
Huh, I broke backticks
@asadyarahmadi You'll need to be more specific sorry. Please give a MCVE. If it's quite a long snippet of code, please be mindful of the room rules and link to the code hosted off-site
 
@roganjosh Here, I provided more description.
 
If you had read the room rules I linked, you'd see that we ask that people don't post questions here that are less than 48 hours old on main
That question was marked as a dupe and got an answer. It could also have been closed because you just included screenshots of the table, meaning that anyone who wanted to help would have to build their own example from scratch. That's not a good way to get help
Heading out for a bit, back later
 
@roganjosh Thanks, just I looked for more help. Sorry, as it violated the considered rules.
 
10:43 AM
@roganjosh jump at the chance and add async as well. That's an instant 20% more awesome for practically* no work! *Disclaimer: Practically practical practicality practically estimated.
 
 
1 hour later…
11:58 AM
@MisterMiyagi Oh man, now I have an idea. The geofencing module can be stackenblocken!
Nobody will really appreciate it other than me at work, but that's the only thing that counts :P
*h not k. It would be helpful if I could spell my own module name :P
 
ok, i'm done answering questions by new users. 3rd time in 2 days that they've deleted their question after getting an answer. and someone had given their question an upvote - so it's not like they were pressured to do it
 
Chances are that they should have been closed as dupes and not answered anyway, no?
 
If they're good questions that are worth having around, I'm sure we can rally enough people to get them undeleted
 
There's a whole heap of presumption on my side with that statement, but I've spent an awful lot of time on the main feed
 
12:13 PM
@roganjosh wasn't a dupe. had enough difference to warrant being answered
@Aran-Fey nah, not that good a question to worth undeleting
 
In which case - what Aran said. If you think one should be resurrected then maybe ask here for help in that
Can you link an example? (not necessarily with the intention of calling for it to be undeleted, just so others can see the type of question [rep permitting])
 
yup, got it. but doesn't need a forced un-deletion
there was another question earlier which got downvoted, wasnt a bad question, just trigger happy fast viewers. i felt bad for the guy. and i was in the process of answering it when it happened
@roganjosh for the current one: stackoverflow.com/questions/66398172/…
 
In that case, it was a typo. You don't need == in SQL, and pulling everything only to then filter in pandas isn't great advice btw
 
@aneroid for the one where it wasn't a bad question, just incorrect code (which is 50% of SO questions) - stackoverflow.com/posts/66395704
@roganjosh it was a pandas exercise. ie, he was probably expected to do the filtering in pandas. (yeah, huge tables shouldn't be loaded into pandas but once the "smaller subset of data" is queries, the analysis is better done in pandas rather than writing crazy complex queries). and he commented that my solution worked. i didn't notice the == in SQL.
 
I can see why you'd be frustrated with that being deleted as it's clear that you put effort in there, but I would have closed that question as a typo. My previous statement is more for your benefit.
 
12:21 PM
thanks, i get your point. i didn't see the typo though, must've glossed over it
does pandas not raise the SQL error if the query was wrong? coz the error message he posted was a pandas error, not an SQL syntax error
 
I never execute SQL via pandas so I don't know sorry
You could probably test easily enough with SQLite if you wanted to know. It might not even be a syntax error in SQL (I haven't tested) but it definitely isn't necessary and I think it is an error
 
whenever i have, it's been large enough that i need to chunk the results anyway so I load it via append rather than directly from the cursor. so i guess i too have never executed SQL via pandas..
 
Oof, how do you append to your df?
 
query data loaded as list of namedtuples, then to df
 
The df is going to be in memory anyway, so I'm not sure what you're buying yourself by chunking, and append forces a copy of the whole df each time
 
12:25 PM
coz querying 10,000 rows from a server across the world takes 30 mins but if i chunk it in 300's; it gets done in 7 mins
 
:O
 
and that's a summary query i wrote which executes on the server. the base data is 1M+ rows
 
Querying 10K rows takes sub-seconds or at least close to that. That's an insane network latency (if that really is the cause!)!
 
yes, it wa network issues - and the way the admins have set the buffer size on the db side. so if i tries to send me 10k rows in one shot, it gets massively buffered
@roganjosh (maybe it was 30K rows, but same issue; this was 5+ yrs ago so i don't recall)
 
@aneroid gross!
 
12:28 PM
i tested with different buffer sizes to select 300. i recall that being a loooong (boring)10 hours...
 
10K rows or 30k, it's in the same ballpark. I'm surprised they'd want to buffer that
 
to make matters worse, the "summary query" itself takes 3 mins to run when directly on the server :-/ so awesome hardware there. later, an "oracle veteran pro" with 20 yrs of experience, joined for a related project. took a look at it - had 0 improvements to suggest.
if I never work with Quality Center again, it will be too soon.
 
"veteran pro" mind if I use this to refer other "veteran pro" at workplaces I might meet ? :P that is a word I did not know I needed
 
Never heard of Quality Center. From Wiki: "Micro Focus Quality Center runs on the Windows platforms with an Internet Explorer browser." No point reading on from that point!
 
yup, that's the one. but it's been bought by HP 10-15 yrs ago. it's been rebranded a couple of times. HPQC, HP ALM (application lifecycle management)
 
12:52 PM
@roganjosh I'm now able to "see" that question, even though it's still deleted. (and i'm not 10k yet)
 
The second one you linked, I can't see it either. I guess a mod took it down (no idea)
 
the first one i mean - is now visible to me - stackoverflow.com/questions/66398172/…
 
Refresh the window? I don't think you should be able to see it, but I can't remember off the top of my head whether the fact you answered is significant. I don't think so.
If you have the tab open at the time of deletion, I think it's an AJAX update so you can still see the Q/A in a faded state. It's gone once you refresh
 
yeah, previously it was faded, so that's when i linked it here. but it's totally visible/un-faded but all 'reddened'. even did a shift+reload
anyhoo, doesn't matter now. probably just a js/ajax error. can't do any actions
 
Interesting. Does this link work for you?
 
12:58 PM
@roganjosh yup. and i see the typo in his query. lol
 
That doesn't make sense, but ok :P Maybe I misremember how it works when you're <10K rep but posted an answer
 
i should be getting the "page not found" error, right?
 
You'd normally get a "this question has since been deleted by the user" (paraphrased) message
 
i know that if i look at it from my history, then i can see it...maybe that's the reason - coz i'd answered it nope, it's not in my history.
if i have a link to the question and have answered it, then even after deletion, i can see it. for the more interesting questions, i save the link with the code i wrote for the solution
"viewable after deletion" was a side-effect
 
 
2 hours later…
2:34 PM
with self.locks[first]:
    with self.locks[second]:
how does this bit work?
I dont see how the locks are being acquired and released?
 
> Locks also support the context management protocol.
 
so with = try...except?
 
Umm... with can catch exceptions, so yeah, I guess?
 
i still dont see with can do acquire or release?
 
It can do both. It acquires the lock when the with block is entered, and releases the lock when the with block is exited
 
2:42 PM
magical
Acquire a lock, blocking or non-blocking.
under acquire()
why would you want a nonblocking lock?
 
no clue
 
seems like opposites
 
that if philosopher != 0 block is confusing
 
In computer science, the dining philosophers problem is an example problem often used in concurrent algorithm design to illustrate synchronization issues and techniques for resolving them. It was originally formulated in 1965 by Edsger Dijkstra as a student exam exercise, presented in terms of computers competing for access to tape drive peripherals. Soon after, Tony Hoare gave the problem its present formulation. == Problem statement == Five silent philosophers sit at a round table with bowls of spaghetti. Forks are placed between each pair of adjacent philosophers. Each philosopher mu...
 
Yes?
 
2:55 PM
resource hierachy solution I think if what its doing here
section in that wiki
@AndrasDeak linked to this
 
OK?
 
that whole code is confusing
 
yeah i dont understand the if else statements function now
 
@Trajan well it's probably doing the resource hierarchy solution
 
ive drawn out some diagrams and still confused
ill draw them out again
 
3:03 PM
@Aran-Fey the only thing better than "random questions about random coding challenge problems" is "random questions about some random guy's solution to a random coding challenge problem"
 
3:13 PM
So much use of "random" outside of the context of non-determinism <cue cartoon robot meltdown>
 
Trajan's questions are probably closer to true randomness than a PRNG, for what it's worth
 
lol
 
after all human consciousness is pretty quantum
 
3:35 PM
@AndrasDeak does this potentially answer my question?
 
well the other alternative is Paul Panzer so I'll go with caching
 
It does kinda make sense if the slicing allows the array to fit into L1/L2 cache and I do a lot of hand-waving and pretending I know what I'm talking about
 
Well yeah, pretty much all vectorization speed-up is due to optimal cache use. There are also things like vectorized CPU instructions but I'm not sure how common those are. I definitely see them mentioned on the mailing list and PRs but I don't know how well numpy can make use of them.
 
What's the test for this? I should see a step-change in runtime with 64,000 elements and 64,001?
or is 64k a ballpark figure do we know?
 
There are probably more moving parts for that. I don't know. But this is the basic rule of thumb that contiguous arrays are faster to work with. Copies are contiguous. arr[:, 25:75] is not contiguous.
I'd have to understand how CPUs work and I don't. I only know the hand-waving "if the memory comes from one place it'll probably be faster"
considering PNumPy 64k is probably a good guess
I didn't read your question in detail now (I did upvote it at one point), but it reminded me of this:
In [57]: arr = np.arange(2000_000).reshape(1000, 2000)[:, ::2]
    ...: arr_c = arr.copy()
    ...: %timeit arr.sum(-1)
    ...: %timeit arr_c.sum(-1)
1.08 ms ± 154 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
639 µs ± 4.05 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
 
3:51 PM
...wow, ok.
 
In [58]: %timeit arr + 1
    ...: %timeit arr_c + 1
1.95 ms ± 96.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
1.31 ms ± 14.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
 
It's gotta be overhead on strides then?
 
very often a single .copy() call can make a numpy-heavy function faster
@roganjosh yeah
 
Neat, thanks! That's super-useful knowledge!
 
@roganjosh Not necessarily literal overhead, just that if the elements have to be picked out from memory the CPU won't be able to shove it all into its cache. That's my lies-to-children level understanding.
 
3:53 PM
The observation itself is enough for me to make use of even if our collective understanding is very high-level
 
Yeah, I don't need a better understanding because at the end of the day I'll just profile my code and see if copying makes it better or worse.
(Because making copies has its own obvious overhead)
 
Well that was the double-whammy for me, because I was sucking up the cost of a copy and still winning
 
@roganjosh one important thing which is easy to understand if you think about row-major (C) order is that slicing with step=1 along the first dimension will keep your array contiguous. Other slices will break contiguity.
obviously for column-major (fortran) order it's the other way around: last dimension can be contiguously sliced
 
I did understand the differences between C and F, but I've clearly underestimated the overhead in traversing slices of either array
In other words, I thought the np.where issue I had was an isolated quirk. It's clearly not from the example you've given, which is far more valuable than the actual answer on that question
 
well...
 
4:15 PM
Did you create that example from memory or do you have the actual question? I'd like to review the answer there (if there is one) because we could potentially dupe my question for something better than "I've eyeballed it and it's a longer code path because the file is bigger"
 
4:36 PM
hello is anyone online ?
 
Nope
What's up? :)
 
Hi everyone, does anyone know how to debug a cron job(executing a python script) where the job is starting fine but is executing the script only till a point?
 
Hi, are you familiar with pytorch ?
 
@MayowaPaul I'm not, but that doesn't necessarily mean we can't help. However, I note that you asked a question on main less than 48 hours ago and we ask in the room rules that you wait at least that long before bringing the same question here. Is it the same question?
@RaphX Have you ensured that the script runs fine outside of the cron call?
 
I have a coordinate:
>>> c = (-1, 0)
 
4:40 PM
@rog
 
And I want the value and index of the non-0 value:
>>> v = max(c, key=abs)
>>> i = c.index(v)
Or:
>>> v, i = max(enumerate(c), key=lambda x: abs(x[1]))
 
@roganjosh yes, but with some additional things as well
 
Or I can swap out the abs for bool.
 
@roganjosh yeah it runs perfectly when I execute in the following way: 'python <script_name>.py'
 
Is there a better way of doing it?
 
4:42 PM
How do I reshape a rank 1 tensor to rank 3 using the view() function ?
 
There will always be one 0 and one of either -1 or 1 in the coordinate.
 
rank1 = torch.arange(24). This creates a tensor of 1rank
 
@RaphX I'm gonna guess that it's a permission error - does the script have to access other files?
@MayowaPaul what have you tried?
 
yeah it has to access some csv files @roganjosh
 
@AnnZen Oops, I meant
>>> i, v = max(enumerate(c), key=lambda x: abs(x[1]))
 
4:46 PM
Ok, and have you looked into how to redirect stdout from crontab to a file @RaphX?
@AnnZen open question to others too: I suspect this lambda defeats any speed increase from using a builtin (max) and it might as well just be a for loop?
 
@roganjosh rank3 = rank1.view(2, 2, 3)
 
And what went wrong with that?
 
@roganjosh I get this output error
RuntimeError: shape '[2, 2, 3]' is invalid for input of size 24
 
@roganjosh Oh. Is a for loop my only best option?
 
@MayowaPaul I think its because 2 * 2 * 3 is not equal to 24
@roganjosh I am redirecting the stdout to a log file , its showing output of only some print lines at the beginning there
 
4:52 PM
thx
 
@AnnZen That's why I've left it an open question to others :) I don't think you're gonna win once you throw the lambda in
@MayowaPaul that's not specific to torch. You can't do that in numpy either. What structure do you want?
@RaphX it should also show any traceback?
 
[[ 0, 1, 2, 3, 12, 13, 14, 15],
[ 4, 5, 6, 7, 16, 17, 18, 19],
[ 8, 9, 10, 11, 20, 21, 22, 23]])
@roganjosh
 
Well, that's 2D, so why are you specifying 3 dimensions?
 
sorry let me rephrase my question properly
 
Maybe my linear algebra fails me, but I don't understand that output @MayowaPaul
 
4:57 PM
right now I have a tensor that looks like this @roganjosh
 
in fact, I'm almost sure that isn't any kind of standard transformation
 
tensor([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18, 19, 20, 21, 22, 23]) @roganjosh
 
You don't need to keep pinging me btw. I'm the only person replying to 3 people
 
([[ 0, 1, 2, 3, 12, 13, 14, 15],
[ 4, 5, 6, 7, 16, 17, 18, 19],
[ 8, 9, 10, 11, 20, 21, 22, 23]]) I want the order of elements to look like this
 
But that's a 2D array, with some kind of mix of elements
 
4:59 PM
don't see any traceback in the log file that I am creating, do I need to do something specific? in the past when they failed the log files were not updated which is how I used to debug
 
Yet you were specifying 3 dimensions. So, from the start, your approach couldn't work. But I don't know a clean way of getting what you want, @MayowaPaul
 
its no problem
 
Is there a name for the particular transform you're trying to do?
@RaphX did you look at this?
I can't say I've had a cron job that prints to an output file but won't print a traceback. It could be that the process hangs waiting on some HTTP request or... well, anything that ties the process up. Or, I'm missing something
 
I didn't check that although I came across it as the log file shows the latest date from which I thought that the cron job started but didn't go till the end as 60% of the print statements were not getting logged
should I still check that?
 
Check what?
 
5:10 PM
the link that you shared with me
 
... I didn't link it for fun...
 
cool, let me try out the options listed there then, thanks
 
5:24 PM
I'm now imagining an easy way for your script to fail and not log, @RaphX. Do you log anything on exceptions?
Based on our history, I imagine there is a lot of:
except:
    pass
In which case, you can import traceback and then use print(traceback.format_exc()) on each exception
 
5:45 PM
thanks! I will try this as well @roganjosh
 
 
2 hours later…
7:42 PM
@MayowaPaul that's more than a reshape
 
Is there a better way of doing:
all(a == b for a, b in zip(sorted(a_list), sorted(b_list)))
 
you need to reshape to 3d, permute some of the axes, then reshape back to 2d
@AnnZen Depends. Is that code buggy?
If that code is buggy then you want set(a_list) == set(b_list)
 
@AndrasDeak but what if a_list contains duplicates?
 
@AnnZen try and see
Don't expect us to fix your vague questions and convince you about them.
Aug 20 '19 at 9:35, by PM 2Ring
Imagine that you won a competition, and the prize is a free session with a Python think-tank who normally charge $1000 per hour. Don't waste that prize!
 
It doesn't work:
 
7:45 PM
@AnnZen consider ^ that, and when you ask here be more efficient
 
>>> a = [(0, 0), (0, 0)]
>>> b = [(0, 0)]
>>> set(a) == set(b)
True
 
@AnnZen indeed. But what if b_list = a_list + [float('nan')]?
 
The right match should be
>>> a = [(0, 0), (0, 0)]
>>> b = [(0, 0), (0, 0)]
@AndrasDeak I get False
@AndrasDeak ohk
 
@AnnZen and do you want to get False?
 
I want to make it so that the...
sorted list a and sorted list b...
 
7:52 PM
Stop
 
@roganjosh huh?
 
There's absolutely no reason not to post this in full sentences
 
I'm worried that i'll be too slow :(
 
There is no race
 
@AndrasDeak <<< just considering
I want to make it so that the sorted list a and sorted list b aligns perfectly like two identical lists.
lists of (x, y) tuples
 
8:00 PM
So you're trying to say "I want to test if the two lists are each other's permutation".
 
oh yeah...
thanks!
 
In which case your code is buggy. But no, it won't be any better than what you have I think.
I guess you could use collections.Counter and consume both lists, but that would assume that all values in both lists are hashable.
@AnnZen but you lied
In [74]: a_list = [2, 42, -1]
    ...: b_list = a_list + [float('nan')]
    ...: all(a == b for a, b in zip(sorted(a_list), sorted(b_list)))
Out[74]: True
 
AAB
8:55 PM
hi all,
In Django I have some views that I don't want the user to access and be redirected to the default page.
I am custom views(django portal disabled)
Is this decorator necessary? can't we just have a function that takes user object as a function argument and returns the path based on if he is authenticated or not?
 
9:14 PM
@AndrasDeak I meant with the set option.
 
I see...
 
9:41 PM
@AnnZen but sets are unordered. When Andras mentioned "more efficient" in an earlier comment, it doesn't just mean "post faster". It means being more direct/specific with requirements so that people know exactly what you want as an output from your input. It's not helpful for anyone otherwise
I can understand how you misread the comment, though
 

« first day (3787 days earlier)      last day (1144 days later) »