« first day (2950 days earlier)      last day (1995 days later) » 

12:39 AM
@jpp OK, I mentioned the np.array "constructor" in the tl;dr. Guess it can't hurt.
 
wim
12:55 AM
>>> s = 'a=1|b=2|c=3|d'
>>> dict(x.partition('=')[0::2] for x in s.split('|'))
{'a': '1', 'b': '2', 'c': '3', 'd': ''}
 
jpp
@AndrasDeak, Yep, I saw that, deleted my comment :)
 
wim
any cooler way to do that?
I don't like the slice thing much
 
jpp
@wim, I'm guessing you don't want str.replace + ast.literal_eval ?
 
wim
of course not
 
jpp
actually that would be messy with that lone d.
of course i'll let you wait for other answers, good luck!
 
1:00 AM
can pep-8 please be revised to specify that inline comments should begin with a capital letter and end in a period?
 
# This is a comment. versus # this is an ew comment
 
no, ew the other way around
 
why are you like this
:P
 
>>> {k:v for x in s.split('|') for k,_,v in [x.partition('=')]}
{'a': '1', 'b': '2', 'c': '3', 'd': ''}
no slicing :P
also, ew
also also, rhubarb
 
1:40 AM
Is there a way to specify render order for Jinja2 blocks?
 
 
1 hour later…
wim
2:51 AM
@AndrasDeak even worse
 
 
3 hours later…
5:30 AM
@jdehesa well looking forward.
 
 
4 hours later…
9:32 AM
is it just me, or did this user's question just get an unnaturally large number of upvotes within minutes of posting? Maybe it's just my paranoia but I smell something fishy...
 
@coldspeed flag it and leave it to the mods?
also, Tuesday cabbage :)
 
(Already done)
just wanted to make sure my suspicions weren't unfounded :)
but yeah, not much else to do besides flag
 
@coldspeed become a mod :-p
 
@coldspeed you may or may not be right but discussing this with the suspect is pointless at best and harmful at worst
 
9:58 AM
Cbg
 
cbg, Professor!
how are things? :)
 
@JRichardSnape cbg!
 
10:27 AM
cbgh
 
10:48 AM
@JRichardSnape cbg severus
 
cbg-ning
kind of urgency
I've made some devs, git add the files, did a commit and push it to a new branch
however, now my devs are gone from the branch
my colleague did a fetch from my branch and nada
any ideas of what might have happened?
 
11:15 AM
do a git log on the branch
and see whats in there
 
@marxin thanks, we're good
things have been settle
 
 
2 hours later…
1:13 PM
cbg
 
cbg
 
cbg
 
2:21 PM
I solved my problem reading two serial ports in separate threads. Looks like the multiprocessing was a better solution for high loads so it can read from both ports without interfering with each other
I had to share it even after one month :)
 
I would've bet that AD would be the one to derail the cbg train
 
2:35 PM
I'm hurt
 
uh oh
 
2:53 PM
I decided I'd be smarter with CSV file processing and use csv.Sniffer to detect the dialect. Did you guys know that if you sniff the header of the format something\r\n (i.e., the file has just one column), it'll consider \r a separator? So in the end you get two columns, and one of them is 100% empty
Thank you Python, very cool
 
\o cbg
 
@vaultah blame windows ;)
 
@AndrasDeak the best part is: I'm using newlines='' which is recommended by the documentation and is supposed to solve the issue with line endings
So it's entirely CSV's fault
 
3:08 PM
Hehe
Does newlines='\r\n' work?
And what does it use by default if not ''? :D Not the sniffer?
 
3:25 PM
Hmm maybe the newline argument plays no role here... It's set for the file object, and in my example CSV functions only receive the string
As I see it, the issue is that Sniffer thinks \r is a sensible delimiter
 
4:01 PM
Seen on reddit /r/python a few weeks ago:
> "Though if you don't like writhing async code, you might not enjoy using it."
 
wim
strangely appropriate for async/await coding in Python
 
 
1 hour later…
5:26 PM
@RobertGrant Cbg @Antti, @Andras and Bob. Sorry - as soon as I made my brief appearance here, students arrived. Still at least I have converted two of them to using Python for their project, so time was not entirely wasted.
I shall return tomorrow, refreshed.
 
time well spent!
 
:) I hope so
 
Forgive my ignorance. I am having trouble deciphering a formula that describes a number sequence. The given formula is as follows:

sum(k>=0, x^((k^2+k)/2) / prod(j=1..k, 1-x^j)) – 1/(1-x)

I am trying to utilize this formula in python code but cannot because I don’t understand the finer points of the above expression.

My interpretation is that if I want the tenth number in this series I should insert 10 for x but then what is k? I get that j is a list of integers from one to k but I can’t decide what to use for k.
 
@JRichardSnape :D well good to see that you're alive :D
 
5:43 PM
Yes - I battle on. Until tomorrow at least :D
 
@W.Dodge sum(k>=0 might be an infinite summation? That is, "The sum of all values from k=0 to infinity"
Hopefully not since Python can't sum an infinite number of values in a reasonable amount of time
With any luck the value converges fairly quickly and you can stop at k=100 or whatever
 
Ok, that makes sense. The formula describes "Number of partitions of n into at least two distinct parts" sequence found here oeis.org/…
 
cbg
 
Maybe if you plugged that mathematica section into Mathematica, it would render the formula in a human-readable way, with nice formatted sigmas and such? I would try it, but I don't have Mathematica.
 
If given a number, there x number of combinations of incrementally smaller numbers that add up the the given number, without containing the given number or any duplicates. That is what the series represents. So 5 becomes 2 and 6 becomes 3.
 
5:54 PM
The OEIS page links to arxiv.org/abs/1308.4945, which contains some formulas, but not ones that resemble what I had in mind
Actually reading the PDF and comprehending which formula matches the thing we're looking for is way out of the question
 
hoho
I guess you can change the variable from 'x' to 'c'. 2c=20c+3->-18c=3->-c=1/6->c=-1/6Christian Gibbons 1 min ago
 
Okay, thanks Kevin. I'll try the Mathematica approach. Eventually I will get the implementation right and then I'll understand how to decipher these types of expressions. I think it is probably simple once you see what is what.
 
(OP asks how to solve 2x = 20x + 3 with c :D)
 
6:07 PM
I know some of y'all have o p i n i o n s on Ansible, but look at Michael DeHaan's follow up to it: medium.com/@michaeldehaan/…
> embraces python 3, uses python as the DSL (no YAML). A type/provider separation. Type checking. Emphasis on performance.
 
@JRichardSnape \o/
 
wim
@KevinMGranger interesting, thanks!
if it's so modern, why is he still using distutils and a Makefile, LOL.
 
Hello
I have an 'algorithm' question
I have a bunch of files (thousands) that I want to distribute into multiple folders
but I want them to be organized in such a way that I can easily find them later on
this is necessary because you can't have more than 64K files per folder in Windows
is this a common problem? are there well known solutions for it?
 
When you say you want to find the files easily, does that mean a computer should find the file or a human?
 
My intuition is that I should organize the directory as a balance BST, but i'm not sure if that's a good idea
 
6:18 PM
Lazy solution: Create folders "A" through "Z", and place each file into a folder in accordance with the first letter of its name.
 
The thing is I'd like the folders to be balanced. For example, if there are 65K files starting with "A", how would I deal with this?
 
@wim makefiles rock!
even warehouse is using a makefile
 
Within folder "A", create folders "A" through "Z", and place each file into a folder in accordance with the second letter of its name.
 
Perhaps using a balanced BST as the folder structure and splitting folders when it nears 64K files is a good idea?
oh
 
Following this idea to its extreme, this is a radix tree
 
6:21 PM
@KevinMGranger I liked it all the way up to
> There’s very strong object-orientation, where everything is subclassable. You can provide your own implementations for system types, or write your own types. You can subclass literally every piece of the API.
 
@Kevin, thanks for the idea. Is this a realistic (sane) solution given that the nodes will be folders?
so when the nodes are being split, thousands of files will have to be moved, etc?
 
> the quality of the base class API for when we start taking pull requests sets the standard for everything.
 
Well, I expect you'll be moving thousands of files no matter what directory structure you come up with
 
Or maybe you're saying "won't I have to move thousands of files thousands of times, since every new insertion into the tree will change its structure, possibly to a very deep level?". I have a feeling that in the average case, any single insertion is reasonably fast. That said, it might be a good idea to calculate the final folder structure first, and only then start creating folders.
 
6:27 PM
> The application will CD into the path specified, so all paths referenced will be relative to there. You can see we have a "files/" and "templates/" subdirectory in that same directory but there is no enforced convention.
@KevinMGranger arghh
 
@Kevin yeah, but I will be inserting an unknown number of files, so the structure is hard to know to begin with
 
@J.L.Louis Are you sure about this? superuser.com/questions/446282/… would lead me to believe that NTFS allows upwards of four million files per folder.
FAT32 only allows 65,534 files, but if I understand correctly, FAT32 hasn't been used by Windows for a long time
 
oh
 
fat32 is being used for all external storage almost everywhere :d
 
yeah
I think that's the problem
external storage (not the system's drive)
 
6:33 PM
but even then, usually for laaarge amounts of files, it is common to organize directories based on some hashes...
makes navigating easier :D
 
I feel that there should be a common solution to this, since I presume everyone working with lots of files comes across this issue eventually?
@AnttiHaapala, could you point me to some resource that explains this?
 
I don't suppose putting everything in a zip archive would solve the problem?
 
files are being inserted and accessed all the time, zipping them wouldn't be practical I think
 
I just created a directory and put 70,000 empty text files into it. Windows 10 seems to have no problem with this. I think we all agreed that this would be the case, but I may as well say it for the record.
 
6:36 PM
I would use 2 hex digit branching, perhaps, though...
 
@Kevin, I can't assume that the file system is NTFS
The software will run on Windows, but the drive will be accessible by any client. So I guess they're formatted as FAT32 to keep it neutral
 
Ok. Just making sure we're all working from the same factual basis :-)
 
alright :)
 
The post Antti linked to suggests a solution pretty similar to my radix tree idea. As an added bonus, you can reasonably expect the files to be mostly evenly distributed among the folders, even if all of the filenames are very similar. Good hashing algorithms are uniform in this way, and md5 is generally considered a good hashing algorithm. (not to be confused with a good cryptographic hashing algorithm)
 
yes
avalanche
 
6:44 PM
Luckily we don't give a fig about cryptography for this problem
 
recbg
 
@Kevin, right :)
 
7:04 PM
I'm half tempted to write an example implementation but I'm so frazzled by my current work project I don't know if I can look at a text editor for more than 45 seconds in a row right now
 
No worries :)
More constraints: I can't use a database :D
So solutions involving hashing are a no-go since I won't be able to do a look-up
 
The database isn't strictly necessary. I think that's just there so you don't have to run md5 on a filename more than once ever. But md5 isn't that expensive, so whatever.
Not having hashing means that the tree will be more lopsided, and that rebalancing will be more common, especially if you have a lot of files with similar names. But it's still possible without it.
On second read, I'm not sure whether you're saying "I can't use a database, and I assume I can only perform a hash using a database, therefore I can't use a hash either", or "I can't use a database, and I can't use hashing, even though I recognize that hashing can be done without a database"
If it's the first one, good news! You can do md5 right within Python with the built-in hashlib module, no databases required.
The reply I'm dreading is, 'actually, I can't use hashing because I'm sharing this external storage device with other people, some of whom have no programming experience. We have to maintain the tree manually, and if the instructions are more complicated than a two-line README file, then most likely Karen from Accounting is just going to throw her hands up and put all her stuff in a directory named "Karen's stuff"'
 
7:39 PM
yep, the latter :)
Karen will need to know how to find the files manually in an apocalyptic scenario
But maybe she can just use Windows explorer search? lol
 
do you guys know how to make python use more cpu resources in a multi core environment?
 
The multiprocessing module is pretty good at that
 
i'm training an ner model using spacy and I tested my script on a single core Xeon server and then a 10 core Xeon server and the 10 core is running about the same speed as the single core server
 
7:55 PM
Based on your previous interactions here: you're probably doing it wrong. MCVE please.
 
um no because the script was provided by spacy developers so they would be doing it wrong if anything
 
Do the developers claim that their program leverages multiple cores? Because in the absence of any promises like that, I assume that any given piece of Python code uses exactly one core
 
no they don't and I'm well aware of that
 
of course we already agreed yesterday that things can't be your fault
yesterday, by erotavlas
its not me its python
probably that pesky python again
 
7:59 PM
ok so you guys are being hostile now instead of helping?
 
only me
and I can help if you provide an MCVE which I already asked for
 
that was a joke...you know "its not you its me" when people break up
 
I'm not trying to be hostile, I just want to ask questions and make observations. Blessed neutrality.
IIRC there are some programming languages that use some extremely clever optimizations so that ordinary single-threaded code can be made faster by employing more than one core. But CPython is not one of those languages.
 
@AndrasDeak and fyi I was asking question about sinple strings yesterday which don't really require an MCVE
 
sure they don't
 
8:00 PM
the string itself is its own mcve
 
yesterday, by Andras Deak
Yeah, that doesn't mean anything. Good luck.
The thing with parallelization is that it's not magic pixie dust you can sprinkle on your code. You need to find subtasks the code can perform in parallel and then write code that handles doing that parallelization is some way
 
I like to have MCVEs for even very simple problems. The only thing I can trust in this world is the output my REPL produces.
 
ok so in the code linked above perhaps the part that does the 'minibatch' can be parallelized
 
Hmm, unsure. the for batch in batches: loop makes a reference to a single nlp object, which is updated each time. Having shared state like that is a potential indicator that parallelization may be difficult or impossible.
 
yeah it needs to keep updating that object
 
8:08 PM
... Which is not to say that all loops that mutate an object can't be parallelized. For example, seq = []; for x in y: seq.append(f(x)) can be parallelized effectively, doubly so if you don't care about the final order of the list.
 
I have a pandas DF with timestamp as index. I need to plot most of the parameters against timestamp, with auto y-scaling (because ranges are varied), and select/deselect y-variables as required.
Is there an existing code that can do this, or I need to write / modify my own. I went through the matplotlib examples, but couldn't find a very convincing match.
 
"most of the parameters", "select y-variables as required". That's quite vague
df.plot should handle the simplest use cases
 
Let's switch it to 'all the paremeters/columns/series' and be able to deselect parameters by clicking on their names in a 'legend'
 
Oh, interactively? That's not easy I think
 
Something I can handle in Tableau quite easily. I could switch to that for now, if coding it is time consuming.
 
8:22 PM
there's machinery in matplotlib to handle events like that, but that's a bit more esoteric than basic plotting
 
I was fishing for an existing piece of code
 
yeah, I don't know about that
 
Thanks
it's like viewing data change with time (index). And choosing y-parameters against time
 
8:38 PM
I've heard that function calling is expensive in Python. Admittedly, I've not spent any time with a profiler. Question: are there any savings made when using map? Meaning do we get to spare the time of calling the function x - 1 number of times?
 
how would using map spare that?
 
wim
no
 
i don't know. My guess was that maybe it gets called once in some mysterious C layer and awaits inputs.. or something. That's why I asked (-:
Thanks @wim
 
I'd expect map to have very little C-level magic.
 
Anything obviously dumb with this string manipulation? stackoverflow.com/a/53288971/2336654
 
8:45 PM
eh, [*map(...)] is just a list comp, right?
 
yes... but golfier
 
Here is the implementation of map. I can't read all of it, but I'm pretty sure it's not doing anything especially fancy
 
wim
This is the good map.
 
I can't discount the possibility that C is literally magic and therefore map is fast because it's in C despite the fact that it's doing a regular old loop
 
wim
(for other pointless distractions, see np.vectorize)
 
8:50 PM
no point ^
(-:
 
I've been seeing [*thing] more and more often these days
 
Interesting (length 10k df)
%timeit list(map(rm , df.GOOD_ADR1, df.BAD_ADR1))
%timeit [*map(rm , df.GOOD_ADR1, df.BAD_ADR1)]
%timeit [rm(*t) for t in zip(df.GOOD_ADR1, df.BAD_ADR1)]
%timeit [t for t in map(rm, df.GOOD_ADR1, df.BAD_ADR1)]

# 100 loops, best of 3: 6.27 ms per loop
# 100 loops, best of 3: 6.31 ms per loop
# 100 loops, best of 3: 7.25 ms per loop
# 100 loops, best of 3: 6.56 ms per loop
 
list(map())
 
Can't believe there isn't a df.reticulate method that does it in 0.06 ms per loop
 
^ I love that method. But it seems unfair to always win with it. So I leave it be to give myself a challenge
 
8:57 PM
The most dangerous game: Not using dataframes
 
9:13 PM
>>> file = StringIO('stuff\r\ntest\r\n', newline='')
>>> dialect = Sniffer().sniff(file.getvalue())
>>> [*csv.reader(file, dialect)]
[['', 'tuff'], ['te', 't']]
¯\_(ツ)_/¯
 
nice
 
The documentation has an example where they pass 1024 characters to Sniffer().sniff, so maybe that would be enough
>>> file = StringIO('stuff\r\ntest\r\n' * 1000, newline='')
>>> dialect = Sniffer().sniff(file.getvalue())
>>> [*csv.reader(file, dialect)]
[['', 'tuff'], ['te', 't'], ['', 'tuff'], ...
 
wim
to be fair, it's not obvious to me what your delimiter is there either.
I suppose it guessed s because it occurs the same number of times on each row
 
There're no delimiters because there's just one column
One column is a valid special case of CSV, right?
 
wim
no
 
9:22 PM
comma non-separated values
 
wim
it's just a bunch of lines why would you use a csv parser for that.
 
Because I want a general solution
 
wim
the S in CSV kinda suggests more than one column, no?
 
Yes?
 
wim
so, I disagree that one column is a valid special case for csv
 
9:29 PM
Thank you for your input
 
wim
Sarcasm detected in violation of CoC, this incident has been reported to meta.
 
^ twice
 
import copy
def mul_mat_by_scalar(mat, alpha):
    # Write the rest of the code for question 5 below here.
    new_mat = list(mat)
    for n in range(len(mat)):
        for m in range(len(mat[0])):
            new_mat[n][m] = (alpha*new_mat[n][m])
    print('orginal',mat)
    return new_mat

mat = [[1,2], [3,4], [5,6]]
print mul_mat_by_scalar(mat, -1.5)
print mat
orginal= [[1,2], [3,4], [5,6]]
mat = orginal[:]


print mul_mat_by_scalar(mat, -1.5)
this this keeps modifying the original list
in the function
why
 
Both list(mat) and orginal[:] make shallow copies
 
mat keeps getting modified even after I slice it
so i need to make a deep copy
ho do I do that
 
wim
9:33 PM
guess what module you should import and you're probably right
 
got it thanks guys :)
 
was that a more polite way of saying lmgtfy?
 
With Tkinter, when doing a Textwidget.window_create to insert objects, does anyone know how to get the index of that object?
 
How do I flatten a 2D numpy array into a 1D python list (or numpy array) column by column?
I have something like this:
 
arr.T.ravel().tolist()?
 
9:45 PM
a = np.array([[1, 2],[3,4]]) # example input
and I want this:
 
if an array works, keep the array
 
[1, 3, 2, 4] # desired output
 
derp
 
so a.T.ravel() is what you propose?
 
try and see, but yes
 
9:47 PM
is there a way to do it with a.stack()?
 
why would you want that?
I don't know of an ndarray.stack
 
there's a np.stack() function I found. I'm translating some R code to python and my coworker said it is called "stacking". I'm completely cargo culting here.
 
if you want to lay out the data in a multidimensional array, .ravel() is the best thing (it creates a 1d view)
but it goes according to row-major order, hence the need to transpose
 
also maybe a.ravel('F').tolist()
 
ok, I'll give that a try
 
9:49 PM
I've always kept myself away from alternative layouts in the call to ravel
 
for what its worth, saves a call to transpose /shrug
 
200 quatloos for @AndrasDeak
 
also [*chain(*a.T)]
 
@piRSquared transpose doesn't copy memory, just changes strides
 
not sure if I need the tolist(). Gonna keep it as an array for now
 
9:52 PM
though yeah, I guess one less attr lookup
@Code-Apprentice yup
 
good call. @AndrasDeak
 
10:03 PM
@Code-Apprentice btw you are too free with your quatloo distribution. I hate to see inflation wreak havoc on the purchasing power of the quatloo.
 
10:22 PM
@piRSquared that was my first and only quatloo distribution
 
10:50 PM
cbg
 
cbg
 
I'm curious with whether anyone here has interviewed candidates for a Python position and has some ball-park questions they would actually use from the online answers to that question? When I'm Googling, there's a whole spectrum of stuff from people I don't know, so I was wondering whether people I have some familiarity with actually think e.g. expect that knowing about the mutable default argument, or the GIL, was a decent benchmark.
Or a different line of attack; I've spent that long working alone that I have absolutely no idea what my level is compared to industry levels that are more development-focused, so I'm curious for a yardstick that might be used :)
 
11:15 PM
we do a lot of django here, but my participation in interviews has been as a relatively junior member of the team and I didn't ask any python-specific questions.
I figured my lead dev asked about those kinds of things...and the candidate already showed their programming chops in the coding exercise.
I ask about their experience with automated testing and version control instead
 
@roganjosh I've done quite some interviewing but I'm relatively inexperienced (~2-4 years) to be taking interviews. Anyway, what I've found is that generators or list / dict comprehensions are a relatively okay benchmark for decent candidates. If they know / can tell more about nested comprehensions, it's an added bonus. GIL & mutable default args is something I haven't asked yet, mostly because of the positions I was interviewing for.
@roganjosh same :|
 
@Code-Apprentice can you say roughly what that challenge was?
@shad0w_wa1k3r What level were they interviewing for?
 
11:32 PM
junior / mid-level
 
Ok, that bases my inference a bit better, thanks :)
 
huh
 
... I was waiting for a grand revelation to the "huh". Can I stop holding breath? :P
 
> Anyway, what I've found is that generators or list / dict comprehensions are a relatively okay benchmark for decent candidates. If they know / can tell more about nested comprehensions, it's an added bonus.
huh ^
 
Same boat
 
11:47 PM
then again I've reassessed my view of programming candidates since I heard a few stories and read Atwood's "goats vs sheep" blog post
 
For first technical round, good for early rejections, doesn't guarantee candidate is good :-p (benchmark since law of large numbers)
 
I've built my solver and was trying to think how to word the CV for the supporting infrastructure so I was wary of overstating anything
 
CV?
 
currviculum vitae
 
@roganjosh building a blogging platform
 
11:48 PM
What you send off to apply for jobs for
 
CV for the supporting infrastructure?
 
there's a missing comma after the CV
 
As in, I have a solution algorithm, but I built a huge Flask app to allow users to configure it
 
Built your solver? Sorry, someone's jamming the line :D
 
I'm with Andras...not really sure what you are saying. I know what a CV is. I don't get the "for the supporting infrastructure" modifier.
 
11:50 PM
I thought I'd be beating a dead horse to say that my main jobs are in optimisation. used to be vehicle routing, now machine scheduling
But in my current position I had to build all the front and back end for people to interact with the optimiser
 
sounds like something to include on your CV/resume
 
In essence, I was trying to benchmark myself against a "full-stack" developer because I don't want to overstep the mark
 
I thought it's completely normal to be bold in your CV
not outright lying, but being, uh, assertive in your phrasing
 
My job is the solver. Whether I can build the UI around it is something I can't benchmark properly
 
I keep hearing that the vast majority of programming candidates can't even write a for loop, so you shouldn't worry too much about the exact phrasing of your real, existing programming knowledge
 
11:55 PM
Yeah, which is why I was asking about the questions I might face on Python understanding in general.
 
but again I remind all those present that I'm probably the furthest from industry here
@roganjosh I'm starting to realize that you're preparing for an interview yourself, rather than wanting to interview others. For some reason I thought it's the latter. Probably a good sign that I should call it a (UTC) day
 
@AndrasDeak academics just throw £50K into a training budget to get slav students trained up :P
Yeah, I've totally screwed my question up, sorry
I'm thinking of preparing a new CV and going back out again for a new job
 
@roganjosh here it's more like £5K per annum :P
 

« first day (2950 days earlier)      last day (1995 days later) »