« first day (2721 days earlier)      last day (2225 days later) » 

4:00 PM
>>> '%3        a' % 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: unsupported format character ' ' (0x20) at index 2
so, that's weird, but seems in line with what I'd expect
 
Cabbage everyone!
 
cbg @Prateek
 
cbg
 
is there meaning for avocado and guacemole in salad language BTW?
 
Meaning is highly fluid outside of the three most used words
 
oops..yeah..avocado is there..my bad
@Kevin what are other 2 most used ones? besides cbg
 
cbg and rbrb are unquestionably #1 and 2, in my mind... I'd say Yam is in the #3 slot at the moment
Cussing never goes out of style
 
@Kevin cabbage melon
 
wim
@WayneWerner yes, it is, but look at this
>>> '% d' % 1
' 1'
>>> '{: d}'.format(1)
' 1'
>>> '%  d' % 1
' 1'
>>> '{:  d}'.format(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Invalid format specifier
it lets you use invalid format specifiers
that's the "bug", the consequence being that any % signs inside a logging message need to be escaped, even if they are not ambiguous (can't be misinterpreted as a format spec)
 
cbg guys, currently I'm rsyncing tons of files onto the cloud(100's of GB's). However, only a few files are really changed/added/removed.
 
wim
4:18 PM
If that's not a bug, it should at least be documented.
 
Is there a faster alternative?
 
wim
that's pretty much what rsync does best. what's your problem?
 
Well, it still takes an extremely long time, like hours. I'm thinking in my head whether there's a hacky alternative that marks modified/created/deleted files and/or enumerates them in a log or something of the sort
so that rsync knows that there's only ~100 files that really need to be synced when I need to run it
or maybe a smarter alternative is available and I'm just too thick to come up with it
 
At some point you have to compare the last-modified-times/contents of files, you can do that via rsync or you could read them all and then only pass in your N files to rsync directly. Either way you have to pay the price.
 
Doing research now on it but figured someone might know something I can use. I am using python 3.6 and trying to make an exe file. Py2exe wont work due to a change in I believe cPython? What are some other options you have used that work with 3.6? Working through this now, if you know something better feel free to share.
 
4:22 PM
it would be much slower if you had to transfer all the unchanged files, assuming you're using rsync right
rsync even tells you the "speedup" obtained by skipping existing files
 
rsync does a timestamp + size check and then scrolls through the contents. I think it doesn't transfer the unchanged files
 
No it won't transfer unchanged files
 
Yep; I'm guessing the walk it takes is the bottleneck(hence I want to remove as much of the walk as possible)
 
You're probably being slowed by lots of small files. I wonder if it'd just be quicker to zip, transfer single file, and unzip.
 
that is true. I think I'll try that, very good idea
 
4:26 PM
40
Q: Create a single executable from a Python project

ShadowFlameI want to create a single executable from my Python project. A user should be able to download and run it without needing Python installed. If I were just distributing a package, I could use pip, wheel, and PyPI to build and distribute it, but this requires that the user has Python and knows ho...

 
@OneRaynyDay one thing, your cloud may charge for uploads, using this you'll be uploading the full size again even though it's not necessary.
So you may have to take that into consideration, you've got to pay a price at some point, whether that's in time or in money.
 
@davidism Thanks! Figured it was worth asking
 
@Ffisegydd right, I know :)
Thank you for the advice regardless
 
Why you shouldn't charge your equipment during the nights:
 
Id rather it do that when Im not in the office then when my hands are on the keyboard though haha
 
wim
4:35 PM
wow is that a macbook
the white flash looked like a nuclear explosion
 
Hmm, trying to figure out the best way to export as CSV a pair of tables that have a many-to-one relationship
Each Widget has many comments, and each comment has one parent widget. I could make each row represent a widget, with a single "comments" column containing all of the widget's comments in a single delimited field. Or I could make each row represent a widget or comment, with comments appearing directly below the widget they belong to. Or I could have all the widgets in their own rows, then have a page break or something, then have all the comments in their own rows.
 
I believe its the new Macbook Smoke
 
Each of these are uniquely horrible in their own way
 
DSM
Do you have control over the parser side as well?
 
Most likely the only system that will need to import these CSVs will be... The very project that's exporting them. So yes, I do control the parser.
#2 or 3 would probably work better with C#'s existing csv parsing libraries. Flat-but-heterogeneous collections require less work than homogeneous-but-nested.
 
4:52 PM
Oh it's apparently an HP laptop which should've been returned (recall of hp few months ago for exactly this reason)
 
#2 is more human-readable, provided I can style things so that widget and comment rows are easily distinguishable. But #3 more closely resembles how I would like to implement "export to excel", i.e. one worksheet for widgets and one worksheet for comments.
Ultimately I'm probably just going to ask the client and do whatever they say, but I need to present the options in a way that reduces the likelihood of them picking the most painful option
A lesson I learned when I asked "hey, are we going to need exporting capability? Like to csv and excel and stuff" and they replied "uhhh sure, do both of those"
 
DSM
Heh. "He who learns must suffer," as the poet said.
 
Gather requirements like the supple willow, which bends in the storm but never breaks
Interview tip: when asked your greatest strength, explain at length how supple you are
 
I believe your problem relies on the fact that you want to make a CSV of something that shouldn't be made a CSV. ;D
 
Best case scenario: client says "actually, leave the comments out entirely. Just the main widget data is fine."
 
DSM
5:03 PM
"And really, who's going to look at all of them individually? Just give us the number of them and maybe what they're called."
 
Well, I said my last sentence thinking that you could comment the comments. Like in Facebook, you could comment a status, and then comment the comment. And this, in a CSV, would be so painful to read / painful to compute.

But making a CSV with the first column as the parent widget, and the next columns as the comments shouldn't be that hard, right ?

I might be mistaken about what you are trying to do.
 
I'm 70% sure the client won't ask for the ability to comment on comments. Threads are currently totally linear.
 
Does anyone know why this code:
Gives this error:
    x1 = np.fromiter(get_x(), dtype=int)
TypeError: 'int' object is not iterable
 
And if they are linear, then it's not too hard. Just a little more tedious than if I somehow was able to get a homogeneous and flat structure, in which case it would be one line of code.
 
When the int type should work?
 
5:08 PM
So I've determined what I really dislike most about python
I really do not like the logo.
 
@3141 because you return from the loop. I think you need to yield int(round(...)) instead
 
DSM
@3141: get_x() returns an int. What are you hoping fromiter will do?
 
This is pretty simple. np.fromiter() takes as first parameter an iterable. docs.scipy.org/doc/numpy-1.13.0/reference/generated/…
 
Question about pyinstaller. I am having trouble getting a few modules to load into the exe. (At least I think thats my issue) My project can be seen here and the error message and spec file here. The issue is with the PAHK module but Ive added it to the spec. Do I need to give it a file path or something to find it?
 
@3141 I suspect you want your get_ functions to be generators, yes? They should yield multiple values from within their for loops? Then you should be using yield instead of return inside them.
 
5:09 PM
and get_x() returns an int.
Oh, right. The return is in the loop. I second the others about yield.
 
Putting an unconditional return inside a loop should always trigger your code smell receptor. There are very few* legitimate reasons to do such a thing
( * I would say "there are no legitimate reasons", but I don't deal in absolutes ** )
 
Thanks for all the help, that was a pretty stupid moment...
 
DSM
Moments like this make me appreciate that yield was a good addition to the language. :-)
 
(** ... Except when I wrote "always" in the previous line)
 
DSM
In before someone well-actually'd you on grounds of inconsistency? Nice.
 
5:12 PM
BTW, has anyone here used GDAL that can give me any pointers on it? I'm going to be using it quite a bit for this upcoming project, but my experience with GIS stuff is... limited
 
The good old "no word in edgewise" footnote chain
 
@JGrindal me neither
 
Perhaps I should switch to "†" and "‡" for my footnotes so I don't have to work around the markup engine turning my asterisks into italics
Additional bonus: † will repel vampires
 
Why not just use numerics for footnotes?
@Kevin it's a dagger, not a cross!
 
DSM
@JGrindal: everything I know about GIS was from when I had to make some choropleths a few years back, and that wasn't very much, unfortunately. :-)
 
5:18 PM
backslash asterisk works for escaping here
 
Hmm it did not occur to me to try that
@paul23 Don't tell the vampires that.
 
I *very often* use that
 
The quick* brown** fox jumped over the lazy*** dog
(* scholars are divided over whether this term means "fast" or "cunning", or both)
(** more of an auburn, really)
(*** disputed by the dog; the libel lawsuit is still in court)
 
* or liquid?
 
Ew, typographically I'd want asterix, dagger, double dagger there. Or superscript numbers.
 
DSM
5:22 PM
Or living.
 
Perhaps it was never liquid, it's rather living?
Oh, also "quicken" as in someone's womb, as in living?
 
The living fast cunning liquid auburn fox jumped over the allegedly lazy dog
 
no more 'k'
 
Survey results are in: 9 out of 10 jackdaws love my big sphinx of quartz
 
@Kevin wonder if this is kept here
 
5:25 PM
@paul23 Just as well, the original one didn't have an S. So the property of being 25/26ths of a pangram is preserved.
 
DSM
You escaped this time, Kevin. This time!
 
The rare situation where two wrongs cancel out
 
DSM
@AndrasDeak: "quicken" re: kids is not a word I typically expect non-native speakers to know. Point to you. ;-)
 
While we're on the topic of foxes, have y'all seen ethiopian wolves
user image
5
basically fox-wolves in appearance
 
DSM
Those are adorable!
 
5:28 PM
@DSM most of the credit goes to George R.R. Martin :)
 
Adorable in the category of "would watch 45 minutes of nature documentary showing their antics, would not want them in my living room"
 
(but thanks, I'll keep the point on his behalf :P)
@Kevin your loss
 
My question got pushed up so figured Id see if anyone had any idea on my pyinstaller issue. More just making sure it didnt get pushed up and missed :)
 
DSM
Turning Python code into distributable executables is one of those Python skills I've somehow never needed to use. I can't even remember which one is the one we're supposed to use and which ones we're not to.
 
there's that post that sometimes gets linked here...
 
5:32 PM
I hoard all my programs like a covetous dragon, so distribution isn't a thing I worry about
 
I use a preprocessor to just concatenate all of the code into a single python file and ducks
 
maybe this one, the formatting is familiar
 
@ZackTarr I had a lot of issues with pyinstaller too, however I solved them by downgrading my python version to 3.5.
 
@KevinMGranger I wonder if they stink like foxes apparently do
 
boo, sadly I need it to be a standalone. Getting around my IT department and their No Admin Rights for users.
 
5:33 PM
It might not work but worth giving it a try.
 
@WayneWerner probably
 
Well you guys are no help at all.
WAIT
Does this make me the room's GDAL champion?
 
By my reckoning, it does
 
@SebastianNielsen exactly
 
5:34 PM
If you ask "any XYZ experts in here?" and there aren't, then that makes you the king of the XYZ hill just by dint of knowing that XYZ exists
 
@SebastianNielsen Geospatial Data Abstraction Library
 
I knew that GIS was a thing
 
Ahah I get it.
 
but.. never used it
 
It's a fancy library for making fancy maps.
 
5:35 PM
almost die
did
 
@SebastianNielsen Any negatives to rolling back a version? Just curious on others thoughts.
 
DSM
Hail our new king of GDAL! I look forward to the Speech from the Throne.
 
@ZackTarr losing those sweet-sweet features
 
Yeah, you can't run python 3.6 syntax in 3.5
eg f strings
You would have to re-write your code.
 
f-strings are blessed
f-strings are f-ing awesome
 
5:36 PM
I dont think I have any f strings. Is that like printf or what?
 
No it's just a neet way of formatting strings in python 3.6
 
they're closer to 'some {} thing'.format(stuff)
but they look like f'some {stuff} thing' instead
 
f strings are a departure away from the convention of printf-like syntax. Percent-style formatting is the method that most resembles printf, and it's the oldest and least-encouraged approach for that reason
 
"for that reason"^[citation neeed]
 
Yep just looked it up. I dont have any of those.
 
5:37 PM
and you can do stuff like f'some {stuff.upper()} thing'
 
and they're fast \o/
 
DSM
Missed a chance to write f'some {stiff.upper()} lip', there.
 
whereas you'd have to do .format(stuff.upper()), otherwise
@DSM know what I mean, nudge nudge, wink wink
 
DSM
@AndrasDeak: I think Antti played a role in their speed. Pretty sure he opened a ticket on bugs about it, back in the day.
 
yup, he did
 
DSM
5:39 PM
Aww, it's old news :-(
 
he squeezed a new bytecode thingy out of the cpython devs
I only learned about it a few months ago or so:) Close to when I learned about poke's involvement in raise from None
 
Does anyone join competetions on kaggle?
 
Okay well that being said. Rolling back just means uninstall 3.6 and reinstall 3.5 right? No fancy wizardry I dont know of right
 
@Kevin which reminds me of NatGeo trending video on Youtube on large snakes, especially python swallowing a deer !
 
I would suggest installing anaconda
@ZackTarr
So that you don't have to uninstall python 3.6
 
5:41 PM
@ZackTarr are you on windows or linux?
 
Windows 10.
 
@SebastianNielsen Not ot Kaggle, but I've been known to do CodeFights and HackerRank
 
on Window you can easily install side by side and then just run py -3.5 <your stuff>
 
Well kaggle is more data science oriented as opposed to HackerRank.
 
Neat, didnt know the python command could take in version numbers.
 
5:42 PM
I have never heard of codefights before.
 
@SebastianNielsen why do you ask?
 
I was just wondering.
It would be fun if I could team up with someone.
 
Fair enough. I do enter them, but alas I'm not too interested in teaming up.
 
wim
@Kevin I can give you a legitimate usecase
 
biggest advice: dont team up with someone that you barely know over the internet for kaggle
 
wim
5:45 PM
for thing in lazy_iterable_you_expected_to_be_empty:
    return "I didn't expect this"
 
Yeah that's why I'm not interested in teaming up :P
 
yeah, poking @ sebastian
 
@Ffisegydd Are you good at it? Have you ever won anything?
I am not interested in the money, I am interested in learning.
 
wim
I had this once, though it was with a raise not a return. Close enough.
 
I thought teaming up with someone would be an unique opportunity to maybe accelerate the learning process.
 
5:46 PM
@SebastianNielsen what are you trying to learn?
 
Data science.
 
I am a good data scientist, I have never won though because 1) I'm not as good as the people that win and 2) I never really have time to proper dive into it.
 
Atm. I am studying different types of deep learning models
 
Team programming is a unique opportunity etc, but you gotta find someone that's compatible with you. It's like Pacific Rim.
 
And don't stick your brain in 10k pounds of alien cerebral matter
 
5:47 PM
I am sorry, I have never watched Pacific Rim.
 
don't be too sorry...
 
What do you mean?
 
Short synopsis: giant robots piloted by two humans that have a special psychological connection. Sticking complete strangers in the robots tends to have poor outcomes.
 
@SebastianNielsen some people enjoy it more than others :P
 
5:48 PM
the winning model* on kaggle is 90% of the time not a deep learning model FWIW, it's more like you stack random stuff on top of each other and hope it gives you 0.001% edge over someone else
 
Yeah definitely.
 
I don't think it's that easy @OneRaynyDay
What, really? You agree with him @Ffisegydd
 
I don't think it's as easy as you think it is @SebastianNielsen
 
nobody said it's easy
 
wim
you had me at giant robot
 
5:49 PM
A slightly better summary of the movie is 2 hours of giant robots fighting aliens. What's not to like?
 
I definitely agree, that sounds like a pretty good summary of most Kaggle competitions.
 
@OneRaynyDay Sorry you just made it sound easy when you said" it's more like you stack random stuff on top of each other and hope it gives you 0.0001% edge over someone"
 
Charlie Day as a scientist is the only thing that broke my suspension of disbelief in that film.
 
Ensembles usually win, features are thrown at the wall until something sticks.
 
5:51 PM
don't be too sorry
 
Looking forward to Pacific Rim 2, here's hoping Ballistic Gel Torso does a good job
 
And usually you're trying to get it to stick 0.0001% above someone else on the public board, in the hopes that it's not too sticky to make it onto private.
 
Apparently it's better than the first one @Kevin.
 
@Ffisegydd In that case it sound easy to get to the top. The one who puts the most time in it will mathematically speaking have a better chance right? Because he has more time to throw random stuff together and test it.
 
It's also worth noting that a lot* of the code that does well in Kaggle competitions would probably get someone fired from an actual data science job, so I wouldn't take them as too great a role model. [*unquantified opinion]
 
5:53 PM
I loved Charlie Day as a scientist, as long as you think of him as that guy from the interviews about IASiPhiladelphia instead of as the character from that show.
 
definitely takes a very specific set of skills to win at kaggle, but at what cost
 
@SebastianNielsen I think it's more like "create a versatile stack of carefully chosen components, then twiddle the hundreds of dials that change the model in minuscule ways until it just happens to produce a result slightly better than your opponent"
Twiddling dials is simple in the sense that anybody can twiddle a dial. Knowing what dials to twiddle, and in what direction, is the hard part
 
It just takes som trial and error - I guess.
 
@abarnert I utterly failed to make that distinction :-)
 
If you're looking to learn data science, don't bother with deep learning.
 
5:56 PM
What, why?
 
There are much better things to start with.
 
As I understood it, deep learning is the core of a datascientists toolbelt.
 
@WayneWerner Say I need to install py2exe for my 3.5 version. Can pip install it to the 3.5 version. Because it currently tries to install to the 3.6 still.
 
hahahahahahahahahaha no.
 
lol
What would you suggest then?
 
5:57 PM
What is your aim? Why do you want to learn data science?
 
To become a data scientist - unbelievable right? :p
 
@ZackTarr py -3.5 -m pip install py2exe
at least if you can pip-install that
@davidism well, yes, if your dataset isn't large you shouldn't be using Hadoop, right?
 
@Kevin Real-life Charlie Day's mannerisms remind me of a friend of mine from college who was a chemistry major, and who in turn reminded me of the nerd girl from Real Genius, which probably helps.
 
Wow - okay so I thought the network throughput might be a bottleneck (it usually is)
turns out the throughput at work is 10GBs/s
f me
 
5:58 PM
IO on disk from lots of small file lookups?
 
@WayneWerner it's a toy example, but gets the point across.
 
no - rsyncing to the cloud
 
@OneRaynyDay are you opening a lot of network connections?
 
@Ffisegydd I would also like to develop my skillset to be able to participate in competetions.
 
cause opening a TCP/IP connection is pretty slow
 
5:59 PM
I know what your use case is, I'm saying "I'd imagine your bottleneck would be IO on disk..."
 
Anecdotally, for every gov project we've been on that wants to use the crazy distributed tool of the moment, it's been entirely unnecessary and counterproductive.
 
That could be a neat way of earning some cash
 
Sure, if you can win...
 
14 mins ago, by Sebastian Nielsen
I am not interested in the money, I am interested in learning.
 

« first day (2721 days earlier)      last day (2225 days later) »