« first day (4936 days earlier)      last day (26 days later) » 

3:34 AM
@Aran-Fey Yes I know, Linux is your helpful open-source friend that doesn't slap telemetry on everything, build an AI profile of your behavior and try to serve you ads... also has downsides...
@roganjosh The camelCase comes from the Spark/Scala influence.
@roganjosh Well, how many rows, and how large was the resulting DataBricks instance in memory? Were the queries and files amenable to simple columnar queries or not? Sounds like your office IT setup is not very clued-in for DataBricks. Maybe you can find find agreement on common user pain-points and get IT to at least address those. (make a benchmark of a few simple queries and track it ongoing?)
 
4:18 AM
@NordineLotfi The XDG answer is probably the best option, AFAIK most distros follow freedesktop here.
 
5:07 AM
@smci I really cannot stress enough how big this company is and how little influence I have. They were bought out and have to migrate literally every data system they have off their old owner - they're now overdue on the migration and being charged £1m per day that they remain using the old systems. I cannot have a chat with a friendly IT department as there are hundreds of people on a long-term plan
That's not to say that the plan really aligns with what we actually need but this is what we've got.
The benchmarks will be basically pointless too. The fact I have to download 70GB files from MS SQL, upload them to Azure, then pull them in to DataBricks to run a spark query that can be run quicker locally, then dump it back to Azure and download it again... if there was someone to talk to, you'd bet I'd have done it by now
 
5:36 AM
@roganjosh I hear ya. I've been in that position too. It's not your job description to survey DataBricks users and figure out the minimum use-case to get basic stuff done... but if not, running all their analytics on your laptop sounds wack. Sounds a right pain.
 
Oh, it's awful. But there has been some catharsis in just voicing how bonkers it it :D I'm still on a 4G connection and my download failed last night after just over 24 hours of running on MS SQL so I've had a big injection of grump recently!
We had an "all hands" meeting on Wednesday and I really wanted to ask the head of this migration if there was a face to it all. Like, someone just dedicated to answering questions about how everything is going to link together. But 150 people in one room and me potentially looking like a fool for not knowing the systems was too much. At least I know now who to grab if I see him walking about
Separately this question really confuses me. I don't get why keras would be calling some encoding module for cp1252 on an image. Particularly since it didn't blow up when doing numerical calcs on the previous lines before the error
Or, the traceback message is failing to encode itself? That's a bug in the library?
 
 
2 hours later…
8:00 AM
My guess is that keras somehow modifies the traceback (filtered_tb) and doesn't do that correctly, causing Exception.with_traceback to crash
 
It's the encoding_table bit that confuses me. You don't need text as cp1252 to display in a Windows cmd do you?
 
Good question. No idea how the terminal encoding works on Windows
 
i think it's going to be a bug in keras itself. It mentions pulling out a traceback so some weirdness between C/C++ and Python. <insert hand waving>
 
 
1 hour later…
9:26 AM
Asking something about general SO etiquette: the title of this question is grossly misleading, can I edit in the words in italics "Python default dictionary seems to be giving duplicate key - what is happening?"
 
I personally don't see any issue with that. It's a low-volume question anyway
 
9:50 AM
As far as etiquette is concerned I think it's fine, but I don't see much of a benefit in changing the title. It may be "wrong", but it's short and googleable.
 
@Aran-Fey It's actively unhelpful, because the title as-is states Python's defaultdict doesn't work properly, rather than simply that the OP is looking at temporary variables from multiple calls on the output of a Spark job. Their first sentence if factually wrong: "When I pass data in default dictionary in python, I get an out put as : ..." But they aren't passing defaultdicts in, they're getting them as output.
 
Honestly, I don't think it's worth cleaning up. It doesn't have an MRE, has nothing to do with pyspark, and the problem was that OP had a misconception about how functions work. The odds of someone finding this question and benefiting from it are pretty close to zero
 
@Aran-Fey The odds are high, not zero: I found it via Google as the #6 hit or higher for Python defaultdict. I was looking for how to use defaultdict
 
But did you benefit from finding it?
That only makes it worse. Much worse
(In case it wasn't clear, I think we should delete it)
 
@Aran-Fey My point was that it's not useful or generic for learning how to use defaultdict; hence needs retitling. But it is a legit newbie example of confusion, so definitely should not be deleted, just retitled so as to be very clear it's not the usual scenario, nor that defaultdict is broken.
 
 
1 hour later…
11:02 AM
Lol, truncation can be fun. I just got home insurance and in my inbox the email I got from them is titled "If you are unable to view this mess...".
 
"With age, a mess can turn into a message" –A wise guru
2
 
Confuciusbot 5000?
 
More like Confuciusbot 0.0.1 I'd say
 
Huh, you have high expectations. I thought it was pretty profound as it was :)
 
I don't think it's very clever, but then again I don't live on a mountain, so what do I know? :P
 
 
4 hours later…
3:27 PM
Hi 😊
I'm trying to follow the Computerphile HTTP server in 25 lines in Python, in the video they did it in rust, wish me luck
 
3:39 PM
I almost spit out my water
hliter = self.results.highlighter
My brain is too german for this
 
 
2 hours later…
5:32 PM
My brain is too canadian for this. I want to correct it to "hlitre".
 
lol
 
@Aran-Fey I'm not even German but I still got the same first impression before rereading it multiple times
 
Glad I'm not the only one!
 
yeah, this is related to something I found a couple months ago: Typoglycemia or en.wikipedia.org/wiki/Transposed_letter_effect
 
@NordineLotfi I heard the effect works as long as the string is an anagram and the first and last letters are the same. But I'd imagine the "love is in the air" meme also has some relation.
 
5:43 PM
yeah but I found that the effect still work even when it's not an anagram. Just shuffle the middle "letters" (except first and last letter) and it should work most of the time, in theory
 
Oops getting my phonemes mixed up. "luck be in the air tonight"
@NordineLotfi Are you removing letters to make the shuffle not an anagram?
 
ah, got things mixed up too. You're right this still hold true since they're anagrams
I have been playing around with permutation since a long time, but I never noticed the fact they were all anagrams now that I think about it
 
6:33 PM
Speaking of shuffling letters...
Jan 28, 2017 at 9:34, by PM 2Ring
def shuffle_vowels(s, vowels=set('aeiou')):
    a = list(s)
    shuffle(a)
    cons = (u for u in s if u not in vowels)
    return ''.join([c if c in vowels else next(cons) for c in a])
 
 
3 hours later…
9:04 PM
Can git not pull/merge the upstream changes into my uncommitted code? Do I really have to make a commit just so I can get the changes from upstream?
 
--autostash if your friend
also, merge commits in feature branches are awful, git pull --rebase --autostash upstream master is your friend ;)
 
Neat, thanks!
 

« first day (4936 days earlier)      last day (26 days later) »