« first day (2549 days earlier)      last day (2624 days later) » 

00:05
does this look right? np.sqrt(np.dot(np.transpose(xi), xi))
00:31
@kush probably too verbose
depending on shapes that's np.sqrt(xi.T.dot(xi)) or something
but you can always check a vectorized approach with a loop
I don't even know what vectorize means >_>
I don't think I am doing it right
so I have a 100x10 matrix and I think each row of the 100 rows is a vector which would fit nicely with the result I expect for distance matrix which is 100x100 matrix
but a distance matrix is supposed to have all zeros in the leading diagonal, no?
like the distance from me to me is zero?
 
7 hours later…
07:47
I made someone mad I think. I don't like feeling like a bully. But sometimes right is right. At least I think I'm justified.
08:11
@piRSquared well, it wasn't you who was the bully there.
08:46
Hi guys, do you happen to know how to create a graphviz legend in python?
 
2 hours later…
10:34
@kush you're calculating pairwise scalar products; that's 1 when you apply it to itself. As I said before, you need to understand the math first, otherwise you're just cargo culting and you'll never know if your result is right
Find the definition of the distance matrix, then implement it (or try using scipy.spatial.distance.cdist if applicable)
@piRSquared you were right, don't worry about that
It must count as something if the biggest bullies here think you're fine :P
 
3 hours later…
13:14
Guys I would be glad if you helped me with this a bit - unfortunately the documentation on the internet is pretty bad :(
0
Q: Using Graphviz in Python - creating a legend

SlowerPhotonThis is a part of my graph made with graphviz in Python: Now I would like to add a description (legend) for the graph. I want to express what color corresponds to what branch (of OpenSSL in my example). I.e. something like: Branch 1 Branch 2 (written in their corresponding colors) How can...

13:27
@SlowerPhoton 1. you already asked about that 5 hours ago, 2. you're explicitly asked not to post new questions here, 3. your question seems to be a duplicate
Interestingly if I google "create a graphviz legend in python" I get the duplicate that Martijn found, which raises the question what you tried to do before asking here
Its not a duplicate as Ive explained in my comment
Ive been asking for an answer for more than two hourse I va really done a bit of reasearch
and sorry for breaking the rules you mention it was not on purpos
you have shown absolutely zero indication of "really done a bit of research"
I am unsure how to demonstrate that - the answers I have found were of no use to me becuase in 90 % of the time they were not related to python at all but the .dot files for graphviz
I discovered the graph.attr(label = "my label") method but this doesn't work for the problem I described
because it needs to be in different colors
the linked duplicate's answer seems to be a pretty hacky way of creating a legend, so are you sure graphviz supports legends in the first place?
if graphviz itself doesn't do that it's no surprise that the python interface doesn't do that either
13:49
Yes, they seem to be creating a subgraph for that kind of thing and I am fine with that
if only it worked somehow in Python :(
they create a table which means hardcoding the creation of table somehow
but even if I wanted to do that in such an ugly way I don't know how
(in Python)
if all else fails you can dump your graph to a .dot file, edit that, then read it back
Yes, but but the process needs to be automatic and work for any general graph
thats why i cant hardcode those things
14:07
cbg
14:58
Thx @AndrasDeak and @AnttiHaapala
Cabbage
hey guys, anyone familiar with gcloud and kebernetes?
@SlowerPhoton I'm glad that you didn't break our rules on purpose, but it's expected that people posting here have actually read those rules...
i just keep running into problems trying to set it up
here's the output, which occurs even though i think i've set up the gcloud sdk:

$ ./kubernetes/cluster/kube-up.sh
... Starting cluster in us-central1-b using provider gce
... calling verify-prereqs
missing required gcloud component "alpha"
missing required gcloud component "beta"
@SlowerPhoton How was Martijn to know that you're using some Python GraphViz library? You didn't mention that anywhere in your question. FWIW, there a few different Python libraries with different capabilities. However, I've never used them since I find it easy enough to generate DOT files directly. I don't use GraphViz very often, but I was using GraphViz via other languages long before I started learning Python.
One neat thing I only learned recently about the GraphViz command line programs (dot, neato, etc) on Linux: you can specify x11 as the output type and it will render the graph directly to an X window. To see the output formats your version supplies, pass an invalid format specifier, eg dot -T?
@user2047228 Sorry, I know nothing about those things.
15:35
I just noticed a little mistake in one of the AES-ECB files I posted on GitHub the other day. It's all fixed now, but some of the old code used the key length (16, 24, or 32 bytes) in some places where it should've used the AES block size (16 bytes).
FWIW, that code lets you make an AES key and do AES-ECB encryption & decryption by calling the OpenSSL library directly via ctypes, so you don't need to install a crypto module.
16:05
@AndrasDeak @piRSquared :'D
LGTM
@AndrasDeak X0 is [x0,0 x0,1 x0,2 ... x0,d-1] but regardless of what X0 is, X0 - X0 should be zero and ||X0 - X0|| should also be zero, no?
@piRSquared Sorry, I have to agree with the middle-schooler. Sure your code is self-explanatory and very short. But it's still not good to post code-only answers, you should always add a little bit of text, even if it's only something like "You can do this using groupby:". High-rep users are supposed to set a good example for the newbies...
I did cryptopals 2/11 a couple of hours ago. At first I thought it was impossible, but then I realised that I'd misunderstood what encryption_oracle() was supposed to do. I thought it randomly chose between ECB and CBC for each block of a given plaintext. :)
16:30
@kush yes
but np.dot computes X0.X0 etc., the scalar product
what you're talking about is the norm of the distance of two vectors
as I said: understand the math, then compute it right
yesterday, by Andras Deak
instead of randomly looking at functions, try to understand the math first :P
A little while ago I wrote some code to test the speed of XORing two bytes strings together. I expected it'd be a little faster to do it by converting to int and back, rather than looping over the strings at Python speed, but I was surprised by how much faster it actually was. See here for timeit code & typical output: Test speeds of performing XOR on two bytes strings of equal length
import numpy as np
arr1 = np.array(list(buff1))
arr2 = np.array(list(buff2))
res = bytes((arr1 ^ arr2).tolist())
my version ^
I expect it to be definitely slower for small strings, and maybe better for really large ones?
the need to convert to a list before getting a bytestring is bugging me, but I couldn't figure out a better way in the 5 minutes I considered the problem two days ago
@AndrasDeak Give me a minute & I'll let you know. Of course, results may vary when running the code on a more modern machine. :)
sure thing, thanks :)
I could run it myself to see what a more modern machine does
don't know if numpy's xor can make use of multiple cores
evening
16:43
cbg
how's it going :)
I'm playing around on Kaggle
quite fun
You played with it?
16:45
not yet
it's one of those "oh this would be great to play with" things for me
I probably know enough to start, but I have little internal motivation and zero experience in data mining
I've actually learned quite a bit. I did a machine learning module in university but all of the data sets we played around with were already 'complete' so to speak.. Never introduced to feature engineering etc
but yeh took me a month to get going as well xD
I can see a red "report gist" link on a github gist. Has that always been a thing?
@AndrasDeak Yeah. OTOH, you can replace bytes((arr1 ^ arr2).tolist()) with (arr1 ^ arr2).tobytes(), which improves the speed a little. But it's still very slow. For small bytes strings, it's about 3 or 4 times slower than a list comp or gen exp. It manages to catch up when the size is 512.
@PM2Ring but .tobytes() does something weird, does it not?
In [10]: buff1 = b'asdf'; buff2 = b'ghij'

In [11]: arr1 = np.array(list(buff1))
    ...: arr2 = np.array(list(buff2))
    ...: res1 = bytes((arr1 ^ arr2).tolist())
    ...: res2 = (arr1 ^ arr2).tobytes()
    ...:

In [12]: res1
Out[12]: b'\x06\x1b\r\x0c'

In [13]: res2
Out[13]: b'\x06\x00\x00\x00\x00\x00\x00\x00\x1b\x00\x00\x00\x00\x00\x00\x00\r\x00\x00\x00\x00\x00\x00\x00\x0c\x00\x00\x00\x00\x00\x00\x00'
it includes the 64-bit default size complete with zeros
guess I could use np.int8?
In [16]: arr1 = np.array(list(buff1),dtype=np.int8)
    ...: arr2 = np.array(list(buff2),dtype=np.int8)
    ...: res1 = bytes((arr1 ^ arr2).tolist())
    ...: res2 = (arr1 ^ arr2).tobytes()
    ...:

In [17]: res1
Out[17]: b'\x06\x1b\r\x0c'

In [18]: res2
Out[18]: b'\x06\x1b\r\x0c'
16:55
@AndrasDeak Ah, ok. :) Yeah, I just noticed that my verify function is returning False. Oops!
but my handwavy FUD-based opinion is that this is ascii-specific
Can bytes values be >255? They can, for non-ascii input, right?
oooh stupid me
it's called bytes for a reason
well there we go
@AndrasDeak I did a quick test before I tried using .tobytes, but I did it with an array that was np.uint8.
then again I wouldn't be surprised if somehow 64-bit ints were faster to work on by the cpu
@AndrasDeak yeah I am trying to write something better than github.com/78036/one/blob/master/ComputeMatrices.py#L18 though
@kush OK, so try to do that. Why did you think that sqrt(sum((xi-xj)**2)) would be equivalent to sqrt(xi.xj)?
16:59
my bad I was just looking on stack overflow for a quick answer
exactly my point from earlier
as I also said earlier, you can probably use scipy.spatial.distance.cdist for a quick answer
and if you want to do it by hand, you can vectorize your loopy version too
what does that mean?
I could help but I believe you should really start to understand it yourself first
@kush meaning things like using np.dot rather than looping over elements and computing the scalar product yourself
@AndrasDeak Now that I've restored the code to your original syntax the Numpy version manages to catch up to the gen exp at size=512, but it's still 25% slower than the list comp by size=16384. And about 50 times slower than xor_bytes_I :)
loopy: sum(x[i]*y[i]). vectorized: x.dot(y)
@PM2Ring interesting, thanks:)
could you update your gist? I can try running it on my laptop
17:02
No worries.
how would I go about learning how to vectorize something?
I guess I could, but it might be easier if you just paste the Numpy version in yourself. :)
import numpy as np

def xor_bytes_np(a, b):
    arr1 = np.array(list(a))
    arr2 = np.array(list(b))
    return bytes((arr1 ^ arr2).tolist())
@PM2Ring making me do actual thinking, eh? How rude!
And just add xor_bytes_np to the funcs tuple.
OK, will do, thanks
@PM2Ring I don't think that's the best place :)
np.vectorize will mostly hide the loop from the user
but googling "numpy vectorization" gives a lot of relevant results
Good point. Sorry, I haven't really been following your conversation with kush. I just saw the word "vectorize" and that was the first thing that popped into my mind. :)
@PM2Ring you’ve convinced me. I’ll edit my post when I get the chance 🙂
this is with uint8 values and .tobytes(), which gives me True for the test
(that's faster than my original)
@piRSquared Excellent. :)
@AndrasDeak Yep, it's a definite improvement. Surely there's some way to do this with Numpy that's faster than plain Python... but I guess the from_bytes & to_bytes methods are running at C speed anyway.
17:20
yeah, I guess
Perhaps this OP will say something to clarify his question, but frankly the odds aren't looking good. But I reckon we should give him another 5 minutes or so before CVing. stackoverflow.com/questions/46633774/…
it's all really fast so the small overhead of allocating temp arrays is significant
17:38
HAHAHAHA
import numpy as np

def xor_bytes_np(a, b):
    arr1 = np.frombuffer(a,dtype=np.uint8)
    arr2 = np.frombuffer(b,dtype=np.uint8)
    return (arr1 ^ arr2).tobytes()
@PM2Ring I've got it ^
Size: 1024 Loops: 64
Verify True
xor_bytes_I    : [0.0003000569995492697, 0.00030087801860645413, 0.0003281980170868337]
xor_bytes_np   : [0.00037117296596989036, 0.00041500298539176583, 0.00047412700951099396]
xor_bytes_BLC  : [0.006089195027016103, 0.00628484005574137, 0.007016002025920898]
xor_bytes_BGE  : [0.00804398802574724, 0.008173542039003223, 0.008524519973434508]
Size: 1048576 Loops: 10
Verify True
xor_bytes_np   : [0.001925402961205691, 0.0020514189964160323, 0.002331484982278198]
xor_bytes_I    : [0.035676010011229664, 0.037150073039811105, 0.04015611403156072]
xor_bytes_BLC  : [0.9035593890002929, 0.9192670349730179, 0.9252126920036972]
xor_bytes_BGE  : [1.2421691989875399, 1.250712821027264, 1.252681304991711]
bytes -> list -> array was the real overhead
this feels much better
@AndrasDeak Yay!
On my machine, the crossover point where xor_bytes_np beats xor_bytes_I is around size=1100
I'll add this new version to my Gist shortly. I need to eat something first. :)
17:55
have a good meal :)
18:11
I'm just having some toast. Updated
a good thoroughly garliced buttered toast can make a really awesome supper
thanks for the update :)
I like it how numpy goes from awfully slowest to awesomely fastest
That's what I was hoping for on the previous version, but it just never took off. ;)
@AndrasDeak I'm not a huge fan of garlic. I do eat it, but I don't seek it out. And I'm not fond of garlic bread. I'm having peanut butter & Vegemite. The next piece may have blackberry jam.
@PM2Ring yeah, it sucked :)
I don't think I'd like proper garlic bread (never had it), we just grind raw garlic on our toast (but I do like garlic in moderate amounts)
and I'm pretty sure I wouldn't like vegemite :)
As I said last time we discussed Vegemite here, it's rather unusual for adults to like it if they weren't exposed to it during childhood.
yeah, I vaguely remember :)
that reminds me, I discovered the lyrics of Down Under by Ment at Work (I hadn't understood a single word) and I love it; I wanted to share this but I think you might have already been on SO-hiatus when this happened
18:28
@AndrasDeak It's one of our unofficial national anthems. :) I've never consciously tried to learn the lyrics, but I reckon I could probably sing most of them.
I had stumbled upon a great 70's-80's greatest hits playlist, and this was one of many gems on it
It was a good era for Oz rock. I'll try to post a couple of the more obscure gems from that time. I can't guarantee that you'll like them. Or even understand them. But they'll be very Australian. :)
hehe, thanks, I'll gladly take a listen later:)
considering my experience with (not) understanding Aussie, I probably won't now either
*give them a listen, I guess
18:49
The Triffids - Wide Open Road (1986) Lyrics
The Go-Betweens - Cattle and Cane (1983) Lyrics. I really love Lindy Morrison's drumming on this rather complex track.
19:10
Here's one that's fairly obscure, and it won't surprise me if you don't like it, or just don't get it. It didn't chart well at the time, but the hard-core fans of the band love it. Hunters & Collectors - The Slab (aka Betty's Worry) lyrics I like it for its quirky rhythm, and the weirdly humorous and sexual lyrics.
And now for something really left-field, from the era of acid rock. Company Caine - The Day Superman Got Busted (1971). I'm pretty sure you won't like this one Andras, but I think it might appeal to Corvid.
19:27
But enough of that silliness. Here's another one from Hunters & Collectors, a beautiful love song called Throw Your Arms Around Me (1986). This is, without doubt, their greatest hit.
And one of my favourites that I posted a couple of years ago:
Nov 7 '15 at 13:15, by PM 2Ring
But speaking of prog rock, here's a great keyboard-driven example that you may not be aware of, although it was a minor hit here. It's from a local band called Healing Force with mostly Australian members, but featuring a great singer from New Zealand. Golden Miles (audio + still image)
Jan 26 '16 at 11:00, by PM 2Ring
Some classic Aussie rock: Cold Chisel - Bow River - Live in Hamburg, Germany, 1982.
19:47
thanks, I'll check them out tomorrow
Something a little more modern: Powderfinger - Sunsets with special guest Missy Higgins. Originally recorded in 2004. Lyrics
 
2 hours later…
21:29
weekend cabbage!
 
2 hours later…

« first day (2549 days earlier)      last day (2624 days later) »