« first day (4360 days earlier)      last day (587 days later) » 

7:22 AM
@Warcaith I don't think this will work. The StopIteration of the generator is already consumed by the for loop.
If you want to handle the yields and return as one, I would recommend a helper that puts them all into one stream.
 
What did I even write? I can't see my message, haha.
 
def flatten(generator):
    result = yield from generator
    yield result  # pass on `return` value as regular `yield` value
You can get freaky and write one line body of yield (yield from generator) but that's likely unintelligible to most people.
 
Oh, you mean that thing, no, I saw yesterday that it didn't work.
Wouldn't it work if I manually do "next()" on the generator though?
 
@MisterMiyagi Whoa, I didn't know you could assign yield from
 
@Aran-Fey Same idea as result = await coroutine. ;)
@Warcaith Yes, that would also work well.
 
7:28 AM
But if I would want to put them in a stream, what would that look like?
"yield from" does not solve that I guess :p
 
"stream" == "iterator"
 
Yeah, I see that
Is the flatten method the helper class you mentioned?
I don't get how the "return" value passes on as a regular yield value in the method
 
@MisterMiyagi Thanks for making me feel like "most people" on a Friday morning :P I'm not special? :'(
 
@roganjosh Special to me? :P
 
Works for me!
 
7:43 AM
For the record, I'm one of the people who would be flabbergasted when seeing that in a PR.
@Warcaith Well, the by being yield'ed. ;) I admit it's a bit of a mind-bend but the mechanism is simple.
yield from is effectively a for loop that yields each item. And it also evaluates to the returned value of the generator.
 
Oh, ok, so it doesn't skip and ignores the returned value?
 
what I don't get is how (yield from generator) works. Does this make another generator from a generator (since it use yield)?
@MisterMiyagi ohhh, nvm I got it
but is there any benefit from using yield from instead of for i in?
 
Yes, it also passes on send and throw properly. Plus it's faster.
 
I see :o
 
8:09 AM
@MisterMiyagi Mister, I get "ClassA" object is not iterable with the method above.
If I ONLY return inside the method and never yields.
Should I check if the return value is an Iterable or something?
 
@Warcaith Is your "generator" actually a generator?
 
I have a method that can either (yield, return, yield+return), so the return value can be either a generator or a single item.
I want the "single item" to be an iterable of length 1 I guess, but don't know how to do it really
Maybe like this?

ret = function()
if isinstance(ret, Iterable):
    yield (yield from ret)
else:
    yield [ret]
There's maybe a much cleaner way doing it though
yield ret**
ret = function()
if isinstance(ret, Iterable):
    yield (yield from ret)
else:
    yield ret
This works great :)
yield (yield from ret) <- ugly :')
 
9:11 AM
return_value = yield from ret
yield return_value
Less ugly
 
Cbg
 
 
2 hours later…
11:12 AM
Say I have a bunch of meaningless hashes. The only thing I can do with these hashes is subtract them, which spits out a float. A small float means the hashes are very similar, a large float means the hashes are very different. Given a hash as input, how can I efficiently find similar hashes?
I could've sworn I've heard of a "distance tree" data structure, but I can't find any info about it
 
Can you sort them?
 
Well, the hash is essentially just a bunch of bytes, so... yes?
 
I don't think I have a concept of what 1 would mean here if you had the Avalanche Effect in the hashing algorithm?
1 wouldn't be any closer than 10000000?
 
@Aran-Fey there the hamming distance if that's what you had in mind?
 
There's no avalanche effect
 
11:25 AM
Ok, then that changes my thinking entirely. 1 is clearly closer than 10000000. Forget I said anything :P
 
@NordineLotfi No, I don't need a way to calculate similarity. That feature is already built in. I need a way to efficiently find similar hashes
 
but you could use that to find similar hashes, no?
 
Say I have 1 million hashes. Given a hash x, how do I find every hash y with x - y < z?
 
what's z in this case?
(I feel like it's the third time I'm asking a question instead of answering.)
 
*rolls dice* z is 4
 
11:29 AM
BINGO!
 
Intuition says iterate over the million hashes and keep track of every one that fits the criteria. But I suspect there's something fun you could do with a fully connected weighted graph.
 
(Or FULL HOUSE. I only actually did proper Bingo once and the iPad just did everything for me. I hated it)
It seems like something along the lines of a binary tree?
Though the sticking point is the fact that there's an overlap of 4. Hmm
 
Actually, let me move the goal posts a bit. Instead of finding all hashes that are similar to one specific x, I want to do that for every single hash in my data set. Basically cluster the whole data set into groups of similar hashes
 
@Aran-Fey Then have the sorted hashes and an inverted mapping of that, so hash => index. Given a hash, lookup the index in the sorted hashes; then use binary search to find the furthest hash.
If you want to cluster by similarity (as in DBSCAN or similar), you can move a window over the sorted hashes to check whether the next is still close enough to the pivot.
 
For some background info, my inputs are actually images. I create a list of all images on my PC, and then I want to find images that are similar to each other
 
11:36 AM
that's literally what people use hamming distance for though (in this case, for finding similar images, etc)
 
I think I need to read up more on this kind of hashing algorithm. Would an image and an inversion of itself give the same hash?
 
@MisterMiyagi That won't work properly though. Sorting arranges the hashes in a 1-dimensional way, but I very much doubt that they're actually 1-dimensional
@NordineLotfi Yes, x - y returns the hamming distance between x and y. I still need a way to efficiently find all my ys though
 
the y in this case is "another image" in your dataset right? so, just make a for loop that goes through every image, and compare one to every other one using hamming distance hashing.
 
@roganjosh Not sure tbh. I know it converts the image to greyscale and then looks for "interesting" points in the image. This is resistant to rotation and scaling, but color inversion? Dunno
 
@Aran-Fey Ah, so the sort wouldn't be by similarity? Then it's bogus indeed.
 
11:42 AM
@NordineLotfi But I want to do that for every image, so this would be O(N^2)
 
hmm, true. I guess you could do it using multiprocessing or multithreading (won't give too much as a speed up compared to the former) but I guess it's out of the question, maybe
 
@MisterMiyagi Yeah. If I understand the algorithm correctly, x - y returns the bitwise hamming distance, and the hashes are 8 bytes long per default, so... that makes the hashes effectively 64-dimensional?
 
If x-y is small, and y-z is small, does that imply that x-z is also small (give or take)? More formally, is it guaranteed that x-z <= (x-y) + (y-z)?
Evidently this is the case when x and y and z are regular numbers, but I imagine subtraction of images might not play by those rules
 
@Aran-Fey I think there a way to do that without comparing each file to every other one. One way would be to just get the hash of each files, then only compare each hash to the file to get the percentage of similarity. There is some existing hamming distance based hashes algorithm that do this iirc. I played around with one like that one at least
this would be faster since on the second pass, you would just compare a small string, being the hash, to each file
 
I can only compare the files by their hash though? What does it mean to "compare each hash to the file"?
 
11:50 AM
There might be a formal term for this kind of property. If you imagine that each image is a point in space, and a straight line drawn between two points has a length corresponding to the image's bitwise hamming distance... x-z <= (x-y) + (y-z) holds true as long as the shortest distance between two points is a line
 
@Aran-Fey it means exactly what it means :) this is mostly dependent on a specific hashing algorithm I found once, but I need to remember the name of the project
 
If the problem is isomorphic to this kind of geometric representation, then you may be able to make use of data structures that work well on spatial data. For example an octree.
 
Is the problem too complicated to duplicate in a MCRE with a stubbed find_similar() function?
 
Reading en.wikipedia.org/wiki/Metric_space, I see it mentions hamming distance. Encouraging, that we're on well-trod ground
 
@Kevin I think x-z <= (x-y) + (y-z) does indeed hold true, but that thing you said about the shortest distance sounds like a red herring. Because hamming distance is essentially traveling across the axes, right? We're not taking the direct route from x to y
 
11:57 AM
True. I must ponder this.
 
I wonder where Kevin goes to ponder?
 
it probably doesn't always travel across the axis though. It's highly dependent on what we're comparing here. For example, most of the perceptual hashing algorithm use the binary representation of the "hashes", then use their hamming distance to say "okay, they are X time far from each other", or I guess the word "similar" work here too
you could implement your own way that use the hamming distance of some other properties of your data, which doesn't need to be binary, from the hashes, or etc
@roganjosh on top of a secret montain?
@Aran-Fey anyway, here an example of the algorithm that I can't remember the name of: lvngd.com/blog/…
 
Too mundane
 
We know that I have a "prototyping cave", which I imagine is a literal cave. Maybe with a stalagmite sanded flat into a primitive work surface.

Logically, the pondering area should be near the prototyping cave. Likely a place of undisturbed wilderness. A forest, or perhaps a rocky plateau. It would be nice if it was picturesque, but it doesn't have to be.
 
Bodies of water are good for pondering in my experience.
 
12:05 PM
I like to walk when pondering, a lots.
 
The forest has a stream, but I can only stumble upon it by chance. It never seems to be in the same place twice.
 
ha, POND-ering
 
hashes = range(10)
max_difference = 3

# An example of a valid result
result = [[0, 1, 2, 3], [4, 5, 6, 7, 8, 9]]

for group in result:
    assert max(group) - min(group) < max_difference * 2
@0x263A Does that help?
 
Yes, now I can start to throw things at the wall :)
 
Just keep in mind that it's not entirely representative of my real problem because those are 1-dimensional hashes
 
12:08 PM
If an octree can represent 3d space, I wonder if you could make an eighteenSeptillionTree to represent 64d space
Going by the name, I'm guessing it would be a memory hog
 
Sounds about right
But the good news is that every dimension only has 2 different coordinates, 0 and 1
 
here another example based on the link I posted earlier: apiumhub.com/tech-blog-barcelona/…
this one try multiple different hashing algorithm, practicaldatascience.co.uk/data-science/…
still can't remember the name of the one I can't remember though
 
I wonder if a Vantage-Point Tree would work well enough
 
this was also mentioned in the first link I posted earlier
among other ones like, K-d_tree, Ball_tree
 
Interesting
 
12:26 PM
looks decent although using opencv: pyimagesearch.com/2019/08/26/…
 
VP-Tree exists as a pypi package and Ball-Tree doesn't. Looks like we have a clear winner
 
"Forgot" is perhaps not the right word. If I say "the grocery store doesn't have coconuts", it would not necessarily be accurate to say "you forgot to fly to hawaii and visit a coconut grove"
 
And somehow the coconuts on hawaii are of a lower quality than those in the grocery store
 
I may have considered the possibility of flying to Hawaii the entire time I was searching fruitlessly through the grocery store. But alas, my lunch break is 60 minutes long and a flight to Hawaii is about 11 hours. So whether I remember Hawaii or not, my lunch will have no coconut.
 
12:41 PM
morning cabbages, folks
 
We do coconuts now, get with the program!
 
But if I were to fly to Hawaii on a Saturday, and return Monday morning with a heap of coconuts that I somehow got through customs...
 
ah of course. Let's go nuts for coconuts
 
@PM2Ring Forgot to say thank you for pointing me at this. I went down quite a rabbit hole following this sound.
 
 
1 hour later…
1:58 PM
@0x263A My pleasure! Larkin Poe are pretty special. Rebecca has an incredibly powerful & expressive voice. She's an excellent guitarist, but she's also an award-winning mandolin player. And of course Megan is the Slide Queen. :)
Speaking of mandolin, check out this recent performance of Whitewater, featuring bluegrass royalty. I love how Bela & Sam watch Sierra's fingers as they dance over the fretboard.
 
 
2 hours later…
4:07 PM
patterns = list(
    map(
        lambda rule: [[rule[2]][0].replace("\\", "\\"), [rule[1]][0]],
        rules,
    )
)
Very strong Python culture here
Package this into your src package and ship it!
 
Meant ironically? where did the code come from?
rather, what's the context
stuff like [rule[2]][0] is amusing, I guess (in the same way as int('1'))
 
Strong python culture in the sense that it makes you want to strangle someone?
 
I wager that the original author had a regex with backslashes in it, and he knew it was broken because he had too many backslashes, or not enough.
Maybe he wrote .replace("\\\\", "\\") in the hopes this would fix everything. When it didn't, he changed it to .replace("\\", "\\"), knowing it was a no-op. Better to keep the general structure around for his next crack at the problem
 
4:23 PM
@KarlKnechtel it's a snippet of what is running in production in this company right now. The context: get the list of 5-tuples that represent regex rules with certain associated values, keep the regex rule and one of the values, but gather them in reverse order
 
>>> import re
>>> s = r"hack\slash"
>>> re.search("\\\\", s)
<re.Match object; span=(4, 5), match='\\'>
The ol' "need four backslashes to match one backslash" conundrum
 
@vaultah mm. not sure how someone comes up with something that convoluted; I assume it's a multi-step process of repeatedly missing the obvious
>Better to keep the general structure around for his next crack at the problem
No; better to actually understand how the system works and create strings that don't require a patch on the end.
that also doesn't justify the useless list wrapping and unwrapping, or the use of map (perhaps repeatedly adapted from a very old codebase?)
 
[thinking about the dozen times this year management told me I should stop trying to understand how the system works]
 
patterns = [[rule[2], rule[1]] for rule in rules] is all we need (probably more readable and expressive than the slice version)
this is what I like about being the only dev on my project :D
 
@KarlKnechtel nope this file is two years old
 
4:27 PM
oof
I've definitely used things like list(map(str, x)) or sum(map(len, x)) but when you have to write the lambda anyway.... ew
(map is also nice for the implicit zipping, sometimes)
(but mainly I like it because a list comprehension can't be eta-reduced)
@wjandrea I should put it here instead instead of cluttering the comments.
Most people who ask about "iterables" don't seem to realize that it doesn't mean the same thing as "sequences". In both of these questions, the example is constructed with a list, even. It's not that hard to create a non-indexable iterable for the purpose of a MRE; a file open for reading works, for example.
in more extreme cases, I've seen questions un-duped or argued about because of an entirely-meaningless-in-context distinction between lists and tuples.
 
5:12 PM
@KarlKnechtel Agreed, lists and tuples are the same in this context
 
5:23 PM
@KarlKnechtel So what's the problem? Any solution that will work for any iterable will work for a list. If OP's confused, it's beside the point.
What I would do is put a comment under the question saying, "You said iterable, but you put a list, and lists actually support more solutions since they're sequences too. See How to iterate over a list in chunks."
And for the other one, "You said iterator, but you put a list, which isn't an iterator, though you can get an iterator out of it by calling iter() on it. As well, lists support more solutions since they're sequences too. See How to iterate over a list in chunks."
 
5:42 PM
Hmm, you may be right. I'd like to have a better system in general for organizing questions that are this closely related, though.
 
 
4 hours later…
9:14 PM
Itchy geums and blank fjord win again
 
 
2 hours later…
11:26 PM
small rant: I have a policy of not directly answering fizzbuzz-level questions, but instead working through them in the comments - for the reason that, if the asker cannot solve a fizzbuzz-level question independently, why should I expect anything I write for an answer to be comprehensible?
 
11:40 PM
larger rant: there are people who have answered thousands of questions on Stack Overflow, with accounts older than mine, who nevertheless don't seem to understand the most basic things about the site policy (not answering dupes and typos).
 

« first day (4360 days earlier)      last day (587 days later) »