« first day (3097 days earlier)      last day (1852 days later) » 

5:27 PM
Ah, UNPACK_EX is not so arcane. docs.python.org/3/library/… indicates that its low and high byte correspond to the count of non-star variables on either side of the starred variable.
But if this is the case I'm unsure why dis.dis("a,*b,c = f()") has an EXTENDED_ARG instruction between CALL_FUNCTION and UNPACK_EX. UNPACK_EX's argument only needs two bytes, and EXTENDED_ARG is ostensibly for arguments that require four bytes.
 
@Kevin I remember my mum saying something about inking pretty patterns on long decks, but then, that was the 60s...paisley card stacks?
 
wim
I edited the question with an example use-case
 
5:42 PM
Is it possible to extract audio from a webm in memory?
 
I've got something that half-works for simple multiple assignment. Starred assignment is still a hot mess: dpaste.com/0QN94DX
 
My original plan was for my server to pull down an audio file, but it turns out the file is a webm. its small (30 seconds long at most) and its just audio, but since it was created from html5 media recorder its a webm. I need to convert it to wav and id like to do it all in memory so im not creating a bunch of files
But everything ive searched up says to just use subprocess and ffmpeg
 
I haven't tried calling diabolically_retrieve_target_list_size from within a class' __iter__ but I bet I don't have the right number of f_backs for that
Does itertools have a "skip first N elements" function or did I hallucinate that?
 
wim
@Kevin 🤯
@Kevin islice ?
 
Yeah I'm using islice now but I'm irritated that I have to supply a stop argument
dpaste.com/0FHSQS7 supports starred assignment, based on the completely unfounded assumption that EXTENDED_ARG always appears before UNPACK_EX and that it contains no useful data
 
wim
5:52 PM
that's because the itertools module has to confirm to the irritator protocol
 
I think it might make sense for diabolically_retrieve_target_list_size to return something more complicated than an integer. The "shape" of the target list is not completely representable by its length.
 
wim
you might have more fun with ast than dis
 
Yeah, now that my fugue state is over I think I'll investigate saner approaches
 
wim
as you go down source -> tokenize -> ast -> compile, you're losing context and information
and also CPython peephole optimizer can really wreak havoc at the compile stage
 
I assumed that the bytecode had all the context and information I'd need, since ceval.c only uses bytecode to decide when to emit "not enough / too many values to unpack" ValueErrors
 
wim
5:57 PM
yeah, in this case, it would have it all. it's more machine-friendly than human friendly that's all.
 
It is a tad dangerous for me to assume that the instructions between CALL_FUNCTION and UNPACK_WHATEVER are predictable by humans
Also I'm pretty sure that everything falls apart if the right hand side of the assignment statement has anything other than a single get_diabolically_sized_list() call
 
I'm certain I've read this very short story in the last year or so, and it's neither of the stories in those 2 answers, but I can't remember the title, author, or even if I read it in one of my old books or magazines, or online. :(
 
An ast-based approach might be less brittle in that regard
 
I've Googled, and checked a dozen or so likely suspects on my shelves, to no avail.
 
@PM2Ring Hmm, this strikes a faint recognition within me. It's not A Sound of Thunder, I don't think...
 
6:02 PM
I'm almost certain it was published in the fifties, give or take a decade.
And it was really short, only a page or two.
 
I don't recall that humanity was replaced with lobsters specifically, or even necessarily anthropods. So searching for those terms in particular might be a blind alley.
I do think they were some kind of body-horror-y spindly things with a resemblance to an extant life form. I just forget which one.
 
Yeah, lobster & crab just confused Google in my searches.
 
There might have been a passage where the time traveler returns to find things mostly similar except humanity has a disturbing number of teeth.
Or maybe I'm thinking of the Treehouse of Horror episode where Homer returns to a reality where everyone has a proboscis and he shrugs, saying "eh, close enough"
en.wikipedia.org/wiki/Treehouse_of_Horror_V indicates that the segment's only known inspiration is Sound of Thunder, which I already discarded as a candidate
 
>>> x="a {b}"
>>> y="hello {x}".format(x=x,b='c')
>>> y
'hello a {b}'
is it possible to get: "hello a c" in the output ?
 
without first calling x.format(b='c')?
 
6:17 PM
yes
i don't want to do this:
>>> x="a {b}"
>>> x=x.format(b='c')
>>> y="hello {x}".format(x=x)
>>> y
'hello a c'
 
why not, though? Seems like a straightforward solution...
 
one extra line of code.
 
Or call it on the result
@SusheelJavadi no...
 
though this is not code-golf. it seems un-necessary
 
1 line of code per thing you want to do *shrug*
 
6:19 PM
slightly related, but tangent: i dont understand this obsession with shrinking every code well beyond sanity that seems to come up every now and then
 
Well, I do in the context of code golf
but not in production code
 
I just thought this was much clearer:
y="hello {x}".format(x=x,b='c')
 
@SusheelJavadi Don't aim for minimal line count. Aim for maximal readability.
 
the issue is, that line of code hides that x contains a positional arg for "b".
 
yes, true.
 
6:23 PM
that loses out on readability and makes maintenance down the line tougher, especially for someone else
@WayneWerner for the sake of code golf i guess :(
 
point noted. thanks.
 
x = 'a {b}'

f"hello {x}".format(x = x, b = 'c')
Out[2]: 'hello a c'
 
ugh
 
I guess you could assign b first, then make x a f-string.
 
f strings are nuts. but i wouldn't recommend doing that in anywhere except code golf..but at that point, you wouldnt be using format at all
 
6:24 PM
@PM2Ring Unless Wikipedia is wrong, Muppets were from 1955, while Lamb Chop appeared in 1957 :P
 
@PM2Ring I'd rather chain the format calls
"hello {x}".format(x=x).format(b='c') seems more readable to me
 
@ParitoshSingh That's not how f strings are meant to be used
I mean, you could. But that's a terrible idea
 
oh, absolutely.
 
Also, no different than ^
in particular it's basically this:
 
@AndrasDeak Me too. Or do it all with f-strings. I'm not fond of mixing two different strategies for one task.
 
6:26 PM
x = 'a {b}'.format(b='c')
print(f'hello {x}')
 
@PM2Ring yup, very confusing
 
@WayneWerner Really? Wow! We didn't get Sesame Street in Oz until the 70s, IIRC. But I remember Shari Lewis from the early 60s.
 
Well, Sesame Street is younger than the Muppet's first appearance, which was apparently on en.wikipedia.org/wiki/Sam_and_Friends
 
thanks everyone for your time!
 
Hmm, I did find it curious that PM shared a fond childhood memory that I also experienced, since that didn't fit into my existing data about our respective chronologies
 
6:30 PM
(though, re f-strings, that's actually more like print('hello {b}'.format(b='c')))
 
Ah, I see the source of my confusion. Although Lamb Chop's Play-Along premiered in '92, the character itself existed before then.
 
in fact, the format(x=x) part is entirely redundant
 
that does make it clearer as to what's going on
not enough to merit using it, but still :P
 
>>> x = 'a {b}'
>>> f'hello {x}'.format(x='what is this madness?', b='nothing')
'hello a nothing'
Unused values are ignored, but you have to supply at minimum the required values
 
f"hello {x}".format(b = 'c', d = 'never used :( sad ')
'hello a c'
 
6:33 PM
@ParitoshSingh *PEP 8 slap*
 
eep hides and cries in the corner
 
>>> x = 'a {0}'
>>> f'hello {x}'.format('okay')
'hello a okay'
>>> f'hello {x}'.format()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: tuple index out of range
f strings are evaluated before other things happen
 
formatting an f-string is a bit confusing, though. Doing it in two lines might honestly be more readable
 
yup
 
>>> def one():
...  print('It is time')
...  return 'one'
...
>>> def two():
...  print('It is also time')
...  return 'two'
...
>>> def do_it():
...  print('Doing it')
...  x = 'some thing {a}'
...  return f'{one()} {x}'.format(a=two())
...
>>> do_it()
Doing it
It is time
It is also time
'one some thing two'
 
6:42 PM
Have we eliminated the possibility of y = "hello a {b}".format(b=c) yet? How essential is it that x exists at all?
 
I don't have a clue what started this whole conversation, lol
 
one-liner frenzy
 
[I place my ear flat on the ground and listen intently.] Hmm. I sense an XY problem nearby.
 
I learned many things about f-strings a while ago. A surprising number of things are possible and a surprising number of things are impossible
>>> x = {'this': 'thing'}
>>> '{x[this]}'.format(x=x)
'thing'
>>> f'{x[this]}'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'this' is not defined
For instance ^
 
For me, in that example it's the str.format call that's behaving unintuitively.
.format's rules for indexing is rather unlike Python's native indexing syntax
 
6:46 PM
@WayneWerner Thanks. Some of that sounds slightly familiar, but I'm almost certain it never aired over here. Live to air kids shows with puppets &/or people in animal suits were invariably local. We saw Shari Lewis as a segment in a variety show, like Val Doonican.
 
As opposed to this
 
Kids' shows are relatively cheap to make, and were a good way for the networks to get their mandated quota of local content. Also, there was pressure that edutainment for pre-school & primary school kids exposed them to Aussie accents & culture. We did get some British stuff though, like Sooty.
 
>>> x = '\n'
>>> print(f'{x.join([str(n) for n in range(10)][::-1])}')
9
8
7
6
5
4
3
2
1
0
 
wim
@WayneWerner wtf
 
Which is entirely improbable to do with text in normal python
@wim format strings implicitly wrap indexes with quotes, while f-strings do not
 
wim
6:48 PM
yeah, weird. I never noticed that.
 
At one point I was thinking that you could .access them like in Javascript, but that may have been some kind of Jinja magic, or maybe I was just doing too much with Javascript at the time :P
 
i officially regret the footgun i fired. But this is pretty amazing to see, Thanks Wayne
 
wim
>>> x = X()
>>> setattr(x, "123", "one two three")
>>> '{x.123}'.format(x=x)
'one two three'
 
>>> x = {'this': 'that'}
>>> '{x.this}'.format(x=x)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'dict' object has no attribute 'this'
 
IIRC because of str.format's implicit quoting quirk, you can't use it to access a value from a dict with the key "0"
 
6:49 PM
interesting!
>>> x = {0: 1}
>>> '{x[0]}'.format(x=x)
'1'
 
>>> d = {0:1}
>>> "{x[0]}".format(x=d)
'1'
>>> d = {"0":1}
>>> "{x[0]}".format(x=d)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 0
>>> "{x['0']}".format(x=d)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: "'0'"
 
wim
what if the dict has both 0 and '0' keys
 
Oh - the key '0'
 
Likewise, '{x["0"]}'.format(x=d) doesn't do the needful
 
Yeah, looks like it will get the 0 key and not the '0' key
 
6:51 PM
@wim Then it will get the value associated with the integer 0 key, and the value associated with the string "0" key will remain inaccessible
 
wild and crazy!
 
wim
thats bananas
 
>>> x
{0: 1, '0': '1', "'0'": "'1'", '"0"': "'1'"}
>>> '{x["0"]}'.format(x=x)
"'1'"
lol
You're welcome ;)
 
Indexing within the curly brackets of a format string is mostly a convenience feature though. It's not like it's impossible to pass in d["0"] as an argument in the format call.
More people are convenienced by quote-free string indexing than people are inconvenienced by the impossibility of indexing integer-looking string keys, so there you go
 
True. I mean, arguably if you have string'd integers as keys for your dictionary you probably have done something wrong anyway
 
6:55 PM
Exactly. In that case you're a bad person and deserve misfortune.
 
I can think of basically literally one circumstance where that makes any sort of sense
by_id = {'0': {'name': 'Wayne"}, '1': {'name': 'Kevin'}
 
I must have taken a wrong turn somewhere, I didn't mean to end up in the insane asylum
 
@AndrasDeak This is room 6
like... you might want to be able to have numeric IDs for your whatevers, but a number is a lie.

But if that's the case and you have someone who's literal id is `'0'` instead of `'0000000'` or something, then, heck, lol.
 
wim
>>> d = {0: 'a', '0': 'b', "'0'": 'c', '"0"': 'd'}
>>> "{x[0]}".format(x=d)
'a'
>>> "{x['0']}".format(x=d)
'c'
>>> '{x["0"]}'.format(x=d)
'd'
so how to get 'b' ... impossible?
 
{x[\'0\']}?
 
7:00 PM
Use case: you are writing a phone book app, and {"0005309": "John Smith"} is more semantically correct than {5309: "John Smith"}
@wim The last time I tried to figure this out, my conclusion was "it's impossible".
 
@WayneWerner Keys in JSON must be strings.
 
>>> '{x.get("0005309")}'.format(x=x)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'dict' object has no attribute 'get("0005309")'
o.O
 
It's true. getattr(some_dict, 'get("0005309")') will indeed fail to return a useful value.
 
Even worse:
>>> '{x.get("0.0")}'.format(x=x)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'dict' object has no attribute 'get("0'
 
welp.
I'll take the f-strings ;)
 
wim
7:08 PM
@Aran-Fey wtf
 
json.dump will auto-convert int keys to strings. But don't attempt to pass it a dict with mixed int & str keys, and also pass it sort_keys=True in Python 3. It'll try to sort the keys before converting them, and chuck a wobbly. This is a known bug that the devs refuse to fix.
 
Wouldn't this be getting too close to eval?
 
wim
ohhh... recursive attribute access
 
Yeah
 
Presumably . is booted for a reason
 
wim
7:08 PM
the second . is another "attribute" lookup
 
> chuck a wobbly
I lawl'd
 
wim
aussie slang
 
We have "throw a wobbler" which is pretty close
 
wim
@PM2Ring it's impossible to fix backwards compat.
 
That's pretty fantastic, though
 
wim
7:10 PM
and json.dumps working the same buggy way in both py2 and py3 is more important than correctness here.
so, I begrudgingly agree with devs refusal to fix
 
what would the "correct" behavior be, anyways? Which one of the two identical keys would be kept?
oh wait, this isn't about conflicting keys, just mixed types
never mind
 
wim
the "more correct" behavior would be to convert all to string before ordering
the "most correct" behavior would be to only dump dicts with all stringy keys
 
Yeah, that latter point is fair enough...
 w@gru î‚° ~ î‚° node                                               (3m 46s 580ms)
> x = {0: 0, 1:1}
{ '0': 0, '1': 1 }
 
@wim why is this not log(N) in time?
import math, time, timeit

def worker(lst):
    for n in lst:
        if math.log2(n) % 1:
            continue
        # work here
        time.sleep(1)

lst = [range(1, 2**k + 1) for k in range(8)]
lens = list(map(len, lst))
times = [timeit.timeit('worker(it)', number=1, globals={**globals(), **locals()}) for it in lst]
 
wim
because you didn't go to infinity :P
it's O(n) with a greatly reduced coefficient
 
7:14 PM
>>> import json
>>> x = {0:0, '0': 'a'}
>>> json.loads(json.dumps(x))
{'0': 'a'}
that's fun
 
@wim OK, I see what you mean
so we can agree that for all intents and purposes it's O(log N) :PP
 
@wim python does only dump stringy keys
>>> json.dumps(x)
'{"0": 0, "0": "a"}'
 
wim
@AndrasDeak no, because big O is all about the asymptotic complexity
 
yeah, yeah, I know :P
 
7:15 PM
The JSON module does turn keys to strings
 
@roganjosh It can't eval arbitrary expressions though, so it's deemed to be safe.
 
@PM2Ring my point was more about how far they wanted to push it :)
 
but by the times the linear part starts to win the transition from python 2 will be over...
 
wim
and also the solution: 3rd party json e.g. demjson.
@AndrasDeak a better example would be using a while loop and more rapidly approaching the termination condition with a logarithm
 
but the question was about a for loop I think ;)
 
7:18 PM
Ah. I see what you're saying. I guess it's sort of a bug...
 
@wim Well, yes, because that would let you round-trip dump & load. But the auto-conversion of int keys is too convenient, so I expect removing it would break too much existing code.
 
>>> x = {0:0, '0': 'a'}
>>> json.dumps(x, sort_keys=True)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    json.dumps(x, sort_keys=True)
  File "/usr/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.6/json/encoder.py", line 199, in encoden double quotes:
    chunks = self.iterencode(o, _one_shot=True)ne 257, in iterencode
  File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0) between instances of 'str' and 'int'
 
wim
@AndrasDeak for loop is pretty much O(n)+ by design ... unless you break
 
oh yeah, break is a good way to cut down on complexity...
 
Does JSON allow for multiple keys with the same name? I mean... obviously it's just text
but is there anything that supports having multiple keys with the same name?
 
wim
@PM2Ring if i was BDFL I would add a "strict" kwarg to json.dumps, and deprecate the magic over a minor version, making it opt-in with strict=False
imo this is not convenient, in fact it's inconvenient/dangerous.
 
@WayneWerner Yes, the JSON standard permits multiple keys of the same name, but doesn't encourage it.
 
Interesting.
 
There are a few SO questions asking how to handle JSON with dupe keys because some maniacs use it on their websites.
 
I don't see json parsing an int at all
 
7:23 PM
You'll probably end up trashing the structure without even knowing it if you try to work with it
 
the only thing that I'm seeing is that bork when you do the dumps with the sort_keys
 
As PM said earlier, JSON must have string keys. They won't be parsed to integers when you read the JSON into a dict
Values will be correctly parsed as integers, they aren't affected by serialization
 
json.org doesn't seem to forbid duplicate keys, unless you squint real hard at "An object is an unordered set of name/value pairs"
 
Yeah.
 
7:28 PM
If we interpret "set" there to mean the thing that python's set() is, then you might argue that json forbids duplicate name/value pairs, but not necessarily pairs that have identical names and differing values
 
I thought I read something in here implying a conversion, but the only thing that I'm seeing is the very slight wonkiness with sort_keys
 
{'a': 1, 'a': 1} forbidden, {'a': 1, 'a': 2} allowed
But this is very silly.
 
There is a conversion. All integer keys will be converted to strings if you JSON serialize your data
 
Yes, which is natural and expected and doesn't cause a problem
 
Oh, then I'm not sure what the implied conversion is that you're referring to sorry
 
7:30 PM
I thought I understood a claim of {"0": 0} -> {0:0}, but that's not the case at all that I can see
 
In the contexts of dictionaries here, nothing to do with the format/f-string discussion?
 
right right
and JSON in particular
 
Then no, you get string keys back
 
which is what I had previously understood lol
 
This answer by jfs links to the RFC that talks about dupe keys stackoverflow.com/a/14902564/4014959 I'm pretty sure poke wrote a great answer about handling them properly in Python, but I can't find it.
 
:) You could set up parsing rules, but loads() or load() will not on their own convert the keys back
 
Yeah, which you'd need to do if you wanted to do magic with keeping the duplicate keys anyway
 
wim
complete with upvoted wrong comments from Barmar
this guy is wrong a lot, actually
 
I am, too, I just delete my wrong comments ;)
 
json.loads('{"a": 1, "a": 2}', object_pairs_hook=tuple) is a quick-n-dirty way of preserving duplicate keys in the result, at the expense of not having an easy way of indexing anything.
 
7:35 PM
@wim I'd already upvoted your comment there refuting Barmar.
 
wim
@Kevin no. both are allowed.
 
Yeah, I decided that maliciously misinterpreting a page that isn't even the official spec was probably not the right approach :-P
 
JSON structures have become so large and complicated that I wonder whether there is really any advantage to them these days for more than a few records over JSON Lines
 
I came here to ask about flask.jsonify() and got tons of knowledge + good stackoverlow quest_answer links, thanks everyone !!!
 
@lmao Glad our assorted ramblings could help ;)
 
7:40 PM
At some point there is always going to be iteration over records, might as well load them lazily and grab what you need as you go
 
@WayneWerner: Actually sir, it's just a beautiful room full of lecturers here, upvoting all the answers as I read them., thanks again.
 
@roganjosh I'm pretty sure that lazily loading regular JSON is also possible. But nobody has implemented it yet because they're lazy.
 
Ha, this could be true too :)
 
@WayneWerner I meant json.dumps({1:'one', 2:'two'}) will convert the keys to quoted strings.
 
wim
meanwhile I'm still having fun with str.format
 
7:43 PM
If your JSON object is so large that load()ing the entire thing into memory is a serious resource drain, maybe you should ask yourself whether JSON is an appropriate serialization format for your huge data to begin with.
 
wim
'{[ ]}'.format({"": "empty", '""': "not really empty", " ": "spaaaaaace!"})
 
@Kevin I'm curious what you'd use instead
 
@PM2Ring Yeah. That makes plenty of sense, understanding how JSON works :)
 
@Kevin and my point was not just about space but time complexity too. Presumably you're going to have a nested structure and you can only go so deep before you start iterating. The efficiency of that depends on whoever designed the structure of the JSON but it's not impossible that you'll still end up iterating the entire structure to find what you want
 
wim
'{x. }'.format(x=SimpleNamespace(**{" ": "spaaaaaace!"}))
 
7:46 PM
@Kevin It's hard dealing with partial objects. You don't know how big a dict or list is until you read its closing brace or bracket, which may be gigabytes away.
 
@roganjosh Some kind of packed binary format. The more information I have about the structure of the data ahead of time, the more space I can save.
@roganjosh Yeah, but JSON Lines can have that problem too. "find me the last element of this list" is O(N) for both JSON and JSON Lines, unless your OS has a better-than-linear way of locating newlines in a file.
 
There's also things like BigQuery where it will take JSON Lines but not regular JSON I don't think
 
wim
json lines has the same problem as json, except it has it n times ...
 
@Kevin In the same vein, the consumer can be smart and grab what they need on a single pass and just forget about space complexity (unless what they want to grab is also ridiculously huge).
 
@roganjosh I started work on one, once upon a time, but decided that a generator that yields recursive generators was too confusing to write, or to use. ;)
 
7:52 PM
I think of JSON and JSON Lines as being identical except JSON Lines is always a list object whose brackets are implicit and whose commas are newlines.
The distinction is almost completely cosmetic, to me
 
Except the lazy loading
We already have the tools to go through it line-by-line and, as you say, people are too lazy to make something that can load a JSON file without it going into memory
(I suspect it's much more complex than just being lazy :) )
 
Up here in my academic ivory tower, I don't care that a JSON lazy loader doesn't exist. All that matters is that it could exist. The implementation is left to my undergrads.
... Who also do not exist.
 
Well, I expect big things on this front in the next year then :P
 
If the JSONLines lazy loader is lazy in the sense that it eagerly loads each line, but only one at a time, then there's nothing complex about making a JSON equivalent.
 
wim
[
    {"k": "v"},
    {wtf.bogus}
]
the problem is that you should not eagerly yield {"k": "v"} because it should not be consumed in the context of a broken array
 
7:58 PM
I spent hours searching high & low for a JSON lazy loader in Python, and was puzzled & annoyed by their apparent non-existence. Until I tried to write one, and gave up in confusion. And I'm no stranger to recursive iterators.
 
A "lazy for the top level, eager for all deeper levels" loader wouldn't require recursive iterators, fortunately.
 
@Kevin That's virtually identical to a lazy reader of CSV. You know you have a list of lines, and you can easily read those lines one by one. You don't have that luxury trying to load full JSON of unknown length & nesting depth.
 
Exactly, and JSON Lines, if done properly, gives you some assurance. and as a consumer, it seems that people are better at making it easier to consume
 
@PM2Ring That's OK. I don't mind if my "one layer of laziness" reader has terrible memory usage for certain use cases, as long as all extant JSON Lines loaders also have an equivalently terrible memory usage.
If anyone goes to the trouble of digging up a lazy-all-the-way-down JSON Lines loader, then that's a win for me too, because a lazy-all-the-way-down JSON Lines loader must necessarily have a lazy-all-the-way-down JSON loader embedded inside it.
 
Cognitively I guess that the requirement for each line to "stand alone" and be of a regular structure helps in actually making it so, just like it's easier to fill out a spreadsheet than it is to make a uniform but deeply nested structure
 
8:06 PM
cbg
 
cbg
 
Feb 7 at 20:22, by PM 2Ring
Last year I wrote a crazy infinite prime sieve as a recursive generator filter. Each recursive call passes the parent filter to its child. It's actually quite fast at first, but it gets a bit sluggish once there are a few dozen nested filters. :)
^ That was a little confusing to write, but it was a breeze compared to a JSON lazy loader, IMHO. Of course, I may be totally wrong, and there's a sensible way to write and to use such a thing. But I'll believe it when I see it. ;)
If any multiprocessing experts are about, this visitor would like some assistance:
12 hours ago, by Jonaswg
Anyone here have experience with concurrent filehandling in python using multi processing?
 
8:23 PM
When I started with Python and found the multiprocessing module, I expected that everyone would be using it everywhere. I didn't see that. Now I never bother with it; I thought it might be useful in my last discussion about segmenting files by name, but really I have never used it for files.
Well, other than something like the segmentation. I have never tried multiple processes accessing a single file
 
multiprocessing is more useful when you want to bypass the GIL
not much use anywhere else, AFAICT
 
numpy bypasses the GIL
 
But the overhead of spawning the processes and then joining the results back... it can be faster but it's a lot of overhead too
 
Jonaswg has lots of files that can be processed concurrently, but all the processes need to share the same ginormous dict to perform the desired string replacement.
 
8:33 PM
I think each access to the Manager object will lock it
So all the processes will stall at that point
SQLite would almost certainly be slower than Redis
I haven't used multiprocessing with Redis either so I'd have to set up a full test case to understand the issue, which is a bit of a stretch :/ I guess we will have to wait still for someone else to be able to answer
 
no, no, no MCVE after all, something's off
the offending line is not shown in the code block...
 
the answer showing the typo is marked accepted for whatever reason, just going off that
Oh, I see. The interpreter is throwing an exception on option == 'C', I'm guessing the raw_input() line is just above it
 
8:52 PM
@Jonaswg depending on how ginormous your dict is, maybe have that in its own process and communicate via Queue or something? i.e. work/reader --(queue)--> (translator containing the dict) --(queue/answer)--> work/reader
 
I think there's probably a better tech that we're not aware of. That queue would give random results, no?
Just depends on the luck of the draw in the order in which values are added/taken-from the ground source?
I don't see how reading from files is a dynamic system so IMO the focus should be on getting the analysis correct vs. doing it as fast as possible and being at the mercy of how different processes run at different speeds and things get pushed through a queue
 
If you configure the IPC properly, no...
 
IPC does not dictate the run speed of the individual processes
 
one simple approach would be to just return {'term': 'definition'} to the process
I meant about the randomness :P
 
If values are constantly being updated by new values submitted to a queue to the central resource that they all then read, then it's basically random which one gets picked up by the other processes because they all run at variables speeds
 
9:01 PM
Basically, we're talking about the reactor pattern
 
@roganjosh cue black-and-white clip featuring a programmer in distress. "There has to be a better way!"
 
yeah, but (presumably) nobody cares. I mean, if you're trying to use multiprocessing it's because the GIL is holding you up, which means you're CPU bound.
 
@AndrasDeak fumbling round in a cupboard while Tupperware crashes into them
 
You don't care that terms are getting translated unordered... if you do, then you probably want to do something way different.
 
But I'm thinking we should care here
 
9:03 PM
Why?
 
it's static analysis. You bite the bullet when you have to respond quickly. This is just files on a disk.
Then again
 
if there's an ordering issue then multiprocessing is not a good tool to use. Neither is threading, for that matter.
If the problem can be parallelized then there's either no ordering, or you can at least break things up into chunks in some way
 
The order of the files is just as likely to be arbitrary so my argument probably holds no water unless the files are generated through some refinement process
Yeah, I conceeed
 
Granted, the entire amount of information I have is Anyone here have experience with concurrent filehandling in python using multi processing? :D
So I'm making a ton of assumptions. It might be an XY problem
 
my solution is can we not do it with just multiple files instead of writing to the same file eyyy.
 
9:07 PM
shrugs only Jonaswg can tell us what they really need :P
 
ive never yet actually made myself implement data sharing when it came to multiprocessing. Whenever a problem initially seemed to ask for it, i just found some workaround. Maybe the whole thing intimidates me, but i have lucked out so far with fortunately having such workarounds.
 
I'm not clear on how @Jonaswg populates the translation dict, and I didn't think at that time to ask about that, sorry.
 
essentially came back to what you said though, found a way to logically chunk the tasks
 
No need to be sorry
Surely it's their responsibility to ask. There's been plenty of discussion since if they respond to the ping :)
@ParitoshSingh no zombie processes arising from blocked queues? You haven't lived; at least the zombies did at some point.
 
Yeah, I mean I don't know for sure about threading, but from what I can infer about multiprocessing your data sharing... wouldn't. And if you've got each process building up a huge dict, depending on your system that could be a bad thing.
couple hundred MB? Meh, whatever. Eat the cost, unless you're trying to run like 10,000 files or something
closer to GB? Yeah, you probably don't want to do that in all your processes lol
 
9:12 PM
@roganjosh hah :P ironic, considering.
 
:P
I can't tell you the fun I had with whack-a-mole in Windows Task Manager for hours on end closing them all down after each run. Quite the hoot.
 
I don't think he mentioned the typical number of files he needs to process, but "For a "normal" file set I usually have some 10 million reads and writes".
 
400,000 writes, 390,000 reads I have in my mind
 
@roganjosh That was just a small test run.
 
Gah, 390,000 writes, 400,000 lookups. I got the numbers right but on the wrong things
 
9:22 PM
On the one hand, it sounds like he just wants to do simple string substitution on his files, but I also get the impression that the key: value pairs of his dict are somehow read from the files he's reading. Maybe he's just reading IP addresses from each file, and assigning an ordered series of strings like "IP-00001" etc for each unique IP.
 
If that is the case then it could be done easily enough in post-processing with pandas after the files are read
Then there wouldn't need to be a Manager between processes, just gather the info from each file (but not by appending to a dataframe within the process)
They could collate it into a list within each process, probably have to append to a Manager object at the end rather than push through a queue, and then create the DF
But there would be no lookup for each value from some central resource
In other words, the Manager List is just used to get the data out of each process once it's done, not for checks on each iteration of each Process. But this is mega speculation on what they need and probably full of hurdles
 
9:37 PM
You guys have a favorite way to read an xlsx in python?
 
Why do you ask?
 
"favourite" is very much not a word that comes to mind when I hear "excel"
 
I like using excel to organize my data, especially when I need to annotate it with notes and make certain rows certain colors.
 
pd.read_excel() will do it, but it doesn't guarantee great results
@Mikhail I don't know any way to make that work sorry so I'm out
There is styling that might help with colours, but for the notes I really don't know how they can be handled
 
9:47 PM
Oh, I don't care about reading the styling in python. I just don't like working with csv, especially when I need to collaborate with biologists.
 
I was about to ask ^
 
Why can the notes not be their own column?
 
probably per-cell resolution
 
I suspect that more can fit in a cell than a note. I think I'm too tired to try searching exactly
 
I mean one note per cell rather than one note per row. If you have a note column you can only have per-row resolution easily.
 
9:52 PM
support.office.com/en-ie/article/… "Total number of characters that a cell can contain: 32,767 characters"
 
@wim But does it handle numeric data where the numerals are clip art?
Feb 27 at 11:53, by PM 2Ring
Some real life Excel horror stories in connection with the recent xkcd comic: http://forums.xkcd.com/viewtopic.php?p=4435060#p4434879 Especially the Twitter link in the 2nd post.
 
War and Peace in the bacterial world, they probably wouldn't live long enough to read it to the end
 
wat
is it Friday already, @roganjosh? :P
 
@AndrasDeak one wishes so!
The point being that an excel column for comments should easily contain all the note contents and would be so much easier to handle
 
Ugh. I walked 20 minutes in the boiling sun to my library which is now freezing cold.
wish I had a change of clothes.... or a coat
 

« first day (3097 days earlier)      last day (1852 days later) »