« first day (3054 days earlier)      last day (2116 days later) » 

00:44
cbg folks
I was wondering if its possible to print the scope (or rather at least the function executing in the current scope) in some way that recursively applies to child functions/routines.

For example I can run it on a seperate thread, and print out something like `__name__` during the main and/or any method/class or submethod called
but instead of ` __name__ ` its whatever tells me the deepest current scope function
 
9 hours later…
09:42
I'm having quite a hard time figuring out how to make gstreamer's appsrc work with python/opencv
anyone got some experience in this?
10:08
turns out opencv doesn't come with python support in pip, and when trying to use it, it just continues silently. No errors or anything. Took 2 hours to figure out
so now I'm building opencv from source -_-
10:29
opencv seems like a good library, but I'd really love it if they could fix their english
almost all opencv tutorials I could find also seem to have pretty bad english
yup, a good chunk of the commits come from people with russian-sounding names
 
1 hour later…
11:36
cbg, so, strange question, potentially XY problem. (let me know if it is). I want to create a dictionary that has column names for keys, and some computations on some function args you can say. however, to create the df initially, i need just the column names. I was wondering if i could get around having the write the dict keys separately in a list, and having to rewrite the keys.
So, essentially, is there a way to only write it for dict keys, but have the dictionary not try to compute the values?
hello , i'm submitting Form with action = "http://127.0.0.1:5000/submitAnswer"

my python function :

`@app.route("/submitAnswer" ,methods = ["POST"] )
def submitAnswer():
Print('working')
render_template("submitAnswer.html")`

i keep getting this error instead.
`Method Not Allowed`

any idea why ?
how does the form submit action look? im guessing its written in html in one of your templates? show the html?
its created via Javascript but i can read it from console log function
<form class="answerFormClass" action="http://127.0.0.1:5000/submitAnswer" method="POST">

</form>
it has other inputs and 1 submit input.*
@ParitoshSingh Sorry, I don't understand the question. As far as I can tell, you're trying to create a dictionary with only keys and no values, which isn't possible (and also doesn't make much sense), but you also seem to be using pandas dataframes (which I know next to nothing about), so if "dict" actually means "dataframe" then maybe that actually does make sense after all. So... I think it would help if you could maybe show us some example code to illustrate what the problem is?
both :P uh, let me see if i can sorta do explain it clearer.
11:49
any idea why i get 405?
import pandas as pd
import numpy as np
train = pd.DataFrame({"col1":[1, 2, 3, 4], "col2": [0, 0, 0, 0]})
rows = 2
segments = int(np.floor(train.shape[0] / rows)) #segments = 2

X_tr = pd.DataFrame(index=range(segments), dtype=np.float64,
                    columns=['ave', 'std'])
for segment in (range(segments)):
    seg = train.iloc[segment*rows:segment*rows+rows]
    x = seg['col1'].values
    y = y = seg['col2'].values[-1]
    X_tr.loc[segment, 'ave'] = x.mean()
    X_tr.loc[segment, 'std'] = x.std()
127.0.0.1 - - [25/Feb/2019 13:51:34] "POST /submitAnswer HTTP/1.1" 405 -
so, something like that, the aim is to aggregate the dataframe somehow. now: writing mean and std, i had to write column names up top, and write the column names while writing their actual functionality.
this dataframe is getting out of hand REALLY quickly.
so i wanted to convert it to a function. 1 place to define the functionality. and i will be reusing the function for another dataframe too
so i wanted the function to only work on one "segment" or slice of dataframe at a time, regardless of where it came from
@za001a try replacing methods=["POST"] with methods=["GET", "POST'] & see if the issue persists. That way we might know if the server reloaded successfully, and the trouble is elsewehere
If it works, revert back & see if your server reloads similarly & retry the submission
@shad0w_wa1k3r i've tried that already right now , and that didnt fix it .
11:55
My primary guess is that the code looks correct, so the server may not have reloaded nicely
are you sure the website you're viewing is running off of the flask code that you think its running off of?
Surprise of the day: having a directory.rglob('SOMETHING*') much slower than the equivalent os.walk
(ive had issues like that before.)
flask or django i suppose.
11:56
The reason was probably that I only wanted directories and thus didn't try to match the files in the returns of os.walk, but directory.rglob('SOMETHING*/') is not a thing apparently :/
@ParitoshSingh it's a chrome extention's form
oh! hmm.
can you get a barebones post to work on it? remove render template and everything else thats not needed. Just test a post.
@ParitoshSingh ok let me try
127.0.0.1 - - [25/Feb/2019 13:59:13] "POST /submitAnswer HTTP/1.1" 405
127.0.0.1 - - [25/Feb/2019 13:59:13] "GET /favicon.ico HTTP/1.1" 404 -
i would be inclined to think at this point somethings either wrong in the tool or in the configuration somewhere.
can you test with a simple requests or something like postman that pushes get or post requests?
@ParitoshSingh not sure what you are talking about , i'm not the original developer of this extention so iam not sure what to look for
12:07
im trying to say that you should consider ditching it :P
@ParitoshSingh Sorry, I think I'm a bit out of my depth here (too much pandas). Not really sure what parts of the code you want to improve or make modular
ah no worries no worries.
i have a kind of crappy idea. but it should hopefully be good enough for now. can improve upon it later
sure hit me
sorry, uh, that was a reply to Aran
lol it's ok sorry
12:10
for your case, i think yes, ditch the form app. Make a simple requests code in python
just test that your server's get and post are working fine.
once you do that, only then start adding more code
something as simple as return "hello" on a post request, etc
(thats usually how i debug things, i just keep chucking away stuff until i can get to a state where something works. then just nail down 1 issue that breaks it all)
@ParitoshSingh so you are suggiesting that the input might be the one that is blocking the whole thing ?
because all i have is form with minmum input.
i have just found on minifest file :

"content_security_policy": "script-src 'self' https://ssl.google-analytics.com; object-src 'self'"
do you think that might be blocking the connection in a way ?
12:29
hmm
i am afraid this is all out of my depth. i dont understand much of web development related things
i see , thank you , i will keep trying and invistigating too
good luck! i will say, once again, the most barebones example SHOULD work just fine in your local system. i usually test a "get" by just opening the brower with a link, and a post by using the python "requests" library.
12:45
@ParitoshSingh thank you i will start with that
:45455540 okay you might be interested in giving this a look perhaps. My hacky solution so far:
def compute(df_segment = None, names_only = True):
    '''Takes a segment of the dataframe to aggregate into a single row as dict'''
    if not names_only:
        x = df_segment['col1'].values

    result = {
            'ave': None if names_only else x.mean(),
            'std': None if names_only else x.std(),
            'max': None if names_only else x.max(),
            'min': None if names_only else x.min(),
the only part im disliking still is having to write None if names_only else #whatever_calculation
Anyone knows if there is a better way to do this portion perhaps?
The goal is to only need to write the dict's keys once, but be able to get just their keys, or keys with computation for values, as needed.
13:05
@za001a You need to return render_template. It's possible that issues like this are causing the route to be incorrectly registered
@ParitoshSingh I'm not following this properly. Why are there so many checks for names_only?
potential xy problem, i dunno. I want to not want to write the key names separately, but there is atleast 1 instance where i will not have any df_segment but need the key names.
But the names are hard-coded anyway?
the original problem sort of link
yep. and i'll be adding and removing them fairly often
it was just getting to a point where its a nightmare to manage
both the names and their corresponding computations would be essentially hardcoded by nature
But result is a custom dictionary that basically has nothing to do with the DF?
@roganjosh yes but i have a returning template for it
13:18
yes, and i'll be iterating through the dict to update the df's slice that im working on
In terms of whether names_only is True
so, its in a way tightly coupled.
the df's column names were what i wanted to use "name_only" for
essentially, the df has to be created with those column's names, and updated part by part with the column's values.
Let me have another look at your link. Something is missing in my understanding here
or in my explanation, sorry for that.
I'm still not properly following I don't think, but you could initialize the dict with just None values and have a single check for names_only to populate it with values
13:25
so, the population step needs some kind of mapping though right
I'm on my phone so it will be a right hash if I have a go at coding this :/
if i initialize the dict separately with Nones, (or essentially just have a list with column names), i'd have to create the mapping between those names and their corresponding computations
lol, nah nah, i wouldn't want you to endure something like that.
col_name = ["mean", "std"]
result = {k:None for k in col_name} #at this point, this really is redundant
if not name_only:
    result["mean"] = x.mean()
    result["std"] = x.std()
you were thinking of something like that im assuming?
at that point, theres duplication in col names. When im axeing columns or adding new ones, it starts to get hard to track things.
though, tbh, maybe if its the best option i might go for it
results = {'ave': None,....}. Then a single if not _names_only: with x = df_segment[col_name].values. Then overwrite the values in results
Yes, that
No more redundant than your current code
yep, im starting to realise that
was wondering if there was something better than both these versions
And your dict comprehension could also be inside an if check and just return col_name
13:33
yeah, i think i'll settle for this then
i mean, the alternative is making me think of eval. and if theres anything SO has taught me so far, is that if you're thinking of eval, you're doing it REALLY wrong.
Oh my. Disaster averted.
i could put every computation in strings instead, and only evaluate it if name_only is false. you know, for science :P
(its really tempting too...)
We're talking like 4 lines of perfectly legible and logical code. Anything else would be over-indulgence IMO
oh. ah. i should probably mention i have about 32 columns like these by now
@za001a The fact that you have the template doesn't mean that your route can end without an explicit return
13:36
and 3 dataframes that use these during their creation.
(or rather, will use. its all hardcoded in loops right now. yikes)
@ParitoshSingh I'm thinking this whole thing might benefit from a small MCVE. Although we're talking about that one function, I genuinely haven't grasped the wider goal
the original link was my attempt at an mvce i suppose.
@roganjosh

@app.route("/test" ,methods = ["POST , GET"] )
def test():
Print("got it" )
return render_template("test.html")
the wider goal? feature engineering. im trying to make some features on a dataset
lots of trial and error involved, same calculations needed during training and testing, but the dataframes differ in how they are given to me for training and testing.
and the initial code is just giving me headaches when im trying to add things on my own. so im trying to get 1 place from where i can control any new features to add, alongwith the calculations involved for those features
without following what the previous base code did, which was basically copy paste large chunks of code, and slightly changing things so it worked on test files.
@za001a POST, GET as a single string isn't valid I don't think
13:42
return "got it". i'd eliminate the render template part entirely until you get that portion to work.
and yes, methods = ['POST', 'GET']
@ParitoshSingh Just for curiosity, could you not use df.describe for this?
uhm.. i dont think so i dont think.
so, the original df on which i am doing the aggregations would only have 2 or 3 columns.
these 32 columns or features would all be created by me. im essentially creating a new df that is aggregated off of chunks of the original dfs.
Ok, but once you've built the final df then all the columns are in-place?
That would then allow you to get your stats from describe
13:50
ah. i see what you mean. yes for some of them.
for some others, it wouldnt work out.
essentially, the basic stats, mean std max min etc, they're covered. Some dont really "exist" per se, they are things that i am writing as i go along, with the expectation that they help my model learn certain things better.
say, for example, taking the std deviations of only outliers in a chunk.
Ok, makes sense
I would suggest result = defaultdict(lambda: None) if I didn't suspect that you were doing something with it and pandas that requires a plain dict
im writing the function based on your suggestion now. Atleast one thing is for sure, its going to be a massive improvement over the current way its done
Fellow elsewhere asked "is there still a reason to define functions instead of using arrow functions / lambdas / anonymous functions everywhere", to which I replied, yes if you value your sanity:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in <lambda>
File "<stdin>", line 1, in <lambda>
ZeroDivisionError: division by zero
Devs, please implement named anonymous functions
14:03
yikes
They have. ;^P They're called functions. At least this isn't Pascal, even though I liked the differentiation between functions and procedures…
you must love fortran
Less so. I like my lower-case alphabetics. XP (Had enough fun with AS/400 integrations in the past… never again.)
fortran is case-insensitive
Python at least discourages "anonymous stack trace hell" by forbidding statements inside lambdas, so you're more likely to write complicated functions with def rather than lambda
14:06
Yes. Why I mentioned I like my case. ;^P The last AS/400 I touched literally couldn't encode lower-case at all…
@amcgregor AS400 continues to plague my life
FTP upload CSV, RCMD to trigger remote procedure work across that data, FTP download the resulting CSV. :mimes a self-inflicted finger-gun shot:
Meanwhile in JavaScript, if you want to make an AJAX request, you're going to have four nested anonymous callbacks
Weird; that's not how my XHRs work. XP At most one anonymous state change handler, but usually those are bound methods on whatever object is managing the interaction.
@Kevin ah yes, that horror game is beyond something. It's okay though once you get acclimatised to it.
14:09
I imagine that JS's weird binding rules don't help the situation one bit
can confirm, they don't
Correction: if you want to make an AJAX request, and you're as ignorant as I am about JS best practices, then you're going to have four nested anonymous callbacks.
well, nested or not, you're going to have a lot of lines here & there which feels like writing C code.
14:12
I've watched Madara explain JS lambda binding to the same guy twice, so I'm almost an expert
Unfortunately what constitutes "JS best practices" changes every 17 minutes as new libraries become trendy, so I haven't got the energy to keep myself educated
17.4
You need to keep up with this or your numbers will be wrong!
17.4000000000067
/me coughs darnfloatingpoint.
Had a 1.70000000000068x Uber demand multiplier this morning. When will this no longer plague humanity? ;^P
it's the universe telling you to buy a bike
Uberverse*
14:17
@amcgregor Thanks JSON
I survived a near-death experience a bit over a year ago with a blood vessel popping in my head; nature's way of telling you to slow down. The ice today encourages me to not ever bike here during the winter. ;^P
I didn't realise people even wrote JS these days. I copy/paste, change some names, get behaviour that kinda embodies the spirit of what I wanted it to and move on.
@amcgregor I hope the 'higher" part is what's hidden behind the ...
@amcgregor are there any further surprises in your brain?
@amcgregor Oh wow.. that could theoretically even be “1.70000000000068 million dollars”
Anyone here going to or know someone going to PyTexas?
@AndrasDeak Billions, but they rise to awareness semi-randomly based on ancillary factors. :P The color "hot pink" / "magenta" or "T-mobile pink" (yes, they own a trademark on that color) isn't actually a color (as wavelength of light), it's a visual hallucination caused by the interference between bright blue and bright red, on opposite sides of the spectrum.
well that goes for most of color vision
Not quite in the same "it's a color, but not a wavelength" way. ;^P
14:36
management: we need a method that serializes the Sprocket object.
Me: ok, maybe we can use JSON if all of the class' properties are simple. Let's see: ten strings, six bools, two ints, and... One DynamicSplineQualityCollectionBeanFactory.
@amcgregor "blue + yellow == green" is the same thing. So is brown.
I can reconstruct the state of the DynamicSplineQualityCollectionBeanFactory from the other properties, but it fills me with sadness.
@AndrasDeak Subtle distinction between visual hallucination (inventing a "color" that does not exist in the spectrum) and perceptual interference (RGB additive). Green is still a wavelength for individual photons to transmit. "Magenta" is not. (Think of sound. Two tones will be heard as the average; so too, albeit visually, in the case of many colors when combined, but not this one.)
Yeah, I guess. So brown it is :P
The true glory of subtractive color: it all ends up brown in the end. ^_^
14:44
I suspect there's a lot of "color depth" (I'm not familiar with the proper lingo) beyond what's present in the rainbow
magenta and brown and a few others are just easier to identify than "that one greenish colour of blue"
Hehehehehe… it's a wonderful rabbit-hole to dive into when you have spare time to burn. There's wavelength (true color), polarization (which does have an impact on human eyes, though subtle), color cast (the temperature of the lights in an environment will tint your perception of objects in that environment), radiosity (objects in an environment act as reflected light sources of tinted light), and gamut…
Psychovisual modelling for fun and profit!
It's going to be awkward when aliens make contact, and when they try to integrate into our society it turns out they have perfect spectrogram-like perception of light wavelength, which makes it more complicated to (for example) interpret traffic lights
(Color is the primary reason I'm the asshole who gets people to clarify what they mean by "4K". 4K is resolution. UHD includes color space and other parameters of the video stream. ;)
"I understand that 500nm means go. But this signal is displaying 565nm and 450nm. What does that mean?"
"to humans, pure green is indistinguishable from pure yellow plus pure blue"
"... Do I really need to be a master of xenobiology to get a driver's license?"
@Kevin You can already get a feel for the experience of "talking like a dumb to aliens": say hello to a cuttlefish. Start with hands flat on the table in front of you, touch your thumbs together, raise the inner thumb side and overlap your right index finger over your left. That's "hi". Now, you could complete the statement by flashing blue a few times, but… (as an aside: cuttlefish have excellent senses of humour, but are somewhat restricted to slapstick.)
14:52
in before blue ring octopus
(if you're thinking "but wouldn't traffic lights use additive color rules, where blue plus yellow doesn't make green?", maybe. Substitute in "red plus green makes yellow" if that works better.)
I have not had nearly as much success communicating with other cephalopod species. XP
@Kevin there's probably a reason why it's up/down/middle as well as colourful. Quite a few humans are bad at colours yet they should be able to drive.
I suppose it wouldn't be too hard to develop alien driving glasses that effectively give them human vision. That's essentially what digital cameras already do -- they convert rich spectrums of color into a mere 16 million combinations of just R G and B
@amcgregor caming back to this, colour vision and colour theory in general is a can of worms I don't intend to familiarize myself with ;) I'm happy with what I can take home from viridis
15:07
My first internship involved automated QA at an industrial printing company and my takeaway is "color is always more complicated than you think it is"
\o cbg
@AndrasDeak Most lights in my area are horizontal, with no strong L-R R-L color conventions. Sucks to be color blind in this environment
15:20
I wanted to see if there were per-country restrictions on driving for the color blind, and one search result was quora.com/Can-color-blind-people-drive, whose answers average out to "it's not a problem, we can figure out what the light is most of the time", which is slightly unnerving to me
@PaulMcG light are horizontal? Where is this?
"traffic lights usually orient their colors in a particular way" implies that occasionally they do not orient their colors that way.
@roganjosh Got a bunch of different arrangements here in Montréal, including central vertical trees, as well as horizontal bridge lane traffic lights, plus the standard single-arm hanging vertical arrangements; a few can be seen in this time-lapse.
I guess if you have R/G colorblindness, you can assume 90% of traffic lights use the correct orientation, and 75% of them use LEDs that are distinguishable to you, and 80% of the time there's a driver who's right in front of you so you can follow his lead, so it's only the remaining 0.45% of lights that you drive through thinking "I hope I don't get T-boned here"
@PaulMcG ugh
@MooingRawr cbg
15:26
We also have some weird ones. White-bar ("green warning"), red, yellow ("red warning"), green (one or more of: general, straight, turn-only, busses/taxis-only).
Probably an alert driver with R/G colorblindness has a better potential accident rate than an able-bodied driver who looks at internet memes at red lights
UK traffic lights are boring
it is at this point that i have realised that i need to be writing functions for dictionary values. i could have avoided ALL this pain. sigh.
(And, just in the city, no idle right turns, e.g. turning right at a stop. Too many cyclists getting run over, but idle-right is OK in the rest of Canada, pretty much.)
i think this is my time to take a break. :)
15:28
I loved when I was in Melbourne and the crossing signal sounded like a some kind of laser weapon
I've only heard such a laser weapon sound here in Budapest at a pedestrian crossing with blind-friendly audio signals, when a blind guy activated the audio signal with some kind of remote control
There's an intersection in my town where the crossing signal talks, and beeps with two different tones/frequencies, but I wouldn't describe it as laser-like
the one I mentioned went "pew!"
I like the beeping system, since you can tell how much time is left to cross based on how close together in time the two tones are. I hope the guy that invented that got a raise.
regular blind-friendly crossings (we don't have many...) just go tick tock
15:35
@Kevin The ones in my home town go "pew pew pew" (more like a chirp than a laser, but I can see it), then towards the end (last 10s) it goes "tick tick tick" (more of a machine clicking sound).
Ours beep, but yeah, in Melbourne they go "pew!". A lot of them here have a dial on the underside that spins for deaf/blind people
@AndrasDeak :shudders and recoils in horror:
PythonScript has no dictionaries other than globals(). That'd be a massive hit with plenty of people I've seen on SO.
"You can have any variable name as long as it's n."
15:40
I pat myself on the back for having the ego foresight to name KevinScript after its author and not its antecedents
I named my esolang "Clueless". Because you would have to be to use it for anything real. :P
(Also the fact that it's a language that basically defines no language. ¬_¬ Its only loop is forever, there isn't even a return statement (or continue, break, …) or function definition statement out of the box. ;)
That'd come in handy for the question a few days back of whether 10^42 calculations was reasonable. I bet forever gives really clean syntax for that one
@roganjosh If you're of the opinion that exceptions are only for exceptional events (errors), you may wish to avert your gaze. Exceptions are the basic unit of stack-frame-popping flow control. >:D
16:15
cbg. Is there a nice one liner to get the standard deviation of time series data in pandas at an annual window? Rolling could work if we subsequently reindex. But I am thinking there must be a better way to do it.
I think resampling with annual frequency first and then taking index of that and using in the above should be good
cbg
16:40
@amcgregor not sure I was suggesting that. I use exceptions for controlling flow roo. More that it would be nice to have a forever loop for a ridiculously long calculation. I certainly wouldn't even be alive to see the error at the very end :P
Am I reinventing the wheel here? The goal is to make it possible to write functions that recurse as deeply as they need to, without having to worry about the maximum depth of Python's call stack. And ideally this should be attainable without making large structural changes to the function. Is there already a built-in way of doing this? Something in asyncio maybe?
1) discovers immortality, 2) sets off solver for combinatorial problem, 3) hits exception after 10^42 years. Could you imagine the horror?
@Kevin I typically pivot my recursive code to be progress or candidate tracking. E.g. have a list of candidates, initially populated with only candidates = [root]; iterate while candidates (possibly using the sexy new assignment expression syntax ;), left pop, iterate children, and add them to the list of candidates. Rinse, repeat.
@isquared-KeepitReal I don't think I understand. Wouldn't you just group by year?
17:01
I, too, try to make my recursive code iterative where possible, but sometimes it's a pain. My most recent frustration was while writing a binary tree walker. Preorder traversal is easy enough, but my in-order implementation was decidedly unbeautiful.
@roganjosh Austin, TX (was afk a bit, forgot to rbrb)
I'm looking for a Python solution that precomputes a nested directories of JSON files that can be searched via AJAX calls from JavaScript -- does this exist?
I know of lunr.py and lunr.js, but they assume one is going to build one massive tree file, which is way too big for the browser
(I feel like tool-based questions like this should be permitted on StackOverflow. I can build this but don't want to is someone else has done so well)
Path9'root').rglob('*.json') will give you a generator that you can then page through
@PaulMcG You're in Austin? Are you going to PyTexas?
I looked at it, haven't registered yet (some stuff happened)
17:14
@duhaime Rats; I formerly had a project which indexed multiple Sphinx documentation indexes and would allow for php.net/<term> style searching and automatic redirection to the real documentation page. I hope GitHub hasn't eaten the project. :|
What type of search is needed, though? Full-text? Faceted? Simple?
I'm wanting only to support simple string searching with substring matching in the simplest case, where we assume the user query ends in *
I'm working towards building a tool that can be run off a static file server -- so the goal is to precompute the index / trie with search phrases, persist that to disk, then use ajax to query the data structure from disk to get matching doc ids
That's a bit easier, then. (Avoiding having to worry about things like Okapi-BM25 inverse document frequency ranking algorithms, and depluralization, and stemming, and… is always nice. That being a parallel Python implementation of that algorithm. ;)
example. This code should be able to visit trees of arbitrary depth, while still having a concise implementation. I wanted to write a normal iterative implementation for comparison, but I couldn't come up with one on the spot.
Ok, I looked one up. Here is a proper comparison. I think it would be hard to claim that in_order_traverse_iterative is more comprehensible than in_order_traverse_NN.
It's easier to argue that exec_with_non_native_stack is as opaque as in_order_traverse_iterative, but I think the former does a better job of abstracting away the complexity. You don't need to understand it beyond "this is the magic function that makes my recursive generators give me what I want"
And if you have three recursive functions, writing three iterative implementations is three times as much work as writing one magic recursive generator evaluator
17:42
Open problem: But what if I want a generator that yields values of the tree as I walk over it? in_order_traverse_iterative can do this simply by changing callback(node[0]) to yield node[0]. But any yield inside in_order_traverse_NN is interpreted as a control signal to exec_with_non_native_stack rather than a "real" yield.
@PaulMcG I'm planning on going. I've never been to a con before...you're now literally the only Python programer I know in Tx, lol. Hoping to make some contacts :)
@Kevin Bonus points for sensical names. I had some code to manipulate a combined hereditarily finite set + adjacency list model (one heck of a queryable hierarchy!) and the best name I could come up with for the method to "open and close holes" in the nested set model was stargate.
I was a sad girl in snow, that day.
18:11
missed opportunity with whack-a-mole
I take it you're not a snow angel kind of person :P
Unless I misunderstood that idiom
It's originally from the web-comic Megatokyo (at least, as far as I'm aware).
18:26
even much less obscure manga references are lost on me :P
Heh, also having had a distinct effect on me, Sluggy Freelance, whose character "Bun-bun" (the homicidal rabbit) has… well, switchblades. "You're in my seat, toots." :ka-click: XP
19:25
For those who do this, how do you manage "versioning" Jupyter notebooks? And how do you deal with metadata diff annoyances when re-running a notebook and such?
does "i convert notebooks to python code and delete them forever" count? :P
Yes, it is a valid way to do it. I really like the HTML output that I can get from it so I like to keep the ipynb around, though.
But easiest is probably to put it all into a script, have each output I want as just a single function call, so that I still get the display that I want, but the actual code is version controlled as usual..
wim
wim
@ParitoshSingh +1
unless I've missed something, the .ipynb format is really just a json hack and hence does does not have any convincing diffs / vcs / docs / comments story.
@wim that is correct AFAIK as well.
To be more clear, my workflow might be something like: use Jupyter just to create the visual report that gets exported to HTML after running through some data. But when I update the tool and maybe there's new data to display, I'd update the ipynb accordingly. I might use the same notebook to generate HTML files for different input datasets. As such I'm never sure how to include them in the VCS flow. I guess since the HTML outputs have the Python code displayed in them, those could suffice.
wim
wim
the "output" is completely canned, e.g. here is a KeyboardInterrupt stack trace committed in version control 😒
19:40
But if I change the way some data is accessed in the same function name...would be hard to know which version of the tool the report came from.
Ah, I could have the notebook print the tool version up at the top. I don't really want to commit each report output to VCS as the HTML files get large fro the SVGs being embedded in them..I guess I don't need to put them into VCS for git since they're for non-devs mainly. They can go up to google drive or w/e
Holy moly. I just realized how awesome f-strings are for cleaning up my minimal examples. Application("Hello!").serve() (all requests return "Hello!") is okay, but Application(lambda ctx, name: f"Hello {name}!").serve() is so much sexier and dynamic. (/world, /?name=Alice, etc.)
20:01
@amcgregor Yeah and I find they are generally more friendly to new users, as in it's readable and obvious what it's doing. .format() is usually fine if you name your variables, but lots of people just do empty {} inside the strings and go positionally for simple examples.
.format() suffers from the visual duplication issue. Writing .format(record=record, title=title, hgralphbatlam=hgralphbatlam, …) feels so… dirty?
wim
wim
you could format with **locals() since forever
@amcgregor yes indeed. On one hand the string is more readable, on the other the kwargs are long and duplicated. Anyways, f-strings FTW
@wim And I've avoided all locals() use in my career so far. Searching the totality of my checked out public repos, the only locals() use is that of dependencies, one test, and my template engine benchmark suite (not part of the installable package). :^P
On the gripping hand, my template engine disabling the GIL gives it a nice 15% performance boost. :cough:
20:18
how much does disabling gc help?
wim
wim
I get it, I'm just saying the functionality provided by f-strings is not really just substituting in for local names. It's the power to use expressions.
@AndrasDeak Since the GIL is freed every N executed opcodes, and template rendering tends to be a very tight loop spending an alarming amount of time in byte code, disabling the GIL allows this tight loop to execute without jumping out to other threads for the duration of the operation. (Which ordinarily might free the GIL thousands or tens of thousands of times during a single render.)
OK but how does that answer my question? :P
@AndrasDeak Ah, sorry, ignored the "much" part because the answer is literally the message previous to the one of yours. 15%. XD
I don't get it, but it's probably me (also have to go for a while)
20:30
gc? the garbage collector?
not talking about gil right?
yup, nope
@ParitoshSingh I was not referring to the GC at all. GIL. And how turning it off can improve tight loop performance, at the cost of giving up (entirely) on threading.
yeah, so andras was wondering about gc then :D
yup :P
But if it keeps on running in a tight loop than that's probably not a good idea. I didn't know if individual renders were individual runs of the interpreter.
Quite so. I'd have to benchmark that case; I only have my w/GIL numbers handy. Each render is one function call to trigger (see the linked file), no interpreter restarts across all benchmarked engines. It's assumed that all template engines tested here will most often be used in long-lived WSGI applications, thus multiple renders per process invocation.
20:35
my understanding of the Gil is very limited. but since we're talking about it, i have always wondered
what makes the GIL so good for single threaded applications? (or rather, apparently removing GIL makes performance drop for single threaded applications)
@ParitoshSingh The GIL ensures the single-threaded operation of C code, treating each executed bytecode operation as an atomic, thread-safe operation. (Peeps correct me if I'm wrong!) While executing a single operation, no other thread can possibly become active. (Thread election occurs when the "remaining ops" counter hits zero.)
wim
wim
the only viable replacement for the GIL is many smaller locks
Or full STM (software transactional memory).
wim
wim
and your "single threaded application" doesn't know that it's not running as 1 thread under some other bigger app..
But STM tends to be… slow. Hardware TM is infinitely superior.
20:39
so, the issue with removing GIL is more so of having to implement additional safety logics?
essentially doing what the GIL did anyways?
Aye; either old-school locks and mutexes and semaphores everywhere, or "revolutionary new ideas" like STM. It won't be easy to eliminate.
wim
wim
you can do what the GIL does on a finer grained scale, many little locks rather than one big lock.. it's a lot of work, and for questionable benefit
i see, that does make sense
A global lock is generally "effective enough".
Albeit penalizing threads because they literally can't execute simultaneously. ;^P
how do other languages do it? say, C or java?
20:41
Either it's a Wild West of "good luck with that", or you use certain multiprocessing libraries/preprocessors that add most of the important glue to your code for you.
you know, i am always amazed by how much python actually abstracts away. Its amazing to find out.
Here's a complete playlist of Parallelism in C++ if you're interested in digging. Some of the preprocessing stuff is magical.
wim
wim
C does it by not doing it, the responsibility to write code that treats memory safely and doesn't crash is on the developer. Java has the virtual machine, is really good for threading, at the cost of being a much more formal and strict language in general
@wim But, by gum, that virtual machine is a work of art.
Even if I feel the Java language itself has earned the shade thrown its way.
i laughed at that more than i thought i would considering i dont know any java
21:01
@ParitoshSingh It's often more elaborate of a punchline than that: now I have an AbstractCompositeProblemFactoryObserverBuilderSingletonFacadeProxy… >:D
… and a problem.
haha, well i can definitely say it worked
@ParitoshSingh In Java, it's not uncommon to not just want, but need an IDE with refactoring tools. "Make an abstract observer factory for my current class" and so forth. A warning sign. ;)
21:19
can confirm, Android Studio wrote the app I wrote a few years back
i have a list of objects with random ids, like mylist = [ob131,ob2312,ob583,ob4794,ob89795,ob78976,ob749,ob867,ob79]
what is the best way to devide this into sublists with a different condition for each sublist.
like sublist_a: only objects with object.id %2==0, sublist_b, only objects with object.id %3==0, sublist_c: only objects with object.id%4==0,, sublist_d: only objects with object.id %5==0
currently i loop over mylist and put the object in every group if it matches the condition... is tehre a better way?
Sounds like reverse fizzbuzz
Well, no. You're always going to need a loop and check a bunch of conditions. That's the part that all the solutions have in common. Maybe if you showed us how exactly you coded it we could suggest improvements
21:35
I'm having a bit of an issue with shutil.move. It moves a directory and it's contents fine (that is, my_folder exists with it's contents inside my_top_folder), but errors with PermissionError: [WinError 5] Access is denied: 'C:\\my_folder' -> 'C:\\my_top_folder\\my_folder' and while it handled that error another occurred: OSError: [WinError 145] The directory is not empty: 'C:\\my_folder'
Any ideas?
@toonarmycaptain Sounds like the system was unable to free (delete/unlink) the original "file handle" (folder) after copying the contents across to the destination, as there may be hidden files (not copied) within that, that aren't cleaned up. Happens often on macOS with hidden .DS_Store files and the like.
@Tweakimp I'm gonna go ahead and say that's fine; most "improvements" that could be made would probably just be overengineered unreadable messes
ok. i managed to remove one if unit.alliance == Alliance.Neutral.value , but it wont get better than this probably.
I'm on Windows. There shouldn't be any hidden files in there (it's a folder I'm creating and using, all within python), unless Windows/python is creating them for me?
I should have normal user permissions there.
21:46
Windows has ACLs; have you examined those to ensure you have permission to manipulate that source directory? Windows itself may construct hidden files, I'm just not sure under what conditions. The macOS .DS_Store files are there to store extended attributes (xattrs and ACLs) on remote filesystems that don't support those advanced features, entirely as an example.
@toonarmycaptain It could also entirely be a race condition of some kind, e.g. files being added to the source folder while the move operation is happening; it'd try to unlink the source folder after moving the files it was aware of, to just fail, since it didn't actually get them all.
If it's a race condition, it's one I'm not aware of/causing, which would be odd.
Looks like every file on my HDD is read only (although this doesn't cause issues during normal usage..) and owned by TrustedInstaller...? Maybe changing ownership of C to Administrators will fix this.
I need Windows because my current job's print server doesn't play well with Linux, but man, it irks me.
What's weird is that when I wrote this method and tests a couple of months ago it tested (and still tests) fine, it just appears to fail when using folders created outside the test. Hmm.
Did you check if there's really anything left in that folder? Does os.listdir('C:\\my_folder') output anything?
@Aran-Fey Just the empty list.
22:01
Huh. Weird.
@toonarmycaptain Woah, slow down there. One does not "fix" permissions issues by throwing the idea of ownership to the dogs. (Non-Windows equivalent: mode 666, UNIX permission of the beast.). :|
I had a visceral reaction when I got back to my desk and read that. (I've deployed systems like KROLL, pharmacy dispensing software, which mandates execution as administrator, and configuration of the DCOM system service to run as administrator. Makes me want to blow someone's brains out, security-wise.)
@amcgregor Should TrustedInstaller be the owner of all my files, rather than me?
Not at all. That's highly suspicious, and worthy of proper investigation.
anything that calls itself TrustedInstaller sounds like a shady guy in a trench coat
Like, if I were still using Windows, at that point I'd "nuke the machine from orbit and start anew" (it's the only way to be sure in re: malware which might mangle permissions like that). OTOH, any time things went south when I was using Windows, my reaction was to nuke it from orbit. (Eventually just roll back my VM.)
22:08
Hmm, C:\\Users is owned by System.
@amcgregor If I didn't need WinOS at work due to print servers/drivers not playing well with Linux, I'd probably have removed it altogether.
@AndrasDeak Despite the inordinately shady name, that may actually be part of Windows itself. Microsoft's attempt at SIP (System Integrity Protection), possibly?
FWIW it was always thus since I first installed Windows, I'd see the READONLY tag in explorer - properties, and occasionally have to change the owner on a file or folder to myself, but it usually didn't seem to affect anything.
rhbrb, thanks :)
Hmm update before I go...reading that these issues can be caused by Windows Updates...zero shock there...
22:25
cbg folks
anybody know why pickle is named pickle
I would guess because that's a traditional method of conservation of certain perishable goods
22:44
oh, i was hoping there was some kind of crappy pun or acronym in there
there could be one

« first day (3054 days earlier)      last day (2116 days later) »