« first day (3097 days earlier)      last day (1843 days later) » 

Hello All!
does anyone know where I can read up on the best performance measures fro KNN?
 
@ronwagner what do you mean?
 
user11093202
@U9-Forward Thank you for that It helped
 
Someone told me that there are specific measures to pick the best number of neighbors. F1, Accuracy, and precision. Just trying to make sure that im being told correct information.
 
@Programmer Your welcome :-)
 
user11093202
12:16 AM
It actually worked @U9-Forward!!!!! No joke am so suprised.
 
So I wanted to read up on it, but im not really finding anything online. I will keep searching tho.
 
@Programmer That's what a link can do
:-)
 
 
2 hours later…
1:48 AM
cbg
 
hi all
i had a function, I want to pass a date to a sql query
like so : def blah("'2019-03-13'") .. is there a way I can just pass 20190313 , just to make it easier to call?
 
this seems to be a pretty popular library for datetime convenience
You can also just do this:
>>> from datetime import datetime
>>> datetime(2019, 3, 13)
datetime.datetime(2019, 3, 13, 0, 0)
most SQL libraries that I know of will work with Python's native datetime formats, so that should be good enough.
 
cool!
so do I have to switch the date in the code if I want to call a new date?
 
note also, if you str() that result, you get it as a string
>>> str(_)
'2019-03-13 00:00:00'
@excelguy not sure what you mean, maybe a code example might help?
 
say i want to call this function in my console:
stack(20190313)
i can do that using datetime?
 
2:02 AM
Well you'd have to put commas in like I did
I mean, you could..parse the number, but that's a bad idea.
 
but what if I want ot change the date each time im calling it? do I have to change my code
like datetime(2019,3,14)
 
I'm still not sure what you're asking, but if I understand you correctly, yes---because what you're trying to do is a bad idea.
so you shouldnt expect to/want to call function(20190314)
 
why not?
 
>>> def dontdothis(date_int):
...   date_str = str(date_int)
...   return datetime(int(date_str[:4]), int(date_str[4:6]), int(date_str[6:]))
...
>>> dontdothis(20190314)
datetime.datetime(2019, 3, 14, 0, 0)
 
ok thanks, ill try it out and read up on that link you sent.
 
2:07 AM
it's too easy to mess up. manual parsing is almost always a bad idea. you can fatfinger long integers way too easy, it's hard to read, and you have to enforce specific enumerations (like you must do 4 numbers for the year, must do 2 numbers for the month, must do 2 numbers for the day)
etc, and it's just a non-standard way to work with things like this, so don't.
is the simple suggestion.
 
ill try to wrap my head around it lol,im still a newb compared to you
 
because you'll have to manually make sure for example 2019/13/55 is an incorrect date and all that.
but the built in libraries will do that for you
 
@excelguy What's wrong with just typing some commas, just three extra characters...
 
>>> datetime(2019, 55, 13)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: month must be in 1..12
>>> datetime(2019, 12, 33)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: day is out of range for month
see, the built-in gives you nice help messages
if you type day 31 for a month that doesn't have 31 days, it'll figure that out for you
and raise an error and such. so its just better to not do parsing and let libraries do it for you :)
 
but then i have to explicity state the date I want in the code base?
 
2:10 AM
well wouldnt you have to do that with function(20190314) anyways?
like...you are still writing the date :)
 
ie datetime would hav eto be configred for 20130313
 
how about a different conversation: what are you trying to do?
 
lol :(
 
like what is this code trying to achieve?
 
just trying to beable to use pythonto do my sql query, but i may want different dates in my query
like i may want 03 13 or 03 14
 
2:11 AM
where are those different dates gonna come from?
 
hmm, i guess a user's input
most likely a month end date
 
like, from a command line application?
or how does the user input get accepted? Or will you just be hardcoding it in a script?
 
no, just a previously defined month end i guess
well the user could use a console like spyder, call the function with the inputted date
then it will return the query output
 
Are you just using a simple SQLite db or connecting to a production database?
 
production database , just reading from it though.
 
2:24 AM
@excelguy Use python to convert user input into the string format you'd like, then pass that string as an argument to your database connection. I'm not sure what the issue is. Are you stuck trying to convert generic date-like input into this YYYYMMDD format? There's python standard libraries for that
And longterm if you are thinking of writing programs that move in-and-out of python and DB work, definitely give the Django ORM (object-relational mapper) a look
 
2:41 AM
@tripleee Congrats for your gold badge
of python
Chee, i was convinced to have 18k, but than a retracted up-vote, made me 2 rep away
Yes! now i did
@coldspeed Lol, you answered too
 
yes, btw yours is incorrect if "df" is a dictionary... you're missing the step of converting to a dataframe
 
@coldspeed Okay, edited mine
 
It still technically isn't correct because the integers are converted to floats. Then again, the only real answer for 0.24 is nullable types and the only answer before that is to convert but then cast to object
 
2:57 AM
@coldspeed Yeah, i voted for yours +1
Btw, this, was my first message in python room that contained edited mine
 
thanks. I am still a bit miffed by the fact that OP has asked a question they don't quite understand themselves
 
Yeah, exactly, dictionary or dataframe...
 
nice to meet you guys, I'm a new guy here, less than two weeks.
and python is my favorite language!
 
3:17 AM
me too
 
I think it is very concise and elegant
let us ignore redundant grammar, and totally focus on algorithm and logic.
 
3:43 AM
If I had to pick one thing I don't like about python, it would be all the possible misspellings of the word "python"
 
4:19 AM
haha
 
4:45 AM
@U9-Forward thanks, much appreciated!
 
wim
5:00 AM
what the dupe for naming your .py file with stdlib name collision? stackoverflow.com/q/55585151/674039
 
stop! hammer time
 
5:25 AM
oh, it's in the canon list anyways
 
 
2 hours later…
7:23 AM
i have a public google sheet with 2 sheets https://docs.google.com/spreadsheets/d/14hFn00O9632n96Z2xGWvfrcY-K4kHiOGR02Rx7dsj54/edit#gid=447738801
I know how to wget if it has only 1 sheet, but is there a simple way of getting sheet1 and sheet 2 as a file in python (maybe csv)
 
7:42 AM
My not-an-expert's opinion: there's probably an API to do that. Have you done some digging?
 
7:55 AM
@tripleee I added idjaw's canonical to the dupe target list for that question.
Even though I know it's in our canonical questions, I tend to search for it in Google via the error message, to help promote it in Google's search results.
 
@PM2Ring thanks, good idea
@pythonRcpp didn't you ask the same thing yesterday?
 
8:30 AM
Anyone here have experience with concurrent filehandling in python using multi processing?
And using shared memory at the same time
 
@Jonaswg Sorry, I know about threading, but not mult-processing. But others who do may be able to help later. But are you sure you need that, and that threading isn't adequate? What's the task? If you're doing lots of IO, threading may do what you need. But if your task is CPU-bound, then I guess you probably need multiprocessing, assuming you task can be adequately split up.
 
I read in many ifles, look for specific patterns and if I find this pattern I replace it with some generated string.
The complexity comes from the fact that files A and B should use the same generated string so I use a dictionary to store key:value pairs
When I used a threadpool I only shaved off roughly 10% of the execution time, while with a process pool a lot of time was saved during execution
But since processes use individual instances of the dict I did not have the dictionary memory shared between the various processes
So I implemented a redis store as a shared dictionary but I cannot pipeline most of the lookups/writes so it is much slower than what I would like it to be
 
8:50 AM
What about manager objects?
disclaimer, not familiar with this. but sharing memory between processes can also be implemented using manager
 
I looked into it, but manager has quite a lot of overhead
It was faster than redis
 
How many items are in your dictionary? And what's the average size of the keys & values?
 
I always start it empty and using the test set about 400 items. The keys are strings with length 3 to 16
The values are pickled lists containing a string length 9 and an integer
During the test I had about 400,000 lookups in the dict and 390,000 writes
But the test set is small in comparison to the sets I will use in the future
For a "normal" file set I usually have some 10 million reads and writes
 
the writes, are they just outputs to be dumped or passed between processes?
 
9:06 AM
@ParitoshSingh I assume Jonaswg means writes to the dict. "files A and B should use the same generated string"
 
hmm. i was going to suggest something like sqlite with locks, but i dont expect it to perform very well if it has to undergo constant updates to data
 
Agreed. If the dict gets mostly populated by processing the first file or so, then that'd work.
@Jonaswg Are the keys general Unicode strings, or would plain ASCII strings work? Why do the value lists need to be pickled?
 
done ^
wait. curious. you'd be able to add the tag yourself right? is there a daily limit?
 
@ParitoshSingh you can't hammer it if you retag it yourself
 
if I add the tag myself then my gold badge hammer doesn't work
 
ah i see
ty
 
not necessarily need to create question for this but how do i set min() to ignore zero/ find > 0 like in this function---- min(lst, key=lambda x: x.num).val)
 
9:28 AM
min((n for n in lst if n>0), key=lambda x:x.num)
 
thanks
 
@ParitoshSingh We like to let the people who don't have the Mjollnir participate in the fun of hammering. :)
 
well, can't say i didn't enjoy it :P
 
@skyline33 Aran-Fey's way is probably the best, but another option is to "clamp" the key function. Eg:
def keyfunc(x):
    n = x.num
    return n if n > 0 else 0
But that will give a wrong result if there are negative values but no zero value.
 
@Aran-Fey n.val > 0, right?
 
9:39 AM
n.num
 
Oops, that
 
OTOH, it saves on the generator overhead, and Aran-Fey's version needs a default arg to min if all values are negative.
 
Your keyfunc should be returning infinity instead of 0 though
>>> min([-1, 0, 1], key=lambda n: n if n>0 else 0)
-1
>>> min([-1, 0, 1], key=lambda n: n if n>0 else math.inf)
1
 
@Aran-Fey Even better!
I'd probably fetch math.inf outside the loop, though. Attribute lookups on a global are a little slow compared to using a local.
But I guess that's not worth worrying about, unless we expect lots of negative values.
 
Or a performance bottleneck :P
 
9:48 AM
Sorry meeting popped up...
All the strings are plain ascii, more specifically IP adresses
And the general idea is to mask them using aliases
 
When I see "IP addresses" my inner child wants to ask "have you seen a doctor about it?"
 
Not yet
 
I haven't done any real coding for a couple of months. I guess I'm getting a little rusty... And I need my first coffee of the day. :)
 
@Aran-Fey Just changed a little to work it for list of objects---- min((n for n in lst if n.num>0), key=lambda x:x.num).num)
 
@Jonaswg Fine. I'm just trying to think of ways to shrink the RAM footprint. Using bytes strings uses less RAM than Unicode text strings. And decoding them to Unicode is fast, if you need to do that.
 
9:54 AM
It may be that I cannot achieve multi processing AND efficiently share a dict between the processes
 
@skyline33 Since you only care about the .num attribute, you might as well just extract those with the genexp, and call min on the genexp output.
 
The fact that f.ex "127.0.0.1" should be IP-00001 for every instance of that ip across all the files is the only factor that makes me unable to solve it in a more straightforward manner
 
@PM2Ring any code examples?, new to python :)
 
@Jonaswg Just to be crystal clear, you're replacing "127.0.0.1" with the alias "IP-00001", right? Are they all IPv4, or do you want to handle IPv6 too?
 
I am handling both
I use some regex patterns and re.sub()
and parse files line by line
 
9:59 AM
@skyline33 min((n.num for n in lst if n.num>0) ...)
 
min(n.num for n in lst if n.num>0, default=0)
 
will try that.. ty
 
Assuming you're using at least Python 3.4, which is when max & min got the default arg
 
yea 3.6 here
 
If you can guarantee that there will always be at least 1 item in the list >0, then you don't need the default. And if there isn't a suitable default value for your task you might want min to raise an error if there are no positive items.
 
10:18 AM
@PM2Ring when multithreading do you use the ThreadPoolExecutor?
 
@PM2Ring default = -1 ;)
 
@Jonaswg I guess you could, but I've never used that stuff, just the basic threading module.
 
And how do you avoid the GIL?
 
@AndrasDeak Sure, but I think 0 works ok, since all the legitimate values are supposed to be >0.
 
@Jonaswg well, ... multiprocessing
 
10:29 AM
you dont avoid the gil if you're using threads in python. So, switch to multiprocessing
or well, cpython i should say
could also go for multithreading in one of those niche implementations that dont have GIL.
 
@Jonaswg If the bottleneck is IO, you don't have to worry about the GIL.
 
The bottleneck is not IO
I thought so at first
Did you guys try out keyvi?
 
But as I said at the start, I don't know much about multiprocessing in Python. I've mostly been trying to get the info from you that I believe will be helpful for someone who is an expert on that stuff, as several room regulars are.
 
the fact that multiprocessing brought improvements is already an indicator that this is not I/O bound -- adding parallelism to something where I/O is the bottleneck usually just makes it slower if you add more resources to compete for the limited I/O bandwidth
 
10:41 AM
@PM2Ring ah, right
 
Yes, the issue I believe is related to having a dict shared between the processes
 
10:57 AM
unless you can fix this question. Has morphed from the original asking about multidimensional dicts, so that part should be deleted, or at least reduced to a footnote and the actual question posted at the start. If the question is reduced to sorting (flat) dicts it's a duplicate and should be closed as dupe.
 
11:13 AM
voted unclear
 
@smci Do you have a dupe target for it?
The OP solved it themself, according to EDIT2.
 
Does anyone else see mobile web gmail sprouting phantom drafts of emails that can only be deleted from desktop? Started last week.
 
Sorry. I have a gmail account, but I never use it. And I don't use my other email accounts much either.
 
11:33 AM
@PM2Ring Yes I know the OP solved it after removing the multidimensional aspect and invalidating the code example and leaving a confusing mess. What the dupe for it should be depends on whether we keep the multidimensional aspect or not. If not, the code example needs a chainsaw taken to it.
 
Hello, I have a pandas dataframe with date index incremented by 1h. How could I duplicate a value to all cells within the same day?
let's say, i have on 2019-1-1 at 12:00:00 some value, and I want to copy it to all cells for the day 2019-1-1
 
m8_
You can use .loc
 
any chance that a slice that contains datetimes works as an indexer?
 
@smci with the latest edit I'm thinking it could be closed as a dupe of anything with "how to sort a list of lists on some element within each member list"
 
@smci Fair enough. How to sort a dict of (Py 3.6+) dicts is a reasonable question. OTOH, the OP's final example data is a dict of single item dicts. I positively loathe single item dicts, but they seem to be popular, especially with Indians, for reasons I'm unable to fathom. I guess they don't mind that each 1 item dict consumes 300 or so more bytes than a (key, value) tuple. ;)
 
11:43 AM
might be some weird JS/JSON thing
 
@AndrasDeak Sure, and that's the fault of whoever decided to use a single item object in the JSON, instead of a [key,value] array.
 
I guess
 
Virtually all single item dicts I've seen on SO are connected to JSON.
 
the good thing about single-item dicts is that you can .pop the one item :>
 
There must be millions of those stupid things out there in the wild. A list of 1 item dicts gives you all the extra RAM overhead of dicts, with none of the benefits of fast dict lookup.
 
11:59 AM
@Capie I don't know Pandas, but in plain Python you can use the datetime.replace method to create a datetime at midnight on the date of the original datetime. Then subtract that midnight datetime from all your datetimes & see if the resulting timedelta is <= 1 day.
 
@PM2Ring single-item dicts is a JSON thing, there's nothing wrong with it per se, in fairness that's a criticism of our memory bloat with Python dicts. Seems inevitable with multidimensional dicts imported from JSON.
@PM2Ring but you don't want to start optimizing assuming you're getting a single (key, value) tuple, because then someone might add something to the incoming dict... I mean strictly if this becomes annoying, do the parsing in C.
 
@smci I don't like to think of it as memory bloat. In a JSON file, it's just a bunch of text. But in Python, it's an actual dict so it needs a working hash table.
 
How do I deal with an empty vocabulary using TF-IDF on text fields where some text fields have only 1 or 2 words? I've tried not removing stopwords.
 
@smci Sure, if those single item dicts get expanded into multiple item dicts, then that's reasonable. It's just that I've rarely seen that. OPs want to parse their JSON, search &/or modify items without expanding the dicts.
 
stackoverflow.com/q/55590188/1222951 typo (the answer is literally just the OP's code with the variable names fixed)
 
12:07 PM
@mtbrands What is your use-case? TFIDF is not the right approach to every use case. Can you post a link to example/pastebin/question? (If you have a finite well-known vocabulary, you could use e.g. Jaccard distance). Also obviously if most terms are stopwords, TFIDF is going to do badly
 
@Aran-Fey also no MCVE
 
Textual descriptions with variable length. Using LSA to get an average description out of multiple. I have no example ready, sorry. The vocabulary is not well known and changes a lot.
 
@mtbrands since the D in tfidf stands for "document", I'd say that the algorithm simply might be the wrong tool for the job. two words hardly make for a good document.
 
Very good point, but so far I've found no more compatible algorithm.
 
@mtbrands I can't understand your use-case without an actual example. Yes, two words is not a proper document. TF is just going to reduce to something like one-hot or Jaccard.
 
12:16 PM
@Aran-Fey you map different characters to the same digits... this "encryption" of yours is not reversible :facepalm:
 
It takes texts like: 'big house, red roof, two windows.' Many variations of this with things added or left out. After using TFIDF to do LSA it will return an approximation of the 'average' text that was entered. And this does work in some examples. However in some other cases it could be like 'red car' and this would result in an empty vocabulary. Even when using CountVectorizer instead of TFIDF
 
@mtbrands if there is just not enough signal, there may not be any reliable algorithm
if the input is "hello", how would you reason about whether "hello world" or "you had me at hello" is a closer match?
 
I have a technique for encrypting data that will never be decrypted. I call it the Algorithm of Amontillado.
 
(you could say shorter is better because less signal too :-)
 
I'm reminded of that SigBovik paper that encodes 1000 words to a single emoji, in accordance with the old saying that a picture is worth a thousand words. ;)
 
12:21 PM
(on the other hand with TFIDF the weights of "you" "had" and "me" will be approx zero)
 
yeah could be that im just trying to find the algorithm that makes the result 'bad' instead of 'utter crap'
 
@mtbrands 'red car' / empty vocabulary. Yes that's called the Unseen Word problem in NLP. What do you want to do? use an ontology to map 'car' to the nearest word you've already seen ('vehicle'/'auto')? Ignore unseen words? Make a wild guess based on Levenshtein distance? You can google on "unseen word NLP", there are 362K hits... ;-) Tell us what you decided to do.
 
Anyhow, thank you, needed some back and forth to get my thoughts straight.
Oh that's a neat pointer!
Big part of knowing what to search for ;)
 
was I too harsh in hammering this? stackoverflow.com/questions/55591926/…
 
Sure. My NLP expert friend told me. But honestly, TFIDF is often the wrong tool. Btw "After using TFIDF to do LSA it will return an approximation of the 'average' text that was entered." sounds like a dodgy way to do clustering. Sometimes your label will not be representative of the whole cluster. It's ok for things to genuine outliers to remain outliers, you know...
 
12:27 PM
@Kevin A couple of days ago, I learned that Larkin Poe chose their name because the sisters are related to Edgar Allan Poe. Here they are performing Son House's cynical Preachin' Blues.
 
Yeah the data I'm working with would fit in nicely at a garbage dump so it's a lot of making do. I'll work with these pointers
 
@tripleee If the OP doesn't agree, we can unhammer, or find a better target.
 
@mtbrands LSA is a clustering-based approach, so if the data has lots of outliers or doesn't really exhibit clustering (at least with a metric like TFIDF on very short sentences), you're using the wrong tool. At least, stop forcing dissimilar things into the same cluster: when you hit a knee on the curve.
 
@PM2Ring it should perhaps remain closed as no MCVE for now anyhow
 
@tripleee That's fine, see my comment backing you up. I don't see how what looks like an internal C bug on paramiko implementation is a Python issue, it's a paramiko bug. Maybe post them the URL to where to report paramiko bugs. Unless the OP can show an MCVE proving otherwise.
@mtbrands Another search term for Unseen Word is "out-of-training-vocabulary (OOTV)"
 
12:37 PM
I don't think this is really a Paramiko problem either ... and "system libraries" is weird and probably incorrect
 
@smci Great, thank you so much
 
@tripleee Agreed.
See this Physics Meta answer for my philosophy on dupe hammering.
 
@PM2Ring Undeniably talented, but not quite in my wheelhouse. Let me know if they make a nightcore remix.
 
12:54 PM
@Kevin Ok. :) They have a pretty wide repertoire, but it is mostly bluesey, or blues-rock. They do lots of original material as well as covers.
 
1:30 PM
Hmm, moderately annoyed that the second highest answer to the canonical "how come for loops don't get their own scope?" post basically says "because Python doesn't have blocks" when in fact Python does have blocks and they are the fundamental basis of scoping
They're in paragraph 1 sentence 1 of docs.python.org/3/reference/executionmodel.html so it's not like the docs are trying to hide their existence
(never mind that I spent twenty minutes looking through the docs for the definition of "block" before I found that passage. Who puts important information in the introductory paragraph, I ask you?)
(Curse my rote literary education that taught me that the first and last paragraph of all essays contain only useless formalities)
 
Ok, I have done so.
 
thanks
 
Today I am annoyed by users that pass a dictionary of length one to dict.update
And also by users that do if my_boolean_variable is False:
 
ugh, latter is much worse
 
1:43 PM
\o cbg
 
I treat my booleans as if the devs will decide tomorrow that they don't want them to be doubletons anymore
If your code doesn't assume that True may have a different id every time it gets evaluated, you're not paranoid enough
 
return True if <insert condition here> else False
 
@Kevin I forgot to mention that in my earlier rant against 1 item dicts.
 
It's kind of justifiable in the case of stackoverflow.com/questions/55594072/…, if OP assumed that dict.update returns a new dict rather than mutating the old one and returning None
 
if <some condition>:
    return True
else:
    raise BlahBlahException
 
1:48 PM
If OP thought return d.update({key:value}) was the correct logic, replacing it with return d[key] = value would obviously not work
 
@Kevin What's wrong with if my_boolean_variable is False?
I do it all the time, I like the explicit nature of the statement.
 
is is the black sheep of the operator family because 99% of the time, Python should pretend that pointers and addresses don't exist. You shouldn't throw away a good abstraction unless you have a good reason.
Also it's discouraged in PEP 8, if you're militant about that kind of thing:
Yes:   if greeting:
No:    if greeting == True:
Worse: if greeting is True:
 
@stschindler Preferred is if not my_boolean_variable: If that doesn't read well, then your variable is poorly named.
 
@Kevin I suppose so, although I take a dim view of OPs who don't spend 30 seconds looking at the docs instead of composing & posting a SO question.
 
Oh, and cbg
 
1:55 PM
Comparing booleans with == doesn't sound like a good idea to me...
>>> 1 == True
True
>>> 1 is True
False
 
hmm?
oh, I see
 
If you've reached maximum levels of paranoia, your if statements have to look like if type(greeting) is bool and True == greeting:
 
And woe betide unto you if you name a variable is_disabled, and even worse if you Javasize this to def set_is_disabled(self, disabled_value): and then later have to untwist obj.set_is_disabled(False)
 
@PaulMcG What if the value can be True, False or None? I've been bitten by that one already.
 
is None -> handle the rest
 
1:58 PM
Sometimes I just want to check for False, for example, so is None isn't needed.
 
When something can be True and False and None that should usually only happen if None is a default. False and None both being valid values is asking for trouble.
 
It's only asking for trouble when you're not being explicit IMO.
 
@Aran-Fey I'm trying to decide whether to point out that isinstance is typically preferable over type(thing) is whatever. I suspect you already know this and have rejected isinstance because it returns True for subclasses.
 
A use-case would be a flag succeeded, that can be None (nothing happened), False (no success) and True (success).
 
if succeeded is not None and not succeeded: should be clear enough for that...
"something happened and it was a fail"
 
2:03 PM
@Kevin Yeah, it's because of the "paranoia" thing. I could take it even further, too: if ((type(greeting) is type(True)) and (True == greeting)):
 
I find if succeeded is False to be more readable.
 
@Aran-Fey Assuming singleton types? Bold!
 
bools are un-subclass-able so in this particular case I think it's safe to use isinstance
@stschindler OK. It's fine to selectively ignore PEP 8 as long as you're aware that you're doing it.
 
:-)
It's not the only PEP rule I'm breaking. ;-)
 
I have never obeyed "keep lines below 80 characters" and only a substantial cash bribe can change my mind
 
2:05 PM
Yeah, it's ridiculous to enforce such things IMO. Although I keep my lines at 79.
 
@stschindler I am deeply unfond of the "nullable boolean" pattern. If there are three possible states, I'd rather use an enum.
 
@AndrasDeak Not as bold as assuming someone might write a metaclass with an overloaded __eq__ implementation and set greeting to an instance of that!
 
@Kevin Sounds like a good alternative.
 
Or possibly I might write my function to return only True or False, and raise an exception for the third state.
But despite all my preaching I bet I've got a couple true/false/nones hanging around in my various code bases.
 
Blasphemy! :-)
 
2:14 PM
@Kevin I'd vote for this is as the canonical True/False/None refactor
But if doing True/False/Maybe fuzzy logic, I'd go with the enum
 
My distaste for nullable bool is probably partially a throwback to my time using languages where "objects" are easily nullable, and "primitives" aren't, and booleans are primitives.
The other half of my distaste is the Oracle database for my work projects where every bool column is nullable for no particular reason and this makes the ORM layer ten times harder to implement
 
So let me get this straight... your distaste stems from the fact that it tastes bad?
 
@stschindler Really? Instead of if not succeeded: ???
 
2:29 PM
In the specific context of nullable bools, if not succeeded: and if succeeded is False: behave differently when succeeded is None.
@piRSquared I typed succeeded = None into an empty Notepad window and licked my monitor and it tastes just awful
Hmm, at what dose does monitor cleaning spray become harmful to humans?
 
Idk, but my colleagues are looking at me like I'm a freak right now.
 
if succeeded in (True, False) and not succeeded: ??? This is the kind of coding up with which I will not put.
 
Someone wrote code like that? Perhaps they were tired and drunk, or both
 
No, that last I haven't actually seen, I was responding to @kevin 's comment about nullable bools being a confounding factor in writing 'if not succeeded:'
 
2:44 PM
maybe someone who wanted to say isinstance(succeeded, bool) but didn't know the idiom
 
Wis
@AndrasDeak i'm only interested in getting the path from the commandline, not parsing all arguments? it's simpler to do the former right?
 
@AndrasDeak just here to say thank you for the suggestion you gave me. using databases instead of pickled files has made my life much easier.
 
@Nasrin glad to hear that, no problem :)
@Wis probably yes but you were asking about "mak[ing] a python commandline parsing library"
 
Python has a reputation for being easy to write and easy to read. Just about every language is easier to write than it is to read, but Python has been designed to reduce that gap, as far as is practical.
 
3:01 PM
And I strive to widen that gap with splats, comprehensions, and named lambdas.
 
One of the core principles in the Zen of Python is "There should be one—and preferably only one—obvious way to do it". True, that principle has eroded a little in a few areas (eg, string formatting), but by & large it's still one of Python's great strengths, speeding up development by making code both easier to write and to read.
Many of the PEP-8 rules, like the one about testing bools, inform users as to what that obvious way is, in case they aren't Dutch. ;) When I read code that disobeys those guidelines, it tells me that the coder is either a clueless newbie, or they're intentionally doing non-standard stuff.
In both cases, it means I have to slow down & scrutinize their code more carefully. They're probably doing other not-so-Pythonic stuff too, like not keeping with the spirit of duck typing.
TL;DR: Try to stick to PEP-8, it'll make it easier for people to read & understand your code. Including yourself, in 6 months.
 
I was introducing Python to my group (of researchers) a few years back. My supervisor suggested that one of the others give an IPython Notebook a whirl while I guide them (horrrrrrible idea). I got stuck trying to instruct them to put spaces after commas. They soon became irritated with me and started doing it intentionally.
 
"Why do I need the spaces? It works without them." "Because you're writing so that humans have an easy time reading it. If you want to make it easy for the machine to read, write in assembler."
And may Grace Hopper have mercy on your soul.
 
I didn't read the style guide until like two years after I started using Python. So you can imagine that I'm a proponent of the learning style of "learn how to write correct programs, then learn how to write pretty ones"
 
Oz takes the top spot for average years of coding exp.
 
3:12 PM
"But it's harder to unlearn bad habits than it is to learn good habits at the start", you might say. To which I say: meh, I think the difference in hardness is overblown
I started with a myriad of bad habits and ten year later I'm almost completely normal.
 
Wis
are one function modules ok?
 
I vote yes
 
I occasionally have one-function modules, sure.
It's not a common occurrence for me though. Don't go out of your way to isolate all your functions in their own file if doing so feels awkward at all.
 
I'd suggest forming a consistent pattern of how you do things. If and when a one function module falls into that well thought out pattern, then you're all good.
 
(here I reveal my ignorance of the fact that when you're in Coding Adolescence, everything feels awkward)
"Just use your instinct for code smells that you have developed after years and years of practice, easy"
 
3:16 PM
Hey, I started out in Dartmouth BASIC. We had single letter variable names. And early C didn't exactly encourage me to use longer names. Several of my early languages permitted longer names, eg PL/I permitted upto 31 chars, but only the first 6 chars were significant. It took me a couple of decades to get into the habit of using more meaningful names, and I still have the tendency to backslide.
 
Nonsignificant chars, ugh
 
@Wis I'd happily do the equivalent in C, but it feels a little awkward in Python. Unless there's a good reason, eg it's a really useful function, & you probably want to use it in several programs. And you can't think of any other functions that make sense to live in the same module.
 
Wis
so a module for killing other instances of the same script shouldn't depend on a module that tests if the cmdline is of a python interpreter with a regex and another which gets the path from the cmdline, but one "pythoncmdline" module which with both functions?
 
@AndrasDeak I'm sure some early Fortrans did that too, back when core memory was actual magnetic cores.
@Wis To me, it makes sense to bundle them together, since the script killer can't work without the script recognizer, and the recognizer isn't much use by itself.
 
@PM2Ring I know 6 chars was a limit; I've never heard of allowed-but-ignored characters beyond. Might easily have been the case, but I'd expect users to not use such ignored characters anyway.
 
3:29 PM
Permitting more characters but clipping them afterwards seems quite odd to me. If you're resource-constrained to the point that you can only have 6-character variable names, surely you could just have a preprocessing step that losslessly maps long variable names to short-yet-meaningless available names? Or are you so resource-constrained that your compiler can't have such a feature?
 
preprocessors were only invented in the eighties #probablyfakenews
 
E.g. if you have variables named a, b, and my_cool_variable, then replace my_cool_variable with c.
 
One thing I loved about the PL/I compiler is that when it hit a syntax error it would try & keep going to identify as many errors as possible. If things were really bad, it would eventually get confused & start spouting rubbish, but often it could find all minor errors, and correctly identify their exact location in the source.
It would print each line of your code (i.e., each punch card), and any lines that had errors would get an error line underneath, consisting of a letter code under the offending source char. At the end of your source printout, all the relevant letter codes would be printed, along with the meaning of the error code.
@AndrasDeak Indeed. It was one of those clever ideas that ended up being deadly in practice.
 
Now I'm leaning towards "preprocessing steps that make a modified copy of the source code is a no-go, because we don't even have enough memory to store the original source code all at once"
You want your compiler to be single pass unless you like feeding in punch cards more than once
On the other hand, "It would print each line of your code" implies that you can store one copy of the source in memory.
 
@Kevin It's like a dict that maps strings to symbol table slots, but only the 1st 6 chars of the string are hashed. Sorry, I can't remember the technical details, but yes, we didn't have a lot of RAM to play with. 64kb was typical, 256kb was a big machine. You could by a block of houses for the same money as one of those mainframes. ;)
 
3:43 PM
Theory 2: all of the cool optimizations I thought of are technically feasible and ultimately were implemented... In other more sophisticated languages. Back in the day, upgrading one's compiler wasn't something you would do just for a couple of QOL improvements.
 
These machines did have tape, or disk, so cards could be read to that storage, I guess. But IIRC, it was single pass compilation, writing an object file to disk, and then running the linker on that object file.
The IBM mainframes were leased, not sold. Along with the hardware, you got an on-site engineer, in the iconic IBM sky blue suit, as a permanent member of your staff for the duration. He'd turn up every day, go to the company Christmas party, etc, but his pay cheque came from Big Blue.
 
which gave rise to the idiom "out of the blue"
 
Maybe the problem is not space but time... It wouldn't take all that much memory to store a table that maps between long descriptive variable names and short-yet-meaningless variable names. But perhaps doing an extra table lookup for every variable in the source bogs things down to a measurable degree.
Especially since the long-to-short lookup requires you to hash the entire string and not just the first six characters.
 
Wis
I got the underscore and lowercase right from PEP 8, but isn't script_path_from(cmdline) more readable than script_path_from_cmdline(cmdline) or should I use of instead of from, or should it just be script_path(cmdline)?
 
We had a backup copy of the PL/I compiler on cards. I think it filled a standard box of cards, about 1 metre long. You wouldn't want to drop that box & spill the cards... Cards did have space in the last 8 columns for sequence numbers, but that feature was often unused.
A couple of friends were writing a Minitran compiler (a cut-down Fortran). One day, they accidentally dropped around 2000 cards of assembler code, without sequence numbers. They just looked at each other with a look of painful resignation & started throwing the cards into the waste paper bin.
 
3:58 PM
@Wis Perhaps extract_script_path
 
@PM2Ring so, so preventable...
like not backing up your data...
 
I know that sometimes devs would draw a diagonal stripe along the top of their punch card deck, to make it less impossible to restore order after a spill. But maybe that's not so practical for meter-long decks.
 
"This is statistically unlikely with a sample of over 70,000 developers who answered this question, to put it mildly."
SO Savage.
 
@Kevin neat workaround
 
@AndrasDeak Very. And I'd even written a very efficient assembler program that could print a listing of a deck of cards & produce a duplicate deck with sequence numbers. But they declined to use it. Yamming cowboys!
 
4:02 PM
A sufficiently slanted diagonal will become hard to distinguish by eye when the difference between two punch cards is narrower than the margin of error due to ink bleed-through
 
@PM2Ring it must have been educational...
 
@Kevin Yep, that was pretty standard. And my Minitran friends did that. It's quite helpful for high level source code, where you can easily correct minor sequence errors, but not so useful for large assembler sources.
@AndrasDeak It was. ;)
 
Mm hmm, I was just thinking that it would depend on the language whether you could manually turn an almost-ordered program into an actually ordered program.
 
I was quite proud of my duping & sequencing program. It used a little bit of self-modifying code so I could reuse the same loop that dumped a card to the line printer to punch the duplicate card. This is before device drivers were invented, so you had to talk to IO hardware directly.
I assembled & linked the assembler program to machine code by hand, to squeeze it down to 120 bytes, so it could fit on a single card using a special encoding mode that was normally only used for cold-booting the machine.
Those were the days. :)
 
I'm reading about how the card reader could fail if the punch card lacked structural integrity due to too many holes... I'm getting flashbacks to INTERCAL's "program is insufficiently polite" and "program is too polite" error messages
Your program is syntactically valid, but the compiler has elected to barf anyway. Try something else.
 
4:17 PM
I suppose the closest modern equivalent is hacks that get read from a disk's MBR. Or on a slightly higher level, programs like Memtest, that run in the same sort of mode as the BIOS startup stuff.
@Kevin Yes. That was the main reason that high density mode was rarely used. The cards could end up with the structural integrity of a paper doily. Luckily, the bit pattern of my program wasn't too fragile, but you still had to be gentle with such cards. And we always had a couple of spares of the bootloader card on hand, since they had a fairly short useful life.
 
Bit rot, meet thy ancestor: actual rot.
 
4:47 PM
@wim interesting Q. Simple multiple assignment statements like a,b,c = 1,2,3 appear to store the target list size in the argument to its UNPACK_SEQUENCE bytecode instruction. I think you can extract that from the frame object. More complex assignment statements, such as a, *b, c = 1,2,3 appear to use UNPACK_EX instead, whose argument is more arcane to me.
But perhaps you already know this and are more interested in whether the inspect or ast modules make this information available in a neat little package. I haven't found anything during my perusal.
 

« first day (3097 days earlier)      last day (1843 days later) »