« first day (4538 days earlier)      last day (401 days later) » 

1:47 AM
@AndrasDeak--СлаваУкраїні no
 
 
6 hours later…
 
2 hours later…
9:32 AM
cbg
I am trying to append a dataframe to the end of another
both dataframes have the same index - and that is throwing the following error - Reindexing only valid with uniquely valued Index objects
I don't care for the indices, they are just numerical starting from 0
would ideally reindex as part of the append
any thoughts?
 
Did you set ignore_index=True?
 
9:57 AM
yes
self.df_raw_data = pd.concat([self.df_raw_data, df], ignore_index=True)
 
Index values can be non-unique? What the heck is the point of an index, then? O.o
 
The point would be on an index collision and just have it re-indexed. They don't really serve anything close to the purpose of an index in, say, SQL when it's just a standard numerical index for each row
Do you have a multi-index @Andy?
 
11:09 AM
@MisterMiyagi That seems a really fair appraisal of the situation :)
 
11:37 AM
@roganjosh no
i am just reading excel sheets into dfs and then trying to combine them
this is why the indices all started at 0,1, 2, etc - but I want them appended not joined on index
 
 
3 hours later…
2:24 PM
What's the standard way to keep logs regarding actions in my API? E.g. someone edits his user account on "2023 Jan 30". Where should I save that?
 
@Andy Did you try this answer?
@Riya what do you mean by "standard" here? There are a bunch of ways of storing logs for different things e.g. you might want it in a format for massive analysis with something like Splunk. Or perhaps you just want logs for the last few days, in which case you can just store them into a designated directory somewhere as a text file
If you use flask, for example, then there is built-in logging functionality, or you could implement your own like a RotatingFileHandler for something a bit more "advanced"
 
2:46 PM
There are 3 tables in my DB: User, Service, and UserService which links User and Service and its only useful data is 'purchased_date'. So I was wondering if the 3rd table is useless and can be deleted. But to do that, I need to first separate the logging.
 
An entry in a log versus database are quite different things. Are you sure logging is appropriate? If the data isn't needed, are you sure you are allowed to log it?
 
do you mean application logs like "[INFO] successfully connected to database, now reading data.." or a database history log that lives in a db and archives updated/deleted rows?
 
@Riya is that an Association Table?
Agree with Arne and MisterMiyagi here, and I now realise it probably couldn't be an association table here because the linking field is a timestamp. These are database histories. While UserService is something I find to be an odd name (rather than ServiceHistory), deleting that data seems like a really risky thing to do to me and not something I would consider to be "logging"
 
3:05 PM
The UserService table contains 3 columns. User UUID (foreignkey), service ID (foreignkey) and date purchased. (and it's row id which i dont use at all)
 
The flip side to "are you sure you're allowed to log it?" would be "are you sure you're allowed to not log it?". There could be tonnes of different legal obligations behind the need for timestamps and it looks like it could be standard database normalization
@Riya which looks a lot like the association object pattern. That's what I meant to say, not "association table" which confused me when it didn't meet my expectations on searching it
A really simple, real world example in the media a lot right now is all the enquiries into social media harm to teens. Even when it's not a paid-for service, companies can be hauled up to enquiry and they have to give solid figures on patterns of users (which have an ID) interacting with different post contents (also have an ID), and a time they did. They have to have these numbers.
That's not at all meant to spook you, just to say - tread carefully before you consider that such tables can just be removed because there could be obligations behind its existence
 
Thanks everyone. Your feedback is much appreciated.
 
@Riya see also Chesterton's Fence.
> Do not remove a fence until you know why it was put up in the first place.
Chesterton went on to explain why this principle holds true, writing that fences don’t grow out of the ground, nor do people build them in their sleep or during a fit of madness.
Of course this often doesn't apply with software due to the sheer amount of incompetence in the industry
 
 
2 hours later…
4:53 PM
@roganjosh thanks will try
 
5:30 PM
@Andy I tried to repro it in a few different ways and couldn't do it. I know that the problem exists, that much is true, but I couldn't reverse engineer the problem in a way that I thought was sensible
Some context is missing
Completely separately; is there a way to try screw with pre-commit hooks? I leave my current position at the end of this week and they're joking about adding black the first thing on Monday morning on my library. I wonder whether there's a way to put a jokey PR in that can get at that machinery
For educational purposes
 
5:57 PM
Yo, still haunted by OrderedDict with Dict
Does the insertion order preserving version of Dict still use PYTHONHASHSEED ?
Trying to find differences between OrderedDict and Dict with the intention of removing all non deterministic code paths
 
@Mikhail I'm sure it does, because it uses hashes. In fact dicts are probably the reason for PYTHONHASHSEED.
 
but OrderedDict doesn't?
 
That's not what you asked nor what I said
 
So, I'm confused. How does PYTHONHASHSEED effect Dict order post 3.7?
 
Same way as pre 3.7?
Dict uses hash, hash gets randomized.
Or so I assume :P
@roganjosh do you already have pre-commit hooks? And if yes, is it voa the pre-commit library?
 
6:10 PM
Hello. I have 4 bytes - e8 03 48 b7 . This 4 bytes contains valid lolngitude (which is around 30.xxxxxx) . In docs written "It is a value calculated by converting to decimal which is further divided by 1,800,000" and I have no clue how to interpret that statement as I cannot obtain correct value.

int.from_bytes(bytes.fromhex("e8 03 48 b7")) / 1_800_000 # wrong, both byte orders
 
@AndrasDeak--СлаваУкраїні Neither. I just badger people in their PRs
 
I'm sitting with this crap 2 hours and can't decypher damn 4 bytes :D
 
Hmm, so pre 3.7 the order of the dict was purely determined by the hash. Post python 3.7 dict is now insertion ordered (perhaps order is implemented as a supplementary lists IDK). So what does the hash effect on the interface side?
 
@Mikhail Affect. And you're looking at it the wrong way IMO
The key is always going to get hashed. Pre- or post-3.7. The ordering of the dictionary is stored in a separate, ordered, collection in the latest implementations.... I was going to try dig up various articles on why this is, but this answer seems to cover most bases, so that might be a better start?
Actually, it doesn't answer your question. I think there are two parts; the key is hashed (1) and the order can be preserved based on that hash (2). The former can be trivially set at runtime for repeatability simply by setting a seed. That defeats the point of the seed being randomised each time you invoke the interpreter, but there's still utility in making such things repeatable
 
One reading is that for regular Dict order is preserved but not across PYTHONHASHSEED numbers?
But in OrderedDict order is preserved independent of PYTHONHASHSEED?
Yeah I saw the SO link but it doesn't answer the critical question of "is ordered preserved" across runs (where PYTHONHASHSEED is changed)
 
6:23 PM
Order is preserved because each run is internally consistent?
The interpreter is invoked -> it gets randomly seeded -> everything that happens in that session is consistent -> the interpreter dies and you're back to your shell?
 
So one reading (not sure if right) is that the hash value (modified by PYTHONHASHSEED) is only used internally so the interface is still insertion order preserving and thus invariant between runs?
 
The state of a dictionary is not typically conferred between runs of a program. You have persistence for that, and it shouldn't be the default python hash algo of the data (unless I'm missing some edge case)
@Mikhail The dictionary doesn't exist before you run the program. The dictionary is entirely within the python session. If you pull the data from an external, persistent, source, it'll all be consistent for the lifetime of that program. Once it dies, it'd be unintelligible to another run of the python program. But that wouldn't matter, because you have some persistent storage somewhere else
 
Sure... But I guess I'm trying to figure out if ordering of items() will change between runs on 3.7. Before it would change depending on the seed...
 
The ordering doesn't have to change because it has another construct that maintains the order for the lifetime of that program
 
Yeah I was also kinda thinking that now order is preserved across runs, but wanted somebody to confirm that this is correct.
 
6:33 PM
Within each run of the python program, it has ways to be internally consistent. In an imaginary world where you invoked the python interpreter twice on the same memory, it wouldn't work. But it lives and dies in a consistent state each time
 
So internal ordering of the dict doesn't really matter that much compared to ordering on the interface level (which is now maintained as separate insertion order list?)
 
@Mikhail Don't know about dicts changing order across versions, but I do remember we talked about set() here, which does have different ordering behavior across one or two versions, I think.
 
Yeah set isn't ordered yet, although hope is that OrderedSet is ordered
 
it isn't, but it technically does for most ints (see discussion linked) although it's only a byproduct of the way it hash them.
 
Yeah so this is really confusing.
From this chat we still don't have a direct answer if insertion ordering is a guarantee of interface level ordering (from .items(), etc) across python script invocations.
 
6:39 PM
It is guaranteed from 3.7 onwards
But I'm failing to give an intuitive explanation of why, so maybe someone else should pick it up
 
@Mikhail the only direct answer I could think of is to thoroughly study the Cpython code across versions
 
Each invocation of the script performs insertion in the same order so order is preserved across program runs? Even if hash changes, the hash order is no longer used as part of the interface?
 
No, because you're imagining state being transferred between program runs. You cannot do that in base python. Each run is independent and only has to be internally consistent
a = {'a': 1, 'b": 2}
print(a)
 
I don't understand how "state transfer" is relevant. I would expect the order to be independent of state, but only dependent on insertion order?
 
I think what Mikhail meant was "are each invocation giving the same ordering across different runs?", so when thinking that it could use seed (without first checking and only as assumption) and if that seed is a fixed seed, then yes, it doesn't change
 
6:43 PM
Guess what - I created that dict within the interpreter run. I didn't pull it from anywhere. It makes no difference to me how the keys are hashed, because if ordering is guaranteed, I'll always get the same order back out
Each invocation creates its own ancillary collection to monitor ordering, regardless of the hash used. But it won't transfer between invocations.
 
@Mikhail it depends on how it's implemented. I don't think I know how dict is clearly implemented to give good feedback, aside from a couple of test cases and empirically say that "the ordering doesn't change". Now whether or not that's true for all versions or all inputs...that's up in the air unless you check Cpython's repo.
that's only if you really doubt or want to make sure. Otherwise, taking the docs at heart might be good enough (even if it tends to be not documented enough to me, but that's beside the point)
 
I don't think Mikhail is questioning the veracity of the docs
 
right, I was mostly the one saying that...I mean, the docs isn't necessarily diving too much into how the seed/ordering part of the dict work across version
 
I've now managed to get myself confused about the temporal side of things within a session, so I definitely am backing out for a bit
 
7:00 PM
I really hate it when this happens. You spend a while crafting a good answer to what appears to be a specific question, then the OP comments that their example isn't what they're actually looking for. Grrrr....
 
7:15 PM
@MattDMo, to be honest it's not a big deal in this certain case as it will require 1 more line of code to satisfy "new" requirements
 
What new line would you put in? I'm thinking of making a new list of replacements as well as a list of sample input, zipping them all together, then using raw f-strings to create the search and replace patterns.
 
@MattDMo, maybe I missed something, but generic answer will be to make dictionary mapping original - replacement and pass lambda which calls dict.get on matched string.
And use ignore case flag
 
I think using zip will accomplish essentially the same thing. I'm testing code now...
 
7:40 PM
@MattDMo tell them to ask a new question
 
@AndrasDeak--СлаваУкраїні I had thought of that. I will now.
 
@MattDMo Probably 80% of all parsing questions are this. Welcome to my world.
 
"Oh, I forgot to tell you, there's also this: ..." Yeah, I feel for you.
 
8:08 PM
If anyone's interested, I came up with a more general solution for replacing bits of strings that contain braces - just use re.escape. I've edited my original answer.
 
8:38 PM
@MattDMo I'm wondering why do you need re there at all
 
@OlvinRoght They need to ignore case, which you can't do with str.replace().
 
 
2 hours later…
10:32 PM
Folks, I need a schema (JSON or XML) for storing information about books.
I can devise my own, of course. But I bet something like that already exists in the public domain.
Is there a catalog of common or standard schemas out there somewhere?
 
Isn't that going about it backwards? Wouldn't you usually start with a class or database table, and create your schema based on that? JSON/XML are for representing data as text; why are you starting with those rather than the data itself?
 
10:52 PM
This will be a data collection effort by quite technical people. The users are electrical and mechanical engineers.
Initial "generation" of data will be collected using common text editors (think Notepad++ or VS Code).
Processing will come later. In this case figuring out what should be collected is harder than figuring out how to process.
 
11:38 PM
@roganjosh so what would "put a joke PR in" entail? Probably not, anyway. Pre-commit hooks live in .git/ which is not versioned. You need opt-in from each user by pip installing pre-commit or using some other means to install hooks versioned somewhere else. I don't think you can trigger having client-side hooks with a single PR being merged. You can still have a CI task running black :P
or similarly, server-side hooks
 

« first day (4538 days earlier)      last day (401 days later) »