« first day (4955 days earlier)   

3:03 AM
@deostroll Parameters after “*” or “*identifier” are keyword-only parameters and may only be passed used keyword arguments. stackoverflow.com/questions/2965271/…
@KarlKnechtel Just call them 'kwargs', already. Like the doc does. Keep the name distinct to 'keyword', as Aran-Fey says.
 
@Aran-Fey if, while, for, def, import, etc are reserved words.
 
3:20 AM
@PM2Ring The doc calls them keywords, and most other languages call them keywords, so we should too. (We can also draw a distinction between actual keywords vs conventional names: 'self', 'args', 'kwargs', 'cls'...
...And then there are the conventional names and aliases for packages/modules, which it's recommended practice not to shadow: string, import numpy as np, import matplotlib.pyplot as plt,import seaborn as sns, import pandas as pd...
 
4:06 AM
@smci Yes. That's unfortunate. But it does say "reserved words, or keywords".
 
 
1 hour later…
5:10 AM
Generally programmers call them keywords. If you type help() then keyword Python lists you all 35 hard keywords. If you import keyword you can then see the 35 hard keywords inkeyword.kwlist and the 4 extra in keyword.softkwlist: _, case, match, type
155
Q: Is the list of Python reserved words and builtins available in a library?

Neil GIs the list of Python reserved words and builtins available in a library? I want to do something like: from x.y import reserved_words_and_builtins if x in reserved_words_and_builtins: x += '_'

(doesn't include builtin types, or builtin functions)
(doesn't include builtin types, or builtin functions, or builtin exceptions. Can access those with import builtins: 158 more as of 3.12)
 
5:33 AM
(and then there all the conventional names for dunder methods and dunder attributes. See simonwillison.net/2024/Mar/20/every-dunder-method-in-python )
 
 
2 hours later…
7:56 AM
I notice that django is terribly slow, like really really slow taking even for a simple request multiple milliseconds to make a response.
But worse is that when you start having database with gigabytes of data the orm just blocks up everything for even a simple delete of ±100k items, it first tries to verify in code that you can actually delete and built up the models.

Is there a framework in python that is meant for speed, and big data? where requests for a server on the same physical server actually become sub millisecond overhead?
 
8:13 AM
@paul23 milliseconds? Oof.
 
8:55 AM
@paul23 What's your use-case? Are you often deleting ±100K items?
 
Well we havea lot of financial data. Tertiary parties sometimes delete/add those weekly to their ERP, so we need to update then our database and since some erp have no identifier to the old data we do that. And yes some companies have 100K+ invoice "elements" per week.
It's mostly a cost analysis, of course we can buy bigger azure clients but that becomes expensive, and really: why does a delete have to take more than 5 seconds anyways? In sql it takes about 5-15 seconds, the rest of the 15 minutes is django doing things.
 
@paul23 Why are you using Django for that? This is cloud, not local, I take it. Postgres should be better.
 
uhhh wut?
of course we also use postgres, but you kind of need an interface as application?
and we use celery to run our daily tasks.
-- but it also needs to be called "on demand"
 
If all you're doing is searching by (say) company and date-range, that can be done with a Postgres command, probably a one-line one. Django is adding overhead.
 
@smci ..
Of course I know that, but we obviously need a web interface. On top of that it's not a matter of a "single place single command", it's like 100s of places and queries to delete are build up based on what parameters are given (time period to delete, can you delete data that is being used somewhere? etc etc)
and of course we can use a raw sql statement, we are thinking about that.. But there is a reason you want to have query builders and an ORM; so I wonder if there is an ORM that is built for speed.
 
9:13 AM
What % of those ±100K deletions could be handled by raw SQL statement? Why not do the bulk deletion in raw SQL, and the corner-cases in Django then?
 
9:29 AM
because that is a lot of maintenance in code
you now have to make sure both keep working
when changes happen you have to verify that all the raw sql statements work, and that for example the "filter", which is done by django statements, correspond to the raw sql delete (as they are named the same)
 
10:05 AM
@paul23 Performant alternative: every Saturday at 2am (or whenever), take the Django offline, lock the database, do the mass deletion in SQL, unlock the database. If your use-case allows. What % of those ±100K deletions will this handle?
Try to reduce the mass deletion to a simple common case statement that catches most of the use-case.
 
10:29 AM
@smci good luck explaining that to a global company that they can't handle their invoices nor compete in the high frequency trade market.
we already have a working system, but the clients do need 100% (or well 98.5% is the actual number we have to provide) uptime
 
@paul23 They're doing trades 24/7? Not just in trading hours?
 
Yes the automatic trading does, and they are not located to a single country so it's always at some place in the world trading hours.
given global customers have databases based typically for each country separate as otherwise the lag is too much.
 
11:25 AM
@paul23 not that I know of. You're still instantiating thousands of python object. I don't see how you can have it both ways
Even if someone follows the recent trend of writing the core engine in rust, it still needs to pass it back to python so that seems like a non-starter, at least to my understanding
 
12:12 PM
@paul23 98.5% uptime in a week still allows it to be offline for up to 2.5hrs/week. You could find the quietest 10-30min interval over a weekend across all timezones to do the deletion. Saturday evening UTC-5 is Sunday morning UTC+9.
 
1:00 PM
How do you guys handle if you have some calculation which you need in the frontend and backend to avoid having the code duplicated? We could do the request to the backend with ajax, but that feels wasteful, is there a better way to avoid the duplication but to still have it efficient?
 
 
2 hours later…
2:47 PM
Do you guys prefer for a checksum to get the 4 first and last digits and ... inbetween or get the first or last 8 digits of a 64bit checksum?
 

« first day (4955 days earlier)