« first day (4545 days earlier)      last day (389 days later) » 

1:05 AM
I just saw a link to a Python compiler project called "Codon" - here is an MIT Press article, with link to the GitHub repo. (Make special note of the license, it comes with strings attached.)
Lots of breathless statements ("10X faster!", "faster than C"), but they do do some actual timings of Python genomics programs, and ran 5-10 times faster when compiled with Codon.
 
has anyone seen "shadowranger" or "tjdelaney" around here? They seem to be active and insightful curator types in the Python tag but I don't think I've noticed them in the chat
 
 
3 hours later…
4:37 AM
@KarlKnechtel take the user profile and prepend "chat." to the url, then you see some stats
 
 
2 hours later…
7:02 AM
@roganjosh I agree. I'd love to have a single source of truth. But team has a different point of view on how to build it due to time constraints. My DB1 doesn't have UUIDs, while DB2 does. I need to store that mapping somewhere. Would you consider storing it in DB1 a good idea? Or do I store it locally as it was suggested to me?
E.g. I have a Cats table with IDs (primary keys) in DB1. Does adding an extra column with UUIDs in DB1 make sense? It will not be used at all in the first project, other than for transferring the data to DB2.
 
user17135505
7:23 AM
Good morning pythonistas, anyone knows how to fill in close to X GB of memory?

I do currently use [[f"_{i}" for i in range(size)] for _ in range(8)] to quickly fill in memory but cannot control the size. Is there potentially something quicker?
 
How about 'x' * size?
 
7:46 AM
If you are just worried about the total size, replacing the f-string with f"{i:08d}"[:8] should work.
Depending on what you want to do, it might make quite a difference whether you have many small pieces or one large one (as Aran suggested). So choose wisely.
@PaulMcG I've seen the same article but haven't been able to give it a try yet. Did you by any chance?
 
user17135505
Sorry, I was wondering if there is a way to fill in x GB of memory?
 
user17135505
One could measure memory consumption and append to the object until size is reached I guess
 
b'm' * (gigabytes * 1_000_000_000)
 
 
1 hour later…
9:10 AM
int also has pretty reliable scaling characteristics.
A list's pointers as well. So something like [None] * size should give a good scaling in terms of sizeof(ssize_t).
 
 
2 hours later…
11:21 AM
hi
 
user17135505
12:16 PM
hey
 
12:38 PM
@MisterMiyagi I haven't yet.
GitHub is having troubles this morning: githubstatus.com
 
1:13 PM
@Riya What does "locally" mean here? If you're looking to dump the mapping to JSON or something then it's going to be pretty poor performance mapping between the two. However, adding the UUID to DB1 is going to be a sync issue. Given that, IIRC, you only have 10k records, you might be able to weather the transition without syncing the two DBs but, at some point, this needs to live in one place
I'm struggling to see a happy path here, if I'm honest. I don't think you can get the mappings to work cleanly between two isolated DBs in the short term
I'm interested why you need the UUID in the short term. Without a data warehouse like Redshift/Snowflake, I think you can rely on the auto-generated DB primary key to be an as-strong-if-not-stronger identifier for the row
 
2:02 PM
Aaaaand, GitHub is back up
 
2:20 PM
@roganjosh the UUID is used extensively as a FK in DB2. So, let's say I have a Cats table (primary key: ID in DB1, primary key: UUID in DB2). Cats are a foreign key in Owners table.
Then I am going to transfer the tables 1 by 1. First Cats, then Owners. To transfer Owners in DB2, Cats must already exist (with their UUID). Meaning I need to match IDs to UUIDs somewhere outside DB1 and DB2 (unfortunately, locally on my laptop).
 
2:54 PM
cbg folks! It's been a while. Potato?
 
Cbg
 
@inspectorG4dget Banana, and you?
 
banana, thanks. Tired, but banana. I miss writing code
 
 
1 hour later…
3:58 PM
Anyone have any ideas where I could permanently host a 42Mb text file? Right now it's linked in a question to a Google Drive file, but I'd like to back it up in case the OP at some point decides to delete it. The question is over a year old and the link still works, but I'd still like a permalink somewhere. Pastebin died when I tried to paste it in, and dpaste doesn't have a permanent option.
It's a bioinformatics file, in case you're wondering. stackoverflow.com/questions/70219758/…
 
Put it in a GitHub repository.
 
A gist would probably work, I think
 
That's not a bad idea
 
Put a sample of the data into the question. SO questions shouldn't rely on external data at all.
It doesn't look like the question even needs all data and frankly it would be improved significantly by actually having some data in text form.
 
OK, I'll do that
Ah. The file format uses tabs as delimiters, which won't copy from the formatted question unless you edit it and get it from the original markdown, which I'm guessing 100%-1 viewers won't do. I'll just link to the gist, and note that you don't need the full file.
 
5:05 PM
@Riya This doesn't sound too horrendous as long as you're backing up the DBs with something like an interim CSV and you don't make a throw-away script. Pandas would be a reasonable interim in this case, and you can map the ID: UUID over the column
I'm not aware of any automated tool for this but it's perfectly possible to build a robust transfer via Pandas as long as you don't delete anything until the transfer is complete and you can always roll back
 
10,000 records would be manageable with CSVs and littletable...
 
I think it's only fair here to disclose that you are the author of littletable. The pandas approach that I suggested would also be in memory
In the more-general sense, I suspect that UUIDs for keys will impact performance on JOIN in the long run. It's certainly going to be negligible here, but I'm trying to find benchmarks for big databases relying on primary keys
The benchmark at the bottom of this suggests that the slowdown isn't as much as I thought it might be (at least for inserts, not joins), but it's not catastrophic
 
5:28 PM
@roganjosh Oh yes, of course - thank you.
 

« first day (4545 days earlier)      last day (389 days later) »