« first day (4833 days earlier)      last day (339 days later) » 

00:05
All: I went looking for Python f-string parameterize length or "How to use string.len() function inside f-string formatting?" and surprised that the only half-decent result I found was this, which isn't great, too case-specific, and had -2 votes before I upvoted it. Anyone know of better?
 
3 hours later…
03:03
eventually I found this: stackoverflow.com/questions/36962995/… which was from 2016 but didn't prominently recommend f-strings
There is also this very old 2011 How can I fill out a Python string with spaces?
 
2 hours later…
 
2 hours later…
06:55
@AndrasDeak--СлаваУкраїні I wouldn't have phrased it that way and it's not a great wording: "parameterize (f-string) length" says what it does; "pass string format as a variable to an f-string" is getting hung up on a specific implementation.
07:26
@smci I have a fairly thorough example of nested formatting, with both format() & f-string: stackoverflow.com/a/47712457/4014959 Here's a more arcane example: stackoverflow.com/a/40246013/4014959
But I guess neither of those is a great dupe target.
08:02
@smci Actually my brother was the one who initially wrote the number parsing/formatting code with babel, and we had problems were babel simply couldn't tell us what the decimal separator character for the given locale was. So I was actually looking for a more reliable library, before I realized that babel had built-in functions for the stuff we coded manually...
But while I was rewriting the code, my brother ripped out babel and made Javascript straight up send us the decimal separator character... so we'll never know if the builtin functions were more reliable
08:21
@smci I've gone over that Q&A once now, editing especially the prominent answer that suggested downgrading by just reordering things. But it the entire Q&A could certainly do with some good ol' vote and flag love.
 
4 hours later…
12:31
@smci it's the same issue for those too
 
5 hours later…
17:39
I'm pretty sure Databricks is going to give me a brain heamorrhage before this week is over. I don't think it would be possible for them to have designed a worse ecosystem if they tried.
I feel bad now about the fights I used to have with our platform developers at my last place. I'd even take boto3 back with a smile over this dumpster fire. It's just impossible to do anything serious in a stack of notebooks where we can't even import things from another "module" (another notebook), so instead you just have to use a magic command to run it and dump its entire namespace in your current notebook
 
3 hours later…
20:27
so ArneCorp has to make the call soon-ish between which platform to pick for datascience stuff, and it boiled down to (..drummroll..) Databricks and Sagemaker. I'm getting curious Mr Josh, are you mostly venting, or can I take your comments as anonymous data points against Databricks?
@Arne I'm happy to enumerate them properly for you if you need to make a case
I'll put a gist together every time it makes kittens cry. It goes way beyond what I just vented about. Previous colleagues of mine have text me totally unprompted and out of the blue basically to say "wtf is this? Do you have any idea how to do X?" --> "no. No I do not"
The DS introducing the setup to me on my first day actually said "I'm embarrassed to tell you this is how we do things". He'd never worked anywhere else, he was a fresh grad
"Previous colleagues" being people that used to battle our old system with me, then have moved on to different companies that also use databricks. So it's not isolated to the company setup at my current place (and some of that contributes terribly to a bad situation with broken pipelines). They're independently hitting the same nonsense in other companies
20:54
that's good to know. If it's not too much trouble I would actually like to have a list of the worst offenders.
21:22
Are you going to back it with azure? @Arne
21:40
@roganjosh Didn't this whole discussion play out last October? "Databricks notebooks are a manual flow and you can't share/import to/from other users?" and the answer from blogs seemed to be "Instil a methodology where people identify common company-internal code and move it to a package?"
Also, I seriously encourage you to do a talk or session on this: PyData London 2024 currently has CFP, or else PyCon UK 2024 is presumably September. Or if you consider notebook paradigm to be broken wrt path to production, you could say that too (note: Databricks was a 2023 platinum sponsor). Here's the 2023 schedule, for comparison.
It did and it just got worse. I've since pulled all the data off as parquet files out of our data storage just process the entire thing locally now with polars. I can't back any of my code up or use the central version control but I was about to punch my screen. We can't make a package any more because the CI/CD pipeline is broken and I can't do anything about it
This is at new company? What do you need to get done to fix things?
Anyway, you make good methodolgoy points about the hidden overhead and lifecycle of knowledge-sharing stuck inside notebooks in a piecemeal way, esp. wrt employee turnoer. Any half-sensible recommendations should be better than none.
All I can do is wait. I can do literally nothing else. The higher-ups just say something better is coming from an unknown, uncontactable team for me
I love theology
We talked about me doing a talk but right now it's depressing me if I'm honest rather than enthusing me to try change things. I've spent the whole day fighting a deadline and I've run out of ways to rename my df to stop trampling global names without making myself even more confused during debugging
21:55
Ok. Whenever your batteries are recharged, maybe you can identify a tractable example. (run out of ways to rename my df to stop trampling global names"??) What's the better practice than every user calling their dataframe df in a global namespace?
"Trampled a variable name somewhere? Tough ****, that's an indelible mark on your workbook. Back to the start of the 10 min pipeline for you, Minster!"
If you play your cards right you might become Databrick certified flow consultant ™... definitely doesn't sound like you...
Did anyone here attend PyData London 2023 and what's the general comment? For better or worse, DS users hacking up individual notebooks, instead of using repos, sounds like the quantum of independence that they want.
Requoting you...
Oct 25, 2023 at 21:39, by roganjosh
If my animus holds after a good sleep, I will seriously consider writing an abstract for a pycon of some kind. I both love and hate you for suggesting it
I know. I just don't think I can at this point. 99% of my enthusiasm is in my side project that I'm racing for, and the databricks BS for my actual job has ground me down. The fact I can pull thr whole lot out as parquet files (not easy in itself...) and run the whole thing locally at least 10 times faster is kinda getting me through, but does little for the despair the I'm gonna miss my deadline
Man, my mobile typing is not good tonight. Apologies for the typos. My grump will hopefully pass
22:14
Ok. If faster data import is one pain point, what does "pull the whole lot out as parquet files" mean, what format is it stored as?
You can "back" databricks with some storage medium. In our case it's an azure datalake. When you pull the data from that storage into a notebook, it becomes some quasi-dataframe. You can use pyspark against it (which doesn't match pandas or polars syntax in the slightest) or you can create a "view" on it that acts like a quasi-SQL database
So you get some strange hybrid object on the underlying data. After you read the file, you can call .createOrReplaceTempView() on it, and then it's some global database table that you can query against in strings (without any kind of string formatting - you just have an SQL-like table name floating in the ether now forever - congratz). I run my query, export the result back out to the data storage, then download it and do all subsequent processing locally
So, you can do:
1) Export some data from a completely detached data source. Save to CSV, load back into the datalake separately from databricks
2) Connect databricks to that data source (don't forget to _mount_ every directory individually because otherwise it's invisible)
3) Read that data as a spark object in a cell and convert immediately to an "SQL table"
4) Read happily from this table anywhere you like in any other cell anywhere in the notebook
product_details = read_dataframe(spark, 'tmp/shrink/ProductDetails.parquet').createOrReplaceTempView('product_details')
Then 50 cells later
%sql
SELECT * FROM product_details

« first day (4833 days earlier)      last day (339 days later) »