« first day (4757 days earlier)      last day (416 days later) » 

02:50
i found this line of code, is it correct?
x = map(str, range(2002, 2013))
 
2 hours later…
04:58
@discoMonkey Yes, that's just the functional-programming equivalent of the list comprehension. It produces a map object that yields str(x) for x in range(2002, 2013)
05:23
People usually write list-comprehensions, they're clearer.
 
7 hours later…
12:43
I have seen things. I want to unsee things. Every answer vulnerable to SQL injection and it's the first result on Google. Then I find parameterization was added in Spark v3.4 and we're only on 3.5, so that seemingly was the accepted approach. As a little treat, they chose to break the Python DB API in the process
You cannot have your cake and eat it, sir
Attempts to have his cake and eat it
The recipe for the cake was injected with very high concentrations of SQL. I would not recommend it but, hey, it's your call
It was in the oven for like 2 hours, my guy. Nothing bad is still alive in there.
13:06
dies
 
1 hour later…
14:36
Ok, I have a question that makes my own eyes bleed so I apologise in advance, and it'll take me a few posts to explain. Basically, I've just found out that our entire team has a single monorepo covering every single project. Each person has their own "project_a" directory etc. and inside each folder it's just a flat structure packed full of notebooks. This is actually the seemingly intended workflow of Databricks. You create a DAG stringing each of the notebooks together in order in a GUI
However, this repo itself does get compiled into a whl and does so through a Makefile. So while everything is a standalone workbook, there's is a "library proper" that you can import. I've sketched the directory structure here (apologies, it was done by hand because I'm not allowed to install anything atm)
Monorepo
+---project_a
|    notebook_a.py
|    notebook_b.py
+---project_b
|    notebook_c.py
+---actual_library_name
|    __init__.py
|   +---etl
|   |    __init__.py
|   |    aggregation.py
|   |    transformation.py
+---Makefile
+---requirements.txt
I cannot work like this. What I do know at this stage is that within a notebook, from actual_library_name.aggregation import aggregator is valid, so there is some semblance of packaging possible without me taking a chainsaw to the entire thing. Is there some spooky magic to hijack this to make my own package without obliterating this setup?
Monorepo
+---project_a
|    notebook_a.py
|    notebook_b.py
+---project_b
|    notebook_c.py
+---MY_PROJECT
|    __init__.py
|   +---some_folder
+---actual_library_name
|    __init__.py
|   +---etl
|   |    __init__.py
|   |    aggregation.py
|   |    transformation.py
+---Makefile
+---requirements.txt
For example:
Ordering got messed up there. But something that would allow me to do something like:
from actual_library_name.projects import MY_PROJECT as my_project
from my_project.some_folder import X
You will probably have to look into the Makefile. There's no problem for a Python distribution package to install multiple code packages at once.
Aha, so it could be that simple?
The first lines of the Makefile are below, which made me assume that it could only install a single package, but I guess I can just repeat certain steps and swap those values out in a single file?
.PHONY: help default tests install uninstall clean


PROJECT_NAME = actual_library_name
PYTHON_INTERPRETER = python3
REQUIREMENTS = requirements.txt
14:59
I assume so, yes. You may want to check where PROJECT_NAME is actually used. Most likely you can inject your own there.
Now I look more, it looks like most of the globals are in ECHO actually. I'm not familiar with make but there's a few simple things I could try, to see if I'm following it properly. It kinda suggests that I should be able to from project_a import * with no changes
Maybe not. I'll need to clone it but I'm still banned from installing anything until I get admin rights so I can make changes to the repo. A nervous wait. Thanks for clarifying that it's possible to have multiple packages installed in one go, though. I was thinking of hacks trying to look up the directory tree
Hello.
Hello :)
15:28
Hi
o/
(I'm helping)
16:09
Thank you kindly :)
16:47
You can only post once every 40 minutes.
whaaaat, since when is that a thing? So annoying
Is that since gpt?
Ask GPT? Get the tools and give it a full interrogation
Ok, well, I know now why the Makefile confused me... it's a relic that isn't used. There's a different deployment pipeline elsewhere in the system and it looks like my simple fix might have worked. Thanks for the rubber-ducking guys
17:18
Is there a way to turn code like
foo: Annotated[int, DefaultsTo(3)]
into something like
foo: DefaultsTo[int, 3]
without upsetting static type checkers, in particular pyright?
 
2 hours later…
19:33
@roganjosh I normally use a src directory because 1. I know I'm running tests from a setuptools package and 2. packages=find_packages("src") means anything under src becomes import xable. However I'd assume from actual_library_name.projects import MY_PROJECT as my_project would be a little harder to do.
I do exactly the same; I follow flask's example. I've just been thrown into this one which has been built up over 3 years. I have zero experience with make so it looked like a horrible challenge
Turns out, the team I joined 3 days ago has been struggling for years and all they needed was __init__.py in each of their projects. The installation is greedy and adds any top-level directory to PYTHONPATH... they just never flagged it as such
Hopefully I earned some brownie points with that change across the team so early on, while I sit waiting to be able to install basically anything :P
All of the notebooks use magic e.g. # MAGIC %run ./config which just injects things into the global namespace of each notebook. It's nearly impossible to follow because you can't know where any of the methods come from
19:55
It's an interesting "frog in the pan" scenario tbh. Everyone lives with this horrendous setup where everything has to be global functions from unknown sources (each notebook runs like 10 different modules and throws all the functions into global with no trace) and then convinced me that there was no better way as part of my induction. All I'm left with is a feeling of "screw jupyter and the idea that no data scientist could know better". That doesn't help anyone
20:09
I might take @smci up on the suggestion of blogging because I really want to rage at this point and I need an outlet. I'm thoroughly sick of how tools are positioning themselves to "help" the poor little data scientists that can't fend for themselves, like productionise something and just need notebooks and GUIs. </rant>
20:23
@roganjosh How did that break the Python DB API?
Because {} is definitely not part of the specified param style
@roganjosh Lotta small companies have sacred crufty scripts with globals, magic variables and paths, things that only work for certain userids or groupids...
My grumps come from the fact that I don't even have to be able to run something (I still can't) to see that the tool is being horribly misused, and not from ill-will. Pitching stuff to data scientists to "make life easier" causes so many borked setups
@roganjosh Hey, don't blame the tool for one user's permissive workflow. Just gently show them the better alternative. So what's the better approach than "# MAGIC %run ./config ", without you administering the PYTHONPATH?
__init__.py being in the directory
You know, the thing we all got used to before a platform came along and forced everything to be a jupyter notebook because "that's what they need"
20:34
@roganjosh You and I have touches of perfectionism. They're not paying you to rearchitect their Python/jupyter flow and they don't care that it's held together with chewing-gum and bits of string; you need to figure out the minimum quantum of change that they will actually swallow and make the case for it, without nonverbally communicating the emotional content of the Edvard Munch painting. (Gradually wean off dependency to global symbols, use imports and fully qualified names, I suppose)
@roganjosh Before you start "Confessions of an opinionated scientific notebook administrator..." :)
You might like to check out Noah Gift on LinkedIn and Twitter, he's been beating up mercilessly on notebook paradigms as not being a production-capable flow.
I'm ruminating on it. Honestly. There are too many angles to my anger so I'm just gonna go out for a pint. I encounter broken systems every day but 3 days in, when I don't understand really anything and I get rid of their biggest bugbear for years that was created by the misunderstanding the platform gave the team....
rbrb
@roganjosh Yes, a pint is the correct Douglas-Adams-grade answer to the initial shock. Once the outrage subsides/ transforms into daily procedural dread, figure out how to make the case to them to wean off keeping every import in the global namespace. They don't care about injection attacks by outsiders, but they should care about namespace collisions. I'm sure other people have written advice at more length.
...a pint, with an optional curry chips.
@smci I have had no trouble selling "correct" way. The team wanted it all along, they just bought the idea that what they had is the best they could have
The latest tools suggest to people that they are the best way to do things. They're helping you, right? They're designed for data scientists. Screw learning all the fluff; what you need is right here. But it's garbage if you don't know what you're doing - you can bork it. And bork it bad.
Oh great, well then your scientists are more highly evolved... Did ya ever watch 'Prometheus' and treating it as a drinking game: one shot each time you predict the identity and manner of the next demise.
@roganjosh Notebook paradigm remains a great way to share code and data, get visualizations up fast, develop intuition. It's not a reproducible or production-ready flow, and the further you go down that path the more debt and lock-in you can accrue. (e.g. importing all identifiers into the global namespace and not using qualified names is bad practice; you don't know which versions of which packages the thing is reliant on, or trace when something breaks.)
...Someone out there must have written a good talk on getting the genie back in the bottle. Guidelines for coding style in notebooks, plus solid advice on when and how to productionize the code. (if not, why don't you write one for PyCon UK or EuroPyCon 2024?)
21:02
@smci you're scaring me now. It's been years since I've done public speaking but I reckon I could pull something together on this theme
Do not tempt me, smci :P
e.g. look in the EuroPython 2023 stream "PyData, software packages & Jupyter" ep2023.europython.eu/sessions
or see noahgift.com , noahgift.com/#tags or PragmaticAI (he stopped posting on Twitter/X). Back in 2018 he wrote "Here Come The Notebooks" calling Jupyter "same workflow as the majority of data scientists around the world", recently he's been much more critical. But the right code migration and production flow depends on your individual company.
Hey if anything the rise of AI in coding IDEs will make it worse and non-deterministic... you think importing globals is bad practice... wait until users say "then call that function that breaks the data out by categoricals, aggregates then transforms into tables... no, the other way..."
Trying to find you more specific talks but not seeing anything great. It helps if you can tell us your company's workflow, toolset, what their main activity is, at what level they want to collaboratively share code and data, what the production flow is. Honestly I think you just make a list of the worst practices and tell people what to avoid doing.
21:25
I have a catalogue in my head of things that go wrong
Ok here is the most (ahem) harshly worded critique: "Jupyter Notebook is the Cancer of ML Engineering" - Or Hiltch, 2019 medium.com/skyline-ai/…
> The problems begin when this story needs to interact with a production application. The fun & easy platform used by the data scientist needs to be integrated into a production-grade data pipeline. That’s where nearly all of the benefits of Jupyter become drawbacks, turning the life of the ML Engineer into a living hell. If the company does not have an ML engineer, almost every single line of code written on Jupyter can become a bad case of tech debt, slowing the company’s time to market and plaguing it with hard to debug issues both in the research and the production phase.
And I'm not always looking for answers here when they do. I'm really tempted to lead a charge for sensibility on what actually helps DS. I'll see how I feel tomorrow but maybe you're right that I should do a talk. I think I probably do have enough experience in this space now
(oh by the way, you can't use harsh titles like that at confs these days, it violates CoC)
That talk seemingly fell on deaf ears for Databricks
"We shall fight them on the notebooks, we shall fight them in the sandboxes, we shall fight them in the conference halls..."
21:29
is there a unittest.mock.patch context manager that isn't specific to a module?
something like:


with unittest.mock.patch as patcher:
patcher("module.function")
patcher("module.function")
pytest's monkeypatch has something like that
@smci ... but we will never productionise
@roganjosh "someone else's department". But seriously, tell us more about what your current crew do. In order to know which way to point them.
I was just completing the speech :) I will figure it out. I'm just tired of tools that are ostensibly there to help us
They don't.
@roganjosh I don't necessarily agree; I think it's understood that notebooks (rather than git) are for when you want to quickly share code, data, visualizations, discuss, brainstorm esp. using open-source packages, with people (esp. outside your org or dept, no permissions required). Once you decide what to productionize, you stop and migrate back to git repos and whatever the approved IDE is, to avoid accruing technical debt.
Not true. Look into databricks
If my animus holds after a good sleep, I will seriously consider writing an abstract for a pycon of some kind. I both love and hate you for suggesting it
21:40
@roganjosh But the words "production flow", "production pipeline" or "software eng" don't occur once on docs.databricks.com/en/notebooks/index.html . It says "for developing [DS] code and presenting results... collaborating". See Reddit discussion "For those who use Databricks, do you use notebooks in production?"
Then "we" are using it wrong and I'm the devops
This is a company with >100k employees and this is the team I joined. Survivor bias is something to consider
Well just go with the flow and identify the worst practices to avoid
That would kinda be the premise of my talk. The tools are marketed as making life easier for DS but they go wrong
I don't wanna labour the point anyway. I'll collect my thoughts and see if I can make some coherent story
Ok. I gotta get back to stuff now. rbrb

« first day (4757 days earlier)      last day (416 days later) »