« first day (4934 days earlier)   

5:02 AM
@smci For me, I cannot think of a single thing it would help me with. From being forced to use notebooks, in a browser, using some pseudo-SQL-DataFrame hybrid mess with camelCase syntax, to its own version of git, having to deploy wheels to be able to do imports properly, I could go on...
Some pain points can be reduced, for example soon we will be able to work locally and with real git, and deploy jobs to the spark cluster when needed, which will be a neat feature. However, like I said, I got sqlite to beat its processing (yay, indexes) speed and polars will do larger-than-memory processing much faster for anything I've needed. If we went into billions of rows I might change my tune, but the fact is I'm rarely doing that. And if I was, I'd take redshift any day of the week
I think I fundamentally don't understand the data lakehouse concept. Just because you can store hundreds of parquet files and blobs somewhere and then pretend it's some single SQL entity doesn't mean you should. Like, you're surely making a conscious choice to create all these files rather than using a proper columnar database
 
5:58 AM
What am I missing with g in Flask here? I don't get why that's an AttributeError: start_time. Looking at dir(g) it looks like a dict (it contains methods like pop and setdefault) but it won't work as a dict, and it doesn't store the attribute I put on it
 
6:39 AM
I'm guessing that the request context is already gone by the time we get to @after_request, so I can register that with @app.teardown_request but I can't find in the API where I could set g attributes, assuming that @app.before_request suffers the same problem of not being in the request context
 
 
1 hour later…
7:48 AM
@roganjosh with app.app_context() is unnecessary:
@app.before_request
def log_route_start():
    g.start_time = time.time()
@app.after_request
def log_route_end(response):
    print(f"ended after {time.time() - g.start_time}")
    return response
 
hmm, now I'm wondering what it was that pushed me to use the context as I was getting errors about being out of context. But indeed, that works, thanks!
Ah, I know. It was when I wasn't using it as a decorator and I ended up conflating the two solutions
 
 
2 hours later…
10:20 AM
@roganjosh Same, that's what I thought the first time I heard about it. Good to know that my intuition was halfway right
 
10:30 AM
"halfway right" suggests that I am at least a partial authority on this :P Next week I might embrace the whole notion - "just chunking this down into hundreds of parquet files, because reasons. Gotta get our money's worth out of Spark!"
 

« first day (4934 days earlier)