« first day (4766 days earlier)      last day (407 days later) » 

08:20
@shintuku No problem. I feel like I talk too much about database normalisation but I would go far as to say it's the fundamental technical consideration for a business. When you create that items table, you better know that it's gonna be there in 10 years' time with barnacles and tentacles all over it, and everyone moaning about how all the systems are hamstrung by relying on it. I guess it's like the inevitable fate of all databases but you can try slow the entropy with a good design
08:39
@AnilSarode wow, I just plotted my first graph in Excel for like 5 years after grabbing that data :)
And its linear regression doesn't match either of your outputs. The plot thickens...
09:00
Ok, your first method is correct, the difference is just because of the random sampling. Your manual implementation gives a clearly incorrect result but I won't have chance right now to look into why exactly
Although, maybe it's just me being stupid, but x_train = scaler.fit_transform(x_train) makes no sense to me. You're not even regressing against the same x values between the two so why should you expect the same gradient and intercept?
In fact, that probably explains why your gradient is near-as-damn-it 0
 
1 hour later…
10:37
hello chat! how are you all. I have discovered that chat gpt is awesome when it comes to a solving a coding problem or explaining a line of code. It is saving me a hours. i encourage you all to do so.
Oh my. The band wagon serving your area must have been in the garage for the last 9 months :P
@roganjosh while I agree, there is this to consider :P xkcd.com/1053
@discoMonkey I'll pass. The things it can explain, I can understand by checking the documentation.
@matszwecja when was the last time you used it? I thought so too but I was using 3. 3.5 really has gotten better and I improved my prompting a bit, so I would give it a shot again if it has been a while since you used it
10:50
Around the initial big boom, so yeah, it probably did get better
Can you ask it something for me? "What is the easiest way to handle postgresql connections? Should I use a context manager?". I never signed up but I want to see what it has to say
I'm basically interested in the code for a toy example, but I know nothing about how to prompt it
But I do have an idea of the mistake it will make. One that will crash my old central cluster and pull down every customer project. We shall see...
Yep, it's bang wrong
As yet it's a specific tool with a specific use case
To be fair, I made this mistake myself, but looking at the docs you have a chance to spot the big red warning box at the bottom
10:56
Quick and easy hypothesis generation or it can help you areas where your expertise is not strong, I used it for bash and c and matplotlib
@roganjosh I wouldn't be surprised if the mistake was learned from actual mistakes one finds on the internet.
Oh, for sure. I was just curious there. Had it known the issue that is quite clear in the docs, I might have had to rethink my stance. Instead, I will take a slice of that smug pie into my meeting now :P
@Hakaishin The problem is, in such areas you have no way to tell if what it is saying is correct or a pure hallucination or learned from purely incorrect code. It's primary goal is to sound convincing.
That's much more difficult for human to do, make a statement that sounds like you know what you're doing but you really don't
"correct or a pure hallucination or learned from purely incorrect code" That's my feeling during code review for ages already. D:
@matszwecja not really, you just run it quickly and check the result, easy peasy
11:08
The example roganjosh provided has just proven that's not enough
The output code will work just fine, and then some time later you start getting unexplicable random issues, because connection is not closed.
@matszwecja well depends on the context, for bash the result is immediate and checking it is easy, whereas writing it is hard. So yeah, a specific tool with a specific use case
12:15
@Hakaishin this is not some techno-backwater edge case, though. Postgres is one of the most popular databases in the world and vast swathes of companies rely on it
Now I wish I knew regex just to see if I can get catastrophic backtracking in a general answer. Alas, I only know the exciting terms and nothing about how to write it myself. Cheating GPT still isn't enough to convince me to try learning it
12:48
@Hakaishin if I click "continue this conversation" does it cost you? It might flat-out reject me anyway, but I think I should probably ask before I try play with it
@roganjosh nono, costs nothing on my end
thanks for asking though
Yeah, it immediately sends me to a login screen anyway, which I expected :)
 
7 hours later…
19:35
@Hakaishin bash is easy though
maybe I'm biased since I knew it before I tried Python...
@roganjosh reminds me of what happened with Cloudflare and their regex
Just couple it with a bajillion idle connections that we just proved it would recommend. Then you can have a tea party while the world burns. I really would like to prompt it innocently to make garbage regex but I don't have the knowledge. I wonder if I can lay its flaws bare in other ways
async is probably another good approach, but I don't know enough of that either to make things go boom with innocent questions
"But roganjosh, this is just a helper tool to make people more productive". BS. I spent months waiting for working code that was required for my model from one person who kept pushing GPT. Nothing worked, nothing could be integrated and I never finished my model and yet he committed twice as many lines as anyone. I would be less bitter if he wasn't my boss at the time
I mean, it's like using google on steroid. Technically vastly different. But the way you query it is based on the same workflow for me: Imagination
you just play around and find out
how much you're willing to play around, will dictate how much you can get out of it
In their cases, it can be either 3 things: 1. They thought they wouldn't need to do due diligence, and just trusted the output blindly, and thought "looks good to me". 2. They didn't prompt it the right way. 3. Might be unlucky and the subject of their prompt must not have been trained enough, or they didn't win the probabilistic lottery
@NordineLotfi what would you have done with this answer? If you're being really honest.
19:51
Honestly, I wouldn't have prompted it about SQL. But if I did, and used the same prompt (which I probably wouldn't have) I wouldn't trust the output. I would test it and use the minimum knowledge I have on SQL, which isn't that much compared to normal Python or Linux...
There is no SQL in it. That's all python
I thought psycopg2 was SQL related?
I guess I really don't know anything about it hmm
That's a python driver to connect to the server. It's no different than requests setting up a session
ah, got you. Never used it
but my initial point still stand, they definitely did something wrong, most likely 1. but also maybe 2.
20:37
@Hakaishin "result is immediate and checking it is easy, whereas writing it is hard". Surely by "checking" you mean "run and see"? Because that's "easy" in the same way as it's easy to tell a poisonous mushroom by eating it. SQL injection vulnerabilities "work". So does most bash code until you get a space or an empty variable.
Bash was easier to me than Python. Now I don't know what to think, but maybe this was a good thing I knew it before the other
I really don't like to pretend that gpt and friends are a plausible alternative to thinking and learning about code you actually write. It "works" if you're already knowledgeable enough to tell crap from correct code. And it's a lot of effort to do so.
And we sort of have a responsibility here to protect newbies and our own future data from the crap written by said newbies at some crap gpt-accelerated software favtory.
I picked my postgres question for a reason. It will work just fine without you knowing there's an issue (the server will kill connections over time). But as soon as you exceed the death rate of the connections, everything will explode. Yet the GPT answer confidently told me that it killed the connections, so the problem can't be there. How far do you expect your oversight of its answers to go?
... before it's worth just actually understanding the thing yourself and writing it
Interestingly, our very own Steve Holden has posted on LinkedIn which is on my main feed now
21:00
@roganjosh looking at your post on Linkedin, I hope I wasn't the enthusiast in question :P
Nope, but no points for guessing who it is :P In any case, I'm not out to shame anyone for using the tools available, I only wish to create a dissenting voice to the deluge of crap about how good LLMs are
all good, was just curious
either way, I think I proven enough I have a distaste for this thing. If it can't understand anything, and doesn't handle context like I do, then what's the point? Only good thing might be as toy to play around with maybe, but never depending on it
Given that you've raised that, I suppose I should post the link for full disclosure. Again, it's not to take a pop at anyone and I don't believe any of it is identifiable other than me
unrelated, but I just found out you could get the difference between two set() by using "-" :stackoverflow.com/questions/30986751/…
it's so obscure I never thought this was possible. Might be used in golfing
It's what a mathematical set does.
21:08
interesting :o I have more to learn I guess
It has use cases wherever sets do. Not obscure at all unless you treat sets as unordered lists.
Got you :)
22:03
@roganjosh I totally get that your comment was about the code ChatGPT emitted not explicitly closing the connection, not about the design of pyscopg context-manager does not call closing(), and hence may leak idle connections if called repeatedly. But a couple of remarks:
The suspense...
is palpable
a) what do you think of this blogger's suggestion to explicitly declare the context-manager as: with closing(psycopg2.connect(...)) as conn: (Maybe ok for the common-case, hard to debug if exceptions occur.)
b) Is this only a design decision specific to psycopg, or other DB packages?
I can stop you there if you're gonna go on this theme
c) Here are some relevant Q&A I found, some may be dupes or should be referenced (or closed):
22:11
I use SQLAlchemy to create an engine. That does clean up after itself in a with block`
7
Q: Using a context manager with mysql connector python

CallumI'm moving my code across from an sqlite database to mysql and I'm having a problem with the context manager, getting the following attribute error. I've tried combinations of mydb.cursor() as cursor, mydb: etc... mydb = mysql.connector.connect( host="localhost", user="root", passwd="",...

(that one was MySQL, not psycopg)
1
Q: With psycopg2 how to avoid using the connection context manager

yigalWith psycopg2, connection and querying the database works like so conn = psycopg2.connect('connection string') with conn: cur=conn.cursor() cur.execute("SELECT * FROM pg_stat_activity") #simple query rows = cur.fetchall() for row in rows: print (row) After trial and err...

1
Q: Should I manually close a database connection if using context managing (with-as)?

bjarkemoenstedI'm learning how to use a PostgreSQL database with Python (3.6)'s psycopg2 module. I'm having trouble figuring out exactly under which circumstances I should manually close my connections. According to the docs: unlike file objects or other resources, exiting the connection’s with block doe...

Just to be clear - I'm not a damsel in distress here. I know how to fix it. I just made ChatGPT give noob-me a bad answer
The bottom line is that it lied to me, and I foo'd up by taking things for granted. Far from helping me, it actually just codes out my own (incorrect) instinct.
I know you're not in distress. I'm saying a) important to distinguish if that's mainly a psychopg design quirk, not all DB connections. b) it turns up Q&A which should either be cited/closed as dupe
Well, actually, that messes up the timeline. I crashed the company systems a good year or so before ChatGPT even existed
No, it's every DB
@roganjosh "Hallucinating in code... that a DB context-manager closes its connection after use, instead of just idling". Hey you could have used that as a LinkedIn comment, the general audience on LI would understand it easier.
22:17
Maybe. I don't really fit in with LinkedIn since I generally just tear people apart when I'm bored so I can't say that I'm overly invested in likes
Ok how bout ask ChatGPT to write code which deliberately leaks idle DB connections. Or when it gives you the comment that the connection was closed, ask it which line or method call it thinks was responsible for the close.
Or.... don't use it at all and just write it!
@roganjosh I wasn't talking about LI likes, just a clearer first sentence.
I "knew" ChatGPT would fail on the question I asked it. I also knew it would give a convincing answer
@roganjosh unfortunately that's becoming the programmer equivalent of telling people to eat their spinach or broccoli. But what's achievable is you can point out the dangers of overly relying on it.
I wonder what happens if you ask ChatGPT to write code which generates lots of idle connections as a side-effect.
22:21
It just did. It told you they were dead but... they're not. Is it too late for zombies now?
I don't care if people on linkedin listen to me. I participate in some chats about polars etc but most of the rest is nauseating garbage. My message was clear. If people wanna take it on board, huzzah! Else, have fun when it all blows up
In defense of ChatGPT on this particular issue, the heresy in question was started by humans, proliferated by humans...
...Look at the results for "Do DB context managers close the connection?" e.g. towardsdatascience.com/… "Thanks to the context managers, we can read data from external resources and rest assured that the connections to underlying databases or files are closed, even if we encounter some unhandled exceptions in our code."
Small comfort when I actually did bring our entire system down
...so maybe in future ChatGPT will learn a concept of reliability/accuracy/completeness for each individual code source (just like we currently expect with news sources, although obviously there aren't huge monetary or political incentives to disseminate incorrect code... Butlerian Jihad...)
I don't mean to spoiler Dune on you, but the Butlerian Jihad was a reaction against friendly helpful code-writing bots... ("Hi I'm Sammy, and I'm here to crash your SQL server... let's get started on that query of yours")
In my defence, I didn't actually make things worse but I got nearly 50 people to review their code and make it just as bad in the space of 2 days. It takes a lot of credibility to mobilise so many people and if you foo it... well, that's a lot of social currency lost
Well here's a concept: manually seed a Code LLM with a ranking of which code sites/blogs are believed to be 'good' vs 'bad' (e.g. for Python), then have it learn which questionable practices appear on the bad sites but not the good ones. Human-review the initial site rankings, rinse and repeat...
22:32
This was before LLMs
I won't use them
@roganjosh I understand that. But millions will, and we'll be dealing with tidal waves of them... at least SO will...
Meh, not my problem really
Civilization collapsing isn't our individual problem. Oh: Dune 2 (film) was pushed from 11/23 to 3/24. I guess in part due to the media pipeline being full due to the screenwriters' strike. But it misses the 2023 Oscar eligibility deadline.
At the end of the day, people seem proud to say that GPT hasn't answered their problems, or that my solutions line up with what it told them. If that's what they take as a arbiter of truth or good practice, so be it
@roganjosh Right, but you have survivorship bias; you're not a non-technical hiring manager in a bad SW company. I keep hearing the opposite take on GPT from non-technical people, tales bragging of epic corner-cutting and boosted productivity from people who didn't read the docs or get certifications. (Before that triggers you, consider it a new opportunity for us...)
22:46
@smci you could reverse that and say people saying that have their own bias (probably not selection bias, but there must be a name for this) where their need or queries/prompt would be so trivial, that the answer would be verbatim on SO or other website.
To truly know if it's useful, you would need to know exactly the entire dataset, and then check empirically if N output are just strictly from the dataset, or entirely new and valid output. More than half of the time, I would bet you that this would be just the dataset itself.
In the training phase, as well as the reinforcement learning phase, they do use a bunch of tricks to "augment" the dataset, like synonyms, low paid/non paid worker that supply conversation/instruct data, label, etc.
Without reinforcement and the bag of tricks used in training, it's not far from a normal autocomplete or markov model. Just look at gpt1 or gpt2 "quality" of output. It's night and day
How would you call that bias/expectation? The expectation that SO covers the use-cases which each user considers common, and moreover that the answers have cookie-cutter code (clear, short but complete, correct, no pitfalls, works (and continues to work) across all versions of software, packages, dependencies, platforms, OSes)? "Implicit fitness"? completeness? bug-freeness?
I don't think there is ever a "continue to work" answer. This is context dependent, like everything else. I don't know of a good name for this, but I wouldn't be surprised if one already exist...
@AnilSarode One tip: like rj said, whenever debugging code that uses anything with randomness (a training-test(/-holdout) split, a sampling, a shuffling, an ML classifier/regressor, a grid-search, then explicitly pass a random seed. To ensure your results are deterministic.
@NordineLotfi See, none of us are going to win enterprise support gigs with attitudes like that ;-)
ah, I don't know what to say to that :P
23:04
It is the year 2049: you wake up on schedule in orbit round Alpha Centauri, but you dimly remember that in all that hectic rush before entering cryosleep, you left your packing to EnterpriseLogisticsGPT... and it seems to have sourced its concept of people's everyday apparel from the 3rd generation of the Kardashians and Duck Dynasty, mashed up... It is The Singularity. Maybe Amazon Predictive has already dispatched a resupply convoy to arrive in a year or three... or maybe it hasn't...
Does anyone know a good profiler for python code? I'm mucking about with cProfile, but the documentation and functionality are both lacking
I recall there was a couple on github that used google chrome/chromium as interface/viewer. Maybe not what you want?
it basically plot the performance and then use chrome for viewing the plot/benchmark/profiler data
That sounds great
recall there were a lot more, but here is two: github.com/kwlzn/pytracing and github.com/jiffyclub/snakeviz
Thanks
23:14
I didn't test those, but I did try this one: github.com/nvdv/vprof also use the browser
concur with snakeviz, it eats cprofile output
it doesn't have too much "functionality" so YMMV

« first day (4766 days earlier)      last day (407 days later) »