« first day (4077 days earlier)      last day (870 days later) » 

1:08 AM
cbg
 
 
5 hours later…
5:39 AM
Wow. New feature! Stack chat!! :fire
 
@Davidkim chats been here for almost a decade now.
 
6:30 AM
@Sankar Oh I didn't know that! Thank you bro!
 
6:59 AM
cbg all
 
7:20 AM
@WayneWerner What is cdg?
 
 
1 hour later…
8:25 AM
Cbg
@Davidkim Salad
 
9:02 AM
morning cbg
 
 
1 hour later…
10:18 AM
Hi mates, do those database batch insert/update methods provided by database connection library e.g.bulk_update xxxx, bulk_insert xxxx, executemany eventually use this sql syntax to do a quick batch insert stackoverflow.com/a/5526937/5983841 ?
 
11:09 AM
PSA: h2o, a huge machine learning library for Python and R implements a vulnerable version of log4j.
Even better, their open PR is only looking to implement snyk to check for the vulnerability... which is just going to fail. I'd raise an issue on github if they hadn't just disabled that for some reason
 
Who in their right mind would use SOAP to make REST calls on the same API ?
 
11:27 AM
@roganjosh I see you are having fun...
 
<cries in grype>
 
11:57 AM
Hello everyone!
Can you please share the techniques with me? How to speed up the flask?
 
step 1 of any speedups: identify what's the real bottleneck
 
@Partiosh I want to optimize the code that I have written
 
take a look at code profiling. you will just waste your time if you don't know "which part" is the slow part
you need to identify what's the problem before you try solving it
 
you're right
I think you didn't get my point. I just want to speed up the flask. How can I do it? what kind of techniques can I apply?
 
why do you think flask is the one that's slow
 
12:02 PM
I want to get the quick response.
I have gone through with this link;
https://github.com/sunary/flask-optimize
 
I guess let me speak even more plainly. there's a good chance flask isnt the issue, and your code is.
if you want to see how fast flask is, just try running a hello world app on flask. is that slow? if its not slow, then the problem isnt with flask
you need to profile your code if you want to understand what's actually causing the slowdown, instead of (mostly likely wrongly) assuming it's flask.
 
I have checked my all code. It's working fine. Actually, I am assigned with the task and I have to speed up the flask.
 
any code is as fast as the slowest link. dont be the blind leading the blind, your "end goal" is to get the overall response delivered faster, flask is just the mode of communication*. It's like youre expecting home deliveries to become faster by making checkouts faster...the primary time taken is still the actual time taken to deliver the goods.
 
@ParitoshSingh can I have skype ID please if you don't mind?
 
i dont have one, and even if i did, i'd most likely not share that.
 
12:12 PM
That's quite enough @HaseebTariq. Paritosh has already explained what you need to do. It's not his job (or anyone's here) to fix your code
 
I am just seeking help as I am beginner in the world of Python. @ParitoshSingh Thank you! for helping me out.
 
Np. identify what's slow. look at code profiling. "make flask faster" is a pointless request if flask isnt the problem. I can always give you answers like : buy a better pc, change flask to fastAPI..but none of those are truly answers that will help you. what you ask is...ill informed. Find what the actual problem is.
 
Please take some time to understand how the chatroom works @HaseebTariq rather than flagging my comment as offensive. I'm a room owner, which means that it's my job to try keep order in the room
 
Bundle of thanks @ParitoshSingh
 
@roganjosh you've used this library yourself? Any opinions about it so far?
 
12:18 PM
I haven't, no, sorry. I don't really do any machine learning these days tbh
 
ah i see, no worries
 
@ParitoshSingh which library you want to use?
 
Better get on fixing that vulnerability then!
 
I remember dabbling with h2o very very briefly 3-4 years ago or so, so it was like a blast from the past, hearing about it again.
 
12:20 PM
Sorry bro!
 
Actually they (h2o, the company) seemed to be focused on pushing for things akin to AutoML even back then, and make a product out of it. I forgot what name they gave it
Not sure how they're doing now though
 
I think this came up from the rocker/ml stack which seems to include it
I don't really know what I'm supposed to be doing once I find these vulnerabilities. I'm going to track them, but then I can't actually do anything other than check when they get updated. We can't exactly rip these things out of our workflow; our customers would love that
It was fine for my own Java libraries because I could just bump the version, but in these cases all I seem to be able to do is cry and wait
 
yeah, this is a rough vulnerability for sure
if its an option, you could isolate the machines where these programs run, but i have a feeling thats a logistical nightmare
maybe chuck all these under docker deployments and pray thats enough of a sandbox perhaps
 
In theory they already are since everything runs in docker containers, and it's not really clear to me how someone could get such strings running through these kind of libraries, especially into the logs. But not knowing doesn't mean not possible, and the exposure would be huge
Plus, I imagine that security auditors would want evidence that this stuff was at least tracked in order to maintain certain accreditations
 
 
1 hour later…
1:36 PM
what's the best way to integrate a matplotlib graph into a website?
@roganjosh ah the world of certifications...
 
@Hakaishin there isn't one. Unless you're totally wedded to matlab for specialist functionality, I would suggest changing to something like plotly/plotlyjs
 
Yeah, that's what google just told me too :P
Makes sense
 
Don't know why I just said matlab and not matplotlib there :/
 
well, to be fair, matplotlib is essentially a static image, so you could also always go that route if you dont really care about interactions or what not
 
1:59 PM
That has the possibility of being cached by the browser, so it'd be fine for static images but it falls down if you want to load things that can be changed (not just animated plots, but literally any graph that might get updated)
 
@ParitoshSingh not to be a smart-ass but matplotlib is everything but static. It has its own event loop. Default backends allow you to interact, show you cursor coordinates etc. It's only static if you use a static backend (e.g. pdf)
or if you don't show() your figures
 
nope, perfectly glad to be corrected, i should have been much more precise
 
there are also a bunch of widgets for it
 
matplotlib essentially makes static plots. That's correct isnt it?
 
if you mean "if you save them as static image files" then yes, otherwise no
depending on the definition of "static" of course
 
2:03 PM
hm, then there may be more to this than i know of. I essentially have always just used plt.show() or something along those lines to use matplotlib, and always assumed i get a fixed image out
 
You know, there's probably some niche pdf rule that would allow you to embed matplotlib in your document. :P
 
the p in pdf stands for "lol"
@ParitoshSingh depends on "fixed image". You can move your mouse and see the coordinates. You can pan the figure or zoom in/out in it. If there are widgets you can interact with them.
perhaps some of this is borked in jupyter notebooks, so if that's your only entry point it might explain things
 
o.o
yeah, i dont think i've ever had any of those functionality..and that sounds rather handy
 
if you don't do %matplotlib inline or whatever you probably get the classical behaviour of a tk/qt backend
oh yeah, and you can move 3d plots around too
 
oh
this is like a revelation to me, i need to play around with this.
 
2:09 PM
haha then you have been missing out, matplotlib is way more than a static image. But yeah I'm gonna use highcharts
 
import matplotlib.pyplot as plt
plt.plot([1, 2, 3, 4])
plt.ylabel('some numbers')
plt.show()
 
import matplotlib.pyplot as plt
import numpy as np

fig = plt.figure()
ax = fig.add_subplot(projection='3d')

x, y = np.mgrid[-10:10, -10:10]
z = x**2 - y**2
ax.plot_surface(x, y, z)
plt.show()
 
This should theoretically be all i need right?
 
yes
 
Where do you see your plots usually? It looks like no dice for me
 
2:11 PM
print(plt.get_backend())?
@ParitoshSingh new window popping out
try running it in vanilla python
 
ah, let me try that
i got module://ipykernel.pylab.backend_inline for that get_backend command (spyder right now)
 
lol, spyder
no wonder
 
Qt5Agg...and... it's magical. (cmd)
 
You're like Goku with his wrist and ankle weights. Now you can fly.
 
import antigravity this is amazing
 
2:14 PM
lol i don't get how spyder survives next to jetbrains prof/community and even eclipse. It's a miracle
 
yeah okay, so even in jupyter notebook i get the same thing as with spyder. module://ipykernel.pylab.backend_inline and no interaction. even without doing a matplotlib inline magic command
 
@Hakaishin i'll admit i actually love spyder for what it is
 
A burning mess?
 
heh
a convenient way to keep an ipython kernel next to your code and freely inter-op as needed.
 
2:19 PM
my terminal has tabs ;)
 
haha, touche
 
@Hakaishin Wait, you program Python in Eclipse?
 
okay, looks like %matplotlib before doing a plot gets me to Qt5Agg backend in both jupyter notebooks and in spyder
so it was just a case of the default behaviour being inline
 
I thought... I thought you were a VSCode guy @ParitoshSingh?
 
@roganjosh Nah, i gave VSC 2 good tries. It just never stuck with me
 
2:22 PM
In fact, I think it was you that convinced me to drop Spyder!!
 
I've mostly been using spyder through and through
Well, im glad i was a positive influence on this world, even if i couldn't save my own soul :P
 
:P
I couldn't go back at this point. I've learned to live without IPython tbh, but it'd be totally impractical for any of the work I'm doing which is across multiple languages
 
cbg
 
@roganjosh not me, but I guess it's possible? Or is it only for Java?
 
2:38 PM
How do i find the wiki edits? there isn't a filter on my activity page
 
@roganjosh yeah, fwiw it was most likely my IT infra team's fault for most of the VSC issues i faced..but it just never became a seamless experience
for me, the final straw however was VSCode refusing to honour the virtual environments that were available, but "pretend" it did
 
Was this the desktop version or the server?
 
desktop
 
If it's the server version, I could fully appreciate all of these issues and more. That's the one that still auto-corrects import pandas as pd to import pandas as pandas, no matter how many times you tell it "no, bad VSCode"
 
or well
what is the server version?
(i presume its vscode only accessible via browser?)
 
2:45 PM
Basically, it's the one that runs in a browser. I think there are a few implementations
 
yep got it, then no, it was vscode desktop
I will also say, something about vscode, it just never felt snappy
not quite sure why, i couldn't put my finger on one single thing
 
It used to be a mess, but it's improved a lot. In its early days it was practically unusable without installing a bunch of addons and tweaking some dumb settings, but thankfully that's no longer the case. It's actually turned into a respectable editor IMO. (Can't say how well it interacts with virtualenvs though)
 
It auto-detects if there's a new virtualenv in the project root (assuming that you opened the editor at that point)
 
@AndrasDeak Sage can display that, but it's not interactive. OTOH, it's fairly easy to display that array using Sage's 3D plotting stuff, which is interactive:
Ok, it's not very interactive, but at least you can pan, rotate, & zoom. :)
 
3:04 PM
that's almost as interactive as matplotlib's
 
In Sage, you wouldn't normally plot the function using an array. You'd just pass the function as an arg, and specify the x & y domains, and let Sage figure out the step size, (although there's an arg to adjust that if the result isn't smooth enough).
Sage can do 3D anims. I haven't played with that much, but I used it here: astronomy.stackexchange.com/a/47554/16685
With SageMathCell, I guess proper interactive widgets would put an insane load on traffic to & from the server, so all interaction has to be client-side.
 
 
1 hour later…
4:37 PM
Hello friends. I am officially looking for job opportunities. If you know of any that seem like I would be particularly suited to, let me know. Either via ordinary chat ping, or email to Kevin953482<at>gmail<dot>com.
15
I spent thirty minutes trying to decide if this message violates SO's self-promotion guidelines, and I settled on "crass, but technically allowed". If sentiment in the room seems to indicate otherwise, I will send it to the cornfield Ouroboros room, for reconditioning
 
Given the sheer amount of time you've spent self-promoting by helping probably thousands of people for nothing... I think you're probably safe :)
This doesn't sound like the best circumstances for you so I hope things are ok. We are actually starting a massive US drive next year which might be an option for you. In terms of competency, you'd smash the job. I don't know whether the demands of customer work in a scale-up might kinda crush your free-spirited approach to coding, though
 
Hmm, pretty close to the response I predicted, which was "we'll let this blatant commercialism slide, just this once, because on average your existence is still useful to Us"
 
I really don't think that is what I was shooting for!
 
That's the "royal us", there. The speaker is a victorian queen on a fat pony, languidly playing polo
 
Apologies if that is the way it has come across.
 
4:47 PM
Hehe, not really :-) in turn, I apologize for making it seem that this was the way it came across.
It's really only similar in terms of... "actionability"? Both you and the victorian queen have indicated that you won't call the self-promotion police on me. A splendid outcome IMO.
In terms of niceness they're on opposing ends of the spectrum
Regarding my circumstances, the short version is: health good, finances OK for a couple months, personal life currently free of catastrophe
 
That's good to read, TBH.
 
5:03 PM
It's good to read that you consider it good to read
 
5:19 PM
I'm still toying with the idea of doing some kind of freelance thing... I have little clue on how to get something like that set up, though
My three main categories of ignorance are,
- necessity -- how do I send the results of my work to customers? How do they send me money?
- visibility -- when people need work done that I can do, how will they find me?
- regulatory -- what forms do I need to send to the government so the Small Business Administration doesn't send its infraction team to seize all of my belongings?
"Oi mate, you can't run a software business within 100 yards of a natural body of water, the EPA says that wifi is bad for fish. Take him away, boys"
 
5:35 PM
- visibility -- If you have a decent linkedin that lists explicit skills (at least in text; I haven't filled out all my actual Skills section for years), you won't need to. There's a whole recruitment industry that will find you
- necessity -- You'll probably get access to their existing systems and it's not your concern before you even have to consider an opportunity
- regulatory -- pointless me even trying to answer this because I'm in the UK
 
sba.gov/business-guide/launch-your-business/… essentially says, the federal government doesn't care if you register, if you're a one-person business operating under your own name. The state and city governments, on the other hand...
NJ's website: "you probably need to register as a computer consultant with the Division of Consumer Affairs". Consumer Affairs website: "no search results found for: computer consultant"
A month from now, the consumer affairs enforcement team, while they press their jackboot on my neck: "oh yeah, that's a common problem. You have to search for 'computer consultancy' to find the right form. Oi! Stop resisting! Stop resisting!" [they tase me]
 
5:57 PM
What is a "computer consultant"?
 
Personally, I'd consider that to be someone that gives advice on computers, but doesn't take any concrete action themselves. I'm the opposite, I cryptically leave an executable file on your doorstep, and vanish with no explanation, like the morning dew.
 
This is so different from how I launched as a consultant (and I can't claim to have some illustrious career with lots of contracts) that I don't know what to say. In the UK, you just set up a Limited company for (IIRC) £50 and off you go
 
Maybe it's like that here too, but my personal overcomplication field is interfering
 
common beginner FAQ, there is probably a better duplicate than the one I found stackoverflow.com/questions/53727948/urllib3-parse-quote
 
6:26 PM
I do like to worry about worst case scenarios... More likely, the consumer affairs enforcement team will send me an email saying "we noticed you started a computer business, please submit forms X Y and Z in the next 60 days", and they only break out the jackboots after the third notice goes unheeded
 
That's why you get an accountant
When I was contracting, my company charged 2.5 times what I get paid now. You can afford an accountant
 
Sensible
 
6:50 PM
I can balance a checkbook but I need someone to serve as a regulations sherpa. Guides me away from dangerous routes, or gets eaten by the regulations yeti while I make a quick retreat
 
Accountants can do that
Put it this way; I got in a hell of a mess (in my head) with HMRC (which is about the most powerful non-military force in then country) and I was owed £21k from the contract before I called it in. I really struggled to write the invoice because I just doubted myself... I only put an invoice in when I had almost exhausted a £2K overdraft and I didn't have enough money to get a train ticket to get to work tomorrow
None of this was necessary. Get an accountant and just stick to the terms
 
7:13 PM
As a seasoned expert at doubting myself, I like the idea of avoiding the scenario you describe
My spirit animal will be the guy that sent $10,000 invoices to Google/Facebook/etc, even though he never worked for them, in the hopes that they'd shrug and pay him anyway. Channel that level of entitlement.
 
I don't think that's useful (even though I guess it was a joke)
If you want to be independent: you will have to deal with this in one way or another, in whatever workplace you go to
 
Man, why can't things be easy and profitable and fun all at once :-(
(Not sarcasm, I really want society to be like that)
 
7:29 PM
I'm thinking very hard about this. I don't think we're very different (I hope that's not an insult!) but my role gets kinda crushing even though the company is fantastic.
 
I thought "yep, this guy gets me" when you said in your first message, "[it] might kinda crush your free-spirited approach to coding"
All of my previous employers were basically humane and well-meaning, but the environment might not have been good for my productivity soufflé
I remember my very first job, which was at a company of about 40 people. The CEO would frequently come into my office* and ask how my day was going. Very nice guy. But it still jumbled up my coding brain afterwards.
(*actually a storage room with a reasonably-sized empty portion that they put a desk into)
 
8:01 PM
So what do you want? Lock you up in a room and let you concoct things? (genuine question). We have a "labs" functionality that is away from customers. I'm trying to gauge how best to use your talent and not put you in a corporate loop (and it's certainly not restricted to what our company is doing)
 
"what do you want" -- now there's a real puzzle
 
8:24 PM
@Kevin nope, all good
@Kevin I read about "visibility" on the internet. It works like this: you do free work for people and that gives you exposure. Perfect system!
 
I've got about 70,000 points worth of free work accumulated, so I'll just skip to the next part thanks
 
@Kevin to be honest it's always sounded to me that you were too deep down in the industrial machine compared to your (presumed by me) personal preference. Weird proprietary databases and things... then again if your work is more than your hobbies then your hobbies kind of get ruined.
 
I've heard many stories of people burning out on a hobby they liked because they monetized it. Perhaps it can be a good thing for one's day job to have a different tech stack than one's hobby projects.
 
half the words in my previous two sentences are a bit off but I hope you can get the gist
 
8:39 PM
I thought the dots didn't connect perfectly, but I do think I got the gist.
But yeah, weird proprietary databases are not something I want to work with, given the choice.
Although 90% of my work-related tales involve weird proprietary things, they probably made up only 10% of my work responsibilities. Just because perfectly ordinary .NET web development isn't fun for me to complain about.
 
I didn't realize .NET web development was ordinary O.o
 
"What a day -- the customer wants to be able to create, read, update, AND delete data!"
 
That doesn't seem like a particularly popular choice for web servers
 
Oh fascinating Kevin, tell us more, how did you escape from this predicament
@Aran-Fey This site that I picked at random from google, has two .NET frameworks in the top ten. Add them together, and they're in third place!
 
@Kevin Wait! Let me grab some popcorn
One of them's more popular than flask, wow
 
8:49 PM
When I'm frightened by a large customer, I activate my natural defense mechanism, which causes my arm to bloodlessly pop off and wriggle on the ground vigorously. This usually causes enough of a distraction that I can escape.
 
Note to self: don't hire Kevin for that customer service position
 
Fascinating. Does the arm grow back or do you need to wait for the customer to leave and pick it up?
 
@Aran-Fey I'm sorry but anything that puts jquery and flask in some bar chart is utter bull
 
It grows back, but it's good to retrieve the old one. That's good composting material right there.
 
Is jquery not a web framework? I don't know much about it except that it has way too many features, so I wouldn't be surprised if that's one of things it can do
 
8:55 PM
They aren't even comparable. One is a web framework and the other is a JS-only toolbox. Jquery can't serve requests on its own (I'm gonna live to regret that statement, I'm sure)
jquery is not a web framework
 
perhaps it's the "have you tried jquery?" joke taken too far
 
I... will politely excuse myself. I appear to have egg on my face
 
Have you tried cleaning it off with jquery?
 
Good shout!
 
I love learning new english idioms. Although it's usually funnier to take them literally
 
9:00 PM
en.wikipedia.org/wiki/Web_framework#Client-side considers Jquery UI to be a "client side" framework, and likewise for ReactJS. I believe JQuery UI is considered distinct from JQuery, even though they're developed by the same team.
 
I hope it's a guy called Jake Weiry
 
And then there's en.wikipedia.org/wiki/Template:Web_frameworks which lists just plain old JQuery
 
@Aran-Fey I mashed things up here. The trope is women leaving a table saying "I need to powder my nose" (which just isn't a thing now) as an excuse to go to the bathroom, and, well, "egg on my face"
 
My personal definition of a web framework is something like "software on a web server that generates html in response to user requests". I don't think JQuery qualifies under that definition, and maybe React doesn't either.
 
Imagine a bloke getting up from dinner going "Please excuse me for a moment, I need to egg my face"
 
9:04 PM
in conclusion, .NET must be the #1 real web framework. Thank you for coming to my TED talk.
 
@Aran-Fey What amuses me is that you used "bloke" here, not "guy". Now you're getting right down into pub-talk!
You can't have picked that up online from something other than satirical articles, I assume?
 
The policemen in all of my hypotheticals today kept saying "Oi mate" because I saw it in a funny meme about british TV licenses
It's a bit too NSFW to share in here, but it ends with "[I] stab [the policeman] with my unregistered butter knife"
 
@Kevin I'm sorry to say that TV licenses have nothing to do with our police :/ (they call me the Thought Crusher)
 
@roganjosh I was under the impression that "guy" is more american and "bloke" is more english (in other words, a bit fancier). Pretty sure I got it from english dudes like Tom Scott or David Mitchell
 
It's rather more boring. They're not going to smash down you door, take your TV and lock you up
 
9:14 PM
I suppose the meme never explicitly says that the inspector is a policeman. He gets called "bobby" but maybe that's just his name.
 
@Aran-Fey "dudes" definitely isn't going to sound natural. "Bloke" is probably Northern and even then, I don't think it's common in written text at all
 
"guy" is a pretty ordinary term in my area, FWIW
 
I'd never write "bloke" (I'd never say it, either) but I hear it a bit. I guess when it's written it's a bit more serious and seems odd
By "serious" I just mean, my brain has to process a bit more
 
If you prefer I can go full australian and use "mate"
 
I use "mate" every day. It's not Australian
 
9:19 PM
Every language and dialect has unwritten rules, mostly because they're really hard to write down
The precise scenarios for using "bloke" may be perfectly intuitive and consistent, but you can only acquire the knowledge by living in roganjosh's neighborhood for 15 years.
 
@Aran-Fey same word, different context
 
Hmm, looks like I'm not as well-informed as I thought. Guess I'll play it safe then and go austrian with "Oida"
 
I'm told Aussies use mate when they don't like you, the "like you" word of address is grossly forbidden here :P
 
"Bloke"; A indescript male you're referring to, to your mates in the pub... 10 years ago. "Mate"; And indescript person that you may, or may not, have forgotten the name for, but they aren't horrendous
 
First name "You Wot"
 
9:24 PM
@AndrasDeak It's probably the one that was in the TV license meme, and why I didn't link it
The author may have mixed Australian slang into the ostensibly British scenario in order to look dumb on purpose. Many such cases on the internet.
 
@AndrasDeak I'm fick, me
 
I imagine some confused German speakers :'D
 
can confirm
 
9:40 PM
The earlier mention of TV licences (which we used to have in Australia until the early 1970s) reminded me of the classic Monty Python Fish Licence sketch. montypython.50webs.com/scripts/Series_2/84.htm
 
@Aran-Fey What does "fick" mean here? It's hopefully less bad than the phonetic approach I've taken
 
@AndrasDeak It's complicated, and varies with region, and over time. You can use "mate" to a stranger in a perfectly neutral way, eg to a bus driver. Older Australians tend to say "pal" rather than "mate". Younger Aussies are quite likely to say "dude", possibly with an exaggerated fake American accent. ;)
 
Oook, no need to translate @Aran-Fey. I apologise for that.
 
No need to apologize. Especially since it wasn't even intentional
 
I can give you some context; there are people that say "fick" when they mean "thick" because they can't differentiate between "f" and "th". They also wear it as some weird badge of pride that they don't understand what you're saying because they're "thick". Watch Jeremy Kyle clips and you'll (hopefully) get what I mean
 
9:56 PM
Ah, thanks for the explanation. I thought that might be what you were going for, but wasn't entirely sure. Maybe I'm a bit thick ¯\_(ツ)_/¯
My Youtube results for "jeremy kyle" look very much like stuff I don't want to watch ("You're not the father!", really?)
 
They annoy me to no end. Incidentally, I'm not welcome in several pubs round here. Correlation/causation strikes again
 
There's that American bloke with similar shows whose name eludes me right now... wonder which one came first. Jerry Springer?
@PM2Ring good to know, thanks :)
 
By the way, it is now my headcanon that male Lego figures are referred to as "blocks"
 
hehe
 
10:31 PM
Hello, does anyone know if it's possible to do a merge of pandas dataframes using SQL syntax? Alternatively, is there any easy way to accomplish a merge like this with pandas directly?

select * from DataframeA a
left join DataframeB b ON a.startDATE <= b.someDate AND b.someDate <= a.endDATE
 
Why even bother? You have the SQL already, and it's trivial to get it to a df
It's better to push this into SQL and get a df at the end because the query planner can optimise it
 
Huh, hadn't considered that, sqlite is always an option I guess
 
If you think I'm suggesting dumping to sqlite3 as an intermediary then I've misunderstood
I've read your question 5 times (literally) and I don't think you can get what you want. Why should it be SQL syntax?
 
Because I know how to do what I want to do in sql syntax
Alternatively, (if possible, but I don't think it is) Is there a way just using pandas directly I can say "I want to merge where b.someDate is greater than a.startDate and less than a.EndDate"
 
Rewind a bit. I actually appreciate your answer, but pandas and SQL are not mutually exclusive here
Having the SQL right is, at least IMO, the right thing. Don't do a post-fix in pandas
 
10:44 PM
Yeah, I've verified the sql works the way I want it to in SSMS with temp tables
 
Sidney has a dataframe as square 1, I think.
 
It's trivial to make a df from the result. How did you run the query?
 
they just don't know how to make pandas do what they want, but they know how they'd do it in SQL, if the data were in SQL rather than pandas to begin with
that's my interpretation
 
Andras is correct
 
In which case, I can't help without an MCVE as I'm tired. Also, your database presumably hasn't just vanished. If you're hoping for pandas to be more efficient then, I'm sorry to break the bad news
 
10:47 PM
What database? :P
 
Well, I assume the dfs that they want to merge come from databases
 
Yes, sorry.

I have two dataframes
Data frame A has two relevant columns, startDate and endDate
Data frame b has one relevant column, someDate

I want to do a fancy join where I join rows where someDate is between startDate and endDate

In sql that's pretty easy
Anyway, yeah, no it's from a CSV
😊🔫
 
This would be useful info to include from the start
 
I don't know how a merge like that would work, unfortunately. I could only guess the behaviour if A and B have the same number of rows which correspond to one another.
@12944qwerty not even close
 
10:50 PM
@Sidney What is that in SQL?
 
He said he wanted to insert someDate in between startDate and endDate right?
 
@12944qwerty unless you know pandas and SQL you should probably let roganjosh handle this
@12944qwerty no, please don't help
 
Found the same module with this question stackoverflow.com/questions/30627968
Not ideal, but hey it gets the job done
This answer specifically stackoverflow.com/a/45510620/2254971
 
suspiciously specific
the code in the question shows you how to do it with native pandas at least
 
10:54 PM
Nothing new under the sun, glad to know some other denver coder has solved the problem.
 
If they solved the problem; what are we answering?
 
I think they found that answer after they asked here
 
I spent time reviewing that answer and then backtracking on your question. I don't know what I'm supposed to help with
Fair enough. I guess it's solved?
 
I think so
hopefully if they'd known about that answer they'd have asked "I have this problem, are there any better solutions today?", sparing everyone some work
 
Happy days :)
 
11:08 PM
@Sidney I need to look into this in pandas. The concept is so utterly weird for me; how are conflicts resolved, etc. I honestly didn't know this was possible so thanks for bringing it to my attention
 
@roganjosh Glad I could share something new, and sorry for the bother
 
Not just pandas, SQL too. How can you join on a range?
@Sidney no bother :)
 
select * from DataframeA a left join DataframeB b ON a.startDATE <= b.date AND b.date <= a.endDATE
SQL is scary-powerful-weird
 
Sure, but what happens here algorithmically?
 
11:12 PM
You can't be guaranteed any order in what gets joined. How is that query even stable between multiple runs?
 
@roganjosh just to double-check, did you see the two-liner in the linked question? Merge (join) then filter.
 
Here's the query plan from my MVCE in ssms (bear in mind it's t-sql vs whatever that library is using, but it's simple enough)
 
@AndrasDeak yes
 
OK. That two-liner seemed straightforward to me. But I don't see in Sidney's question that there was a common column to join on, but again I don't know SQL.
 
I'm not sure how it wouldn't be stable, every join has a condition on which to join, I've just provided a composite condition (which is just a condition)
(just to add linguistic variety, I spent roughly 0% of my time in the python world, I'm a .net guy who's helping a friend)
 
11:15 PM
You are getting into python, so it sounds like they are helping you! :P
 
Actually, what you've said is totally fair. It's me that's mixing words
But I don't understand how the overlapping date ranges work in a join, and especially not when they get to pandas
 
@roganjosh is your concern that one given row in B might match multiple ranges in A?
 
Something like that. If you have overlapping ranges then I don't even understand how the SQL works tbh. I might have to push this to tomorrow, though
 
I don't think overlapping ranges should be a problem. Rows in A are treated independently, aren't they?
My 100% uneducated idea is this: the usual join does "value in B equals value in A", and instead we want "value in B is between this value in A and this value in the same row of A"
I mean, yeah, you probably need overlapping ranges to generate weird cases, but I don't think the overlap itself is the issue. If no B falls in overlapping regions, the overlap itself is not a problem. Right?
I have no idea if I understand correctly, I just want to know if we have the same picture here
 
Well, I now want to join something between Mon and Fri, and something else between Tue and Sat. How does it reconcile that? (There's not necessarily any collision at all, but what's going on there in the query?)
 
11:24 PM
I would think that either there's also a column to join on (like in the linked pandas question) that pins down a row in A uniquely, or a row in B on Wednesday matches both rows in A.
What happens with a regular left join when there are duplicate values in the column of A to join on?
 
In the event of an overlap you'd just get a join on both rows
Which yeah wouldn't be great
 
Ok, so I'm, not going mad :P
 
Well we don't know that...
 
But of course this is a light-version of a bigger problem, there are other columns we're joining on too that make it so there won't be overlap
 
so that's actually exactly what the linked pandas question needs
 
11:32 PM
I've gone back through all of this, and are you sure that this is what you meant by the query @Sidney?
Actually, that's not fair to say @Sidney and I don't have time to work through this with you tonight. My apologies
 

« first day (4077 days earlier)      last day (870 days later) »