« first day (4059 days earlier)      last day (896 days later) » 

12:00 AM
I'll try it out some time. I'd agree on that, except for the fact that you can't set indexes or foreign keys in redshift. It treats them as "advisory" (or some other similar, but non-concrete) term
 
depending on the DB though... sometimes even if you have an index on columns... it may just decide based on the cardinality of such columns and the size of the table and how it wants to work it out - it'll just ignore it and hash the other table in someway and do whatever
Amazon Redshift does not enforce unique, primary-key, and foreign-key constraints. - errrrrrrrrrr....
 
The Redshift docs just annoy me: "Define primary key and foreign key constraints between tables wherever appropriate" --> "Do not define primary key and foreign key constraints unless your application enforces the constraints."
 
yeah... the whole point for any normal RDMS is actually to enforce those constraints
(hence the word constraint)
 
EXPLAIN sounds like it came from a programming language called Who. Like ArnoldC.
 
It's a columnar database, so it's for bulk storage. It sucks that it backs our webapps too because it has terrible latency. On the upside, it'll do mega queries in seconds
 
12:06 AM
sounds like Hadoop
well.. sounds like a Hadoop/Cassadra/Pig/Spark setup could be good
shrugs
 
I'm very soon to get redis for all our users. After that, it'll be postgres
One battle at a time
 
any experience using redis? 'cos that can be a pain in the yam itself... what'll you be using that for?
 
Briefly. I don't need to manage it, it's just hot storage for me and it'll be backed on an SSD. They've already written me a wrapper for it
I need to run a process that'll take 7-14 days to run (I don't have enough data to estimate properly) but it'll give me two floats that I need to store against int keys
Backing it with redshift has bumped my solver/API response time from 0.5 secs to 5, so I need something else to store these values
 
okay... I love redis and use it quite a bit in projects - just don't expect it to be ACID as proper RDMS's are
 
I don't. It's just a dict as far as I'm concerned
 
12:20 AM
that's the proper way of thinking about it
just didn't want some disappointment is all
(it does store on disk sometimes, but umm... it's not a DB)
 
That's why I said it's backed by an SSD. The data is around 30k keys, so they'll have it periodically dump to disk and we can just pull it back up if needed
 
30k keys is nothing
 
... exactly
 
unless they're stupidly large keys and you have stupidly large values
 
I'm not the one suggesting problems with the setup :P
I have to work on a platform that gives me basically no privileges (sudo? lol) and we have to run everything through defined data stores... of which there is redshift. Alone. So I have to think of ways around that
Now everyone gets redis because I need hot storage and failover. They're all welcome
 
12:28 AM
think my use for redis now is limited... mostly caching but also got 1.8 million postcodes with lat/lon in it and it's GEO* (check out redis.io/commands) are extremely fast and useful
 
For the other users, it will indeed just be a cache for API calls. As for postcodes, we pull them from Ordnance Survey and OSRM is stupid fast for all the matrix calls I need (though, alas, the project is basically dead now)
I didn't realise that Mapbox runs OSRM development and Valhalla. And they're shifting to Valhalla, which is slower
They also named all the components after Norse gods, making it very difficult to get my head around the source code without constantly cross-referencing
 
I won't mention that I call my current computer Minerva then
@roganjosh yeah... haven't had a copy of the official PAF since start of this year
 
@JonClements unsurprisingly, not in the list. My old computer tower was The Frobnicator. I haven't named my recent laptops
 
one of my first computers was called SELMA after en.wikipedia.org/wiki/Time_Trax - I need to find that on YT or somewhere
 
1:09 AM
Sorry to barge into a conversation like this without much context. My first computer was an intel Pentium which was amazing!
 
 
7 hours later…
8:37 AM
cbg guys
 
9:05 AM
cbg
 
 
5 hours later…
2:02 PM
@Hakaishin Relatable. "Knowing I could do it is enough" seems to be a common sentiment among mathy personalities. Consider the old joke:
> An engineer is working at his desk in his office. His cigarette falls off the desk into the wastebasket, causing the papers within to burst into flames. The engineer looks around, sees a fire extinguisher, grabs it, puts out the flames, and goes back to work.

A physicist is working at his desk in another office and the same thing happens. He looks at the fire, looks at the fire extinguisher, and thinks "Fire requires fuel plus oxygen plus heat. The fire extinguisher will remove both the oxygen and the heat in the wastebasket. Ergo, no fire." He grabs the extinguisher, puts out the flames
 
I know a variant of that joke with prison cells and an unopened can of food. The mathematician ends up dead, with a note on the wall saying "let us assume the can is open".
 
Whenever I start a project in order to optimize/automate some other hobby of mine, such as a video game, I need to be careful not to overdo it. If I understand the problem space too well, or automate away all of the annoying parts, I'll lose interest in the underlying topic
 
2:18 PM
especially with idle games where the whole point is getting marginally more gratification than nerve-wracking menial labour
at least that's my impression as a non-idle-game-player
 
I remember one notable case where I did this intentionally in order to free myself from a game I found to be harmfully addictive. Much of the challenge of high level play revolved around efficient and precise inventory management under time constraints. I wrote a macro system to do the hard parts for me. I played for a day or two more and abruptly stopped. It just wasn't compelling any more.
 
reverse nerd snipe
 
A curious weapon, it can only be used on the wielder
@AndrasDeak A valid perspective. While many games of many genres are like that, idlers are unusually unapologetic about being human Skinner Boxes.
One of the most popular idlers, NGU, has it right there in the title: Number Goes Up. Come, O gamer, and click this button, and lo the number will rise
A bit tongue-in-cheek in their case because NGU has more game mechanics and meaningful strategy than the next ten most popular idlers combined
 
heh
 
2:36 PM
It's actually a little too meaningfully strategic for me. I only got through half of the available content before I thought "ok, I get the idea"
 
oppsies...
 
no worries
 
"Enough of this inventory management and development of heuristics to predict the relative feasibility of completing various voluntary quest lines, I want to push a lever and receive a reward pellet"
 
Cookie clicker?
 
if a question is 2 days old and have not been answered I can post it here...right.?
 
2:39 PM
indeed
 
Cookie clicker is more to my tastes, yeah :-)
 
@Tushar even if it is answered, if you have a reasonable question about it. The point is just that in the first 2 days we expect the main site to handle most questions.
 
Pretty good gratification/menial labor ratio, and polished presentation
 
@AndrasDeak ooh okay gotcha.!
0
Q: Subtracting/Adding the data from previous row in a different column in a pandas dataset

TusharI have the following dataset: Id Company SD ED A111 ABC 1/1/2000 1/1/2001 A111 ABC 1/1/2001 1/1/2005 A111 DEF 1/1/2005 1/1/2010 A111 GHI 1/1/2010 B111 JKL 1/1/2006 1/1/2007 B111 JKL 1/1/2010 1/1/2011 B111 JKL 1/1/2011 1/1/2012 B111 JKL 1/1/2012 1...

 
@AndrasDeak *wrecking
@Tushar adding an MCVE to your question, i.e. a runnable snippet that loads the input dataframe, would benefit people looking at your question.
 
2:44 PM
@AndrasDeak ooh okay...cool....I am new to stack overflow so thanks for the advice.! :) will keep it in mind when posting questions next time.!
 
Can confirm. I am precisely good enough at pandas that I can do basic-to-intermediate data manipulation, but it takes me half an hour to figure out how to create the dataframe in the first place. (not a joke)
"will keep it in mind when posting questions next time" terrific :-) but could you also keep it in mind this time? In other words, can you edit your existing post to add the MCVE?
 
@Kevin oh yeah sure will do....on my work laptop so will edit it once I am on my personal laptop. I have to edit one more question that is also not answered so faar so will edit both of them :)
 
Ok, cool
 
excellent
 
You're already halfway to a good question since you've got neatly formatted expected output. I see a lot of pandas questions where it seems like the asker doesn't even know what they want
I have some sympathy for those askers because "not knowing what I'm trying to accomplish" is my default state in both coding and in life. But if you want to convey the problem to Internet strangers, you gotta solidify things a bit
I went on a tangent there, a bit. Anyway: MCVEs are good.
 
3:00 PM
I understand that now since I started posting questions a bit more actively....I am learning how to post good questions.!
 
"I am learning" is my favorite thing to hear :-)
 
I have 2 data frames of different lengths and both have Id columns....whenever I try to map df1["Id"] == df2["Id"] in np.where function but I get an error (ValueError: Can only compare identically-labeled Series objects) because the data frames are of unequal length.
result["New_End_Date"]=np.where((ghi["max"] == result["Rank"]) & (result["CLEAN_NAME"] == "HONEYWELL / HON") & (result["Employee_Status"] == "Active"), '', result["New_End_Date"])
this is the code I am trying....where I am trying to make New_End_Date column as NULL based on the conditions in where function
 
You should ask yourself what kind of object you'd expect df1["Id"] == df2["Id"] to be
 
here instead of Id i just tried a different column called max and rank....both these columns have same values but different lengths of data frames
@AndrasDeak i just tried after you asked and realised it's returning boolean value and not the Ids that are common in both the data frames
 
yeah, comparison operators on numpy arrays and pandas Series(es) give you bools for each pair of elements (with some leeway in interpreting "each pair of elements")
 
3:16 PM
For me, using a comparison operator in pandas is like plugging in a USB stick. The first three times I guess how they work, I'm wrong.
 
@AndrasDeak ooh okay....cool....so what can I try to get the required result.?
 
Looking at the documentation is the equivalent of peering intently into the USB port to see how the recessed-yet-protruding bit is oriented
 
@Tushar You also have to ask yourself what "the required result" is.
You have two dataframes where the rows have little to do with each other. I assume they are tied together by some column value, probably exactly the Id. You probably need some kind of join. But I'm not a pandas user.
 
Wouldn't it be funny if I had 15 years of professional pandas experience and I was merely pretending to be a humble code janitor
Casually solving half-finished formulas on chalkboards while I mop the classroom floors
"I don't know what a Riemann Hypothesis is, but this reminds me of some homespun country wisdom my momma told me once about the precise summation of the reciprocals of the squares of the natural numbers"
 
 
2 hours later…
5:22 PM
@Aran-Fey Thanks, that fixes the logic there
 
 
3 hours later…
8:21 PM
I was making an app and I had to store the last directory the user used, so I saved it onto a file with the script. But now after making it an app and installing it, it requires admin privileges to write into a file in the installation directory. Then I came across somewhere that such type of practices are bad. So any other alternative suggestions?
 
How do you identify a user?
 
Identify? Anyone that uses the app is a user..?
 
*the user, then. But I think I missed the point of the initial question and I'm gonna sit this one out
 
Sure. I was thinking of maybe storing into the temp folder in windows, I do not think it requires admin privileges.
 
8:39 PM
Firefox sync requires you to enter an email verification code now, so I'm locked out of most of my accounts for the next 3 days cabbage
@CoolCloud Store it in %APPDATA%/name-of-your-program on Windows or ~/.config/name-of-your-program in linux
 
Well, open('%APPDATA%/folder') wouldnt work right?
 
check out appdirs on pypi
 
Managed to save into temp folder with tempfile though
 
that's, like, the worst possible location :P
 
Lol, I could give documents a try with appdirs.
 
8:58 PM
@Aran-Fey that doesn't sound like a very good kind of cabbage
 
Could be worse, but yeah
On the flip side, my ancient relic of a laptop has an SSD now. I can't be mad
off to install linux rbrb
 
...do you also have a dead HDD?
have fun
 
I have a working HDD, but I'm not about to swap it in again just so I can grab my email password
 
Ah, OK, much better news
I have an old HDD that I put in a cheap external HDD enclosure. Could just plug it in.
it was something like 10 euros
Unless the 3 days is the time until you get your hands on an HDD case...
 
9:42 PM
That's when I get back home to my desktop PC. Not planning to use the old HDD for anything, really
 
@Aran-Fey Usefull, this helped, thanks :)
 

« first day (4059 days earlier)      last day (896 days later) »