« first day (4522 days earlier)      last day (442 days later) » 

9:25 AM
@KarlKnechtel uh, that is indeed amusing
only thing I could find is: pypi.org/project/papajohnsapi-mehtaarn000 but then again it doesn't use requests, just selenium
 
 
6 hours later…
3:11 PM
I am absolutely new to pandas. I have the following two lines of code:
df=pd.read_csv('data.csv')
df1=pd.to_datetime(df[['Year', 'Month', 'Day', 'Hour', 'Minute','Second']])
In my original dataframe, df, I have several columns specifying the date, which I want to change into one column. However, I also have my measurements, of say temperature, that do not show up in df1.
 
4:12 PM
Add df1 as a new column to df and remove the unnecessary ones
 
@matszwecja That is my current solution, but I currently need three lines of code for that :(
df=pd.read_csv('data.csv')
df['Date']=pd.to_datetime(df[['Year', 'Month', 'Day', 'Hour', 'Minute','Second']])
df=df.drop(['Year', 'Month', 'Day', 'Hour', 'Minute','Second'], axis=1)
 
4:37 PM
3 lines of code! The horror! All that typing!
At the risk of adding yet another line of code, you could make this a little nicer/DRYer by moving that list of field names to a variable:
df=pd.read_csv('data.csv')
timestamp_fields = ['Year', 'Month', 'Day', 'Hour', 'Minute','Second']
df['Date']=pd.to_datetime(df[timestamp_fields])
df=df.drop(timestamp_fields, axis=1)
 
@PaulMcG OK, I give up. Thanks! :)
 
I never remember when pandas creates a copy vs a slice. Does df[timestamp_fields] create a new DataFrame, to then be passed to pd.to_datetime?
("slice" being what I recall Wes McKinney calls these view-like things in his book - not to be confused with a Python slice)
Also, would this work if timestamp_fields were a tuple instead of a list? I'm starting to bias toward tuple literals vs list literals, since being immutable, they and their initialization get baked into the bytecode directly.
 
5:01 PM
@schn There is nothing wrong with this code. Dropping columns is cheap. I would be more concerned about the runtime than the lines of code, if I were you
@PaulMcG Just to doubly confuse you :D stackoverflow.com/questions/50956542/…
I think it would be a view at first, but it would become irrelevant as soon as you tried to do anything with it
 
5:24 PM
That's numpy, not pandas
In numpy it would be a copy, and a tuple would behave differently (probably raise). With integer indices, of course.
pandas has different (more complex I think) copy vs view semantics which is why you need the whole "setting a copy with a view" or whatever warning
 
I wasn't really addressing the tuple case because that shouldn't exist in a df in the first place, but fair enough
 
I think his point was to do df[('Year', 'Month', 'Day', 'Hour', 'Minute','Second')]
 
"they and their initialization get baked into the bytecode directly" perhaps I'm missing some nuance in this part, then, because I'm not following the suggestion properly sorry
 
That tuple of strings would be stored more optimally than the equivalent list of strings, presumably
Concern outside of pandas, other than whether it would still work
 
 
2 hours later…
7:17 PM
Hi guys, any numpy experts here?
I'm looking to speed up the np.where usage here, it always looks into the array for a value that's unique and only one cell contains it:

for p in big_array_of_unique_values:
  i, j = np.where(big_2d_ndarray_with_unique_values == p)
  results.append(some_fn(i.item(), j.item()))
 
7:31 PM
@FaridNouriNeshat is there any relationship between the two arrays? np.unique can return some handy indices. And can some_fn() be vectorized to work with arrays efficiently?
And how big is big_array_of_unique_values? And the other array?
 
I don't fully understand the logic what I'm optimizing here, I found i, j = np.unravel_index(np.argmax(big_2d_ndarray_with_unique_values), big_2d_ndarray_with_unique_values.shape) does the same thing, and that provided a 13 times speed up already
I found the np.where usage was the top bottleneck
I think it can be vectorized but I'm not smart enough to figure it out.
Basically from what I understood, big_2d_ndarray_with_unique_values is a 2d sequence of numbers and big_array_of_unique_values or path in the code, is a subset of those values, but not super sure.
 
So you don't know the sizes?
 
If I understand correctly, you regularly have to scan through a large array to find specific values? That sounds like a job for a dict
 
7:48 PM
@FaridNouriNeshat the fact that this does the same thing means I'm not willing to chime in without more information
Lots of assumptions missing
 
@Aran-Fey Yeah, that might be better
Actually yeah, sorry I used you guys as my rubber duck. :) I think I got what I wanted, with a couple of other improvements now it's only takes 3% of total time.
 
 
2 hours later…
9:41 PM
Given these two functions, look at the dis.dis disassembly of their bytecodes:
def a_literal_list():
    return ['a', 'b', 'c']

def a_literal_tuple():
    return ('a', 'b', 'c')
a_literal_list:
  9           0 RESUME                   0

 10           2 BUILD_LIST               0
              4 LOAD_CONST               1 (('a', 'b', 'c'))
              6 LIST_EXTEND              1
              8 RETURN_VALUE
a_literal_tuple:
 12           0 RESUME                   0

 13           2 LOAD_CONST               1 (('a', 'b', 'c'))
              4 RETURN_VALUE
 
all the nanoseconds :)
sorry, bytes
 
This code used to use lists for all the spinner characters. For those spinners whose contents (labeled as "frames") are just one character wide, they got converted to strs. In retrospect, the lists could have been tuples.
 
and "aesthetic" could be non-cyclic
 
10:02 PM
Here is the before-and-after.
The spinners are fun to write - I posted this gist with spinners that show "the computer is thinking" with random hex characters, random Braille characters (looks like a War Games style display), and a row of random single dots (like the LEDs on the HP3000 computer, tied to the address bus). I also contributed the idea for the clock spinner.
 

« first day (4522 days earlier)      last day (442 days later) »