« first day (2289 days earlier)      last day (2663 days later) » 
00:00 - 23:0023:00 - 00:00

11:03 PM
cbg
I have a dataframe that is 135x2548, and each cell contains a value like (2358063, 0.014136)
 
debian has no fsck wat??
@Gary cbg
tuple-valued dataframe?
 
I would like to take the sum of each row.
But only the second item (index 1) in each cell.
I am having some trouble writing my for loop.
 
why don't you have a 135x(2*2548)-element dataframe?
 
I guess that's what I have. In Jupyter it tells me "135 rows x 2548 columns"
I am trying to create a new column that contains the row-sum of just the second element in each cell.
 
if an element among those "135 rows x 2548 columns" is "like (2358063, 0.014136)", then you have tuple-valued elements
 
11:07 PM
Ah, okay :) New vocabulary today
 
(2358063, 0.014136) is a tuple
if it really is (2358063, 0.014136)
 
Right, it is a tuple.
 
you should consider transforming your dataframe to have those two elements in different columns, then you can vectorize calculations over them
(and I suspect it'll be more memory-efficient as well, based on my experience with numpy)
 
I can try to do that.
def MergeOnDiff(a, diff):
    b = [a[0]]
    for i in range(len(a)-1):
        if a[i+1][0][0] - a[i][-1][0] < diff+1:
            b[-1] += a[i+1]
        else:
            b.append(a[i+1])
    return b

diff = 60
b = MergeOnDiff(chunk_i_water, diff)
print('merged with diff = ', diff, *b, sep='\n')
 
typically, if you're using numpy/pandas and you're looping over your indices, you should refactor
 
11:10 PM
That's the current routine.
 
isn't that damned slow?
and is your dataframe really a dataframe?
 
:/ yeah...
 
as in pandas dataframe?
(just to be sure)
 
I turned it into a dataframe to visualize it better
Yeah
 
I see
you should turn it into a dataframe to make it faster and more convenient;)
although depending on what you want to do, a numpy array is an alternative option
I'd put this into a 3d numpy array if I were you
135x2548x2
 
11:12 PM
Do you know the command off-hand to do that?
 
then you can just say a[:,:,1].sum(axis=1) or something
@Gary if your data is a list of list of lists, just import numpy as np; arr = np.asarray(a) :)
 
Okay, so I just tried that, and all looks okay. Now, to sum each row, I do a[:,:,1].sum(axis=1) ?
 
you should play around with it, maybe with a 2d array first:)
try M=np.arange(12).reshape(3,4) to get a simple 3x4 matrix
then look at M[0,:], M[:,2], M.sum(axis=0), M.sum(axis=1)
so a[:,:,1] takes the 2d subarray where the third index is 1, i.e. all the 135x2548 second elements of those tuples you had
and .sum(axis=1) will take the row-wise sum of this subarray, which should be a 135-element 1d array
the only problem is that you won't be able to concatenate this 135-element 1d array to your 135x2548x2-element 3d array, but this should still be better than using tuple-valued arrays/dataframes
 
Okay, I will take a look at that.
Thanks!
I am getting an error too many indices for array
 
make sure that it really is a 3d array
print(arr.shape)
 
11:26 PM
Hmm it's 2d: (135,906)
 
yeah, that's not right
your array is probably not homogeneous
I mean, it's not an n x m x l-shaped list of list of lists
you probably have tuples in that 2d array
 
So is there a way for me to write a for loop that just goes over the data frame?
I'm not too concerned about speed.
 
@Andras Deak when I run
pyinstaller --onedir --name=testexe --debug "textexe.py"
I can finaly see what errors and it is
ImportError: No module named PyQt5.QtCore
I have no idea how to fix it o.o
I use this to import qt
from PyQt5.QtCore import *
 
@Gary of course there is, but you don't want to do that:P
why not loop over your native list-of-list-of-lists?
looping over the dataframe won't be any faster, I think
if your dataframe named df really is shaped what you're saying (I'm skeptical), you need something like df.applymap(lambda x: x[1]).sum(axis=1)
the applymap will pull out the second element of each of those tuples, and the resulting 2d dataframe gets summed with .sum()
but if numpy croaked on converting your original list, it's highly possible that your dataframe is corrupted as well, i.e. not all of its elements are 2-element tuples
which scenario should either lead to an error, or wrong result
@Dariusz does that work in a REPL?
 
11:49 PM
REPL ? not sure what you mean. But I tried to change me script to
from PyQt5 import QtCore
and do
--hidden-import=PyQt5
Still nothing works :- (
 
the interactive python shell
I suspect that your program should first work as a python program, and then you should try to compile it into a standalone application (unless I'm misunderstanding your problem)
 
Yeah if I run program from Pycharm everything works. But when I try to run from Exe I get the error above from debug
 
00:00 - 23:0023:00 - 00:00

« first day (2289 days earlier)      last day (2663 days later) »