« first day (858 days earlier)      last day (1330 days later) » 

12:09 AM
0
Q: Pandas dataframe: how to cluster together groups with groupby?

ShanZhengYangI have the following pandas DataFrame. import pandas as pd df = pd.read_csv('filename.csv') print(df) A B C D 0 2 0 11 0.053095 1 2 0 11 0.059815 2 0 35 11 0.055268 3 0 35 ...

 
12:21 AM
0
Q: Replace all 1's at a given index - numpy

akshayI have a binary matrix of size 10-by-10. I wanted to change all the 1's in the matrix at a given index to -1 I was able to get something like this import numpy as np mat = np.random.randint(2, size=(10, 10)) index = [6,7,8,9] mat[(mat[index,:] == 1).nonzero()] = -1 print(mat) when I print th...

 
12:42 AM
0
Q: How to return certain rows from two different dataframes that meet certain conditions in Python?

Li ShuweiI'm new to Python, and here I have a project that I am struggle with that needs your help. Basically I've created two dataframes, first dataframe is called dfm_2013, which looks like this name births 0 Aadhya 135 1 Aadya 69 2 Aahana 15 3 A...

0
Q: Access a list within an element of a Pandas DataFrame

MichaelI have a Pandas DataFrame which has a list of integers inside one of the columns. I'd like to access the individual elements within this list. I've found a way to do it, by using tolist() and turning it back into a DataFrame, but I am wondering if there is a simpler/better way. In this example...

 
1:04 AM
1
Q: Pandas: convert categories to numbers

Sachin_rukSuppose I have a dataframe with countries that goes as: cc | temp US | 37.0 CA | 12.0 US | 35.0 AU | 20.0 I know that there is a pd.get_dummies function to convert the countries to 'one-hot encodings'. However, I wish to convert them to indices instead such that I will get cc_index = [1,2,1,3]...

 
1:30 AM
0
Q: Pandas fill cells in a column with NaN values, derive the value from other cells in the row

harshitI have a dataframe: one two three 0 1 2 3 1 1 1 1 2 3 7 NaN 3 2 3 5 ... I want to fill column "three" inplace (update the values) where the values are NaN using a machine learning algorithm. I don't know how to do it inplace. I tried: df[~d...

0
Q: Python Matplotlib How to create subplots?

Shawn ZhangI am having quite a bit of trouble understanding how to create good subplots. I want to create a figure that is similar to the one shown below. Does anyone know how I could set up a similar template as this?

 
1:55 AM
0
Q: Merge a list of pandas dataframes

JakeThere has been many similar questions but none specifically to this. I have a list of data frames and I need to merge them together using a unique column (date). Field names are different so concat is out. I can manually use df[0].merge(df[1],on='Date').merge(df[3],on='Date) etc. to merge each...

 
2:13 AM
0
Q: How to delete entire row of data set given a condition on a column in csv file?

Technolohic27Here is a snippet of the following data-set in csv format: quantity revenue time_x transaction_id user_id 1 0 57:57.0 0 0 0 1 0 18:59.0 0 1 I want to delete the entire row when the user_id is empty. How do I do this in python? ...

 
2:24 AM
0
Q: How to parse json files faster using pandas

user113531I have multiple compressed files containing lines of nested json objects. I would like to extract the information as a pandas dataframe or some table structure. The current workflow I am using using pandas json_normalize: data = [] for file in files: with gzip.open(file ,'rb') as f: ...

 
2:47 AM
0
Q: How to select column names that have non-zero values for a given row in a pandas dataframe?

AlanHI have a pandas dataframe and I need to find out the labels of the columns of nonzero elements of a particular row. So for example: columns = ['a', 'b', 'c'] df = pd.DataFrame([[1, 0, 1], [0, 1, 0], [3, 5, 2]], index=columns, columns=columns) For column a, I would want to have ['a', 'c'] be r...

 
3:40 AM
0
Q: Plotting exponential function python

Prince PI get a linear graph when trying to plot exponential function: `def graph(formula, x_range): x = np.array(x_range) y = eval(formula) plt.plot(x, y) graph('100*(np.power(0.8, x))', (0,100))` what am I doing wrong? output_image

 
3:54 AM
0
Q: How to generate a list of all (x, y, pixel value) tuples from a 2D image matrix with numpy?

Tim CronUsing python and probably numpy, how can I generate a list of all coordinates with a ( x, y, pixelValue(x,y) ) format? I can achieve this with for loops using the following: img = cv2.imread('messi5.jpg',0) data = [] for x in range(img.shape[1]): for y in range(img.shape[0]): data....

0
Q: Multiplying arrays in python

A1122I have a simple question. I have an array with shape (12,2) and another with shape (2,). I am just trying to apply multiplication on the two arrays. I am coming from Matlab/R where matrix multiplication is straightforward. When I try to multiply the two arrays using np.dot() I get the following e...

 
4:39 AM
0
Q: What is the simplest way to make matplotlib in OSX work in a virtual environment?

Charlie ParkerI just discovered that matplotlib has issue with virtual environments. I tried the solutions in the FAQs but they didn't work. An ideal solution should only involve pip commands, but it might be hard or unrealistic to request that. Anyway, I tried: $ pip install TKAgg Collecting TKAgg Could no...

 
4:51 AM
0
Q: Pandas hdf cannot match existing table structure on appending data

user113531First I ensured all types are the same: df[['col1', 'col2', 'col3', 'col4']] = df[['col1', 'col2', 'col3', 'col4']].astype(np.float64) This is how I called the append function with pd.HDFStore(sPath, complevel=5, complib='blosc', encoding='utf-8') as store: store.append(self._sNode, df, m...

 
5:47 AM
0
Q: How can Increase the efficiency of this code? Reddit Bot

user502301import praw import time import database # Create a user agent and log into Reddit UA = 'OZBOZZ v0.1' r = praw.Reddit(UA) # Log UA into reddit with my personal user name and password r.login('', '', disable_warning='True') subreddit = r.get_subreddit('listentothis') current_time = int(time.time(...

 
6:05 AM
0
Q: How to drop rows not containing string type in a column in Pandas?

Harsh WardhanI have a csv file with four columns. I read it like this: df = pd.read_csv('my.csv', error_bad_lines=False, sep='\t', header=None, names=['A', 'B', 'C', 'D']) Now, field C contains string values. But in some rows there are non-string type (floats or numbers) values. How to drop those rows? I'm ...

 
6:31 AM
0
Q: Efficient GROUP BY query on numpy recarray

schwiftybotI have a dataset of product purchases logs with 6 columns: purchase_date, user_address, user_id, product_id, brand_id, retailer_id. All contain integers, except user_address which is a string. I need to get the top 5 brands selling the most items over the whole dataset, i.e. the ones having the...

 
7:26 AM
0
Q: flexible query based on two numpy arrays

ClausI would like to create a somewhat dynamical query based on numpy array. Ideally, the indices of one array (uu) should be returned, given the conditions for each column in a second array (cond). The sample below demonstrates what I have in mind, and it works using a loop. I am wondering if there ...

0
Q: How to set path to matplotlib custom style?

RezneyAs per matplotlib documentation: "You can create custom styles and use them by calling style.use with the path or URL to the style sheet." I tried: plt.style.use('/usr/share/mygraph/mystyle.mplstyle') but it is returning: ValueError: '/usr/share/mygraph/mystyle.mplstyle' not found in the st...

0
Q: Python: Pandas dataframe sum

SpeedEX505I try to learn how to work with pandas dataframes. My dataframe has 4 columns A,B,C,D. For index (A,B,C) there are multiple values of D. I want to merge these rows and sum the values of D. I have: ╔═══╦═══╦═══╦═══╦═══╗ ║ ║ A ║ B ║ C ║ D ║ ╠═══╬═══╬═══╬═══╬═══╣ ║ 1 ║ 1 ║ 2 ║ 3 ║ 5 ║ ║ 1 ║ ...

0
Q: Transposing the data in python

Puneet TripathiMy Data looks something like this cust c1 p1 c2 p2 c3 p3 1 2 3 4 5 6 7 8 9 10 11 12 13 14 I want to transpose this data and return in below format: cust c p 1 2 3 1 4 5 1 6 7 8 9 10 8 11 12 8 13 14 Please Can someone suggest what to use?

0
Q: ValueError when apply function to data frame in python-pandas

user5779223Let's say I have a very simple data frame: import pandas as pd df = pd.DataFrame(np.full((6), 1)) Now I am going to define a function that generate a numpy array with random length and add the given value to the tail: import numpy as np def func(row): l = np.full((np.random.random_integet...

 
7:59 AM
0
Q: Double graph with matplotlib: Attribute error

PapieI have got a pandas dataframe like this: nan 0 ingredient contribution count 0 0.0 cracker 0.088844873 11 1 2.0 water 0.044386494 125 2 3.0 oil 0.034567456 10 3 4.0 flour 0.030855063 186 ... I would like to create a dou...

 
8:13 AM
0
Q: Python dimension mismatch for multiplication

A1122I have a few variables listed below. How come I receive the error message at the bottom when I try to run this block of code? Ind_var[i,:] should be of shape (1,2) so by matrix multiplication (1,2) * (2,2) is (1,2)? Sorry I'm new to python. import numpy as np spread_len = 10 n_param = 2 Q = np...

0
Q: How can I reduce time complexity from O(N^2) here?

Kaushal28Problem from Hacker Earth: Inverted GCD: Given an array a of N numbers , you have to find the number of pair of indices i and j that satisfy the following relation: i < j ai > aj gcd( ai , aj )=1 Input: The first line of the input contains a single integer N - d...

0
Q: Group by with Panda

l4nd0Hi I have a dataframe and I get the following index1 index2 results 0 0 0 8.605016 1 0 1 8.600410 2 0 2 8.585075 3 1 0 8.550355 4 1 1 8.557221 5 1 2 8.557640 6 2 0 4.360486 7 2 1 4.362304 8...

 
8:36 AM
0
Q: numpy indexed list error

BaRudI am trying to create an indexed np.ndarray as: dos2[q] = np.empty((dos[q].shape[0], dos[q].shape[1] + 1), dtype=dos[q].dtype) with q properly defined. I am getting error: NameError: name 'dos2' is not defined The shape and type of dos[q] is: <class 'numpy.ndarray'> (301, 18) If I create...

 
8:53 AM
0
Q: pandas delete row if zero counts too more

liu gangI have a DataFrame like this: a b c d 1 0 0 0 0 1 0 7 5 2 0 4 6 3 0 0 0 0 8 8 0 7 7 7 0 0 0 1 1: fow each row, if the counts of 0 is >90% of the column counts, then delete the row. 2: fow each column, if the counts of 0 is >90% of the row counts...

0
Q: Pandas with Decimal?

cjm2671I'm calculating some standard deviations which are giving FloatingPointErrors. I wanted to try converting the data series to Decimal (using https://docs.python.org/3/library/decimal.html), to see if this fixes my issue. I can't seem to make a pandas series of decimal. How can I take a normal pd...

 
9:11 AM
0
Q: Why is numdifftools so inaccurate? How does it work?

luongminh97I'm using python's numdifftools library to perform derivatives. However, a few tests prove the library is highly inaccurate: import numpy as np from numdifftools import Derivative # Result should be 1/2 or 0.5 Derivative(np.log, i=1)(2.0) >>> array(0.5493061443340549) Is there a way to fix thi...

0
Q: pandas delete the 0 appears most times

liu gangI have a DataFrame like this: a b c d 1 0 0 0 0 1 0 7 5 2 0 4 6 3 0 0 0 0 8 8 0 7 7 7 0 0 0 1 1: fow each row, if the counts of 0 is >90% of the column counts(in this case: mean: 0.9*4 ), then delete the row. 2: fow each column, if the counts of...

 
9:26 AM
0
Q: Python plot of a piecewise defined surface

IgnatiusI am trying to make a 3d plot of a surface that is defined in different ways for different regions. As an example, take f(x,y) that is defined as 1 if x > y and as x^2 if x <= y. I defined f with logical operators, and tried to plot it with the "plot_surface" function, evaluating it in a grid. U...

0
Q: Is this a bug in Pandas? FloatingPointError on ewm().std()

cjm2671When I execute the following, I get a FloatingPointError. import traceback import warnings import sys import pandas as pd import numpy as np np.seterr(all='raise') def warn_with_traceback(message, category, filename, lineno, file=None, line=None): traceback.print_stack() log = file if ha...

2
Q: Save a pandas dataframe as table in Image or pdf document with nice multi index display

Luce PhilibertI'am trying to include a dataframe with multi index in a report in pdf. I would like to have a nice table output. I have found this 2 solutions: pandas.df -> HTML -> pdf import pandas as pd from IPython.display import HTML import pdfkit # df generation df = pd.read_csv(path...

0
Q: Python pandas 'ffill' method not working

2dorIn the book 'Python for Data Analysis' there is an example using pandas' Series data structure for reindexing. I copied this simple code into an iPython notebook and run it but it does not change obj3. obj3 = Series(['blue', 'purple', 'yellow'], index=[0, 2, 4]) print(obj3) obj3.reindex(range(6)...

 
9:58 AM
0
Q: Create a column based on condition pertaining to 2 other columns

neuronI have two columns in a pandas DataFrame (let's call the 'col1' and col2'). Both contain True/False values. I need to create a third column from these two ('col3'), that will have a True value for a record if one or the other of the two columns has a True value in that record. Currently, I'm d...

0
Q: What is the cleanest way to use truth operations on multiple Pandas series?

draco_alpineI currently have two bool pandas.series objects. They are derived from the same source so I have no worries about indexing etc. What I need to do is combine them within an or truth table; ie) T T = T T F = T F T = T F F = F What is the cleanest way to achieve this? Thank...

 
10:09 AM
0
Q: How can i reduce running time significantly, pandas data frame slicing

VinceFollowing are two test code As i tested test function are bit faster than test1 Running time are around 40ms Total row are 1000 Is there any good way to reduce running time?? Integer i is essential condition def test(data): a=data['Stock_On'][i - 1] data['Stock_On'][i] = a...

 
10:34 AM
0
Q: Efficient method to transpose submatrix in a numpy array

happy fishI have a very large numpy array of matrix that has the structure: np.array([ [[1, 2], [3, 4]], [[5, 6], [7, 8]], ]) my expected output is np.array([ [[1, 3], [2, 4]], [[5, 7], [6, 8]], ...

 
10:46 AM
0
Q: How to work with 14-bit data in Python?

Low Yield BondMy current workflow involves dealing with 8-bit resolution data, written in 16-bit format and 64-bit format. For example: b = np.fromfile(file, dtype='uint8') data1 = b[:2].view(np.int16) data2 = b[2:10].view(np.int64) # alternative: struct.unpack I have just received 14-bit resolution data -...

 
10:57 AM
0
Q: attaching "metavariables" to variables in theano to implement Adam or RMSProp

fstabAdam and RMSProp are variants of SGD that require storing adaptive quantities to each parameter that is going to be optimized. I wonder: is there any way to solve this problem in an effective and elegant manner instead of just keeping explicitly cumbersome replicas of all the variables that cont...

0
Q: Scipy.misc import not working

J.DoeI'm having problems with importing scipy.misc in my python script. Now I know other people also posted questions concerning this, but their solutions are not working for me. The program I am writing begins like this: import matplotlib as plt from matplotlib.pyplot import imshow from PIL import ...

0
Q: Ploting colorbar with Basemap and Matplotlib error

AmulusI made a prototype of application that's plot wrf data using python and display the figures on a web interface as a project of my internship. I made it on my local computer and it worked well but when I tried to install my application and make it work on the server I got this error : AttributeE...

 
11:22 AM
0
Q: Inherite from matplotlib

Steff80I´d like to create my own plotting class as follows but I cant find a way to inherite from Figurewhile using the plt module, see below. Either it inherits from Figureor it changes the tick_params. Figure is a class so i can inherite but plt ist a module? I am just a beginner trying to find my way...

0
Q: how o convert datetime.time columns in dataframe to string

Houda LI've been struggling to convert two columns in a pandas. The frame contains many columns, and 2 columns with dates: 'datelog'(is a date) and 'Timeofday' (is a time). The column datelog is a string. The column Timeofday is a datetime.time() format. The dateframe display is as folows: datel...

0
Q: connecting strings in pandas

EnergyNetI try to sum up columns with string data. The Problem is that I want to ignore the NaN, but I didn't find a solution. The Dataframe look like this: s=pd.DataFrame({'A':['(Text,','(Text1,'],'B':['(Text2,','(Text3,'],'C':['(Text4,','(Text5,']}) A B C 0 (Text, (Text2, (...

0
Q: Getting error while making column using DataFrame and Pandas

InamI am following this example tutorial I am using spark1.5.1 and python 3.5 anaconda distribution.My code was running fine untill I reached at 7th cell this pd.DataFrame(CV_data.take(5), columns=CV_data.columns) I am getting error on this cell Py4JJavaError Traceback (most recent call last) <...

 
12:29 PM
0
Q: Making lists from pandas.dataframe.iloc slice doesn't work while pandas.dataframe.values slice work

Ando JuraiI want to agregate columns from a dataframe into a new columns So I used: newdataframe["agregate1"]=list(df.iloc[:,0:4]) but it doesn't work. It returns an ["intensity_x"] list (with x as column index), these strings being the names of my dataframe columns. While newdataframe["agregate1"]=l...

0
Q: Quotation marks in XML text interfering with subsequent JSON parsing

SANBI samplesI have this helper function that gets rid of control characters in XML text: def remove_control_characters(s): #Remove control characters in XML text t = "" for ch in s: if unicodedata.category(ch)[0] == "C": t += " " if ch == "," or ch == "\"": t ...

0
Q: Python optimize to short version perhaps using intertool/filter/lambda

Andrew NodermannThe point is that the filter discards a string contains numbers or words in the string except the first word does not begin with a upper letter. How to make a short version? raw_list = ['Apple FResh', 'Americano', 'animator Bob', 'tea$.sugar Menu', 'tea And Sugar', 'hello Tom And jeRRy', ...

 
12:58 PM
0
Q: Can conversion from a pandas DataFrame to a raw numpy array improve ML performance?

Brian BienA pandas DataFrame has the limitation of fixed integer datatypes (int64). NumPy arrays don't have this limitation; we can use np.int8, for example (we also have different float sizes available). Will scikit-learn performance generally improve on large datasets if we first convert the DataFrame to...

0
Q: how to make a null query selecting all the items

Ruggero TurraI really like the query method. What is the null query that select all the items? I have tried df.query("True"), but it doesn't work. The only thing I have found to work is df.query("index == 0 | index != 0").

 
T J
Duplicate, OP re-posting same question within hours jQuery live search - Go back to first result after the page has found all available results - Matty - 2016-06-29 02:04:56Z
 
0
Q: Separating 3-d array into 2-d arrays along the respective columns

Dhanajit Brahmatemp1 = tempObj[0][0] temp2 = tempObj[0][1] if len(tempObj) > 1: for i in range(1, len(tempObj)): temp1 = np.vstack((temp1, tempObj[i][0])) temp2 = np.vstack((temp2, tempObj[i][1])) The code is in Python. In the above code, I am trying to separate the numpy 3-d array (tempObj) into ...

 
1:32 PM
0
Q: Replace cells in row if condition met

SLEIs there a way to for Less, Middle Greater columns. If it is greater than 0 to replace the ROW with 1. So the top row would become? Conc Less Middle Greater Date 2005-03-02 00:00 10.3 0.000000 1 1 This is th...

0
Q: Changing multiple column names but not all of them - Panda Python

Antonio López RuizI would like to know if there is a function to change specific column names but without selecting a specific name or without changing all of them. I have the code: df=df.rename(columns = {'nameofacolumn':'newname'}) But with it i have to manually change each one of them writing each name. ...

0
Q: How to write output to pandas

e3e4sSo the output of a short little iteration of an xml file I did looks like this: {'LegalNm': 'LAUNCH ANGELS MANAGEMENT COMPANY, LLC', 'SECRgnCD': 'BRO', 'SECNb': '802-79280', 'FirmCrdNb': '170562', 'BusNm': 'LAUNCH ANGELS'} {'LegalNm': 'JACOBSEN CAPITAL MANAGEMENT, LLC', 'SECRgnCD': 'DRO', 'S...

0
Q: Matplotlip bar plot not working as expected

Sijan BhandariI have a dataset of bike trips for a month. I am trying to observe the peak hours of trips. So, I think the good way is to get the frequency of hours along the whole month and plot " hours vs. frequency " using bar plot as follows : times = pd.DatetimeIndex(trip_data.starttime) plt.figure() plt...

 
1:47 PM
0
Q: Python Dict to numpy multidimensional array

MechanicI have a python dictionary with keys (0,1,2....n) with each key holding the location co-ordinates in a tuple such as {0:(x1,y1), 1:(x2,y2), 2:(x3,y3), ....., n-1:(xn,yn)} I want to create a multidimensional array like coordinates= np.array([ [x1, y1], [x2, y2], ... ...

0
Q: Initialization of Xgboost DMatrix reduce features number

SpanishBoyI am trying to understand following case: when I create new xgbost DMatrix xgX = xgb.DMatrix(X, label=Y, missing=np.nan) based on input data X with 64 features I got the new DMatrix with 55 features What the magic is doing here?

0
Q: Find first occurrence of any string in a specific column of a dataframe pandas python

brandogI have a pandas dataframe and need to find the first occurrence of any string in a specific column (lets say column 0) of the dataframe. Then drop all data including and after this string. How would I go about this?

1
Q: Pandas: reorganizing a dataframe

LucyI have this dataframe with four columns: df = pd.DataFrame({'name': {1:'Aaron', 2:'Bobby', 3:'Carl'}, 'job': {1:'mechanic', 2:'carpenter', 3:'plumber'}, 'experience': {1:12, 2:23, 3:14}, 'type': {1:'owner',2:'contractor',3:'assistant'}}, ...

0
Q: Pandas dataframe.to_csv file rows out of alignment with each other

adrian.berryI want to write my DataFrame to a csv file. The result is shown below. The raws are out of alignment with each other. Do you have any suggestion to solve this problem ? thank you ! Code : dataframe.to_csv(data,sep="",index=False,header=False,encoding='ascii', float_format='%10...

 
2:27 PM
0
Q: Numpy and diff()

J.JI'm trying to create a diff of my sorted numpy array such that if I record the value of the first row, and the diffs, i can recreate the original table but store less data. So here's an example of the table: my_array = numpy.array([(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), ...

 
2:55 PM
0
Q: Python - Easily importing csv arrays from multiple column sets

DarthLazarI'm trying to import this data set via csv. import numpy as np import csv Snap1 = [] Snap2 = [] with open('BrownM.csv', 'r') as f: reader = csv.reader(f, delimiter=',') for row in reader: Snap1.append(row[0]) Snap2.append(row[1]) Snap1 = np.array(Snap1) This works f...

0
Q: How do I create a wheel from a django app?

thebjornThe reusable app docs (https://docs.djangoproject.com/en/1.9/intro/reusable-apps/) tells you to list template and static files in MANIFEST.in, but it doesn't look like python setup.py bdist_wheel looks at that file at all. I've seen references to data_files but those files go in directories rela...

 
3:07 PM
0
Q: Rescaling to (0,1) certain columns from Pandas Python dataframe

Antonio López RuizI have the following type of dataframe: Channel Region Fresh Milk Grocery Frozen Detergents_Paper Delicassen 0 2 3 12669 9656 7561 214 2674 1338 1 2 3 7057 9810 9568 1762 3293 1776 2 2 3 ...

0
Q: Making custom signal processing block in Gnu Radio

Maunil VyasI am trying to make a signal processing block in gnu radio with the help of python language. Functionality which i want from this block is: ex .Input as a vector [1,2,3] ->Output (1+2+3)=6 Here i am attaching my code, i won't able to get the desirable output so please help me with this concer...

0
Q: Plotting parallel coordinates in pandas/python

C. StuckiI am trying to use pandas in python to plot the following higher-dimensional data: data = pandas.read_csv('ParaCoords.csv',sep=',') data x1 x2 x3 x4 y 0 0.5917 0.7165 0.1809 0.4262 377.9537 1 0.5540 0.9420 0.3518 0....

 
3:29 PM
0
Q: matplotlib animations from a remote hosted ipython notebook

keithSchumacherMy home desktop is the guts of my workstation (good CPU + memory). I often ssh in from my chromebook and port forward ipython notebooks in order to work remotely. However, I have run into issues with displaying matplotlib animations remotely (static plots work fine). The animation appears as a ...

 
3:45 PM
1
Q: Merging two Pandas dataframes on multiple differently-named columns

Sir Larry WildmanI have two Pandas dataframes that I would like to join on two columns. The columns are named differently in each df. In [11]: df1 Out[11]: ID_1 ID_2 Hour 132 235 1 133 236 2 134 237 3 In [12]: df2 Out[12]: ID Hour Price 132 1 17.2 133 2 14.6 134 3 21.3 I wou...

0
Q: Splicing MultiIndex (.loc not working)

CarolineI am having trouble with slicing a MultiIndex. I have tried several tecniques by now but escept for one I don't get them to work. My DataFrame is composed of several rows like this: means Out[25]: Total DL DM Mouse Genotype Intensity ...

0
Q: Using numpy data types in argparse

patrickI am setting up a Argparse parser to read some user input through the shell. The input will be used to extract data from a pandas DataFrame containing strings and numbers. I want to automatically set the type= argument in Argparse.add_argument() to match the datatype of the respective column. M...

 
4:29 PM
0
Q: SQLAlchemy connection goes to sleep without query returning

adamHI'm trying to read a large amount of data from a db, >10,000,000 rows. I connect to the database like so: def mysql_conn(self): self.connect_string = 'mysql+mysqlconnector://{}:{}@{}:{}/{}'.format(self.user, self.dbPwd, self.host, self.port, self.dbName) self.dbconn = create...

0
Q: Solving a large number of small linear systems

Nico SchlömerI need to solve a large number of 3x3 symmetric, postive-definite systems with Python. So far, I did res = numpy.zeros(n) for k, obj in enumerate(data_array): # construct A, rhs, idx from obj res[idx] += numpy.linalg.solve(A, rhs) This produces the correct result, however is also qui...

0
Q: Pandas Dataframe column getting implicitly converted to Nan after merge

vishal bhattBelow are small samples of my 2 pandas dataframe: In [65]: df1 Out[65]: Send_Agent Send_Amount Country_Code 0 AWD120279 85.99 KW 1 API185805 22.98 PH 2 ANO080012 490.00 NO 3 AUK359401 616.16 GB 4 ...

0
Q: Python : Failed to interpret as pickle when loading .dat file?

RangooskiI have a numpy, saved as data.dat file in python 3+ . When I load it in python 2.7 import numpy as np data = np.load("data.data") it throws error as --------------------------------------------------------------------------- IOError Traceback (most recent c...

3
Q: Downloading large file in python error: Compressed file ended before the end-of-stream marker was reached

ClickyButton.comI am downloading a compressed file from the internet: with lzma.open(urllib.request.urlopen(url)) as file: for line in file: ... After having downloaded and processed a a large part of the file, I eventually get the error: File "/usr/lib/python3.4/lzma.py", line 225, in _fill_b...

0
Q: Zodb killing process after reading multiple fs

notaryaI have used ZODB BTrees to store some data (~1-2GB per file). I created the ZODB files using: storage = FileStorage(os.path.join(zodb_dir, 'trainPatches.fs')) db = DB(storage) connection = db.open() root = connection.root() root.patches = BTrees.OOBTree.BTree() # take columns from a pandas dat...

0
Q: find the biggest string in each row in a Pandas DataFrame

GuilhermeI am new to Pandas and I am trying to get the biggest string for every row in a DataFrame. import pandas as pd import sqlite3 authors = pd.read_sql('select * from authors') authors['name'] ... 12 KRISHNAN RAJALAKSHMI 13 J O 14 TSIPE 15 ...

0
Q: Delete rows from pandas DataFrame with non-unique index

J JonesI am looking for a way to delete rows in a pandas DataFrame when the index is not guaranteed to be unique. So, I want to drop items 0 and 4 from my DataFrame df. This would be the typical code you would use to do that: df.drop([0, 4].index) If each index is unique, this works fine. Howeve...

 
4:59 PM
1 message moved from SO Close Vote Reviewers
 
0
Q: How to multiply each row in pandas dataframe by a different value

johnchaseI am trying to multiply each row of a pandas dataframe by a different value and wondering what the best way to do this is. For example if I have the following dataframe: import numpy as np import pandas as pd df = pd.DataFrame(np.random.randn(2, 3)) df 0 1 2 0 -1.28331...

1
Q: pandas groupby aggregation results from another dataframe

JBrI'm having trouble using groupby and aggregation when working w/ multiple dataframes. I'm trying to calculate num_maint_over_$90 from two different dataframes. cars_dict = {"ABC123": ["Ford", "Compact_Car"], "XYZ148": ["Chevy", "Truck"], "ASX133": ["Ford", "Truck"], "ADS111": ["Porsche", "Sports...

0
Q: Reading CSV with Python's Panda Error

Antonio López RuizI want to import a csv into Python with Pandas, also be able to use one as the columns as a transaction ID in order for me to make association rules. (link: https://github.com/antonio1695/Python/blob/master/nearBPO/facturas.csv) When I try to import the csv with the following code: import pan...

0
Q: concat turns values to nan data

SLEI have this code: gg=df_met[['Less','Middle','Greater']].resample('h').mean() Filtered_mean=Filtered[['Conc']].resample('h').mean() result = pd.concat([Filtered_mean, gg], axis=1, join_axes=[df1.index]) Reduced_result=result.dropna(axis=0,how='any') Where gg is a file: ...

 
5:20 PM
3
Q: Python Spark / Yarn memory usage

Joe KI have a spark python application that is being killed by yarn for exceeding memory limits. I have a step that involves loading some resources that are a bit heavy (500+ MB), so I'm using mapPartitions. Basically: def process_and_output(partition): resources = load_resources() for record...

 
5:30 PM
0
Q: Numpy flatten array of array from cv2

WboyWait! Before you 'harrumph' and downvote, this seems like a repeat question but I've looked at the other questions and they don't really fit my use case. I was reading images one by one into a list with cv2, with the intention of running a classifier over them. eg: lst = [] for picture in dire...

 
5:42 PM
0
Q: How do I get periodic curve fit from discrete data in python?

PyNoobI wand to do integration on a dynamic system like to the ODE x_ddot + d*x_dot + k*x = a*sin(omega*t) with one modification: The external force a * sin(omega * t) must be replaced by another periodic signal which is based on some measurement data. That means: I need do curve fitting on my discr...

-3
Q: How to create a new dataset by writing regex using python?

Lakshmi NarasimmanThis is my dataset(dataframe) having 10000 records having three columns as below: Product_id order_id item_id 0 120GB Hard Disk Drive with 3 Years Warranty fo... 3950 8 1 TOSHIBA SATELLITE L305-S5919 LAPTOP LCD SCREEN... 35099 324 2...

0
Q: Customize marker style with shapes and letters in Python?

pl0222is there a way to customize markers in a scatter plot to combine letters and shapes? For example, a filled circle with a number in it? The documentation on markers (http://matplotlib.org/api/markers_api.html) doesn't seem to include this info. I found something using the pyx package (http://pyx.s...

0
Q: matplotlib - How to save an array as image with overlayed text?

hazrmardI have a 2x2 array that I need to save as a png. I also need to add a text label to the image. So far, I have tried two approaches, none of which is optimal: I use the matplotlib.image module to save the array directly as an image: matplotlib.image.imsave(FILENAME, ARRAY, cmap=plt.cm.binar...

0
Q: Plotting Multiple Histograms in Matplotlib - Colors or side-by-side bars

ScientistGirlProblem : When Plotting Multiple Histograms in Matplotlib, i cannot differentiate a plot from another Problem as Image : ** **Minor Problem : The left label 'Count' is out of the image, partially. Why? Description I want to plot the histogram of the 3 different sets. Each set, is an array wit...

 
6:00 PM
0
Q: Django Left Outer Join

RodericDayI have a website where users can see a list of movies, and create reviews for them. The user should be able to see the list of all the movies. Additionally, IF they have reviewed the movie, they should be able to see the score that they gave it. If not, the movie is just displayed without the sc...

0
Q: Bokeh - matplotlib - histogram - blank plot

George C I have the following code that runs in a jupyter notebook with python 3: from bokeh import mpl from bokeh.plotting import figure, show, output_notebook import matplotlib.pyplot as plt output_notebook() plt.hist([1,2,3,3,3,3,4,5,4]) show(mpl.to_bokeh()) This plots me a blank plot with no dr...

0
Q: How to turn Pandas' DataFrame.groupby() result into MultiIndex

taerathSuppose I have a set of measurements that were obtained by varying two parameters, knob_b and knob_2 (in practice there are a lot more): data = np.empty((6,3), dtype=np.float) data[:,0] = [3,4,5,3,4,5] data[:,1] = [1,1,1,2,2,2] data[:,2] = np.random.random(6) df = pd.DataFrame(data, columns=['kn...

 
6:18 PM
0
Q: Trying to learn Conway's Game of life in python

RaymondHi there I was wondering if there are any really good tutorial videos or websites that walks you through the implementation process of conway's game of life in python? Any links at all would be appreciated.

0
Q: is it possbile to edit matplotlib plot interactively?

BaRudI am not sure if this is an acceptable question in SE. I am wondering if it is possible to edit matplotlib plot interactively. i.e., # plot plt.plot(x, y[1]) plt.plot(x, -1.0*y[2]) plt.show() will open up a tk screen with the plot. Now, say, I want to modify the linewidth or enter x/y label. ...

1
Q: Pandas: Changing a specific row to percentages

KevinI have a row in a Pandas Data frame that contains the sales rate of my items. A look at my data: block_combine Out[78]: END_MONTH 1 2 3 4 5 Total Listings 168 219 185 89 112 Total Sales 85 85 84 41 46 I can easily calculate the sales % by doing the followi...

1
Q: Merge 2 Pandas DataFrame, Take One Record Over Another if Index Matches

getgladI want to merge two Pandas DataFrames, but anywhere an index matches I only want to merge in the row from a specific df. So if I have df1 A B apple 10 xyz orange 11 pqs df2 A B apple 11 xyz grape 12 def I would get df3 A B apple 10 xyz orange 11 pqs grape 12 de...

0
Q: Python, Matplotlib: Suptitle with arbitrary number of vertical figures

The Quantum PhysicistI have a website that produces (depending on available data stations running) an arbitrary number of plots (as an image), that are vertically stacked over one another. An example is the following: The problem is that depending on the number of vertical plots, the suptitle (top title) goes to a...

1
Q: Memory order behavior issue when converting numpy array to QImage

AjeanI have a simple viewer in PyQt4, and I'm having some strange behavior when converting from a numpy array (8-bit grayscale) to a QImage to display. It seems to me like something is going haywire when I try to construct a QImage from a transposed array, even though the memory order should have bee...

0
Q: How do you use numpy's fromfile to store files in an ndarray?

LouI'm still pretty new to python. I've been able to write a program that will read in a file from binary and stores the data that's there in a few arrays. Now that I've been able to complete a few other tasks with this program, I'm trying to go back through all my code and see where I can make it m...

0
Q: NumPy: Sort and sum data into array

Nico SchlömerI have a (large) data array and a (large) list of lists of (a few) indices, e.g., data = [1.0, 10.0, 100.0] contribs = [[1, 2], [0], [0, 1]] For each entry in contribs, I'd like to sum up the corresponding values of data and put those into an array. For the above example, the expected result...

 
6:45 PM
0
Q: Column to Transacction ID for association rules on dataframes from Pandas Python.

Antonio López RuizI imported a CSV into Python with Pandas and I would like to be able to use one as the columns as a transaction ID in order for me to make association rules. (link: https://github.com/antonio1695/Python/blob/master/nearBPO/facturas.csv) I hope someone can help me to: Use UUID as a transaction...

0
Q: drop_duplicates() got multiple values for argument 'keep'

FunnyChefI'm getting and error that I haven't been able to resolve that I'm hoping someone can assist with. The error is; "drop_duplicates() got multiple values for argument 'keep'" When I execute the following: hardware_df.Tags.str.contains(r'hardware').drop_duplicates('Id',keep='last').count() hard...

 
7:02 PM
0
Q: From tuples to linear equations with numpy

Alexi need help in the following topic. Lets say i have three points, each with x, y coordinates and a corresponding z value, e.g.: p_0 = (x_0, y_0, z_0) : coordinates of first point p_1 = (x_1, y_1, z_1) : coordinates of second point p_2 = (x_2, y_2, z_2) : coordinates of third point Later on, i ...

0
Q: How to get values that meet a condition

Yared J.I have a really long data set of transcription factors (TF) and a long list of essential genes, and what I've done so far is determine if a gene that is regulated by a TF is an essential gene or not. Like this: In: df2 = regulon.copy() lista_genes = (genes['Esenciales'].values.tolist()) df2['Es...

 
7:29 PM
1
Q: Arranging numpy arrays in a block matrix

lballesI have 3 numpy arrays A, B and C. For simplicity, let's assume that they are all of shape [n, n]. I want to arrange them as a block matrix A B B^t C where B^t shall denote the transpose of B. Of course, I could do this via a series of concatenations top_row = np.concatenate([A, B], axis=1...

 
7:41 PM
0
Q: NumPy: Sort and sum data into target indices

Nico SchlömerAnother data summation problem. I have a (large) data array and and array with indices of the same length. data = [1.0, 10.0, 100.0] indices = [0, 2, 0] The goal is to sum the entries of data into a new array such that the entry at index k contributes to out[k]. For the above example, the re...

0
Q: on click pause the real time plot of data

nezzI simply want to put a pause clause to my plot where I can pause the real time stream voluntarily. This is what I am trying: def onClick(event): global pause pause ^= True fig = plt.figure() fig.canvas.mpl_connect('button_press_event', onClick) plt.show() for t in range(n): pause=Fa...

 
8:05 PM
0
Q: Combining two series into a dataframe row wise

Calvin KuI'm trying to merge two series into one single row in a new dataframe and then add more rows into it. For example, I have series as below: s1 var1 10 var2 9 var3 2 s2 var4 1 var5 81 var6 13 I want to make them into a dataframe like this: df var...

0
Q: Deleting DataFrame rows in Pandas based on column value - multiple values to remove

GDICommanderI have a list of values (not known in advance, in a Python list) that a column in my Panda DataFrame must not have for all rows. All recipes on the Web (like this one) show how to do it with only one value to exclude, but I have multiple values to exclude. How do I do it? Please note that I can...

 
8:53 PM
0
Q: Map a number to an id in python

tnabdbSuppose I have a numpy array like: [11, 30, 25]. These numbers represent categories of the objects corresponding to the indices. I know there are just 20 categories but for some reason they are numbered from 11 to 29. I'd like to convert them to numbers in 0:19 and back. What would by a pythonic ...

0
Q: How to do arithmetic on tidy data in pandas?

caseygrunI have a DataFrame in "tidy" format (columns are variables, rows are observations) containing time series data for several different conditions. I'd like to normalize the data to the zero-hour time point for each condition. For example, lets say I fed two different animals two different kinds of...

0
Q: How to add a ceiling to a cylinder in matplotlib

DmanI am trying to make a 'closed' cylinder in matplotlib but I am not sure how to go about doing this. So far I have a cylinder with the ends open, the code for this is as follows: #make a cylinder without the ends closed import numpy as np from matplotlib import cm from matplotlib import pyplot as...

 
9:20 PM
-1
Q: why does dropna() not work at all?

Al_IskanderI have imported a dataframe from csv (with delimiter ';') where the final several rows end up as NaN due to ';;;;;;' rows at the end of the csv. When I try to apply df.dropna() the rows with the missing values (NaN) are not being deleted as it is usually the case. Any idea what's going wron...

0
Q: Transpose list subset dimension?

xmanI have a list with 2 cells, each of them is 3 dimension array, one is 3 by 4 by 5, the other is 6 by 7 by 8. Need to transpose the subset in the list to be like: 5 by 4 by 3, and 8 by 7 by 6, I used array.transpose([2,1,0]), got the error axes don't match.

0
Q: NumPy: Dot product for many small matrices at once

Nico SchlömerI have a long array of 3-by-3 matrices, e.g., import numpy as np A = np.array([ [[1, 2, 3], [3, 4, 5], [4, 5, 1]], [[2, 0, 3], [5, 1, 5], [4, 2, 1]] ]) and for each of the small matrices, I would like to perform an outer product dot(a, a.T). The list comprehe...

0
Q: In Pandas, how to convert a string to a datetime object with milliseconds?

banananaI have a column of event's timestamp in my dataframe, looks like "2016-06-10 18:58:25:675", the last 3 digits are milliseconds, is there a efficient way to transform this column to a Pandas datatime type? When I try: pd.to_datetime('2016-06-10 18:57:35:317') An error occurs saying "Unknown s...

-1
Q: Connection Refused Error while trying to open Python Console in PyCharm

GieneI am new to PyCharm/Python. Installed successfully PyCharm (Communities edition) on MAC ensured Python is installed and available (Version 2.7.11). Issue I am facing is when I am trying to open Python Console from PyCharm, running into Connection Refused error. Tried google'ing have no luck and ...

0
Q: Can someone explain how arrays and scalars are handled in a Python code snippet

sammanI have this code snippet I am trying to understand that is in python. I don't understand how scalars operate on arrays in all cases. In most code I read it makes sense that operations work on each value of an array. sig_sq_samples = beta*invgamma.rvs(alpha,size=n_samples) var_norm = sqrt(sig_sq_...

 
10:22 PM
0
Q: how to index a numpy array using conditions?

tnabdbSuppose I have an array like this: a = np.array([[2,1], [4,2], [1,3],...] I want to retrieve the elements of the second column where the corresponding elements in the first column match some condition. So something like a[a[:,0] == np.array([2,4]),1] (?) should g...

 
10:40 PM
3
Q: SQLAlchemy automap data and override some columns

MarkIs it possible in SQLAlchemy to automap existing database tables into classes but override some some fields of some tables? I'm pickling the MetaData object, since it takes some time, it contains all the tables, but then when I'm trying to override some objects, it raises an exception that MetaD...

 
10:58 PM
0
Q: How can I sum up specific rows in Pandas based on conditions?

HarrisI am trying to think of a way to manipulate a Pandas table as follows: Table <table style="width:100%"> <tr> <td>Date</td> <td>Quantity</td> <td>Unit Cost/Unit Retail Price</td> </tr> <tr> <td>01-01-2016</td> <td>100</td> <td>1.00</td> </tr> <tr>...

0
Q: Is it possible to reset a Cycler object

gboffiE.g., if shape of lines1 and lines2 were (3,101) with plt.plot(x, lines1, linewidth=3) plt.plot(x, lines2, linewidth=1) I have a blue, green, red line with width equal to 3, and a cyan, magenta and yellow line with width equal to 1. Because the data in lines1 is connected pairwise to data i...

 
11:29 PM
0
Q: Python loop slow in some cases

HighwayJohnI am having an issue with a loop in my following code u = numpy.zeros((l, l)) wp = numpy.zeros((l,2)) # some code which edits u and wp m = u * wp[:, 1] while wv > acc: wavg = numpy.dot(wp[:, 0], wp[:, 1]) wp[:, 0] = 1.0 / wavg * numpy.dot(m, wp[:, 0]) wp[:, 0] = numpy.tensordot(numpy...

0
Q: AttributeError: 'numpy.ndarray' object has no attribute 'plot'

René GWhy is this matplotlib code giving me a weird exception? I'm going for two rows of plots. The top row is supposed to show true vs. pred and the bottom row is supposed to show percent error. yy = func(*X) fig, axes = plt.subplots(1, len(X)) for ax,_x in zip(axes,X): ax.plot(_x, y, 'b.') ...

 

« first day (858 days earlier)      last day (1330 days later) »