« first day (846 days earlier)      last day (1342 days later) » 

12:33 AM
1
Q: Why am I returned an object when using std() in Pandas?

William The print for average of the spreads come out grouped and calculated right. Why do I get this returned as the result for the std_deviation column instead of the standard deviation of the spread grouped by ticker?: pandas.core.groupby.SeriesGroupBy object at 0x000000000484A588 df = pd.read_csv('C...

0
Q: Replace values in column of Pandas DataFrame using a Series lookup table

BillI want to replace a column of values in a DataFrame with a more accurate/complete set of values generated by a look-up table in the form of a Series that I have prepared. I thought I could do it this way but the result is not as expected. Here is the DataFrame I want to fix: In [6]: df_normali...

 
12:56 AM
0
Q: How can I do computations on dataframes or series that have different indexes in PANDAS?

Jeffrey DeroseI have two Series that are of the same length and datatype. Both are float64. The only difference are the indexes both are dates but one date is at the beginnning of the month and the other is at the end of the month. How can I do computations like correlation or covariance on Series or datafra...

 
1:28 AM
0
Q: Plot list of lists in histogram - x and y values

tuco94I have a list of lists: [[Decimal('3.69578'), Decimal('5.12519')], [Decimal('3.69586'),Decimal('2.50457468')], [Decimal('3.69587'), Decimal('72.33894358')], [Decimal('3.69629'), Decimal('54.2359599')], ...] and I would like to plot it in a histogram with matplotlib using one of the values in t...

0
Q: Easiest way to select the subset of 2-d array by 2 corners

Han ZhengzuWith an global distributed variable in the shape of (len(latitude), len(longitude)), I want to get an subset corresponding to the area of interest. The specific area is defined by 2 corners(left-low latitude/longitude, right-top latitude/longitude). So, this array is what I have now: VA...

 
2:14 AM
1
Q: Youtube playlist saver with Google API

MajiickI have a problem with Youtube videos being deleted from my big playlist and not knowing which videos got removed. So, I decided to write a Python script to save certain information(title etc.) about the videos in a playlist into a text-file. I particularly don't like the setChannelNames() func, b...

 
2:59 AM
0
Q: Pandas: to_excel(): How can I express Time as a format to TimeSeries index?

HaggarTheHorribleI have a DataFrame with an index as: Index([00:00:00, 00:05:00, 00:10:00, 00:15:00, 00:20:00, 00:25:00, 00:30:00, 00:35:00, 00:40:00, 00:45:00, ... 23:10:00, 23:15:00, 23:20:00, 23:25:00, 23:30:00, 23:35:00, 23:40:00, 23:45:00, 23:50:00, 23:55:00], dtype='object', length=288) I w...

 
3:24 AM
0
Q: Reverse column values in pandas.DataFrame

wl2776I've got a pandas DataFrame with boolean column, sorted by another column and need to calculate reverse cumulative sum, that is, amount of true values from current row to bottom. Example In [13]: df = pd.DataFrame({'A': [True] * 3 + [False] * 5, 'B': np.random.rand(8) }) In [15]: df = df.sort_...

0
Q: pandas plot timeseries for overlapping hours

user3299288I am trying to do a visual analysis of load variation. I've read in the data from CSV files, converted the index to datetime format, which works great. The data is per 5 minutes for a whole year, with the daytime (currently selected as 07:00-18:00) hours of interest. I can plot the timeseries da...

0
Q: Apache Spark Dataframe from Pandas Dataframe Error

rahnamI am trying to create spark dataframe from a pandas data frame.But i am getting an error as shown.Any help? pandadf = pd.read_csv('http://s3.amazonaws.com/assets.datacamp.com/course/Kaggle/train.csv') sparkdf = sqlContext.createDataFrame(pandadf) error sceenshot

 
3:46 AM
0
Q: Numpy.matrix doesn't own its data. Can't figure out why

user2465510Most of what I could tell you is in the comment at the top, but basically I'm trying to dynamically create a matrix representation of derivation as a linear transformation (basically every nxn space will have a 1 and the rest will have zeros). I can't figure out why DMatrix doesn't own its own da...

 
4:31 AM
0
Q: How do I create a new dataframe using grouped output from another dataframe?

William I have tick data containing symbols, bid prices, and ask prices. I was able to find the average spread and standard deviation of each symbol. I'd like to create a confidence interval for each symbol and have the final DataFrame output have the columns ticker symbol average spread lower bou...

 
 
1 hour later…
5:36 AM
0
Q: Varying search results in different environments

Kartik SibalI am learning Pandas and while doing a question, I came across a weird observation. The question was to print the number of occurrences of the string 'Soup' on the Beautiful Soup home page, using python and pandas. The weird thing being that, the number of occurrences varies in Pandas and in Pyth...

 
6:34 AM
0
Q: how to plot a plan using a normal vector and having width and length of it win python?

Saeed Soltani MoghadamI'm wondering how to plot a plan (3D) using its normal vector and having its length and width. Lets assumed i want to plot the following: n = [1,2,3] # normal vector plane_width = 2 plane_length = 2 i' have tried the following but the result is mot cocsistent: import numpy as np import matpl...

 
6:54 AM
0
Q: Copy a range of text from csv column in pandas and python

Mukesh Kumar MishraI have a csv file that I have imported into Pandas. Now it has almost 45 columns of data and each column has more than 100 lines of information. Now I need to select only the range of text that starts with a Date Stamp at the start and ends with a Date Stamp. Ex : datestamp Hi, -----...

 
7:50 AM
0
Q: Incorrect blob contour detection python opencv

Shalin ShahI would like to detect the contour of the giant black blob in the following image: So I used Canny edge detection to find the edges using the following code: edged = cv2.Canny(image, 30, 200) Then, when I try to find the contour, it gives me the little white hole inside the black blob. Th...

 
8:15 AM
0
Q: How to add element to matrix?

Eli BorodachI saw many posts on the same issue but using numpy.array, whereas here I am talking about numpy.matrix. How can I add an element to a matrix? For example: my_matrix = [[1 2 3 4 5]] How can I make it: my_matrix = [[1 2 3 4 5 6]]?

0
Q: Matplotlib interfering with namespace

Felixfrom matplotlib.figure import Figure import numpy as np def f(ax, x, y): x /= 10 ax.plot(x, y) datx = np.linspace(0, 1, 10) daty = np.sin(datx) fig = Figure() ax = fig.add_subplot(111) print datx f(ax, datx, daty) print datx Hello, Normally I would expect datx to stay untouched when...

 
8:32 AM
0
Q: Using pandas isin with selecting dates

pymatA similar question like mine has been asked before, but unfortunately the answer was unable to help my problem. I have two sets of data columns: 1. the first is generated as follows: t = pd.bdate_range(start='2016-05-16', end='2016-06-15') This gives: DatetimeIndex: DatetimeIndex(['2016-05-1...

 
8:48 AM
0
Q: 3D Plotting of multiple data matrices in a single graph in Python

yoyoI have two different types of data which i want to plot in a single 3D graph in Python. I have tried some methods from matplotlib but I it isn't helping. Note: I am able to get the graphs separately but failing in combining them. Please can anyone suggest me a function to do it. I tried using ax....

 
9:20 AM
0
Q: HOW TO LABEL the FEATURE IMPORTANCE with forests of trees?

ElenaPhysI used the sklearn code to plot the feature importance for a forests of trees. My dataframe is named 'heart'. Below you can find the code with which I exctracted the list of the sorted features: importances = extc.feature_importances_ indices = np.argsort(importances)[::-1] print("Feature rankin...

0
Q: Drop rows while iterring through groups in pandas groupby

Jonathanfor a dataset like: te = {'A':[1,1,1,2,2,2],'B':[0,3,6,0,5,7]} df = DataFrame(te, index = range(6)) vol = 0 I'd like to groupby A and iter through the groups after the groupby function: for name, group in df.groupby('A'): for i, row in group.iterrows(): if row['B'] <= 0: ...

0
Q: How to turn a pandas dataframe row into a comma separated string

Blue MoonI need to iterate over each row of a pandas df and turn this into a comma separated string. example: df3 = DataFrame(np.random.randn(10, 5), columns=['a', 'b', 'c', 'd', 'e']) a b c d e 0 -0.158897 -0.749799 0.268921 0.070035 0.099600...

0
Q: How to calculate average value from gridded datasets in python

bikuserHello I have a gridded datasets in .nc format. I tried to read .nc file for different combination of latitude and longitude with the code given below, but i know this is not the ideal way to do that. import netCDF4 from netCDF4 import Dataset f= Dataset('data.nc') data1 = f.variables['asa'][:,...

 
9:44 AM
0
Q: How to get Time offset information off datetime64[ns]

rojasI am converting a datetime[ns] column to a different timezone and the resulting column looks like this: 0 2011-05-25 20:30:00+02:00 1 2011-05-24 20:30:00+02:00 2 2011-05-20 20:30:00+02:00 3 2011-05-19 20:30:00+02:00 4 2011-05-15 13:30:00+02:00 I am trying to pull out the offset infor...

0
Q: Python Pandas: Set frequency as month-end today

RuaridhI'm looking for an equivalent specification to W-MON (weekly, ending monday) for monthly data. Specifically, I have a pandas data frame of daily data, and I want to only take monthly observations, starting with the most recent date and going back monthly. So if today is 17/06/2016, my date ind...

0
Q: how to parse an np.argwhere array in python numpy

jagI am a newbie, I was working with raspberry pi camera to capture a frame from loop and evaluate to see if have the pixel color I defined in a range. I got until line dumping the coordinates of the acceptable range but I still want to parse further to get the x,y coordinates and get the position i...

1
Q: Why is pandas.DataFrame().groupby().apply(func) called more often than there are groups?

GerenukI've written a test script to understand pandas.DataFrame().groupby().apply(). df=pd.DataFrame([("a",1),("a",2),("b",1)], columns=["idx", "col1"]) def my_apply(group): print(group) return group df.groupby("idx").apply(my_apply) As a result the dataframe for the group "a" is printed t...

0
Q: Year 0: Instruction Follower

Joe WallisHaving played Human Resource Machine for over a day, and wanting to improve my interpreting / tokenizer skills. I made a Human Resource Machine interpreter. The pipe line is quite hard to explain with the types that I add. But it follows the following design: HRM class: HRM functions wrapped ...

 
10:14 AM
0
Q: matplotlib animation ArtistAnimation failure due to ffmpeg unavailable

OhmI am trying to call matplotlib.animation.ArtistAnimation to make an animation from a series of png images, but I get an error that is connected to the ffmpeg. I have Anaconda installed on my Ubuntu 14.04. The error message that I get is: UserWarning: Matplotlib is building the font cache using f...

0
Q: python matplotlib multiple lines animation

Marina VabistsevitsI'm trying to create an animation with matplotlib to plot several datasets simultaneously in one animation. The problem is that two of my datasets have 50 points in them, and the third one has got 70000 points. Therefore, simultaneous plotting (with the same interval between the points) is usele...

0
Q: Indexing one array by another in numpy

vooSuppose I have a matrix A with some arbitrary values: array([[ 2, 4, 5, 3], [ 1, 6, 8, 9], [ 8, 7, 0, 2]]) And a matrix B which contains indices of elements in A: array([[0, 0, 1, 2], [0, 3, 2, 1], [3, 2, 1, 0]]) How do I select values from A pointed by B, i.e.: ...

1
Q: Groupby df column using pandas

ldevyataykinaI have data 1 member_id application_name active_seconds 2 192180 Opera 6 3 192180 Opera 7 4 192180 Chrome 243 5 5433112 Chrome 52 6 5433112 Opera 34 7 5433112 ...

0
Q: How does numpy polyfit work?

C HeyerI've created the "Preciptitation Analysis" example Jupyter Notebook in the Bluemix Spark service. Notebook Link: https://console.ng.bluemix.net/data/notebooks/3ffc43e2-d639-4895-91a7-8f1599369a86/view?access_token=effff68dbeb5f9fc0d2df20cb51bffa266748f2d177b730d5d096cb54b35e5f0 So in In[34] and...

0
Q: Unable to import even after isntalling the packages

user123I have installed numpy, scipy, scikit learn and other required packages. Still when I try to import them in python, it shows no module found. pip install -U numpy scipy scikit-learn Requirement already up-to-date: numpy in /usr/lib/python2.7/dist-packages Collecting scipy Downloading scipy-0.1...

0
Q: How to change the value of an entire array based on one value in Python

LouiseLet's say I have this array: import numpy as np np.array([[0,2,7],[-3,4,0],[12,10,12]]) I'm trying to change the values of one column if one or more values in the column are less than zero. The code should be general and not just for this array How do I set all of the values in the middle vect...

0
Q: Find values in a pandas dataframe containing a substring

AlexI have a pandas dataframe with string values and I would like to be able to return a subset of the dataframe where the values contain some substring. This is easy to do on a series in this way (example adapted from pandas documentation): import pandas as pd import numpy as np s4 = pd.Series(['A'...

0
Q: Python 2.7 - Help on pandas_reader

Sean FallonAm replicating the code I have got from another website where in "pandas.io.data" has been commented out. Instead am using "pandas_datareader" # Commodity Channel Index Python Code # Load the necessary packages and modules import pandas as pd #import pandas.io.data as web import pandas_dataread...

 
10:43 AM
0
Q: Creating a 3D grid using X,Y,Z coordinates at cell centers

HeliI have a question, I have been given x,y,z coordinate values at CELL CENTERs of a grid. I would like to create structured grid using these cell center coordinates. Any ideas how to do this? Thanks A lot in Advance

0
Q: to_csv keeps on looking for a file and not creating one

entercaspaif os.path.isfile('%s.csv' %code): print('located %s, next step, update.. \n') update_stocks(code=code) else: print('data for %s does not exist, downloading data...\n' %code) print(code) df = quandl.get('%s' %code, authtoken=auth...

 
10:57 AM
0
Q: How can I change datetime format of x-axis values in matplotlib using python?

Dot_PyHow can I make this plot's x-axis looks better? I tried using plt.xticks(rotation=70) but it still cuts my numbers.. As you can see, it shows the date in the format MMM DD YYYY. I wanna something like MMM DD or DD MMM. Or even writing in 2 lines DD MM \n YYYY. How can I achieve tha...

 
11:07 AM
0
Q: 3D matplotlib rotation matrix

pepijntje02Is it possible to use the rotation matrix for a 3D object? I have created an ellipsoid with three different axes, but now it is aligned with it coordinate system, but that is not what I want. I want to use roll, pitch and yaw; y-axis, x-axis and a z-axis rotation, to rotate the ellipsoid. I am ...

0
Q: Concatenation of pandas DataFrames including rows' source-dataframe/-category

Pierre SchroederI need to pandas.concatenate(...,axis=0,...) multiple DataFrames so that the resulting DataFrame has a new column holding information about which dataset a row belongs to, but drop the implicit indices of the original DataFrames. In this MWE for example, we have from multiple countries (A and B,...

1
Q: Python property AND getter in a single class - bad practice?

RemcoWI've got a base class that a lot of other classes inherit from. This class has the following property: @property def element(self): wait = WebDriverWait(self.driver, self.timeout) return wait.until(ec.presence_of_element_located(self.locator)) Now I've got the problem that ...

 
11:31 AM
0
Q: 2D discrete colour plot using discrete variables

J Taylor-ImsI am looking for a way to graph some data in a human readable format. I have data from simulations that contains a frame number, protein residue number and protein secondary structure state. Frame number as integers and the secondary structure state as strings. For example: 0     1     2     3     

0
Q: How to Preserve the index from df.pivot data back to df.stack() or pandas.melt in with Pandas?

toasteezI have a dataframe (fx_return): index_date ccy rate ==================================== 2016-06-01 CHF 1.013017272 2016-06-01 EUR 1.1178 2016-06-01 GBP 1.44255 2016-06-01 USD 1 2016-06-02 CHF 1.010560356 2016-06-02 EUR 1.11615 2016...

1
Q: Multiplication of two positive numbers gives a negative output in Python 3

Michael_OI have a DataFrame df1 : df1.head() = wght num_links id_y id_x 3 133 0.000203 2 186 0.000203 2 5 6 0.000203 2 98 0.000203 2 184 0.000203 2 ...

 
12:23 PM
0
Q: Reshaping Numpy Arrays to a multidimensional array

SruliFor a numpy array I have found that x = numpy.array([]).reshape(0,4) is fine and allows me to append (0,4) arrays to x without the array losing its structure (ie it dosnt just become a list of numbers). However, when I try x = numpy.array([]).reshape(2,3) it throws an error. Why is this? An...

0
Q: How do I convert a numpy ND-array to a CFFI C++ array and back again?

user1021819I want to pass a numpy array into some(one else's) C++ code via CFFI. Assume I cannot (in any sense) change the C++ code, whose interface is: double CompactPD_LH(int Nbins, double * DataArray, void * ParamsArray ) { ... } I pass Nbins as a python integer, ParamsArray as a dict -> structure...

 
1:08 PM
0
Q: Color-coded histogram

Py-serI want to plot a color-coded histogram, where I input an array of arrays to represent the elements on the y-axis, while a simple 1D array on the x-axis to represent a phase. The array of arrays to plot on the y-axis has a dimension, let's say, (100, 25), while the phase on the x-axis has 25 ele...

0
Q: Mandrill and pandas libray in python are compatible?

Maryam PashmiI would like to send the report via Mandril: def make_excel_report(res): df = pd.DataFrame(res) df = df.fillna('-') df = df.T df.to_excel('suspected_user.xlsx', sheet_name='Sheet1', engine='xlsxwriter', columns=['id','100_credits','300_credits' ,...

0
Q: openpyxl: Formulas read as blanks in some (key use) cases

Kathryn CogertMy code downloads a .xlsx file from google drive (using pydrive), finds some blank cells with pandas, and fills in those blank cells with openpyxl. When I open the openpyxl altered file, everything looks great. However, when I use the pandas read_excel function, all cells that have equations ar...

-1
Q: How to save df to different file with condition

ldevyataykinaI have data member_id event_path domain subdomain search_engine search_term event_time 123 https://yandex.ru/search/?text=%D0%9F%D1%80%D0%BE%D0%BC%D1%8B%D1%88%D0%BB%D0%B5%D0%BD%D0%BD%D1%8B%D0%B9%20%D0%B6%D1%83%D1%80%D0%BD%D0%B0%D0%BB&clid=1955453 yandex.ru yandex.ru None None...

 
1:33 PM
0
Q: scrapy - how to populate item with data from pandas dataframe?

kanimblaSuppose the following CrawlSpider: import scrapy from scrapy.loader import ItemLoader from scrapy.spiders import CrawlSpider, Rule from scrapy.linkextractors import LinkExtractor from tutorial.items import TestItem from scrapy.http import HtmlResponse class TestCrawlSpider(CrawlSpider): na...

0
Q: Pandas: rolling_apply bottleneck

Steve LorimerI am calculating a dataframe of profit/loss amounts for every row in a dataframe of prices. The requirements for this piece of logic are as follows: We buy/sell the asset at the current time period. We hold the asset for holding_period. If, during the holding period, the price exceeds take_pro...

0
Q: is this correct use of argparse in python?

Sens4I want to get some Website information by using a python script were python does the logic and C# is used for GUI. I am calling my python script within c# + process parameters (username password, cookies). Now at some point, I just need the login credentials and not any information from any webs...

 
1:57 PM
0
Q: Filter a pandas dataframe based on a derived field

user3447653I have the below query that I execute and store in a pandas dataframe. SELECT mn as MachineName, dt as DateTime, REGEXP_EXTRACT(path, 'Name:([\\s\\S\\w\\W]*?)Domain:') AS Name1, FROM TABLE DataGallery Output of the query is: MachineName DateTime Name1 GOG1 2016-12-13 ...

0
Q: How to remove labels from one of two y-axes on a subplot

evtohI'm making a plot with two subplots, one of which has two sets of data plotted on different y-axes. This is the code I have rn for the subplot in question: ax2 = plt.subplot(2, 1, 2) plt.plot(time, temp, color = 'r', label = "Temperature") plt.yscale('linear') plt.ylim(0, 10) plt.ylabel("Tempera...

0
Q: numpy.insert() invalid slice -- Trying to Insert NaN in Numpy Array

T. AtoI know there are already lots of questions about this, but none of the answers I've seen have solved my problem. I have a pandas DataFrame with 10 columns for data, but on some rows I have just 9 columns-worth of data. For the rows with just 9 datapoints, I need the data to be in the last nine co...

0
Q: Python Numpy: array assignment using slicing

Moon ZoeWhen I was using array assignment using slicing, there is some thing strange happened. The source code is below: import numpy as np a = np.array([1,2,3,4]).reshape(2,2) b = np.array([5,6,7,8]).reshape(2,2) print(id(a)) print(id(b)) b = a[:] b[1,1] = 10 print(b is a) print(id(a)) print(id(b)) pri...

1
Q: Accessing a pandas.DataFrame column name with a '.' in it

Patrickc01I have a pandas dataframe 'df'. One of the columns is 'Project.Fwd_Primer' I would like to access that column, however when I use df.Project.Fwd_Primer I get an AttributeError. Is there another way I can access this column, or do I need to get rid of the period in it?

 
2:32 PM
0
Q: Tensorflow Dimensions are not compatible in CNN

Andrea PavoneThis is main.py: # pylint: disable=missing-docstring from __future__ import absolute_import from __future__ import division from __future__ import print_function import time from six.moves import xrange # pylint: disable=redefined-builtin import tensorflow as tf from pylab import * import cn...

0
Q: matplotlib 2D slice of 3D data

bob.sacamentoI haven't been able to find anything on this, maybe because I don't have the right nomenclature (i.e. I don't know exactly how to ask for it), but anyway, I have a 3D numpy array "a". I would like to identify and plot the 2D surface where a=0. Does matplotlib have a ready-made routine for doing...

 
2:49 PM
0
Q: How to append reshaped / flattened images to a python list without consuming too much memory?

W. HawkI'm trying to read around 2,000 grayscale images (640 x 480) from a directory using opencv, scale down the intensity values between 0 and 1, flatten the images as one-dimensional numpy arrays and append them to a list for feeding them to a Neural Network later. This is my code : import os import...

0
Q: Changing the value in a dataframe column depending on the value of two columns in a diffrent dataframe

GreenGodotI have two dataframes made with Pandas in python: df1 id business state inBusiness 1 painter AL no 2 insurance AL no 3 lawyer OH no 4 dentist NY yes ........... df2 id business state 1 painter NY 2 painter...

 
3:16 PM
0
Q: How do I make Tkinter faster

Sruthi KuradaI am trying to make a scrolling game in Tkinter. However, with all my for loops and functions it is extremely slow. I am reading the tutorial from Khan Academy's advanced JS. In their JS game everything is fine, so is it just a problem of python is slow or am I doing things inefficiently. import...

1
Q: DataFrame array modification (loc) on boolean array

pymatI have a series (business_day) which was generated via isin in order to create the series with dtype: bool: Series: 0 True 1 True 2 True ... 563 False 566 False Name: Date, dtpye: bool I then tried to update another array by using: myArray.ix[business_day] and myArray.loc[business_day] ...

1
Q: Sum and count data from columns using pandas

ldevyataykinaI have data like 1 member_id application_name active_seconds 2 192180 Opera 6 3 192180 Opera 7 4 192180 Chrome 243 5 5433112 Chrome 52 6 5433112 Opera 34 7 54...

 
3:34 PM
0
Q: How would you simplify this program? Python

Ekaterina1234I wrote this program, which purpose is to visit the 18th link on the list of links and then on the new page visit the 18th link again. This program works as intended, but it's a little repetitive and inelegant. I was wondering if you have any ideas on how to make it simpler, without using any f...

0
Q: Trying out perceptron learning in python, keep getting "no labeled objects" error

Tylr DurdenI'm trying to use the perceptron model in python on the classic iris example. For some reason at the end of this early process im getting the no labelled objects error. Could you please take a look and see where the error might be? import matplotlib.pyplot as pltimport numpy as np class Perceptr...

0
Q: Are Python's map and filter really so ridiculously fast or are my measurements wrong?

MichaelI do some speed tests for operations on vectors/lists. Suprisingly, map and filter seem to beat numpy by a factor of 5-10x. See the following short code sample with times given (Full code below.): n = 10000000 a = np.random.rand(n) b = np.random.rand(n) c = a + b # time = 0.07 s d = a[a < 0.3] #...

 
3:59 PM
0
Q: pandas dataframe.resample() Silently Failing

Colorful EdI have multiple dataframes that represent years worth of data that are sampling on average every 30 minutes. I am trying to graph the years worth of data and due to fluctuations with the observations/equipment, there are some anomalies that appear, making my graph appearing to be an area graph ra...

0
Q: How do I optimize a pandas apply lambda that looks at all records?

pandasEverywhereI have a dataframe that looks like this: ID YEAR AMOUNT 1 | 2001 | 4340 1 | 2002 | 5460 1 | 2004 | 1245 1 | 2006 | 6000 2 | 2003 | 5000 2 | 2006 | 3059 .... I would like to add a column that computes the highest amount thus far, as in: ID YEAR AMOUNT AMT_MAX 1 | 2001 | 4340 |...

 
4:25 PM
-1
Q: How to get brighness percentage level from a histogram?

RichI wrote a python program that used scikit-image to return the histogram of an image: def get_hist(image): image = img_as_float(image) hs, dis = exposure.histogram(image) cdf = np.cumsum(hs) #cumulative distribution function cdf = 255 * cdf / cdf[-1] #normalize return ...

0
Q: Exiting program for any input that is not a list

B.ghoMy goal is to create a little program that converts angle from radiant to degree and vice-versa. I need the program to close with no error message from python if the user enters the information to convert in the wrong format. After assigning the variable ā€˜angleā€™ to both values of the input. The ...

-1
Q: numpy where operation on 2D array

ZanamI have a numpy array 'A' of size 571x24 and I am trying to find the index of zeros in it so I do: >>>A.shape (571L, 24L) import numpy as np z1 = np.where(A==0) z1 is a tuple with following size: >>> len(z1) 2 >>> len(z1[0]) 29 >>> len(z1[1]) 29 I was hoping to create a z1 of same size as A...

0
Q: Member functions for numpy records

al.zatvI have to use numpy arrays of records to save RAM and to have fast access. But I want to use member functions on that records. For example, X=ones(3, dtype=dtype([('foo', int), ('bar', float)])) X[1].incrementFooBar() For ordinary python class,I can make class QQQ: ... def incrementFooBar(s...

 
5:02 PM
0
Q: Visualize 5-D feature set and regression results

wolframalphaI am using sklearn.svm Support Vector Regression to solve a regression problem on a continous dataset where feature set has 5 dimensions and 1 dimension label set, below is the how the dataset looks like: >>> type(feature_set) 55: <type 'numpy.ndarray'> >>> type(label_set) 56: <type 'numpy.ndarray'

0
Q: What are the drawbacks to storing a single value as a property of a DataFrame instead of as a column?

Michael AI came across some code that calculates a single value, based on a DataFrame, and stores it as a property of the DataFrame instead of a separate variable or a new column. Here's an example. The portfolio has two securities, each with a price and quantity. import pandas data = {"Prices" : [83.2...

0
Q: Spark + Zeppelin/Toree + Scala + Matplotlib

M.RezI am enjoying the Zeppelin's nice presentation of data in %sql mode, but I would prefer much if I could use matplotlib style of plotting in zeppelin + spark with scala language. I know how to use matplotlib (or any other nice presentation libraries) using python and zeppelin but I am not famili...

0
Q: using OR in arange function in numpy

Nodar95I have the following code, trying to get 2 and 3 out. arr = np.arange(-10,10,0.5) arr = arr[arr == [2|3]] print(arr) But I only get 3 out. What did I get wrong? Thanks.

 
5:29 PM
1
Q: Python curve fitting on a barplot

Euler_SalterHow do I fit a curve on a barplot? I have an equation, the diffusion equation, which has some unknown parameters, these parameters make the curve larger, taller, etc. On the other hand I have a barplot coming from a simulation. I would like to fit the curve on the barplot, and find the best para...

http://www.commitstrip.com/en/2016/06/07/good-code/
CommitStrip
Good code
CommitStrip
1465320845
http://www.commitstrip.com/en/2016/06/09/coder-1-corrupt-file-0/
CommitStrip
Coder: 1. Corrupt file: 0.
CommitStrip
1465468090
http://www.commitstrip.com/en/2016/06/13/the-end-of-an-expensive-era/
CommitStrip
The end of an expensive era
CommitStrip
1465841605
http://www.commitstrip.com/en/2016/06/16/incognito-mode-freedom-at-last/
CommitStrip
Incognito mode : freedom, at last
CommitStrip
1466069934
 
Sam
4 messages moved from The Ministry of Silly Hats
 
Sam
3 messages moved from The Ministry of Silly Hats
 
0
Q: pandas series add fill_value from left side only

piRSquaredWhen applying an arithmetic operator via methods on a a pandas series or dataframe, you can pass an argument fill_value to specify how to handle missing values. http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.add.html For example: import pandas as pd s1 = pd.Series([1, np.n...

 
6:00 PM
0
Q: How do you add pandas series filling each series' missing values separately

piRSquaredIf I have two series: import pandas as pd import numpy as np s1 = pd.Series([1, np.nan, 2, np.nan]) s2 = pd.Series([1, 2, np.nan, np.nan]) I want to fill s1's missing values with 0 and s2's missing values with 1. But I want to preserve missing if both are missing. I could: s1.fillna(0).add...

 
6:12 PM
0
Q: pandas DataFrame add fill_value NotImplementedError

piRSquaredI have: import pandas as pd import numpy as np np.random.seed([3,1415]) df = pd.DataFrame(np.random.choice((1, 2, np.nan), (5, 5))) s = pd.Series(range(5)) I want to add s to df and broadcast across rows. Usually I'd: df.add(s) And be done with it. However, I want to fill missing values ...

 
6:24 PM
0
Q: How do I organize my project to accomodate GIT pull requests for source installed via PIP?

Brian BienI want to structure my Python project in a manner such that others can use a package manager (PIP) to easily clone the environment, and GIT to contribute to a library dependency through pull requests. My confusion relates to organizing a project around PIP, GIT (I'm familiar with SVN). The proje...

0
Q: Compiling Python code into an exe file calling just the required DLLs

MattThere seems to be little information out there on this subject - I have tried py2exe to no avail, as well as pyinstaller. The latter seems to include EVERY extension module installed on the system unless they are specifically excluded during build time, which ends up being 50+ on even a small bu...

0
Q: pandas series bfill first half, ffill second half

piRSquaredSuppose I had: import pandas as pd import numpy as np np.random.seed([3,1415]) s = pd.Series(np.random.choice((0, 1, 2, 3, 4, np.nan), (50,), p=(.1, .1, .1, .1, .1, .5))) I want to back fill in missing values for the first half of the series and forward fill for...

 
6:41 PM
0
Q: Error installing matplotlib via pip in OSX El Capitan

rawbeansI am trying to install matplotlib via pip on OSX (El Capitan). I have installed Freetype via homebrew, and linked it using: ln -s /usr/local/opt/freetype/include/freetype2 /usr/local/include/freetype when running pip install matplotlib inside my virtualenv im am getting the error: error: comm...

0
Q: displaying x-axis labels properly in matplotlib plots

et3I'm trying to plot electricity usage against time. I'm using this script: from datetime import datetime import matplotlib.pyplot as plt import numpy as np timelist = [] valuelist = [] # Logic that populates timelist and valuelist timeaxis = np.array(timelist) valueaxis = np.array(valuelist) ...

 
7:10 PM
0
Q: Pandas: Filter out all rows preceding a string match

WinterflagsI'm attempting to get Pandas DataFrame to discard all rows that precedes the place where there is a string match in one of the columns. In other words: All rows after, and including the row with the string match, should be kept. import pandas as pd df = pd.read_csv(file_path) test_string = "...

 
7:22 PM
0
Q: How to split a dataframe into multiple parts with copying comments in python pandas?

Bhishan PoudelI have a datafile like this: # coating file for detector A/R # column 1 is the angle of incidence (degrees) # column 2 is the wavelength (microns) # column 3 is the transmission probability # column 4 is the reflection probability 14.2000 0.531000 0.0618000 0.938200 14.200...

 
7:53 PM
0
Q: Appending to a Pandas Dataframe From a pd.read_sql Output

ClintWeathersI'm coming from R but need to do this in Python for various reasons. This very well could be a basic PEBKAC issue with my Python more than anything with Pandas, PyODBC or anything else. Please bear with me. My current Python 3 code: import pandas as pd import pyodbc cnxn = pyodbc.connect(DSN...

 
8:28 PM
0
Q: Find rank and percentage rank in list

ajspI have some very large lists that I am working with (>1M rows), and I am trying to find a fast (the fastest?) way of, given a float, ranking that float compared to the list of floats, and finding it's percentage rank compared to the range of the list. Here is my attempt, but it's extremely slow: ...

-1
Q: Pandas Apply Syntax

HoosierCoderI can't figure out how to apply a simple function to every row of a column in a Panda data frame. Example: def delLastThree(x): x = x.strip() x = x[:-3] return x arr = ['test123','test234','test453'] arrDF = pandas.DataFrame(arr) arrDF.columns = ['colOne'] arrDF['colOne'].apply(del...

 
8:40 PM
0
Q: pandas date assignment destroy other columns

user2658138In Python 2.7.11 + Pandas 0.14.1, enlargement assignment with a column of datetime64[ns] seems to corrupt existing columns. Anyone can kindly explain what happened here ? Thank you. import pandas as pd import numpy as np ss=pd.Series(data=['1/1/2000','1/2/2000','1/3/2000','1/4/2000','1/5/2000',...

 
9:03 PM
0
Q: Computing Cosine Similarity of a Matrix in Python

BaktaawarI have a matrix of size 56K X 8.5K. It is binary preference matrix. Now I want to get the cosine similarity of each columns and some more operations as follows: 1). Before taking cosine similarity, normalize the rows of the matrix to unity (L2 norm). 2). Take cosine similarity 3). Make th...

0
Q: Coloured Plot Including Percentages

seemoI have a data frame containing a range of positive and negative values: LineLength = pd.read_pickle('Line_length') type(LineLength) pandas.core.frame.DataFrame I have split the positive and negative values to different columns and need to show this data in a histogram: LineLength Pos...

0
Q: Python reshape array

ZanamI have a very unique error I feel. I am working with an array 'A' of shape >>> A.shape (1L, 1823L, 24L) I am trying to get rid of first dimension as it is empty. So, I do something as: >>> z1 = A[0,:,:] >>> z1.shape (1253L,) >>> z1[0].shape () >>> z1[0] 10411.505889359611 I don't understand...

0
Q: Most efficient way to fill missing elements of dataframe with a function of column and row indices

piRSquaredI have a dataframe with missing values. import pandas as pd import numpy as np np.random.seed([3,1415]) df = pd.DataFrame(np.random.choice((0, np.nan), (5, 5))) print df 0 1 2 3 4 0 0.0 NaN 0.0 NaN 0.0 1 0.0 NaN 0.0 NaN NaN 2 NaN NaN 0.0 NaN NaN 3 0.0 NaN 0.0...

0
Q: How to include end date in pandas date_range method?

srodriguexFrom pd.date_range('2016-01', '2016-05', freq='M', ).strftime('%Y-%m'), the last month is 2016-04, but I was expecting it to be 2016-05. It seems to me this function is behaving like the range method, where the end parameter is not included in the returning array. Is there a way to get the end...

0
Q: Increment matplotlib color cycle

MikeIs there a simple way to increment the matplotlib color cycle without digging into axes internals? When plotting interactively a common pattern I use is: import matplotlib.pyplot as plt plt.figure() plt.plot(x,y1) plt.twinx() plt.plot(x,y2) The plt.twinx() in necessary to get different y-sca...

 
9:24 PM
0
Q: Python plotting percentile contour lines of a probability distribution

neither-norGiven a probability distribution with unknown functional form (example below), I like to plot "percentile-based" contour lines indicating boundaries around, for example, 50% volume inclusion, 70%, 80%, 90%, etc. ## example of an "arbitrary" probability distribution ## from matplotlib.mlab import...

0
Q: How do I specify na_values in pandas using read_sql?

Feynman27I'm trying to create a pandas dataframe from a sql table. I read in the data using data=pd.read_sql(query,con=con), and this works just fine. However, I wish to set which type of elements in the data frame is NaN. When reading a csv, this can be set using pd.read_csv('file.csv',na_values=['',[]']...

0
Q: How do you normalize an FFT in python?

LouI am trying to plot frequency vs power in python of a complex valued data array. However, I can't seem to quite get the values correct. I believe there is a normalization error somewhere. Using numpy, what I have is datafft = np.fft.fft(dataArray) #FFT of the data array (units of...

0
Q: Implementing a 2D recursive (FIR) filter using Scipy

0xf8Minimally, I would like to know how to achieve what is stated in the title. Specifically, signal.lfilter seems like the only implementation of a difference equation filter in scipy, but it is 1D, as shown in the docs. I would like to know how to implement a 2D version as described by this differe...

 
9:52 PM
0
Q: Keep as numpy array type when returning a scalar

Alexander McFarlaneAn array a = np.array(1) is convenient as it allows a.shape ** How can I preserved the numpy class type when doing an operation that returns a scalar e.g. b = 0. c = np.dot(a.T, a) b+=c b.shape --------------------------------------------------------------------------- AttributeError ...

0
Q: Is the object data type static in Python?

TfovidI have a data frame which contains two columns: one with strings and the other with integers. As expected, the data type of the integer column is int64. For the string column, however, it is object. Now I wanted to convert the string column to an integer column by assigning each string a given i...

0
Q: Pandas Series: Log Normalize

BenniI have a Pandas Series, that needs to be log-transformed to be normal distributed. But I canĀ“t log transform yet, because there are 1 < values < 1. Therefore I want to normalize the Series first. I heard of StandardScaler(scikit-learn), Z-score standardization and Min-Max scaling(normalization). ...

 
10:13 PM
-1
Q: Difference between a view and assignment

Abhishek BhatiaI can understand the difference between an assignment, shallow and deep copy. But I am still unclear what is the difference between a view(c=a) and an assignment(c=a.view()). Both reflect changes and seem the same. Please give examples if possible.

0
Q: Display the matching strings if at least half of string length matches

user6461192I have two data frames built using pandas with more than 13 columns each. In df1,one of the columns is "company_name_x". In df2, one of the columns is "company_name_y". Both the columns in their respective frames contain plenty of company names which are strings. As output, I want to display the ...

0
Q: scipy gaussian_kde image display: transpose vs. rotation vs. origin location

neither-norI'm trying to understand Scipy's gaussian_kde function using the documentation's example (pasted here for convenience): from scipy import stats import numpy as np np.random.seed(0) def measure(n): "Measurement model, return two coupled measurements." m1 = np.random.normal(size=n) m2...

0
Q: Is there some better way just to keep the duplicated rows in Pandas?

Lucas MeierI am just curious if pandas has a easier way just to keep the duplicated rows ? Pls dont understand me wrong. I have such a problem, because I unintented get the Pandas Dataframe wrong and have to work with the duplicated rows manuelly. Thank you all!

0
Q: Using fuzzywuzzy to create a column int he data frame

RedVIII'm running into a challenge with using the FuzzyWuzzy library to store all my results in a data frame column (I'm guessing it might require a loop?) I've been scratching my head over this all day, now I want to see if any of you can help me with the solution! Would be super helpful! As an exa...

 
10:36 PM
0
Q: How to plot a function on the secondary y-axis of a histogram in python?

samthebrandLet's use the famous Titanic dataset, found here: http://biostat.mc.vanderbilt.edu/wiki/pub/Main/DataSets/titanic3.xls And read it in as a dataframe: df The features that I'm interested in are 'age' (a float) and 'survived' (also a float, but binary 1 or 0). I know how to plot a histogram to ...

 
10:50 PM
0
Q: Droppin rows in pandas with a certain condition

robinpiksiI have a list of ID's and a dataframe, where one of the columns is ID. I want to drop all rows in the dataframe where the ID is not one of the ID's in the list of ID's. This is the code I use: df = df.drop(df[df.ID not in list_IDs].index) but I get this error message: ValueError: The truth va...

 
11:02 PM
0
Q: Is my merge sort python implementation efficient?

Orestis""" merge sort algorithm in python """ def merge_sort(array): if len(array) == 1 or len(array) < 1: # base case return array int_mid = len(array) / 2 left = merge_sort(array[:int_mid]) right = merge_sort(array[int_mid:]) return merge_arrays(left, right) def mer...

 
11:20 PM
0
Q: How can I make this run faster?

Will WangThis is question #10 from project euler. I just started programming and read that a sieve of eratosthenes is the fastest way to find primes however this code is taking forever to run. The sum of the primes below 10 is 2 + 3 + 5 + 7 = 17. Find the sum of all the primes below two million. end = F...

0
Q: pandas: How to get .to_string() method to align column headers with column values?

yobogoyaThis has been stumping me for a while and I feel like there has to be a solution since printing a dataframe always aligns the columns headers with their respective values. example: df = pd.DataFrame({'First column name': [1234, 2345, 3456], 'Second column name': [5432,4321,6543], 'Third column ...

 

« first day (846 days earlier)      last day (1342 days later) »