last day (15 days later) » 

12:10 AM
0
A: Why Pandas is not creating a new sheet in excel?

BiarysThe reason you have only one tab is because thru each iteration you create a new file called "Stock-{}.xlsx" with one tab in it. Depending on your code, you will either end up with multiple excel sheets, or one sheet that gets rewritten multiple times. What you should do is the following: for in...

 
filepath = "Stock-{}.xlsx".format(index.strftime("%m")) in this statement, index is generated from the for loop, so it can not be written outside. I have tried using with Excelwritter within the for loop, and it doesnt work, still creating a new file rather than reading my existing file
 
right, didnt notice it. so for each month you want to create a new file?
 
basically, I have already created the empty spreadsheet as a container, then when I run through my historical data, i extract the date, find out what month it belongs to, open the corresponding spreadsheet, wanna write the entire dataframe into a new sheet in this workbook (this is where I failed to do so, save and close the workbook, be ready for the next iteration
 
so if you have 2 months of january, in the second iteration it needs to open up an existing sheet and modify it? That's quite problematic as far as I know
 
yes, so let say i have written 1st Jan in the Jan book, then when I run into 2nd Jan, it should open the Jan book which has 1st Jan page created, then create the page of 2nd Jan in the book. So just keep appending the excel with sheet
 
12:10 AM
so 1 excel sheet with 6 tabs?
 
yes, i may not be using the same terms as you, but i get what you mean. Workbook = excel file, Sheet=tab=page
 
Then why excel file name changes month to month? filepath = "Stock-{}.xlsx".format(index.strftime("%m")) will result in a new file each time. If you wanna keep one file, I recommend you use filepath = "Stock.xlsx"
 
Hi Biarys
thanks for helping me
 
np
 
I have the following structure
 
12:13 AM
i had my share of pain working with excel in pandas
 
Stock-01, Stock-02, Stock-03 n etc
until Stock-12
so 12 months in total, 12 files
 
does each file have many months or just one?
 
each one of these excel file, will have 28 to 31 sheets/tabs/pages
so in Stock-01, it has 31 sheets, and each sheet has the intraday data for the date
this is the end result i want to obtain
because now I have one single giant csv file that has a year worth intraday data
hope it makes clear
 
so you have 1 file with a year worth of intraday data
you wanna break it into 12 files by months?
i am confused now :)
 
exactly
sorry to make you confused
but does it makes sense now?
 
12:18 AM
and each file needs to have 28-31 tab, which denotes to a day worth of data?
 
yes
 
i see
do you only have 1 giant csv file that has a year worth intraday data or many?
 
only one
 
ok
can you pd.read_csv(giant_excel) into memory or its too big to read as a whole?
 
no problem reading it
 
12:23 AM
then, the way i'd do it is
df=pd.read_csv(giant_excel)
 
yes
 
then a list of days and months
for each month create new file
for each day, create new tab
can you update your question cuz its different from what you originally asked
i'll check the ecode meanwhile
 
ok
i have edited the question
once again appreciate for your time helping me
 
np
try this one
for month in months:
    filepath = "Stock-{}.xlsx".format(month)
    with ExcelWriter(filepath) as writer:
        for day in days:
            name = "Stock_" + "-".join(df.index.year.unique()[0], month, day)

            temp = df.loc[(df.index.month == month) & (df.index.day == day)]
            temp.to_excel(writer, index = True, sheet_name=name)

        writer.save()
let me know if it works
what do you think?
 
12:56 AM
it doesnt quite work, it does create a few sheets now, but didn't populate the data
but probably is the loc function
i will take a deeper look into it first
it is actually 1am here, may I come back to you tomorrow?
will definitely let you know the result
probably can you explain to me your approach please?
 
can u wait 5 mins?
 
yes
 
the script works for me
i fixed it
oh sorry
i know why it doesnt work
full script
df = pd.read_csv("D:/AmiBackupeSignal/AMGN.txt", index_col="Date/Time")

df.index = pd.to_datetime(df.index)

days = df.index.day.unique().sort_values()
months = df.index.month.unique().sort_values()

for month in months:
    filepath = "Stock-{}.xlsx".format(month)
    with pd.ExcelWriter(filepath) as writer:
        for day in days:
            name = "Stock_" + "-".join([str(df.index.year.unique()[0]), str(month), str(day)])

            temp = df.loc[(df.index.month == month) & (df.index.day == day)]
that one created 12 excel files
with 31 tab in each
 
so what is the logic here? are we trying to creating the excel file while we are generating the data?
rather than reading an existing excel?
 
first we read all data into a dataframe
then we modify index to be datetime index
we get all months and days we have
for each month, we create new excel file
for each day we create new tab
temp has data for the month/day
we write temp to the excel file
 
1:03 AM
so we never read an existing excel file?
 
not really
you could do it
 
is it not possible with pandas to read an excel and modify it?
just interested to know
 
it is possible, but fucked up
pandas has problems with modifying existing sheet, as far as i know
 
i guess it is not what pandas is used for lol
 
what you will need to do is to read that sheet
then append to the existing dataframe
and write back to it
 
1:05 AM
make sense
 
i think it might be easier to simply append to a csv file
with open(csv_file, "a+"):
as long as you dont have charts/macros
there seem to be an updated answer where a guy talks about using xlswings or smth like that to modify
 
next step is to create chart, hence i want it to be excel
 
but when I was doing it, it was quite a pain
 
well, then i don't think i want to look into it
lol
 
not sure why u wanna do what u do anyway
 
1:10 AM
what you mean? the approach that i initially took?
 
yea, like why u wanna separate this sheet into 12 sheets with 30 tabs
 
because one giant file is too large for selecting the right data to generate chart
i think i can break it down, then use excel to plot the OHLC chart for each day
 
well, good luck
 
btw, one last question
i got my months using months = pd.Series(data.index.strftime("%m").unique())
this gives me 01, 02, 03 n etc
and data.loc[(data.index.month == month doesnt work because index.month return 1, 2, 3 n etc
this is why i got empty spreadsheet
how should I resolve it?
 
i modified my answer
months = df.index.month.unique().sort_values()
df.index = pd.to_datetime(df.index)
then
months = df.index.month.unique().sort_values()
your index is in string format
so when you do data.loc[(data.index.month == month)] you get empty dataframe
you can either change index to datetime index as i did
or change data.loc[(data.index.month == month)]
 
1:19 AM
got you
 
actually not sure you would change data.loc[(data.index.month == month)]
keeping index in datetime format for time series data is a good thing
pls dont forget to upvote/accept my answer if u are satisfied
 
will do
many thanks
 
np
 

last day (15 days later) »