last day (15 days later) » 

8:36 PM
-1
Q: Regression data analysis of csv file data using R

user449355Hi I have large data sets in a csv file which is thousands of server memory usage through out day and months and years. For e.g. serverData <- read.csv("server_mem_usage.csv") head(serverData) It prints the following: date,server,datatotal(GBs),dataused(GBs) 10 Jun 2014 17:00:50,server1,800,5...

 
How will your desired output will look like?
 
I want in csv file or graph/plot anything is fine. Not able to figure out ideal output in R. If I write Java program then I can have Map of server name to dataused values.
I want data for each server for all the date ranges availables for that server in my data sets I have server1 has two dates 10 Jun 2014 17:00:50 and 10 Jun 2014 19:00:50. I want in the end data for each server for all date ranges and dataused and datatotal columns.
 
Is the output in my answer is what you are looking for?
 
thanks yes that looks good actually in the end what I want is one graph which will tell me server names with the most used data. Can you please guide me how should I get that kind of plotting?
 
So you don't need the datatotal(GBs) at all, and all you need is to sum dataused(GBs) per server and plot it?
 
8:37 PM
yes you are right datatotal will always be same for each server but I want to find sudden increase or decrease in dataused for each server and plot it.
 
Try
library(data.table)
library(ggplot2)
temp <- setDT(serverData)[, list(`dataused(GBs)`= sum(`dataused(GBs)`)),
by = server]
ggplot(temp, aes(server, `dataused(GBs)`)) + geom_point()
still not sure if this is what you want though
 
ok I will try and let you know thanks so much for your time and guidance.
 
also, are you sure your variables names are dataused(GBs)? This is usually not a valid column name in R
OR maybe you want a distribution plot per every server?
 
yes those are not correct names actually I cant put my company data into this forum so I created name myself. With given datasets of say 6 months we have around 45 thousand servers and so huge data sets. Every two minutes we collect data for each server so you can imagine how much data and I need to find glitch/hump for each server. Yes that sounds good each server plot for data usage.
I am sorry for all the confusion I just started working in R and nothing makes sense to me I am Java developer its completely new world of R for me so not able to understand logic.
 
dont think 45K plots will help you in any way
 
8:48 PM
Based on your expertise please suggest me the ideal solution.
 
First thing to do will be to define a "glitch/hump". Afterwords it is easy
I.e., if server uses over 95% of its capasity is a glitch
or if server uses significantly more than it is usually uses, then it is a glitch
or if a server uses more that the overall servers average is a glitch
something like that
 
yes that sounds good approach
 
I just named 3 different approaches...
 
ok where do I specify one of this approach? like this temp <- setDT(serverData)[, list(dataused(GBs)= sum(dataused(GBs))),
 
it won't be that simple
 
8:58 PM
:( then how do I solve this?
 
So which approach do you want?
for example
do you want the 90% usage approach?
 
over 95% of its capacity
 
do you want the plot to be per date?
like everyday you will want to see the names of the server who were above 95% capacity?
or you want to see the actual values?
 
yes per date would be a good idea
 
and all you need is the server names?
in that case you won't need a plot
you can get just a data set similar to what i gave u in the answer
which will give u a list os servers reached 95% capacity and above per date
Would that be sufficient for u?
u need to make some decision cause im running out of time
 
9:02 PM
its ok if server names
are coming as output I will think about making it plot
thanks a trillion for spending time with me.
 
plot of what
what do you want on your x/y axis
server names wont look good on a plot
 
yes you are right
 
of you have tens of them
or hundreds
 
I am sorry I am new to this plotting, graphs,data analysis so not able to give you intelligent answers
please suggest me working solution and I will work on it to optimize it in case I will doubt I will ask you later.
 
Ok
were u able to run my code that i previously provided?
with your real data?
 
9:09 PM
no I did not try as I am out of office but I will try first thing tomorrow morning and let you know
 
do you know how to install packages with R?
 
yes I have R studio installed basic commands I know
 
are you using any IDE?
ok
So the following code will show you per server per date if the server reached 95% capacity in any part of that date
library(data.table)
temp <- setDT(serverData)[, list(ServersReachedMaxCapacity = any(`dataused(GBs)` > `datatotal(GBs)`*.95)),
by = list(server, Date = as.POSIXct(date, format = "%d %b %Y"))]
you will need to adopt your real column names to the code
 
sure I will do that change variable and will try thanks a lot I wish I could give you 100 upvotes
I really appreciate your time have a good night
 
k
np
after you will do that, you will have to decide where you want to proceed with information
you can do another aggregationn that will give you all the names of the servers that reached the capacity
per date
 
9:18 PM
yes right
 
temp2 <- temp[ServersReachedMaxCapacity == T, list(servers = paste(server, collapse = ", ")), by = Date]
something like
That will give the list of all the servers which reached maximum capacity per date
Dont be scared of that syntax
this is not a normal R syntax
this is a data.table package syntax
which a bit different
 
ok
I see
 
but data.table is the most efficient package in r
so if you want your code to fly
instead of waiting hours
it is best to use data.table
 
hmm nice to know that
 
9:24 PM
g/n
 

last day (15 days later) »