last day (16 days later) » 

9:19 AM
0
A: Out of memory running Tensorflow with GPU support in PyCharm

Steve CarterTo wrap up our conversation as per the comments, I'm do not believe that you can allocate GPU memory or desktop memory to the GPU - not in the way that you are trying to. When you have a single GPU, Tensorflow-GPU in most cases will allocate around 95% of the available memory to the task it runs...

 
I am running on Kubuntu, see my edition above. Why does only 1 process use the GPU memory in your case, but in my case quite a few processes are using the GPU memory? In TensorFlow GPU, models only use GPU's memory? What about other RAM? My computer has a total of 64G RAM. Are they almost not used?
 
Yes, in Tensorflow-GPU, models only use GPU memory. If you want to use your desktop memory, then you would not use GPU support for the model. Keep in mind that the CPU will be the bottleneck if you do your training on the CPU, not the memory. The main reason for using your GPU is because it is much faster than CPU's in most situations. In my case, I run Ubuntu server 16.04 and I have no graphical interface so there is only 1 process running on the GPU which is a Tensorflow program I am developing.
I see in your update that you have one PID 20728 using most of your GPU's resources. This is your program already running in the background. If you were to end that program, then resources would be available for the one you are trying to run. In your case, if you are trying to run 2 training scripts on the same GPU, this is not going to be possible with the current memory allocation.
 
The PID 20728 is the process I am running. Somehow it has not out of memory anymore, but very very slow. I am testing CPU to train the same model. It may be much faster in this case.
You have a only 4G gpu, how fast is it in your case? Did you compare it with CPU training?
 
Is was a GTX970, it was okay for training - still faster than my 4Ghz CPU, 2 of them was better again. Now, I keep that one GTX970GPU for training & testing of models that I don't care as much about. When I have something I really need to get done, I use my main PC with twin 1080ti cards. They have 11gig each. Its important to note that you can also run out of memory if your config is wrong. Say you double your memory, then you can train larger of batches of data because more can be loaded into memory. If you get an out of memory error, reducing the batch size can help to resolve that.
Hi Ling, happy to give you any more advice if needed, but I think continuing this discussion on the actual question will violate the forum rules - feel free to ask me anything else - I'll try to help out as best I can.
 
Hi, Steve, I am have a little code for learning purpose. It takes a lot of time to train. I suspect something might be wrong. Can I send the code to you and you test on your 4GB GPU how much time it needs to train?
Do you compile TensorFlow from source by yourself, or you just install by pip install tensorflow?
 
9:46 AM
I just install tensorflow using pip. Im just off to a movie now but ill discuss when im finished. About 3 hrs
 
9:57 AM
Enjoy! I am going to sleep, and it's too late here. My code is here from a book and the author says it takes about 5 minutes for the grid search using CPU. But my computer takes too much time and not sure why. The code this here:
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import GridSearchCV

import time

import numpy as np

start_time = time.time()

# Function to create model, required for KerasClassifier
def create_model(optimizer='rmsprop', init='glorot_uniform'):
model = Sequential()
model.add(Dense(12, input_dim=8, kernel_initializer=init, activation='relu'))
model.add(Dense(8, kernel_initializer=init, activation='relu'))
If you have a result, please leave a message for me. Or email me: lingvisa@gmail.com. Thank you so much!
 
 
3 hours later…
12:30 PM
I've got your code and am ready to run it.. I need pima-indians-diabetes.csv though. Can you provide it?
my email is kangarooted@gmail.com
 
12:42 PM
NVM, I downloaded it from the net
Epoch 46/50

5/512 [..............................] - ETA: 0s - loss: 0.5550 - acc: 0.8000
100/512 [====>.........................] - ETA: 0s - loss: 0.5110 - acc: 0.7700
200/512 [==========>...................] - ETA: 0s - loss: 0.4830 - acc: 0.7900
305/512 [================>.............] - ETA: 0s - loss: 0.5195 - acc: 0.7443
415/512 [=======================>......] - ETA: 0s - loss: 0.5327 - acc: 0.7349
512/512 [==============================] - 0s 494us/step - loss: 0.5457 - acc: 0.7285
I'll let you know how it went in the morning - i'll run it overnight
 
 
5 hours later…
5:34 PM
Thank you. My GPU run finished at 59 minutes, and the best result:
Best: 0.744792 using {'batch_size': 10, 'epochs': 150, 'init': 'normal', 'optimizer': 'rmsprop'}
 
 
5 hours later…
10:09 PM
768/768 [==============================] - 1s 884us/step - loss: 0.4666 - acc: 0.7799
Best: 0.746094 using {'batch_size': 5, 'epochs': 150, 'init': 'uniform', 'optimizer': 'rmsprop'}
0.696615 (0.014731) with {'batch_size': 5, 'epochs': 50, 'init': 'glorot_uniform', 'optimizer': 'rmsprop'}
0.669271 (0.037377) with {'batch_size': 5, 'epochs': 50, 'init': 'glorot_uniform', 'optimizer': 'adam'}
0.692708 (0.013279) with {'batch_size': 5, 'epochs': 50, 'init': 'normal', 'optimizer': 'rmsprop'}
0.708333 (0.010253) with {'batch_size': 5, 'epochs': 50, 'init': 'normal', 'optimizer': 'adam'}
Hope that is of use to you!
 
10:40 PM
So it also takes about 1 hour. Your GPU is 4G ram?
 
I'm just running this on my CPU now. Initial numbers is that for this type of training is best served to run on your CPU. My guess is that the data is not real complex. On my CPU vs GPU - the CPU training is around 5 times faster. The main reason for this is the complexity of the data - or lack there of. I'll let you know the results of running on my CPU
The GPU I am running is actually 11G - its a 1080ti
I've a few GPU's here :-D
But based on what I'm seeing here - if the csv file is the typical data you are using for your training, then I don't think there is any benefit to you using your GPU. GPU comes into its own when data is large and complex and batch size is of a decent size. When you think about it, your csv data is 1 x array of 768 x 9. Thats not a lot of data on the grand scheme of things.
When I train data on my GPU, it might be 100,000 images around 600 x 600 pixels in size. In my case for the training I am doing, GPU runs rings around CPU, but I don't thing you will see the benefit. What is the actual goal of your project anyway?
 
11:11 PM
useful info about training on GPU vs CPU
23
Q: Choosing between CPU and GPU for training a neural network

StatsSorceressI've seen discussions about the 'overhead' of a GPU, and that for 'small' networks, it may actually be faster to train on a CPU (or network of CPUs) than a GPU. What is meant by 'small'? For example, would a single-layer MLP with 100 hidden units be 'small'? Does our definition of 'small' c...

FYI 24 mins on my CPU
Best: 0.748698 using {'batch_size': 5, 'epochs': 150, 'init': 'uniform', 'optimizer': 'rmsprop'}
0.692708 (0.012075) with {'batch_size': 5, 'epochs': 50, 'init': 'glorot_uniform', 'optimizer': 'rmsprop'}
0.679688 (0.046983) with {'batch_size': 5, 'epochs': 50, 'init': 'glorot_uniform', 'optimizer': 'adam'}
0.696615 (0.015733) with {'batch_size': 5, 'epochs': 50, 'init': 'normal', 'optimizer': 'rmsprop'}
0.717448 (0.009744) with {'batch_size': 5, 'epochs': 50, 'init': 'normal', 'optimizer': 'adam'}
 

  last day (16 days later) »