GPUメモリの使用率監視はnvidia-smi
コンソールから以下を打つ。
> nvidia-smi -l Sat Jul 18 12:57:58 2015 +------------------------------------------------------+ | NVIDIA-SMI 346.46 Driver Version: 346.46 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla K20c Off | 0000:02:00.0 Off | 0 | | 42% 55C P0 67W / 225W | 2136MiB / 4799MiB | 34% Default | +-------------------------------+----------------------+----------------------+ | 1 GeForce GT 610 Off | 0000:03:00.0 N/A | N/A | | N/A 39C P8 N/A / N/A | 37MiB / 1023MiB | N/A Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 3318 C /home/XXXXXX/anaconda3/bin/python3 2121MiB | | 1 C Not Supported | +-----------------------------------------------------------------------------+
一旦リセットしたかったら使用しているprocessを単に止めればOK.それでもだめなら下記で。マルチGPUの時はID要チェック。
> sudo nvidia-smi --gpu-reset -i 0
