Memory usage when training on gpu

[mohgh]

I'm using this code for conversational modeling with Subtle corpus containing approximately 5.5 million message response pairs. When I train the model with GPU with medium model params, python process allocates nearly 26 gigabytes of ram at step 100000 and got killed by the kernel. Is it normal or there's memory leakage?
What data is lost when I restart the training from saved checkpoint?