OOM while training LLM

Leonhard_Piff · April 15, 2024, 1:59pm

Im working on trying to fine tune a 7 Billion parameter model in TF keras but I’m having the problem that I can’t distribute the model across multiple GPUs.

I read some about the parameter server strategy but I can’t really get it to work the way I want to.

If anyone has any insights they would be greatly appreciated.

Kiran_Sai_Ramineni · April 24, 2024, 9:04am

Hi @Leonhard_Piff, Please refer to this document for running Gemma 7b on multiple GPU’s. Thank You.

Topic		Replies	Views
Distributed inference with JAX: GPU/TPU interconnect TensorFlow distributed-training , gpu , tf-probability	0	33	August 23, 2024
Parallelising model with multiple inputs Keras distributed-training , keras , custom-loss , gpu , model-training	3	453	May 21, 2024
Parallel inferencing - image classification model General Discussion gpu , help_request	1	486	June 28, 2022
Model Maker TF Lite Slow Inference General Discussion models , tflite , model_maker , help_request	11	5450	September 6, 2021
Hardware recommendation for a box to train object detection models General Discussion models , tflite	2	579	March 7, 2023

OOM while training LLM

Related topics