MedGemma 27b text model to Vertex AI Endpoint Deployment Failure – “Model server exited unexpectedly”

HAI-DEF
1

Description:
I have a MedGemma 27b text fine-tuned model packaged as a .mar file (TorchServe format). The .mar file is uploaded to Google Cloud Storage and successfully imported into Vertex AI Model Registry.
When I attempt to deploy this model to an endpoint in Vertex AI, the deployment fails with the generic error message:
Model server exited unexpectedly
Additionally, the UI shows an “image not correct” type of message, even though the container image URI I specified exists and is accessible.
Details:
Model format: TorchServe .mar
Deployment target: Vertex AI Endpoint
Container image: Tried multiple images, below

“us-docker.pkg.dev/vertex-ai/prediction/torch-gpu.2-4:latest”

us-docker.pkg.dev/deeplearning-platform-release/vertex-model-garden/pytorch-inference.cu125.0-4.ubuntu2204.py310

“us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20250430_0916_RC00_maas”
Machine type: a3-highgpu-1g
Region: us-central1
Service account permissions: [confirm if Artifact Registry access is granted]
Issue:The error message is too generic, and I cannot determine the root cause. I need guidance on:
How to get detailed logs or failure reasons for this deployment (e.g., container logs, health probe failures).
Any known compatibility requirements for deploying TorchServe .mar models on Vertex AI (e.g., container entrypoint, health/predict routes, port configuration).
Steps to troubleshoot “image not correct” errors when the image exists in Artifact Registry.
Please advise on how to retrieve detailed error information and confirm the correct configuration for TorchServe-based custom containers on Vertex AI.

Below command to push to model registry then create endpoint
gcloud ai models upload
–display-name=“MedGemma_FineTuned_LoRA_Model_Test”
–artifact-uri=“gs://project-xxx/models/torchserve/fine-tuned-0812”
–container-image-uri=“us-docker.pkg.dev/vertex-ai/training/pytorch-gpu.2-4.py310:latest”
–project=“xxxx-itpr10996”
–region=“us-central1”
–model-id=“medgemma-lora-model-v1-test”
–description=“Fine-tuned MedGemma 27B model with LoRA adapters for obesity stage prediction.”
–labels=“model_type=llm,fine_tuned=true,task=medical_qa”

@tiffanychen Can you please help

3

Hi there, just acknowledging the thread. We will look into it and respond soon.

Fereshteh on behalf of the HAI-DEF engineering team

5

Hi @Ankit_Mishra The main reason for the failure is that the gcloud ai models upload command specified a training container .../training/pytorch-gpu... instead of a prediction container. Vertex AI Endpoints require a prediction container that runs an active HTTP server on port 8080 responding to health checks.

For large language models like MedGemma 27B (and LoRA fine-tunes), vLLM / Triton Server is the recommended high-performance serving architecture rather than legacy TorchServe .mar files. Either you can choose Vertex AI Model Gardento Deploy your MedGemma model or LoRA adapters directly via Vertex AI Model Garden, which automatically provisions optimized vLLM containers on GPU instances (L4/A100). Check our official notebook guide: quick_start_with_model_garden.ipynb or use a Custom Container with vLLM / Triton If deploying custom Docker images to Vertex AI. Ref.

Thank You.