ATRIA | EchoTrace: Fine-Tuning MedGemma 1.5 for Polygon-Based Heart Structure Contouring

Esteemed HAI-DEF & MedGemma Team,

I’m Shehab Anwer, a cardiologist. I recently shared some of my progress with Rory Pilgrim, who kindly suggested I post here to connect with the right folks for your valuable insights and feedback! Please read the disclaimer and references for more information on ATRIA and EchoTrace prototype (sections under my signature.)

I’ve been exploring the fine-tuning of the google/medgemma-1.5-4b-it model for endocardial tracing. Specifically, my project focuses on polygon-based contouring for the left ventricle (LV) endocardial border and the left atrium (LA) - and can be adjusted to any contouring tasks.


The Clinical Impact:
Automating this process addresses a critical, day-to-day task in Cardiac Imaging. Accurate contouring is the essential precursor for downstream management of cardiovascular disease, specifically enabling:

  • Geometric Assessment: Evaluating anatomical features, such as chamber area and estimated volume.

  • Mechanical Function: Assessing physiology, including the fraction at the end of contraction versus the end of relaxation.

  • Criteria-Based Decision Making: Applying gender-based normal limits for anatomical/functional parameters and executing guidelines-based decision workflows.


A quick overview of my workflow and setup:

  • Framework Integration: DEITY Principles Framework - The Adimension’s Foundational Framework to cover the pillars (Data, Ethics, Informatics, Technology and You) to healthcare operations towards human-machine synergy.

  • Dataset Integration: I am using pre-processed echocardiographic frames (ED and ES instants as PNGs) paired with serialized JSON traces.

  • Coordinate Geometry: The ground-truth segmentation masks are pre-computed into polygons featuring 30 points each for the LV and LA. These polygon coordinates are normalized to a [0, 1000] scale.

  • Prompting & Training: My prompt engineering was heavily inspired by the box localization notebook, but I have directed it specifically toward polygon geometry. I am currently utilizing JSON structured outputs from QLoRA training.

  • Structured Output in JSON array: MedGemma exceptionally capable of following the prompt command post-finetuning, and programmtic extraction of coordinates into JSON array follows the inference. Specifically to enable further dataset iteration, integration with PACS/DICOM packages, and enable further AI tasks (further tuning, reinforcement learning, directing towards another model architecture).

  • Human-in-the-Loop UI: I have built an interface to correct the model’s outputs. This allows the corrected data to be reused in continuous fine-tuning, which is incredibly helpful for managing inter- and intra-operator variability!


What Does ATRIA EchoTrace Currently Targets?

Internal development and validation within the ATRIA cardiac AI research initiative. The platform is designed for immediate collaborative adoption by academic hospitals, echocardiography core laboratories, and medical AI research groups working with CAMUS, EchoNet-Dynamic, or institution-specific echo datasets. Open to structured pilot programs focused on annotation efficiency, ground-truth quality improvement, and responsible AI governance.


What Are the Current Challenges Facing ATRIA EchoTrace?

  • Transitioning the interactive revision UI from Colab notebook to a lightweight, production-grade web or desktop clinical application
  • Extending the pipeline to full video/sequence modeling, temporal consistency, and additional cardiac structures (RV free wall, valves, strain curves)
  • Clarifying regulatory pathways and intended-use boundaries if the tool evolves from research annotation aid toward clinical decision-support components
  • Building larger-scale clinical validation studies quantifying time savings, inter-observer variability reduction, and downstream impact on ejection fraction/strain measurements
  • Further lowering resource barriers and optimizing for real-time or edge deployment scenarios

I would highly appreciate any insights, feedback, or best practices the team might have regarding polygon geometry generation, adapter training, or UI integration with MedGemma 1.5.

I am really grateful for the foundation you all have built—thank you to the moon and back!

Best regards,
Shehab Anwer, MD
Cardiologist & Medical AI Developer
The Adimension | Founder, CEO.

  • Google Scholar: [ scholar .google.com/citations?user=W-rpYiUAAAAJ&hl=en]

Important Disclaimers

I. INTENDED USE
ATRIA EchoTrace is currently classified as a Prototype / Advanced Minimum Viable Product (MVP). The platform, including its MedGemma-based models, LoRA adapters, pipelines, and user interfaces, is explicitly intended for research, academic collaboration, and internal validation. It is NOT a cleared or approved medical device by regulatory bodies at the time of this document generated.

II. NON-CLINICAL USE
Not for Direct Clinical Diagnosis.ATRIA EchoTrace is an AI-powered annotation and drafting tool designed to support and augment medical research and dataset creation. The software must not be used as a standalone diagnostic tool, for direct patient care, or as the sole basis for clinical decision-making. Any downstream metrics derived from its outputs (such as ejection fraction, cardiac volume, or strain calculations) are strictly for research and validation purposes.

III. MANDATORY HUMAN OVERSIGHT
In strict adherence to the DEITY Principles Framework (specifically the “You” pillar), this system relies on a Human-in-the-Loop (HITL) architecture. All AI-generated structural contours (LV/LA) and annotations are preliminary proposals. They must be independently reviewed, verified, and revised by qualified echocardiographers, cardiologists, or trained medical personnel.

IV. DATA PRIVACY
Users of ATRIA EchoTrace are solely responsible for ensuring that all echocardiographic inputs (DICOM, AVI, PNG) are properly anonymized and de-identified prior to processing.

V. REGULATORY COMPLIANCE
Users must ensure their use of the platform complies with all applicable local, national, and international healthcare data privacy regulations (e.g., HIPAA, GDPR).

VI. “AS IS” PROVISION
The ATRIA EchoTrace codebase, models, artifacts, and documentation are provided “AS IS” without any warranties of accuracy, reliability, or clinical fitness, either express or implied.

VII. LIMITATION OF LIABILITY
The developers, contributors, and affiliated research initiatives assume no liability for any direct, indirect, or consequential damages, or clinical outcomes arising from the use, misuse, or inability to use this platform.


REFERENCES

  • The Adimension & DEITY Principles Anwer, S. (2026). The Adimension: Bridging human ingenuity and machine intelligence through the DEITY principles framework. European Heart Journal - Imaging Methods and Practice, 4(1), qyaf038.

  • CAMUS Dataset Leclerc, S., Smistad, E., Pedrosa, J., Østvik, A., Cervenansky, F., Espinosa, F., Espeland, T., Berg, E. A. R., Jodoin, P.-M., Grenier, T., Lartizien, C., Dhooge, J., Løvstakken, L., & Bernard, O. (2019). Deep learning for segmentation using an open large-scale dataset in 2D echocardiography. IEEE Transactions on Medical Imaging, 38(9), 2198–2210. [

  • EchoNet-Dynamic Ouyang, D., He, B., Ghorbani, A., Yuan, N., Ebinger, J., Langlotz, C. P., Heidenreich, P. A., Harrington, R. A., Liang, D. H., Ashley, E. A., & Zou, J. Y. (2020). Video-based AI for beat-to-beat assessment of cardiac function. Nature, 580(7802), 252–256.

  • Google MedGemma 1.5 Google. (2026). MedGemma 1.5: Technical reports and model card (google/medgemma-1.5-4b-it). Hugging Face.

  • LoRA: Low-Rank Adaptation. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA: Low-rank adaptation of large language models. arXiv.

The Adimension’s ATRIA EchoTrace notebook is available online at:

Hello Shehab,

Thanks for building with MedGemma! I hope I don’t miss anything:

  • We’ve not trained MedGemma for segmentation and contouring tasks, but if I’m reading it right, you’ve managed to adapt it for the task? Can you share your performance metrics and how much labeled data you used?
  • As for adaptation, given how different the task is from baseline capabilities, the only viable solution would be fine-tuning, which I believe you’re doing already. Nevertheless, this FAQ lists more techniques.
  • As for UI integration, if the app can be online, then you’d deploy your adapted model as a service endpoint and have your front-end app use the model as an API. It will become typical app development you could even vibe code. This FAQ could be relevant.

Hope this helps!

Fereshteh

1 Like

Hi Fereshteh (@fmahvar),

Thank you so much for the thoughtful reply and the FAQ pointers — much appreciated!

To address your questions, please find a point-by-point response below, in addition to these two videos that explains the work, and show a screen recording of the HITL interface:

Examples of output, also more are visible on the Google colab notebook:


1. Adaptation for contouring + metrics / labeled data
Yes — we have adapted MedGemma-1.5-4B-it for polygon-based endocardial contouring of the left ventricle (and planning into the left atrium). This is not traditional semantic segmentation (i.e., we are not generating pixel-wise masks). Instead, we are building directly on MedGemma’s demonstrated strength in anatomical localization (as shown in the official MedGemma notebook for CXR anatomy localization: medgemma/notebooks/cxr_anatomy_localization_with_hugging_face.ipynb at main · Google-Health/medgemma · GitHub ).

We fine-tune the model with QLoRA (extensive application to the vision tower’s attention and MLP layers across nearly all 27 encoder layers, plus language components) so that it directly outputs normalized 30-point closed polygon coordinates as structured JSON for the endocardial border on apical 2CH/4CH echocardiographic frames at ED/ES phases.

Each adapter was trained on the preprocessed frames as PNG with the tracings normalized in 30-point polygon annotations. Full details of preprocessing, splits, and metadata are in the resources - the preprocessing script will be uploaded to the GitHub repo soon.


2- Performance metrics:
As this remains an early prototype focused on the complete annotation workflow, we have not yet published formal quantitative benchmarks (Dice, IoU, Hausdorff distance, etc.). The emphasis is on producing high-quality initial contours that meaningfully reduce expert annotation time through the human-in-the-loop process. The notebook contains evaluation code so metrics can be computed on held-out data. We are happy to discuss results or run additional evaluations. The following screenshot is from the notebook, they show the variability in performance at this stage given the short training cycles (due to computational resources, that follows experimentation with prompt to reach a stable training, and now planning to run it longer on the most stable hyperparameters to properly report the performance metrics stated earlier.





3. Fine-tuning approach
We fully agree that fine-tuning (via QLoRA) was the right approach given the shift in task and output format. The complete fine-tuning configuration, training script, and hyperparameters are openly available in the notebook, and the various experimentations on dataset and the training/eval loss on TRL are available upon request.

We have released both adapters so the community can continue training on in-house EchoFirst data without starting from scratch:


4. UI integration & deployment
Your recommendation aligns perfectly with our direction. The Google Colab notebook contains the entire end-to-end pipeline — from data preprocessing, through the exact QLoRA fine-tuning setup, to inference, and all the way to the interactive 4-panel human-in-the-loop (HITL) revision interface (Original | Model prediction in red | User revision in green | Overlay) with vertex-by-vertex polygon editing.

All outputs are saved as clean structured JSON (model + revised polygons) plus visualizations, making downstream integration straightforward. For a production web or clinical app, serving the model/adapter as an API endpoint (as you suggested) is the natural next step, and we are actively planning that transition.


All resources are on:

This work follows the DEITY Principles Framework, with mandatory clinician oversight for all outputs. The goal is to accelerate the creation of high-quality, reusable training data while keeping humans firmly in control: doi..org/10.1093/ehjimp/qyaf038

We would greatly value feedback from the MedGemma and Google Health community. If you or others would like to experiment with either adapter on private data, collaborate on evaluation, or contribute to the deployment path, please let us know — we’re very open to collaboration.

Thank you again for the helpful questions & guidance!

Best,
Shehab Anwer
The Adimension

Update on ATRIA EchoTrace (since the earlier exchange)

Several pieces are now public and more complete:

Technical implementation

The Colab notebook, CLI/API (atria with the full pipeline), and previous fine-tuning session logs are available. Everything is on GitHub:

(notebook is runnable in Colab; platform, benchmark harness, and preprocessing scripts are included).

Segmentation accuracy (official CAMUS test split — 50 patients, 200 frames, never trained on)

  • Dice 0.744 ± 0.135 · IoU 0.609 ± 0.156 · MAD 6.22 mm
  • Primary metric is point-to-curve distance (median 4.98 mm), since the model emits coordinates rather than a mask
  • 200 / 200 outputs parsed to a valid in-bounds polygon

Full results, overlays, and the evidence room (training runs, metric agreement, limitations) are here:

Gallery: ATRIA overlay review — 200 frames

Evidence: Evidence — ATRIA EchoTrace

Developer utility (the item we’d most like surfaced)

After a transformers upgrade renamed the vision-encoder module path, 324 of the adapter’s 802 tensors were loading into nothing — silently. PEFT warned and continued, so roughly 40 % of the vision-tower fine-tuning was inert while the model still produced plausible contours.

This can affect any HAI-DEF developer fine-tuning a vision tower. The detection-and-repair pattern is documented on the site and is now a regression test.

Small correction

The adapter produces 29–53 points (median 31), not a fixed 30. “Structured coordinate generation” is the accurate framing; this will be tightened in future prompt hardening.

Limitations are published alongside the results. Happy to adjust length or emphasis if useful for the showcase template.

All materials remain research-use only, with the same DEITY / clinician-oversight stance as before.

Thanks again for the amazing work that enabled ATRIA-EchoTrace to be built in the open, and it is open invitation to collaborate and improve the current state.

Shehab