U.S. Navy PMS 408 Reduces Time to Update ML Models by 97% Using Fiddler
The U.S. Navy’s NAVSEA Expeditionary Missions Program Office, or PMS 408, plays a critical role in safeguarding national security across maritime environments. One of their responsibilities is underwater explosive ordinance disposal (EOD), leveraging unmanned underwater vehicles (UUVs) to identify potential threats.
These UUVs rely on computer vision based Automatic Target Recognition (ATR) machine learning (ML) models to assess and classify underwater threats. However, monitoring and improving these ML models post-deployment proved challenging.
To keep models performing at their best, the Navy and the Defense Innovation Unit (DIU) collaborated in building the Automated Machine Learning for Mine Countermeasures Operations (AMMO) MLOps prototype, selecting Fiddler AI as a key participant.
Key Results from the Fiddler and DIU Collaboration
Fiddler’s collaboration with the U.S. Navy and DIU on Project AMMO delivered significant outcomes:
- 97% Reduction in Update Time: A robust MLOps pipeline was established, slashing model retraining and deployment times and increasing efficiency and enhancing operational readiness.
- Improved Explainability: Explainability features provided granular insights into the ATR models’ decision-making processes, guiding model developers in keeping their models up to date.
- Shipped to Production: Following the successful prototype, Project AMMO transitioned to production with the Naval Information Warfare Center Pacific (NIWC).
The Challenge of Maintaining ML Models
The Navy and DIU collaborated on the AMMO program in order to ensure ATR models maintain high accuracy, while remaining up to date with evolving threats. This task proved challenging as ML model outputs drift over time, and new technology is introduced by adversaries at a rapid pace.
The legacy process for retraining these ATR models was time and effort intensive, taking up to 6 months to identify areas of improvement and deploy updates. This timeframe risked operational readiness and model performance in mission-critical scenarios.
Building an MLOps Pipeline with Fiddler
In order to reduce their model retraining and deployment times, the Navy partnered with the DIU and Fiddler to build a robust MLOps pipeline, automatically surfacing issues and highlighting potential areas of improvement.
Monitoring integrated with visual debugging of image embeddings using UMAP helped identify operational changes in the ATR model’s behavior and perform root cause analysis quickly, enabling model developers to:
- Improve target accuracy across different battlefield environments
- Receive real-time alerts for potential threats from image data inconsistencies
- Make better decisions with deep understanding of UUV sensor data
Explainability in AI helped Navy developers and mission operators understand and trust the decisions of object detection and classification ATR models, enabling them to:
- Identify anomalies and anticipate potential threats
- Track differences in sensor or image data compared to training-time
- Enhance human decision-making in high-stakes post-mission analysis
Shaping Model Management for AI in National Security
With machine learning at the core of PMS 408’s EOD operations, Project AMMO required a robust solution to ensure the accuracy, transparency, and efficiency of their ATR models.
The Fiddler AI Observability platform, in collaboration with the DIU, played a pivotal role in shaping the Navy’s approach to AI model management and monitoring. As a result, the DIU has awarded Fiddler with a Success Memo to reflect the positive results of the program, and has since transitioned the prototype into production with the Naval Information Warfare Center Pacific (NWIC).
To learn more about AI in national security and how the Fiddler AI Observability Platform can help you improve LLMs and ML models in production, book a demo, or read additional case studies.
