Algorithm and Software group collaboration project

5/26/2022 Pamela S Bedient

Artificial Intelligence algorithms are numerous widely used in a number of applications. However, comparing algorithm performance fairly can be challenging due to inconsistencies in datasets, frameworks, and outputs.

Written by Pamela S Bedient

Classifying Airplanes
Classifying Airplanes

Recently, the Applied Research Institute (ARI) partnered with Sandia National Laboratories on a Laboratory Directed Research and Development (LDRD) project focused on training, testing, and preparing an object detection algorithm for inclusion in their SeaScape framework which allows head-to-head comparisons between similar algorithms using the same dataset and API.

The Rapid Automation Validation and Verification (RAVVE) project developed by Sandia National Labs established a common API and labeled dataset for use by object detection algorithms. The challenge posed to the researchers was to train, test, and prepare an algorithm for use in this project. The object detection problem selected was to detect and classify three types of airplanes (passenger cargo, small aircraft, and fighter jets) from overhead imagery.

The team selected DetectNet as the base object detection algorithm. To assess algorithm performance, the team studied the effects of object class size (number of airplanes for each class), image size distribution (camera resolution), and proximity of adjacent airplanes on the algorithm’s ability to detect individual aircraft and properly classify them. The team tuned algorithm parameters and modified image pre-processing to maximize algorithm performance. As part of this effort, the team also characterized the algorithm limitations and data requirements, developed a confidence value metric, and repackaged the algorithm in a Docker container to prepare it for inclusion in the SeaScape framework.

By the end of the project, the team was able to improve the recall for passenger cargo, small aircraft, and fighter jets from 41%, 29%, and 12%, respectively, to 78%, 77%, 39%. Precision was improved from 41%, 36%, and 15% to 90%, 93%, and 96% for the three aircraft classes respectively.


Share this story

This story was published May 26, 2022.