Real-time semantic segmentation can help autonomous robots interpret their surroundings, but running dense image analysis onboard a micro aerial vehicle (MAV) is challenging. The processor, memory and power budget are constrained by the platform’s size, weight and power (SWaP) limits, while offloading images to the cloud may not be practical when connectivity is unavailable. MAVNet addresses this problem with a compact segmentation network designed for onboard use, including target tracking and infrastructure inspection.
Why onboard segmentation matters for MAVs
Semantic segmentation assigns a class to each pixel in an image. Unlike image-level classification, which labels an entire frame, segmentation can distinguish an object from its background and identify relevant regions such as corrosion, rivets or water. For a MAV, this pixel-level information can support tasks such as locating another aircraft, building a semantic map or identifying areas that merit closer inspection.
These tasks bring demanding visual conditions. A target MAV can appear at different distances and orientations against changing backgrounds. Infrastructure imagery may be noisy, poorly illuminated or captured by grayscale and fisheye cameras. Processing these images in real time is especially demanding when the vehicle cannot rely on a powerful desktop GPU or a reliable connection to remote computing resources.
MAVNet was developed for this constrained setting. The reported design uses approximately 4,300 parameters—about 400 times fewer than the ERFNet reference model—and aims to balance model compactness, inference speed and segmentation quality. Its results are not uniformly best across every accuracy measure; rather, they show why model selection should reflect the operational task.
How MAVNet balances speed and accuracy
Reducing a network’s parameter count alone does not guarantee efficient inference or useful predictions. MAVNet instead uses a purpose-built encoder-decoder architecture inspired by ERFNet. Two convolution-convolution-pooling blocks reduce the image’s spatial dimensions early, lowering the computational burden of later layers. Downsampling also lets subsequent filters work with a broader field of view, which can help account for objects that vary in apparent size.
The encoder then uses four depth-wise feature aggregation blocks, with dilated convolutions at rates of 2, 4, 8 and 16. Dilation expands the receptive field without requiring the same increase in parameters as adding conventional convolution layers. That broader context can help a network interpret objects in relation to their surroundings. The experiments also found that removing dilation reduced performance on the penstock dataset, while the modified network did not converge on the MAV dataset.
A short decoder restores the segmentation to the input image size. Rather than using transposed convolution, its upsampling blocks use nearest-neighbor upsampling followed by regular convolution. This design is intended to limit checkerboard artifacts associated with deconvolution, while avoiding the computational and memory demands of a longer decoder with extensive skip connections.
Task-specific datasets reveal different challenges
Performance was evaluated on two datasets that reflect distinct MAV applications, rather than relying only on conventional driving-image benchmarks.
- MAV tracking: Indoor images show other MAVs against varied backgrounds. The training labels cover about 1,000 pixel-wise samples from 12 flights; the test set contains about 300 images from a different, more cluttered indoor location. The target aircraft was also modified for testing, introducing a change from the training examples.
- Penstock inspection: Fisheye-camera imagery collected during flights inside a dam penstock is labelled for background, corrosion, rivets and water. The images include challenging lighting and occlusion, and the training, validation and test sets contain 39, 64 and 35 images, respectively. The split was designed so that areas visible in training were not seen in testing.
These datasets matter because visual conditions and class balance influence what a model learns and how its results should be interpreted. For example, the MAV images contain far fewer target pixels than background pixels. The training approach therefore applies focal loss, which gives relatively greater weight to difficult examples, with an additional class weight for MAV pixels in the highly imbalanced tracking dataset.
Evaluate the model against the operational objective
On the MAV dataset, ENet achieved the highest reported intersection over union (IoU), a measure of overlap between predicted and labelled object pixels. MAVNet did not lead on that metric. However, it recorded the lowest centroid distance error: 3.36 pixels, compared with 3.46 pixels for ENet. Centroid distance measures how far the predicted object’s center is from the labelled center, making it relevant when the task is to localize a target for tracking. MAVNet also had a 3.34% false-negative rate, while its false-positive rate was comparatively high.
On the penstock dataset, MAVNet-Focal achieved the highest mean IoU across the three object classes: 36.24%. Its performance varied by class, as it did for the comparison models. For corrosion, for example, its IoU was 57.66%, below UNet’s 61.67%, while its results across the classes contributed to the best overall mean. The authors also report a tendency toward more false-positive pixels for MAVNet and some other compact models using dilated convolutions. That trade-off may matter when false alarms create extra review work or trigger unnecessary actions.
Embedded inference and deployment considerations
At an input size of 1024 × 1280 pixels, the paper reports MAVNet running at 4.4 frames per second on the Falcon 250 platform equipped with a Jetson TX2, and 8.9 frames per second on a Jetson Xavier. The reported rate reaches up to 48 frames per second on an NVIDIA 1080Ti. These are measurements from the evaluated implementations; the models were not optimized with tools such as TensorRT, and results depend on the hardware and test conditions.
For engineering teams, the key lesson is to assess more than frame rate or parameter count. Validation should match the deployment camera, lighting, target classes and hardware. Teams should also choose metrics that reflect the job: IoU for pixel-level overlap, false-positive and false-negative rates for error patterns, and localization measures where tracking depends on an object’s position.
Conclusion
MAVNet demonstrates a practical approach to real-time semantic segmentation under tight onboard-computing constraints: use an architecture designed for efficiency, test it on representative imagery and evaluate it against the operational objective. Its compact size and embedded inference results are promising for MAV applications, but its accuracy trade-offs—including false positives—remain important deployment considerations. For automation and inspection teams, task-specific testing is essential before selecting a model for an autonomous system.