SUMMARY of RESULTS
The main objective of CORTEX is to define a workflow easing the integration and reduction of complex deep neural networks (DNN) models on SoC FPGA devices embarked in spatial platforms. This workflow has been demonstrated on a deep learning (DL) image processing pipeline devoted to feature extraction and tested on an FPGA representative of the on-board hardware.
In order to define a generic pipeline, three use cases have been selected: 1) ships detection in Sentinel-2 images with a DNN trained with transfer learning technics, 2) oil spills and ocean features detection in Sentinel-1 images, two separate models are developed to demonstrate the flexibility offered by easy processing updating on-board, 3) Sentinel-1 to Sentinel-2 transformation applied on specific regions (refugees camps), demonstrative use case for future applications based on generative adversarial networks.
Deep Learning models have been implemented and delivered as well as generated/modified data bases and training software. The networks trained for use cases 1 and 2 showed very good performances (F1-score over 80%) compared to available public results.
The results obtained for use case 3 were encouraging even so they would need a more extended study.
The distillation process showed to be fairly robust and provided low performance loss with a drastic reduction in the number of parameters of factor 52 on the use cases 1 and 2. A small performance drop was observed for the oil spill case, but we believe that more parameter exploration should solve the case. The techno push use case 3 proved to be difficult but we gained insights on the distillation of GAN approaches: first we managed to use a loss which is more focused on structures in the images.
The inference code for the simplified/distilled DNN models was ported to be executed in a middle/low range FPGA representative of the devices used on-board small sats. Eventually, the quantization result has shown a minimal F1 score loss around 1% (1.33% at worse) for the use cases 1 and 2 from the distilled models. These results are fairly good and prove that the hard part lies in the distillation process, justifying our methodology and approach. Moreover, our student architecture can be ported on the selected hardware, enabling our pipeline to bring deep learning models on (small) satellites abiding by COTS hardware constraints.
We continue to work, we will keep you informed of our progress...
Thank you to ESA and specially to ESA Ф-Lab (https://blogs.esa.int/philab/) for its support and recommendations.
