×
 

CORTEX

Deep Learning simplification for on-board processing

Objectives

The main objective of the project is to define a workflow easing the integration and reduction of complex Deep Neural Networks (DNN) models on SoC (Sytem-on-Chip) FPGA platforms.

Context

Image processing on-board (HR & VHR & Hyperspectral)

This workflow will be demonstrated on a Deep Learning (DL) image processing pipeline devoted to feature extraction in Earth Observation images on-board of small satellites.

This pipeline will be as generic as possible to be able to update, in flight, the performed recognition tasks (clouds, floods, planes, ships or more generic objects identification) with new models required by new applications or users (replace ships detection by oil spills detection, or even fires).

The main challenge of this activity is to define the most suitable combination of methods for DL networks simplification (pruning, compression, …) allowing to execute efficient but complex networks in on-board hardware resources (spatialized or not, including FPGA). These methods are generic enough to be applied to networks or aggregate of networks with different types of architectures. For example:

As a summary, the main objectives of the project are:
Reduce DNN free parameters to fit in existing devices suitable for cubesat platforms, with minimal performance loss & high throughput
Propose and benchmark generic methodologies for DNN simplification
some applications
surveillance
to identify ships, planes or vehicules for sites monitoring
agriculture
in agriculture, to analyse cultures and soil evolution
forest fire
to identify and fight forest fires
icebergs
to identify evolution of glaciers and movement of drifting icebergs
other
for any other application
acknowledgements
This work is funded by a contract in the framework of the EO SCIENCE FOR SOCIETY PERMANENTLY OPEN CALL FOR PROPOSALS EOEP-5 BLOCK 4 issued by the European Space Agency.
Results

The main objective of the project is to define a workflow easing the integration and reduction of complex deep neural networks (DNN) models on Soc FPGA platforms.

Result
14
Dec
2020

SUMMARY of RESULTS

The main objective of CORTEX is to define a workflow easing the integration and reduction of complex deep neural networks (DNN) models on SoC FPGA devices embarked in spatial platforms. This workflow has been demonstrated on a deep learning (DL) image processing pipeline devoted to feature extraction and tested on an FPGA representative of the on-board hardware. In order to define a generic pipeline, three use cases have been selected: 1) ships detection in Sentinel-2 images with a DNN trained with transfer learning technics, 2) oil spills and ocean features detection in Sentinel-1 images, two separate models are developed to demonstrate the flexibility offered by easy processing updating on-board, 3) Sentinel-1 to Sentinel-2 transformation applied on specific regions (refugees camps), demonstrative use case for future applications based on generative adversarial networks. Deep Learning models have been implemented and delivered as well as generated/modified data bases and training software. The networks trained for use cases 1 and 2 showed very good performances (F1-score over 80%) compared to available public results. The results obtained for use case 3 were encouraging even so they would need a more extended study. The distillation process showed to be fairly robust and provided low performance loss with a drastic reduction in the number of parameters of factor 52 on the use cases 1 and 2. A small performance drop was observed for the oil spill case, but we believe that more parameter exploration should solve the case. The techno push use case 3 proved to be difficult but we gained insights on the distillation of GAN approaches: first we managed to use a loss which is more focused on structures in the images. The inference code for the simplified/distilled DNN models was ported to be executed in a middle/low range FPGA representative of the devices used on-board small sats. Eventually, the quantization result has shown a minimal F1 score loss around 1% (1.33% at worse) for the use cases 1 and 2 from the distilled models. These results are fairly good and prove that the hard part lies in the distillation process, justifying our methodology and approach. Moreover, our student architecture can be ported on the selected hardware, enabling our pipeline to bring deep learning models on (small) satellites abiding by COTS hardware constraints. We continue to work, we will keep you informed of our progress... Thank you to ESA and specially to ESA Ф-Lab (https://blogs.esa.int/philab/) for its support and recommendations.

Publication
4
Nov
2020

Ф-WEEK 2020: CORTEX Presentation

CORTEX team participated to ESA EO Ф-WEEK 2020 organised in a virtual form this year. Our presentation shows CORTEX results in DNN training and simplification for several use-cases addresed with classification and GAN architectures. Our presentation is available in the ESA Open Science group: Even if the phi week is over now, we are always at your disposal to answer your questions.

CORTEX
Publication
3
Nov
2020

Ф-WEEK: CORTEX Open datasets

CORTEX team participated to ESA EO Ф-WEEK 2020 organised by ESA-EOP in a virtual form this year. Our presentation shows the databases generated and published and the results on DNN training for these and other use-cases addresed by classification and GAN architectures. Our presentation is available in the ESA Open Science group. Even if the phi week is over now, We are always at your disposal to answer your questions.

Open training databases
See all results
Contact
  • Contractor:
  • Agenium Space
  • 1 avenue de l'Europe, Bâtiment 1
  • 31400 Toulouse