Results of Cortex extension project
The CORTEX CCN is an extension of the CORTEX project capitalizing on the developments in Artificial Intelligence (AI) performed in the previous phase. In the previous phase AGENIUM Space developed a complete pipeline for DNN simplification and AI-based image analysis on board the satellites. This extension tackles key operational elements of TinyML. It specifically attempts to:
1) improve semi-automatic sample synthesis with generative AI models, in particular for hyperspectral use-cases,
2) incorporate some interpretability in the Deep Learning (DL) models through trustworthy AI, and
3) allow transfer DL models between various sensors.
First objective listed above was implemented through the exploration of new ways to generate synthetic images with associated labels for hyperspectral use cases. Such a method is interesting in the case where too few annotated data are available to train a deep neural network. Then, it is interesting to train a generative model, for example using a set of unlabeled data and a small amount of labeled data. The context of hyperspectral images is particularly suited for this problem since labeled datasets of hyperspectral images are scarce and generally of very small size. First step of the project consists in defining an hyperspectral dataset along with associated ground truth in order to correctly train the models. We have constituted a segmentation dataset based on PRISMA products and IGN BD Forest V2 (PRISMA HSI Forest dataset, https://zenodo.org/record/7230134).
To correctly match the ground truth and the images, an important work was done about the improvement of the geolocalization of the PRISMA images. The small patches of PRISMA images were coregistered with Sentinel-2 images (with very accurate geolocation) to have a better estimation of the position of the patches. Then, a segmentation model was trained on the dataset to assess its quality and the feasibility of the task of forest-type segmentation. Good results were obtained using a Unet-EfficientNet segmentation network. It showed that the dataset is globally coherent in terms of association between image and ground truth.
Finally, an important research work was conducted to develop a Generative Adversarial Network (GAN) method able to generate synthetic hyperspectral images. The final goal was to generate synthetic ground truth masks alongside the images and the method SemanticGAN was elected to address this problem.
The study showed good results regarding the generation of HS images up to a certain number of bands. Regarding the generation of masks, the initial expectation was that it would help stabilizing the generation of images, but the experiments showed the contrary. More research will be necessary to obtain couples of images and masks that could be used to train a DNN. Overall, the increase of the spectral dimension is a key difficulty of the problem.
Second objective of the CORTEX project was to investigate the possibility to associate a confidence score to the predictions of a Deep Neural Network (DNN) in Earth Observation (EO) scenario. Most of the DNNs are designed to predict a class, a segmentation map or detections, no matter it is interpolation or extrapolation. Then, a confidence score answers to the need of having interpretable outputs and it could help an AI4EO end-user to take a decision. We have investigated one method: the confidNet approach on two use cases, one based on S2 tiles containing ships or not (Ship S2 AIS dataset, https://zenodo.org/record/7229756) and the other one is classification of 10 geophysical phenomena from Sentinel-1 wave mode. The main results obtained in this study are the relevance to utilize the confidNet approach in AI4EO scenarios, the possibility to reduce the network in an on-boarding interest, and a first warranty that the confidNet approach can learn in a different way from classification networks, with interesting properties of generalization. Also, an important work has been done to produce and publish the ARD database of the first usecase. It is a good outcome of this study, and a good contribution to open science.
Finally, we've also addressed the topic of transferring DL models through the on boarding of a DL Model on the Unibap/ION mission. We' ve sent on board the ION satellite a forest and clouds segmentation DNN trained on known S2 images (that were uploaded on board as well). This experiment was very conclusive and allowed us to prove that we were CPU flight proven.
As a conclusion the work performed on CORTEX project has allowed us to investigate innovative AI methods and as a consequence has enlarge our competencies in the domain. This kind of exploratory activities are an essential part of Agenium Space AI solution development.
