Efficient AI4EO OpenSource framework
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.


Neat-EO Quality Analysis

This tutorial allow you in few hours (depending on your hardware),
to see how you can validate Quality of a Dataset, easily and at scale.

Check Neat-EO.pink installation and GPU

neo info

Configuration file

echo '''

### Inputs
  name   = "images"
  bands = [1, 2, 3]

### Outputs
  title = "Background"
  color = "transparent"

  title = "Building"
  color = "deeppink"

### AI stuff
  nn = "Albunet"
  encoder = "resnet50"
  loader = "SemSeg"

  bs = 4
  da = {name="RGB", p=0.5}
  loss = "Lovasz"
  optimizer = {name="Adam", lr=0.000025}
  metrics = ["QoD"]

''' > config.toml

export NEO_CONFIG=config.toml

Define our area of interest

neo cover --bbox 4.795,45.628,4.935,45.853 --zoom 18 --out gl/cover.csv

Retrieve Imagery

neo download --rate 20 --type WMS --url "https://download.data.grandlyon.com/wms/grandlyon?SERVICE=WMS&REQUEST=GetMap&VERSION=1.3.0&LAYERS=Ortho2015_vue_ensemble_16cm_CC46&WIDTH=512&HEIGHT=512&CRS=EPSG:3857&BBOX={xmin},{ymin},{xmax},{ymax}&FORMAT=image/jpeg" --cover gl/cover.csv --out gl/images

Retrieve and generate labels

wget -nc -O gl/lyon_roofprint.json "https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&REQUEST=GetFeature&TYPENAME=ms:fpc_fond_plan_communaut.fpctoit&VERSION=1.1.0&srsName=EPSG:4326&outputFormat=application/json; subtype=geojson"

neo rasterize --type Building --geojson gl/lyon_roofprint.json --cover gl/cover.csv --out gl/labels

Train a model, check metrics and perform prediction

neo train --dataset gl --epochs 3 --out gl/model
neo eval --checkpoint gl/model/checkpoint-00003.pth --dataset gl
neo predict --metatiles --checkpoint gl/model/checkpoint-00003.pth --dataset gl --out gl/masks
neo cover --dir gl/masks --out gl/cover.csv

Quality Analysis compare between model prediction and so-called GroundTruth

neo compare --cover cover.csv --mode stack --images gl/images gl/labels gl/masks --out gl/compare
neo compare --cover cover.csv --mode list --labels gl/labels --masks gl/masks --max Building QoD 0.80 --out gl/outliers.cover.csv
neo cover --cover gl/outliers.cover.csv --type geojson --out gl/compare/tiles.json
  • Pink pixels: predicted by model
  • Green pixels: labels
  • Grey pixels: both model and labels agree
  • Pink squares: significant differences between both

Manually Filter training DataSet by selecting unwanted outliers

neo compare --cover gl/outliers.cover.csv --mode side --images gl/images gl/compare --out gl/compare_outliers

It would take about an hour to a human, to do so.
Here a possible result:

wget -O gl/remove.cover.csv https://datapink.net/neo/qa/remove.cover.csv

Then filter our dataset ro remove the unwanted outliers.

neo subset --dir gl/images --cover gl/cover.csv --out gl/filter/images
neo subset --dir gl/labels --cover gl/cover.csv --out gl/filter/labels
neo subset --dir gl/filter/images --delete --cover gl/remove.cover.csv  
neo subset --dir gl/filter/labels --delete --cover gl/remove.cover.csv  

Relaunch a train on a filtered DataSet and display QA results

neo train --dataset gl/filter --epochs 25 --out gl/filter/model
neo eval --checkpoint gl/filter/model/checkpoint-00025.pth --dataset gl/filter
neo predict --metatiles --checkpoint gl/filter/model/checkpoint-00025.pth --dataset gl --out gl/filter/masks
neo compare --cover gl/cover.csv --mode stack --images gl/images gl/labels gl/filter/masks --out gl/filter/compare
neo compare --cover gl/cover.csv --mode list --labels gl/labels --masks gl/filter/masks --max Building QoD 0.80 --geojson --out gl/filter/compare/tiles.json