Quality and splits

Checking label quality and organising data for honest model evaluation

1. Motivation¶

A dataset that looks fine at a glance can still have systematic problems that quietly limit model performance, incomplete labels, misaligned masks, or a class imbalance so severe the model learns to ignore the class you actually care about. Separately, how you split your data into training, validation, and test sets can make a model look far better than it actually is, if you are not careful about spatial separation.

2. Core ideas¶

Data-quality problems in GeoAI training sets tend to fall into a recognizable set of categories. A weak point anywhere in this chain can affect model performance.

source imagery quality → label alignment → label completeness → class balance → tile boundaries → image-mask pairing → split strategy → documentation

In general machine learning, training data fit model parameters, validation data support model selection and tuning, and test data provide a final held-out evaluation. In spatial projects, this logic still applies, but the split must also account for spatial dependence. Geospatial train/validation/test splits need spatial separation, not just random shuffling, to give an honest estimate of performance.

3. Quality workflow¶

A. Check label quality¶

Issue	Why it matters	First check
Incomplete annotations	The model learns that missing objects are background.	Inspect several tiles and look for visible but unlabelled objects.
Spatial misalignment	The model learns shifted boundaries.	Overlay labels on imagery and zoom in.
Class imbalance	The model may ignore rare classes.	Count pixels, objects or chips per class.
Edge effects	Objects cut by tile boundaries can confuse training.	Inspect chips near tile edges.
Outdated labels	The model learns from objects that changed or disappeared.	Compare imagery date and label source date.
Ambiguous classes	The model receives inconsistent examples.	Write clear class definitions and inspect edge cases.

B. Check class balance¶

Many spatial datasets are imbalanced. Background pixels may greatly outnumber building pixels. Common classes may dominate rare classes.

Accuracy can be misleading in this situation. Later lessons will introduce task-specific metrics, but training-data preparation already needs basic class awareness.

Useful checks include:

number of chips with and without target features,
number of objects per class,
approximate pixel counts per class,
geographic distribution of rare classes,
whether rare classes appear in training, validation and test areas.

Possible responses include:

keeping some empty tiles but not too many,
oversampling rare classes,
using class weights or focal loss in later training,
reducing the number of classes,
collecting more labels for underrepresented classes.

C. Check edge effects¶

Objects at chip edges may be incomplete. This matters for buildings, roads, fields, vehicles and other structured features.

Overlap helps because the same object may appear fully in another chip. However, overlap increases the number of chips and can create strong similarity between neighbouring examples.

Use overlap deliberately:

more overlap
→ fewer cut objects
→ more files
→ more spatial similarity between chips

For evaluation, do not let overlapping chips from the same area appear across training and validation sets.

D. Organise the dataset¶

For segmentation-style datasets, a common structure is:

dataset/
  images/
    tile_000000.tif
    tile_000001.tif
  masks/
    tile_000000.tif
    tile_000001.tif

For separate splits:

dataset/
  train/
    images/
    masks/
  val/
    images/
    masks/
  test/
    images/
    masks/

For detection datasets, the layout may follow formats such as YOLO:

dataset/
  images/
    train/
    val/
  labels/
    train/
    val/
  data.yaml

The main principle is simple: image and label files must be paired unambiguously.

E. Split spatially¶

In ordinary machine learning examples, data are often randomly split into training and validation sets. For spatial data, this can be risky.

Nearby chips are often similar because they share the same landscape, lighting, sensor, season and annotation style. Randomly splitting neighbouring chips can create Data Leakage: the validation set may be too similar to the training set.

Better options include:

Split strategy	Use when
Region-based split	You have several geographic areas.
Scene-based split	You have multiple satellite or aerial scenes.
Spatial buffer	You must split within one scene but can leave distance between areas.
Time-based split	You want to test generalisation to another date.

Common split ratios such as 70/15/15 or 80/10/10 are starting points, not rules. The spatial logic matters more than the exact ratio.

4. Python reactivation¶

You can quickly inspect generated folders with Pathlib.

from pathlib import Path

output_dir = Path("output")
image_tiles = list(output_dir.glob("images/*.tif"))
mask_tiles = list(output_dir.glob("labels/*.tif"))

print("Image tiles:", len(image_tiles))
print("Mask tiles:", len(mask_tiles))

Image tiles: 18
Mask tiles: 18

Matching counts are not enough, but they are a useful first check.

You can also store split logic explicitly:

split_plan = {
    "strategy": "region-based",
    "train": "western part of study area",
    "validation": "central part with spatial buffer",
    "test": "eastern part of study area",
    "reason": "reduce leakage between neighbouring chips",
}

split_plan

{'strategy': 'region-based',
 'train': 'western part of study area',
 'validation': 'central part with spatial buffer',
 'test': 'eastern part of study area',
 'reason': 'reduce leakage between neighbouring chips'}

5. Common pitfalls¶

Splitting after tiling, at random. As above, this is the single most common source of an artificially optimistic validation score in student projects.
Not tracking class balance until after training already looks disappointing. Check it as part of your quality-control pass, before you invest time in training.
Using the test set to make any decisions before the final evaluation. Even a single peek to “just check” quietly compromises how honest that final number really is.
Inconsistent file naming between images and labels. Decide on a convention early and stick to it across your whole pipeline.

6. Mini task¶

For your own project’s tiles (or the sample tiles from earlier pages), describe how you would split them into training, validation, and test sets while avoiding spatial leakage, and name the split ratio you would use.

Sample solution

For a citywide building-footprint project covering four distinct neighborhoods: assign two neighborhoods to training, one to validation, and one to test, rather than mixing tiles from all four neighborhoods and splitting randomly. This keeps roughly a 50/25/25 split by area, close enough to common ratios, while guaranteeing that no validation or test tile is spatially adjacent to a training tile.

7. Key takeaways¶

Check systematically for incomplete labels, misalignment, class imbalance, and edge effects before trusting a generated dataset.
Random, tile-level splits usually leak information because of spatial autocorrelation between nearby tiles.
Split at the region or scene level, with a spatial buffer if splitting within a single scene.
The test set should stay untouched until your final evaluation; using it earlier undermines the honesty of that final check.
Consistent file naming between images and labels is necessary for data loaders to pair them correctly.

1. Motivation¶

2. Core ideas¶

3. Quality workflow¶

A. Check label quality¶

B. Check class balance¶

C. Check edge effects¶

D. Organise the dataset¶

E. Split spatially¶

4. Python reactivation¶

5. Common pitfalls¶

6. Mini task¶

7. Key takeaways¶

Further reading¶