Compare and reuse - SDS320 Spatial Data Analytics

Comparing classifiers fairly and sharing a trained model

1. Motivation¶

“Model A scored higher than Model B” is only a meaningful statement if everything else was held constant. Otherwise, you cannot tell whether the architecture caused the difference, or something else did, a different random split, a different number of epochs, or a different batch size.

2. Core idea¶

A fair architecture comparison changes exactly one thing, the architecture, and holds the data split, number of epochs, batch size, image size, and random seed identical across runs.

3. Comparison workflow¶

Previous code

This page assumes that result is the trained ResNet-50 output from the previous page.

# Import training functions and data

from geoai.recognize import (
    train_image_classifier,
    evaluate_classifier,
    predict_images,
    plot_training_history,
    plot_confusion_matrix,
    plot_predictions,
)
from geoai.utils import download_file
import os
import random
from pathlib import Path
import matplotlib.pyplot as plt

DATA_DIR = Path("data/raw")
DATA_DIR.mkdir(parents=True, exist_ok=True)

url = "https://data.source.coop/giuz/sds320/L05/data/EuroSAT-RGB.zip"
data_dir = download_file(url, output_path=str(DATA_DIR / Path(url).name))

# Train a ResNet-50 classifier

result = train_image_classifier(
    data_dir=data_dir,
    model_name="resnet50",
    num_epochs=5,
    batch_size=32,
    learning_rate=1e-3,
    image_size=64,
    in_channels=3,
    pretrained=True,
    output_dir="image_recognition_output/resnet50",
    num_workers=4,
    seed=42,
)

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
GPU available: True (mps), used: True
TPU available: False, using: 0 TPU cores
💡 Tip: For seamless cloud logging and experiment tracking, try installing [litlogger](https://pypi.org/project/litlogger/) to enable LitLogger, which logs metrics and artifacts automatically to the Lightning Experiments platform.

Metric val_acc improved. New best score: 0.942
Epoch 0, global step 540: 'val_acc' reached 0.94236 (best 0.94236), saving model to '/Users/wulf/Documents/Documents_work/SDS/02_teaching/01_courses/SDS320/26HS_SDS320/06_jupyter_book/sds320-jb/book/04_lessons/L05/image_recognition_output/resnet50/models/resnet50_epoch=00_val_acc=0.9424.ckpt' as top 1

Metric val_acc improved by 0.012 >= min_delta = 0.0. New best score: 0.955
Epoch 1, global step 1080: 'val_acc' reached 0.95486 (best 0.95486), saving model to '/Users/wulf/Documents/Documents_work/SDS/02_teaching/01_courses/SDS320/26HS_SDS320/06_jupyter_book/sds320-jb/book/04_lessons/L05/image_recognition_output/resnet50/models/resnet50_epoch=01_val_acc=0.9549.ckpt' as top 1

Epoch 2, global step 1620: 'val_acc' was not in top 1

Epoch 3, global step 2160: 'val_acc' was not in top 1

Metric val_acc improved by 0.008 >= min_delta = 0.0. New best score: 0.962
Epoch 4, global step 2700: 'val_acc' reached 0.96250 (best 0.96250), saving model to '/Users/wulf/Documents/Documents_work/SDS/02_teaching/01_courses/SDS320/26HS_SDS320/06_jupyter_book/sds320-jb/book/04_lessons/L05/image_recognition_output/resnet50/models/resnet50_epoch=04_val_acc=0.9625.ckpt' as top 1
`Trainer.fit` stopped: `max_epochs=5` reached.

# Evaluate the ResNet-50 on the test set

eval_result = evaluate_classifier(
    model=result["model"],
    dataset=result["test_dataset"],
    class_names=result["class_names"],
)


### A. Train EfficientNet-B0

EfficientNet-B0 is a lighter architecture than ResNet-50. Use the same settings as the ResNet-50 run and change only `model_name` and `output_dir`.

# Training a classifier takes time. The code took 15 minutes to run on my laptop.

result_effnet = train_image_classifier(
    data_dir=data_dir,
    model_name="efficientnet_b0",
    num_epochs=5,
    batch_size=32,
    learning_rate=1e-3,
    image_size=64,
    in_channels=3,
    pretrained=True,
    output_dir="image_recognition_output/efficientnet_b0",
    num_workers=4,
    seed=42,
)

GPU available: True (mps), used: True
TPU available: False, using: 0 TPU cores
💡 Tip: For seamless cloud logging and experiment tracking, try installing [litlogger](https://pypi.org/project/litlogger/) to enable LitLogger, which logs metrics and artifacts automatically to the Lightning Experiments platform.

Metric val_acc improved. New best score: 0.938
Epoch 0, global step 540: 'val_acc' reached 0.93843 (best 0.93843), saving model to '/Users/wulf/Documents/Documents_work/SDS/02_teaching/01_courses/SDS320/26HS_SDS320/06_jupyter_book/sds320-jb/book/04_lessons/L05/image_recognition_output/efficientnet_b0/models/efficientnet_b0_epoch=00_val_acc=0.9384.ckpt' as top 1

Metric val_acc improved by 0.016 >= min_delta = 0.0. New best score: 0.955
Epoch 1, global step 1080: 'val_acc' reached 0.95463 (best 0.95463), saving model to '/Users/wulf/Documents/Documents_work/SDS/02_teaching/01_courses/SDS320/26HS_SDS320/06_jupyter_book/sds320-jb/book/04_lessons/L05/image_recognition_output/efficientnet_b0/models/efficientnet_b0_epoch=01_val_acc=0.9546.ckpt' as top 1

Epoch 2, global step 1620: 'val_acc' was not in top 1

Metric val_acc improved by 0.008 >= min_delta = 0.0. New best score: 0.962
Epoch 3, global step 2160: 'val_acc' reached 0.96227 (best 0.96227), saving model to '/Users/wulf/Documents/Documents_work/SDS/02_teaching/01_courses/SDS320/26HS_SDS320/06_jupyter_book/sds320-jb/book/04_lessons/L05/image_recognition_output/efficientnet_b0/models/efficientnet_b0_epoch=03_val_acc=0.9623.ckpt' as top 1

Epoch 4, global step 2700: 'val_acc' was not in top 1
`Trainer.fit` stopped: `max_epochs=5` reached.

B. Plot training history¶

# Compare training curves side by side
fig = plot_training_history("image_recognition_output/efficientnet_b0/models")
plt.show()

Comparing this against the ResNet-50 curves from the training page shows how quickly each architecture converges, and whether either shows signs of overfitting.

C. Evaluate EfficientNet-B0¶

eval_effnet = evaluate_classifier(
    model=result_effnet["model"],
    dataset=result_effnet["test_dataset"],
    class_names=result_effnet["class_names"],
)

print(f"ResNet-50 accuracy:       {eval_result['accuracy']:.4f}")
print(f"EfficientNet-B0 accuracy: {eval_effnet['accuracy']:.4f}")

ResNet-50 accuracy:       0.9615
EfficientNet-B0 accuracy: 0.9520

Model	Parameters	Typical EuroSAT accuracy	Relative speed
ResNet-50	~25M	~93–97%	Baseline
EfficientNet-B0	~4M	~93–97%	Faster

Your exact numbers will vary with the random split, training duration, and hardware. For most geospatial classification tasks, the two architectures perform similarly once fine-tuned on a well-curated dataset. A model that scores one or two percentage points lower in accuracy can still be the better practical choice if it is several times smaller, trains faster, or fits on more limited hardware.

D. Plot the normalised confusion matrix¶

fig = plot_confusion_matrix(
    eval_effnet["confusion_matrix"],
    result_effnet["class_names"],
    normalize=True,
)
plt.show()

Compare this matrix with the ResNet-50 matrix. Focus on class confusions, not only the overall score.

E. Compare overall accuracy¶

print(f"ResNet-50 accuracy: {eval_result['accuracy']:.4f}")
print(f"EfficientNet-B0 accuracy: {eval_effnet['accuracy']:.4f}")

ResNet-50 accuracy: 0.9615
EfficientNet-B0 accuracy: 0.9520

A slightly lower accuracy may still be acceptable if the model is smaller, faster or easier to run with your hardware.

F. Record the comparison¶

comparison = [
    {
        "model": "resnet50",
        "accuracy": eval_result["accuracy"],
        "epochs": 5,
        "batch_size": 32,
        "learning_rate": 1e-3,
    },
    {
        "model": "efficientnet_b0",
        "accuracy": eval_effnet["accuracy"],
        "epochs": 5,
        "batch_size": 32,
        "learning_rate": 1e-3,
    },
]

comparison

[{'model': 'resnet50',
  'accuracy': 0.9614814814814815,
  'epochs': 5,
  'batch_size': 32,
  'learning_rate': 0.001},
 {'model': 'efficientnet_b0',
  'accuracy': 0.952037037037037,
  'epochs': 5,
  'batch_size': 32,
  'learning_rate': 0.001}]

This simple record is useful for your project report and repository.

4. Optional: publish and reuse¶

Sharing a trained model lets collaborators run inference immediately, without your original training data or compute. push_classifier_to_hub uploads the best checkpoint’s weights and a small configuration file to a versioned repository; it needs a free Hugging Face account and a write-access token.

This is optional for SDS320. Use it only if model sharing is useful for your project and you are comfortable managing access tokens.

from huggingface_hub import notebook_login

notebook_login()

from geoai.recognize import push_classifier_to_hub, predict_images_from_hub

repo_url = push_classifier_to_hub(
    model_path=result["checkpoint_path"],
    repo_id="your-username/eurosat-resnet50",
    model_name="resnet50",
    num_classes=len(result["class_names"]),
    in_channels=3,
    class_names=result["class_names"],
    commit_message="EuroSAT ResNet-50 classifier trained for 5 epochs",
)
print(repo_url)

predict_images_from_hub downloads the model back down and runs inference, without needing the local checkpoint at all, which makes it easy to hand a working classifier to anyone with the repository ID.

hub_result = predict_images_from_hub(
    image_paths=test_paths[:10],
    repo_id="your-username/eurosat-resnet50",
    image_size=64,
)

5. Common pitfalls¶

Changing more than one variable between comparison runs. If epochs, batch size, or the random seed differ between two runs, you can no longer attribute a performance difference to architecture alone.
Judging architectures on accuracy alone. Training time, model size, and inference speed often matter as much for a real deployment or a compute-limited student project.
Treating Hugging Face Hub sharing as required. It is a convenience for collaboration and reuse, not a core part of the SDS320 workflow.

6. Mini task¶

Sketch a fair comparison plan for two architectures on your own project: which single variable would you change, and which five or six variables would you deliberately hold fixed?

Sample solution

Variable changed: model_name (resnet50 vs. efficientnet_b0). Held fixed: data directory and split, num_epochs, batch_size, learning_rate, image_size, in_channels, and seed. Documenting this list explicitly, before running either experiment, makes it much easier to trust and explain the comparison later in a project report.

7. Further reading¶

8. Key takeaways¶

A fair comparison changes only the architecture and holds every other setting, including the random seed, identical.
ResNet-50 and EfficientNet-B0 often perform similarly on a well-curated dataset like EuroSAT; the practical difference is usually speed and size, not accuracy.
Judge architecture trade-offs on the full picture: accuracy, training time, model size, and inference speed.
Publishing a model to a hub like Hugging Face is a useful, optional way to share or reuse a trained classifier, not a required course workflow.