On this page

Finishing image annotation does not mean a marine segmentation model is ready to train. The masks still need validation, acquisition groups must be preserved, classes must be mapped consistently, and the experiment needs an evaluation set that represents deployment.

This article connects Getting Started with Marine Image Annotation Using Label Studio and SAM to a reproducible model-development workflow, focusing on the data contracts and checks required for training and evaluation.

What Segmentation Learns

Semantic segmentation assigns a class to each pixel. Instance segmentation also separates individual objects of the same class. Marine applications include fish measurement, benthic habitat mapping, organism counts, marine-debris surveys, and inspection of submerged infrastructure.

The annotation type must match the output. A semantic mask that marks every fish pixel can train a fish-versus-background model, but it does not identify where one overlapping fish ends and another begins. Instance masks preserve that separation.

An underwater fish image with a segmentation mask overlaid for marine image annotation

Use a Public Dataset as a Contract Example

The DeepFish computer vision dataset paper describes a fisheries dataset for fish instance segmentation, classification, and size estimation. The published dataset contains 1,320 JPEG images, 1,291 annotated images, 7,339 specimens, and 59 species represented as 60 labels, collected over six months. Its annotations include pixel-wise instance masks and specimen information. The data are linked from the paper through Zenodo.

This makes DeepFish useful for discussing dataset design: images, instances, classes, size reference, and collection dates must stay connected. Before downloading or redistributing any public dataset, verify the current repository record, license, citation request, and any restrictions for the intended use.

A market-tray dataset and an underwater ROV dataset are different domains. Success on one does not demonstrate performance on the other. Background, lighting, camera distance, occlusion, turbidity, and target pose all change the problem.

Freeze the Dataset Contract Before Training

Create a machine-readable manifest that links each image to its labels and acquisition context:

image_id
image_path
mask_path or annotation_path
class_map_version
site_id / trip_id / dive_id / batch_id
observed_at
annotation_version
split

The manifest should enforce:

  • unique, stable image IDs;
  • masks that match image dimensions;
  • known class IDs only;
  • instance IDs that do not collide within an image;
  • acquisition groups available for splitting;
  • an explicit annotation and class-map version;
  • no file referenced by more than one partition.

Do not infer all grouping from directory names inside the training loop. Resolve it once, validate it, and keep the manifest with the experiment artifacts.

Audit Annotations Before Training

Annotation errors become training signals. Run automated checks first:

  • image and mask can both be decoded;
  • width and height match;
  • labels are integers from the declared class map;
  • each declared instance has pixels;
  • masks do not contain impossible IDs;
  • empty-background images are handled intentionally;
  • polygons or run-length encodings survive export conversion;
  • excluded or uncertain images are not silently treated as background.

Then review a stratified visual sample. Include every class, crowded scenes, image edges, low contrast, small objects, partial targets, and multiple annotators. Overlay masks with transparency and inspect the original at full resolution.

Agreement is not proof of truth, but disagreement can reveal unclear instructions. Revise the annotation guide before scaling instead of letting each annotator invent a boundary rule.

Split by Acquisition Group Before Augmentation

Divide the data into training, validation, and test sets before any step that multiplies it. Assign original images, and images from the same acquisition group, to a partition first, then create crops, tiles, flips, and color transformations. Every derivative of one source image stays in the partition its source went to.

Otherwise nearly identical images land on both the training and the evaluation side, and the measured performance overstates what the model can actually do.

For video, keep frames from one sequence or dive together. For a fixed-camera project, split by deployment period. For tray images, date or batch may capture background and handling conditions. The correct group follows the intended deployment claim.

Data Leakage in Marine Machine Learning: Spatial and Temporal Splits Explained explains grouped, spatial, and temporal splits in detail. Record class and instance counts per group before finalizing the split; otherwise a rare class can disappear from the test set or occur in only one acquisition condition.

Use three conceptual partitions:

  • Training: fit weights and training-only transformations.
  • Validation: choose checkpoints, thresholds, architecture, and hyperparameters.
  • Test: estimate the selected method once; do not use it to choose the method.

Build a Reproducible Training Run

A training run is more than a script. Record:

  • code revision;
  • container image or locked environment;
  • dataset and split-manifest versions;
  • model architecture and initialization source;
  • input resolution and crop policy;
  • augmentation configuration;
  • loss function and class weighting;
  • optimizer, learning-rate schedule, batch size, and epoch limit;
  • random seeds and deterministic settings where practical;
  • hardware and mixed-precision settings;
  • checkpoint-selection rule.

How to Make a Research Analysis Reproducible with Docker covers the environment layer. How to Keep Research Code Reproducible with Git and GitHub covers reviewable code and experiment metadata.

Reproducibility does not mean that every GPU produces bit-identical floating-point output. It means the inputs, rules, environment, and expected tolerance are explicit enough to rerun and investigate differences.

Keep the Data Loader Boring and Testable

The loader should have one clear contract: given a manifest row, return the image tensor, target mask or instances, and identity metadata.

Test it with small known cases:

  • one foreground object;
  • multiple touching instances;
  • a background-only image;
  • a mask with the maximum class ID;
  • an image whose dimensions require padding;
  • a sample that an augmentation moves near the border.

Apply geometric augmentation to the image and mask together with nearest-neighbor interpolation for categorical masks. Photometric augmentation changes the image but not the mask. A resized class mask must not acquire fractional class IDs.

Normalize using documented values. If normalization statistics are estimated from the dataset, calculate them from training images only.

Choose Metrics That Match the Task

A five-stage segmentation workflow leads from versioned images and masks through group splitting and training to untouched-test evaluation with IoU, Dice, and visual review

For a class mask, intersection over union is:

IoU = true-positive pixels / (true-positive + false-positive + false-negative pixels)

Dice is:

Dice = 2 × true-positive pixels / (2 × true-positive + false-positive + false-negative pixels)

Every reported segmentation score must name the dataset version, split, class handling, averaging method, and treatment of absent classes.

For semantic segmentation, report per-class IoU and a clearly defined macro or frequency-weighted aggregate. Pixel accuracy alone can look high when background dominates.

For instance segmentation, use an instance-aware evaluation appropriate to the output representation and deployment question. A mask can have good pixel overlap while merging two fish into one instance, which matters for counts and measurements.

Handle Empty and Rare Classes Explicitly

Suppose no pixel of class coral appears in a test image. Some metric implementations ignore the class, some return an undefined value, and some treat an empty prediction as a perfect match. Declare the rule before aggregation.

Also report the number of evaluated images and instances per class. A mean based on two rare specimens is not comparable to one based on thousands.

If a class occurs in only one site or batch, class performance and acquisition condition are confounded. Collect more data or narrow the claim instead of repairing the table with a random split.

Visual Error Review Is Part of Evaluation

Aggregate metrics do not tell you why a model fails. Create a review sheet containing:

  • original image;
  • ground-truth mask;
  • predicted mask and confidence;
  • false-positive and false-negative overlay;
  • image ID, group, and acquisition metadata;
  • per-image metric only as a sorting aid.

Inspect the best, median, and worst cases as well as random cases. Stratify by class, site, visibility, depth, camera, object size, and crowding where metadata permit.

Look for systematic failures:

  • fins or thin structures omitted;
  • neighboring individuals merged;
  • one individual fragmented;
  • shadows, rulers, vehicle parts, or debris predicted as targets;
  • small or partly visible objects missed;
  • masks following annotation style rather than object boundary;
  • performance collapse in an unseen site or acquisition batch.

Review can reveal that the label policy, not the architecture, is the limiting factor.

Evaluate Size and Count Outputs Separately

If segmentation feeds another measurement, validate that downstream task directly. Good IoU does not guarantee accurate fish length, area, biomass proxy, or count.

For length estimation, state calibration method and compare predicted versus reference length with suitable error summaries. For counts, report merges, splits, missed instances, and duplicate detections. Keep the segmentation metric alongside the downstream error so tradeoffs remain visible.

Package the Evidence

For each candidate release, retain:

  • dataset and annotation version;
  • split manifest and group summary;
  • training configuration and logs;
  • selected checkpoint hash;
  • evaluation code revision;
  • machine-readable metrics by class and group;
  • representative visual review sheet;
  • known limitations and intended deployment boundary.

Do not retain only the “best model” file. Without its data and evaluation context, the file cannot support a scientific claim or a safe review workflow.

A Minimum Credible Experiment

For a first segmentation study:

  1. Define one narrow class set and boundary policy.
  2. Validate masks automatically and visually.
  3. Reserve groups that represent the intended unseen condition.
  4. Train one documented baseline before extensive tuning.
  5. Select with validation data only.
  6. Evaluate once on the frozen test set.
  7. Report class-level metrics and group variation.
  8. Review representative failures and update the limitation statement.

That baseline is more informative than a large architecture search built on a leaking split.

If running this training and evaluation workflow inside your own team would be difficult, the inquiry link below can also be used to ask about support for model training and evaluation design.

Contact

Implementation support and annotation services

Ask about setup, design, and implementation support, or annotation services.

Go to the contact form

Frequently Asked Questions

How many annotated images are enough?

There is no universal count. Diversity across classes, sites, conditions, and acquisition groups often matters more than raw image count. Start with a learning curve and error review.

Should SAM-generated masks be treated as ground truth?

No. They are proposals that require review against a written annotation policy. Retain review status and corrections.

Is mean IoU enough?

No. Include per-class results, group variation, empty-class handling, and visual failures. Add downstream measurement error if masks support counts or size estimates.

Can I compare my score directly with a paper?

Only when dataset version, split, preprocessing, class mapping, and metric implementation match. Otherwise the numbers answer different questions.

References and Next Steps