On this page
A marine machine-learning model can score well on a test set and fail on the next survey. One common reason is data leakage: the model was trained with information about the test data, or with fields that will not exist at deployment time, so the test score reflects something the next survey cannot reproduce.
Leakage is especially easy to create in ocean and marine datasets. Consecutive video frames look alike. Profiles from one float are related in space and time. Images from the same camera share lighting, altitude, and background. Nearby grid cells are not independent samples.
Splitting samples at random ignores those relationships. This article shows how to design an evaluation split around the scientific deployment question instead.
What Data Leakage Means
Data leakage occurs when model development uses information that crosses the intended boundary between training and evaluation. It can enter through the split itself, feature construction, preprocessing, labels, or repeated tuning on the test set.
Two forms are useful to distinguish:
- Direct leakage: the same observation, crop, duplicate, or target-derived field appears on both sides.
- Dependence leakage: samples are technically different but so closely related that the test set does not represent the promised generalization, such as adjacent frames from one video.
The second form is the subtle one in marine work. A random split may be implemented correctly in software and still answer the wrong scientific question.
Why Marine Samples Are Often Related
Underwater imagery
An ROV or towed camera records a sequence. Adjacent frames can contain the same animal, seafloor patch, color cast, and vehicle shadow. Extracting frames every second and randomly splitting them can place near-duplicates in training and test sets.
Plankton and microscopy images
Images from one bottle, tow, plate, instrument run, or preparation batch share acquisition conditions. A model may recognize the batch background instead of the organism.
Acoustic surveys
Neighboring pings and transects share vessel, instrument, weather, and water-mass conditions. Random segments may leak cruise-specific signatures.
Profiles and gridded ocean data
Cycles from one Argo float form a trajectory through connected water masses. Adjacent spatial cells and dates are autocorrelated. Randomly holding out individual samples can test interpolation among familiar conditions rather than performance at an unseen place or future time.
Research on spatial prediction has shown that conventional random cross-validation can be optimistic when spatial dependence is ignored. The exact bias depends on the data and objective, so spatial blocking is not a ritual; it is a way to align evaluation distance with the claimed use. See the open Nature Communications study on random and spatial cross-validation for a detailed analysis.
Match the Split to the Claim
The three rows answer different questions:
| Split | Example grouping | Question it can answer |
|---|---|---|
| Random sample split | Individual independent specimens | Can the model handle another sample from the same population and acquisition process? |
| Group holdout | Site, cruise, transect, dive, float, video, tow, or batch | Can it generalize to an unseen group of this type? |
| Temporal holdout | Earlier data for training, later data for testing | Can it generalize to future observations? |
Random splitting is not always wrong. It is appropriate only when samples are sufficiently independent and deployment will draw from the same acquisition process. The burden is to justify independence, not to use a more complicated splitter by default.
Define the Unit of Generalization First
Write one sentence before touching the code:
The model will be used on ______ that were not represented during training.
Possible completions include:
- later frames from the same fixed camera;
- a new dive using the same camera system;
- a new reef site;
- a future season;
- a new vessel or instrument;
- floats or regions not present in training.
The blank determines the group. If the claim is “new dive,” keep every frame from a dive on one side. If the claim is “new site,” grouping only by video is too weak because several videos from the same site can still cross the boundary.
Sometimes two axes matter. A credible final test might reserve both a later period and unseen sites. That evaluation is harder and usually yields fewer independent groups, but it matches a stronger claim.
Build a Split Manifest, Not Just Arrays of Indices
A split should be a versioned research artifact. Store one row per sample with fields such as:
sample_id, source_id, site_id, deployment_id, observed_at, label_version, splitThe exact identifiers depend on the dataset. The important properties are:
- sample IDs are stable;
- every group used for separation is explicit;
- the split is generated once for an experiment family;
- code checks that no protected group crosses partitions;
- the manifest is reviewed and committed or stored with the run artifacts.
This makes leakage auditable. It also prevents two training scripts from quietly producing different “test sets” from the same random seed convention.
How to Keep Research Code Reproducible with Git and GitHub explains how to record code and small metadata artifacts without pretending that Git alone versions every large dataset.
Grouped Splitting in Practice
Scikit-learn’s GroupKFold keeps a group from appearing in both train and test for a fold. For a marine image dataset, a group might be a dive:
from sklearn.model_selection import GroupKFold
splitter = GroupKFold(n_splits=5)
for train_index, validation_index in splitter.split(
samples,
labels,
groups=dive_ids,
):
train_model(samples[train_index], labels[train_index])
evaluate(samples[validation_index], labels[validation_index])The syntax is the easy part. dive_ids must actually represent the intended independence boundary. If the same site appears in several dives and deployment is to a new site, use site IDs or a combined site-period group instead.
For class-imbalanced data, examine class counts by group before splitting. A rare class present at only one site cannot support both an unseen-site test and a reliable class-specific estimate. That is a data-coverage limitation, not a reason to leak the site across partitions.
Temporal Splitting in Practice
For forecasting or future-deployment claims, sort by time and reserve the latest period. Scikit-learn’s TimeSeriesSplit maintains time order and provides a gap between train and test.
A gap is useful when observations near the boundary remain highly related. It does not solve seasonality or long-lived group overlap by itself. A float, site, or animal can still appear on both sides, so temporal and grouped rules may need to be combined in a custom manifest.
Do not choose the cutoff only after seeing which date produces the best score. Set it from the deployment scenario or a predeclared experimental plan.
Spatial Blocking Without False Precision
Spatial splitting can hold out geographic cells, buffered regions, sites, or entire survey areas. The block size should be related to:
- sampling footprint and positional uncertainty;
- expected spatial autocorrelation;
- distance between deployment locations;
- number and distribution of independent groups;
- the scale of the scientific claim.
There is no universal “correct” number of kilometres. Try defensible block sizes as sensitivity analyses, report them, and avoid claiming performance beyond the distances evaluated.
For a drifting platform, a simple latitude-longitude grid may cut one trajectory into artificial groups. Platform, mission, time window, ocean region, or combinations of these may better represent how the model will be used.
Leakage Can Happen After the Split
A correct manifest does not protect the rest of the pipeline automatically.
Fit preprocessing on training data only
Compute normalization statistics, imputers, feature selection, dimensionality reduction, and learned color correction from training data. Apply the fitted transformation unchanged to validation and test data.
Split before augmentation and patch extraction
Do not create crops, rotations, or overlapping tiles and then split the derived files. Split source images or acquisition groups first; generate augmentations only inside the training partition.
Detect duplicates before finalizing the split
File hashes catch exact duplicates. Perceptual hashes or embeddings can help find resized, recompressed, or nearly identical frames, but candidates still need review.
Keep target information out of features
A post-survey species code, reviewer decision, or filename constructed from the label can reveal the answer. Audit metadata and paths, not only pixel tensors.
Do not tune on the final test set
Repeatedly viewing test performance and changing the model turns the test set into validation data. Keep a final partition frozen until the method is selected.
Report Results at the Right Level
Thousands of frames from ten dives do not provide thousands of independent deployment tests. Report both sample-level metrics and variation across meaningful groups.
Useful reporting includes:
- number of sites, cruises, dives, videos, or floats in each partition;
- date and spatial coverage;
- class counts by group;
- exact grouping and buffer rules;
- aggregate metric plus group-level distribution or confidence interval;
- performance on important conditions such as turbidity, depth, or camera type;
- a statement of what the split does not test.
A model evaluated on unseen dives from familiar sites should not be described as validated for unseen regions.
Image Annotation Decisions Affect Leakage
Leakage prevention begins before model training. Annotation projects should retain source-image, video, dive, site, and acquisition metadata. Exporting only image files and masks can destroy the grouping information needed later.
Getting Started with Marine Image Annotation Using Label Studio and SAM explains how labels and segmentation masks are created. Add stable acquisition identifiers before the project grows; trying to reconstruct them from filenames later is unreliable.
A Practical Review Checklist
Before accepting a marine ML evaluation, review the points below. They are not the only valid procedure; they are prompts for checking whether the split matches your own deployment question. If an item does not apply, be ready to explain why.
- The intended unseen unit is written in one sentence.
- Original acquisition groups are known before splitting.
- Near-duplicates and derived crops cannot cross partitions.
- Training-only transformations are fitted without validation or test data.
- The final test set is frozen before model selection.
- Group, spatial, and date coverage are reported.
- Metrics are summarized across meaningful independent groups.
- The stated conclusion is no broader than the split supports.
If designing or reviewing these splits for your own project would be difficult, the inquiry link below can also be used to ask about support for evaluation design.
Contact
Implementation support and annotation services
Ask about setup, design, and implementation support, or annotation services.
Frequently Asked Questions
Is a random 80/20 split always leakage?
No. It can be suitable for genuinely independent samples from the same deployment distribution. It is risky when repeated, adjacent, or group-related observations are treated as independent.
Should I split by site or by survey?
Use the unit that will be new at deployment. If both will be new, design a test that reserves both or clearly state the narrower condition evaluated.
Can cross-validation replace a final test set?
Grouped cross-validation helps model selection and uncertainty assessment, but a separately frozen test set is valuable when you need a final estimate untouched by iterative decisions.
What if there are too few groups?
Report the limitation, use group-aware resampling cautiously, and collect broader data if the claim requires it. Row-level randomization cannot manufacture independent evidence.
