On this page
One way to annotate marine imagery more efficiently is to let a model propose the labels and have people review them. A subset of images is annotated by hand and used to train a model, that model runs over the images nobody has inspected yet, and it predicts masks showing where each target is and what shape it has. A reviewer checks the predictions, corrects what needs correcting, and adds the settled annotations to the next round of training. Repeating this cycle labels far more images than working through every one from scratch.
Combining model predictions with human review and correction in this way is called human-in-the-loop (HITL). It is not simply a person clicking approve. It governs which images are reviewed, what is corrected, and how those results feed the next round of training and evaluation. Making it work requires written review criteria, traceable identifiers, label provenance, quality checks, and an evaluation set that is never used for development.
What the Human Reviewer Decides
In HITL, a person checks the masks the model produced. That work is not limited to confirming the extent of the region the model drew. Reviewers also decide how the image should be treated as research data:
- whether the target belongs to a declared class;
- whether two touching organisms are one region or separate instances;
- how to handle partial visibility, blur, turbidity, and occlusion;
- whether the image is usable as study data;
- whether the model found a genuinely difficult case or only a broken input.
To keep these decisions consistent across reviewers, write the criteria down in advance. Repeating the loop without them raises throughput while inconsistent judgments accumulate.
The Improvement Cycle
HITL is more than repeating prediction and correction. Decide in advance which data is reviewed, how the results return to training, and how model improvement is confirmed. The basic order is:
- Fix the model version so the same conditions can be reproduced, and apply it to unlabeled or partially labeled images to generate predictions.
- Select which images a person reviews, following a selection policy decided in advance.
- Present the original image and the model prediction to a reviewer.
- Have the reviewer record the decision, validate the corrections, and publish the settled annotations as a new version.
- Retrain from a manifest that names that version explicitly.
- Compare the models before and after the update on evaluation data that was never used for training.
Run the cycle when the added data is expected to improve a documented problem. Retraining after every small batch often produces no new evidence for evaluation while adding operational effort and variance in the results.
Keep Three Things Separate
To make the cycle auditable afterward, the model prediction, the human review, and the published annotation used for training must not be stored as the same piece of data. A reliable system keeps three records apart:
Model prediction
The model version, the mask and class it output, the confidence, and the inference parameters. It is not rewritten, so what the model proposed stays recoverable.
Review action
Who reviewed it, when, under which guideline version, and whether they accepted, edited, rejected, or deferred it. A review action does not overwrite the prediction.
Published annotation
The label adopted into a dataset version after required review and validation. Later training manifests reference this version.
If predictions, review history, and published annotations all overwrite one file called mask.png, the team cannot measure acceptance rate, reconstruct what humans corrected, or determine which model seeded a label.
Choose What Humans Review
Review capacity is limited, so the selection policy matters.
Uncertainty sampling
Select low-confidence predictions, or predictions where the model is torn between classes and assigns them similar probability (high entropy). This finds images whose boundary or class is hard to decide, but model confidence is not necessarily well calibrated as a probability. Predictions that are confident and wrong are not caught this way.
Disagreement sampling
Select cases where model versions, augmentations, or ensemble members disagree. This identifies unstable predictions but costs extra inference and can overrepresent one kind of difficulty.
Diversity or novelty sampling
Use metadata or embeddings that quantify visual appearance to cover different sites, dates, cameras, depths, or visual clusters. This avoids reviewing thousands of nearly identical frames.
Known-risk sampling
Deliberately sample conditions that matter operationally: rare species, low visibility, small targets, image edges, fouling, glare, or a new instrument.
Random audit sampling
Keep a random stream. Without it, the team cannot estimate how common errors are in the ordinary pool, and confident failures go unnoticed.
In practice these sources are combined, limiting bias in what gets reviewed without letting cases slip through. Recording which rule selected each image makes the composition of the queue auditable later.
Prevent Sequence and Group Bias
Marine image pools often contain long runs of consecutive frames extracted from video. If model uncertainty alone drives selection, hundreds of near-identical frames from one difficult moment can enter the queue. Review effort is duplicated, and the next training set becomes dominated by a single scene.
Apply group and spacing constraints when the queue is built:
- cap candidates per video, dive, transect, site, or day;
- enforce a minimum frame or time distance;
- deduplicate near-identical crops;
- sample evenly across acquisition conditions;
- preserve the original group IDs in the annotation export.
Recording the original acquisition group also matters when the data is later split into training and test sets. Data Leakage in Marine Machine Learning: Spatial and Temporal Splits Explained explains why images from one group must not fall on both sides of that boundary.
Design the Review Interface Around Decisions
The interface needs whatever a reviewer requires to check a prediction, make the necessary corrections, and record the decision. It should make the scientifically relevant action easy, and make an accidental change both visible and reversible.
Useful elements include:
- zoom and contrast controls on the original image, with a prediction overlay that can be hidden;
- class and instance identity, with editable boundary tools;
- acquisition context and neighboring frames when they are needed;
- explicit accept, edit, reject, exclude, and defer actions;
- links to the guideline and edge-case examples, plus somewhere to record why a correction was made.
Confidence is not a value that guarantees an annotation is correct. It can order the queue or explain why a candidate was selected, but it can also anchor a reviewer to an incorrect proposal. For a quality audit, hiding confidence, or comparing samples reviewed with and without the prediction shown, is useful.
Getting Started with Marine Image Annotation Using Label Studio and SAM walks through annotating with model proposals. The same separation applies there: SAM proposes a region, and the annotator decides whether it matches the project’s class definition and boundary policy.
Measure the Review Process
Whether review has actually become more efficient is judged from decision quality and dataset balance, not throughput alone. Useful measures include:
- time per reviewed image or instance;
- the distribution of accept, edit, reject, and defer decisions;
- how much the boundary moved when it was edited;
- the rate at which corrections or a second review were needed;
- review coverage by class and acquisition condition.
Acceptance rate and review time alone cannot show whether assistance helped. The queue may simply have collected easy images, or reviewers may be over-trusting the proposals. Faster review can come with lower annotation quality. Pair these measures with blinded audits or double-reviewed samples.
Do not rank individual reviewers from raw disagreement alone. They may receive different classes or difficulty levels, and frequent disagreement can expose an unclear guideline rather than careless work.
Version Labels and Decisions
For each published annotation version, keep the source image and acquisition group, the model prediction it started from, who reviewed and corrected it and how, and the final class and geometry all traceable. Record the guideline and class-map version used and the validation status as well, along with the reason for an exclusion or deferral where that applies.
Training then references a fixed annotation version named in the manifest. Reading a “latest” pointer that can change midway through a run makes the training impossible to reproduce under the same conditions.
Code, schemas, guidelines, and small manifests can live in Git, as described in How to Keep Research Code Reproducible with Git and GitHub. Large image and mask objects usually go to a dataset-versioning tool or object storage, with immutable identifiers recorded on the Git side.
Protect the Evaluation Set
As the improvement-cycle diagram shows, the evaluation-only test set sits outside the loop. It must not be used to select annotation candidates, to tune the class definition, the boundary policy, or a confidence threshold, to compare model variants, or to retrain. Avoid reusing it for demonstration images that influence development decisions.
If reviewing the test set turns up a labeling error, correct it through a written policy. Record the change, explain whether earlier comparisons remain valid, and treat the benchmark as a new version when that is warranted. Quietly fixing only the inconvenient labels invalidates every comparison that follows.
For newly arriving data, maintain a separate operational audit stream. It reveals shifts in the data distribution without turning the frozen benchmark into a training pool.
Decide Whether the Next Model Is Better
After a reviewed batch is published, compare the previous model with one trained under the same conditions. The point is to confirm on evaluation data that the added labels actually improved a documented problem.
- Record the new annotation version and train with the same pipeline as before. For the first comparison, keep the training conditions as close as possible.
- Evaluate the old and new models on the same frozen test set.
- Compare class- and group-level metrics, not only one aggregate.
- Inspect which failures were fixed, which were newly introduced, and which remain.
- Check whether the conditions targeted by the new batch improved, and record the evidence for promoting the updated model.
From Marine Image Annotations to a Segmentation Model: Training and Evaluation explains metric definitions, group-aware testing, and visual error review.
Adding labels does not guarantee improvement. Class imbalance, inconsistent correction criteria, overrepresentation of hard cases, or changed training settings can reduce ordinary-case performance.
Watch for Feedback-Loop Failure Modes
Confirmation bias
Reviewers assume the model proposal is correct and accept its errors along with it. Mix unassisted images into the queue for audit, and let the overlay be hidden when needed.
Sampling collapse
Only uncertain images are labeled, so common conditions become underrepresented. Mix uncertainty, diversity, known risk, and random sampling.
Unknown class suppression
The model cannot propose a class it does not know, and reviewers who only check proposed objects will keep missing it. Allow “unknown/other” and image-level issue reports.
Stale predictions
Proposals from several model versions are treated alike, so old predictions leak into new review queues. Store the model version per prediction and rebuild queues after a model update.
Silent guideline drift
Reviewers change their interpretation over time. Version the instructions, announce changes, and re-review affected samples only when necessary.
Benchmark contamination
Test examples enter the correction and retraining pool. Enforce dataset roles in manifests and storage permissions, then test that the separation holds.
A Small First Implementation
Rather than covering every class and image from the start, run the whole cycle on a manageable scope. For example:
- Pick one target class and start from a few acquisition groups.
- Decide how accept, edit, reject, and exclude are applied.
- Generate proposals with one fixed model version and build a queue mixing uncertainty, diversity, and random audits.
- Double-review a small subset and publish an immutable annotation version.
- Retrain once on that data and compare quality and review effort on a test set drawn from acquisition groups never used in development.
Adding classes, adding reviewers, and automating prioritization can wait until the basic cycle is confirmed.
If building this review loop inside your own team would be difficult, the inquiry link below can also be used to ask about design and implementation support, or about outsourcing the annotation work.
Contact
Implementation support and annotation services
Ask about setup, design, and implementation support, or annotation services.
Frequently Asked Questions
Is human-in-the-loop the same as active learning?
Active learning is one candidate-selection strategy. Human-in-the-loop is broader: it includes review decisions, annotation policy, provenance, quality control, versioning, retraining, and evaluation.
Should reviewers see confidence scores?
Only when they help the task, and with awareness that a model’s number can anchor the decision. Confidence can explain why a candidate entered the queue, but it must not be treated as truth.
Can accepted predictions go directly into training?
Only after the acceptance policy, reviewer identity, prediction version, validation result, and dataset version are recorded. Spot-check or double-review the images that fall into higher-risk conditions.
How often should the model be retrained?
Group work into meaningful versioned batches and judge against a declared evaluation gate. There is no universal schedule; retrain when the new data and review results justify the cost of retraining and comparison.
