Skip to content
Back to Blogs

Data Annotation

What Is Data Annotation? Types, Tools and Quality Checks for Computer Vision

Data annotation turns raw images and video into training data for computer vision. Learn the main annotation types, tools and the quality checks that matter.

Written byCTO
  • Published
  • Updated
  • 9 min read
Labelled car wash camera frame from our training data: boxes around two cars, three staff members, a cleaning cloth and the wash head
Table of Contents
  1. What is data annotation for computer vision?
  2. What are the main types of image and video annotation?
  3. Which annotation type does your project need?
  4. Which annotation tool should you use: CVAT, Label Studio or Roboflow?
  5. How do you check annotation quality?
  6. What does annotation look like on a real project?
  7. Which annotation mistakes cost the most?
  8. Should you annotate in-house or outsource?
  9. How does ML Bench approach data annotation?
  10. Get a free annotation sample

Every computer vision model learns from examples. Before a model can spot a shoplifter, count passengers or find a tumour on an MRI, someone has to show it thousands of correctly labelled images. That labelling work is called data annotation, and it often decides whether an AI project succeeds or fails.

This guide explains what data annotation for computer vision is, the main annotation types, how to choose a tool, and the quality checks that separate useful training data from expensive noise, with examples from projects our team has delivered.

What is data annotation for computer vision?

Data annotation is the process of adding labels to raw data so a machine learning model can learn from it. In computer vision, that means marking what appears in an image or video frame and where: a box around each car, an outline around each cell, points on a player's joints.

The model then learns the pattern between the pixels and the labels. If the labels are inconsistent, the model learns inconsistency. That's why annotation quality matters more than almost any other part of the pipeline: a better model architecture rarely makes up for labels that disagree with each other.

What are the main types of image and video annotation?

The six main types are bounding boxes, polygons, segmentation, keypoints, video tracking and OCR labelling. Which one you need depends on what the model must output: a box for detecting and counting, an outline or pixel mask for exact shapes, points for pose, and one ID across frames for movement.

  • Bounding boxes: rectangles around objects. Fast and cost-effective, ideal for object detection and counting (people, vehicles, products).
  • Polygons: precise outlines that follow an object's shape. Used when boxes include too much background, such as irregular products or vehicle damage.
  • Semantic and instance segmentation: every pixel is labelled. Needed for medical imaging, road scenes and quality inspection.
  • Keypoints and pose: points on joints or landmarks. Used for sports analytics, ergonomics and behaviour analysis such as theft detection.
  • Video tracking: the same object keeps one ID across frames, so the model learns movement, not just appearance.
  • OCR and text labelling: boxes and transcriptions for text in documents, number plates and labels.
Overhead car wash camera frame with labelled bounding boxes: two cars, four staff members, three cleaning cloths and the wash head
One training frame from our car wash dataset: 10 boxes across four classes (car, person, cloth, wash head). Faces, plates and signage blurred.
TypeWhat it marksTypical useEffort per image
Bounding boxA rectangle around each objectDetection, countingLowest
PolygonThe object's outlineIrregular shapes, damageMedium
SegmentationEvery pixel's classMedical imaging, inspectionHighest
KeypointsPoints on joints or landmarksPose, sports, behaviourMedium
Video trackingOne ID per object across framesMovement, dwell timeMedium to high
OCR labellingText boxes and their transcriptionDocuments, plates, labelsMedium
Annotation types at a glance

Which annotation type does your project need?

Start from the decision the model has to support, not from the tool. If you need a count, boxes are usually enough. If you need a size, area or shape, you need outlines or masks. If you need to know how something moves, you need tracking. Here's how that played out on our own projects.

  • Boxes and tracking IDs for sports analytics. On our padel analytics project, each player gets a box and keeps one ID through the rally, so positions can be projected onto the court as movement heatmaps. The detector reached 85%+ accuracy on in-game video.
  • Instance segmentation for cell analysis. On our cell segmentation project, every cell needed its own outline, because the client wanted area, shape and counts per cell, and cells often touch or overlap. We benchmarked MEDIAR, StarDist and Cellpose and chose Cellpose, which reached 90% segmentation accuracy across cell types.
  • Voxel-level labels for medical imaging. On our brain MRI project, the labels are whole brain regions in 3D. Before any segmentation, the MRI and PET data from public datasets such as ADNI, OASIS and AIBL had to be converted into one standard (BIDS), because each dataset used its own formats.
  • Tiny boxes for aerial video. On our drone detection project, targets can be just 3–5 pixels wide in high-resolution footage, so every box has to be exact. That footage was annotated in Label Studio, and the system reached 85%+ detection accuracy on aerial video samples.
Padel court camera view with a box and ID around each of the four players, next to a court grid showing each player's position and role
Player tracking from our padel analytics project: one box and one ID per player, projected onto the court to build movement heatmaps.
Microscope image of blood cells, each traced with a coloured outline and labelled RBC or white blood cell with its area in square micrometres
Instance segmentation from our cell analysis project: every cell outlined separately, labelled by type and measured.

Which annotation tool should you use: CVAT, Label Studio or Roboflow?

For most image and video projects, CVAT is the strongest free, self-hostable option. Label Studio suits teams that label several kinds of data (images, text, audio) in one place. Roboflow is fastest when you also want hosted training and deployment. Where your data is allowed to be stored often decides it.

CVATLabel StudioRoboflow
HostingOpen source, self-hosted, or CVAT's cloudOpen source, self-hosted, or a paid cloud editionCloud, hosted by Roboflow
Best forImage and video: boxes, polygons, masks, keypoints, trackingMixed data: images, text, audio and time seriesSmall teams going from dataset to trained model quickly
Export formatsCOCO, YOLO, Pascal VOC, CVAT XML and moreCOCO, YOLO, Pascal VOC, JSON, CSVCOCO, YOLO, Pascal VOC, TFRecord and more
Model-assisted labellingYes: auto-annotation with built-in or your own modelsYes: pre-labels from a connected ML backendYes: label assist and auto-labelling
CVAT, Label Studio and Roboflow compared

Beyond the tool itself, check the annotation types it supports, how it handles video, whether it fits your team size, and whether it exports the format your training pipeline expects (COCO, YOLO or Pascal VOC). For sensitive data such as medical or security footage, self-hosted tools and strict access control are essential. That's why medical and security projects usually run on a self-hosted tool, with the data never leaving agreed storage.

How do you check annotation quality?

Annotation quality is checked in layers: written guidelines, a pilot batch the client signs off, a second person reviewing every label, agreement checks on shared samples, and a baseline model that points to labels it disagrees with. Each layer catches mistakes the one before it misses.

Five quality layers in a row: guidelines, pilot batch, second review, agreement checks and a model check
The five quality layers we use on annotation projects.
  1. Clear labelling guidelines with visual examples of correct and incorrect labels, and rules for edge cases (occlusion, blur, partial objects).
  2. A pilot batch reviewed with the client before scaling up.
  3. Multi-stage review: annotator → reviewer → sample audit by a senior.
  4. Agreement checks: the same images labelled by two people, with disagreements resolved and guidelines updated.
  5. Model-in-the-loop review: train an early model and inspect where it disagrees with the labels; those are often labelling errors.
Scanned transaction form shown twice: fields detected by the model on the left, and the reviewed golden dataset of labelled fields on the right
Model-in-the-loop review on a scanned form: fields the model detected (left) against the human-checked golden dataset (right).

What does annotation look like on a real project?

On our car wash monitoring project, frames from the site's own cameras were labelled with the objects the system needs to follow: cars, staff, cleaning cloths and the wash head (the first image in this article). The frames were picked from different days, cars and stages of the job, not just one clip, so the model sees the site as it really is. Faces, number plates and site branding are blurred in anything we publish.

The same car wash camera on another day: an SUV with its boot and doors open, a staff member, a cloth and the wash head, each in a labelled box
The same camera on another day: a different car, open doors and floor mats. Variety like this is what lets the model work on real footage.

On a document project, a model first detected the fields on scanned transaction forms, and reviewers corrected its output into a human-checked golden dataset (the image above). Correcting a model's suggestions is usually faster than labelling every field from scratch, and the places where model and reviewer disagree show where the guidelines need to be clearer.

Sometimes the lesson is about what to label. On our retail theft detection project, detecting theft only from products disappearing produced too many false alarms. The system had to learn customer behaviour and action sequences instead, and we evaluated 8 approaches on real surveillance footage before choosing the combination that worked best.

Which annotation mistakes cost the most?

The expensive mistakes are rarely bad drawing. They're decisions made too late: classes that overlap, a schema that changes halfway through, or training data that doesn't look like the footage the model will actually see. Each one means relabelling work that has already been paid for.

  • Vague classes. If annotators can't tell "damaged" from "dirty", neither can the model. Define every class with examples before labelling starts.
  • Changing the schema mid-project. Adding a class after 5,000 images means going back through all of them. The pilot batch is the time to settle it.
  • More precision than the model needs. Pixel masks cost far more than boxes. Only pay for them when the output needs shape or area.
  • Training data that doesn't match the site. Night footage, rain, wet lenses and odd camera angles must be in the training set if they'll be in production.

Should you annotate in-house or outsource?

In-house annotation gives full control but ties up engineers on repetitive work. Outsourcing to a specialist team is usually faster and more consistent, as long as the partner understands the end model, not just the drawing tool. Look for a partner that can show sample work, explain its QA process and keep your data secure.

How does ML Bench approach data annotation?

Because our team also builds computer vision models, we annotate with the model in mind: we agree the label schema with the engineers, run a pilot batch, use multi-stage review, and test the data on a baseline model before delivery. We work with bounding boxes, polygons, segmentation, keypoints, video tracking and OCR labelling.

Labelled data is also what makes automation work further down the line. If you're weighing up what to automate next, read AI Agents vs Chatbots.

Get a free annotation sample

Have images or video that need labelling? Request a free annotation sample of 100 images.

About the author

Waqas Sultani, PhD

CTO, ML Bench

Dr. Waqas Sultani is ML Bench's CTO. He leads the technical direction of the company's computer vision and AI work, from data annotation and model training to deployment, and is the final technical reviewer for articles on this blog.

Have Questions?

Find the right solution
for now

Book a Quick Call (opens in a new tab)

It depends on the annotation type and image complexity. Bounding boxes are fastest; pixel-level segmentation takes the longest. A pilot batch gives a reliable estimate.

Next step

Get a free annotation sample

Have images or video that need labelling? Request a free annotation sample of 100 images.

  • UI/UX Design
  • Web Development
  • Mobile Apps
  • SEO
  • AI/ML Solutions