Data Annotation
What Is Data Annotation? Types, Tools and Quality Checks for Computer Vision
Data annotation turns raw images and video into training data for computer vision. Learn the main annotation types, tools and the quality checks that matter.
- Published
- Updated
- 9 min read
Table of Contents
- What is data annotation for computer vision?
- What are the main types of image and video annotation?
- Which annotation type does your project need?
- Which annotation tool should you use: CVAT, Label Studio or Roboflow?
- How do you check annotation quality?
- What does annotation look like on a real project?
- Which annotation mistakes cost the most?
- Should you annotate in-house or outsource?
- How does ML Bench approach data annotation?
- Get a free annotation sample
Every computer vision model learns from examples. Before a model can spot a shoplifter, count passengers or find a tumour on an MRI, someone has to show it thousands of correctly labelled images. That labelling work is called data annotation, and it often decides whether an AI project succeeds or fails.
This guide explains what data annotation for computer vision is, the main annotation types, how to choose a tool, and the quality checks that separate useful training data from expensive noise, with examples from projects our team has delivered.
What is data annotation for computer vision?
Data annotation is the process of adding labels to raw data so a machine learning model can learn from it. In computer vision, that means marking what appears in an image or video frame and where: a box around each car, an outline around each cell, points on a player's joints.
The model then learns the pattern between the pixels and the labels. If the labels are inconsistent, the model learns inconsistency. That's why annotation quality matters more than almost any other part of the pipeline: a better model architecture rarely makes up for labels that disagree with each other.
What are the main types of image and video annotation?
The six main types are bounding boxes, polygons, segmentation, keypoints, video tracking and OCR labelling. Which one you need depends on what the model must output: a box for detecting and counting, an outline or pixel mask for exact shapes, points for pose, and one ID across frames for movement.
- Bounding boxes: rectangles around objects. Fast and cost-effective, ideal for object detection and counting (people, vehicles, products).
- Polygons: precise outlines that follow an object's shape. Used when boxes include too much background, such as irregular products or vehicle damage.
- Semantic and instance segmentation: every pixel is labelled. Needed for medical imaging, road scenes and quality inspection.
- Keypoints and pose: points on joints or landmarks. Used for sports analytics, ergonomics and behaviour analysis such as theft detection.
- Video tracking: the same object keeps one ID across frames, so the model learns movement, not just appearance.
- OCR and text labelling: boxes and transcriptions for text in documents, number plates and labels.
| Type | What it marks | Typical use | Effort per image |
|---|---|---|---|
| Bounding box | A rectangle around each object | Detection, counting | Lowest |
| Polygon | The object's outline | Irregular shapes, damage | Medium |
| Segmentation | Every pixel's class | Medical imaging, inspection | Highest |
| Keypoints | Points on joints or landmarks | Pose, sports, behaviour | Medium |
| Video tracking | One ID per object across frames | Movement, dwell time | Medium to high |
| OCR labelling | Text boxes and their transcription | Documents, plates, labels | Medium |
Which annotation type does your project need?
Start from the decision the model has to support, not from the tool. If you need a count, boxes are usually enough. If you need a size, area or shape, you need outlines or masks. If you need to know how something moves, you need tracking. Here's how that played out on our own projects.
- Boxes and tracking IDs for sports analytics. On our padel analytics project, each player gets a box and keeps one ID through the rally, so positions can be projected onto the court as movement heatmaps. The detector reached 85%+ accuracy on in-game video.
- Instance segmentation for cell analysis. On our cell segmentation project, every cell needed its own outline, because the client wanted area, shape and counts per cell, and cells often touch or overlap. We benchmarked MEDIAR, StarDist and Cellpose and chose Cellpose, which reached 90% segmentation accuracy across cell types.
- Voxel-level labels for medical imaging. On our brain MRI project, the labels are whole brain regions in 3D. Before any segmentation, the MRI and PET data from public datasets such as ADNI, OASIS and AIBL had to be converted into one standard (BIDS), because each dataset used its own formats.
- Tiny boxes for aerial video. On our drone detection project, targets can be just 3–5 pixels wide in high-resolution footage, so every box has to be exact. That footage was annotated in Label Studio, and the system reached 85%+ detection accuracy on aerial video samples.
Which annotation tool should you use: CVAT, Label Studio or Roboflow?
For most image and video projects, CVAT is the strongest free, self-hostable option. Label Studio suits teams that label several kinds of data (images, text, audio) in one place. Roboflow is fastest when you also want hosted training and deployment. Where your data is allowed to be stored often decides it.
| CVAT | Label Studio | Roboflow | |
|---|---|---|---|
| Hosting | Open source, self-hosted, or CVAT's cloud | Open source, self-hosted, or a paid cloud edition | Cloud, hosted by Roboflow |
| Best for | Image and video: boxes, polygons, masks, keypoints, tracking | Mixed data: images, text, audio and time series | Small teams going from dataset to trained model quickly |
| Export formats | COCO, YOLO, Pascal VOC, CVAT XML and more | COCO, YOLO, Pascal VOC, JSON, CSV | COCO, YOLO, Pascal VOC, TFRecord and more |
| Model-assisted labelling | Yes: auto-annotation with built-in or your own models | Yes: pre-labels from a connected ML backend | Yes: label assist and auto-labelling |
Beyond the tool itself, check the annotation types it supports, how it handles video, whether it fits your team size, and whether it exports the format your training pipeline expects (COCO, YOLO or Pascal VOC). For sensitive data such as medical or security footage, self-hosted tools and strict access control are essential. That's why medical and security projects usually run on a self-hosted tool, with the data never leaving agreed storage.
How do you check annotation quality?
Annotation quality is checked in layers: written guidelines, a pilot batch the client signs off, a second person reviewing every label, agreement checks on shared samples, and a baseline model that points to labels it disagrees with. Each layer catches mistakes the one before it misses.
- Clear labelling guidelines with visual examples of correct and incorrect labels, and rules for edge cases (occlusion, blur, partial objects).
- A pilot batch reviewed with the client before scaling up.
- Multi-stage review: annotator → reviewer → sample audit by a senior.
- Agreement checks: the same images labelled by two people, with disagreements resolved and guidelines updated.
- Model-in-the-loop review: train an early model and inspect where it disagrees with the labels; those are often labelling errors.
What does annotation look like on a real project?
On our car wash monitoring project, frames from the site's own cameras were labelled with the objects the system needs to follow: cars, staff, cleaning cloths and the wash head (the first image in this article). The frames were picked from different days, cars and stages of the job, not just one clip, so the model sees the site as it really is. Faces, number plates and site branding are blurred in anything we publish.
On a document project, a model first detected the fields on scanned transaction forms, and reviewers corrected its output into a human-checked golden dataset (the image above). Correcting a model's suggestions is usually faster than labelling every field from scratch, and the places where model and reviewer disagree show where the guidelines need to be clearer.
Sometimes the lesson is about what to label. On our retail theft detection project, detecting theft only from products disappearing produced too many false alarms. The system had to learn customer behaviour and action sequences instead, and we evaluated 8 approaches on real surveillance footage before choosing the combination that worked best.
Which annotation mistakes cost the most?
The expensive mistakes are rarely bad drawing. They're decisions made too late: classes that overlap, a schema that changes halfway through, or training data that doesn't look like the footage the model will actually see. Each one means relabelling work that has already been paid for.
- Vague classes. If annotators can't tell "damaged" from "dirty", neither can the model. Define every class with examples before labelling starts.
- Changing the schema mid-project. Adding a class after 5,000 images means going back through all of them. The pilot batch is the time to settle it.
- More precision than the model needs. Pixel masks cost far more than boxes. Only pay for them when the output needs shape or area.
- Training data that doesn't match the site. Night footage, rain, wet lenses and odd camera angles must be in the training set if they'll be in production.
Should you annotate in-house or outsource?
In-house annotation gives full control but ties up engineers on repetitive work. Outsourcing to a specialist team is usually faster and more consistent, as long as the partner understands the end model, not just the drawing tool. Look for a partner that can show sample work, explain its QA process and keep your data secure.
How does ML Bench approach data annotation?
Because our team also builds computer vision models, we annotate with the model in mind: we agree the label schema with the engineers, run a pilot batch, use multi-stage review, and test the data on a baseline model before delivery. We work with bounding boxes, polygons, segmentation, keypoints, video tracking and OCR labelling.
Labelled data is also what makes automation work further down the line. If you're weighing up what to automate next, read AI Agents vs Chatbots.
Get a free annotation sample
Have images or video that need labelling? Request a free annotation sample of 100 images.
Waqas Sultani, PhD
CTO, ML Bench
Dr. Waqas Sultani is ML Bench's CTO. He leads the technical direction of the company's computer vision and AI work, from data annotation and model training to deployment, and is the final technical reviewer for articles on this blog.
