insonl
← journal

Build a Manufacturing Visual Inspection Dataset Before Picking a Model

date
22 November 2026
by
insonl
read
8 min read

Illustrative example. A plant team captures defect and pass photos on Line 1 during day shift, splits them randomly 80/20, trains a vision model, and sees high offline scores. After deployment on Line 2 on night shift, overhead glare and a new raw-material coil send acceptable parts to rework while small border scratches pass unchecked.

That failure starts before model selection. Building a visual inspection dataset for manufacturing defects is a taxonomy, metadata, and evaluation-split problem first; model choice comes after.

This guide uses three third-party public industrial inspection benchmarks from MVTec Software—MVTec AD, MVTec AD 2, and MVTec LOCO AD—plus the MVTec AD and MVTec AD 2 papers, to specify a dataset for computer vision services. All MVTec datasets are third-party public benchmarks under CC BY-NC-SA 4.0 (non-commercial use only); MVTec is not an Insonl client, and every factory example in this guide is illustrative.

Define defect categories before labeling a single frame

A binary good / defect label hides operational rules. Different defect families require different optics, annotations, and shop-floor dispositions (scrap, rework, or log-only).

For a plant quality specification, distinguish structural defects, logical defects, and acceptable variation. The three MVTec benchmarks illustrate these distinctions:

  • Structural defects: Local physical flaws—scratches, dents, holes, contaminations, or deformations. The MVTec AD paper covers 5,354 high-resolution images across 15 object and texture categories, with 73 defect types spanning 1,888 pixel-annotated anomalous regions.
  • Logical defects: Violations of rules about presence, position, or count. Models limited to local appearance can miss defects that depend on relationships across the image. MVTec LOCO AD covers 3,644 images across 5 categories separating structural and logical anomalies.
  • Acceptable variation: Cosmetic or process variation that must pass. The MVTec AD 2 paper (8,004 high-resolution images across 8 categories) uses fabric and sheet metal to illustrate variability in normal textures, and rice, walnuts, and wall plugs to illustrate overlapping or occluded bulk items.

Before labeling, write a defect taxonomy specification. The table below is an illustrative, made-up example for a stamped aluminum housing station (not from a real plant or Insonl client):

defect_code Family Disposition Annotation format Minimum size or rule
SURF_SCRATCH_CRIT Structural Scrap Pixel mask Length ≥ 2.0 mm on sealing face
EDGE_BURR Structural Rework Pixel mask Protrusion ≥ 0.8 mm on rim
CLIP_MISSING Logical Rework Region + count rule 4 clips present in slots 1–4
ORIENT_INVERTED Logical Stop line Image tag + keypoint Alignment notch at top-left
GRAIN_VAR_OK Acceptable variation Pass (hard normal) Image tag Mill-finish shift within spec

Follow two labeling controls documented in MVTec AD 2’s ground-truth protocol:

  1. Refine labels against written per-object rules. MVTec AD 2 captured coarse labels on-site during acquisition; annotators refined the initial labels using object-specific guidelines, checked for incomplete or spurious regions, and adjusted masks when lighting significantly changed a defect’s appearance.
  2. Audit defect-free pools as strictly as defective pools. MVTec AD 2 manually screened its defect-free training and validation images for missed defects so hidden anomalies would not corrupt the baseline.

Log capture metadata so normal variation is measurable

A flat folder of pass/ and fail/ images discards the variables behind production errors. Every frame needs an immutable sidecar record tied to a unique physical part ID.

In MVTec AD, every image depicts a unique physical sample rather than multiple poses of one object. Logging part_instance_id enforces the same boundary on a factory line: if a station captures multiple frames of one housing, all frames stay in one split so the same part never leaks across training, validation, and test.

Below is an illustrative, made-up capture manifest record (not from a real factory or Insonl client):

{
  "image_id": "img_20261122_L1S2_004812",
  "part_instance_id": "HOUSING-20261122-09142",
  "sku_revision": "ALU-H42-REV3",
  "line_id": "LINE-1",
  "station_id": "STA-02",
  "camera_serial": "CAM-GIGE-1104",
  "lighting_profile": "dark_field_low_angle",
  "exposure_us": 4200,
  "supplier_lot_id": "COIL-8841B",
  "tool_die_id": "DIE-04",
  "capture_utc": "2026-11-22T01:14:22Z",
  "native_resolution_px": [2448, 2048],
  "px_per_mm": 18.5,
  "ground_truth_status": "defect",
  "defect_codes": ["EDGE_BURR"],
  "label_guideline_version": "v1.2"
}

Log native resolution and px_per_mm alongside optics settings. Table 3 of the MVTec AD 2 paper lists frames from about 2.3 to 5.0 megapixels, with anomalies across the full image including borders—exposing center-crop and padding artifacts. The authors hypothesized that some small, low-contrast defects could disappear when resized to 256×256; in their Rice experiment, scaling up to half native dimensions roughly doubled AU-PRO0.05 for INP-Former, PatchCore, and MSFlow at the largest tested input, while raising latency and peak GPU memory by more than an order of magnitude (about 400 ms for INP-Former and 2 seconds for PatchCore on an NVIDIA RTX 2080 Ti). Recording scale early lets teams evaluate tiling or optics before committing to cycle times.

Separate evaluation by production line, lighting condition, and normal variation

Frame-level random splitting can inflate offline accuracy when related shots from the same shift, fixture, lighting state, and material coil land on both sides of a split.

MVTec AD 2 shows how to structure holdout evaluation for real shifts. Training and validation contain only defect-free images, with training under regular lighting. The two private test sets show the same scenes: TESTpriv uses training-matched lighting, while TESTpriv,mix includes both seen and unseen lighting. Each scene was captured under at least four lighting conditions—varying exposure and adding spotlights or alternate ring-light angles to induce reflections and uneven gradients.

Separating lighting conditions exposed stark model differences at 256×256 in the MVTec AD 2 paper. Across eight benchmarked methods on MVTec AD 2, mean AU-PRO0.30 stayed below 60%, and under AU-PRO0.05 on TESTpriv, INP-Former led with a 31.6% mean. Moving from matched (TESTpriv) to mixed lighting (TESTpriv,mix), Reverse Distillation fell 1.4 percentage points while MSFlow dropped 12.4 percentage points (24.3% to 11.9%).

Build four frozen holdout slices before comparing models. The matrix below is an illustrative, made-up evaluation design for a two-line housing plant (not from a real manufacturer or Insonl client):

Evaluation slice Split rule (disjoint from training) What it measures
1. In-distribution baseline Line 1, regular lighting, known supplier lots, later shifts, unseen part_instance_ids Core defect recall under nominal conditions
2. Cross-line transfer Line 2 held out from training (separate camera, fixture, background) Generalization to a different line
3. Lighting-shift stress Over/underexposure, shift glare, alternate light incidence angle Optical robustness before recalibration
4. Hard-normal variation Defect-free parts from a new supplier_lot_id, worn tool_die_id, or borderline grain False-reject rate under acceptable variation

Enforce three split rules in code: keep every part_instance_id and production run on one side of the training, validation, and test boundaries, freeze all four test slices before model tuning, and report metrics per slice instead of one pooled average.

Pick evaluation metrics and thresholds before comparing models

Freeze the test slices and define acceptance targets before training. Calibrate each candidate’s cutoff on a separate validation set, then lock that cutoff before evaluating the frozen test slices. Two findings from the MVTec papers explain why default metrics mislead buyers:

  1. Per-pixel AUROC masks missed small defects. Only 2.7% of test pixels in MVTec AD are anomalous (MVTec AD paper). Pixel-level metrics weight every pixel equally, so catching one large smudge offsets missing several small cracks. Both MVTec papers evaluate Per-Region Overlap (PRO / AU-PRO), weighting each connected defect region equally. MVTec AD 2 tightens the false-positive integration cap from 0.30 to 0.05 (AU-PRO0.05): for a 10-pixel defect in a 5-megapixel frame, even a 5% pixel false-positive rate flags about 25,000 times the defect area in false-positive pixels.
  2. Threshold-independent rankings do not predict reject decisions. Production lines act on a fixed cutoff, not a curve area. MVTec AD found that an outlier can make a maximum-score threshold too conservative and that a Gaussian-based k-sigma cutoff can miss its target false-positive rate. MVTec AD 2 found that its baseline validation cutoff varied in effectiveness across methods and input sizes. Under MVTec AD 2’s baseline protocol, PatchCore and MSFlow scored strongly on threshold-independent metrics yet lagged on thresholded image-level classification F1; for PatchCore, normalizing anomaly maps across the entire input dataset worked on test data containing both normal and anomalous images but yielded an excessively high cutoff on defect-free validation data alone.

Treat cutoffs as an operational cost decision, mirroring setting OCR confidence thresholds for invoice review. Report critical-defect escape rate and false-reject rate separately on every slice containing defects; on the defect-free hard-normal slice, report false rejects and mark escape rate not applicable. Calibrate thresholds on a separate validation split containing hard normals—and labeled critical defects when available—never on frozen test slices, and log latency and memory at the resolution required for your smallest defect.

Only after freezing taxonomy, acceptance targets, and test slices should you compare models. MVTec AD benchmarked an aligned statistical variation model alongside deep models; geometric checks are another candidate where tolerances can be expressed directly. Keep licensing clean: MVTec AD, MVTec AD 2, and MVTec LOCO AD carry CC BY-NC-SA 4.0 (attribution, non-commercial use only, ShareAlike for distributed adaptations). Use these datasets for non-commercial benchmarking under CC BY-NC-SA 4.0, and use factory-owned or appropriately licensed data for commercial training and evaluation.

Next step

Before buying cameras or training models, scope defect tolerances, lighting constraints, and evaluation slices—the operational grounding described in how an AI workflow audit works, alongside Insonl’s computer vision services. Insonl offers a free AI workflow audit delivering a written workflow map, recommendations, and a fixed-scope estimate; bringing sample images, capture setup details, and error-cost assumptions grounds the review. The audit is free; Insonl charges only for follow-on engineering work you choose to proceed with.

Sources