To illustrate the algorithm for assigning ground-truth bounding boxes to anchor boxes, consider an IoU matrix $$\mathbf{X}$$. First, if the absolute maximum value in the matrix is $$x_{23}$$, we assign ground-truth bounding box $$B_3$$ to anchor box $$A_2$$, and then discard all elements in row $$2$$ and column $$3$$. Next, we search the remaining valid elements; if the largest is $$x_{71}$$, we assign $$B_1$$ to $$A_7$$ and discard row $$7$$ and column $$1$$. We repeat this process: finding $$x_{54}$$ assigns $$B_4$$ to $$A_5$$ (discarding row $$5$$, column $$4$$), and finding $$x_{92}$$ assigns $$B_2$$ to $$A_9$$ (discarding row $$9$$, column $$2$$). Once all ground-truth boxes are assigned, we traverse the remaining unassigned anchor boxes (e.g., $$A_1, A_3, A_4, A_6, A_8$$) and assign them to their highest-IoU ground-truth box only if that specific IoU exceeds a predefined threshold.

Ground-Truth Bounding Box Assignment Example

To assign ground-truth bounding boxes to anchor boxes, we use an iterative algorithm based on their Intersection over Union (IoU). Given $$n_a$$ anchor boxes and $$n_b$$ ground-truth bounding boxes (where $$n_a \geq n_b$$), we compute a matrix $$\mathbf{X} \in \mathbb{R}^{n_a 	imes n_b}$$ where each element $$x_{ij}$$ is the IoU between anchor box $$A_i$$ and ground-truth box $$B_j$$. First, we find the largest element in $$\mathbf{X}$$, assign the corresponding ground-truth box to the anchor box, and discard its entire row and column. We repeat this process of finding the largest remaining element and discarding its row and column until all $$n_b$$ ground-truth boxes are assigned. Finally, we evaluate the remaining $$n_a - n_b$$ unassigned anchor boxes, assigning each to the ground-truth box with the highest IoU in its row, provided that this maximum IoU is strictly greater than a predefined threshold.

Claude

In an object detection training dataset, every generated anchor box is treated as an individual training example. To train the model effectively, each anchor box must be assigned two specific types of supervision labels: a class label and an offset label. These labels are derived from the closest ground-truth bounding box assigned to the anchor box. Together, these labels provide the necessary signals for the model to simultaneously predict object categories and refine the bounding box coordinates.

Labeling Anchor Boxes in Training Data

In object detection, the similarity between two bounding boxes is measured using their Jaccard index, which is commonly referred to as the intersection over union (IoU). Treating the pixel area of any bounding box as a set of pixels, the IoU is calculated as the ratio of their intersection area to their union area: $$J(\mathcal{A},\mathcal{B}) = \frac{|\mathcal{A} \cap \mathcal{B}|}{|\mathcal{A} \cup \mathcal{B}|}$$. The IoU value ranges from $$0$$ to $$1$$, where $$0$$ indicates that the bounding boxes have absolutely no overlap, and $$1$$ indicates that the two bounding boxes are exactly equal.

Intersection over Union (IoU)

Dive into Deep Learning

At inference time, an anchor-box-based object detection model follows a multi-step prediction pipeline for each input image. First, the model generates multiple anchor boxes at various positions and scales across the image. Next, it predicts a class label and a set of bounding-box offsets for every anchor box simultaneously. The predicted offsets are then applied to adjust the positions and sizes of the corresponding anchor boxes, producing a set of predicted bounding boxes. Finally, a filtering step retains only those predicted bounding boxes that satisfy certain criteria (such as confidence thresholds or non-maximum suppression), which are output as the final detections.

Object Detection Prediction Pipeline Using Anchor Boxes

Ground-Truth Bounding Box Assignment to Anchor Boxes

To illustrate anchor box labeling, consider an image containing ground-truth bounding boxes for a dog and a cat, alongside a set of five generated anchor boxes, denoted as $$A_0, \ldots, A_4$$. The labeling process evaluates the pairs of anchor boxes and ground-truth bounding boxes based on Intersection over Union (IoU). For example, if the IoU between anchor box $$A_4$$ and the cat's ground-truth bounding box is the largest among all pairs, $$A_4$$ is labeled as the cat. Removing all pairs containing $$A_4$$ or the cat's bounding box, if the remaining pair with the largest IoU is $$A_1$$ and the dog's bounding box, $$A_1$$ is labeled as the dog. For the remaining unlabeled anchor boxes (e.g., $$A_0, A_2, A_3$$), they are assigned to the ground-truth bounding box with the highest IoU only if that IoU exceeds a predefined threshold (e.g., $$0.5$$). Thus, if $$A_2$$ has an IoU with the cat exceeding the threshold, it is labeled as the cat; conversely, if $$A_0$$ and $$A_3$$ have maximum IoUs below the threshold, they are labeled as the background class.

Learn Before

Related

Learn After