The Faster R-CNN model is trained end-to-end, meaning the region proposal network (RPN) is jointly trained with the rest of the object detection architecture. Its objective function combines four components: the standard object class and bounding box predictions used in Fast R-CNN, along with the binary class (object versus background) and bounding box predictions for the anchor boxes in the RPN. This joint training allows the RPN to learn from data how to generate high-quality region proposals, thereby maintaining detection accuracy while reducing the total number of proposals.

Faster R-CNN Joint Training Objective

The region proposal network (RPN) is an internal component of Faster R-CNN that generates region proposals by processing CNN-extracted feature maps in four steps. First, a $$3 	imes 3$$ convolutional layer with a padding of $$1$$ transforms the feature maps into a new output with $$c$$ channels, providing a length-$$c$$ feature vector for each spatial unit. Second, multiple anchor boxes of varying scales and aspect ratios are generated and centered on each pixel. Third, these length-$$c$$ feature vectors are used to predict a binary class (object or background) and a bounding box for each anchor box. Finally, non-maximum suppression is applied to the bounding boxes predicted as objects to remove overlapping results, yielding the final region proposals required by the region of interest pooling layer.

Claude

To reduce the number of region proposals without losing accuracy, the Faster R-CNN model improves upon the Fast R-CNN architecture by replacing the external selective search algorithm with an internal region proposal network (RPN). Aside from this change in the region proposal method, the rest of the object detection pipeline remains identical to Fast R-CNN.

Faster R-CNN

An anchor box (also called a prior box) is a predefined bounding box that is centered on every pixel of an input image and serves as a candidate region for detecting objects. Rather than searching for objects at arbitrary positions and sizes, an object detection model generates a fixed set of anchor boxes with varying shapes across the entire image and then predicts how each anchor should be adjusted to tightly fit nearby objects. Each anchor box is fully determined once its center position, scale, and aspect ratio are specified. Anchor boxes provide the initial spatial hypotheses that a detector refines, enabling efficient coverage of potential ground-truth bounding boxes without exhaustive search.

Anchor Box in Object Detection

When an object detection model generates numerous anchor boxes, it often outputs multiple predicted bounding boxes that surround the same underlying object, leading to significant overlap and redundancy. To simplify the output and ensure each object is detected only once, a technique known as non-maximum suppression (NMS) is employed. NMS merges or suppresses similar predicted bounding boxes that belong to the same object by retaining only the highest-confidence predictions and discarding the overlapping duplicates.

Non-Maximum Suppression (NMS)

In object detection, a region proposal is a candidate bounding box extracted from an input image that potentially contains an object of interest. Models typically extract a large number of these proposed regions (e.g., $$2000$$ per image) to process further. Additionally, predefined anchor boxes can also serve as region proposals.

Region Proposal

Dive into Deep Learning

Region Proposal Network

The mask R-CNN is an extension of the faster R-CNN object detection architecture. It is designed to effectively leverage detailed pixel-level labels in a training dataset to further improve object detection accuracy. By modifying the base architecture, mask R-CNN is capable of predicting both traditional bounding boxes and the precise pixel-level positions of objects.

Mask R-CNN

An anchor box centered on a pixel of an image with height $$h$$ and width $$w$$ is parameterized by two quantities: a scale $$s \in (0, 1]$$ and an aspect ratio $$r > 0$$, defined as the ratio of the box's width to its height. Given these parameters, the width of the anchor box is $$ws\sqrt{r}$$ and the height is $$hs/\sqrt{r}$$. This formulation ensures that larger scales produce bigger boxes and that the aspect ratio controls the elongation of the box — values of $$r > 1$$ yield wider boxes, while values of $$r < 1$$ yield taller boxes. Since the scale is expressed relative to the image dimensions, the resulting anchor box dimensions are directly proportional to the image size.

Anchor Box Parameterization by Scale and Aspect Ratio

The multibox_prior function returns anchor box coordinates in a normalized form, where the $$x$$-axis values are divided by the image width and the $$y$$-axis values are divided by the image height. This means all coordinate values fall within the range $$[0, 1]$$, making them independent of the absolute pixel dimensions of the input image. Each anchor box is stored as four values representing the upper-left $$(x, y)$$ corner and the lower-right $$(x, y)$$ corner in this normalized coordinate system. To recover the original pixel coordinates for tasks such as visualization, a scaling tensor bbox_scale of the form $$(w, h, w, h)$$ is element-wise multiplied with the normalized coordinates to restore them to the image's native resolution.

Normalized Anchor Box Coordinates

In an object detection training dataset, every generated anchor box is treated as an individual training example. To train the model effectively, each anchor box must be assigned two specific types of supervision labels: a class label and an offset label. These labels are derived from the closest ground-truth bounding box assigned to the anchor box. Together, these labels provide the necessary signals for the model to simultaneously predict object categories and refine the bounding box coordinates.

Labeling Anchor Boxes in Training Data

The non-maximum suppression (NMS) algorithm filters out redundant predicted bounding boxes based on their confidence scores and spatial overlap. For a given predicted bounding box $$B$$, its confidence is the largest predicted class likelihood, denoted as $$p$$. First, all non-background bounding boxes on an image are sorted by confidence in descending order to form a list $$L$$. The algorithm iteratively selects the bounding box $$B_1$$ with the highest confidence in $$L$$ as a basis and removes any remaining boxes in $$L$$ whose Intersection over Union (IoU) with $$B_1$$ exceeds a predefined threshold $$\epsilon$$. Those with non-maximum confidence scores are thus suppressed. This process is repeated using the next highest confidence box as the new basis until every box in $$L$$ has been used as a basis, resulting in a final set of predictions where no two boxes have an IoU greater than $$\epsilon$$.

Non-Maximum Suppression Algorithm

Selective search is a computer vision algorithm utilized to extract multiple high-quality region proposals from an input image. By selecting proposed regions at multiple scales and with varying shapes and sizes, the algorithm provides a set of candidate bounding boxes that can subsequently be evaluated by an object detection model.

Learn Before

Related

Learn After