mAP is the metric to measure the accuracy of object detectors. It is the average of the maximum precisions at different recall values.
Precision & recall
Precision measures how accurate is your predictions. i.e. the percentage of your positive predictions are correct. If you have a precision score of close to 1.0 then there is a high likelihood that whatever the classifier predicts as a positive detection is in fact a correct prediction.
Recall measures how good you find all the positives. For example, we can find 80% of the possible positive cases in our top K predictions. If you have a recall score close to 1.0 then almost all objects that are in your dataset will be positively detected by the model.
Here are their mathematical definitions:
IoU (Intersection over union)
IoU measures how much overlap between 2 regions, This measures how good is our prediction in the object detector with the ground truth.
AP (Average Precision)
AP calculation with precision-recall curve
The calculation if the AP score is to take the average value of the precision across all recall values (see explanation in section 4.2 of the Pascal Challenge paper).
However the actual calculation is a little different.
We set the precision for recall r to the maximum precision obtained for any recall r' > r. Then we compute the AP as the area under this curve (shown in light blue) by numerical integration. No approximation is involved since the curve is piecewise constant.
Code for AP calculation
def compute_ap(recall, precision):
""" Compute the average precision, given the recall and precision curves.
Code originally from https://github.com/rbgirshick/py-faster-rcnn.
# Arguments
recall: The recall curve (list).
precision: The precision curve (list).
# Returns
The average precision as computed in py-faster-rcnn.
"""
# correct AP calculation
# first append sentinel values at the end
mrec = np.concatenate(([0.], recall, [1.]))
mpre = np.concatenate(([0.], precision, [0.]))
# compute the precision envelope
for i in range(mpre.size - 1, 0, -1):
mpre[i - 1] = np.maximum(mpre[i - 1], mpre[i])
# to calculate area under PR curve, look for points
# where X axis (recall) changes value
i = np.where(mrec[1:] != mrec[:-1])[0]
# and sum (\Delta recall) * prec
ap = np.sum((mrec[i + 1] - mrec[i]) * mpre[i + 1])
return ap
import numpy as np
recall = [0.2,0.4,0.4,0.4,0.4,0.6,0.8,0.8,0.8,1.0]
precision = [1.0,1.0,0.67,0.5,0.4,0.5,0.57,0.5,0.44,0.5]
AP = compute_ap(recall, precision)
print('AP = {}'.format(AP))
# AP = 0.728
# If we calculate using 11 spaced recall values,
# AP = (5 × 1.0 + 4 × 0.57 + 2 × 0.5)/11 = 0.753.
# Check the this blog: mAP for Object Detection
# Intermediate results
def compute_ap(recall, precision):
# correct AP calculation
# first append sentinel values at the end
mrec = np.concatenate(([0.], recall, [1.]))
mpre = np.concatenate(([0.], precision, [0.]))
# mrec = [0. 0.2 0.4 0.4 0.4 0.4 0.6 0.8 0.8 0.8 1. 1. ]
# mpre = [0. 1. 1. 0.67 0.5 0.4 0.5 0.57 0.5 0.44 0.5 0. ]
# compute the precision envelope
for i in range(mpre.size - 1, 0, -1):
mpre[i - 1] = np.maximum(mpre[i - 1], mpre[i])
# mpre = [1. 1. 1. 0.67 0.57 0.57 0.57 0.57 0.5 0.5 0.5 0. ]
# to calculate area under PR curve, look for points
# where X axis (recall) changes value
i = np.where(mrec[1:] != mrec[:-1])[0]
# i = [0 1 5 6 9]
# mrec[1:] = [0.2 0.4 0.4 0.4 0.4 0.6 0.8 0.8 0.8 1. 1. ]
# mrec[:-1] = [0. 0.2 0.4 0.4 0.4 0.4 0.6 0.8 0.8 0.8 1. ]
# and sum (\Delta recall) * prec
ap = np.sum((mrec[i + 1] - mrec[i]) * mpre[i + 1])
return ap
Reference
mean Average Precision-github
mAP for Object Detection
Evaluation 12: mean average precision
Understanding the mAP Evaluation Metric for Object Detection
Discussion on fast.ai