Object Detection: YOLO and R-CNN

Show a classifier a street photo and it says: "car, 97%."

Useful? For an album, yes. For a self-driving car, useless. The car needs:

• how many cars,
• where each one is,
• which ones are the pedestrian and which is the hydrant.

So the question becomes

That is object detection: localize AND classify, simultaneously, for an unknown number of things. Its output is a set of tuples:

(class, confidence, x, y, w, h)

x, y = top-left corner; w, h = width, height — together a bounding box, the rectangle that just contains the object.

And there is real difficulty hiding in that one line:

• The number is unknown. Zero objects, one, or forty — the network must say "that's all, folks." This is called the set-prediction problem.
• Two jobs at once. Every answer needs precise geometry (4 numbers) and a correct label — classification errs gently, but a box off by 40 pixels puts the car in the next lane.
• Mostly background. In a 40x40 grid of places to check, maybe 300 cells hold objects and 1,300 hold nothing. The model will drown in "empty" examples unless you handle the imbalance.

Evaluation uses two metrics: IoU (box overlap, worked below) and mAP — mean Average Precision, averaged over classes and over several IoU strictness levels.