Complete Technical Paper Content — Ssd Object Detection
1. Executive Abstract, Problem Formulation & Background
Object detection is a computer technology related to computer vision and image processing that deals with detecting instances of semantic objects of a certain class (such as humans, buildings, or cars) in digital images and videos. Well-researched domains of object detection include face detection and pedestrian detection. Object detection has applications in many areas of computer vision, including image retrieval and video surveillance.
For object localization, true positive is often measured by the thresholded intersection over union. For example, if there is a traffic sign in the image, with a bounding box drawn by a human ("ground truth label"), then a neural network has detected the traffic sign (a true positive) at 0.5 threshold if it has drawn a bounding box whose IoU with the ground truth is above 0.5. Otherwise, the bounding box is a false positive. If there is only a single ground truth bounding box, but multiple predictions, then the IoU of each prediction is calculated. The prediction with the highest IoU is a true positive if it is above threshold, else it is a false positive. All other predicted bounding boxes are false positives. If there is no prediction with an IoU above the threshold, then the ground truth label has a false negative. For simultaneous object localization and classification, a true positive is one where the class label is correct, and the bounding box has an IoU exceeding the threshold. Simultaneous object localization and classification is benchmarked by the mean average precision (mAP). The average precision (AP) of the network for a class of objects is the area under the precision-recall curve as the confidence threshold is varied. The mAP is the average of AP over all classes.
Non-neural approaches: Viola–Jones object detection framework based on Haar features Scale-invariant feature transform (SIFT) Histogram of oriented gradients (HOG) features Neural network approaches: OverFeat. Region Proposals (R-CNN, Fast R-CNN, Faster R-CNN, cascade R-CNN.) You Only Look Once (YOLO). Single Shot MultiBox Detector (SSD) Single-Shot Refinement Neural Network for Object Detection (RefineDet) Retina-Net Deformable convolutional networks Detection transformer (DETR), which uses vision transformers.
In reality, such textures and outlines would not be represented by single nodes, but rather by associated weight patterns of multiple nodes.
Small object detection is a particular case of object detection where various techniques are employed to detect small objects in digital images and videos. "Small objects" are objects having a small pixel footprint in the input image. In areas such as aerial imagery, state-of-the-art object detection techniques under performed because of small objects.
Small object detection has applications in various fields such as Video surveillance (Traffic video Surveillance, Small object retrieval, Anomaly detection, Maritime surveillance, Drone surveying, Traffic flow analysis, and Object tracking.
SSD: Single Shot MultiBox Object Detector, in PyTorch A PyTorch implementation of Single Shot MultiBox Detector from the 2016 paper by Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang, and Alexander C. Berg. The official and original Caffe code can be found here . Table of Contents Installation Datasets Train Evaluate Performance Demos Future Work Reference Installation Install PyTorch by selecting your environment on the website and running the appropriate command. Clone this repository. Note: We currently only support Python 3+. Then download the dataset by following the instructions below. We now support Visdom for real-time loss visualization during training! To use Visdom in the browser: # First install Python server and client pip install visdom # Start the server (probably in a screen or tmux) python -m visdom.server Then (during training) navigate to http://localhost:8097/ (see the Train section below for training details). Note: For training, we currently support VOC and COCO , and aim to add ImageNet support soon. Datasets To make things easy, we provide bash scripts to handle the dataset downloads and setup for you. We also provide simple dataset loaders that inherit torch.utils.data.Dataset , making them fully compatible with the torchvision.datasets API . COCO Microsoft COCO: Common Objects in Context Download COCO 2014 # specify a directory for dataset to be downloaded into, else default is ~/data/ sh data/scripts/COCO2014.sh VOC Dataset PASCAL VOC: Visual Object Classes Download VOC2007 trainval & test # specify a directory for dataset to be downloaded into, else default is ~/data/ sh data/scripts/VOC2007.sh # <directory> Download VOC2012 trainval # specify a directory for dataset to be downloaded into, else default is ~/data/ sh data/scripts/VOC2012.sh # <directory> Training SSD First download the fc-reduced VGG-16 PyTorch base network weights at: https://s3.amazonaws.com/amdegroot-models/vgg16_reducedfc.pth By default, we assume you have downloaded the file in the ssd.pytorch/weights dir: mkdir weights cd weights wget https://s3.amazonaws.com/amdegroot-models/vgg16_reducedfc.pth To train SSD using the train script simply specify the parameters listed in train.py as a flag or manually change them. python train.py Note: For training, an NVIDIA GPU is strongly recommended for speed. For instructions on Visdom usage/installation, see the Installation section. You can pick-up training from a checkpoint by specifying the path as one of the training parameters (again, see train.py for options) Evaluation To evaluate a trained network: python eval.py You can specify the parameters listed in the eval.py file by flagging them or manually changing them. Performance VOC2007 Test mAP Original Converted weiliu89 weights From scratch w/o data aug From scratch w/ data aug 77.2 % 77.26 % 58.12% 77.43 % FPS GTX 1060: ~45.45 FPS Demos Use a pre-trained SSD network for detection Download a pre-trained network We are trying to provide PyTorch state_dicts (dict of weight tensors) of the latest SSD model definitions trained on different datasets. Currently, we provide the following PyTorch models: SSD300 trained on VOC0712 (newest PyTorch weights) https://s3.amazonaws.com/amdegroot-models/ssd300_mAP_77.43_v2.pth SSD300 trained on VOC0712 (original Caffe weights) https://s3.amazonaws.com/amdegroot-models/ssd_300_VOC0712.pth Our goal is to reproduce this table from the original paper Try the demo notebook Make sure you have jupyter notebook installed. Two alternatives for installing jupyter notebook: If you installed PyTorch with conda (recommended), then you should already have it. (Just navigate to the ssd.pytorch cloned repo and run): jupyter notebook If using pip : # make sure pip is upgraded pip3 install --upgrade pip # install jupyter notebook pip install jupyter # Run this inside ssd.pytorch jupyter notebook Now navigate to demo/demo.ipynb at http://localhost:8888 (by default) and have at it! Try the webcam demo Works on CPU (may have to tweak cv2.waitkey for optimal fps) or on an NVIDIA GPU This demo currently requires opencv2+ w/ python bindings and an onboard webcam You can change the default webcam in demo/live.py Install the imutils package to leverage multi-threading on CPU: pip install imutils Running python -m demo.live opens the webcam and begins detecting! TODO We have accumulated the following to-do list, which we hope to complete in the near future Still to come: Support for the MS COCO dataset Support for SSD512 training and testing Support for training on custom datasets Authors Max deGroot Ellis Brown Note: Unfortunately, this is just a hobby of ours and not a full-time job, so we'll do our best to keep things up to date, but no guarantees. That being said, thanks to everyone for your continued help and feedback as it is really appreciated. We will try to address everything as soon as possible. References Wei Liu, et al. "SSD: Single Shot MultiBox Detector." ECCV2016. Original Implementation (CAFFE) A huge thank you to Alex Koltun and his team at Webyclip for their help in finishing the data augmentation portion. A list of other great SSD ports that were sources of inspiration (especially the Chainer repo): Chainer , Keras , MXNet , Tensorflow
A PyTorch implementation of Single Shot MultiBox Detector from the 2016 paper by Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang, and Alexander C. Berg. The official and original Caffe code can be found here .
2. Related Work, Literature Survey & Algorithmic Foundations
# First install Python server and client pip install visdom # Start the server (probably in a screen or tmux) python -m visdom.server
To make things easy, we provide bash scripts to handle the dataset downloads and setup for you. We also provide simple dataset loaders that inherit torch.utils.data.Dataset , making them fully compatible with the torchvision.datasets API .
# specify a directory for dataset to be downloaded into, else default is ~/data/ sh data/scripts/COCO2014.sh
# specify a directory for dataset to be downloaded into, else default is ~/data/ sh data/scripts/VOC2007.sh # <directory>
# specify a directory for dataset to be downloaded into, else default is ~/data/ sh data/scripts/VOC2012.sh # <directory>
mkdir weights cd weights wget https://s3.amazonaws.com/amdegroot-models/vgg16_reducedfc.pth
You can specify the parameters listed in the eval.py file by flagging them or manually changing them.
If you installed PyTorch with conda (recommended), then you should already have it. (Just navigate to the ssd.pytorch cloned repo and run): jupyter notebook
3. Dataset Ingestion, Preprocessing & Stochastic Augmentations
# make sure pip is upgraded pip3 install --upgrade pip # install jupyter notebook pip install jupyter # Run this inside ssd.pytorch jupyter notebook
We have accumulated the following to-do list, which we hope to complete in the near future
Note: Unfortunately, this is just a hobby of ours and not a full-time job, so we'll do our best to keep things up to date, but no guarantees. That being said, thanks to everyone for your continued help and feedback as it is really appreciated. We will try to address everything as soon as possible.
Single Shot MultiBox Detector (SSD300) from Scratch in PyTorch 📖 Accompanying Technical Article : Read the deep-dive architectural walkthrough on Medium: SSD: Single Shot MultiBox Detector Explained by Dheeraj Kumar Maradana 📌 Overview This repository provides a modular, clean PyTorch implementation of the SSD300 (Single Shot MultiBox Detector) object detection architecture from scratch, trained and evaluated on the PASCAL VOC 2007 benchmark. Unlike two-stage detectors (e.g., Faster R-CNN) that utilize separate Region Proposal Networks (RPN), SSD achieves high inference speed by predicting bounding box offsets and category scores directly across multi-scale feature maps in a single feed-forward pass. 🏗️ Architecture & Pipeline flowchart LR A["Input Image<br/>(300x300x3)"] --> B["VGG-16 Base Backbone<br/>(Truncated at Conv5_3)"] B --> C["Auxiliary Conv Layers<br/>(Conv6 to Conv11_2)"] subgraph MultiScale_Feature_Pyramid["Multi-Scale Feature Pyramid"] F1["Conv4_3 (38x38x512)"] F2["Conv7 / FC7 (19x19x1024)"] F3["Conv8_2 (10x10x512)"] F4["Conv9_2 (5x5x256)"] F5["Conv10_2 (3x3x256)"] F6["Conv11_2 (1x1x256)"] end B --> F1 C --> F2 C --> F3 C --> F4 C --> F5 C --> F6 F1 --> D["MultiBox Predictor Heads<br/>(Loc: 4 coords & Conf: 21 classes)"] F2 --> D F3 --> D F4 --> D F5 --> D F6 --> D D --> E["Loss Formulation<br/>- Smooth L1 Localization<br/>- Hard Negative Mining (3:1)<br/>- Softmax Cross-Entropy"] E --> G["NMS Post-Processing<br/>Final Detections"] Loading Key Technical Elements: Multi-Scale Detection : 6 distinct feature map scales ranging from high-resolution shallow layers ( $38 \times 38$ for small objects) down to coarse semantic layers ( $1 \times 1$ for large objects). Default Box Generators : Configured with aspect ratios ${1:1, 1:2, 2:1, 1:3, 3:1}$ and scale ratios $s_k$ according to the original formulation. Hard Negative Mining (3:1) : Eliminates severe foreground-background class imbalance by selecting only the highest-loss background negative samples in a 3:1 ratio to matched positives. MultiBox Objective Loss : $$\mathcal{L}(x, c, l, g) = \frac{1}{N} \left( \mathcal{L} {\text{conf}}(x, c) + \alpha , \mathcal{L} {\text{loc}}(x, l, g) \right)$$ 📁 Repository Structure ├── ssd_from_scratch.py # Complete scratch implementation of SSD300 model & MultiBoxLoss ├── train_voc.py # Custom training harness on Pascal VOC 2007 (SGD with warmup) ├── train_voc_torchvision.py # Comparative training harness using TorchVision's reference SSD ├── test_voc.py # mAP evaluation and qualitative visualization engine ├── utils.py # Default box matching, IoU computation, NMS, and VOC dataloaders ├── ssd_orig.py # Structural reference comparison └── 1512.02325v5.pdf # Original Liu et al. research paper 🚀 Getting Started 1. Dataset Setup (Pascal VOC 2007) # Download VOC 2007 trainval and test splits wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar # Extract archives tar -xf VOCtrainval_06-Nov-2007.tar tar -xf VOCtest_06-Nov-2007.tar # Extracted data is available at VOCdevkit/VOC2007/ 2. Training # Train the from-scratch implementation python3 train_voc.py # Or train the TorchVision reference baseline for comparison python3 train_voc_torchvision.py 3. Evaluation & Visualization # Evaluate scratch model mAP on test split python3 test_voc.py --checkpoint checkpoints/ssd_best.pth # Evaluate TorchVision model python3 test_voc.py --checkpoint checkpoints_tv/ssd_best.pth --torchvision # Save visual predictions with bounding boxes python3 test_voc.py --checkpoint checkpoints/ssd_best.pth --save-vis --num-vis 50 📚 References & Attribution Paper : Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., & Berg, A. C. (2016). SSD: Single shot multibox detector . ECCV 2016. Author : Dheeraj Kumar Maradana ( IIT Bombay )
📖 Accompanying Technical Article : Read the deep-dive architectural walkthrough on Medium: SSD: Single Shot MultiBox Detector Explained by Dheeraj Kumar Maradana
This repository provides a modular, clean PyTorch implementation of the SSD300 (Single Shot MultiBox Detector) object detection architecture from scratch, trained and evaluated on the PASCAL VOC 2007 benchmark.
Unlike two-stage detectors (e.g., Faster R-CNN) that utilize separate Region Proposal Networks (RPN), SSD achieves high inference speed by predicting bounding box offsets and category scores directly across multi-scale feature maps in a single feed-forward pass.
flowchart LR A["Input Image<br/>(300x300x3)"] --> B["VGG-16 Base Backbone<br/>(Truncated at Conv5_3)"] B --> C["Auxiliary Conv Layers<br/>(Conv6 to Conv11_2)"] subgraph MultiScale_Feature_Pyramid["Multi-Scale Feature Pyramid"] F1["Conv4_3 (38x38x512)"] F2["Conv7 / FC7 (19x19x1024)"] F3["Conv8_2 (10x10x512)"] F4["Conv9_2 (5x5x256)"] F5["Conv10_2 (3x3x256)"] F6["Conv11_2 (1x1x256)"] end B --> F1 C --> F2 C --> F3 C --> F4 C --> F5 C --> F6 F1 --> D["MultiBox Predictor Heads<br/>(Loc: 4 coords & Conf: 21 classes)"] F2 --> D F3 --> D F4 --> D F5 --> D F6 --> D D --> E["Loss Formulation<br/>- Smooth L1 Localization<br/>- Hard Negative Mining (3:1)<br/>- Softmax Cross-Entropy"] E --> G["NMS Post-Processing<br/>Final Detections"]
4. Neural Network Backbone, Layer Specifications & Loss Formulations
├── ssd_from_scratch.py # Complete scratch implementation of SSD300 model & MultiBoxLoss ├── train_voc.py # Custom training harness on Pascal VOC 2007 (SGD with warmup) ├── train_voc_torchvision.py # Comparative training harness using TorchVision's reference SSD ├── test_voc.py # mAP evaluation and qualitative visualization engine ├── utils.py # Default box matching, IoU computation, NMS, and VOC dataloaders ├── ssd_orig.py # Structural reference comparison └── 1512.02325v5.pdf # Original Liu et al. research paper
# Download VOC 2007 trainval and test splits wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar # Extract archives tar -xf VOCtrainval_06-Nov-2007.tar tar -xf VOCtest_06-Nov-2007.tar # Extracted data is available at VOCdevkit/VOC2007/
# Train the from-scratch implementation python3 train_voc.py # Or train the TorchVision reference baseline for comparison python3 train_voc_torchvision.py
# Evaluate scratch model mAP on test split python3 test_voc.py --checkpoint checkpoints/ssd_best.pth # Evaluate TorchVision model python3 test_voc.py --checkpoint checkpoints_tv/ssd_best.pth --torchvision # Save visual predictions with bounding boxes python3 test_voc.py --checkpoint checkpoints/ssd_best.pth --save-vis --num-vis 50
Single Shot MultiBox Detector (SSD300) implemented from scratch in PyTorch on Pascal VOC
🎯 Object Detection using SSD (Single Shot MultiBox Detector) A computer vision project focused on detecting and classifying objects in real-time using the SSD (Single Shot MultiBox Detector) algorithm. This project leverages Python and OpenCV for implementing a robust object detection model capable of identifying multiple objects in images and video streams with high accuracy. 🌟 Features Real-Time Detection : Capable of detecting multiple objects in real-time using a single forward pass. Pre-trained Models : Utilizes pre-trained SSD models such as SSD300 and SSD512 for efficient object detection. Bounding Boxes : Draws bounding boxes with labels and confidence scores around detected objects. Customizable Classes : Supports detection of multiple classes, including custom object categories. Cross-Platform Compatibility : Works with both images and live video feeds from cameras. 🚀 Technologies Used Python : Core programming language for model development. OpenCV : For image processing and real-time video capture. TensorFlow & Keras : For implementing the SSD architecture and loading pre-trained models. NumPy : For numerical operations and data manipulation. Matplotlib : For visualizing the results of object detection. 📂 Project Structure data/ : Contains the dataset used for testing and evaluation. models/ : Includes pre-trained SSD models and weight files. scripts/ : Python scripts for object detection on images and videos. notebooks/ : Jupyter notebooks with step-by-step implementation and visualization. output/ : Stores output images and video files with detected objects. README.md : Detailed documentation and setup instructions. 🎯 Use Cases Ideal for applications in surveillance, autonomous vehicles, and smart systems requiring fast and accurate object detection. This project serves as a great learning resource for anyone interested in deep learning and computer vision.
A computer vision project focused on detecting and classifying objects in real-time using the SSD (Single Shot MultiBox Detector) algorithm. This project leverages Python and OpenCV for implementing a robust object detection model capable of identifying multiple objects in images and video streams with high accuracy.
Ideal for applications in surveillance, autonomous vehicles, and smart systems requiring fast and accurate object detection. This project serves as a great learning resource for anyone interested in deep learning and computer vision.
5. Training Protocol, Convergence Behavior & Experimental Evaluation
SSD (Single Shot Multibox Detector) is a real-time deep learning model for detecting and classifying objects in images by performing object localization and classification in a single forward pass.
SSD-Object-Detection I am working on a SSD Object Detector using fastai and pytorch This work was inspired from fastai's course Part -2 on 2018. https://www.youtube.com/watch?v=0frKXR-2PBY&list=PLfYUBJiXbdtTttBGq-u2zeY1OTjs5e-Ia&index=3&t=0s . And the code was inspired from their courses repository on github. https://github.com/fastai/fastai/blob/master/courses/dl2/pascal-multi.ipynb I tried to increase the grids from just 4, 2 and 1 to 28, 14, 7, 4, 2, 1. I am trying to create a SSD Model which infers classes and bounding boxes from 6 different grid sizes. In the original SSD paper they have used grid sizes of 19, 10, 7, 4, 2, 1. I have used a slightly different architecture. I have hooked outputs of the specified grid size from a pretrained model and pass those hooked outputs thorugh a ResBlock and a Out Conv layer which provides the classification and bounding box outputs.I also have a custom head to the pretrained model where I get inferences on grid sizes smaller than 7. Thanks to @muellerzr 's repo https://github.com/muellerzr/Practical-Deep-Learning-for-Coders-2.0.git from where the 'imports' folder is taken from his computer vision section . This imports folder contains the RetinaNet Model and RetinaNet local Loss and other inference tools for object detection. The Architecture image is below: Some Additional functionality added are : 1) A Callback for recording hyper parameter changes for every layer 2) A Interactive Learning Rate Scheduler using jupyter widgets to schedule interactively
This work was inspired from fastai's course Part -2 on 2018. https://www.youtube.com/watch?v=0frKXR-2PBY&list=PLfYUBJiXbdtTttBGq-u2zeY1OTjs5e-Ia&index=3&t=0s . And the code was inspired from their courses repository on github. https://github.com/fastai/fastai/blob/master/courses/dl2/pascal-multi.ipynb
I tried to increase the grids from just 4, 2 and 1 to 28, 14, 7, 4, 2, 1. I am trying to create a SSD Model which infers classes and bounding boxes from 6 different grid sizes. In the original SSD paper they have used grid sizes of 19, 10, 7, 4, 2, 1. I have used a slightly different architecture.
I have hooked outputs of the specified grid size from a pretrained model and pass those hooked outputs thorugh a ResBlock and a Out Conv layer which provides the classification and bounding box outputs.I also have a custom head to the pretrained model where I get inferences on grid sizes smaller than 7.
Thanks to @muellerzr 's repo https://github.com/muellerzr/Practical-Deep-Learning-for-Coders-2.0.git from where the 'imports' folder is taken from his computer vision section . This imports folder contains the RetinaNet Model and RetinaNet local Loss and other inference tools for object detection.
Some Additional functionality added are : 1) A Callback for recording hyper parameter changes for every layer 2) A Interactive Learning Rate Scheduler using jupyter widgets to schedule interactively
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Model Figures & Evaluation Curves (4 Images)
Deep Learning Execution Pipeline
(Preprocessing)
(Feature Extraction)
(Self-Attention)
(Loss Optimization)