Complete Technical Paper Content — Semantic Segmentation Ai
1. Executive Abstract, Problem Formulation & Background
Semantic Segmentation model within ML pipeline This repository shows how to build a Machine Learning Pipeline for Semantic Segmentation with TensorFlow Extended (TFX) and various GCP products such as Vertex Pipeline , Vertex Training , Vertex Endpoint . Also, the ML pipeline contains a custom TFX component that is integrated with Hugging Face 🤗 Hub - HFPusher . HFPusher pushes a trained model to 🤗 Model Hub and, optionally Gradio application to 🤗 Space Hub with the latest model out of the box. NOTE : We use U-NET based TensorFlow model from the official tutorial . Since we implement an ML pipeline, U-NET like model could be a good starting point. Other SOTA models like SegFormer from 🤗 Transformers or DeepLabv3+ will be explored later. NOTE : The aim of this project is not to serve the most SoTA segmentation model. Our main focus is to demonstrate how to build an end-to-end ML pipeline for semantic segmentation task instead. Update 17/02/2023 : This project received the #TFCommunitySpotlight award . Update 18/01/2023 : We published a blogpost on the TensorFlow blog discussing this project: End-to-End Pipeline for Segmentation with TFX, Google Cloud, and Hugging Face . Project structure project │ └───notebooks │ │ gradio_demo.ipynb │ │ inference_from_SavedModel.ipynb # test inference w/ Vertex Endpoint │ │ parse_tfrecords_pets.ipynb # test TFRecord parsing │ │ tfx_pipeline.ipynb # build TFX pipeline within a notebook │ └───tfrecords │ │ create_tfrecords_pets.py # script to create TFRecords of PETS dataset │ └───training_pipeline └───apps # Gradio app template codebase └───models # contains files related to model └───pipeline # definition of TFX pipeline Inside training_pipeline the entrypoints for the pipeline runners are defined in kubeflow_runner.py and local_runner.py . Instructions The TFX pipeline is designed to be run on both of local and GCP environments. On local environment $ cd training_pipeline $ tfx pipeline create --pipeline-path=local_runner.py \ --engine=local $ tfx pipeline compile --pipeline-path=local_runner.py \ --engine=local $ tfx run create --pipeline-name=segformer-training-pipeline \ --engine=local On Vertex AI environment There are two ways to run TFX pipeline on GCP environment(Vertex AI). First, you can run it manually with the following CLIs. In this case, you should replace GOOGLE_CLOUD_PROJECT to your GCP project ID in training_pipeline/pipeline/configs.py beforehand. $ cd training_pipeline $ tfx pipeline create --pipeline-path=kubeflow_runner.py \ --engine=vertex $ tfx pipeline compile --pipeline-path=kubeflow_runner.py \ --engine=vertex $ tfx run create --pipeline-name=segformer-training-pipeline \ --engine=vertex \ --project=$GCP_PROJECT_ID \ --regeion=$GCP_REGION Using GitHub Actions You can use workflow_dispatch feature of GitHub Action to run the pipeline on Vertex AI environment as well. In this case, go to the action tab, then select Trigger Training Pipeline on the left pane, then Run workflow on the branch of your choice. The GCP project ID in the input parameters will automatically replace the GOOGLE_CLOUD_PROJECT in training_pipeline/pipeline/configs.py . Also it will be injected to the tfx run create CLI. For further understading about how GitHub Action is implemented, please refer to its README document . To-do Notebook to prepare input dataset in TFRecord format Upload the input dataset into the GCS bucket Implement and include UNet model in the pipeline Implement Gradio app template Make a complete TFX pipeline with ExampleGen , SchemaGen , Resolver , Trainer , Evaluator , and Pusher components Add necessary configurations to the configs.py Add HFPusher component to the TFX pipeline Replace SchemaGen with ImportSchemaGen for better TFRecords parsing capability (Optional) Integrate Dataflow in ImportExampleGen to handle a large amount of dataset. This feature is included in the code as a reference, but it is not used after we switched the Sidewalk to PETS dataset. Misc notes On the use of two different datasets Initially, we started our work with the Sidewalks dataset . This dataset contains different stuff and things and is also very high-resolution in nature. To keep the runtime of our pipeline faster and to experiment quicker, we settled with a shallow UNet architecture (from this tutorial ). This is why, we also downsampled the Sidewalks dataset quite a bit (128x128, 256x256, etc.). But this led to poor quality models. To circumvent around this, we used the PETS dataset where the effects of downsampling weren't that visible compared to Sidewalks. But do note that the approaches showcases in our pipeline can easily be extended to high-resolution segmentation datasets and different model architectures (as long as they can be serialized as a SavedModel ). Acknowledgements We are thankful to the ML Developer Programs team at Google that provided GCP support.
This repository shows how to build a Machine Learning Pipeline for Semantic Segmentation with TensorFlow Extended (TFX) and various GCP products such as Vertex Pipeline , Vertex Training , Vertex Endpoint . Also, the ML pipeline contains a custom TFX component that is integrated with Hugging Face 🤗 Hub - HFPusher . HFPusher pushes a trained model to 🤗 Model Hub and, optionally Gradio application to 🤗 Space Hub with the latest model out of the box.
NOTE : We use U-NET based TensorFlow model from the official tutorial . Since we implement an ML pipeline, U-NET like model could be a good starting point. Other SOTA models like SegFormer from 🤗 Transformers or DeepLabv3+ will be explored later.
NOTE : The aim of this project is not to serve the most SoTA segmentation model. Our main focus is to demonstrate how to build an end-to-end ML pipeline for semantic segmentation task instead.
Update 18/01/2023 : We published a blogpost on the TensorFlow blog discussing this project: End-to-End Pipeline for Segmentation with TFX, Google Cloud, and Hugging Face .
project │ └───notebooks │ │ gradio_demo.ipynb │ │ inference_from_SavedModel.ipynb # test inference w/ Vertex Endpoint │ │ parse_tfrecords_pets.ipynb # test TFRecord parsing │ │ tfx_pipeline.ipynb # build TFX pipeline within a notebook │ └───tfrecords │ │ create_tfrecords_pets.py # script to create TFRecords of PETS dataset │ └───training_pipeline └───apps # Gradio app template codebase └───models # contains files related to model └───pipeline # definition of TFX pipeline
Inside training_pipeline the entrypoints for the pipeline runners are defined in kubeflow_runner.py and local_runner.py .
$ cd training_pipeline $ tfx pipeline create --pipeline-path=local_runner.py \ --engine=local $ tfx pipeline compile --pipeline-path=local_runner.py \ --engine=local $ tfx run create --pipeline-name=segformer-training-pipeline \ --engine=local
First, you can run it manually with the following CLIs. In this case, you should replace GOOGLE_CLOUD_PROJECT to your GCP project ID in training_pipeline/pipeline/configs.py beforehand.
$ cd training_pipeline $ tfx pipeline create --pipeline-path=kubeflow_runner.py \ --engine=vertex $ tfx pipeline compile --pipeline-path=kubeflow_runner.py \ --engine=vertex $ tfx run create --pipeline-name=segformer-training-pipeline \ --engine=vertex \ --project=$GCP_PROJECT_ID \ --regeion=$GCP_REGION
You can use workflow_dispatch feature of GitHub Action to run the pipeline on Vertex AI environment as well. In this case, go to the action tab, then select Trigger Training Pipeline on the left pane, then Run workflow on the branch of your choice. The GCP project ID in the input parameters will automatically replace the GOOGLE_CLOUD_PROJECT in training_pipeline/pipeline/configs.py . Also it will be injected to the tfx run create CLI.
For further understading about how GitHub Action is implemented, please refer to its README document .
2. Related Work, Literature Survey & Algorithmic Foundations
Initially, we started our work with the Sidewalks dataset . This dataset contains different stuff and things and is also very high-resolution in nature. To keep the runtime of our pipeline faster and to experiment quicker, we settled with a shallow UNet architecture (from this tutorial ). This is why, we also downsampled the Sidewalks dataset quite a bit (128x128, 256x256, etc.). But this led to poor quality models.
To circumvent around this, we used the PETS dataset where the effects of downsampling weren't that visible compared to Sidewalks.
But do note that the approaches showcases in our pipeline can easily be extended to high-resolution segmentation datasets and different model architectures (as long as they can be serialized as a SavedModel ).
We are thankful to the ML Developer Programs team at Google that provided GCP support.
Machine Learning Pipeline for Semantic Segmentation with TensorFlow Extended (TFX) and various GCP products
Semantic Segmentation using Deep Learning Repo contains outcomes from IMAGE SEGMENTATION COURSE offered at thinkautonomous.ai . This post is a gist of what the course teaches for anyone willing to learn about Semantic Segmentation using Modern Deep Learning For people already aware of fundamental concepts, here's a deepdive into SEGFORMERS -one of SOTA for Segmentation tasks For people interested in Deployment strategies - here's a deepdive into Neural Optimization techniques CLICK HERE TO CHECK OUT ALL OF MY PROJECTS Agenda Problem Statement Deep Learning Project Components Dataset Loss function Metric Models HyperParameters Results Model wise comparison Costa Rica Challenge Things that didn't work Annexure References Applications Semantic vs Instance Segmentation Problem Statement Multi-class Segmentation Problem - specifically to classify each pixel in an image to one of following 3 classes: Direct / current lane (label 0) Alternative lane (label 1) Background (label 2) Deep Learning Project Components Following are the key areas of focus when trying to solve any Deep Learning Project PoC Dataset We use the Driveable Area segment from BDD100K dataset for our project 3k labeled images split randomly into train, validation and test images (2.1k, 0.6k and 0.3k) Input Image -> (360, 640, 3) RGB image and labels -> (360, 640) uint8 datatype Labels are converted to type A as its smaller and efficient to work with As with most semantic segmentation datasets, there is data imbalance Class % in Dataset Direct lane 11.7% Alternate lane 4.7% Background 83.6% We'll need to account for class imbalance when selecting the loss function and metric No Data Augumentation was applied other than Normalization using Imagenet mean, standard deviation Loss function As mentioned earlier, Loss function must be able to handle class imbalance Problems in Medical image processing face similar issues and we can take cues from approaches taken there Loss function Use case Multi-class Cross entropy Multi-class version with option for weights for different classes Focal Loss Highly-imbalanced dataset, down-weight the contribution of easy examples, enabling model to learn hard examples Dice Loss Inspired from Dice Coefficient, a segmentation metric. Suitable for imbalanced datasets Tversky Loss Variant of Dice Coefficient. Add weight to False positives and False negatives. Considering the imbalance in the dataset and literature review, Dice loss is chosen as loss function . Dataset is imbalanced, but not to the extreme levels of Medical Image segmentation where Focal loss could have better results Metric As Semantic segmentation is an extension of classification problem, classification metrics like Accuracy, Precision, Recall, F1 score can also be applied Considering imbalanced nature of dataset, mean IoU is chosen as Evaluation Metric. Also, it's the industry standanrd for most segmentation tasks Image reference Models Following model architectures were considered: FCN with resnet50 backbone UNet with 4-level Encoder Decoder PSPNet with resnet50 backbone, aux branch Deeplabv3+ with resnet50 backbone Segformer MiT B3 Models using resnet50 backbone used the same pretrained weights for justified comparison. All of these models try to tackle the following problems in their own way: Multi-scale nature of objects Finding Global context of environment Tradeoff between global context and local semantics UNet uses Encoder-Decoder architecture with skip-connections to get accurate borders PSPNet uses Spatial Pyramid Pooling to set Global context prior and auxiliary branch from 3rd layer to speed up training Deeplabv3+ uses Atrous convolutions, Atrous Spatial Pyramid Pooling and Encoder-Decoder architecture Segformers uses Efficient Self-Attention along with Mix-FFN, Overlap Patch embedding HyperParameters All models had similar training procedures as follows: Batch size of 8 Backbones initialized with pretrained weights and no layers were frozen during training Adam optimizer with OneCycle LR Scheduler with initial learning rate = 3e-4, cosine annealing with 30% ascend 12 epochs of training where model with lowest Dice loss was taken as best model Results Model wise comparison Model meanIoU (%) Weight Size (MB) FCN-resnet50 69.56% 133 MB UNet 65.44% 118 MB PSPNet-resnet50 75.49% 187 MB Deeplabv3+-resnet50 73.05% 154 MB Segformer MiT B3 77.07% 170 MB As seen in table, Segformer achieves the highest meanIoU with model weight size ~170MB with its Multi-scale Transformer + all MLP decoder Deeplabv3+ scores 2% meanIoU lesser than PSPNet, is ~20% lighter. The lesses meanIoU might be down to Overfitting as Deeplabv3+ is more SOTA than PSPNet. There's also a variant with Xception backbone which is faster, memory-efficient too Costa Rica Challenge As a fun challenge, trained models were evaluated on unseen / very different scenario of Streets in Costa Rica Inspite of different environment and lot of shadows, the model performance is very robust Things that didn't work Freezing backbone and training the Classification head alone Cross-Entropy / Weighted CE loss functions peformed poorly compared to diceloss Different Optimizer like SGD with momentum, AdamW, Adagrad didn't improve results Different LR schedulers like CosAnnealing with Warm Restarts, PolyLR also couldn't provide better meanIoU Annexure Applications Semantic segmentation can be applied wherever Image and Image-like data is available, hence has numerous use-cases some of which are highlighted below Image source Semantic vs Instance Segmentation Semantic Segmentation doesn't differentiate across different instances of the same object while Instance Segmentation does
Repo contains outcomes from IMAGE SEGMENTATION COURSE offered at thinkautonomous.ai . This post is a gist of what the course teaches for anyone willing to learn about Semantic Segmentation using Modern Deep Learning
Following are the key areas of focus when trying to solve any Deep Learning Project PoC
Models using resnet50 backbone used the same pretrained weights for justified comparison.
PSPNet uses Spatial Pyramid Pooling to set Global context prior and auxiliary branch from 3rd layer to speed up training
Deeplabv3+ uses Atrous convolutions, Atrous Spatial Pyramid Pooling and Encoder-Decoder architecture
Actua como un radiologista y revisa la seguintes radiografias para mi: [ paciente de 6 años con crisis bronquiales repetitivas, con presencia de sibilancia desde hace 24h. Optimizalo para hacerlo lo mejor y haz las preguntas que consideres necesarias antes de proceder]
3. Dataset Ingestion, Preprocessing & Stochastic Augmentations
Act as a radiologist and review the following x-rays for me: [6-year-old patient with repetitive bronchial attacks, wheezing for 24 hours. Optimize it to do the best and ask any questions you feel necessary before proceeding]
Act as a radiologist and review the following x-rays for me: [6-year-old patient with repetitive bronchial attacks, wheezing for 24 hours. Optimize it to do the best and ask any questions you feel necessary before proceeding]
Act as a radiologist and review the following x-rays for me: [6-year-old patient with repetitive bronchial attacks, wheezing for 24 hours. Optimize it to do the best and ask any questions you feel necessary before proceeding]
ultralytics / ultralytics Star 63k Ultralytics YOLO27, YOLO26, YOLO11, YOLOv8 — object detection, instance segmentation, semantic segmentation, image classification, pose estimation, object tracking python tracking machine-learning computer-vision deep-learning pytorch yolo image-classification object-detection object-tracking semantic-segmentation pose-estimation instance-segmentation ultralytics rotated-object-detection yolov8 segment-anything yolo11 yolo26 yolo27 Updated Oct 29, 2026 Python
Ultralytics YOLO27, YOLO26, YOLO11, YOLOv8 — object detection, instance segmentation, semantic segmentation, image classification, pose estimation, object tracking
HumanSignal / label-studio Star 28.6k Label Studio is a multi-type data labeling and annotation tool with standardized output format computer-vision deep-learning image-annotation annotation annotations dataset yolo image-classification labeling datasets semantic-segmentation annotation-tool text-annotation boundingbox image-labeling labeling-tool mlops image-labelling-tool data-labeling label-studio Updated Oct 29, 2026 TypeScript
Label Studio is a multi-type data labeling and annotation tool with standardized output format
amusi / CVPR2026-Papers-with-Code Star 22.9k CVPR 2026 论文和开源项目合集 python machine-learning computer-vision deep-learning paper image-processing transformers transformer object-detection image-segmentation visual-tracking semantic-segmentation cvpr cvpr2020 cvpr2021 cvpr2022 cvpr2023 cvpr2024 cvpr2025 cvpr2026 Updated Mar 8, 2026
cvat-ai / cvat Star 17k CVAT is a leading data annotation platform for image, video, audio, and 3D datasets. It offers open-source, cloud, and enterprise products, as well as labeling services, with AI-assisted labeling, quality assurance, team collaboration, analytics, and developer APIs. computer-vision image-annotation annotation video-annotation annotations dataset yolo image-classification object-detection labeling semantic-segmentation annotation-tool boundingbox image-labeling labeling-tool computer-vision-annotation image-labelling-tool audio-annotation-tool audio-transcription audio-annotation Updated Oct 29, 2026 Python
CVAT is a leading data annotation platform for image, video, audio, and 3D datasets. It offers open-source, cloud, and enterprise products, as well as labeling services, with AI-assisted labeling, quality assurance, team collaboration, analytics, and developer APIs.
wkentaro / labelme Sponsor Star 16.3k Image annotation with Python. Supports polygon, rectangle, circle, line, point, and AI-assisted annotation. python computer-vision deep-learning image-annotation video-annotation annotations classification semantic-segmentation instance-segmentation Updated Oct 25, 2026 Python
Image annotation with Python. Supports polygon, rectangle, circle, line, point, and AI-assisted annotation.
4. Neural Network Backbone, Layer Specifications & Loss Formulations
microsoft / Swin-Transformer Star 16.1k This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows". imagenet image-classification object-detection semantic-segmentation mscoco mask-rcnn ade20k swin-transformer Updated Jul 24, 2024 Python
This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows".
qubvel-org / segmentation_models.pytorch Star 11.8k Semantic segmentation models with 500+ pretrained convolutional and transformer-based backbones. computer-vision models image-processing transformers pytorch imagenet segmentation image-segmentation unet dpt semantic-segmentation pretrained-weights pspnet fpn deeplabv3 unet-pytorch deeplab-v3-plus segmentation-models unetplusplus segformer Updated Oct 27, 2026 Python
Semantic segmentation models with 500+ pretrained convolutional and transformer-based backbones.
milesial / Pytorch-UNet Star 11.7k PyTorch implementation of the U-Net for image semantic segmentation with high quality images deep-learning pytorch kaggle tensorboard convolutional-networks convolutional-neural-networks unet semantic-segmentation pytorch-unet wandb weights-and-biases Updated Aug 11, 2024 Python
PyTorch implementation of the U-Net for image semantic segmentation with high quality images
mrgloom / awesome-semantic-segmentation Star 10.8k 🤘 awesome-semantic-segmentation benchmark evaluation deeplearning semantic-segmentation Updated May 8, 2021
OpenGVLab / InternVL Star 10.2k [CVPR 2024 Oral] InternVL Family: A Pioneering Open-Source Alternative to GPT-4o. 接近GPT-4o表现的开源多模态对话模型 image-classification gpt multi-modal semantic-segmentation video-classification image-text-retrieval llm vision-language-model gpt-4v vit-6b vit-22b gpt-4o Updated Sep 22, 2025 Python
[CVPR 2024 Oral] InternVL Family: A Pioneering Open-Source Alternative to GPT-4o. 接近GPT-4o表现的开源多模态对话模型
open-mmlab / mmsegmentation Star 10k OpenMMLab Semantic Segmentation Toolbox and Benchmark. pytorch transformer image-segmentation semantic-segmentation vessel-segmentation pspnet medical-image-segmentation deeplabv3 retinal-vessel-segmentation realtime-segmentation swin-transformer Updated Aug 13, 2024 Python
PaddlePaddle / PaddleSeg Star 9.4k Easy-to-use image segmentation library with awesome pre-trained model zoo, supporting wide-range of practical tasks in Semantic Segmentation, Interactive Segmentation, Panoptic Segmentation, Image Matting, 3D Segmentation, etc. transformer image-segmentation autonomous-driving lane-detection semantic-segmentation video-segmentation image-matting medical-image-segmentation interactive-segmentation panoptic-segmentation human-segmentation Updated Feb 5, 2026 Python
Easy-to-use image segmentation library with awesome pre-trained model zoo, supporting wide-range of practical tasks in Semantic Segmentation, Interactive Segmentation, Panoptic Segmentation, Image Matting, 3D Segmentation, etc.
5. Training Protocol, Convergence Behavior & Experimental Evaluation
dmlc / gluon-cv Star 5.9k Gluon CV Toolkit machine-learning computer-vision deep-learning neural-network mxnet gan image-classification object-detection gluon semantic-segmentation action-recognition pose-estimation person-reid Updated Nov 25, 2024 Python
ashishpatel26 / Tools-to-Design-or-Visualize-Architecture-of-Neural-Network Star 5.4k Tools to Design or Visualize Architecture of Neural Network tools tensorflow keras cnn machinelearning resnet alexnet deeplearning semantic-segmentation visualize visualize-data resnet-50 visu tensorflow2 visualize-networks visualization-neural-network Updated Aug 1, 2025
CSAILVision / semantic-segmentation-pytorch Star 5.1k Pytorch implementation for Semantic Segmentation/Scene Parsing on MIT ADE20K dataset pytorch semantic-segmentation scene-recognition ade20k Updated Jan 15, 2024 Python
weiliu89 / caffe Star 4.8k Caffe: a fast open framework for deep learning. real-time deep-learning object-detection semantic-segmentation Updated Apr 21, 2023 C++
HumanSignal / awesome-data-labeling Star 4.4k A curated list of awesome data labeling tools awesome deep-learning image-annotation annotation video-annotation lidar awesome-list labeling semantic-segmentation annotation-tool 3d-annotation image-labeling labeling-tool label-images bounding-box image-labeling-tool label-videos data-labeling audio-annotation-tool audio-annotation Updated Jun 17, 2024
NVlabs / SegFormer Star 3.7k Official PyTorch implementation of SegFormer transformer semantic-segmentation cityscapes ade20k Updated Aug 2, 2024 Python
luigifreda / pyslam Sponsor Star 3.4k pySLAM is a hybrid Python/C++ Visual SLAM pipeline supporting monocular, stereo, and RGB-D cameras. It provides a broad set of modern local and global feature extractors, multiple loop-closure strategies, a volumetric reconstruction module, integrated depth-prediction models, and semantic segmentation capabilities for enhanced scene understanding. slam place-recognition semantic-segmentation 3d-reconstruction depth-estimation visual-odometry feature-matching rgbd-slam instance-segmentation depth-prediction loop-closure local-features semantic-mapping global-features scene-understanding volumetric-reconstruction stereo-slam semantic-understanding gaussian-splatting end-to-end-reconstruction Updated Oct 25, 2026 Python
pySLAM is a hybrid Python/C++ Visual SLAM pipeline supporting monocular, stereo, and RGB-D cameras. It provides a broad set of modern local and global feature extractors, multiple loop-closure strategies, a volumetric reconstruction module, integrated depth-prediction models, and semantic segmentation capabilities for enhanced scene understanding.
meetps / pytorch-semseg Star 3.4k Semantic Segmentation Architectures Implemented in PyTorch deep-learning pytorch semantic-segmentation fully-convolutional-networks Updated Oct 11, 2023 Python
To associate your repository with the semantic-segmentation topic, visit your repo's landing page and select "manage topics."
Semantic-Segmentation-Architecture A repository contains the code for various semantic segmentation in TensorFlow and PyTorch framework. Research papers U-Net ResU-Net MultiResU-Net Star History
A repository contains the code for various semantic segmentation in TensorFlow and PyTorch framework.
Model Figures & Evaluation Curves (4 Images)
Deep Learning Execution Pipeline
(Preprocessing)
(Feature Extraction)
(Self-Attention)
(Loss Optimization)