Neural Architecture Search | AI Final Year Project Report

1. Executive Abstract, Problem Formulation & Background

microsoft / nni Star 14.4k An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning. python data-science machine-learning deep-learning neural-network tensorflow machine-learning-algorithms pytorch distributed hyperparameter-optimization feature-engineering nas bayesian-optimization hyperparameter-tuning automl automated-machine-learning model-compression neural-architecture-search deep-neural-network mlops Updated Jul 3, 2024 Python

An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.

keras-team / autokeras Star 9.3k AutoML library for deep learning python machine-learning deep-learning tensorflow keras automl automated-machine-learning neural-architecture-search autodl Updated Nov 25, 2025 Python

hibayesian / awesome-automl-papers Star 4.2k A curated list of automated machine learning papers, articles, tutorials, slides and projects hyperparameter-optimization automl neural-architecture-search automated-feature-engineering Updated Jun 11, 2024

A curated list of automated machine learning papers, articles, tutorials, slides and projects

quark0 / darts Star 4k Differentiable architecture search for convolutional and recurrent networks deep-learning language-modeling pytorch recurrent-networks image-classification convolutional-networks automl neural-architecture-search Updated Jan 3, 2021 Python

tensorflow / adanet Star 3.5k Fast and flexible AutoML with learning guarantees. python machine-learning deep-learning tensorflow gpu ensemble automl learning-theory neural-architecture-search distributed-training tpu Updated Nov 30, 2023 Jupyter Notebook

carpedm20 / ENAS-pytorch Star 2.7k PyTorch implementation of "Efficient Neural Architecture Search via Parameters Sharing" pytorch neural-architecture-search google-brain Updated Jul 6, 2023 Python

PyTorch implementation of "Efficient Neural Architecture Search via Parameters Sharing"

D-X-Y / Awesome-AutoDL Star 2.3k Automated Deep Learning: Neural Architecture Search Is Not the End (a curated list of AutoDL resources and an in-depth analysis) awesome deep-learning nas automl neural-architecture-search hyper-parameter-optimization autodl Updated Sep 26, 2022 Python

Automated Deep Learning: Neural Architecture Search Is Not the End (a curated list of AutoDL resources and an in-depth analysis)

chaoyanghe / Awesome-Federated-Learning Star 2k FedML - The Research and Production Integrated Federated Learning Library: https://fedml.ai machine-learning privacy computer-vision semi-supervised-learning transfer-learning wireless-communication distributed-optimization interpretability neural-architecture-search federated-learning continual-learning vertical-federated-learning non-iid decentralized-federated-learning hierarchical-federated-learning adversarial-attack-and-defense communication-efficiency straggler-problem computation-efficiency incentive-mechanism Updated Sep 3, 2022

FedML - The Research and Production Integrated Federated Learning Library: https://fedml.ai

rentruewang / aioway Sponsor Star 1.8k AI on the way. An auto deep learning pipe dream. An RDBMS approach to deep learning. Declarative, explainable, scalable, optimizable, easy to deploy, all that good stuff. python sql compiler pytorch lazy-evaluation relational-algebra neural-architecture-search explainable-ai auto-deep-learning Updated Oct 29, 2026 Python

AI on the way. An auto deep learning pipe dream. An RDBMS approach to deep learning. Declarative, explainable, scalable, optimizable, easy to deploy, all that good stuff.

kubeflow / katib Star 1.7k Automated Machine Learning on Kubernetes kubernetes machine-learning ai tensorflow scikit-learn pytorch hyperparameter-tuning automl neural-architecture-search kubeflow jax mlops huggingface llm Updated Oct 25, 2026 Python

2. Related Work, Literature Survey & Algorithmic Foundations

D-X-Y / AutoDL-Projects Star 1.6k Automated deep learning algorithms implemented in PyTorch. pytorch nas automl neural-architecture-search autodl Updated Apr 24, 2022 Python

markdtw / awesome-architecture-search Star 1.2k A curated list of awesome architecture search resources deep-learning neural-architecture-search Updated Sep 15, 2020

THUMNLab / AutoGL Star 1.1k An autoML framework & toolkit for machine learning on graphs. machine-learning deep-learning pytorch automl neural-architecture-search hyper-parameter-optimization graph-neural-networks pytorch-geometric Updated Nov 20, 2025 Python

google-research / morph-net Star 1k Fast & Simple Resource-Constrained Learning of Deep Network Structure python machine-learning deep-learning tensorflow automl neural-architecture-search Updated Oct 5, 2026 Python

gigwegbe / tinyml-papers-and-projects Star 1k This is a list of interesting papers and projects about TinyML. machine-learning computer-vision embedded-systems neural-architecture-search tinyml wake-word Updated Dec 8, 2025

mit-han-lab / tinyengine Star 967 [NeurIPS 2020] MCUNet: Tiny Deep Learning on IoT Devices; [NeurIPS 2021] MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning; [NeurIPS 2022] MCUNetV3: On-Device Training Under 256KB Memory c microcontroller deep-learning cpp pytorch codegenerator quantization edge-computing neural-architecture-search tinyml Updated Nov 27, 2024 C

[NeurIPS 2020] MCUNet: Tiny Deep Learning on IoT Devices; [NeurIPS 2021] MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning; [NeurIPS 2022] MCUNetV3: On-Device Training Under 256KB Memory

joeddav / devol Star 954 Early POC of genetic neural architecture search machine-learning computer-vision deep-learning genetic-algorithm keras automl neural-architecture-search Updated May 25, 2023 Python

JiahuiYu / slimmable_networks Star 925 Slimmable Networks, AutoSlim, and Beyond, ICLR 2019, and ICCV 2019 efficient on-demand adaptive automated neural-architecture-search edge-devices slimmable-networks Updated Mar 9, 2023 Python

guan-yuan / Awesome-AutoML-and-Lightweight-Models Star 855 A list of high-quality (newest) AutoML works and lightweight models including 1.) Neural Architecture Search, 2.) Lightweight Structures, 3.) Model Compression, Quantization and Acceleration, 4.) Hyperparameter Optimization, 5.) Automated Feature Engineering. tensorflow pytorch hyperparameter-optimization awesome-list quantization nas automl model-compression neural-architecture-search meta-learning architecture-search quantized-training model-acceleration automated-feature-engineering quantized-neural-network Updated Jun 19, 2021

A list of high-quality (newest) AutoML works and lightweight models including 1.) Neural Architecture Search, 2.) Lightweight Structures, 3.) Model Compression, Quantization and Acceleration, 4.) Hyperparameter Optimization, 5.) Automated Feature Engineering.

mit-han-lab / mcunet Star 722 [NeurIPS 2020] MCUNet: Tiny Deep Learning on IoT Devices; [NeurIPS 2021] MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning deep-learning pytorch neural-architecture-search tinyml microncontroller Updated Mar 29, 2024 Python

[NeurIPS 2020] MCUNet: Tiny Deep Learning on IoT Devices; [NeurIPS 2021] MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning

To associate your repository with the neural-architecture-search topic, visit your repo's landing page and select "manage topics."

Neural Architecture Search (NAS) Framework A comprehensive framework for efficient deep neural network architecture search, developed as part of a Final Year Project for BSc (Hons) Computer Science. Overview This project implements a modular Neural Architecture Search framework that automatically discovers optimal neural network architectures for given tasks. The framework balances performance (accuracy) with efficiency metrics (parameter count, inference time) through multi-objective optimization. Key Features Modular Design : Easily extensible components for search space, model generation, evaluation, and search strategies Multiple Search Strategies : Random Search (baseline) Bayesian Optimization (sample-efficient) Gradient-Based Search (performance-oriented) Multi-Objective Optimization : Balances accuracy with computational efficiency Progressive Search : Staged approach that gradually expands search space complexity Comprehensive Visualization : Tools for analyzing search results and model performance Project Structure NAS_Project/ ├── src/ │ └── nas/ │ ├── __init__.py │ ├── search_space.py # Defines available operations and hyperparameter ranges │ ├── model_generator.py # Translates architectures to executable models │ ├── evaluator.py # Evaluates model performance │ ├── search_strategy.py # Implements search strategies │ └── visualization.py # Utilities for visualizing results ├── visualization/ # Scripts for generating figures ├── main.py # Main script to run experiments └── README.md Installation # Clone the repository git clone https://github.com/yourusername/NAS_Project.git cd NAS_Project # Install dependencies pip install -r requirements.txt Requirements Python 3.8+ TensorFlow 2.5+ NumPy 1.20+ Matplotlib 3.4+ Seaborn 0.11+ Optuna 2.10+ (for Bayesian optimization) Pandas (for data processing) Usage Basic Usage from src . nas . search_space import SearchSpace from src . nas . model_generator import ModelGenerator from src . nas . evaluator import Evaluator from src . nas . search_strategy import RandomSearch , BayesianOptimization , GradientBasedSearch # Define search space search_space = SearchSpace ( input_shape = ( 28 , 28 , 1 ), # MNIST output_shape = ( 10 ,), # 10 classes max_layers = 5 ) # Create model generator model_generator = ModelGenerator ( search_space ) # Create evaluator evaluator = Evaluator ( x_train , y_train , x_val , y_val ) # Choose search strategy strategy = BayesianOptimization ( search_space , model_generator , evaluator ) # Perform search results = strategy . search ( iterations = 20 ) # Get best architecture best_architecture = results [ 0 ][ "architecture" ] best_evaluation = results [ 0 ][ "evaluation" ] print ( f"Best accuracy: { best_evaluation [ 'val_accuracy' ] } " ) print ( f"Parameters: { best_evaluation [ 'parameters' ] } " ) Running Experiments # Run experiments on MNIST dataset python main.py --dataset mnist --strategy bayesian --iterations 20 # Run experiments on CIFAR-10 dataset python main.py --dataset cifar10 --strategy gradient --iterations 30 Visualization The visualization directory contains scripts for generating various figures: generate_figure_5_1.py : Parameter counts vs. layer composition generate_figure_5_2.py : Convergence of search strategies generate_figure_5_3.py : Pareto frontier of accuracy vs. parameter count generate_chapter4_figures.py : Framework architecture and search space visualization generate_progressive_search_figure.py : Visualization of progressive search approach generate_appendix_f_figures.py : Learning curves and confusion matrices generate_appendix_f_tables.py : Detailed architecture comparison tables Results The framework was evaluated on MNIST and CIFAR-10 datasets: MNIST : Best architecture achieved 98.7% accuracy with 850K parameters CIFAR-10 : Best architecture achieved 86.9% accuracy with 2.2M parameters Compared to baseline approaches (LeNet-5, Simple CNN), our NAS-discovered architectures achieved higher accuracy with fewer parameters. License This project is licensed under the MIT License - see the LICENSE file for details. Acknowledgments This project was developed as part of a Final Year Project for BSc (Hons) Computer Science Special thanks to [Your Advisor/Professor] for guidance and support

A comprehensive framework for efficient deep neural network architecture search, developed as part of a Final Year Project for BSc (Hons) Computer Science.

3. Dataset Ingestion, Preprocessing & Stochastic Augmentations

This project implements a modular Neural Architecture Search framework that automatically discovers optimal neural network architectures for given tasks. The framework balances performance (accuracy) with efficiency metrics (parameter count, inference time) through multi-objective optimization.

NAS_Project/ ├── src/ │ └── nas/ │ ├── __init__.py │ ├── search_space.py # Defines available operations and hyperparameter ranges │ ├── model_generator.py # Translates architectures to executable models │ ├── evaluator.py # Evaluates model performance │ ├── search_strategy.py # Implements search strategies │ └── visualization.py # Utilities for visualizing results ├── visualization/ # Scripts for generating figures ├── main.py # Main script to run experiments └── README.md

# Clone the repository git clone https://github.com/yourusername/NAS_Project.git cd NAS_Project # Install dependencies pip install -r requirements.txt

from src . nas . search_space import SearchSpace from src . nas . model_generator import ModelGenerator from src . nas . evaluator import Evaluator from src . nas . search_strategy import RandomSearch , BayesianOptimization , GradientBasedSearch # Define search space search_space = SearchSpace ( input_shape = ( 28 , 28 , 1 ), # MNIST output_shape = ( 10 ,), # 10 classes max_layers = 5 ) # Create model generator model_generator = ModelGenerator ( search_space ) # Create evaluator evaluator = Evaluator ( x_train , y_train , x_val , y_val ) # Choose search strategy strategy = BayesianOptimization ( search_space , model_generator , evaluator ) # Perform search results = strategy . search ( iterations = 20 ) # Get best architecture best_architecture = results [ 0 ][ "architecture" ] best_evaluation = results [ 0 ][ "evaluation" ] print ( f"Best accuracy: { best_evaluation [ 'val_accuracy' ] } " ) print ( f"Parameters: { best_evaluation [ 'parameters' ] } " )

# Run experiments on MNIST dataset python main.py --dataset mnist --strategy bayesian --iterations 20 # Run experiments on CIFAR-10 dataset python main.py --dataset cifar10 --strategy gradient --iterations 30

Compared to baseline approaches (LeNet-5, Simple CNN), our NAS-discovered architectures achieved higher accuracy with fewer parameters.

NAS-RL: Neural Architecture Search with Reinforcement Learning An RL-based Neural Architecture Search system. A recurrent controller samples network architectures from a defined search space, child models are built and trained, and the controller is updated with REINFORCE using validation performance as the reward. Search runs are orchestrated with Ray Tune and exposed through a FastAPI service. What's implemented Component File(s) Details RL controller src/controller/rnn_controller.py , src/controller/transformer_controller.py LSTM-based (and Transformer-based) controllers that sample operations, skip connections, and hyperparameters step by step Policy training src/controller/trainer.py REINFORCE with a moving-average baseline Search spaces src/search_space/cnn_space.py , src/search_space/transformer_space.py CNN search space (for CIFAR-10) and a Transformer search space (for SST-2) with validation and random sampling Child models src/child/builder.py , src/train/lightning_trainer.py Builds CNN or Transformer models from a sampled architecture and trains them with PyTorch Lightning Distributed search src/ray/run_search.py Ray Tune / Ray Train orchestration of architecture evaluation Model registry src/mlops/model_registry.py Stores discovered architectures and their metrics, and ranks the best ones API src/api/main.py Token-protected FastAPI service: trigger a search, check status, list best architectures, predict, and Prometheus metrics CI and data .github/workflows/ci.yml , dvc.yaml , scripts/download_data.py GitHub Actions test run; DVC tracking for CIFAR-10 and SST-2 data Current status and roadmap ✅ Controller, REINFORCE training, CNN search space, and child-model builder ✅ Ray Tune integration, FastAPI service, and architecture registry ✅ Unit tests for the controller and search space, run in CI ⏳ Benchmark results on CIFAR-10 and SST-2 ⏳ Kubernetes deployment (a starter Helm chart is in infra/k8s/helm/ ) Project structure configs/ search configs (dev, CIFAR-10, SST-2) and a sample search request src/controller/ RNN and Transformer controllers, REINFORCE trainer src/search_space/ CNN and Transformer search spaces src/child/ child-model builder src/train/ Lightning training for child models src/ray/ Ray Tune search orchestration src/mlops/ architecture registry src/api/ FastAPI service tests/unit/ controller and search-space tests infra/ Dockerfile and Helm chart Getting started git clone https://github.com/TRasagna/Neural-Architecture-Search-with-Reinforcement-Learning.git cd Neural-Architecture-Search-with-Reinforcement-Learning make install cp .env.example .env # set API_SECRET_KEY python scripts/download_data.py # CIFAR-10 make test # Start the API, then trigger a search with configs/search_request.json uvicorn src.api.main:app --port 8000 Tech stack Python · PyTorch · PyTorch Lightning · Reinforcement Learning (REINFORCE) · Ray Tune · FastAPI · DVC · Docker · GitHub Actions License MIT. See LICENSE .

An RL-based Neural Architecture Search system. A recurrent controller samples network architectures from a defined search space, child models are built and trained, and the controller is updated with REINFORCE using validation performance as the reward. Search runs are orchestrated with Ray Tune and exposed through a FastAPI service.

configs/ search configs (dev, CIFAR-10, SST-2) and a sample search request src/controller/ RNN and Transformer controllers, REINFORCE trainer src/search_space/ CNN and Transformer search spaces src/child/ child-model builder src/train/ Lightning training for child models src/ray/ Ray Tune search orchestration src/mlops/ architecture registry src/api/ FastAPI service tests/unit/ controller and search-space tests infra/ Dockerfile and Helm chart

git clone https://github.com/TRasagna/Neural-Architecture-Search-with-Reinforcement-Learning.git cd Neural-Architecture-Search-with-Reinforcement-Learning make install cp .env.example .env # set API_SECRET_KEY python scripts/download_data.py # CIFAR-10 make test # Start the API, then trigger a search with configs/search_request.json uvicorn src.api.main:app --port 8000

Python · PyTorch · PyTorch Lightning · Reinforcement Learning (REINFORCE) · Ray Tune · FastAPI · DVC · Docker · GitHub Actions

Automatically designs deep learning architectures using an RL controller that generates network topologies.

Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights Official repository of the paper presented at CVPR 2025 Authors : Ondřej Týbl Lukáš Neumann Paper : CVPR2025 We are proud members of the VRG at Czech Technical University in Prague . Join us if you are interested in cutting-edge research in neural networks and computer vision! Click for more info. Abstract Deep learning has revolutionized computer vision, but it achieved its tremendous success using deep network architectures which are mostly hand-crafted and therefore likely suboptimal. Neural Architecture Search (NAS) aims to bridge this gap by following a well-defined optimization paradigm which systematically looks for the best architecture, given objective criterion such as maximal classification accuracy. The main limitation of NAS is however its astronomical computational cost, as it typically requires training each candidate network architecture from scratch. In this paper, we aim to alleviate this limitation by proposing a novel training-free proxy for image classification accuracy based on Fisher Information. The proposed proxy has a strong theoretical background in statistics and it allows estimating expected image classification accuracy of a given deep network without training the network, thus significantly reducing computational cost of standard NAS algorithms. Our training-free proxy achieves state-of-the-art results on three public datasets and in two search spaces, both when evaluated using previously proposed metrics, as well as using a new metric that we propose which we demonstrate is more informative for practical NAS applications. Method Training-free NAS proxy called \textit{Variance of Knowledge of Deep Network Weights (VKDNW)}. Strong theoretical support and achieved state-of-the-art results on three public datasets and in two search spaces. VKDNW provides information orthogonal to network size unlike previous methods (this helps to easily factor the network performance into \textit{size} and \textit{shape} proxies). Zero-cost network ranking where contributions of network size and architectural feasibility are separated. We demonstrated that previously used correlation metrics for proxy evaluation do not sufficiently assess the key ability to discriminate top networks. To address this, we proposed a new evaluation metric Normalized Discounted Cumulative Gain (nDCG). Results Install and run Clone the repository git clone https://github.com/ondratybl/VKDNW.git Install the requirements: cd VKDNW conda env create -f environment.yml For the NAS-Bench-201 search space, prepare the API file from NATS-Bench (e.g., ./api_data/NATS-tss-v1_0-3ffb9-simple ) and run tss_general_.py for experiments. For the MobileNetV2 search space, prepare the ImageNet dataset and change accordingly ImageNet_MBV2/Dataloader/__init__.py and run ImageNet_MBV2/evolution_search_vkdnw.py for experiments. Acknowledgement The implementation is based on the following projects (please cite the corresponding papers appropriately): NATS-Bench AutoDL-Projects ZenNAS ZiCo TF_TAS Citation @inproceedings{tybl2025training, title={Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights}, author={Tybl, Ondrej and Neumann, Lukas}, booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference}, pages={14881--14890}, year={2025} }

We are proud members of the VRG at Czech Technical University in Prague . Join us if you are interested in cutting-edge research in neural networks and computer vision! Click for more info.

Deep learning has revolutionized computer vision, but it achieved its tremendous success using deep network architectures which are mostly hand-crafted and therefore likely suboptimal. Neural Architecture Search (NAS) aims to bridge this gap by following a well-defined optimization paradigm which systematically looks for the best architecture, given objective criterion such as maximal classification accuracy. The main limitation of NAS is however its astronomical computational cost, as it typically requires training each candidate network architecture from scratch. In this paper, we aim to alleviate this limitation by proposing a novel training-free proxy for image classification accuracy based on Fisher Information. The proposed proxy has a strong theoretical background in statistics and it allows estimating expected image classification accuracy of a given deep network without training the network, thus significantly reducing computational cost of standard NAS algorithms. Our training-free proxy achieves state-of-the-art results on three public datasets and in two search spaces, both when evaluated using previously proposed metrics, as well as using a new metric that we propose which we demonstrate is more informative for practical NAS applications.

For the NAS-Bench-201 search space, prepare the API file from NATS-Bench (e.g., ./api_data/NATS-tss-v1_0-3ffb9-simple ) and run tss_general_.py for experiments.

4. Neural Network Backbone, Layer Specifications & Loss Formulations

For the MobileNetV2 search space, prepare the ImageNet dataset and change accordingly ImageNet_MBV2/Dataloader/__init__.py and run ImageNet_MBV2/evolution_search_vkdnw.py for experiments.

The implementation is based on the following projects (please cite the corresponding papers appropriately):

@inproceedings{tybl2025training, title={Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights}, author={Tybl, Ondrej and Neumann, Lukas}, booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference}, pages={14881--14890}, year={2025} }