Complete Technical Paper Content — Demand Forecasting Machine Learning
1. Executive Abstract, Problem Formulation & Background
These datasets are used in machine learning (ML) research and have been cited in peer-reviewed academic journals. Datasets are an integral part of the field of machine learning. Major advances in this field can result from advances in learning algorithms (such as deep learning), computer hardware, and, less intuitively, the availability of high-quality training datasets. High-quality labeled training datasets for supervised and semi-supervised machine-learning algorithms are usually difficult and expensive to produce because of the large amount of time needed to label the data. Although they do not need to be labeled, high-quality unlabeled datasets for unsupervised learning can also be difficult and costly to produce. Many organizations, including governments, publish and share their datasets, often using common metadata formats (such as Croissant). The datasets are classified, based on the licenses, into two groups: open data and non-open data. The datasets from various governmental-bodies are presented in List of open government data sites. The datasets are ported on open data portals. They are made available for searching, depositing and accessing through interfaces like Open API. The datasets are made available as various sorted types and subtypes.
The data portal is classified based on its type of license. The open source license based data portals are known as open data portals which are used by many government organizations and academic institutions.
The data portal sometimes lists a wide variety of subtypes of datasets pertaining to many machine learning applications.
The data portals which are suitable for a specific subtype of machine learning application are listed in the subsequent sections.
**RTP set** – 35,087 stationary-point geometries (reactant, transition state and product) drawn from 11,961 elementary reactions, each labeled with density-functional energies, atomic forces and full Hessian matrices at the ωB97X-D/6-31G(d) level. **IRC set** – 34,248 structures along 600 minimum-energy reaction paths, used to test extrapolation beyond trained stationary points. **NMS set** – 62,527 off-equilibrium geometries generated by normal-mode sampling to probe model robustness under thermal perturbations. The collection underpins the study Does Hessian Data Improve the Performance of Machine Learning Potentials? and was used to train and benchmark the machine-learning interatomic potentials reported therein. The dataset itself is distributed under a CC licence via Figshare.
OpenML: Web platform with Python, R, Java, and other APIs for downloading hundreds of machine learning datasets, evaluating algorithms on datasets, and benchmarking algorithm performance against dozens of other algorithms. PMLB: A large, curated repository of benchmark datasets for evaluating supervised machine learning algorithms. Provides classification and regression datasets in a standardized format that are accessible through a Python API. Metatext NLP: https://metatext.io/datasets web repository maintained by community, containing nearly 1000 benchmark datasets, and counting. Provides many tasks from classification to QA, and various languages from English, Portuguese to Arabic. Appen: Off The Shelf and Open Source Datasets hosted and maintained by the company. These biological, image, physical, question answering, signal, sound, text, and video resources number over 250 and can be applied to over 25 different use cases.
In machine learning, a neural network (NN) or artificial neural network (ANN) is a computational model inspired by the structure and functions of biological neural networks. A neural network consists of connected units or nodes called artificial neurons, which loosely model the neurons in the brain. These are connected by edges, which model the synapses in the brain. Each artificial neuron receives signals from connected neurons, then processes them and sends a signal to other connected neurons. The "signal" is a real number, and the output of each neuron is computed by some non-linear function of the totality of its inputs, called the activation function. The strength of the signal at each connection is determined by a weight, which adjusts as part of the training process. Groups of neurons are aggregated into layers. Each layer performs a transformation on its inputs. Signals travel from the first layer (the input layer) to the last layer (the output layer), typically passing through multiple intermediate layers (hidden layers). A network is typically called a deep neural network if it has at least two hidden layers. Deep neural networks are capable of learning sophisticated hierarchical representations. Training neural networks is a compute-intensive process, accelerated by the use of graphics processing units (GPUs), and large datasets.
2. Related Work, Literature Survey & Algorithmic Foundations
In reality, such textures and outlines would not be represented by single nodes, but rather by associated weight patterns of multiple nodes. Architectural innovations such as convolutional neural networks (CNNs) significantly improved performance in computer vision tasks, while recurrent neural networks (RNNs) enabled modeling of sequential data such as speech and time-series information. Transformer architectures introduced attention mechanisms that allow neural networks to model long-range dependencies in data and have been the basis of large language models. Artificial neural networks are used for myriad tasks including chatbots, large-scale text, image, and video generation, and robotics.
During the 2010s, the seq2seq model was developed, and attention mechanisms were added. This led to the modern transformer architecture in 2017 in "Attention Is All You Need". It requires computation time that is quadratic in the size of the context window. Schmidhuber's fast weight controller (1992) scales linearly and was later shown to be equivalent to the unnormalized linear transformer. Transformers have increasingly become the model of choice for natural language processing. Many modern large language models such as GPT, Gemini, Grok, DeepSeek, and Qwen use this architecture.
NNs began as an attempt to replicate the architecture of the brain in the digital realm. It immediately showed promise for handling non-linear relationships, but encountered obstacles. Models soon reoriented to applying mathematical insights to improve empirical results, at the expense of biological fidelity.
NNs are composed of digital neurons, conceptually derived from biological neurons. Each neuron has one or more numerical inputs and produces a single numerical output. An input (e.g., an image) is typically parceled across the set of input neurons (each getting a piece of the image). Each neuron's inputs, weighted by the weights of the connections from each of those inputs are summed. A bias term is added to this sum. The result is then passed through a nonlinear activation function to produce the neuron's output. Small enough outputs may be zeroed out (ignored).
Nodes are arranged in layers, with the bottom layers directly addressing raw data, while the top layers reveal the final results. Intermediate layers gradually increase the level of abstraction, so what began as for example, pixels in an image, gradually resolves into things such as object boundaries, and then into real-world objects such as letters and faces. Single layer and unlayered networks are also used. Multiple connection patterns have been used. Traditionally, they are fully connected, with every neuron in one layer connecting to every neuron in the next layer. However, in convolutional neural networks, some layers are convolutional, meaning each neuron in one layer is connected to a subset of neurons in the previous layer, such as those representing one section of an image. In most neural networks, the outputs of the neurons in one layer are connected only to neurons in the immediately following layer (directed acyclic graph), meaning information only flows forward from one layer to the next. These are known as feedforward networks. In contrast, networks that allow connections between neurons in the same or previous layers are known as recurrent networks.
Training/learning involves adjusting the weights of the network to improve the accuracy of the result. NNs typically require vast numbers of sample inputs (far more than biological brains) to achieve a given level of function. This is done by minimizing the observed errors among sample observations. Training takes place before a network is deployed, and (unlike brains) does not continue thereafter. Instead, the network may be retrained from scratch as more sample data becomes available. Empirical risk minimization adjusts node and link weights to minimize the difference (empirical risk), between the predicted output and the known values in the training samples. A defined loss function measures the degree of error. Backpropagation spreads the error (adjusts the weights) from the output nodes across the network to the input nodes. The intent is to allow the network to process data that is not included in the training samples. As long as the value of the loss function (its cost) continues to decline, the network is continuing to improve. The function typically produces a statistic whose value is only approximate. When the cost is low, the difference between the output (e.g. almost certainly a cat) and the correct answer (a cat) is small. Most learning models can be viewed as a straightforward application of optimization theory and statistical estimation. Learning typically ends when additional observations do not usefully reduce the cost. The cost typically approaches, but does not reach, 0. If no feasible amount of sample data yields a low cost, training is labeled a failure.
A loss function measures the discrepancy between a model's predictions and the desired values, usually as a non-negative number where smaller values are better and a perfect prediction incurs a loss of 0. The choice of loss function depends on the type of problem. In regression tasks, where the output is a continuous value, common choices are mean squared error and mean absolute error, which penalize the squared error and the absolute error respectively. In classification tasks, the standard choice is cross-entropy, which treats the model's output as a probability distribution over classes and is equivalent to maximizing the likelihood of the data under the model. Margin-based losses such as hinge loss are also used. Not all losses require labeled targets: self-supervised methods train on augmented views of unlabeled data, and generative models such as variational autoencoders combine a reconstruction loss with a Kullback–Leibler divergence term.
3. Dataset Ingestion, Preprocessing & Stochastic Augmentations
Backpropagation is a method used to adjust the connection weights to compensate for errors found during learning. The error amount is essentially divided among the connections. Technically, backpropagation calculates the gradient (the derivative) of the loss function associated with a given state with respect to the weights. The weight updates can be done via stochastic gradient descent or other methods, such as extreme learning machines, "no-prop" networks, training without backtracking, "weightless" networks, and non-connectionist neural networks.
A hyperparameter is a parameter defining any configurable part of the network and learning process whose value is set prior to training. Examples of hyperparameters include learning rate, sample batch size, number of nodes and layers, etc . The performance of a neural network is strongly influenced by hyperparameter choices, and thus may be adjusted during training (typically between training runs), a process called hyperparameter tuning or hyperparameter optimization.
The learning rate defines the size of the corrective steps that the model takes to adjust for errors in each observation. A high learning rate shortens training time, but with lower ultimate accuracy, while a lower learning rate takes longer, but with the potential for greater accuracy. Optimizations such as quickprop are primarily aimed at accelerating learning. In order to avoid oscillations such as connection weights that cycle between high and low values, and to improve the rate of convergence, refinements use an adaptive learning rate that increases or decreases as appropriate. The concept of momentum allows the balance between the gradient and the previous change to be weighted such that the weight adjustment depends to some degree on the previous change. A momentum close to 0 emphasizes the gradient, while a value close to 1 emphasizes the last change.
In applications such as playing video games, an actor takes a string of actions, receiving a generally unpredictable response from the environment (the game) after each one. The goal is to win the game (get the highest score). The cost is the inverse of the score. In reinforcement learning, the aim is to weight the network to increase the score. After each action the game generates an observation and an instantaneous cost, according to its rules. The rules and the long-term cost can only be estimated. At any juncture, the agent decides whether to explore new actions to uncover their costs or to exploit prior learning to proceed more quickly. Formally, the environment is modeled as a Markov decision process (MDP) with states s 1 , . . . , s n ∈ S {\displaystyle \textstyle {s_{1},...,s_{n}}\in S} and actions a 1 , . . . , a m ∈ A {\displaystyle \textstyle {a_{1},...,a_{m}}\in A} . Because the state transitions (policies) are not known, probability distributions are used instead: the instantaneous cost distribution P ( c t | s t ) {\displaystyle \textstyle P(c_{t}|s_{t})} , the observation distribution P ( x t | s t ) {\displaystyle \textstyle P(x_{t}|s_{t})} and the transition distribution P ( s t + 1 | s t , a t ) {\displaystyle \textstyle P(s_{t+1}|s_{t},a_{t})} , while a policy is defined as the conditional distribution over actions given the observations. Taken together, the two define a Markov chain (MC). The aim is to discover the lowest-cost MC. NNs serve as the learning component. Dynamic programming coupled with NNs (giving neurodynamic programming) has been applied to problems such as vehicle routing, video games, natural resource management and medicine because of NNs' ability to mitigate cost even when reducing the discretization grid density for numerically approximating control tasks.
In situation s perform action a; Receive consequence situation s'; Compute emotion of being in consequence situation v(s'); Update crossbar memory w'(a,s) = w(a,s) + v(s').
The backpropagated value (secondary reinforcement) is the emotion toward the consequence situation. The CAA exists in two environments, its behavioral environment, and its genetic environment, from which it receives initial emotions (only once) about to be encountered situations in the behavioral environment. Having received the genome vector (species vector) from the genetic environment, the CAA will learn a goal-seeking behavior, in the behavioral environment that contains both desirable and undesirable situations.
Neuroevolution can create NN topologies and weights using evolutionary computation. It is competitive with gradient descent approaches. Neuroevolution may be less prone to get caught in "dead ends".
4. Neural Network Backbone, Layer Specifications & Loss Formulations
Learning can be either stochastic or batch. Stochastic learning creates a weight adjustment for each sample. In batch learning, weights are adjusted based on a batch of inputs, accumulating errors over the batch. Stochastic learning introduces "noise" into the process, using the local gradient calculated from one data point; this reduces the chance of the network getting stuck in local minima. However, batch learning typically yields a faster, more stable descent to a local minimum, since each update is performed in the direction of the batch's average error. A common compromise is to use "mini-batches", small batches with samples in each batch selected stochastically from the entire data set.
Applications that must generalize well to unseen examples must avoid over-training. This arises in convoluted or over-specified systems when the network capacity is much larger than needed. Two approaches address over-training. Cross-validation and similar techniques can check for over-training and select appropriate hyperparameters to minimize generalization error. Regularization in a probabilistic (Bayesian) framework can be performed by selecting a larger prior probability over simpler models; but also in statistical learning theory, where the goal is to minimize two quantities: the 'empirical risk' and the 'structural risk', which roughly corresponds to the error over the training set and the predicted error in unseen data due to overfitting.
Supervised neural networks that use a mean squared error (MSE) cost function can use formal statistical methods to determine the confidence of the trained model. The MSE on a validation set can be used as an estimate for variance. This value can then be used to calculate the confidence interval of network output, assuming a normal distribution. A confidence analysis made this way is statistically valid as long as the output probability distribution stays the same and the network is not modified. By adopting a softmax activation function, a generalization of the logistic function, on the output layer of the neural network (or a softmax component in a component-based network) for categorical target variables, the outputs can be interpreted as posterior probabilities. This is useful in classification as it gives a certainty measure. The softmax activation function is:
y i = e x i ∑ j = 1 c e x j {\displaystyle y_{i}={\frac {e^{x_{i}}}{\sum _{j=1}^{c}e^{x_{j}}}}}
Function approximation, or regression analysis, (including time series prediction, fitness approximation, and modeling) Data processing (including filtering, clustering, blind source separation, and compression) Nonlinear system identification and control (including vehicle control, trajectory prediction, adaptive control, process control, and natural resource management) Pattern recognition (including radar systems, face identification, signal classification, novelty detection, 3D reconstruction, object recognition, and sequential decision making) Sequence recognition (including gesture, speech, and handwritten and printed text recognition) Sensor data analysis (including image analysis) Robotics (including directing manipulators and prostheses) Data mining (including knowledge discovery in databases) Finance (such as ex-ante models for specific financial long-run forecasts and artificial financial markets) Quantum chemistry General game playing Generative AI Data visualization Machine translation Social network filtering E-mail spam filtering Medical diagnosis Disaster response NNs have been used to diagnose cancers and to distinguish highly invasive cancer cell lines from less invasive lines using only cell shape data. NNs have been used to accelerate reliability analysis of infrastructures subject to natural disasters and to predict settling in building foundations. They are used to mitigate flooding by modelling rainfall-runoff. NNs have been used for building black-box models in geoscience: hydrology, ocean modelling and coastal engineering, and geomorphology. NNs have been employed in cybersecurity, with the objective to discriminate between legitimate and malicious activities. For example, machine learning has been used for classifying Android malware, for identifying domains belonging to threat actors and for detecting URLs posing a security risk. Research is underway on penetration testing, for detecting botnets, credit cards frauds, and network intrusions. NNs have been proposed as a tool to solve partial differential equations in physics and simulate the properties of many-body open quantum systems. In brain research NNs have studied short-term behavior of individual neurons, the dynamics of neural circuitry arise from interactions between individual neurons and how behavior can arise from abstract neural modules that represent complete subsystems. Studies considered long-and short-term plasticity of neural systems and their relation to learning and memory from the individual neuron to the system level. NNs show promise in profiling a user's interests from photos and discovering new stable materials by efficiently predicting the total energy of crystals.
Error-based monitoring: comparing current predictions against ground-truth labels. This approach directly quantifies predictive performance but may be impractical when labels are delayed or impractical to obtain. Data distribution monitoring: detecting changes in the input data distribution using statistical tests, divergence measures, or density-ratio estimation. Representation monitoring: tracking the distribution of internal embeddings or hidden-layer features. Shifts in the latent representation can indicate nonstationarity even when labels are unavailable. Statistical methods such as statistical process control charts have been adapted for this purpose.
A Brief Introduction to Neural Networks (D. Kriesel) – Illustrated, bilingual manuscript about neural networks; Topics so far: Perceptrons, Backpropagation, Radial Basis Functions, Recurrent Neural Networks, Self Organizing Maps, Hopfield Networks. Review of Neural Networks in Materials Science Archived 7 June 2015 at the Wayback Machine Artificial Neural Networks Tutorial in three languages (Univ. Politécnica de Madrid) Another introduction to NN Next Generation of Neural Networks Archived 24 January 2011 at the Wayback Machine – Google Tech Talks Performance of Neural Networks Neural Networks and Information Archived 9 July 2009 at the Wayback Machine Sanderson G (5 October 2017). "But what is a Neural Network?". 3Blue1Brown. Archived from the original on 7 November 2021 – via YouTube. Hakim MA, Alam MI (2025). "Biologically inspired neural network layer with homeostatic regulation and adaptive repair mechanisms". Scientific Reports. 15 (1). Nature Publishing Group: 33903. Bibcode:2025NatSR..1533903H. doi:10.1038/s41598-025-09114-8. ISSN 2045-2322. PMC 12484884. PMID 41028030.
5. Training Protocol, Convergence Behavior & Experimental Evaluation
Deep Demand Forecasting with Amazon SageMaker This project provides an end-to-end solution for Demand Forecasting task using a new state-of-the-art Deep Learning model LSTNet available in GluonTS and Amazon SageMaker . Overview How Does the Input Data Look Like? The input data is a multi-variate time-series . An example includes hourly electricity consumption of 321 users over the period of 41 months. Here is a snapshot of the normalized data How to Prepare Your Data to Feed into the Model? We have provided example of how to feed your time-series data with GluonTS in the notebook. To convert CSV data or other formats to GluonTS format, please see the customization . What Are the Outputs? A trained LSTNet model and A SageMaker endpoint that can predict the future (multi-variate) values given a prediction interval For example, we can estimate the hourly electricity consumption of 321 users for the coming week. What Algorithm is Used? We have implemented LSTNet which is a state-of-the-art Deep Learning model and is available in GluonTS . What is the Estimated Cost? Running the solution end-to-end costs less than $5 USD. Please make sure you have read the cleaning up part here . How does the Data Flow look like? Solution Details Demand forecasting uses historical time-series data to help streamline the supply-demand decision-making process across businesses. Examples include predicting the number of Customer representatives to hire for multiple locations in the next month Product sales across multiple regions in the next quarter Cloud server usage for next day for a video streaming service Electricity consumption for multiple regions over the next week IoT devices and sensors such as energy consumption Deep Learning for Time Series Forecasting The status quo approaches for time-series forecasting include: Auto Regressive Integrated Moving Average (ARIMA) for univariate time-series data and Vector Auto-Regression (VAR) for multi-variate time-series data These methods often require tedious data preprocessing and features generation prior to model training. One main advantage of Deep Learning (DL) methods such as LSTNet is automating the feature generation step prior to model training such as incorporating various data normalization, lags, different time scales, some categorical data, dealing with missing values, etc. with better prediction power and fast GPU-enabled training and deployment. Please check out our blog post for more details. Getting Started You will need an AWS account to use this solution. Sign up for an account here . To run this JumpStart 1P Solution and have the infrastructure deploy to your AWS account you will need to create an active SageMaker Studio instance (see Onboard to Amazon SageMaker Studio ). When your Studio instance is Ready , use the instructions in SageMaker JumpStart to 1-Click Launch the solution. The solution artifacts are included in this GitHub repository for reference. Note : Solutions are available in most regions including us-west-2, and us-east-1. Caution : Cloning this GitHub repository and running the code manually could lead to unexpected issues! Use the AWS CloudFormation template. You'll get an Amazon SageMaker Notebook instance that's been correctly setup and configured to access the other resources in the solution. Contents cloudformation/ deep-demand-forecast.yaml : The root cloudformation nested stack which creates the AWS stack for this solution deep-demand-forecast-sagemaker-notebook-instance.yaml : Creates SageMaker notebook instance deep-demand-forecast-permissions.yaml : Manages all the permission necessary to launch the stack deep-demand-forecast-endpoint.yaml : Creates demo endpoint using in demo.ipynb solution-assistant : Deletes the created resources such as endpoint, S3 bucket etc. during cleanup src/ preprocess/ container/ : To build and register the preprocessing ECR job Dockerfile : Docker container config build_and_push.sh : Build and push bash scripts used in deep-demand-forecast.ipynb requirements.txt : Dependencies for preprocess.py container_build/ : Uses CodeBuild to the build the container for ECR preprocess.py : Preprocessing script deep_demand_forecast/ : Contains the train and inference code train.py : SageMaker train code inference.py : SageMaker inference code data.py : GluonTS data preparation metrics.py : A training metric monitor.py : Preparing results for visualization utils.py : Helper functions requirements.txt : Dependencies for SageMaker MXNet Estimator demo.ipynb : Demo notebook to quickly get some predictions from the demo endpoint deep-demand-forecast.ipynb : See below What Does deep-demand-forecast.ipynb Offer? The notebook trains an LSTNet estimator on electricity consumption data which is multivariate time-series dataset capturing the electricity consumption (in kW) with 15min frequency from 2011-01-01 to 2014-05-26 . We compare the model performance by visualizing the metrics MASE vs. sMAPE . Finally, we deploy an endpoint for the trained model and can interactively compare its performance by comparing the train, test data and predictions. For example, here, re-training with more epochs would be helpful to increase the model performance and we can re-deploy. Architecture Overview Here is architecture for the end-to-end training and deployment process The input data located in an Amazon S3 bucket The provided SageMaker notebook that gets the input data and launches the later stages below Preprocessing step to normalize the input data. We use SageMaker processing job that is designed as a microservice . This allows users to build and register their own Docker image via Amazon ECR and execute the job using Amazon SageMaker Training an LSTNet model using the previous preprocessed step and evaluating its results using Amazon SageMaker. If desired, one can deploy the trained model and create a SageMaker endpoint SageMaker endpoint created from the previous step, is an HTTPS endpoint and is capable of producing predictions Monitoring the training and deployed model via Amazon CloudWatch Here is the architecture of the inference The input data, located in an Amazon S3 bucket From SageMaker notebook, normalize the new input data using the statistics of the training data Sending the requests to the SageMaker endpoint Predictions Cleaning Up When you've finished with this solution, make sure that you delete all unwanted AWS resources. AWS CloudFormation can be used to automatically delete all standard resources that have been created by the solution and notebook. Go to the AWS CloudFormation Console, and delete the parent stack. Choosing to delete the parent stack will automatically delete the nested stacks. Caution: You need to manually delete any extra resources that you may have created in this notebook. Some examples include, extra Amazon S3 buckets (to the solution's default bucket), extra Amazon SageMaker endpoints (using a custom name), and extra Amazon ECR repositories. Customization To use your own data, please take a look at Extensive GluonTS tutorials Consult with the dataset API Useful Resources Amazon SageMaker Getting Started Amazon SageMaker Developer Guide Amazon SageMaker Python SDK Documentation AWS CloudFormation User Guide License This project is licensed under the Apache-2.0 License.
This project provides an end-to-end solution for Demand Forecasting task using a new state-of-the-art Deep Learning model LSTNet available in GluonTS and Amazon SageMaker .
An example includes hourly electricity consumption of 321 users over the period of 41 months. Here is a snapshot of the normalized data
We have provided example of how to feed your time-series data with GluonTS in the notebook. To convert CSV data or other formats to GluonTS format, please see the customization .
For example, we can estimate the hourly electricity consumption of 321 users for the coming week.
We have implemented LSTNet which is a state-of-the-art Deep Learning model and is available in GluonTS .
Running the solution end-to-end costs less than $5 USD. Please make sure you have read the cleaning up part here .
Model Figures & Evaluation Curves (4 Images)
Deep Learning Execution Pipeline
(Preprocessing)
(Feature Extraction)
(Self-Attention)
(Loss Optimization)