Complete Technical Paper Content — Ai Security Projects
1. Executive Abstract, Problem Formulation & Background
An AI agent, also known as agentic AI, is an artificial intelligence program that can make decisions, pursue goals, and take actions independently of human programmers. Agentic AI contrasts with tool-like AI use for narrow, specific tasks such as answering questions, as with chatbots in the non-agentic forms that were common since 2023, or with traditional machine learning algorithms. While there is no universally agreed-upon definition of an AI agent, common attributes of AI agents include goal-directed behavior, use of external tools, the ability to interact with and modify an external environment, and the ability to autonomously perform multi-step tasks. Their control flow is frequently driven by large language models (LLMs). Agent systems may also include memory components, planning logic, tool interfaces, and orchestration software for coordinating agent components.
The theoretical underpinnings for artificial agents emerged in the mid 20th century, with establishment of cybernetics and artificial intelligence. Oliver Selfridge's 1958 "Pandemonium: A Paradigm for Learning" paper was an important early theoretical contribution in establishing agent oriented architecture. Practical implementations of agents for real world applications began to become widespread in the 1990s, after the introduction of the belief–desire–intention software model (BDI), and agent-oriented programming. Harvard professor Milind Tambe notes that in the 1990s, the definition of an AI agent was not clear. Pure digital agents were deployed in computer infrastructure for purposes such as monitoring, while agents connected to real-world sensors and actuators were increasingly used in industrial control systems. Early artificial agents tended to have simple if then logic, which expanded over time into large decision tree models. By the early 2010s, products like Siri and Alexa were released, and were sometimes called AI agents, though they lacked the general purpose reasoning ability of later agents run by large language models (LLMs). The development of LLMs during the late 2010s and early 2020s introduced new approaches to AI agents. Models such as GPT-3 demonstrated the ability to generate and understand natural language at a much larger scale than many earlier language systems. Academics began to study LLM agents from 2018. Deployment of such agents began to accelerate in late 2023 after OpenAI's "function-calling" API was made available, and especially after Anthropic's late 2024 introduction of Model Context Protocol (MCP), a standardised way for LLM agents to gain contextual awareness, and to act on the world by calling various external tools. The term "agentic" only began to be used with any frequency in 2024, popularised in part by researcher Andrew Ng.
Ken Huang proposed an AI agent reference architecture that consists of seven interconnected layers, with each layer building on the functionality of the layers beneath it:
Layer 1: Foundation models – provide the datasets that power the agent. Layer 2: Data operations – manages the data infrastructure required for AI agent operations, including vector databases, data loaders, and RAG. Layer 3: Agent frameworks – software that manages the AI agents. Layer 4: Deployment and infrastructure – the technical foundation of the AI agents. Layer 5: Evaluation and observability – the safety and performance of AI agents. Layer 6: Security and compliance – a protective framework for safe operation and compliance with regulatory boundaries. At this layer, security and compliance features embedded into all the AI agent stack layers are integrated together. Layer 7: Agent ecosystem – represents the AI agents' interface with real-world applications and users.
An agent harness, or agent scaffold, is the software layer surrounding a large language model that enables it to function as an AI agent. It commonly manages prompts, context, tool use, memory, execution state, operational constraints, sandboxes, permissions, and the processing of results. The harness connects the model to internal and external computer hardware, software, data files, databases, web browsers, command-line interfaces, and application programming interfaces, while controlling how the agent accesses and uses these resources to complete multi-step tasks.
Prompt chaining: A sequence where the output of one step serves as the input for the next. Routing: Directing an input to a specialized downstream task or tool. Parallelization: The simultaneous execution of multiple tasks. Sequential processing: A fixed, linear progression of tasks through a predefined pipeline. Planner-critic: An iterative pattern where one agent generates a proposal and another evaluates it to provide feedback for refinement.
STRIDE: A Microsoft model that identifies threats across six categories: spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege. MITRE ATLAS: A knowledge base of adversary tactics and techniques for AI systems. OWASP GenAI Security Project: Provides guidance on vulnerabilities associated with generative AI and large language model integration, including guidelines specifically for agentic applications. MAESTRO: A framework from the Cloud Security Alliance for assessing risks in AI systems throughout their lifecycle.
Adversarial machine learning is the study of the attacks on machine learning algorithms, and of the defenses against such attacks. Machine learning techniques are mostly designed to work on specific problem sets, under the assumption that the training and test data are generated from the same statistical distribution (IID). However, this assumption is often violated in practical high-stake applications, where users may intentionally supply fabricated data that violates the statistical assumption. Most common attacks in adversarial machine learning include evasion attacks, data poisoning attacks, Byzantine attacks and model extraction.
2. Related Work, Literature Survey & Algorithmic Foundations
Classifier influence: An attack can influence the classifier by disrupting the classification phase. This may be preceded by an exploration phase to identify vulnerabilities. The attacker's capabilities might be restricted by the presence of data manipulation constraints. Security violation: An attack can supply malicious data that gets classified as legitimate. Malicious data supplied during training can cause legitimate data to be rejected after training. Specificity: A targeted attack attempts to allow a specific intrusion/disruption. Alternatively, an indiscriminate attack creates general mayhem. This taxonomy has been extended into a more comprehensive threat model that allows explicit assumptions about the adversary's goal, knowledge of the attacked system, capability of manipulating the input data/system components, and on attack strategy. This taxonomy has further been extended to include dimensions for defense strategies against adversarial attacks.
For instance, intrusion detection systems are often trained using collected data. An attacker may poison this data by injecting malicious samples during operation that subsequently disrupt retraining. A 2025 study by researchers at the AI Security Institute, Anthropic and the Alan Turing Institute found that the number of poisoned documents needed to introduce a language-model backdoor remained approximately constant across the model and training-set sizes tested. The researchers pretrained models with 600 million to 13 billion parameters on datasets containing 6 billion to 260 billion tokens. Adding 250 poisoned documents was sufficient to introduce a denial-of-service backdoor at every tested scale, causing the models to generate gibberish in response to a trigger phrase. In the 13-billion-parameter model, these documents accounted for approximately 0.00016% of the training tokens. The authors noted that the study had not established whether backdoors introduced during pretraining would survive realistic post-training. Co-author Vasilios Mavroudis suggested that backdoors selectively reducing a model's usefulness to particular groups of users could be harder to detect than overt failures. Data poisoning techniques can also be applied to text-to-image models to alter their output, which is used by artists to defend their copyrighted works or their artistic style against imitation. Data poisoning can also happen unintentionally through model collapse, where models are trained on synthetic data.
adversarial training of a linear regression model with input perturbations restricted by the infinity-norm closely resembles Lasso regression, and that adversarial training of a linear regression model with input perturbations restricted by the 2-norm closely resembles Ridge regression.
Gradient-based evasion attack Fast Gradient Sign Method (FGSM) Projected Gradient Descent (PGD) Carlini and Wagner (C&W) attack Adversarial patch attack
Repeat this for v 2 , v 3 , … {\displaystyle v_{2},v_{3},\dots } until the desired level of error in the classifier is reached.It was discovered when the authors designed a simple baseline to compare with a previous black-box adversarial attack algorithm based on gaussian processes, and were surprised that the baseline worked even better.
argmax k = 1 , . . . , K f k ( x ^ ) ≠ y , | | x ^ − x | | p ≤ ϵ and x ^ ∈ [ 0 , 1 ] d {\displaystyle {\text{argmax}}_{k=1,...,K}f_{k}({\hat {x}})\neq y,||{\hat {x}}-x||_{p}\leq \epsilon {\text{ and }}{\hat {x}}\in [0,1]^{d}}
In other words, finding some perturbed adversarial example x ^ {\textstyle {\hat {x}}} such that the classifier incorrectly classifies it to some other class under the constraint that x ^ {\textstyle {\hat {x}}} and x {\textstyle x} are similar. The paper then defines loss L {\textstyle L} as L ( f ( x ^ ) , y ) = f y ( x ^ ) − max k ≠ y f k ( x ^ ) {\textstyle L(f({\hat {x}}),y)=f_{y}({\hat {x}})-\max _{k\neq y}f_{k}({\hat {x}})} and proposes the solution to finding adversarial example x ^ {\textstyle {\hat {x}}} as solving the below constrained optimization problem:
min x ^ ∈ [ 0 , 1 ] d L ( f ( x ^ ) , y ) , s.t. | | x ^ − x | | p ≤ ϵ {\displaystyle \min _{{\hat {x}}\in [0,1]^{d}}L(f({\hat {x}}),y),{\text{ s.t. }}||{\hat {x}}-x||_{p}\leq \epsilon }
3. Dataset Ingestion, Preprocessing & Stochastic Augmentations
The result in theory is an adversarial example that is highly confident in the incorrect class but is also very similar to the original image. To find such example, Square Attack utilizes the iterative random search technique to randomly perturb the image in hopes of improving the objective function. In each step, the algorithm perturbs only a small square section of pixels, hence the name Square Attack, which terminates as soon as an adversarial example is found in order to improve query efficiency. Finally, since the attack algorithm uses scores and not gradient information, the authors of the paper indicate that this approach is not affected by gradient masking, a common technique formerly used to prevent evasion attacks.
Targeted: min x ′ d ( x ′ , x ) subject to C ( x ′ ) = c ∗ {\displaystyle {\textbf {Targeted:}}\min _{x^{\prime }}d(x^{\prime },x){\text{ subject to }}C(x^{\prime })=c^{*}}
Untargeted: min x ′ d ( x ′ , x ) subject to C ( x ′ ) ≠ C ( x ) {\displaystyle {\textbf {Untargeted:}}\min _{x^{\prime }}d(x^{\prime },x){\text{ subject to }}C(x^{\prime })\neq C(x)}
To solve this problem, the attack proposes the following boundary function S {\textstyle S} for both the untargeted and targeted setting:
S ( x ′ ) := { max c ≠ C ( x ) F ( x ′ ) c − F ( x ′ ) C ( x ) , (Untargeted) F ( x ′ ) c ∗ − max c ≠ c ∗ F ( x ′ ) c , (Targeted) {\displaystyle S(x^{\prime }):={\begin{cases}\max _{c\neq C(x)}{F(x^{\prime })_{c}}-F(x^{\prime })_{C(x)},&{\text{(Untargeted)}}\\F(x^{\prime })_{c^{*}}-\max _{c\neq c^{*}}{F(x^{\prime })_{c}},&{\text{(Targeted)}}\end{cases}}}
This can be further simplified to better visualize the boundary between different potential adversarial examples:
S ( x ′ ) > 0 ⟺ { a r g m a x c F ( x ′ ) ≠ C ( x ) , (Untargeted) a r g m a x c F ( x ′ ) = c ∗ , (Targeted) {\displaystyle S(x^{\prime })>0\iff {\begin{cases}argmax_{c}F(x^{\prime })\neq C(x),&{\text{(Untargeted)}}\\argmax_{c}F(x^{\prime })=c^{*},&{\text{(Targeted)}}\end{cases}}}
With this boundary function, the attack then follows an iterative algorithm to find adversarial examples x ′ {\textstyle x^{\prime }} for a given image x {\textstyle x} that satisfies the attack objectives.
4. Neural Network Backbone, Layer Specifications & Loss Formulations
Initialize x {\textstyle x} to some point where S ( x ) > 0 {\textstyle S(x)>0}
Iterate below Boundary search Gradient update Compute the gradient Find the step size Boundary search uses a modified binary search to find the point in which the boundary (as defined by S {\textstyle S} ) intersects with the line between x {\textstyle x} and x ′ {\textstyle x^{\prime }} . The next step involves calculating the gradient for x {\textstyle x} , and update the original x {\textstyle x} using this gradient and a pre-chosen step size. HopSkipJump authors prove that this iterative algorithm will converge, leading x {\textstyle x} to a point right along the boundary that is very close in distance to the original image. However, since HopSkipJump is a proposed black box attack and the iterative algorithm above requires the calculation of a gradient in the second iterative step (which black box attacks do not have access to), the authors propose a solution to gradient calculation that requires only the model's output predictions alone. By generating many random vectors in all directions, denoted as u b {\textstyle u_{b}} , an approximation of the gradient can be calculated using the average of these random vectors weighted by the sign of the boundary function on the image x ′ + δ u b {\textstyle x^{\prime }+\delta _{u_{b}}} , where δ u b {\textstyle \delta _{u_{b}}} is the size of the random vector perturbation:
∇ S ( x ′ , δ ) ≈ 1 B ∑ b = 1 B ϕ ( x ′ + δ u b ) u b {\displaystyle \nabla S(x^{\prime },\delta )\approx {\frac {1}{B}}\sum _{b=1}^{B}\phi (x^{\prime }+\delta _{u_{b}})u_{b}}
The result of the equation above gives a close approximation of the gradient required in step 2 of the iterative algorithm, completing HopSkipJump as a black box attack.
a d v x = x + ϵ ⋅ s i g n ( Δ x J ( θ , x , y ) ) {\displaystyle adv_{x}=x+\epsilon \cdot sign(\Delta _{x}J(\theta ,x,y))}
One important property of this equation is that the gradient is calculated with respect to the input image since the goal is to generate an image that maximizes the loss for the original image of true label y {\textstyle y} . In traditional gradient descent (for model training), the gradient is used to update the weights of the model since the goal is to minimize the loss for the model on a ground truth dataset. The Fast Gradient Sign Method was proposed as a fast way to generate adversarial examples to evade the model, based on the hypothesis that neural networks cannot resist even linear amounts of perturbation to the input. FGSM has shown to be effective in adversarial attacks for image classification and skeletal action recognition.
min ( | | δ | | p ) subject to C ( x + δ ) = t , x + δ ∈ [ 0 , 1 ] n {\displaystyle \min(||\delta ||_{p}){\text{ subject to }}C(x+\delta )=t,x+\delta \in [0,1]^{n}}
Here the objective is to minimize the noise ( δ {\textstyle \delta } ), added to the original input x {\textstyle x} , such that the machine learning algorithm ( C {\textstyle C} ) predicts the original input with delta (or x + δ {\textstyle x+\delta } ) as some other class t {\textstyle t} . However instead of directly the above equation, Carlini and Wagner propose using a new function f {\textstyle f} such that:
5. Training Protocol, Convergence Behavior & Experimental Evaluation
C ( x + δ ) = t ⟺ f ( x + δ ) ≤ 0 {\displaystyle C(x+\delta )=t\iff f(x+\delta )\leq 0}
min ( | | δ | | p ) subject to f ( x + δ ) ≤ 0 , x + δ ∈ [ 0 , 1 ] n {\displaystyle \min(||\delta ||_{p}){\text{ subject to }}f(x+\delta )\leq 0,x+\delta \in [0,1]^{n}}
min ( | | δ | | p + c ⋅ f ( x + δ ) ) , x + δ ∈ [ 0 , 1 ] n {\displaystyle \min(||\delta ||_{p}+c\cdot f(x+\delta )),x+\delta \in [0,1]^{n}}
Carlini and Wagner then propose the use of the below function in place of f {\textstyle f} using Z {\textstyle Z} , a function that determines class probabilities for given input x {\textstyle x} . When substituted in, this equation can be thought of as finding a target class that is more confident than the next likeliest class by some constant amount:
f ( x ) = ( [ max i ≠ t Z ( x ) i ] − Z ( x ) t ) + {\displaystyle f(x)=([\max _{i\neq t}Z(x)_{i}]-Z(x)_{t})^{+}}
When solved using gradient descent, this equation is able to produce stronger adversarial examples when compared to fast gradient sign method that is also able to bypass defensive distillation, a defense that was once proposed to be effective against adversarial examples.
Threat modeling – formalize the attackers goals and capabilities with respect to the target system. Attack simulation – formalize the optimization problem the attacker tries to solve according to possible attack strategies. Attack impact evaluation Countermeasure design Noise detection (for evasion-based attack) Information laundering – Alter the information received by adversaries (for model stealing attacks)
Secure learning algorithms Byzantine-resilient algorithms Multiple classifier systems AI-written algorithms. AIs that explore the training environment; for example, in image recognition, actively navigating a 3D environment rather than passively scanning a fixed set of 2D images. Privacy-preserving learning Ladder algorithm for Kaggle-style competitions Game theoretic models Sanitizing training data Adversarial training Backdoor detection algorithms Gradient masking/obfuscation techniques: to prevent the adversary exploiting the gradient in white-box attacks. This family of defenses is deemed unreliable as these models are still vulnerable to black-box attacks or can be circumvented in other ways. Ensembles of models have been proposed in literature, which have shown to be ineffective against evasion attacks but effective against data poisoning attacks. Moving target defense
Model Figures & Evaluation Curves (4 Images)
Deep Learning Execution Pipeline
(Preprocessing)
(Feature Extraction)
(Self-Attention)
(Loss Optimization)