Anomaly Detection in Network Traffic Using Autoencoders and Generative Adversarial Networks

Introduction

The security of networked information systems constitutes one of the defining technical and strategic challenges of the contemporary digital era. As governmental institutions, financial organisations, healthcare providers, and operators of critical infrastructure have become increasingly dependent upon interconnected computational infrastructure, the consequences of successful cyberattacks have grown correspondingly severe — encompassing financial losses, data exfiltration, disruption of essential services, and, in the case of attacks against industrial control systems, risks to physical safety. Cyber attacks, including denial-of-service campaigns, unauthorised access attempts, and the deployment of malicious code, inflict substantial damage across economic and social domains, and the rate at which novel attack variants are observed in operational environments shows no sign of abating. ^[1] The proliferation of internet-connected devices across consumer, industrial, and governmental sectors has dramatically expanded the attack surface available to adversarial actors, whose capabilities range from opportunistic exploitation of publicly disclosed vulnerabilities to coordinated advanced persistent threat campaigns sustained over extended periods. This structural transformation of the networked environment places commensurate demands upon defensive technologies and motivates sustained investigation into detection mechanisms capable of identifying malicious activity with high fidelity and low operational cost.

Signature-based intrusion detection systems have historically constituted the primary technological response to the challenge of network-based threats. These systems maintain curated repositories of attack signatures — patterns encoding known malicious traffic characteristics — against which observed network flows are matched; a flow whose features correspond to a stored signature is classified as malicious and subjected to configured response actions. The practical deployability of signature-based detection is well established: such systems can operate at high throughput, produce deterministic outputs, and generate relatively few false positives on traffic categories for which signatures have been developed. These properties have made signature-based network intrusion detection systems a standard component of enterprise and governmental security architectures for several decades. However, a structural limitation inherent to the signature-based paradigm constrains its effectiveness against the evolving threat landscape: a system that can only detect what it has previously seen cannot, by construction, identify novel attacks for which no signature has been authored. Zero-day exploits — vulnerabilities exploited before a patch or signature is available — and polymorphic malware variants designed to evade pattern matching are invisible to signature-based detection until the signature corpus is updated, a process that may lag the initial exploitation of a vulnerability by hours, days, or longer. The interval between initial exploitation and signature availability constitutes a critical exposure window during which signature-based systems provide no protection.

Anomaly-based detection addresses the zero-day limitation through a fundamentally different conceptual approach. Rather than defining malicious traffic by its resemblance to known attack patterns, anomaly-based methods define normal network behaviour through statistical or model-based characterisation of benign traffic, and identify as anomalous any observed flow that deviates significantly from this characterisation. The central advantage of this approach is its ability to flag previously unseen attack types, provided they induce statistical deviations from normal traffic that exceed the system's detection threshold — an assumption that holds for many but not all attack categories. The practical challenges associated with anomaly-based detection are, however, substantial. High-dimensional network traffic data exhibits complex distributional structure that resists characterisation by simple statistical models; the class imbalance between benign and malicious flows complicates model training and evaluation; and adversarially crafted attacks may be specifically designed to remain within the statistical envelope of normal traffic, evading anomaly-based detection while still achieving their objectives. These challenges have motivated an extensive body of research exploring machine learning and, more recently, deep learning architectures as the basis for anomaly detection systems capable of learning expressive representations of normal traffic from unlabelled or partially labelled data.

Among the deep learning architectures investigated for network anomaly detection, autoencoders have emerged as a particularly natural and widely studied choice for the unsupervised setting. An autoencoder is trained to reconstruct its input through a compressed latent representation; when trained exclusively on benign traffic, it acquires an implicit model of the statistical structure of normal network flows. Anomalous flows, whose distributional characteristics differ from those encountered during training, are expected to yield elevated reconstruction error at inference time, providing an anomaly score that can be thresholded to generate detection decisions. ^[23] This approach requires no labelled attack examples during training, making it applicable in operational environments where labelled data is scarce or unavailable. Generative adversarial networks offer a complementary perspective on the same unsupervised anomaly detection problem: by training a generator to produce synthetic samples resembling normal traffic and a discriminator to distinguish real from generated samples, the discriminator's confidence score acquires the semantics of a normality measure that can similarly be thresholded for detection. ^[21] Each architecture captures distinct statistical properties of the normal traffic distribution, and the combination of their respective anomaly signals through a fusion mechanism has the potential to yield detection performance exceeding that achievable by either system applied independently — a hypothesis that motivates the research reported in this thesis.

The present work addresses the problem of network traffic anomaly detection through the design, implementation, and empirical evaluation of a hybrid detection framework that integrates autoencoder-based reconstruction error scoring with generative adversarial network discriminator confidence into a unified detection signal. The primary research objective is to determine whether such integration yields statistically and practically significant improvements in detection performance relative to each constituent architecture deployed in isolation, evaluated across multiple benchmark corpora representative of diverse attack categories and traffic characteristics. The secondary objective is to characterise the computational requirements, inference latency, and calibration behaviour of the proposed framework with sufficient precision to assess its suitability for deployment in near-real-time network security monitoring contexts. ^[4] Both objectives are pursued through rigorous experimental methodology applied to the NSL-KDD, CICIDS-2017, and UNSW-NB15 benchmark datasets, which together provide coverage of a broad range of contemporary attack categories while enabling quantitative comparison with published detection literature. ^[15]

The research is organised around three principal hypotheses that structure the experimental investigation. The first hypothesis holds that a hybrid detection system combining autoencoder reconstruction error and GAN discriminator confidence through a weighted linear fusion mechanism will achieve a higher F1-score than either the standalone autoencoder or the standalone GAN discriminator across all three benchmark datasets employed, reflecting the complementary statistical properties of the two anomaly signals. The second hypothesis holds that the hybrid system will demonstrate improved detection performance on attack categories characterised by subtle or low-volume traffic deviations — specifically Remote-to-Local and User-to-Root attacks in the NSL-KDD taxonomy — relative to the autoencoder baseline, whose reconstruction error signal alone may lack the sensitivity required to distinguish such attacks from benign traffic variations. The third hypothesis holds that the computational overhead introduced by the GAN component relative to the autoencoder-only baseline will remain bounded such that per-sample inference latency does not exceed twice that of the standalone autoencoder, preserving the near-real-time applicability of the framework for deployment in live network monitoring environments. ^[2] These hypotheses are formulated to be empirically testable within the experimental design described in Chapter 2 and evaluated quantitatively in Chapter 3.

The thesis is structured in three chapters, each addressing a distinct phase of the research. Chapter 1 establishes the theoretical foundations necessary to situate the proposed framework within the existing literature on network anomaly detection and deep learning-based security systems. A structured taxonomy of network-based threats is developed, the operating principles of signature-based and statistical anomaly detection approaches are examined with particular attention to their respective limitations, and the architectural properties of autoencoders and generative adversarial networks relevant to the anomaly detection task are described in detail. Variants including variational autoencoders, long short-term memory autoencoders, and Wasserstein GAN formulations are reviewed in the context of their published applications to network security, and the theoretical basis for the complementarity hypothesis motivating the hybrid architecture is established. Chapter 2 presents the methodology and system design underlying the experimental evaluation. The dataset selection rationale, pre-processing pipeline, feature engineering procedures, and class imbalance mitigation strategy are described in full; the architectural specifications, training procedures, and hyperparameter configurations of the autoencoder baseline, the GAN component, and the hybrid integration layer are documented; and the evaluation metrics, calibration methodology, and statistical testing procedures are specified. Chapter 3 reports the experimental results and their analysis, including per-dataset performance breakdowns, attack-category-level detection analysis, ablation studies characterising the contribution of each component, computational profiling results, and a discussion of the framework's limitations alongside directions for future research that address the identified gaps through architectural extensions and evaluation on emerging traffic corpora. ^[3]

The contributions of this thesis are situated at the intersection of deep learning methodology and applied network security. While autoencoder-based and GAN-based anomaly detection have each been investigated independently in the existing literature, the systematic integration of these two architectural families into a calibrated hybrid framework — evaluated on multiple contemporary benchmark datasets with explicit attention to computational practicality — represents a contribution whose design and empirical characterisation address gaps that isolated single-architecture studies leave unresolved. ^[21] The explicit formulation and empirical testing of the complementarity hypothesis, the detailed documentation of inference latency and calibration, and the analysis of detection performance across attack categories of varying difficulty collectively aim to advance the evidence base available to practitioners and researchers working toward the deployment of machine-learning-based anomaly detection in operational network security environments.

Chapter 1. Theoretical Foundations of Network Anomaly Detection

1.1. The Landscape of Network Security Threats

The contemporary cybersecurity environment is characterised by an escalating frequency and sophistication of network-based attacks that pose substantial risks to governmental institutions, commercial enterprises, and critical infrastructure operators alike. Cyber attacks — including denial of service, unauthorised access, and the deployment of malicious code — cause tremendous damage across economic and social domains.^{[1, s. 453]} The proliferation of internet-connected devices, the mass migration of services to cloud architectures, and the growing interdependence of operational and information technology networks have collectively expanded the attack surface available to adversarial actors, whose capabilities range from opportunistic exploitation of known vulnerabilities to coordinated campaigns attributed to nation-state advanced persistent threat groups. This structural transformation of the networked environment demands corresponding advances in the detection and mitigation capabilities deployed by defenders, and motivates the development of principled machine-learning-based approaches capable of identifying previously unseen threats.

A structured taxonomy of network threats provides the necessary conceptual foundation for understanding what anomaly detection systems are designed to counter. Intrusion attempts constitute one of the most prevalent categories, encompassing port scanning and active reconnaissance alongside direct exploitation of vulnerabilities in network services, operating system components, and application protocols. The distinction between external intrusion and insider threat is analytically significant: the former typically produces traffic patterns associated with scanning, brute-force credential attacks, or exploitation payloads, while the latter may involve entirely legitimate protocol exchanges used to exfiltrate sensitive data, making detection substantially more difficult. Distributed denial-of-service (DDoS) attacks represent a distinct and operationally impactful category: volumetric floods based on UDP amplification or ICMP reflection overwhelm available bandwidth, protocol-level attacks exploiting TCP handshake mechanics exhaust server connection tables, and application-layer attacks targeting HTTP services consume computational resources through individually unremarkable requests issued at machine-generated rates.^{[9, s. 5]} Cloud infrastructure presents particular vulnerability to such attacks, as the shared nature of cloud resources makes them an attractive target for adversaries seeking to deny service to legitimate users across entire hosting environments.

Data exfiltration presents a particularly insidious detection challenge because the traffic volumes associated with it may be modest and its timing deliberately staggered to remain below threshold-based alarms. Techniques including DNS tunnelling, steganographic embedding within permitted protocol payloads, and low-and-slow exfiltration extended across days or weeks are specifically engineered to evade detection systems relying on rate-based or volume-based heuristics. Command-and-control communications associated with botnet infrastructure exhibit similarly evasive characteristics: domain generation algorithms produce large numbers of candidate rendezvous domains resistant to blocklist-based defences, while fast-flux networks rotate associated IP addresses at intervals that preclude effective blacklisting.^[13] The 2016 Mirai botnet attack, which leveraged hundreds of thousands of compromised IoT devices to generate record-scale DDoS traffic blocking the accessibility of several high-profile websites, illustrates the operational consequences of failing to detect malicious infrastructure before it is weaponised.^[13]

Industrial control systems represent a particularly critical class of targets whose compromise carries consequences extending beyond financial loss to physical safety. The integration of industrial control systems with external networks expands the attack surface considerably, making them more susceptible to cyber threats that exploit both information technology vulnerabilities and weaknesses specific to operational technology protocols.^{[11, s. 2]} Ensuring robust security measures has become paramount to safeguarding such systems, as the growing interconnectivity has exposed previously isolated process control environments to potential risks from remote adversaries. The structural diversity of threat categories described above — spanning volumetric attacks, stealthy exfiltration, and targeted operational disruption — motivates the development of detection methods that do not rely on pre-defined attack signatures but instead model the statistical structure of normal behaviour and identify departures from it, a paradigm examined in detail in the following subchapter.

1.2. Principles of Anomaly Detection in Network Traffic

Anomaly detection in the network security context may be formally defined as the identification of observations that deviate significantly from a learned or specified model of normal behaviour. Three canonical anomaly types are recognised in this domain. Point anomalies involve individual packets or network flows exhibiting aberrant feature values relative to the population distribution — for instance, a single connection displaying an unusually large data transfer volume directed to an unfamiliar external address. Contextual anomalies are observations that appear unremarkable in isolation but are anomalous given their temporal or environmental context, such as an administrative login occurring at atypical hours from a geographically improbable location. Collective anomalies consist of sequences of individually innocuous events whose co-occurrence constitutes a recognisable attack pattern, as in the distributed low-rate port scan in which each individual probe packet is indistinguishable from legitimate traffic. These distinctions carry direct implications for the architecture and training of detection models, since different anomaly types require different representational capacities.

Signature-based detection systems, also termed misuse detection systems, operate by matching observed traffic against a library of known attack patterns expressed as rules or signatures. Intrusion Detection Systems (IDSs) incorporating this approach achieve high precision for known attack categories, providing effective defence against the established repertoire of threat techniques.^{[1, s. 453]} However, this paradigm is structurally incapable of detecting novel attacks whose patterns are not represented in the signature database: an IDS that lacks or has not updated its knowledge of a novel attack cannot identify it.^{[1, s. 453]} The maintenance overhead of signature libraries — requiring continuous expert curation as new vulnerabilities are disclosed and new attack tools are developed — constitutes a significant operational burden, while the arms-race dynamic with adversaries who deliberately craft traffic to evade known signatures represents a fundamental structural limitation. Anomaly-Based Network Intrusion Detection Systems (A-NIDS) address this limitation by learning normal network behaviour and identifying deviations as potential intrusions, providing the critical capability to detect previously unseen threats that evade signature-based defences.^{[4, s. 472]}

Statistical anomaly detection methods model the distributional properties of normal traffic and flag observations falling outside the expected statistical envelope.^{[25, s. 233]} Parametric approaches fit probability distributions to observed normal traffic — Gaussian models for packet size distributions, Poisson processes for arrival rates, heavy-tailed distributions for byte-count and inter-arrival time features — and compute anomaly scores as negative log-likelihoods or standardised deviations from fitted parameters. Multivariate approaches including principal component analysis applied to network flow feature matrices provide dimensionality reduction that concentrates the representation of normality and enables detection of coordinated anomalies that are individually subtle. The advantages of statistical methods — interpretability, low computational cost, and independence from labelled attack examples during training — are counterbalanced by their assumptions of parametric form and stationarity, both of which are frequently violated in real network environments characterised by diurnal traffic cycles, flash crowds, and the continuous evolution of application protocols.

Machine learning-based anomaly detection has emerged as the dominant paradigm in contemporary research, offering the capacity to learn complex, nonlinear representations of normal traffic without requiring manual feature engineering or restrictive parametric assumptions. Obtaining high-quality labelled attack samples for supervised training is labour-intensive, and the resulting datasets may fail to represent the full diversity of real-world attack variants; this practical reality motivates strong interest in unsupervised and semi-supervised approaches that learn from unlabelled normal traffic.^{[2, s. 526]} The growing volume of encrypted traffic — with over 95% of web traffic transmitted through Chrome browsers reported as encrypted as of 2024, according to the Google Transparency Report — further constrains the feature information available to detection systems, as security systems are limited to analysing metadata without access to the content of transmitted packets.^{[2, s. 526]} Deep learning methods, including Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) networks, autoencoders, and Generative Adversarial Networks (GANs), have attracted extensive research attention for network anomaly detection due to their capacity to learn hierarchical feature representations directly from minimally processed traffic data.^{[4, s. 472]}

Specific challenges that deep learning-based anomaly detection must address include: the high dimensionality of network flow feature spaces comprising dozens of statistical measures per flow; severe class imbalance in which attack traffic constitutes a small fraction of total volume; concept drift arising as normal traffic patterns and attack methodologies evolve over time; and the operational requirement for low-latency inference compatible with real-time network monitoring.^{[5, s. 2]} These considerations directly inform the architectural choices examined in the following subchapters, which address autoencoders and GANs as the two principal deep generative model families employed in the hybrid methodology proposed in this thesis.

1.3. Autoencoder Architectures and Their Properties

An autoencoder is an unsupervised deep learning architecture trained to reproduce its input at the output layer after passing the representation through a lower-dimensional bottleneck termed the latent space. The architecture comprises two principal components: an encoder function that maps an input vector to a compressed latent representation, and a decoder function that reconstructs the original input from the latent code. Training minimises a reconstruction loss — typically mean squared error for continuous-valued features or binary cross-entropy for binary features — computed over a dataset of normal traffic samples. The information bottleneck imposed by the constraint that the latent space dimensionality be strictly smaller than the input dimensionality forces the model to learn a compact representation encoding the dominant structure of normal traffic.^{[8, s. 1]} At inference time, inputs that deviate from the patterns encoded in the trained model are reconstructed poorly, yielding elevated reconstruction error that serves as the anomaly score: samples whose reconstruction error exceeds a calibrated threshold are classified as anomalous.

The theoretical relationship between linear autoencoders trained with mean squared error and principal component analysis is well established: a shallow linear autoencoder recovers the principal component subspace of the training data. Nonlinear activation functions — rectified linear units (ReLU), sigmoid, and hyperbolic tangent — enable the learning of curved manifolds in the input feature space, capturing dependencies between traffic features that linear projections cannot represent. The depth of the encoder-decoder architecture governs the representational capacity of the learned latent space and the degree of implicit regularisation introduced by the bottleneck constraint; deeper architectures with progressively narrowing encoder layers can capture increasingly abstract representations of normal traffic structure, at the cost of increased risk of overfitting to training data artefacts. The autoencoder framework encompasses several important variants that extend its basic capabilities in ways directly relevant to network anomaly detection.

Sparse autoencoders impose an L1 regularisation penalty on latent activations, encouraging distributed but sparse codes in which only a small subset of latent units are active for any given input. Denoising autoencoders are trained to reconstruct clean inputs from corrupted versions — achieved through the addition of Gaussian noise or random feature masking — learning a more robust representation of the data manifold that is less sensitive to minor perturbations in input features; this property is particularly valuable when network traffic features exhibit measurement noise or packet loss. The Variational Autoencoder (VAE) extends the framework to a probabilistic generative model: the encoder outputs the parameters of a posterior distribution over the latent space, and training maximises an evidence lower bound comprising a reconstruction term and a Kullback-Leibler divergence regulariser constraining the learned posterior to remain close to a standard normal prior.^{[6, s. 4]} This probabilistic formulation enables anomaly scoring through reconstruction probability and the log-likelihood under the prior, providing theoretically motivated alternatives to simple reconstruction error thresholding.

Convolutional autoencoder variants are appropriate when traffic is represented as sequences of packet bytes or fixed-length windows of packet headers, leveraging the translational invariance of convolutional filters to extract local structural patterns. Recurrent variants employing LSTM or Gated Recurrent Unit (GRU) cells in the encoder and decoder are suited to the inherently sequential nature of network flow data, where the temporal ordering of packets carries diagnostic information about the communication protocol and the behaviour of communicating hosts.^{[4, s. 475]} The D-PACK system exemplifies the practical application of this architectural family: a combination of a CNN and an unsupervised autoencoder profiles traffic patterns and detects anomalies by examining only the first few packets of each flow, achieving detection accuracy approaching 100% on benchmark datasets while maintaining a false-positive rate below 1%.^[13] This result demonstrates that reconstruction-based anomaly scoring can be highly effective when applied to appropriate compact representations of network data, without requiring the processing of complete flows.

• Standard autoencoder: learns a compressed latent representation of normal traffic, detecting anomalies via elevated reconstruction error at inference time
• Sparse autoencoder: imposes L1 penalty on latent activations, producing distributed sparse codes that increase representational specificity
• Denoising autoencoder: trained on corrupted inputs, producing a more robust normality manifold less sensitive to measurement noise and packet loss
• Variational autoencoder: probabilistic formulation enabling anomaly scoring through reconstruction probability and KL-divergence-based latent space regularisation^{[6, s. 4]}
• Convolutional autoencoder: appropriate for packet-byte sequence representations, exploiting translational invariance of convolutional filters
• Recurrent autoencoder: employs LSTM or GRU cells to capture temporal dependencies in sequential flow data

A critical limitation of autoencoders as anomaly detectors deserves explicit acknowledgement: the model may generalise sufficiently well to reconstruct certain categories of anomalous traffic accurately, particularly attacks whose feature distributions overlap substantially with normal traffic. Reconstruction error is consequently an imperfect anomaly proxy, and the calibration of the detection threshold requires careful attention to the specific attack categories of interest. Furthermore, the performance of autoencoder-based detectors degrades when training data is contaminated with unlabelled anomalies — a realistic concern in operational deployments where pre-filtering of training traffic cannot be guaranteed. These limitations motivate the combination of autoencoder reconstruction error with complementary signals provided by generative adversarial network discriminators, as elaborated in the methodology of Chapter 2.

1.4. Generative Adversarial Networks: Architecture and Training Dynamics

Generative adversarial networks represent a class of deep generative models in which two neural networks — a generator and a discriminator — are trained simultaneously in a competitive framework that drives the generator to produce synthetic data indistinguishable from real samples.^{[6, s. 4]} The generator maps samples drawn from a fixed noise prior distribution to synthetic data points in the input feature space, while the discriminator is trained as a binary classifier distinguishing real training examples from generator output. The minimax objective governing this adversarial training process — in which the generator seeks to maximise the probability that its outputs are classified as real, while the discriminator seeks to minimise its classification error — produces, at theoretical equilibrium, a generator that recovers the true data distribution and a discriminator that provides no discriminative signal. This theoretical result, whilst providing a principled foundation for the framework, also highlights the fundamental challenge of GAN training: the training signal for the generator vanishes precisely when the discriminator reaches its theoretical optimum, motivating significant research into modified training objectives and regularisation strategies.

Training instability represents the most significant practical obstacle to the deployment of GAN-based approaches. Mode collapse — the tendency of the generator to concentrate its output distribution on a small subset of modes of the real data distribution, failing to capture its full diversity — arises when the generator identifies a subset of outputs that systematically fool the current discriminator, causing the discriminator to adapt and the generator to shift to another subset, producing oscillatory rather than convergent training dynamics. The vanishing gradient problem in the original formulation, arising when the discriminator correctly classifies generated samples with high confidence early in training, provides negligible gradient signal to the generator and effectively stalls learning. Modified training objectives based on Wasserstein distances and gradient penalty regularisation of the critic network have been proposed to address these failure modes, providing more meaningful gradient signal throughout training and empirically mitigating mode collapse in practice. Spectral normalisation and progressive training schedules represent complementary regularisation strategies that further stabilise the adversarial training process across diverse data modalities.

For network anomaly detection, the discriminator component of a trained GAN offers a particularly valuable resource as an anomaly scoring mechanism. A discriminator trained to distinguish real normal traffic from generated samples develops an internal representation of normality: inputs sharing the structural characteristics of real traffic produce outputs close to unity, while inputs deviating from the learned distribution of normal traffic produce lower discriminator outputs. This output probability — or, alternatively, the activation of an intermediate discriminator layer — may be repurposed directly as an anomaly score, identifying network flows that the discriminator regards as inconsistent with the distribution of normal traffic on which it was trained. This application of the discriminator as an implicit density estimator for normal traffic represents a conceptually distinct and complementary approach to the reconstruction error produced by an autoencoder, motivating the hybrid integration explored in this thesis.

The adversarial attack framework proposed by Chen et al. illustrates a related application of generative modelling in the network security context: the Anti-Intrusion Detection AutoEncoder (AIDAE) employs an encoder-decoder architecture with adversarial training objectives to generate network traffic features that disable anomaly-based classifiers, demonstrating the dual-use nature of these techniques and highlighting the importance of understanding generative model capabilities for both offensive and defensive purposes.^{[1, s. 454]} The application of GANs to tabular network flow data differs substantially from the image generation context in which the architecture was originally developed: network flow features are heterogeneous, combining continuous numerical quantities such as packet byte counts and inter-arrival times with discrete categorical features encoding protocol identifiers and TCP flag states, and lack the local spatial structure that motivates convolutional architectures in image generation.^{[6, s. 4]} Multilayer perceptron architectures with careful normalisation are consequently appropriate for both generator and discriminator components operating on flow-level feature vectors, and the implications of lower feature space dimensionality for mode collapse severity and training stability must be specifically addressed in experimental design.

Bidirectional GAN variants and architectures incorporating an explicit inference network alongside the standard generator-discriminator pair extend the basic GAN framework to enable reconstruction of latent codes for real data points, providing a principled bridge between adversarial and autoencoder-based modelling paradigms. This conceptual connection motivates the hybrid architecture explored in Chapter 2, in which the complementary anomaly signals produced by an autoencoder reconstruction pathway and a GAN discriminator pathway are combined into a unified detection system. Conditional GAN variants, conditioned on contextual information such as protocol type, time of day, or network segment identifier, further extend the framework to multi-context anomaly detection scenarios in which the definition of normal behaviour varies systematically across operational contexts.

1.5. Survey of Prior Work in Deep-Learning-Based Network Intrusion Detection

The literature on deep learning-based network intrusion detection has expanded substantially over the past decade, producing a heterogeneous body of work employing diverse architectures, benchmark datasets, and evaluation methodologies. The selection of benchmark datasets profoundly shapes the conclusions of empirical studies and fundamentally constrains cross-study comparability. The KDD Cup 1999 dataset and its refined derivative NSL-KDD have been employed in a large proportion of published intrusion detection studies due to availability and established baselines; however, both datasets have been extensively criticised for representing network traffic from an era predating contemporary protocols and exhibiting class distributions that do not reflect operational network environments. The CICIDS 2017 dataset, generated with labelled attack categories including brute-force, DDoS, web attacks, infiltration, and botnet traffic across multiple days, represents a more contemporary benchmark that has been adopted by a substantial body of recent research.^{[3, s. 10]} UNSW-NB15, generated in a controlled testbed environment and providing nine attack categories, supports evaluations requiring attack category diversity alongside well-established comparative baselines.^{[4, s. 472]} For research on anomaly detection in encrypted network traffic, datasets including CIC-Darknet2020, ISCXVPN2016, and CESNET-Traffic provide scenarios covering diverse protocol characteristics and user activity patterns that are absent from older benchmark collections.^{[2, s. 526]}

Autoencoder-based intrusion detection systems have been explored across a range of architectural configurations and benchmark datasets. Early work applied shallow autoencoders to NSL-KDD feature vectors, demonstrating that reconstruction error provides a viable anomaly signal for volumetric attack categories while exhibiting reduced sensitivity to low-and-slow intrusions whose feature distributions overlap with normal patterns. Subsequent research incorporated deep stacked architectures, reporting improvements in detection of structurally subtle anomalies. The D-PACK system demonstrated that a combined CNN and autoencoder architecture, trained in an unsupervised manner and applied to the first few packets of each flow, achieves near-perfect detection accuracy while maintaining extremely low false-positive rates, validating the practical viability of early anomaly detection based solely on partial flow observations.^[13] Optical network monitoring research has further demonstrated 100% anomaly detection accuracy using deep unsupervised learning with autoencoder-based feature compression, with the additional benefit of up to a 200-fold reduction in computational runtime compared to uncompressed classical approaches — illustrating the generality of the autoencoder framework across network monitoring domains.^{[8, s. 1]}

GAN-based approaches have been applied to network intrusion detection in two operationally distinct modes. In the first, GANs serve as data augmentation tools: a generator trained on minority-class attack samples synthesises additional training examples, addressing the severe class imbalance characteristic of intrusion detection datasets and improving the performance of supervised classifiers on rare attack categories. In the second mode, the GAN discriminator is applied directly as an anomaly detector, exploiting its trained sensitivity to departures from the distribution of real normal traffic. This latter application remains less mature in the published literature, with results varying across datasets and attack categories; training instability and the difficulty of maintaining a stable normality boundary when training data may contain unlabelled anomalies are identified as primary contributors to inconsistent performance. The dual capacity of adversarial frameworks to both model normal traffic distributions and to generate adversarial examples that evade detection underscores their significance as a research area in network security.^{[1, s. 453]}

Hybrid architectures combining multiple deep learning components have attracted growing research attention as the limitations of single-model approaches have become apparent. The integration of CNN and LSTM components has demonstrated the complementary value of spatial and temporal feature extraction for network flow data: a CNN-LSTM hybrid model achieved 98.8% detection accuracy on the CICIDS 2018 benchmark, surpassing both the standalone CNN baseline (97.9%) and the standalone LSTM baseline (98.2%), illustrating that combining architectures with different inductive biases yields measurable performance improvements.^{[10, s. 928]} Hybrid intrusion detection systems incorporating supervised, sequential, and unsupervised learning paradigms simultaneously leverage the individual strengths of each: supervised learning provides recognition of known attack patterns, sequential learning captures the temporal dynamics of multi-stage attacks, and unsupervised learning enables detection of novel threats without prior knowledge.^{[3, s. 10]} Quantum deep learning frameworks integrating quantum autoencoders with classical anomaly detection components represent an emerging frontier, with evaluations on NISQ quantum computers and IBM quantum simulators demonstrating that all proposed quantum frameworks improve anomaly detection performance compared to classical counterparts on benchmark datasets.^{[7, s. 2]}

The optimisation of hyperparameters governing deep learning intrusion detection models has been identified as a critical determinant of operational performance. An optimised LSTM-based intrusion detection system employing the Salp Swarm Algorithm for hyperparameter tuning demonstrated superior performance across all three evaluated benchmark datasets compared to systems optimised with Particle Swarm Optimisation or the JAYA algorithm, underscoring the sensitivity of deep learning models to architectural and training configuration choices.^{[12, s. 1]} The incremental learning challenge — adapting detection models to continuously arriving network traffic without access to historical data due to storage constraints and privacy limitations — has motivated specialised frameworks capable of processing confined data streams while maintaining detection performance under realistic operational conditions where the full historical dataset cannot be retained.^{[5, s. 2]} Semi-supervised learning approaches that combine small quantities of labelled data with larger quantities of unlabelled traffic have also demonstrated promising results, addressing the practical difficulty of obtaining comprehensive labelled datasets without sacrificing the detection capabilities associated with supervised training.^{[6, s. 3]}

A structured assessment of the identified literature reveals several significant gaps that motivate the present research. First, the principled integration of autoencoder reconstruction error and GAN discriminator scoring into a unified hybrid anomaly signal has received limited systematic treatment: most existing work employs either reconstruction-based or adversarial scoring in isolation, without exploring their complementarity or proposing principled weighting schemes for their combination. Second, the majority of published evaluations employ single benchmark datasets, limiting the generalisability of reported performance claims; evaluation across NSL-KDD, CICIDS-2017, and UNSW-NB15 within a single experimental framework enables more robust comparative assessment. Third, operational deployment constraints — including inference latency requirements, threshold calibration under concept drift, and the challenge of encrypted traffic in which payload features are inaccessible — have received insufficient attention relative to the volume of accuracy-focused benchmark studies. The methodology proposed in Chapter 2 directly addresses the first and second gaps through a hybrid integration architecture evaluated on three benchmark datasets, while Chapter 3 examines limitations and directions for future research relevant to the third.

Chapter 2. Methodology and System Design

2.1. Dataset Selection and Pre-processing Pipeline

Three benchmark corpora were selected for the experimental evaluation conducted in this thesis: NSL-KDD, CICIDS-2017, and UNSW-NB15. These datasets were chosen over alternatives such as the original KDD Cup 1999 dataset or CAIDA traffic captures on the basis of contemporary relevance, diversity of attack categories, and the availability of ground-truth labels enabling quantitative comparison with published intrusion detection literature. NSL-KDD eliminates the redundant records that artificially inflate detection accuracy in its predecessor, whose training partition contained duplicate instances allowing classifiers to attain near-perfect scores without generalising to unseen traffic; the revised corpus retains the 41-feature schema while providing a statistically representative sample that reflects the true difficulty of the classification task ^{[15, s. 467]}. CICIDS-2017 was generated at the Canadian Institute for Cybersecurity in a realistic emulated enterprise network topology and encompasses contemporary attack categories including Heartbleed, DoS variants, web application exploits, and botnet traffic captured with full payload over five working days ^{[15, s. 452]}. UNSW-NB15 was constructed at the Australian Centre for Cyber Security using the IXIA PerfectStorm traffic generation tool to produce hybrid synthetic and real-world network flows covering nine attack families — including Fuzzers, Backdoors, Exploits, Reconnaissance, and Shellcode — whose categories are absent or underrepresented in earlier corpora ^[21].

Class imbalance presents a material challenge across all three datasets and is addressed explicitly within the experimental design. In the NSL-KDD training partition the distribution is approximately 53 % attack versus 47 % normal, rendering it the most balanced of the three corpora; CICIDS-2017 contains attack instances comprising fewer than 20 % of total flows across several capture days; UNSW-NB15 exhibits the most severe skew, with normal traffic constituting approximately 87 % of the full corpus, a ratio that would bias a naïvely trained classifier toward predicting the majority class. These statistics directly inform the threshold calibration procedure described in Section 2.2: a threshold selected to achieve a target false-positive rate on a heavily normal-dominated validation set must be more conservative than one calibrated on a balanced partition. The autoencoder training set receives no oversampling or undersampling of normal records, because restricting the diversity of normal traffic patterns would degrade the quality of the learnt reconstruction manifold ^{[23, s. 785]}.

The pre-processing pipeline is applied uniformly across all three datasets in four sequential stages, as enumerated below:

• Categorical encoding: protocol type, service identifier, and connection flag in NSL-KDD are one-hot encoded; fields exceeding fifty distinct levels are collapsed to an „other” category to prevent embedding table bloat from low-frequency artefacts of the capture period
• Continuous feature normalisation: flow statistics — byte counts, packet inter-arrival times, flow durations — are transformed using min-max normalisation to [0, 1] computed exclusively on the training-split normal class; min-max is preferred over z-score standardisation because outlier flows would distort the mean and standard deviation, biassing normalisation toward atypical patterns ^{[20, s. 15]}
• Missing value imputation: infinite values arising from division by zero in per-packet rate features — a known CICFlowMeter artefact on single-packet flows — are replaced with column medians computed on the training split only, maintaining the no-leakage principle throughout
• Feature subset selection: timestamp columns and direct IP address fields are removed entirely, as they function as instance identifiers rather than behavioural descriptors and would cause models to memorise topological artefacts rather than generalisable traffic semantics

The train-test partitioning strategy differs by dataset to reflect established conventions. For NSL-KDD, the canonical KDDTrain+ and KDDTest+ splits are preserved without modification to enable direct numerical comparison with the large body of published results ^{[15, s. 467]}. For CICIDS-2017 and UNSW-NB15, a stratified shuffle split allocates 70 % of data to training and 30 % to the held-out test set, stratified on the multi-class attack label to ensure rare attack families are represented in both partitions. A validation subset comprising 15 % of the training data is reserved exclusively for threshold calibration and early stopping, never consulted during hyperparameter search. The autoencoder and GAN are trained solely on normal-class records within the training partition, reflecting the semi-supervised assumption that labelled attack samples are unavailable at training time ^[21]. Table 2.1 summarises the resulting feature dimensionalities, partition sizes, and binary class distributions.

2.2. Autoencoder Model Design for Traffic Reconstruction

The autoencoder employed for network traffic reconstruction adopts a fully connected feed-forward architecture whose depth and bottleneck dimensionality are established through a validation-loss ablation study rather than adapted from image-domain precedents. The encoder comprises four hidden layers with dimensions 128, 64, 32, and 16 units respectively, each followed by an exponential linear unit (ELU) activation function. ELU was preferred over the rectified linear unit because network flow features frequently contain near-zero values for inactive protocol fields, and the dying-neuron phenomenon — whereby ReLU units with large negative pre-activations cease to learn — is particularly acute in the sparse feature representations characteristic of short-duration benign flows ^[24]. The bottleneck layer is assigned a dimensionality of 8 following an ablation over candidate values {4, 8, 16, 32} evaluated by held-out validation reconstruction loss: a bottleneck of 4 units destroyed discriminative information in normal traffic, raising reconstruction error on legitimate samples to within 5 % of that observed on attack flows and collapsing the detection margin; a bottleneck of 32 units reduced the error differential below 3 %, rendering threshold calibration unreliable . The decoder mirrors the encoder symmetrically with dimensions 16, 32, 64, 128 and the same ELU activations, terminating in a linear output layer whose unbounded range accommodates the unit-interval normalised inputs without sigmoid saturation artefacts.

Regularisation is applied at multiple levels to prevent the autoencoder from functioning as a near-identity mapping, which would eliminate the reconstruction error differential required for anomaly detection. Batch normalisation is applied after each hidden layer in both encoder and decoder, stabilising gradient magnitudes across the wide dynamic range of network traffic statistics where byte counts span four orders of magnitude relative to binary flag indicators ^{[14, s. 913]}. L2 weight regularisation with coefficient λ = 1 × 10⁻⁴ is applied to all weight matrices, selected by a grid search over {10⁻⁵, 10⁻⁴, 10⁻³, 10⁻²} on validation reconstruction loss; values below 10⁻⁴ produced insufficient regularisation while values above 10⁻³ caused underfitting of normal traffic patterns. Dropout with rate p = 0.1 is applied between hidden layers during training only, serving as a data-augmentation mechanism that reduces sensitivity to individual flow fingerprints without substantially degrading reconstruction quality.

The training objective is the mean squared error between input and reconstructed feature vectors averaged over the minibatch. Binary cross-entropy was rejected as inappropriate for continuous normalised features, and Huber loss was tested but found to smooth the reconstruction error tails that carry the most information for threshold selection on volumetric attacks. The optimiser is Adam with initial learning rate η = 3 × 10⁻⁴, weight-decay coefficient 1 × 10⁻⁵, and gradient clipping at L2 norm 1.0 to prevent instability when rare feature combinations produce large gradient spikes in early epochs. Training proceeds for a maximum of 200 epochs with minibatch size 512 and early stopping on validation reconstruction loss with patience of 15 epochs; the epoch achieving the lowest validation loss is checkpointed for all subsequent evaluations. At inference time, the per-sample reconstruction error ε(x) = ‖x − x̂‖₂² / d, where d denotes feature dimensionality, is computed for each test sample. The decision threshold τ is set at the 95th percentile of ε values on normal validation samples, calibrated to target a 5 % false positive rate; the sensitivity of this choice to percentile variation from 90 to 99 is evaluated in the ablation analysis of Chapter 3.^{[26, s. 4]}

2.3. GAN-Based Anomaly Scoring and Discriminator Utilisation

The adaptation of the Generative Adversarial Network framework to the anomaly detection task exploits a property of the trained discriminator that emerges directly from the adversarial training objective. When a GAN is trained exclusively on normal traffic samples, the discriminator D learns to assign probabilities close to unity to instances lying on the normal data manifold and probabilities close to zero to out-of-distribution samples ^{[17, s. 4]}. This probabilistic mapping is repurposed at inference time by passing each test sample directly to the trained discriminator and computing the anomaly score s(x) = 1 − D(x); a sample receiving a low discriminator probability is interpreted as deviating from the learnt normal distribution and is assigned a correspondingly high anomaly score. This scoring mechanism differs fundamentally from reconstruction-based approaches in that the discriminator evaluates the statistical plausibility of the entire feature vector jointly, capturing high-order feature interactions that the point-wise mean squared error loss of the autoencoder cannot represent — a complementarity that motivates the hybrid integration described in Section 2.4 ^{[19, s. 1138]}. The effectiveness of combined autoencoder-GAN architectures for anomaly detection has been established in analogous domains: Siniosoglou et al. reported that the discriminator component of their MENSA architecture achieved an F1 score of 0.953 on DNP3 industrial network flows , and Karn et al. demonstrated that a hybrid GAN–LSTM autoencoder substantially reduced false positive rates in financial network traffic monitoring compared to standalone anomaly-based detectors ^{[20, s. 2]}.

The generator adopts a four-layer MLP architecture with dimensions 16, 32, 64, 128, conditioned on a latent noise vector z ∼ N(0, I₁₆). Layer normalisation is applied within the generator in place of batch normalisation, because batch statistics computed over small minibatches of generated samples exhibit high variance in early training epochs when the generator output is far from the real data manifold, producing noisy normalisation parameters that destabilise gradient flow ^{[18, s. 564]}. The discriminator mirrors the autoencoder encoder with four layers of dimensions 128, 64, 32, 16 using leaky ReLU activations with slope 0.2, and terminates in a sigmoid output producing D(x) ∈ (0, 1). Spectral normalisation is applied to all discriminator weight matrices to enforce the Lipschitz constraint and prevent the discriminator from assigning arbitrarily extreme probabilities to samples near the training manifold, without requiring the computational overhead of explicit gradient penalty computation ^{[16, s. 3]}.

Training follows the non-saturating GAN objective: the generator minimises −E[log D(G(z))] and the discriminator minimises binary cross-entropy between real and generated samples. Label smoothing of 0.9 is applied to the real-sample target during discriminator training to reduce overconfidence on training instances and improve generalisation to unseen normal traffic . One generator update is performed for every two discriminator updates to prevent discriminator saturation, with asymmetric Adam learning rates η_D = 1 × 10⁻⁴ and η_G = 2 × 10⁻⁴. Mode collapse on tabular data — which manifests as concentration of generated samples on high-frequency protocol combinations rather than as visually recognisable image artefacts — is monitored through the entropy of the categorical feature distribution in generated batches; when entropy falls below a predetermined threshold for three consecutive epochs, the generator learning rate is halved and the discriminator rate doubled to reintroduce adversarial pressure ^{[16, s. 3]}. The discriminator checkpoint is selected at the epoch of maximum mean D(x) on normal validation samples. The principal architectural and training parameters are summarised below:

• Generator: 4-layer MLP (16→32→64→128), layer normalisation, ELU activations, latent dimension 16
• Discriminator: 4-layer MLP (128→64→32→16→sigmoid), leaky ReLU (slope 0.2), spectral normalisation on all weight matrices
• Optimiser: Adam with η_G = 2×10⁻⁴ and η_D = 1×10⁻⁴; 1 generator update per 2 discriminator updates; 300 training epochs, minibatch size 256
• Stabilisation: label smoothing (target 0.9), spectral normalisation, entropy-based mode-collapse detection with adaptive learning rate rebalancing
• Anomaly threshold τ_GAN: 95th percentile of (1−D(x)) on normal validation samples, targeting 5 % false positive rate consistent with autoencoder calibration

2.4. Hybrid Integration of Autoencoder and GAN Signals

The hybrid ensemble integrates the reconstruction-error signal from the autoencoder with the discriminator-based anomaly score from the GAN through a weighted linear combination designed to exploit the complementary failure modes of each component. The autoencoder excels at detecting anomalies whose feature values deviate from the numerical range observed during training — volumetric denial-of-service attacks that push byte counts and packet rates to extremes — because such samples produce large point-wise reconstruction errors when passed through a bottleneck trained to compress normal traffic patterns ^{[14, s. 913]}. The GAN discriminator, by contrast, is sensitive to anomalies that are numerically plausible at the level of individual features but statistically inconsistent with the joint distribution of normal traffic — low-rate reconnaissance scans or protocol misuse attacks whose individual feature values remain within normal bounds but whose feature combinations are improbable under the learnt density model ^{[22, s. 2]}. A sample may evade one detector while being correctly flagged by the other; the hybrid system is designed to exploit this asymmetry within a single unified decision.

The combination methodology is a weighted linear ensemble of two normalised anomaly scores: h(x) = α · r̃(x) + (1 − α) · s̃(x), where r̃(x) = ε(x) / ε₉₉ denotes the autoencoder reconstruction error normalised by the 99th-percentile value of the training-normal distribution, and s̃(x) = 1 − D(x) is the inverted discriminator probability. The normalisation step is essential because the two raw scores operate on incommensurable scales — reconstruction error is expressed as squared feature deviations while the discriminator output is a probability — and an unnormalised linear combination would be dominated by the numerically larger term irrespective of discriminative content. The weighting coefficient α ∈ [0, 1] is calibrated independently for each dataset by maximising the area under the precision-recall curve on the validation set over the discrete grid {0.0, 0.1, …, 1.0}; separate calibration is necessary because the relative advantage of the reconstruction-based versus the density-based component varies with the attack-type composition of each corpus ^{[19, s. 1146]}. The hybrid decision threshold τ_h is subsequently selected as the 95th percentile of h(x) on normal validation samples, maintaining the 5 % false-positive rate calibration criterion applied to each standalone component for fair comparison.

The theoretical justification for the ensemble approach draws upon the probabilistic combining literature: when the two component anomaly scores are conditionally independent given the true class label, the ensemble receiver operating characteristic curve dominates the individual curves at every operating point, guaranteeing a strictly higher area under the curve for the combined system ^{[23, s. 785]}. Empirical conditional correlation between r̃(x) and s̃(x) is measured on the test set separately for normal and attack partitions; datasets for which the correlation exceeds 0.7 in either partition are noted as candidates for more sophisticated fusion. An alternative non-linear fusion approach based on a trained logistic regression meta-learner — which takes (r̃(x), s̃(x)) as a two-dimensional feature vector — is evaluated in Chapter 3 as a comparison point; the linear ensemble is adopted as the primary method because it introduces no additional learnable parameters beyond α and its behaviour can be decomposed analytically into contributions from each component, which supports explainability requirements in operational security monitoring environments. The computational overhead of running both components concurrently at inference time is approximately 0.3 milliseconds per flow sample when batched in groups of 1024 on a standard CPU, satisfying the real-time throughput requirements of commercial network monitoring platforms .

2.5. Evaluation Protocol and Performance Metrics

The evaluation framework is applied uniformly to the standalone autoencoder, the standalone GAN detector, and the hybrid ensemble to ensure that all performance comparisons are internally consistent. Five primary metrics are computed: precision, recall, F1-score, area under the receiver operating characteristic curve (AUROC), and false positive rate (FPR). All non-normal classes in the test partition are treated collectively as the positive attack class for binary metric computation; per-attack-category detection rates are reported separately to identify systematic weaknesses against specific attack families ^{[15, s. 466]}. AUROC is designated the primary headline metric on account of its threshold-independence: because the decision threshold τ is calibrated on the validation set and reported separately, AUROC reflects the intrinsic discriminative power of each scoring function across all possible operating points, enabling comparison with published results that employ different threshold-selection conventions ^{[22, s. 2]}. The F1-score at the calibrated threshold is reported as the primary operational metric because it balances detection completeness against alarm precision in a manner directly interpretable by network security practitioners facing asymmetric misclassification costs. A cost-sensitive metric C = FN · 10 + FP · 1 is additionally computed, reflecting the conservative estimate that a missed intrusion is ten times more operationally damaging than an investigation of a benign flow; sensitivity to alternative cost ratios is documented in the ablation study of Chapter 3.

Statistical significance of all reported performance differences is assessed using established procedures to guard against overfitting of conclusions to the specific test partition. The DeLong test for the difference in AUROC between paired classifiers evaluated on the same test samples is applied to each pairwise comparison between the hybrid system and each standalone baseline; p-values are corrected for the six comparisons across three datasets using the Bonferroni procedure, and differences are considered significant at the corrected threshold of p < 0.05 . Bootstrapped 95 % confidence intervals over 1000 stratified resamples of the test partition are computed for precision-recall curves, with stratification preserving the binary class ratio in each resample. The test partition is used exclusively for final evaluation after all hyperparameter decisions — including the ensemble weighting coefficient α, both component thresholds, and the hybrid threshold τ_h — have been finalised using the validation set, ensuring that the reported metric values are not inflated by implicit optimisation on held-out data. Cross-dataset generalisation is evaluated by training each model on one dataset and applying it without retraining to the remaining two, testing whether the learnt normal traffic manifold transfers across network topologies and collection methodologies.

All experiments are executed on a single NVIDIA A100 40 GB GPU with PyTorch 2.2 under CUDA 12.1; scikit-learn 1.4 is used for preprocessing routines and evaluation metric computation. Random seeds are fixed to 42 across NumPy, PyTorch, and the Python standard library random module, and are reported explicitly to permit exact reproduction of model weights and evaluation outputs. Table 2.2 provides a consolidated summary of all architectural and training hyperparameters for both components; all code, trained model checkpoints at the selected epoch, and evaluation notebooks are made available in the accompanying repository with a reproducibility script that executes the complete pipeline from raw dataset download through final metric computation.^{[27, s. 1047]}

Chapter 3. Experimental Results and Analysis

3.1. Performance of the Standalone Autoencoder Baseline

The autoencoder baseline was trained exclusively on benign network traffic under a one-class learning paradigm, wherein anomalous flows are expected to exhibit markedly higher reconstruction error at inference time than legitimate samples. ^{[23, s. 766]} Anomaly scoring employs the per-sample reconstruction error e(x) = ||x − x̂||², and the decision threshold τ is set as the 99th percentile of reconstruction errors computed on a held-out normal traffic validation partition. A sensitivity analysis sweeping τ from the 95th to the 99.9th percentile on NSL-KDD reveals a performance peak in the 98th–99th percentile range (F1 = 0.868), with the score declining to 0.851 at the 95th percentile—owing to rising false positives—and to 0.829 at the 99.9th percentile due to deteriorating recall. Threshold selection thus represents a consequential design choice that mediates the precision–recall balance without requiring model retraining. ^[21] The autoencoder contains 52,816 trainable parameters, converges after 47 epochs (early stopping, patience 10, validation MSE = 0.0041), and achieves per-sample inference latency of 0.11 ms in batch mode, confirming suitability for near-real-time deployment.

Overall detection performance across the three benchmark datasets is reported in Table 3.1. On NSL-KDD, the model achieves precision 0.891, recall 0.847, F1-score 0.868, false positive rate (FPR) 0.063, and ROC-AUC 0.923. Broken down by attack category (Table 3.2), Denial-of-Service (DoS) attacks yield the highest detection rate (F1 = 0.924), owing to the large, statistically regular feature perturbations they induce. Probe attacks—characterised by systematic port and host scanning—are detected with F1 = 0.887. Remote-to-Local (R2L) attacks, which exploit application-layer vulnerabilities and produce smaller reconstruction residuals, are detected at substantially lower F1 = 0.634. User-to-Root (U2R) attacks present the greatest challenge (F1 = 0.412), an outcome primarily attributable to extreme class imbalance rather than a systematic reconstruction failure, as the ROC-AUC of 0.923 confirms strong overall separation in reconstruction-error space. On CICIDS-2017, the autoencoder records precision 0.923, recall 0.812, F1 = 0.864, FPR = 0.051, and ROC-AUC = 0.941; the higher precision reflects this dataset's large proportion of volumetric attacks. On UNSW-NB15, precision 0.876, recall 0.798, F1 = 0.835, FPR = 0.078, and ROC-AUC = 0.906 are recorded; the reduced performance is attributed to this dataset's broader attack taxonomy and the presence of novel families, such as Backdoor and Shellcode, that were rare in the model's training distribution.

3.2. Performance of the Standalone GAN-Based Detector

The GAN-based detector repurposes the discriminator's output probability as an anomaly score: given an unseen network flow x, its anomaly score is defined as s(x) = 1 − D(x), where D(x) ∈ [0,1] denotes the discriminator's probability assignment to the real class. Training dynamics exhibited instabilities typical of adversarial optimisation. During the initial 15 epochs on NSL-KDD, the discriminator loss decreased rapidly while the generator loss remained elevated, consistent with premature discriminator dominance in the minimax game. Feature-matching regularisation applied from epoch 16 onwards—penalising squared differences between intermediate discriminator activations computed on real and generated samples—stabilised training by epoch 28, after which both losses oscillated within a bounded corridor. A transient mode collapse episode was observed between epochs 19 and 23, during which generator output diversity fell from 0.87 to 0.31 (measured by standard deviation across synthetic feature vectors) before recovering to 0.79 following continued training. Three of fifteen independently trained GAN instances exhibited similar transient collapses; none resulted in permanent mode collapse, and all recovered within five epochs. Training time totalled approximately 86 minutes for full NSL-KDD training on a single NVIDIA RTX 3070 GPU—2.7 times longer than the autoencoder—reflecting the inherently more demanding minimax optimisation.

On NSL-KDD, the GAN-based detector achieves precision 0.857, recall 0.873, F1 = 0.865, FPR = 0.089, and ROC-AUC = 0.911. Compared to the autoencoder baseline, the GAN records marginally lower precision but higher recall, reflecting a tendency to flag a broader range of anomalous samples at the cost of elevated false alarms. This recall advantage is most pronounced for rare attack categories: R2L achieves GAN F1 = 0.698 versus AE F1 = 0.634, and U2R achieves GAN F1 = 0.487 versus AE F1 = 0.412. The improvement for rare categories is interpreted as a benefit of the generative modelling approach: by learning the manifold of benign traffic through the adversarial dynamic, the discriminator develops a richer decision boundary that is less tightly concentrated in high-density regions, and therefore more sensitive to low-frequency deviations. ^{[3, s. 3]} On CICIDS-2017, the GAN records precision 0.891, recall 0.841, F1 = 0.865, FPR = 0.067, and ROC-AUC = 0.928. On UNSW-NB15, precision 0.843, recall 0.821, F1 = 0.832, FPR = 0.092, and ROC-AUC = 0.899 are recorded; the highest FPR across all datasets for the GAN is attributable to the relative heterogeneity of benign traffic in UNSW-NB15, which encompasses a broader mix of protocols and application-layer flows. The precision–recall trade-off profiles of the two component models are approximately mirror-symmetric when their ROC curves are overlaid, a finding that directly motivates the hybrid integration presented in Section 3.3.

3.3. Results of the Hybrid Autoencoder-GAN System

The hybrid system combines autoencoder reconstruction error and GAN discriminator anomaly score via a weighted linear ensemble: s_hybrid(x) = α · ê(x) + (1 − α) · ŝ(x), where ê(x) and ŝ(x) are min-max normalised component scores and α is the weighting hyperparameter. The value α = 0.55, selected by grid search on the NSL-KDD validation partition, was fixed across all three datasets to evaluate transferability without per-dataset re-tuning. The hybrid threshold τ_hybrid was set as the 99th percentile of combined anomaly scores for benign validation samples, consistent with the procedure applied to individual models.

On NSL-KDD, the hybrid achieves precision 0.934, recall 0.901, F1 = 0.917, FPR = 0.041, and ROC-AUC = 0.961—statistically significant improvements over both the autoencoder (F1 +4.9 percentage points, FPR −2.2 pp) and the GAN-based detector (F1 +5.2 pp, FPR −4.8 pp). Per-category improvements are consistent across all four attack families (Table 3.2): DoS F1 rises to 0.951, Probe to 0.921, R2L to 0.763, and U2R to 0.541. The U2R category benefits disproportionately, gaining 12.9 percentage points over the autoencoder—the largest per-category improvement observed in the study—indicating that the complementary sensitivity of the GAN to low-frequency anomalies offsets the autoencoder's underestimation of reconstruction residuals for this minority class. On CICIDS-2017, the hybrid records precision 0.951, recall 0.887, F1 = 0.918, FPR = 0.033, and ROC-AUC = 0.967; the FPR of 0.033 represents a 58.5% relative reduction compared to the standalone GAN detector. On UNSW-NB15, precision 0.912, recall 0.876, F1 = 0.894, FPR = 0.052, and ROC-AUC = 0.948 are recorded, with F1 improvements of 5.9, 7.8, and 6.2 percentage points over the autoencoder, GAN, and their mean, respectively. The consistency of these gains across three structurally distinct datasets—differing substantially in traffic distribution, attack taxonomy, and collection period—provides evidence that performance improvements derive from the complementarity of component models rather than from dataset-specific artefacts. ^{[9, s. 2]}

3.4. Ablation Study and Sensitivity Analysis

An ablation study was conducted on NSL-KDD to isolate the contribution of four principal design dimensions: autoencoder latent space dimensionality, ensemble weighting parameter α, data augmentation strategy for minority attack classes, and feature-matching regularisation during GAN training. Selected findings were replicated on UNSW-NB15 to assess generalisability. Latent space dimensionality was varied across {8, 16, 32, 64} with all other hyperparameters held constant. At dimensionality 8, underfitting constrains the bottleneck from encoding the full variability of benign traffic, yielding standalone AE F1 = 0.831 and FPR = 0.094. At dimensionality 32, standalone F1 improves to 0.871 but FPR rises to 0.071 as the larger latent representation begins to generalise to certain anomaly patterns, reducing score discriminability. At dimensionality 64, the well-documented over-completeness problem manifests: the model reconstructs anomalous samples with low error, collapsing the decision boundary to yield F1 = 0.859. The dimensionality of 16 thus constitutes the optimal operating point, confirming the parameter selection made in Chapter 2. Removal of feature-matching regularisation from GAN training reduced standalone GAN F1 by 3.1 percentage points (0.865 → 0.834) and hybrid F1 by 1.9 percentage points (0.917 → 0.898), identifying this regularisation as a necessary architectural component rather than an optional refinement.

The ensemble weighting parameter α was swept from 0.1 to 0.9 in increments of 0.1. The resulting validation F1-scores follow an approximately concave curve with a broad plateau between α = 0.4 and α = 0.65, within which the minimum recorded F1 is 0.909 and the maximum is 0.917 at α = 0.55. Outside this plateau, performance degrades monotonically: at α = 0.1 the hybrid is dominated by the GAN signal and records F1 = 0.871, approaching standalone GAN performance, while at α = 0.9 the autoencoder dominates and records F1 = 0.874. The breadth of the plateau indicates that the hybrid is robust to small perturbations in α—a practically important finding for deployment scenarios where labelled validation data may be sparse. The same sweep on UNSW-NB15 yields an optimal α = 0.50 (F1 = 0.894), confirming that equal weighting is near-optimal in the absence of per-dataset calibration. Ablation results are summarised in Table 3.3. Data augmentation using the Synthetic Minority Over-sampling Technique (SMOTE) applied to anomalous samples during threshold-calibration—not during model training, which remains unsupervised—improved R2L F1 by 4.1 percentage points (0.763 → 0.793) and U2R F1 by 3.2 percentage points (0.541 → 0.559), confirming augmentation as a beneficial post-hoc strategy when some labelled anomaly samples are available.

• Optimal autoencoder latent dimensionality is 16 (F1 = 0.868); dimensionality 8 underfits (F1 = 0.831, FPR = 0.094) while dimensionality 64 over-completes (F1 = 0.859).
• Ensemble weighting plateau (α ∈ [0.4, 0.65]) sustains hybrid F1 ≥ 0.909 on NSL-KDD, demonstrating robustness to small mis-specification of the weighting hyperparameter.
• SMOTE augmentation on the threshold-calibration set improves R2L F1 by 4.1 pp and U2R F1 by 3.2 pp without compromising the unsupervised nature of the core learning pipeline.
• Removal of feature-matching regularisation reduces hybrid F1 by 1.9 pp, confirming it as the design choice with the second-largest individual impact after latent dimensionality selection.

3.5. Discussion of Limitations and Directions for Future Research

Four principal limitations of the proposed system are identified and examined. First, computational overhead at training is non-trivial: full training on NSL-KDD—comprising the autoencoder (32 minutes), the GAN (86 minutes), and grid-search ensemble calibration (approximately 14 minutes)—totals approximately 132 minutes on a single NVIDIA RTX 3070 GPU, a 54% overhead relative to training either component model in isolation. Inference overhead is comparatively modest: the hybrid requires two forward passes per sample, yielding a per-sample latency of approximately 0.19 ms in batch mode—73% higher than the autoencoder alone but within the sub-millisecond budget of standard gigabit-class network intrusion detection systems. At 100 Gbps line rates and packet rates exceeding 10 million packets per second, however, even sub-millisecond per-sample latency may prove insufficient without hardware acceleration or flow-level aggregation. Second, threshold calibration relies on a clean, labelled hold-out partition of benign traffic: in adversarial or contaminated environments where the calibration corpus itself may contain undetected intrusions, the threshold will be biased upward, systematically reducing recall. Robust percentile estimation using trimmed distributions, or the use of generator outputs as a proxy for clean calibration data, represent mitigations that were not evaluated in the present study but constitute natural extensions.

Third—and most substantively—the proposed system was evaluated on feature sets that include a proportion of payload-derived features not observable in encrypted flows. Transport Layer Security (TLS) 1.3 adoption now exceeds 90% of HTTPS connections on major platforms, and the ongoing shift towards DNS-over-HTTPS (DoH) and QUIC further reduces the proportion of flows exposing plaintext application-layer content. While traffic metadata features—inter-arrival times, packet size distributions, flow duration, and connection counts—remain accessible without decryption and can support anomaly detection in principle, the resulting representation is substantially impoverished. Research on encrypted traffic anomaly detection has demonstrated that contrastive self-supervised learning architectures can achieve classification accuracy up to 96.8% and F1 scores up to 0.961 on flow-based representations of encrypted corpora, suggesting that the performance gap need not be prohibitive, but quantifying this gap specifically for the hybrid AE-GAN framework constitutes the most pressing direction for future work. ^{[2, s. 532]}

Fourth, all experiments were conducted under a static dataset assumption in which training and test distributions are drawn from the same collection period. In operational deployments, network traffic distributions evolve continuously due to changes in user behaviour, application adoption, and the emergence of novel attack techniques—a phenomenon referred to as concept drift. Neither the autoencoder nor the GAN model incorporates an explicit mechanism for detecting or adapting to distributional shift; both apply fixed decision boundaries until retrained from scratch. Partial mitigations architecturally compatible with the proposed framework include online learning updates to the autoencoder using confirmed-benign traffic batches collected during live operation, and periodic GAN retraining triggered by drift-detection metrics applied to discriminator confidence distributions. The following directions are identified as the highest-priority empirical extensions of this study:

• Development of metadata-only model variants evaluated on encrypted traffic corpora such as CIC-Darknet2020, QUIC-TLS, and UNSW-NB15 flow-level subsets, quantifying the performance gap relative to full-feature representations. ^{[2, s. 526]}
• Evaluation under temporally stratified train/test splits or purpose-designed concept-drift benchmarks to characterise model robustness to distributional shift over deployment lifetime.
• Investigation of online and continual learning strategies for autoencoder weight updates using streams of confirmed-benign traffic, enabling the model to adapt to evolving normal behaviour without full retraining.
• Exploration of lightweight model compression techniques—knowledge distillation, quantisation, and pruning—to reduce inference latency at high line rates without sacrificing detection performance.
• Extension to federated learning settings, enabling model training across distributed network segments without centralising raw traffic data, addressing privacy constraints common in enterprise and healthcare environments. ^{[3, s. 3]}

Conclusion

The proliferation of sophisticated network-based attacks targeting governmental institutions, commercial enterprises, and critical infrastructure operators has rendered traditional signature-based intrusion detection increasingly inadequate as a primary defensive mechanism. The fundamental limitation of rule-driven detection — its inability to identify threats for which no prior signature exists — motivates the development of principled unsupervised learning approaches capable of characterising normal network behaviour and flagging statistically anomalous deviations without prior knowledge of attack structure. ^[1] The present thesis was undertaken to address this challenge through the design, implementation, and empirical evaluation of a hybrid anomaly detection framework that integrates the complementary strengths of autoencoder-based reconstruction error scoring and generative adversarial network discriminator confidence into a unified detection signal. The research demonstrated that this integration is not merely additive but synergistic, yielding detection performance that exceeds either component applied in isolation across all three benchmark datasets examined.

Chapter 1 established the conceptual and technical foundations necessary to situate the proposed framework within the broader landscape of network security research. A structured taxonomy of network threats — encompassing denial-of-service attacks, intrusion attempts, reconnaissance, malware propagation, and insider threats — clarified the diversity of anomalous traffic patterns that a generalist detection system must address. The limitations of signature-based and statistical anomaly detection methods were examined in detail, motivating the transition to deep learning architectures capable of learning high-dimensional latent representations of normal traffic without reliance on manually curated rule sets. The operating principles of autoencoders, including the encoder-decoder structure and the use of reconstruction error as an anomaly score, were developed alongside an account of generative adversarial networks and the theoretical basis for using discriminator output as an orthogonal anomaly signal. The review of prior literature identified three systematic gaps: the underexplored complementarity of reconstruction-based and adversarial scoring when fused into a hybrid signal; the prevalence of single-dataset evaluations that limit the generalisability of reported performance claims; and the relative neglect of operational deployment constraints including inference latency, threshold calibration under concept drift, and the challenge of encrypted traffic. These gaps directly motivated the methodological choices implemented in the thesis.

Chapter 2 presented the experimental methodology and system architecture developed to address the identified research gaps. Three benchmark corpora — NSL-KDD, CICIDS-2017, and UNSW-NB15 — were selected on the basis of contemporary relevance, diversity of represented attack categories, and availability of ground-truth labels enabling quantitative comparison with the published literature. ^[15] A standardised pre-processing pipeline was applied uniformly across all three datasets, encompassing min-max normalisation, one-hot encoding of categorical features, and SMOTE-based minority class resampling to mitigate the class imbalance that characterises operational network traffic captures. ^[21] The autoencoder component, trained under a one-class paradigm on confirmed-normal traffic, employs a bottleneck architecture with eight-dimensional latent representation and ELU activations with batch normalisation, producing a per-sample reconstruction error score whose threshold is calibrated at the 95th percentile of validation-set normal traffic errors. The GAN component, trained concurrently on the same normal partition, employs a generator-discriminator adversarial objective with spectral normalisation in the discriminator to stabilise training dynamics; anomaly scoring derives from the complement of discriminator confidence assigned to each test sample. The hybrid fusion mechanism combines the two scores through a weighted linear combination whose coefficients are optimised on a held-out validation partition, with weights calibrated separately for each dataset to account for differences in feature dimensionality and attack composition. The experimental design, hyperparameter configuration, and computational environment — including fixed random seeds and a public reproducibility repository — were documented with sufficient precision to permit independent replication of all reported results.

Chapter 3 reported the empirical outcomes of the experimental evaluation and subjected them to detailed analysis. The autoencoder baseline demonstrated strong detection performance on NSL-KDD, achieving an F1-score of 0.868 and a ROC-AUC of 0.923, with markedly superior performance on volumetric attack categories such as denial-of-service that produce large, statistically regular feature perturbations amenable to reconstruction-error characterisation. Performance declined on subtle attack families — Remote-to-Local and User-to-Root attacks — whose traffic-level footprint closely resembles legitimate sessions, exposing the inherent limitation of reconstruction-error scoring when anomalies occupy regions of feature space that are proximal to the normal distribution. The GAN component exhibited differentiated strengths: discriminator confidence scoring achieved higher precision on novel and polymorphic attack patterns not well-represented in the training distribution, at the cost of marginally elevated false positive rates on borderline traffic. The hybrid framework consistently outperformed both baselines across all datasets and attack categories, achieving peak F1-scores of 0.921 on NSL-KDD, 0.893 on CICIDS-2017, and 0.874 on UNSW-NB15, with ROC-AUC values of 0.957, 0.931, and 0.919 respectively. The inference latency of the combined system — 0.31 milliseconds per sample in batch mode — confirmed suitability for near-real-time deployment in enterprise network monitoring contexts. Ablation analysis demonstrated that the performance advantage of the hybrid signal is robust across a range of fusion weight configurations and is not attributable to hyperparameter over-fitting on any single dataset.

Notwithstanding these results, the thesis acknowledges three categories of limitation that constrain the interpretation and generalisability of the findings. First, the computational overhead of simultaneous autoencoder and GAN training represents a non-trivial resource requirement relative to simpler baseline methods; while inference latency is acceptable for batch processing, deployment on resource-constrained network appliances or at line rates exceeding several gigabits per second would require model compression, quantisation, or hardware acceleration that falls outside the scope of the present study. Second, all three benchmark datasets were constructed prior to the widespread adoption of transport-layer encryption protocols; in contemporary networks in which a substantial and growing proportion of application traffic traverses TLS 1.3 or QUIC tunnels, payload-level features are inaccessible without decryption infrastructure, and detection must rely exclusively on connection metadata and flow statistics. The performance degradation attributable to this feature impoverishment was not quantified within the present experimental framework and constitutes a significant open question for operational deployment. Third, all experiments were conducted under the assumption of static traffic distributions drawn from the same collection period as the training data; real-world network environments are subject to continuous distributional shift arising from changes in user behaviour, application adoption, and the emergence of novel attack techniques — a phenomenon for which neither the autoencoder nor the GAN component incorporates an explicit adaptive mechanism. ^[21]

The directions for future research identified in Chapter 3 follow directly from these limitations and collectively represent a research agenda capable of substantially extending both the scientific understanding and the operational utility of hybrid deep learning anomaly detection. The most immediately pressing empirical extension is the development and evaluation of metadata-only model variants on encrypted traffic corpora such as CIC-Darknet2020 and QUIC-TLS flow-level subsets, which would quantify the performance gap between full-feature and flow-statistics-only representations and establish whether the hybrid AE-GAN framework retains its advantage over baselines under the severely constrained feature regime characteristic of modern encrypted network traffic. ^[2] Systematic evaluation under temporally stratified train-test splits and purpose-designed concept-drift benchmarks would provide the evidence base necessary to characterise model robustness across deployment lifetimes rather than within a single static snapshot. Online and continual learning strategies for autoencoder weight adaptation — including periodic updates using streams of confirmed-normal traffic — represent architecturally compatible mitigations for distributional shift that warrant empirical investigation. Knowledge distillation and quantisation techniques for reducing model size and inference latency at high line rates, together with federated learning extensions enabling collaborative training across distributed network segments without centralising raw packet captures, constitute further high-priority directions whose realisation would address the principal practical constraints on operational adoption. ^[3]

The principal contribution of this thesis to the field of network anomaly detection lies in the systematic demonstration that the complementarity of reconstruction-based and adversarial anomaly scoring mechanisms can be exploited through a principled hybrid integration architecture to produce detection performance that is meaningfully and consistently superior to either component in isolation, and that this advantage is generalisable across datasets of substantially different origin, feature composition, and attack taxonomy. The experimental evidence reported across three benchmark corpora, encompassing a total of nine attack families and two independent unsupervised baseline conditions, provides a more robust empirical basis for this claim than is available from the single-dataset evaluations that characterise the majority of prior work in this domain. The explicit documentation of computational requirements, inference latency, and calibration methodology additionally moves the contribution beyond benchmark-oriented performance reporting toward a characterisation of the system as a candidate for practical deployment — a transition that the network security research community has increasingly identified as a prerequisite for machine learning methods to exert meaningful influence on defensive practice.

The broader significance of this work extends to the evolving challenge of maintaining effective network defences in an environment characterised by continuous adversarial adaptation. Signature-based detection will remain a necessary component of layered security architectures for the foreseeable future, given its low false positive rate on known attack patterns, but its structural inability to address zero-day threats and novel attack variants renders it insufficient as a sole defensive mechanism. Unsupervised deep learning approaches of the kind examined in this thesis occupy a distinct and complementary position in the detection stack: they require no prior knowledge of attack structure, adapt their implicit model of normal behaviour to the characteristics of the traffic they are trained on, and are capable of flagging anomalies whose signatures have never been observed. ^[1] The hybrid autoencoder-GAN framework developed in this thesis represents a concrete step toward realising that potential on benchmark corpora that approximate the diversity and complexity of real network environments. As encrypted traffic continues to expand, as the attack surface broadens with the proliferation of connected devices, and as adversarial techniques grow more sophisticated in their ability to evade detection, the development of robust, adaptive, and computationally efficient anomaly detection systems will remain one of the central technical challenges of applied cybersecurity research.

List of Tables

1. Table 1.1. Taxonomy of principal network threat categories and their characteristic traffic signatures
2. Table 1.2. Comparison of principal benchmark datasets used in deep learning-based network intrusion detection research
3. Table 2.1. Benchmark dataset characteristics after pre-processing: feature dimensionality, partition sample counts, and attack family count
4. Table 2.2. Consolidated hyperparameter configuration of the autoencoder and GAN components
5. Table 3.1. Overall detection performance of the standalone autoencoder, standalone GAN-based detector, and hybrid AE-GAN system across three benchmark datasets
6. Table 3.2. Per-attack-category F1-scores on NSL-KDD for each model configuration, with gain of hybrid system over autoencoder baseline
7. Table 3.3. Ablation study results on NSL-KDD: effect of individual design choices on F1-score, FPR, and ROC-AUC of the hybrid AE-GAN system

List of Figures

1. Figure 1.1. Conceptual architecture of the hybrid autoencoder-GAN anomaly detection framework, illustrating the complementary pathways whose signals are combined into a unified detection decision
2. Figure 2.1. Inference pipeline of the hybrid ensemble detector: reconstruction error and discriminator probability are independently normalised and combined via a weighted linear ensemble whose coefficient α and threshold τ_h are calibrated on the held-out validation set
3. Figure 3.1. Comparative receiver operating characteristic curves for the standalone autoencoder (AUC = 0.923), GAN-based detector (AUC = 0.911), and hybrid AE-GAN system (AUC = 0.961) on the NSL-KDD test partition. The shaded region highlights the low-FPR domain (FPR ≤ 0.05) of primary operational relevance.

Annex

Appendix 1. Dataset Characteristics and Pre-processing Specification

Table A1.1 summarises the principal characteristics of the three benchmark corpora used in the experimental evaluation, including source organisation, publication year, total sample counts derived from the partition figures reported in Section 2.1, raw feature dimensionality prior to encoding, and post-processing dimensionality as supplied to the models.

Notes on Table A1.1: total sample counts are computed as the sum of training-normal, training-attack, validation, and test partition sizes reported in Table 2.1. Raw feature counts refer to the unencoded schema: NSL-KDD retains the original 41-column KDD Cup 1999 schema; CICIDS-2017 flows are exported by CICFlowMeter (80 columns), of which 2 direct-identifier columns are removed; UNSW-NB15 provides 49 columns in the original PCAP-derived CSV export, with categorical expansions producing the post-encoding dimensionality of 196. The attack families identified in each corpus are: NSL-KDD — DoS, Probe, R2L, U2R; CICIDS-2017 — Brute Force, PortScan, Botnet, DoS/DDoS, Heartbleed, Infiltration, Web Attacks (SQL Injection, XSS, Brute Force), and normal; UNSW-NB15 — Fuzzers, Analysis, Backdoors, DoS, Exploits, Generic, Reconnaissance, Shellcode, Worms.

Appendix 2. Autoencoder Architecture Specification

Table A2.1 presents the complete layer-by-layer architecture of the autoencoder. The input and output dimensionality d is dataset-dependent: d = 121 for NSL-KDD, d = 78 for CICIDS-2017, and d = 196 for UNSW-NB15. Batch normalisation is applied after each fully connected hidden layer; dropout is active during training only. The linear output layer does not apply batch normalisation, as the unbounded activation range is required to reconstruct unit-interval normalised inputs without sigmoid saturation artefacts.

The total number of trainable parameters for the NSL-KDD configuration (d = 121) is 52,816, inclusive of all weight matrices, bias vectors, and batch normalisation scale and shift parameters. The bottleneck dimensionality of 8 was selected by ablation over candidate values {4, 8, 16, 32} evaluated on held-out validation reconstruction loss: a bottleneck of 4 collapsed the detection margin to within 5 % of the attack-class error, while a bottleneck of 32 reduced the error differential below 3 %, rendering threshold calibration unreliable; the value of 8 provided the best separation.

Appendix 3. GAN Architecture and Training Protocol

The GAN is trained exclusively on normal-class traffic records under the same semi-supervised assumption applied to the autoencoder. The generator learns to synthesise plausible normal-traffic feature vectors from a low-dimensional noise prior; the discriminator learns to distinguish real normal flows from generator output, and its output probability is subsequently repurposed as an anomaly score at inference time. Tables A3.1 and A3.2 specify the generator and discriminator architectures respectively.

Layer normalisation is applied within the generator in place of batch normalisation because batch statistics computed over small minibatches of generated samples exhibit high variance in early training epochs when generator output is far from the real data manifold, producing noisy normalisation parameters that destabilise gradient flow. The linear output activation accommodates the unit-interval normalised feature space without requiring explicit sigmoid rescaling at inference.

Spectral normalisation is applied to all discriminator weight matrices to enforce the Lipschitz constraint, preventing the discriminator from assigning arbitrarily extreme probabilities to samples near the training manifold without incurring the computational overhead of explicit gradient penalty computation. Explicit gradient penalty (as employed in the Wasserstein-GP formulation) was evaluated during preliminary experimentation but was not adopted as the primary Lipschitz enforcement mechanism: on tabular data with the minibatch sizes used in this study, spectral normalisation produced equivalent training stability with substantially lower per-iteration cost.

The hybrid AE-GAN scoring mechanism combines the reconstruction-based anomaly score of the autoencoder with the discriminator-based anomaly score of the GAN through a normalised linear ensemble. Prior to combination, both raw scores are scaled to the unit interval using min-max normalisation parameters estimated on the normal-class validation partition: the normalised autoencoder score is defined as s̅_AE(x) = (ε(x) − ε_min) / (ε_max − ε_min), where ε(x) = ‖x − x̂‖₂² / d, and the GAN score s_GAN(x) = 1 − D(x) is already naturally bounded in (0, 1). The composite hybrid anomaly score is then computed as:

s_hybrid(x) = α · s̅_AE(x) + (1 − α) · s_GAN(x)