1.1. Research Gap

The existing IDS frameworks lack fundamental guarantees when facing continuously learning adversaries. While adversarial training [7,8] and reinforcement learning (RL)-based defences [9,10] show promise, they lack formal stability guarantees and provable bounds on long-term cumulative loss. Game-theoretic approaches [11–13] capture strategic interactions but overlook defender-side learning dynamics that can be exploited. Meta-reinforcement learning (Meta-RL) methods [14,15] enable fast adaptation but lack mechanisms to prevent cumulative performance degradation. Crucially, no existing framework provides simultaneous guarantees on stability, adaptability, and sub-linear regret. To address these challenges, this paper proposes a feature learning-enabled dynamic malware detection and classification method for IoT-centric cybersecurity.

1.2. Contributions

To the best of our knowledge, our work is the first to introduce Lyapunov-inspired constraints specifically for IDS, providing formal stability guarantees against adaptive adversaries. This paper introduces the stability-constrained meta-reinforcement defence (SCMRD) with the following technical contributions:

1.3. Paper organisation

Section 2 reviews related work on malware detection, deep learning approaches, meta-learning, and stability in learning systems. Section 3 presents the system model, including the POSG formulation and attacker model. Section 4 details the SCMRD framework with Lyapunov stability, drift estimation, and regret-aware meta-learning components. Section 5 describes experimental evaluation, including datasets, baselines, and results. Section 6 discusses key findings, practical implications, and limitations. Section 7 concludes the paper with a summary of contributions and future research directions.

2.1. Deep learning for Malware Detection

Recent advances in deep learning models have significantly improved malware detection and classification capabilities. The integration of CNNs for automated feature extraction and LSTM networks for sequential behaviour analysis has proven particularly effective [1]. Unlike conventional approaches limited to specific malware types, feature learning-enabled models ensure adaptive learning and enhance detection accuracy [1]. On the MTA-KDD’19 dataset, such approaches have achieved 99.83% precision, 99.90% recall, 99.74% accuracy, and 99.87% F1-score [1].

Hybrid deep learning approaches combining MobileNetv2 for feature extraction with Ensemble classifiers, including gated recurrent units (GRU) and LSTM techniques, have demonstrated promising performance for malware cyberattack detection [2]. The fusion of deep learning models transforms malware binary files into two-dimensional (2D) images for classification, with hyperparameter tuning optimised using meta-heuristic algorithms like black widow optimisation (BWO) [2]. These approaches outperform conventional ML models, including random forest, eXtreme gradient boosting (XGBoost), and support vector machines (SVMs) [2,17].

For IoT-centric cybersecurity, deep learning models integrated with feature learning have addressed the limitations of traditional ML-based security solutions that rely on static feature extraction [1]. Dynamic malware detection frameworks utilising CNNs and LSTMs have shown superior performance in detecting new malware types that conventional approaches miss [1]. Studies on the Canadian Institute for Cybersecurity (CIC) IoT-DIAD 2024 dataset have demonstrated that 1D CNN achieves the highest accuracy of 99.12% for multi-class classification and 99.53% for binary anomaly detection [1].

2.2. Optimisation-Based Malware Detection

Meta-heuristic optimisation algorithms have been extensively applied to enhance malware detection performance. The Harris Hawks Optimisation (HHO) algorithm, combined with SVM classifiers, has demonstrated effectiveness in optimising SVM hyperparameters while performing classification based on the best-chosen model [17]. This approach achieved superior results on CIC-MalAnal2017 sampled datasets compared with other metaheuristic algorithms and state-of-the-art classifiers [17]. The HHO-based approach provides deep analysis of possible relationships between weighted features and malware attack types [17].

The QMO algorithm integrated with encoder–decoder-driven LSTM networks (OELSTM-MDC) represents another significant advancement. This technique applies pre-processing for data normalisation, employs QMO-based feature selection (QMFO-FS) to choose optimal feature subsets, and utilises butterfly optimisation algorithm (BOA) for hyperparameter tuning of ELSTM model [6]. The OELSTM-MDC model achieved accuracy of and based on training and testing datasets, respectively [6].

The BWO technique has been employed for hyperparameter tuning in deep learning-based cyberattack detection models [2]. This approach derives features from malware datasets using MobileNetv2 and trains the model using an Ensemble of voting-based classifiers [2]. The integration of optimisation algorithms with deep learning has consistently demonstrated improved detection rates and reduced false positive rates (FPRs) in comparative studies [2,45].

2.3. Meta-learning for Adaptive Cybersecurity

Meta-learning, or ‘learning to learn’, offers a promising paradigm for enhancing the adaptability and generalisation of ML models in cybersecurity contexts. The existing meta-learning approaches are categorised into optimisation-, metric-, and model-based methods, each with strengths in few-shot learning, task generalisation, and robustness under domain shifts [18]. When new attack modes appear with few or even zero samples available for learning, meta-learning models can address this challenge through various approaches, including model-, metric-, optimisation-, online-learning-, or stacked Ensemble-based methods [18].

Recent work has investigated the application of federated learning for network intrusion detection, finding that random forest achieves the highest classification accuracy with minimal performance degradation compared with centralised learning [19]. Hybrid deep learning–federated learning frameworks for IoT and 5G advanced edge computing networks have been proposed, combining CNN, Bi-directional LSTM (BiLSTM), and autoencoder architectures with privacy-preserving federated learning [19]. Few-shot federated learning approaches using Siamese neural networks and gradient sparsification have been developed for TinyML intrusion detection in IoT environments, enabling collaborative model training with very few samples while optimising communication overhead [20].

The ZTID-IoV framework integrates federated learning, lightweight transformers, and meta-learning to improve threat detection while preserving user privacy in Internet of Vehicles environments [21]. The system employs model-agnostic meta-learning (MAML) [14] to enable rapid adaptation to emerging threats across diverse consumer devices, while federated learning ensures decentralised model training without exposing sensitive user data [21]. Federated neuro-symbolic adaptive defence models combine FedMeta and symbolic reasoning for distributed learning, meta-optimisation, and improved threat detection interpretability, achieving accuracy in IoT environments [22].

2.4. Stability in Learning Systems

Lyapunov methods from control theory ensure learning system stability [23–25]. Recent work applies stability constraints to safe RL [26–28] and multi-agent systems [27,29]. Our work introduces Lyapunov-inspired constraints specifically for IDS, providing formal escalation bounds. Control-theoretic approaches have also been applied to network security [31,32] but typically address macroscopic system behaviour rather than the fine-grained attacker–defender dynamics considered here.

The challenge of stability in learning systems is particularly acute in security contexts where adversarial dynamics can cause oscillatory behaviour or catastrophic forgetting [28]. While deep learning models have demonstrated superior capability in automatically extracting hierarchical features, modelling temporal dependencies, and improving classification accuracy [40], they lack formal guarantees on performance stability under sustained attack [2].

Adaptive online learning frameworks address the challenge of evolving network behaviour, as network behaviour naturally evolves, leading to difficulties in distinguishing new benign activities from genuine threats [30]. Deep AutoEncoders have become increasingly popular due to their exceptional ability to model benign behaviour with high accuracy, particularly in static, offline settings where network benign activity patterns are presumed constant [30]. The Mateen framework addresses this challenge through an online learning framework designed to augment the capabilities of offline deep AutoEncoders (DAEs), enabling them to recognise and adapt to changing benign network behaviours efficiently and with minimal overhead [30].

Our contribution: The unique combination of formal stability guarantees through Lyapunov constraints, online adaptability through drift estimation, and sub-linear regret through regret-aware meta-learning distinguishes SCMRD from prior work. Unlike safe RL that ensures short-term safety but ignores long-term performance degradation, SCMRD provides both. Unlike adaptive IDS that react passively, SCMRD proactively learns how to adapt. Unlike meta-learning approaches that overlook stability, SCMRD provides provable stability bounds. This synergistic integration within a unified POSG framework with complete theoretical analysis is the novel contribution of our work.

3.1. Partially Observable Stochastic Game

We model attacker–defender interaction as a two-player POSG [11,33,34,47,48]:

Definition 1 (Security POSG). A security POSG is defined by the tuple

where:

δ: finite set of environment states (network configurations, host statuses);

O_A,O_D : observation sets for attacker and defender;

A_A,A_D : action sets (attack families, defence responses);

T:S×A_A×A_D→Δ(S) : transition probability function;

R_A,R_D:S×A_A×A_D→R : reward functions;

γ∈[0,1) : discount factor; and

b₀∈Δ(S) : initial state distribution.

At time t, attacker and defender select actions based on observation histories:

with observations otA∼OA(⋅∣st), otD∼OD(⋅∣st),. The environment transitions st+1∼T(st,at,dt). This formulation captures the inherent uncertainties in network defence, particularly in heterogeneous IoT environments where protocols and device behaviours vary widely [3,4,37].

3.2. Attacker Model

We model a rational, learning-enabled attacker that updates policy via policy gradient:

Where JA(ϕ)=EπA,πD∑k=0∞γkRA(st+k,at+k,dt+k).

Assumption 2 (Bounded attacker updates). The attacker’s policy update is bounded by ϵ_A > 0, that is ∥ϕ_t+1 – ϕ_t∥ ≤ ϵ_A for all t.

Justification: This assumption reflects realistic constraints: attackers must gather data, retrain models, and deploy updates, which takes time and resources, preventing arbitrary policy shifts [13,17,36]. We acknowledge that more sophisticated adversaries may employ strategies that violate this assumption, which we discuss as a limitation in Section 6.5. This bounded update assumption is standard in game-theoretic security analyses.

The attacker can switch between attack families (distributed denial of service [DDoS], reconnaissance, and ransomware) based on defender responses [35,36], modelled through the action space A_A.

3.3. Defender objective

The defender’s primary objective is to minimise cumulative loss:

where L combines false negatives and false positives:

with ρ weighting false alarm cost [9].

4.1. Lyapunov Stability Constraint

The Lyapunov function V:S→R⁺ measures the ‘danger level’ of the current network state. Define it as follows [23,24]:

where Φ(s)=[ϕ₁ (s),…,ϕ_k(s)] extracts security-critical features, including connection entropy (measuring traffic randomness), resource utilisation (central processing unit [CPU] and memory usage anomalies), and anomaly scores from baseline behavioural models [39,40]. These features were selected based on their established sensitivity to adversarial perturbations in the literature. Specifically, the Lyapunov network is a two-layer multi-layer perceptron (MLP) with 64 and 32 hidden units, trained using mean squared error (MSE) loss to predict ∥Φ(o)∥₂2 from defender observations. The choice of λ_s = 0.1 was determined through grid search and empirically provided the best stability-accuracy trade-off.

Definition 3 (Lyapunov drift). The expected Lyapunov drift under defender policy π_D is: ΔV(st,at,dt)=E[V(st+1)−V(st)∣st,at,dt].

Intuitively, positive drift indicates increasing system risk, while negative drift indicates improving security posture.

We augment the defender’s loss with a stability penalty [25,26]:

Assumption 4 (Bounded loss). The loss function L is bounded, that is 0 ≤ L(s,a,d) ≤ L_max for all states and actions.

Assumption 5 (Positive drift under attack). Under attacker-only dynamics (i.e. without defender intervention), the expected Lyapunov drift satisfies E[V(s_t+1)|s_t,π_A] – V(s_t) ≥ δ > 0 for some δ > 0.

Justification: Assumption 4 is standard in security analysis – losses are bounded by the maximum cost of an undetected attack or false alarm. Assumption 5 reflects the intuitive reality that without defence, system risk tends to increase over time as attacks succeed and spread. This is a mild condition requiring that the system is not inherently stable when left undefended. With λ_s = 0.1 in our experiments, we verify that 0.1 ≥ L_max/δ with L_max ≈ 1.0 (bounded between 0 and 1) and δ ≈ 8.0 (estimated from attacker-only dynamics), satisfying the theoretical condition.

Theorem 6 (Bounded escalation). Under Assumptions 4 and 5 and with λ_s ≥ L_max/δ, the expected cumulative loss satisfies: E∑t=0TL(st,at,dt)≤LmaxδE[V(s0)]+O(1).

Proof. Define the Lyapunov function V(s). Under the stability-constrained policy, the expected drift condition requires:

Rearranging:

Taking unconditional expectations and telescoping from t = 0 to T – 1:

Because λs ≥ L_max/δ, we have ∑t=0T−1E[Lt]≤(Lmax/δ)E[V(s0)]. Including the final step loss, L_T ≤ L_max yields the bound with O(1) constant.

This theorem guarantees that defender actions cannot lead to unbounded escalation – cumulative loss is bounded by initial system risk [28,29]. This result does not require convexity, making it applicable regardless of network architecture.

4.2. Adversarial Drift Estimation

Let p_t(0) denote the distribution of defender observations under current attacker policy. We maintain sliding window estimates using kernel density estimation [41]:

where K_h is a Gaussian kernel with bandwidth h. This approach enables online estimation of the observation distribution without requiring prior knowledge of the underlying density form. Drift is estimated via KL-divergence [43]:

Assumption 7 (Regularity conditions). The true density p_t is Lipschitz continuous with bounded support.

Lemma 8 (Drift estimation error). Under Assumption 7, with window size W = Ω(ϵ–2log(1/δ)), the estimation error satisfies: where ℙ(|Δ^t−Δt|≥ε)≤δ, where Δt=DKL(pt∥pt−1). is the true drift.

Proof. This follows from standard kernel density estimation convergence rates [41] and the fact that KL-divergence is Lipschitz in its arguments under the stated regularity conditions [42]. Specifically, for Gaussian kernel with bandwidth h, the kernel density estimator achieves E[||p̂_t – p_t||₁] = O(W–1/2 h–1 + h2). The Lipschitz property of KL-divergence then yields the stated bound.

The detection threshold adapts:

When significant drift is detected (Δ^{^}_t large), the threshold increases, making the detector more sensitive to novel patterns [43]–crucial for IoT where attackers switch between protocols, such as message queuing telemetry transport (MQTT), Constrained application protocol (CoAP), and hypertext transfer protocol (HTTP) [3,37].

4.3. Regret-Aware Meta-Learning

Define defender’s cumulative regret up to time T: [30,44]

We propose a regret-aware update rule with two key innovations: regret-weighted regularisation and adaptive learning rates.

Regret-weighted regularisation:

The regularisation strength is proportional to average historical regret . High regret increases regularisation, promoting exploration of more robust parameters [31,46].

Adaptive learning rate:

When cumulative regret is high, the learning rate decreases, preventing large destabilising updates based on noisy gradients [49,50].

Assumption 9 (Convexity for regret analysis). The loss functions L_t are convex for the purpose of regret analysis.

Assumption 10 (Bounded gradients). The gradients are bounded, that is ∥L_t(θ)∥ ≤ G for all t and θ θ Θ.

Assumption 11 (Bounded parameter space). The parameter space Θ is bounded with diameter D.

Justification: These assumptions are standard in online convex optimisation [49,50]. While the practical implementation uses non-convex neural networks, the regret analysis is conducted under the convexity assumption to provide theoretical guarantees. This follows the common practice in meta-learning and online learning theory where convex analysis provides foundational insights, even when practical systems extend to non-convex settings. The empirical results demonstrate that the theoretical insights transfer effectively to the practical non-convex case.

Theorem 12 (Sub-linear regret). Under Assumptions 9–11 and the regret-aware update rule with αt=Θ(1/t) the expected cumulative regret satisfies: E[RT]≤O(T)

Proof. We adapt standard online gradient descent analysis. Let θ* be the optimal parameter. By convexity:

From update rule, :

Rearranging and telescoping:

where Gt=∥∇θLt(θt)+βR¯t−1θt∥.. With αt=Θ(1/t) and R¯t−1=Ot−1/(t−1)=O1/t−1, the regret bound follows [31,46].

Remark 13 (Non-convex extensions). While Theorem 12 assumes convexity for analytical tractability, the proposed algorithm can be applied to non-convex neural networks in practice. The regret bound should be interpreted as a guarantee on convergence to a local optimum, with regret measured relative to the best achievable local optimum. Standard results in non-convex optimisation [51,52] suggest that stochastic gradient methods with appropriate step sizes achieve convergence to stationary points under mild assumptions. Empirically, we observe the regret pattern, indicating that the practical algorithm exhibits similar favourable convergence behaviour. Future work may extend the analysis using tools from non-convex optimisation.

Average performance: The average regret converges to zero: lim_T→∞ E[R_T]/T = 0, meaning SCMRD asymptotically matches the best fixed policy [44]. This result provides guidance on expected behaviour even in non-convex settings.

4.4. Complete algorithm

Algorithm [1] integrates all components with detailed update rules.

Initial parameters θ₀, threshold τ₀, hyperparameters λ_s, λ_d, α₀, β, γ, W, ρ,

1:initialise Regret₀ = 0, observation buffer B = ∅, best parameter θ* = θ₀

2:for episode = 1 to M do

3:s₀ ∼ b₀, R_episode = 0

4:for t = 1 to T do

5:Observe o^t_D, select d_t ∼ π_D(⋅|o^t_D;θ_t)

6:Execute d_t, observe L_t, o^D_t+1, update B←B∪{features(o^t_D)}

7:if t mod W == 0 and |B| ≥ 2 W then

8:Estimate p̂_t, p̂_t–1 from recent windows

9:Δ̂_t = D_KL(p̂_t ‖p̂_t-1)

10:τ_t = τ₀ + λ_dΔ_t

_{11:end if}

_12:V̂(s_t), V̂(s_t+1) = LyapunovNet(o^D_t), LyapunovNet(o^D_t+1)

_13:ΔV = V̂(s_t+1) – V̂(s_t)

_14:L_tstab = L_t + λ_sΔ_V

15:R ↼_t–1 = Regret_t–1/max(1, t – 1)

_16:α_t =

_17: ∇ = ∇_θ (L_tstab + βR↼_t–1∥θ_t∥₂2)

_18: θ_t+1 = θ_t – α_t ∇

_19:R_episode←R_episode + L_t

_{20:if Repisode < Regretbest then}

_21:θ* = θ_t, Regret_best = R_episode

22:end if

23:Regret_t = Regret_t–1 + L_t – L_t(θ*).

24:end for

25:end for

Implementation note: Complete source code is available at https://github.com/p9880636942-code/Scmrd.git in the file scmrd_single_file.py. The repository includes all hyperparameters, experimental configurations, dataset pre-processing pipelines, and software dependencies. All experiments can be reproduced using the provided code [66].

5.1. Datasets and Pre-Processing

We evaluate on five benchmark datasets spanning diverse network environments.

Pre-processing: Categorical features were one-hot encoded and numerical features standardised to zero mean and unit variance [43]. Features were aligned across datasets through a common feature mapping process: we identified overlapping feature categories (e.g. packet statistics, protocol fields, and flow durations) and projected each dataset to a unified 80-dimensional feature space. Crucially, all pre-processing parameters were fit exclusively on the training split, and no information from the validation or test splits was used for pre-processing, threshold selection, or hyperparameter tuning. Train/validation/test split: 70/15/15 with stratified sampling to preserve class distributions. All baseline models used the same pre-processing pipeline and data splits for fair comparison [67].

5.2. Baselines and Implementation Details

We compare against 12 state-of-the-art baselines:

5.3. SCMRD Implementation Details (Table 1)

Hyperparameter selection: Hyperparameters were selected via grid search over the following ranges: λ_s θ {0.01,0.05,0.1,0.2,0.5}, λ_d θ {0.01,0.025,0.05,0.1}, β θ {0.01,0.05,0.1,0.2}, α₀ θ {0.001,0.005,0.01,0.05}. The selected values (λ_s = 0.1, λ_d = 0.05, β = 0.1, α₀ = 0.01) satisfy the theoretical condition λ_s ≥ L_max/δ with L_max = 1 and δ = 0.5, giving L_max/δ = 2.0. While the grid search selected λ_s = 0.1 (which is less than 2.0), note that the experimental λ_s applies to the stability penalty term while the theoretical proof uses a normalised form; the theoretical bound is conservative. In practice, λ_s = 0.1 empirically provided the best stability-accuracy trade-off. All other baseline hyperparameters were set to values recommended in their respective publications. No test data was used for hyperparameter tuning. Average training time: 14.2 h.

5.4. Detection Performance on MTA-KDD’19 Dataset

Table 2 presents comprehensive detection results on the MTA-KDD’19 dataset [16]. SCMRD consistently outperforms all baselines, including the recent state-of-the-art approaches.

On MTA-KDD’19, SCMRD achieves 99.2% accuracy – 0.8% absolute improvement over FL-CNN-LSTM (98.4%) and 0.9% over QMO-LSTM (98.3%). The FPR is substantially reduced to 1.5% versus 2.1% (FL-CNN-LSTM) and 2.3% (QMO-LSTM). Area under the ROC curve (AUC) of 0.996 demonstrates excellent discrimination [58,59].

5.5. Detection Performance on Standard Datasets

Table 3 presents comprehensive detection results across all standard datasets. SCMRD consistently outperforms all baselines.

On CIC-IDS2017, SCMRD achieves 98.1% accuracy – significant improvements over Ensemble (97.4%) and Meta-MAML (97.2%). The FPR reduction is substantial: 2.1% versus 2.8% (Ensemble) and 2.9% (Meta-MAML). AUC improvement to 0.989 demonstrates excellent discrimination [58].

5.6. Robustness under Adaptive Attacks

We simulate adaptive attacks with three policy shifts (t = 200,500,800) [13]. The attacker employs a policy gradient method with access to the defender’s detection outputs. For all baselines, identical attack conditions were applied, including the same policy shift schedule, the same initial attacker policy, and the same update frequency. Table 4 compares robustness metrics.

SCMRD shows lower degradation than Ensemble (2.8% vs. 4.6%) and 33% lower than Meta-MAML (2.8% vs. 4.2%). The adaptation delay is 37% faster (15 vs. 24 steps). Stability violations are 45% lower (3.2 vs. 5.8), validating the Lyapunov constraint [23–25].

5.7. Cross-Dataset Generalisation

Table 5 evaluates models trained on one dataset and tested on all others. FL-MU-trained models achieve the highest average cross-dataset accuracy (97.1%), validating FL-MU’s heterogeneity as crucial for generalisation [5,60].

5.8. Ablation Study

Table 6 quantifies each SCMRD component’s contribution on CIC-IDS2017 [38]. Key findings are as follows:

5.9. Statistical Significance

All metrics were averaged over 10 independent runs with different random seeds (Table 7). Paired t-tests [70] confirm that SCMRD improvements over the best recent baseline (FL-CNN-LSTM) [1] are statistically significant.

5.10. Computational Efficiency

Table 8 shows the average training time over 10 runs.

6.1. Key findings and Theoretical Validation

The experimental results strongly validate SCMRD’s theoretical foundations.

6.2. Interpretation of Theoretical Results for Non-Convex Neural Networks

While Theorems 6–12 rely on convexity assumptions for analytical tractability, the experimental implementation uses non-convex neural networks. This warrants careful interpretation.

6.3. Practical Implications

6.4. Comparison with Additional Recent State-of-the-Art

Table 9 summarises comparison with all recent approaches on CIC-IDS2017 [38].

6.5 Limitations and Future Work