Security Vulnerabilities in AI Coding Agent Generated Code: A Large-Scale Empirical Study

Replication package for the paper submitted to EMSE 2026 Special Issue: "Agentic Software Engineering: The Rise of AI Teammates".

This study conducts a large-scale empirical analysis of security vulnerabilities in code generated by five mainstream AI coding agents (GitHub Copilot, Cursor, Devin, Claude Code, and OpenAI Codex), using 25,065 real-world GitHub Pull Requests from the AIDev dataset.

Key Findings

RQ	Question	Result
RQ1	What are the characteristics of security vulnerabilities?	3,965 static findings (0.37/KLOC), 2,305 PRs flagged by five-signal framework (444 high confidence). Top: Cryptographic Failures (A02, 35.9%), Injection (A03, 16.5%).
RQ2	Do different agents differ in vulnerability patterns?	Statistically significant but negligible practical differences (Cramér's V = 0.194, all Cliff's δ ≤ 0.190). Multi-signal flagging rates vary 3.32%–26.14% (χ² = 1340.22).
RQ3	What factors drive vulnerability introduction?	Programming language (Python OR=7.73) and code size (log(LOC) OR=2.17) dominate; agent choice has minimal independent effect (RF importance 5.6%).
RQ4	How effective is code review at catching vulnerabilities?	Dual-channel identification (keyword + LLM semantic): 130 security comments across 70 PRs. Escape rate: Scope A 99.27%, Scope B 87.43%.

Project Structure

shiyan/
├── README.md
├── run_all.py                  # Master script to run all RQs
├── common/
│   ├── __init__.py
│   ├── data_loader.py          # AIDev dataset loading & preprocessing
│   └── security_patterns.py    # 22 custom security detection rules (CWE/OWASP mapped)
├── RQ1V2/                      # Five-signal vulnerability identification
│   ├── scripts/
│   │   ├── run_rq1v2.py        # Multi-signal framework: Bot + Fix PR + Human Review + Static + Labels
│   │   └── rq2_rq3_supplement.py  # Multi-signal supplement for RQ2/RQ3
│   ├── data/
│   │   ├── pr_vulnerability_summary.csv   # 2,305 flagged PRs with signal counts & confidence
│   │   ├── signal1_bot_findings.csv       # Security Bot detection results
│   │   ├── signal2_security_fix_prs.csv   # Explicit security fix PR identification
│   │   ├── signal3_human_reviews.csv      # Human security review comments
│   │   ├── signal4_static_findings.parquet # Static analysis findings (3,965 records)
│   │   └── signal5_labels_issues.csv      # PR labels & related issues
│   └── results/
│       ├── rq1v2_statistics.json          # RQ1 descriptive statistics
│       └── rq2_rq3_supplement.json        # χ²=1340.22, logistic regression (OR=1.85–3.79), RF (AUC=0.728)
├── RQ2/                        # Cross-agent comparison
│   ├── scripts/run_rq2.py      # χ² test, Mann-Whitney U, Cliff's δ, Bootstrap
│   ├── data/                   # Agent metrics, pairwise tests (CSV)
│   └── results/                # rq2_results.json, rq2_bootstrap.json
├── RQ3/                        # Driving factors analysis
│   ├── scripts/run_rq3.py      # Logistic regression + Random Forest validation
│   ├── data/                   # PR-level dataset (CSV/Parquet)
│   └── results/                # rq3_results.json, rq3_rf_extended.json
└── RQ4V2/                      # Code review effectiveness (dual-channel)
    ├── scripts/
    │   └── run_rq4v2.py        # Keyword matching + LLM semantic classification
    ├── data/
    │   ├── comments_on_vuln_prs.csv       # All review comments on vulnerability PRs
    │   └── comments_classified.csv        # LLM-classified security comments
    └── results/
        └── rq4v2_results.json             # Escape rates: Scope A 99.27%, Scope B 87.43%

Requirements

• Python ≥ 3.11
• Semgrep ≥ 1.157.0
• Bandit ≥ 1.9.4

Python Dependencies

pandas
numpy
scipy
scikit-learn
statsmodels

Install

# Create conda environment
conda create -n emse_agentic_software_engineering python=3.11 -y
conda activate emse_agentic_software_engineering

# Install Python packages
pip install pandas numpy scipy scikit-learn statsmodels

# Install static analysis tools
pip install semgrep bandit

Dataset

This study uses the AIDev dataset by Li et al. (2025). Download it before running the scripts:

pip install huggingface_hub
python -c "
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id='hao-li/AIDev',
    repo_type='dataset',
    local_dir='../AIDev_datasets'
)
"

The dataset contains 18 Parquet files covering PR metadata, commit details, code diffs, review comments, and reviews for 5 AI coding agents.

Agent	PRs	Patches	Added LOC	LOC Share
OpenAI Codex	16,297	118,169	3,419,447	32.0%
Copilot	3,411	75,897	2,880,600	26.9%
Devin	3,810	89,353	2,754,089	25.7%
Cursor	1,181	19,889	821,069	7.7%
Claude Code	366	12,586	825,947	7.7%
Total	25,065	315,894	10,701,152	100%

Security Detection Pipeline

Multi-tool Static Analysis (Signal E)

1. Custom Regex Rules (22 rules) — Covers command injection (CWE-78), SQL injection (CWE-89), XSS (CWE-79), path traversal (CWE-22), hardcoded credentials (CWE-798/259), insecure deserialization (CWE-502), sensitive logging (CWE-532), HTTP cleartext (CWE-319), weak hashing (CWE-328), etc. Each rule includes exclude patterns to reduce false positives.

2. Semgrep (v1.157.0, auto ruleset) — Multi-language static analysis with community-maintained rules.

3. Bandit (v1.9.4) — Python-specific security scanner. Filters high-FP rules: B101, B110, B112, B403, B404.

Results are deduplicated by (PR ID, commit SHA, filename, line number, CWE ID) and mapped to OWASP Top 10 categories.

PR Process Data Mining (Signals A–D)

• Signal A: Security Bot automatic detection (highest confidence)
• Signal B: Explicit security fix PR identification (high confidence)
• Signal C: Human security review comments (medium confidence)
• Signal D: PR labels & related issues (auxiliary signal)

Five-Signal Aggregation

All five signals are aggregated per PR with confidence scoring: high confidence (≥3 signals or contains high-confidence signal), medium confidence (2 signals or contains medium-confidence signal), low confidence (single low-confidence signal only).

Statistical Methods

RQ	Methods
RQ1	Descriptive statistics, vulnerability density (findings/KLOC), five-signal aggregation
RQ2	χ² test + Cramér's V, Mann-Whitney U + Cliff's δ, Bonferroni correction (α' = 0.005), Bootstrap downsampling robustness check
RQ3	Univariate tests, multivariate logistic regression (OR, pseudo-R²), Random Forest (5-fold CV AUC, Gini importance, AUPRC)
RQ4	Dual-channel: keyword matching + LLM semantic classification (DeepSeek-V3 via SiliconFlow API), dual-scope escape rate (Scope A: all PRs, Scope B: reviewed PRs only)

Usage

Run Individual RQ

cd shiyan
python RQ1V2/scripts/run_rq1v2.py     # Five-signal vulnerability identification
python RQ2/scripts/run_rq2.py          # Cross-agent comparison
python RQ3/scripts/run_rq3.py          # Driving factors analysis
python RQ4V2/scripts/run_rq4v2.py      # Code review effectiveness (requires SILICONFLOW_API_KEY)
python RQ1V2/scripts/rq2_rq3_supplement.py  # Multi-signal supplement for RQ2/RQ3

Output

Results are saved as JSON in each RQ*/results/ directory. Intermediate data (CSV/Parquet) is saved in RQ*/data/.

Citation

@article{author2026security,
  title={All Agents Sin Alike: A Large-Scale Empirical Study of Security Vulnerabilities in AI Coding Agent Generated Code},
  journal={Empirical Software Engineering},
  year={2026},
  note={Under review, EMSE Special Issue: Agentic Software Engineering}
}

License

This project is licensed under the MIT License.