MaliciousSkillBench

Why MaliciousSkillBench?

Existing malicious Agent Skill resources are fragmented across sources, artifact formats, provenance conventions, label evidence, and benign support. MaliciousSkillBench consolidates these resources rather than concatenating them, with canonicalization, deduplication, structural control, cross-label conflict handling, and benchmark-specific evaluation protocols.

@misc{wang2026maliciousskillbench,
  title={MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection},
  author={Wang, Yue and Liu, Yi and Deng, Gelei and Zhang, Ying and Li, Yuekang and Chen, Zhenyu and Zhang, Leo},
  year={2026},
  eprint={2608.19901},
  archivePrefix={arXiv},
  primaryClass={cs.CR},
  url={https://arxiv.org/abs/2608.19901}
}

Benchmark at a Glance

Benchmark Construction

Existing Agent Skill security resources are fragmented across artifact forms, provenance, label evidence, and benign support. MaliciousSkillBench consolidates them rather than concatenating them.

Threat Landscape

A multi-label harmonized attack taxonomy covers 4,983 of 7,505 malicious identities across 11 categories. This is partial mapping, not complete attack coverage. Figure 2 is the quantitative threat landscape; the miniature matrix in Figure 1 is schematic.

Detection Findings

Word-SVM Macro-F1 remains high under random and structural-disjoint splits, then drops under source shift. This is source-conditioned generalization, not a universal out-of-distribution claim.

Under Source-Disjoint evaluation, Word-SVM retains 95.6% malicious recall but reaches a 62.4% benign false-positive rate. Cross-source degradation is dominated by benign over-flagging while malicious recall remains high.

Dataset Access

9,740 benchmark identities are publicly represented. Exact frozen text is available for 9,735; five sensitive records provide sanitized public representations. Hugging Face is the primary data host.

Dataset: https://huggingface.co/datasets/ProtectSkills/MaliciousSkillBench

from datasets import load_dataset

dataset = load_dataset("ProtectSkills/MaliciousSkillBench")

Getting Started

The GitHub repository is the code and reproducibility hub. It hosts schema, source, taxonomy, and protocol documentation, split manifests, and baseline evaluation scripts. The Dataset button points to the Hugging Face repository. The Paper button points to the public preprint.

Paper

MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection

Yue Wang, Yi Liu, Gelei Deng, Ying Zhang, Yuekang Li, Zhenyu Chen, and Leo Zhang.

Paper: https://arxiv.org/abs/2608.19901

PDF: https://arxiv.org/pdf/2608.19901

Responsible Use

The benchmark contains malicious or adversarial Skill instructions for defensive research. Do not execute untrusted Skills. The public release emphasizes static instruction text; some content is withheld. See RESPONSIBLE_USE.md.