SafeChem: A Benchmark Dataset for Multi-Label Chemical Hazard Prediction and LLM Safety Hallucination Evaluation

<jats:p>We introduce SafeChem, a regulatory-grounded benchmark of 32,211 chemical substances annotated with molecular structure representations, physicochemical descriptors, and 30 binary Globally Harmonized System (GHS) hazard labels. Unlike prior molecular benchmarks queried from pharmaceutical databases, SafeChem is seeded from a curated hazardous materials registry (HMMIS), ensuring coverage of real-world industrial and safety-critical chemicals. We position it as a unified benchmark for chemical safety reasoning across two complementary regimes: predictive performance, measuring multi-label hazard classification from molecular structure, and safety reliability, measuring omission-prone hazard assessment by large language models (LLMs). Under predictive evaluation, a scaffold-split benchmark over 20 GHS categories reveals that all six baseline models achieve meaningful performance on high-prevalence labels (macro-AUPRC 0.45–0.55), while mid-prevalence labels remain substantially harder (macro-AUPRC 0.08–0.28), establishing a difficulty gradient that motivates future work on low-resource hazard prediction. Under reliability evaluation, a 500-substance high-hazard LLM stress test shows that all eight evaluated LLMs exhibit omission hallucination rates above 0.21, with the dominant failure mode being silent abstention: empty predictions indistinguishable from confirmed negatives in automated pipelines, disproportionately targeting the rarest and most dangerous hazard classes. Together, these regimes make SafeChem a benchmark resource for both task-specific molecular models and general-purpose LLMs in chemical safety workflows.</jats:p>