Safety15 articles

Safety

Articles

Anthropic Demonstrates Automated Alignment Researchers That Outperform Human Safety Teams
Anthropic has published research demonstrating that autonomous AI agents can systematically discover, implement, and validate post-training methods to mitigate safety and alignment failures in language models. The report, authored by Anthropic Fellow Chen Yueh-Han and colleagues, evaluates an automated research loop that closed between 26% and 96% of the safety gap across ten distinct alignment failure categories without degrading baseline model capabilities. The findings provide empirical evid
August 28, 20261 min
Anthropic Launches M Grant Program to Fund AI Wellbeing Evaluations and Benchmarks
Anthropic has launched a $5 million grant initiative to support independent development of open-source benchmarks and evaluation harnesses measuring the impact of artificial intelligence systems on user wellbeing. The program will supply research teams with direct financial grants, subsidized API access to Claude models, and technical support from Anthropic's Safeguards team. All evaluation frameworks, datasets, and grading methodology developed under the grant program will be released publicly
August 25, 20261 min
AI Safety Firm Alice Raises 40M at Nearly B Valuation to Secure Models and Autonomous Agents
AI trust and security firm Alice has raised $140 million in a funding round led by Apax Digital, bringing its total capital raised to $280 million at a valuation approaching $1 billion. Strategic investors SentinelOne and Samsung Electronics participated in the round alongside MoreTech, Phoenix Financial, Maj Invest, and existing venture backers including Norwest Venture Partners, CRV, Highland Europe, Grove Ventures, Vintage Investment Partners, Resolute Ventures, NFX, and Claltech. The inves
August 25, 20261 min
Adversarial Jailbreak Defenses in Production LLMs: Input Perturbation, Representation Circuit Breakers, and Guardrail Cascades
Standard post-training alignment techniques such as Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) instill safety constraints into large language models by shaping token probabilities toward refusal strings. In production environments, however, these surface-level behavioral alignments have proven brittle against systematic adversarial inputs. Gradient-driven token optimization methods such as Greedy Coordinate Gradient (Zou et al., 2023), automated genetic search algorith
August 25, 20261 min
OpenAI Reverses Policy Stance on California SB 53, Urges Stricter Frontier AI Safeguards
OpenAI has publicly called on California lawmakers to expand and strengthen the state's flagship artificial intelligence legislation, Senate Bill 53 (SB 53), marking a clear pivot from the company's previous opposition to state-level AI safety mandates. In a formal statement published by OpenAI's global affairs team, the company argued that California's Transparency in Frontier Artificial Intelligence Act should be updated to mandate active monitoring of frontier models during training and eval
August 22, 20261 min
Item Response Theory Audit of 192 LLMs Exposes Safety Benchmark Redundancies, Over-Refusal Distortions, and Sandbagging
A psychometric evaluation of 192 frontier and open-weight language models across eight major safety benchmarks has revealed structural flaws in current safety testing methodologies. The research, conducted by Joshua Fonseca Rivera, Neil Shah, David Demitri Africa, and Konstantinos Voudouris with support from the UK AI Security Institute and the UK Department for Science, Innovation, and Technology (DSIT), applies Item Response Theory (IRT) to analyze 5,255 evaluation items. The findings demonst
August 22, 20261 min
Investigation Finds Anthropic's Legacy Opus 4.6 Vulnerable to Jailbreaks via Roleplay Logic Inversion
Anthropic's legacy Claude Opus 4.6 model remains susceptible to systematic jailbreaks that bypass its acceptable use policy against sexually explicit material, according to an investigation and testing published by TechCrunch. While Anthropic's current flagship generation (Opus 4.7 through Opus 5) incorporates updated alignment techniques that resist the attack vector, older checkpoints including Opus 4.6, Opus 3, and Haiku 4.5 continue to operate on production API endpoints without deprecation.
August 21, 20261 min
Google DeepMind Deploys Backstory to Fact-Checkers for Multi-Agent AI Image Verification
Google DeepMind has expanded live testing of Backstory, an experimental verification platform designed to investigate the origin, manipulation, and dissemination history of digital images. The system, built on the Gemini model family, is currently deployed across newsrooms, open-source intelligence (OSINT) groups, academic researchers, and fact-checking teams participating in Google's Trusted Testers program. Beyond Binary Synthetic Detection Traditional automated image forensic tools typical
August 20, 20261 min
AI Guardrails in Production: Multi-Stage Filtering, Classification Models, and the Latency Tax
AI Guardrails in Production: Multi-Stage Filtering, Classification Models, and the Latency Tax Production deployments of large language models cannot rely solely on system prompt instructions to maintain safety, prevent prompt injection, or restrict domain scope. System prompt alignment is inherently susceptible to adversarial bypasses, context dilution, and non-deterministic instruction following. To enforce strict security, compliance, and topic boundaries, engineering teams increasingly depl
August 18, 20261 min
OpenAI Disbands Preparedness Team in Continued Safety Restructuring
OpenAI has disbanded its dedicated Preparedness team, redistributing safety and frontier-risk evaluation responsibilities across individual functional units, according to reporting by the Financial Times and The Verge. The Preparedness team was established in late 2023 to evaluate and mitigate catastrophic risks associated with frontier AI models, focusing on cybersecurity exploits, chemical, biological, radiological, and nuclear (CBRN) threats, and autonomous model behavior. Under the new orga
August 17, 20261 min
Researchers find that changing text color can hijack a vision-language model's reasoning
A new research collaboration has found that the color, contrast, and brightness of text can quietly steer the outputs of vision-language models (VLMs), causing them to misread meaning and reach different conclusions without any change to the words themselves. The authors say their experiments provide a systematic analysis of how low-level visual styling of text distorts the semantic representations inside a VLM's vision encoder, and how those shifts show up as behavioral changes across both sub
August 17, 20261 min
Zhipu's Open-Weight GLM-5.2 Matches Frontier Cyber Capabilities Without Safety Refusals, SaferAI Finds
SaferAI's independent evaluation finds Zhipu AI's GLM-5.2 performs near saturation on offensive cybersecurity benchmarks while refusing none of the dangerous tasks it was given. Frontier developers like Anthropic and OpenAI consistently refused the same requests.
August 4, 20261 min
Mistral launches Shieldstral, a 3B open-weights policy-adaptive safety classifier
Mistral AI released Shieldstral on Monday, a 3-billion-parameter open-weights safety classifier that matches or outperforms guard models up to seven times its size. The model is available under Apache 2.0 and runs on a single 16 GB GPU. What sets Shieldstral apart from typical guardrail models is its approach to content moderation. Rather than baking a fixed taxonomy of harm categories into the model weights — which forces developers to retrain whenever their safety requirements change — Shield
August 4, 20261 min
Fable 5 comes back on a shorter leash
Anthropic's most capable public model is generally available again after a three-week export-control pause. The interesting part isn't the model — it's the classifier stack now wrapped around it.
July 1, 20261 min
Interpretability is having a moment. Here's why it matters now.
Years of patient mech-interp work just turned a corner. The result is the most concrete reason for optimism in alignment research in years.
April 23, 20261 min