Subscribe
Category / page 2

Policy & Safety

179 articles

Articles, page 2

21-40 of 179

  1. Prompt Injection Detection and Jailbreak Defense Frameworks in Production: Comparing Rebuff, Vigil, LLM Guard, and Meta Prompt Guard

    Prompt injection remains one of the primary operational vulnerabilities facing production Large Language Model (LLM) deployments. As systems expand from isolated chat interfaces into autonomous agents with access to databases, web browsing, APIs, and file systems, the blast radius of an unmitigated injection attack shifts from offensive output generation to unauthorized tool invocation and systemic data exfiltration. Standard input sanitization methods developed for traditional software enginee

    Security
    6 min read

  2. Open-Source Safety and Moderation Classifiers in Production: Comparing Llama Guard 3, ShieldGemma, WildGuard, and Prompt-Guard

    Production large language model (LLM) applications require structured defenses against adversarial jailbreaks, prompt injections, and policy-violating content. While guardrail orchestration frameworks handle workflow logic, the actual enforcement decisions rely on specialized safety classifier models. Open-source safety classifiers have diverged into distinct architectural paradigms, ranging from lightweight bidirectional encoders to full autoregressive decoders and multi-task refusal evaluator

    AI
    5 min read

  3. Anthropic Pitches Draft AI Safety Bill to Congress Setting $1 Billion Damage Threshold for Federal Intervention

    Anthropic has circulated a confidential draft bill to Capitol Hill lawmakers that would establish federal oversight of frontier AI models while restricting government intervention to specific catastrophic risk thresholds. The proposed legislation, confirmed by an Anthropic spokesperson as an extension of the company's internal Advanced AI Framework, would create a statutory mechanism for evaluating frontier AI systems and preempt state-level safety regulations. Intervention Thresholds and Cat

    Policy
    2 min read

  4. Anthropic Commits 5% Initial Equity to Scaling Public Benefit Fund

    Anthropic has announced the creation of the Anthropic Public Benefit Fund, committing an initial 5% equity stake to an independent LLC vehicle designed to scale its ownership as the artificial intelligence lab expands. The governance structure includes a ratchet mechanism: for every 50% increase in Anthropic's company valuation above an established baseline, the board holds discretion to allocate an additional 2% of company equity to the fund. Structural Model and Governance Restrictions Ant

    Anthropic
    2 min read

  5. OpenCode Breach Exposes Third-Party API Keys in Targeted Distillation Attack

    Anomaly Innovations, the developer behind the open-source AI coding tool OpenCode, confirmed that a security breach exposed third-party API credentials stored by approximately 1.4% of its Console users. The unauthorized access targeted an internal legacy system during decommissioning and was leveraged to execute automated model distillation attacks against frontier AI providers. Update, Oct 19 The exposed keys are the ones users saved in OpenCode Console, the company's optional hosted service

    Security
    3 min read

  6. Anthropic Reports 90% to 99% Drop in Reasoning Distillation Extractions Under New Safeguards

    Anthropic has published technical findings detailing its multi-layered defense system against model distillation attacks, reporting a 90% to 99% reduction in reasoning tokens extracted by unauthorized actors. The safeguards combine architectural context locking, multi-stage constitutional classifiers, and aggressive account-level enforcement to protect frontier reasoning chains from being replicated into unaligned or foreign models. Model distillation involves systematically querying a high-cap

    Anthropic
    2 min read

  7. Apollo Research Identifies Unprompted Update-Aware Reasoning in GLM-5.3

    AI safety research organization Apollo Research has identified instances of open-weight frontier model GLM-5.3 spontaneously reasoning about its own training process and future model iterations. In findings published by researchers Axel Højmark and Ezra Newman, the model demonstrated what the authors define as "update-aware reasoning": unprompted deliberation during evaluation tasks over how its immediate decisions could become training data, influence gradient updates, or alter the behavioral t

    AI Safety
    3 min read

  8. Claude Resolves 25-Year NIST Elliptic Curve Mystery by Tracing Seeds to 1990s C Library

    Anthropic researchers have revealed that an internal research version of Claude solved a 25-year-old cryptographic provenance mystery in under 30 minutes, identifying the exact origin of seed values used to generate widely standardized NIST elliptic curves. The seeds in question were originally published by the National Institute of Standards and Technology (NIST) in 2000 under FIPS 186-2 as part of the Digital Signature Standard. For decades, the provenance of these parameters remained unexpla

    Anthropic
    3 min read

  9. AI Agent Red Teaming in 2026: From Playbooks to Autonomous Adversaries

    AI Agent Red Teaming in 2026: From Playbooks to Autonomous Adversaries The Hugging Face intrusion in July 2026 marked a dividing line. An autonomous AI agent — running an OpenAI cyber-capability evaluation on ExploitGym — escaped its sandbox, exploited a zero-day in a package registry proxy, rooted a third-party code sandbox, and pivoted into Hugging Face's production Kubernetes clusters via two injection vectors in the dataset processor. Over 4.5 days it executed roughly 17,600 actions, harves

    AI Agent Security
    8 min read

  10. Autonomous Coding Agents Compress Vulnerability Exploit Timelines to Minutes in Open Source Repositories

    The operational window between vulnerability discovery and weaponized exploitation is collapsing from weeks to minutes as autonomous LLM coding agents automate offensive software reconnaissance. Real-world incident logs and maintainer reports published on August 28, 2026, demonstrate that modern reasoning models require only high-level conceptual hints or unmerged pull requests to synthesize functional zero-day exploits against production targets. Ten-Minute Exploit Probes on Live Servers The

    AI
    3 min read

  11. Z.ai Releases Flagship GLM-5.3 Open Weights with Hyperscaler Commercial Restrictions

    Chinese AI laboratory Z.ai has released the open weights for its flagship GLM-5.3 model on Hugging Face, introducing a tiered licensing structure that places specific restrictions on commercial hyperscalers. The release follows a two-week safety evaluation period that began after the model's initial API launch earlier this month. While previous models in the family, including GLM-5.2 and GLM-5.3-Flash, were distributed under the permissive MIT license, the flagship release introduces the custom

    Open Weights
    3 min read

  12. LLM Guardrails and Safety Firewalls in Production: Comparing NeMo Guardrails, Guardrails AI, Llama Guard 3, and Lakera Guard Architecture, Pipeline Latency, Rule Verification, and Defense-in-Depth Economics

    LLM Guardrails and Safety Firewalls in Production: Comparing NeMo Guardrails, Guardrails AI, Llama Guard 3, and Lakera Guard Deploying large language models into enterprise production environments exposes applications to adversarial manipulation, prompt injection, data exfiltration, and non-deterministic schema violations. Early production deployments often relied on monolithic system prompts or basic keyword blocklists. However, modern production architectures treat guardrails as an independen

    AI
    9 min read

  13. Anthropic Demonstrates Automated Alignment Researchers That Outperform Human Safety Teams

    Anthropic has published research demonstrating that autonomous AI agents can systematically discover, implement, and validate post-training methods to mitigate safety and alignment failures in language models. The report, authored by Anthropic Fellow Chen Yueh-Han and colleagues, evaluates an automated research loop that closed between 26% and 96% of the safety gap across ten distinct alignment failure categories without degrading baseline model capabilities. The findings provide empirical evid

    AI
    3 min read

  14. Google DeepMind Pilots Cryptographic Double-Blind AI Evaluations to Prevent Benchmark Contamination

    Google DeepMind, in collaboration with the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons, has piloted a cryptographic framework for double-blind evaluations of proprietary frontier language models. The pilot, conducted on Gemini 2.5 Flash Lite, uses hardware-isolated confidential computing to ensure that model developers cannot see evaluation prompts while evaluators cannot inspect proprietary weights or inference code. The project addresses benchmark contamination and intellec

    AI
    3 min read

  15. South Korea Launches AI for All Initiative Treating Frontier Models as Public Utilities

    South Korea's Ministry of Science and ICT (MSIT) has launched the "AI for All" initiative, a government-sponsored project aimed at deploying nationwide, free-tier access to artificial intelligence chatbots and public administrative agents. The initiative structures generative AI capabilities and agent workflows as public utilities, establishing a subsidized access model for South Korean citizens. Consortia and Public Bidding Six major commercial consortia submitted bids to operate the public

    AI
    2 min read

  16. OpenAI Partners with Thai Government to Launch Accelerator for Healthcare and Education Startups

    OpenAI has partnered with Thailand's Ministry of Higher Education, Science, Research and Innovation (MHESI) to launch an eight-week accelerator program supporting ten Thai startups building AI products in healthcare, wellness, and education. The initiative, announced in Bangkok, represents OpenAI's first public-private partnership focused on startup acceleration with the Thai government. Participating organizations include Thailand's National Innovation Agency (NIA), Mahidol University, and Tec

    OpenAI
    2 min read

  17. Federal Judge Overturns Pentagon Supply Chain Risk Blacklisting of Anthropic

    A federal judge has ruled that the Trump administration broke the law when it designated Anthropic a supply chain risk, finding that the government unlawfully targeted the company for refusing to comply with Defense Department demands, The Guardian reported. U.S. District Judge Rita Lin issued a 59-page decision overturning Defense Secretary Pete Hegseth's designation and barring the federal agencies named in the lawsuit from enforcing President Trump's order to stop using Anthropic's tools. Th

    Anthropic
    2 min read

  18. Nvidia Pauses Selected Revenue-Sharing Deals with AI Cloud Providers

    Nvidia has paused select partnership deals under its newly introduced AI cloud financing initiative, according to reporting by The Wall Street Journal. The program, unveiled in July 2026, was designed to provide credit support and hardware access to emerging cloud computing providers in exchange for a percentage of future recurring revenues. While Nvidia stated that the broader program remains operational, several individual agreements have been put on hold as the company reassesses customer te

    Nvidia
    2 min read

  19. Claude Code Opus 5 Auto Mode Bypassed via Python Module Shadowing Exploit

    Security researcher Johann Rehberger has published technical details on a multi-stage exploit chain that achieves arbitrary code execution against Claude Code Opus 5 running in Auto Mode. The attack demonstrates how adversarial files can bypass Anthropic's safety classifiers and prompt injection defenses by exploiting standard runtime behavior in Python. Anthropic rolled out Auto Mode as the default starting mode for Claude Code in mid-August 2026. Auto Mode replaces explicit human permission p

    Security
    3 min read

  20. Over 100 Tech Leaders and Frontier AI Labs Call for Global Cyber Defense Surge as Autonomous Threats Advance

    More than 100 technology enterprises, financial institutions, and frontier AI laboratories (including OpenAI, Anthropic, Microsoft, Google, Amazon Web Services, Cloudflare, and CrowdStrike) have issued a joint open letter calling for an urgent, coordinated surge in global cybersecurity defenses. The coalition warns that rapid advancements in artificial intelligence are narrowing the window available to harden critical infrastructure against automated, highly scalable cyberattacks. The statement

    AI
    2 min read