Did AI Safety Become Regulatory Capture?

Last updated: June 26, 2026. Primary keyword: AI regulatory capture.

Editorial note: This article investigates a question. It does not assume that AI safety is a pretext for monopoly power, and it does not accuse any person, lab, policymaker, or advocacy group of improper conduct without evidence.

The most important question in AI policy is not whether artificial intelligence should be regulated.

It should be.

The harder question is whether the rules now forming around frontier AI will protect the public without quietly protecting the companies that already dominate the field.

That distinction matters. A regulation can be justified by real risks and still produce market concentration. A company can sincerely believe in AI safety and still support rules that make life harder for smaller rivals. A policymaker can worry about catastrophic harm and still design a compliance regime that only the largest firms can afford. A startup founder can complain about regulatory capture and still understate genuine dangers. An open-source advocate can defend innovation and still be too casual about misuse.

The phrase “regulatory capture” is powerful because it describes a real failure mode: public rules can be shaped, narrowed, delayed, or weaponized by the industries they are supposed to govern. It is also a phrase that can become lazy. If every unwanted regulation is called capture, the term stops explaining anything.

So the fair question is not, “Did AI safety become regulatory capture?”

The fair question is: where, exactly, do AI safety arguments overlap with incumbent advantage, and where are they simply an appropriate response to unusually powerful technology?

This article separates facts, evidence, expert opinion, and speculation. It looks at the strongest arguments from AI safety researchers, economists, startup founders, open-source advocates, and policymakers. It discusses Dario Amodei’s public views only where relevant and sourced. It avoids personal attacks because the issue is bigger than any one person or company.

It also treats “AI safety” as a broad category. Technical work on alignment, interpretability, evaluations, cyber misuse, biosecurity, privacy, labor harms, and deployment governance are not the same thing. Kingy AI has covered related technical debates in pieces on deliberative alignment, alignment faking, external reviews of alignment-faking research, and AI consciousness and model welfare. This article focuses on the policy and competition layer: licensing, liability, compute thresholds, model evaluations, lobbying, compliance costs, and the balance between incumbents and challengers.

Pull quote: The uncomfortable possibility is that AI safety can be sincere, necessary, and market-concentrating at the same time.

The Short Answer

The short answer is: not exactly, but the risk is real.

AI safety did not simply “become” regulatory capture. Many AI safety concerns are serious, well documented, and not reducible to corporate self-interest. Frontier models can create credible risks around cyber abuse, biological information assistance, persuasion, fraud, autonomy, surveillance, labor disruption, and concentration of decision-making power. Governments have legitimate reasons to demand testing, documentation, incident reporting, security controls, and accountability from the organizations building the most capable systems.

But the regulatory capture concern is also not a conspiracy theory. Frontier AI regulation often targets activities that require huge compute budgets, specialized safety teams, legal departments, security infrastructure, and policy access. Those burdens are easier for large labs and cloud platforms to absorb. Some proposals around licensing, pre-deployment approval, compute reporting, and liability could raise the fixed costs of competing in frontier AI. If badly designed, they could entrench companies that already have capital, data, talent, chips, cloud partnerships, distribution, and direct channels into government.

The most accurate view is a double claim:

1. AI safety is a legitimate public-interest project. Dismissing it as a cover story for monopoly power ignores real technical and social risks.

2. AI safety policy can still become a tool of incumbent advantage. The danger is not only corrupt intent. It is institutional design.

This means the right response is not deregulation. It is better regulation: proportionate, capability-aware, competition-aware, open-source-aware, and transparent about who pays the compliance costs.

First, What Regulatory Capture Means

Regulatory capture describes a situation where regulatory institutions serve the interests of the regulated industry more than the public interest.

The classic modern account comes from economist George Stigler’s 1971 paper, “The Theory of Economic Regulation”, which argued that regulation is often acquired by an industry and designed for its benefit. Later public choice scholars and political economists expanded the idea, and modern policy organizations use “policy capture” more broadly to describe undue influence over rulemaking, enforcement, standards, and information flows.

Capture does not require a secret meeting in a smoke-filled room. It can happen through normal channels:

• Companies hire former regulators and policymakers.
• Agencies depend on industry expertise because the technology is complex.
• Rulemaking processes are dominated by organizations with legal and policy teams.
• Compliance rules are written around existing incumbent practices.
• Smaller firms lack the bandwidth to participate.
• Public fear creates demand for rules that sound tough but favor firms that can afford them.
• Regulators become more worried about visible failures than invisible lost competition.

Historical examples are contested, but the pattern is familiar. The Interstate Commerce Commission was created to regulate railroads, yet over time it became associated with rate structures and entry controls that could protect incumbents. Airline and trucking regulation in the United States limited entry and route competition until deregulation. In finance, complex compliance regimes can be necessary for stability, but they can also create fixed costs that favor large institutions. In telecom, spectrum rules, licensing, and infrastructure requirements can both prevent chaos and create entry barriers.

The lesson is not “regulation bad.” The lesson is that rule design affects market structure.

AI now faces that same test.

Facts, Evidence, Opinion, and Speculation

Before going further, the categories need to be clean.

That separation is not pedantic. It is the difference between analysis and accusation.

Timeline: How AI Safety Became AI Policy

The current debate did not begin with one bill or one company. It formed through overlapping developments in AI capability, public anxiety, research culture, and government response.

This timeline matters because it shows why the debate became so charged. AI safety moved from research labs and online forums into legislation before the public had a stable vocabulary for the risks.

The Strongest Case That AI Safety Is Not Regulatory Capture

The strongest defense of AI safety policy begins with a simple observation: the technology is not ordinary software.

Frontier AI systems are general-purpose, fast-improving, difficult to interpret, cheap to copy at the margin, and increasingly capable of assisting with code, persuasion, synthesis, design, planning, and automation. Even if the most dramatic existential-risk scenarios never arrive, the near-term governance problem is real.

AI safety researchers argue that advanced systems can fail in ways that are hard to detect before deployment. A model may appear harmless in routine testing but behave differently under pressure, in tool-using settings, or when given access to sensitive systems. It may help malicious users scale phishing, malware development, social engineering, or biological research workflows. It may become embedded in business, government, education, and security systems before institutions understand its failure modes.

That is why many safety researchers support evaluations, model cards, red teaming, incident reporting, staged deployment, cybersecurity requirements, and independent audits. A 2023 paper on frontier AI regulation argued that highly capable foundation models can pose emerging public-safety risks because dangerous capabilities may arise unexpectedly, misuse can be difficult to prevent after deployment, and capabilities can proliferate. A 2024 paper on safety cases for frontier AI argued that developers may need structured arguments, backed by evidence, explaining why a system is safe enough for a specific operational context. These measures are not obviously pro-incumbent. A world with no safety standards may also favor incumbents, because users and governments may trust only the biggest brands after a serious failure.

Dario Amodei, CEO of Anthropic, has publicly framed advanced AI as a technology with enormous upside and serious risks. In “Machines of Loving Grace”, he argued that powerful AI could dramatically accelerate progress in biology, medicine, neuroscience, economic development, and governance if handled well. In “The Adolescence of Technology”, he discussed categories of risk including misuse, autonomy, and concentration of power.

Those are public arguments, not proof of capture. One can disagree with Amodei’s policy preferences while acknowledging that his stated view is not simply “regulate my competitors.” Anthropic has also published a Responsible Scaling Policy, a voluntary framework tying safety and security measures to model capability levels. Critics may question whether voluntary policies are enough, but the document reflects a serious attempt to operationalize risk management.

Policymakers have their own reasons to act. If a frontier model is used in a major cyber incident, accelerates a dangerous biological workflow, or causes systemic harm through autonomous deployment, the public will not accept “the market was still experimenting” as an answer. Governments regulate aviation, pharmaceuticals, nuclear energy, banking, medical devices, and automobiles because some failures scale beyond private contracts. AI may not fit any of those categories perfectly, but the analogy is not absurd.

There is also a democratic argument. Without public rules, AI governance is left to private terms of service, lab discretion, cloud provider policies, and opaque safety boards. That is not necessarily more competitive or more accountable. Open-source advocates are right to worry about centralized control, but a completely private governance regime can also centralize power.

In short, the anti-capture case says:

• The risks are real enough to justify public oversight.
• The most capable systems create the greatest externalities.
• Evaluation and reporting duties can make markets more trustworthy.
• Voluntary lab policies are not a substitute for public accountability.
• Capture is a risk to manage, not a reason to abandon safety.

This is a strong case. Any serious article on this topic has to grant it.

Pull quote: The fact that a rule helps large companies comply does not prove the rule was written for them. It may prove only that large companies are the ones building the highest-risk systems.

The Strongest Case That AI Safety Can Become Regulatory Capture

The capture concern starts from a different fact: regulation has costs, and fixed costs change markets.

If a law requires pre-deployment testing, extensive documentation, external audits, cybersecurity controls, legal reporting, safety-case preparation, incident response systems, model lineage records, and senior-officer certifications, the burden does not fall evenly. A trillion-dollar company can hire compliance counsel, policy staff, evaluation teams, and security engineers. A ten-person startup cannot. An open-source research collective may not even have a legal entity capable of carrying the obligation.

Economists usually distinguish between variable costs and fixed costs. Variable costs scale with activity. Fixed costs must be paid before a firm can compete. Regulation that creates large fixed costs can reduce entry even when the rules apply equally on paper. That is why compliance design matters.

AI has several features that intensify this problem.

First, frontier model development already has high fixed costs. Training frontier systems requires chips, data pipelines, distributed systems expertise, safety testing, energy, security, and talent. Regulation can add another layer of fixed cost on top of an already concentrated market.

Second, the largest AI firms often have cloud partnerships, enterprise distribution, policy access, and legal capacity. They can participate in rulemaking, respond to consultations, and shape standards. Smaller companies are busy surviving.

Third, safety language can become a broad political umbrella. “Frontier AI safety” can refer to existential risk, national security, copyright, child safety, election integrity, biological misuse, model autonomy, privacy, and labor harms. That breadth can justify sweeping powers unless lawmakers define the risk surface carefully.

Fourth, licensing proposals can create permission bottlenecks. If a company must obtain government approval before training or deploying advanced models, the approval process itself becomes a scarce asset. Investors may prefer firms that already have licenses, regulators may prefer known institutions, and new entrants may struggle to prove safety without first operating at scale.

Fifth, open-source AI complicates enforcement. A law designed around centralized labs may fit closed model providers better than open-weight ecosystems, decentralized fine-tuning, academic releases, and small specialized models. If policymakers respond by restricting open release, they may reduce transparency, research access, and competitive pressure on closed incumbents.

This is why some startup founders and venture investors have criticized broad AI licensing proposals. They argue that the language of safety can become a moat. The concern is not always that large labs are acting in bad faith. It is that a regulatory regime can make “responsible AI” synonymous with “large enough to maintain a Washington policy shop.”

Open-source advocates add another warning. Open models allow independent research, local deployment, customization, education, security auditing, and competition outside closed APIs. They can reduce dependence on a handful of proprietary model providers. But open models also create real misuse questions because weights can be copied and modified. The policy challenge is to address misuse without treating openness itself as reckless.

The strongest capture case says:

• The largest AI companies can afford compliance better than challengers.
• Licensing and approval regimes can become entry barriers.
• Incumbents have more access to policymakers and standards bodies.
• Safety obligations may be written around closed-lab workflows.
• Open-source restrictions could reduce competition and independent oversight.
• Even sincere safety arguments can produce concentrated markets.

This is also a strong case. Dismissing it as anti-safety rhetoric would be a mistake.

A Comparison: Safety Regulation Versus Capture Risk

The same policy tool can look different depending on design.

The table shows why slogans fail. “Regulate AI” is too broad. “Stop regulation” is too broad. The details decide whether AI safety protects the public, protects incumbents, or does both.

What The EU AI Act Actually Does

The European Union’s AI Act is the most important AI law in force. It does not simply license all AI systems. It uses a risk-based structure: some uses are prohibited, high-risk systems face obligations, and general-purpose AI models face separate requirements.

The final Regulation (EU) 2024/1689 includes obligations for providers of general-purpose AI models. Providers must prepare technical documentation, provide information to downstream providers, respect EU copyright law, and publish summaries about training content. More demanding obligations apply to general-purpose AI models with systemic risk. The European Commission has also worked on a General-Purpose AI Code of Practice to help providers comply.

Supporters see this as a reasonable attempt to separate ordinary AI from models that may create broad downstream risk. Critics worry about administrative complexity, compliance uncertainty, and the possibility that the largest companies will turn compliance into a trust signal that smaller firms cannot match.

The open-source issue is especially delicate. The AI Act contains some differentiated treatment for free and open-source AI components, but the details are technical and do not remove every obligation. That matters because many open-source projects are not companies with compliance departments.

The fair reading is that the EU AI Act is not a cartoon of regulatory capture. It is a serious law responding to real concerns. But it is also a stress test for whether complex AI compliance can be implemented without pushing the market toward the few organizations that can afford specialized legal, safety, and documentation infrastructure.

California: SB 1047, The Veto, And The Next Version

California became the center of the U.S. frontier AI fight in 2024.

SB 1047, the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, would have imposed duties on developers of covered frontier models. The bill became controversial because it linked safety responsibilities to large training runs and created obligations around safety protocols, shutdown capability, and liability. Supporters argued it was a necessary first step for frontier AI accountability. Critics argued it would chill startups, open-source development, and academic work while failing to target the most concrete harms.

Governor Gavin Newsom vetoed SB 1047 on September 29, 2024. His veto message did not dismiss AI risk. Instead, it argued that the bill applied stringent standards based on whether a model was large and expensive, without considering whether the system was deployed in high-risk environments, involved critical decision-making, or used sensitive data. In other words, he objected to the design, not the premise that AI needs governance.

That veto is important because it shows the middle position. One can support AI safety and still reject a particular frontier model bill as poorly targeted.

California later chaptered SB 53, the Transparency in Frontier Artificial Intelligence Act, taking a different approach. The California Legislature’s SB 53 status page records that the bill was approved by the Governor and chaptered by the Secretary of State on September 29, 2025. The chaptered text focuses on large frontier developers, transparency, safety frameworks, reporting to the Office of Emergency Services, whistleblower protections, and a CalCompute study process. It defines a “frontier model” by a training compute threshold above 10^26 integer or floating-point operations and a “large frontier developer” as a frontier developer with more than $500 million in annual gross revenue together with affiliates. Whether SB 53 becomes a durable model or a stepping stone is still an open question, but it confirms that the policy debate did not end with SB 1047.

For startups, the California debate revealed a core fear: if the first serious AI laws are built around frontier labs, the legal definition of responsibility may end up shaped around organizations with billions of dollars in compute and safety staff. For safety advocates, the same debate revealed a different fear: if every frontier bill is defeated as anti-startup, society may wait until after a major incident to build the rules.

Both fears are rational.

AI Lobbying: What We Know And What We Should Not Infer

AI lobbying has increased sharply as AI policy has become more important. That is a fact, but it needs careful interpretation.

OpenSecrets and other watchdogs track lobbying expenditures and registrations across industries and issues. In April 2024, TIME reported using OpenSecrets data that the number of organizations lobbying the U.S. federal government on artificial intelligence rose from 158 in 2022 to 451 in 2023, with 334 of those organizations lobbying on AI for the first time in 2023. TIME also reported that Amazon, Meta, Alphabet, and Microsoft each spent more than $10 million on overall lobbying in 2023, while OpenAI, Anthropic, Cohere, Andreessen Horowitz, Y Combinator, civil society groups, unions, universities, and AI-safety nonprofits appeared in the AI lobbying surge. This is useful evidence that AI policy became a major target of corporate, nonprofit, academic, labor, venture-capital, and civil-society influence after ChatGPT.

The trend did not stop there. In April 2026, Axios reported that Anthropic and OpenAI posted their highest-ever lobbying expenditures in the first quarter of 2026, citing federal lobbying disclosures. Axios reported Anthropic at $1.6 million and OpenAI at $1 million for the quarter, while noting that both still spent less than larger Big Tech companies overall. That newer datapoint matters because it shows AI-specific firms becoming more direct Washington actors, not only technical labs watched by older platform companies.

But lobbying data does not prove regulatory capture by itself. Lobbying can be defensive, educational, self-interested, public-spirited, cynical, or all of the above. A company may lobby for weak rules, strong rules, clearer rules, narrow exemptions, preemption of state laws, open-source protections, export controls, procurement access, copyright positions, safety standards, or liability limits.

The responsible inference is narrower:

• Major AI companies and their partners have strong incentives to shape AI policy.
• They have resources that startups, academics, and civil society groups often lack.
• Policymakers depend on technical expertise that large labs can provide.
• This creates a structural capture risk even without proving corrupt intent.

The last point is the important one. Capture can be structural. If the government hears constantly from the same set of labs because they are the only ones with frontier systems, access to compute data, safety teams, and policy staff, the resulting rules may reflect their worldview even when everyone acts professionally.

That does not make large labs villains. It makes pluralism essential.

The Dario Amodei Question

Dario Amodei has become a central figure in this debate because Anthropic is both a major frontier AI lab and one of the companies most publicly associated with safety-centered AI development.

The fair way to discuss Amodei is to stick to public positions.

Amodei has argued that AI could bring enormous benefits if developed responsibly. He has also warned about risks from misuse, autonomy, concentration of power, and insufficient interpretability. Anthropic’s Responsible Scaling Policy lays out a framework for increasing safety and security requirements as model capabilities increase. Anthropic has also engaged in policy debates, including public discussion around California AI legislation.

Those facts do not establish regulatory capture. They establish that Anthropic is a safety-focused frontier lab with a strong policy voice.

The critical question is not whether Amodei personally believes in AI safety. Based on his public writing, he plainly does. The critical question is institutional: when a frontier lab argues for safety obligations, how should policymakers separate genuine public-risk management from rules that may entrench frontier labs?

This question applies to every major AI company, not only Anthropic. OpenAI, Google DeepMind, Meta, Microsoft, xAI, Amazon, Nvidia, and others all have interests in how AI rules are written. Some are model developers. Some are cloud platforms. Some are chip suppliers. Some are open-weight advocates in some contexts and closed-system operators in others. Their incentives differ, but none are neutral.

So the right standard is not suspicion of one executive. It is consistent scrutiny of all powerful actors.

The Open-Source Counterargument

Open-source AI advocates see the issue from a different angle. They argue that open models are one of the few checks on closed AI concentration.

Open-weight models allow researchers to inspect, test, fine-tune, distill, benchmark, and deploy systems without asking permission from a closed lab. They let startups build specialized products without paying API rents forever. They let governments and enterprises run models in controlled environments. They let smaller countries develop local AI capacity. They let security researchers study failure modes. They create a fallback if closed providers change pricing, restrict access, or become subject to geopolitical controls.

Kingy AI has covered the operational importance of model choice in its AI model selection guide. The broader point applies here: openness is not only an ideology. It is a market-structure tool.

The strongest open-source concern is that frontier safety laws will treat model release as inherently irresponsible. If the law makes open release too legally risky, the result may be a world where only closed, monitored, API-gated models are viable at scale. That world may be easier for regulators to supervise, but it may also concentrate technical power in a small number of firms.

The safety response is that open models can be misused. Once weights are released, they cannot easily be recalled. A highly capable open model could be fine-tuned for cyber abuse, fraud, or other harmful purposes. The open-source community sometimes answers this too quickly by pointing to existing misuse or by saying closed models are also dangerous. That is true, but it does not fully solve the release problem.

The right policy probably needs distinctions that current public debate often blurs:

• Open research code is not the same as open frontier weights.
• A small specialized model is not the same as a general frontier model.
• Publication of a paper is not the same as deployment of an autonomous agent service.
• A model host with usage controls is not the same as a torrentable weight file.
• Academic safety research is not the same as commercial release.

Open-source advocates are right that blunt rules can damage competition and independent oversight. Safety advocates are right that openness changes the risk calculus. The hard work is designing rules that recognize both.

The Startup Founder’s Problem

For startup founders, the debate is practical.

A founder does not ask, “What is the philosophical definition of regulatory capture?” A founder asks:

• Will I need a lawyer before I can train, fine-tune, or deploy?
• Will investors avoid my category because liability is unclear?
• Will cloud providers refuse me because compliance risk is too high?
• Will enterprise customers buy only from vendors with expensive certifications?
• Will open models remain available?
• Will safety evaluations become a procurement toll booth?
• Will policymakers understand the difference between a frontier lab and a small applied AI company?

Many AI startups are not training frontier models. They are building workflow tools, vertical agents, search products, copilots, data systems, evaluation layers, robotics applications, education products, and developer tools. Kingy AI’s AI launch tracker shows how broad the market is. A rule aimed at frontier training should not accidentally sweep in ordinary applied AI.

At the same time, startup founders should not pretend that “startup” means harmless. Small teams can deploy risky systems quickly. A small firm can mishandle sensitive data, automate bad decisions, misrepresent capabilities, or release insecure agentic tools. Some high-risk behavior deserves obligations regardless of company size.

The policy principle should be proportionality. Regulate based on capability, deployment context, access to sensitive systems, foreseeable misuse, and actual control. Do not regulate based only on whether a company is famous enough to be in the hearing room.

The Economist’s Lens: Fixed Costs, Moats, And Dynamic Competition

Economists usually look past moral language and ask how incentives change.

AI regulation can create at least four kinds of market effects.

First, it can create fixed compliance costs. If every serious model provider needs lawyers, auditors, safety researchers, reporting systems, and documentation pipelines, the minimum efficient scale rises.

Second, it can create certification moats. If enterprises, governments, and cloud platforms treat certain safety certifications as mandatory, firms without those certifications may be excluded even when their products are lower risk.

Third, it can create information asymmetry. Large firms can influence technical standards because they know more about frontier systems. Regulators may unintentionally encode incumbent architectures into law.

Fourth, it can create liability uncertainty. Vague duties can deter investment in new entrants, especially open-source and infrastructure projects where responsibility is distributed.

But economists can also make the opposite point. Lack of regulation can create its own concentration. If users fear AI failures, they may buy only from large trusted brands. If harms occur, insurers and enterprise buyers may demand private compliance frameworks. If governments avoid public standards, procurement may favor incumbents with existing relationships. If open competition produces a race to the bottom, responsible smaller firms may lose to reckless ones.

So the economist’s answer is not “less regulation.” It is “watch the cost curve.”

Good AI regulation should:

• Minimize unnecessary fixed costs.
• Use standardized reporting where possible.
• Create safe harbors for good-faith research and low-risk deployment.
• Avoid pre-approval unless the risk justifies it.
• Keep standards contestable and publicly governed.
• Distinguish frontier training from downstream applications.
• Support independent evaluation infrastructure that startups can access.

This is where the policy-design literature matters. A 2024 paper on moving from principles to rules in frontier AI regulation argued that high-level principles and specific rules have different strengths: principles are adaptable but harder to enforce, while rules are clearer but can become stale or box-checking. That tradeoff is also a capture issue. If rules are too specific, incumbents may shape them. If principles are too vague, regulators may rely too heavily on incumbent expertise to interpret them. In other words, competition policy and safety policy should be designed together.

The Policymaker’s Dilemma

Policymakers face an ugly asymmetry.

If they regulate too lightly and something goes wrong, the public will ask why they ignored warnings. If they regulate too heavily and competition quietly dies, the cost is harder to see. There is no front-page headline for the startup that never gets funded or the open-source project that never releases.

This asymmetry pushes governments toward visible action. Hearings, commitments, reporting duties, and named frameworks show the public that officials are doing something. The danger is that symbolic toughness can become substantive concentration.

Policymakers also face an expertise problem. Frontier AI is technically complex. Lawmakers need input from the organizations building it. But the more they rely on those organizations, the more they risk adopting their assumptions.

The solution is not to exclude frontier labs. That would be foolish. The solution is to widen the table:

• Independent AI safety researchers.
• Economists and competition scholars.
• Startup founders from applied AI and infrastructure.
• Open-source maintainers.
• Civil liberties groups.
• Cybersecurity experts.
• Labor representatives.
• Consumer protection experts.
• International partners.
• State and local officials who will administer consequences.

Regulatory capture is less likely when no single worldview controls the policy pipeline.

Where AI Safety Critics Overreach

Some critics of AI safety make the debate worse by flattening it.

They imply that catastrophic risk is invented by companies to scare regulators. That is too simple. Many AI safety researchers raised concerns before today’s leading labs had their current market positions. Some safety advocates are critical of frontier labs and want stronger public oversight, not weaker competition. Others work in academia, nonprofits, or independent research groups with no obvious incumbent interest.

Critics also sometimes treat open-source AI as automatically democratic. It is not. Open release can distribute power, but it can also distribute harm. A model that helps a rural hospital, a solo developer, or a local-language education project may also help a fraud ring, authoritarian surveillance vendor, or cybercriminal group. Openness changes who can use a capability. It does not decide whether the capability is safe.

Finally, critics sometimes understate the public’s demand for accountability. If AI systems become embedded in hiring, finance, education, medicine, law, infrastructure, and defense, “let builders build” will not be a sufficient governance philosophy.

The anti-capture argument is strongest when it targets specific policy designs, not when it dismisses safety itself.

Where AI Safety Supporters Overreach

Some AI safety supporters make the opposite mistake.

They treat scale as a proxy for risk even when deployment context matters. A large model in a controlled research environment may be less immediately harmful than a smaller model embedded in a high-stakes decision system. Newsom’s SB 1047 veto made a version of this point: regulation based mainly on model size can miss actual use-case risk.

Safety supporters can also understate compliance costs. “Just document your model” sounds simple until documentation becomes a legal artifact requiring lawyers, auditors, data governance, red-team logs, benchmark reproducibility, incident tracking, and executive signoff. For a large lab, that may be normal. For a small company, it may be existential.

They can also lean too heavily on frontier-lab expertise. The labs know the technology, but they are not neutral public institutions. Their safety frameworks may be useful inputs, but they should not become law by default.

Finally, safety supporters sometimes frame urgency in a way that narrows debate. If every delay is described as reckless, then questions about competition, openness, civil liberties, and administrative capacity can look like bad-faith obstruction. That is dangerous. A law that cannot survive hard questions is not ready to govern a general-purpose technology.

The safety argument is strongest when it is precise, humble, and competition-aware.

What Would Non-Captured AI Safety Regulation Look Like?

A better AI regulatory framework would not ask society to choose between safety and competition. It would treat both as public goods.

Here are design principles that would help.

1. Tier obligations by actual risk. Use a mix of compute, capability evaluations, deployment context, autonomy, access to sensitive systems, user scale, and misuse potential. Avoid relying on one threshold.

2. Keep low-risk AI outside frontier rules. Most AI startups should not inherit obligations designed for the largest training runs. Applied AI rules should be context-specific.

3. Prefer transparency and post-deployment accountability over licensing where possible. Licensing may be justified for extreme-risk systems, but broad pre-approval regimes should face a high bar.

4. Build public evaluation infrastructure. If evaluations matter, startups and researchers need access to affordable, independent testing. Otherwise evaluation becomes an incumbent service.

5. Create open-source safe harbors. Good-faith research, small models, academic releases, and noncommercial open-source work need clear treatment. Dangerous deployment and negligent release can still be addressed separately.

6. Make standards contestable. Standards should not be written only by frontier labs. They should include independent researchers, civil society, startups, open-source communities, and international experts.

7. Track competition effects explicitly. AI regulators should coordinate with competition authorities. Every major rule should include an entry-barrier assessment.

8. Avoid turning voluntary company frameworks into law without scrutiny. Lab policies can inform regulation, but public law should not simply codify one company’s operating model.

9. Protect whistleblowers and incident reporting. Safety depends on information escaping corporate filters.

10. Review rules frequently. AI capabilities and market structure change too quickly for static thresholds.

These principles do not solve every conflict, but they point toward regulation that is harder to capture.

A Practical Test For Every AI Safety Proposal

When evaluating a frontier AI rule, ask seven questions:

1. What specific harm is this rule trying to reduce?
2. What evidence shows that the rule would reduce that harm?
3. Who can comply cheaply, and who cannot?
4. Does the rule distinguish frontier development from ordinary AI use?
5. Does it preserve independent research and open-source competition where risk is lower?
6. Who writes the standards, and can outsiders challenge them?
7. How will the rule be updated if it entrenches incumbents or misses new risks?

If policymakers cannot answer those questions, the rule is not ready.

So, Did AI Safety Become Regulatory Capture?

The honest answer is unsatisfying: AI safety has not simply become regulatory capture, but AI safety arguments now operate inside a political economy where capture is a constant danger.

There are facts that support the safety side. Frontier models are powerful, general-purpose systems with real misuse and accident risks. Governments cannot ignore them. Technical safety work is not a corporate invention. Many researchers and advocates have spent years warning about risks that markets alone may not handle.

There are facts that support the capture concern. AI policy is being shaped while the market is already concentrated. Large firms have more lobbying capacity, more technical information, more compliance infrastructure, and more direct access to policymakers. Some regulatory tools, especially licensing and heavy fixed-cost compliance, can protect incumbents even when motivated by public safety.

The evidence does not justify accusing AI safety researchers as a class of serving monopolies. It also does not justify trusting frontier labs to define the public interest.

The most useful stance is disciplined suspicion: take the risks seriously, take the incentives seriously, and refuse to let either side use moral language to escape scrutiny.

AI safety should not be a synonym for “whatever the largest labs say responsible development requires.”

Startup freedom should not be a synonym for “no duties until something breaks.”

Open source should not be treated as either salvation or danger by default.

And regulation should not be judged by how tough it sounds in a press release. It should be judged by whether it reduces real harms while preserving the pluralism, competition, research access, and public accountability that a healthy AI ecosystem needs.

Questions Worth Asking Next

The debate would improve if more people asked questions like these:

• Which AI risks justify pre-deployment obligations, and which are better handled through liability, procurement rules, or post-deployment enforcement?
• How can governments regulate frontier systems without turning compliance into a moat?
• Should open-weight models have different obligations from closed API services, and where should the line be drawn?
• What public evaluation infrastructure would let startups prove safety without depending on incumbent-controlled standards?
• How should AI laws account for downstream deployment context, not only model size?
• What role should antitrust and competition agencies play in AI safety regulation?
• How can policymakers hear from frontier labs without letting frontier labs define the entire problem?
• When large AI companies call for regulation, what evidence should the public require before treating their proposals as neutral?
• When startup advocates oppose regulation, what evidence should they provide that their preferred alternative protects the public?
• What would it look like for AI safety, open-source AI, and competition policy to strengthen one another instead of fighting for control of the same conversation?

Those questions do not produce outrage. They produce governance.

And that is what AI needs now: not panic, not complacency, not monopoly by safety language, and not deregulation by startup slogan.

It needs rules good enough for the technology, and humble enough to know who might benefit from them.

Selected References

• George Stigler, “The Theory of Economic Regulation”.
• European Union, Regulation (EU) 2024/1689, Artificial Intelligence Act.
• European Commission, General-Purpose AI Code of Practice.
• Governor Gavin Newsom, SB 1047 veto message, September 29, 2024.
• California Legislature, SB 53 status and chaptered text.
• TIME, “There’s an AI Lobbying Frenzy in Washington. Big Tech Is Dominating”.
• Axios, “Anthropic outspends OpenAI in biggest-ever lobbying quarter”.
• Center for AI Safety, Statement on AI Risk.
• Anthropic, Responsible Scaling Policy.
• Dario Amodei, “Machines of Loving Grace”.
• Dario Amodei, “The Adolescence of Technology”.
• Solaiman et al., “The Gradient of Generative AI Release: Methods and Considerations”.
• Anderljung et al., “Frontier AI Regulation: Managing Emerging Risks to Public Safety”.
• Buhl et al., “Safety Cases for Frontier AI”.
• Schuett et al., “From Principles to Rules: A Regulatory Approach for Frontier AI”.
• Stanford HAI, AI Index Report.