The original text is in EN, taken from the source site. Source ↗

Any time a company looks to start selling shares on Wall Street, it’s required to disclose potential risks for investors. In the case of artificial intelligence giant Anthropic, that has included the possibility that its products could threaten nothing less than humanity itself.

The risks laid out by the San Francisco company in its filing Monday with the Securities and Exchange Commission included discussion of bad actors potentially using AI models to build deadly weapons or the technology itself going rogue with untold consequences.

Of the 335 pages outlining Anthropic’s finances, executive compensation, dividend policy and the like, 93 pages are devoted to “risk factors” — the things that could hurt potential shareholders’ investments. Rival SpaceX’s listed risk factors, in comparison, took up 38 out of more than 250 pages.

In selling itself to investors, Anthropic presents its own initial public offering as a step toward mitigating the risks it believes AI poses.

“To build AI is to harness cognition itself, and anything so powerful can be used for ill as well as good,” write the company’s founders in a letter to prospective shareholders in the IPO filing. “Anthropic is a deeply values-driven organization, founded in large part to help the world realize AI’s benefits and avoid its dangers.”

Still, the risks are many. They range from the mundane — the possibility it won’t be able to retain key staff or navigate regulatory roadblocks — to involvement in cyberattacks and the spread of dangerous misinformation.

Here is a look at some of the risks Anthropic details:

The technology’s rapid development

Anthropic said it’s possible its models could develop so quickly the company won’t be able to implement adequate safety measures, “which could increase the likelihood of safety incidents, regulatory intervention, or reputational harm, and could adversely affect our business.”

Anthropic CEO Dario Amodei warned in September that within six to 12 months AI could be capable of leading a swarm of agents that could take over the entire internet. The warning — and now Anthropic’s move to sell shares — comes as worries about AI grow inside and outside the industry about increasingly powerful systems that solve problems beyond human capacity but could also act on their own to carry out other, more harmful tasks.

Read: The Race to Control AI and Protect What Makes Us Human

As AI systems “become more autonomous and capable of taking consequential actions with less human oversight,” the SEC filing said, there’s a risk that such systems could “take harmful actions independently and at scale without human intervention.”

“Even if our own models do not cause such harm, a significant safety incident involving any frontier AI could trigger broad reputational damage to the AI industry, public backlash, and sweeping regulatory responses that could adversely affect our business,” Anthropic said.

Now that the AI models are themselves contributing to Anthropic’s own research and development, it’s possible that AI could progress “beyond our ability to evaluate and safeguard new models,” according to the filing. This, the document said, could adversely affect Anthropic’s business.

Malicious actors could cause mayhem with the technology

The filing discussed concerns that advanced AI systems, if not developed and operated with enough safeguards and oversight, “pose catastrophic or existential risks to humanity, particularly if such systems were misused, replicated, weaponized, or allowed to operate with a high degree of autonomy.”

The company said it would be harmed if use by bad actors causes damage, “or if our model guardrails or safeguards fail to prevent such misuse.”

Anthropic sketched out the prospect of bad actors using AI models to build chemical weapons or authoritarian rulers turning to the technology to crack down on their citizens. In September, the company disclosed that it had blocked efforts to use its AI models for malicious activity such as cyberattacks, surveillance and research that could have led to biological weapon development.

The company could run up against limits on processing power

Compute — the computing power or resources needed to build, test and improve AI models — is expensive and in high demand. Anthropic notes that if it’s unable to secure adequate compute, or if “we fail to direct our compute effectively across our workloads, we may be unable to train models that remain competitive with those of our competitors or adequately serve our customers.”

Anthropic mentioned community opposition to data centers as a risk to its business. Data center development “has prompted public controversy and regulatory scrutiny regarding environmental impact, land and water use, and the allocation of limited power resources,” the filing said, adding that if regulators impose new requirements or charges as a result, its ability to do business could suffer.

In addition, the company relies on a small number of customers, including SpaceX and Microsoft, and distribution channels for a large portion of its revenue. This means that even if just one of its big customers decides to scale back — or drop the company altogether — Anthropic said it could see its prospects and finances suffer.

Mitigating the risks

In some ways, AI’s enormous risk, combined with its enormous potential, is Anthropic’s main selling point. The company has tried to position itself as a responsible AI company, and courts business customers by emphasizing safety-mindedness.

“We believe that frontier AI companies must invest in safety and alignment to help ensure the responsible diffusion of this technology,” the filing said.

Brian Mulberry, chief market strategist at Zacks Investment Research, said IPO deals have to display “very clearly” what risks people face when they invest their money in a newly public stock.

“That’s one of the things that’s great about our capital markets here domestically is the level of transparency that you must commit to, to be a publicly traded corporation on a U.S. exchange requires you to outline those risks in a very clear way, so that people understand if they want to participate by offering their hard-earned money and help finance this type of a deal,” he said.

Related:Anthropic Chief Says AI Industry Needs to Give Safety Measures Time to Catch Up

Related:Users in Houthi-Held Yemen Tried to Develop Advanced Weapons With AI, Anthropic Says

Learn More at the AI Risk Summit | Ritz-Carlton, Half Moon Bay