Anthropic proposes that the world’s top artificial intelligence companies should come up with a unified way to halt the development of advanced AI systems, which risks humans losing control as the technology advances rapidly.
The company behind the Claude chatbot said in a blog post Thursday that as AI gets increasingly faster at performing sophisticated tasks, the option to slow or temporarily pause its development is good for the world.
Anthropic plans to work with its internal research firm to investigate the issue and “take steps” to help develop systems for reliable slowdowns or pauses.
Anthropology rival OpenAI argued for a different approach in a report released Wednesday, saying that “democratic governments — not private companies acting alone — should ultimately decide the rules, safeguards and accountability mechanisms.”
“Our view is that decisions about the pace of AI innovation should not be left to any one laboratory, institution or special interest group,” it said.
AI models are growing rapidly, with rapid increases in how quickly they can perform software tasks such as coding, Anthropic said in its post. Based on current trends and with sufficient computing power, an AI system can design and develop its own successor, a process known as “cyclical self-improvement”.
Self-managing AI will be a major technological milestone that will bring benefits to science, healthcare and other areas, but it could also “increase the risk of humans losing control over AI systems,” Anthropic said.
Some tech industry figures have long warned of such a scenario.
Anthropic’s post comes after a strange warning from a team of University of Toronto researchers this week who showed how they could use AI tools to create a new type of AI “worm” that spreads from device to device and takes over vast computer networks.
“I think it’s really important that people understand that it’s not just the largest, most powerful language samples that cause security concerns,” lead researcher Nicholas Papernot said in an interview.
Authors of the Anthropic post, company co-founder Jack Clark and head of its research institute Marina Favaro, said the hiatus will be used to enable “social structures and alignment research” to pursue AI advances.
Alignment is industry shorthand for ensuring that technology matches human values and goals.
The proposed integration would allow advanced AI labs to verify that global competitors have actually stopped or slowed down their work, and “a bad actor cannot surreptitiously advance under the auspices of a coordinated slowdown.”
The agency said a unified global mechanism is needed because without a slowdown in AI development, “less cautious” players could add pressure on companies and governments as they make tough choices about AI security.
Anthropic’s post comes as the company and ChatGPT-producer OpenAI compete to sell shares on the stock market, valuing Anthropic at nearly a trillion US dollars in an IPO.
Paperknot notified Canadian cybersecurity officials ahead of the release of its report, which shows how researchers developed the worm in the lab using an “open-source” AI tool that is cheaply accessible and easy for software developers to modify.
“In the past, cyber attackers would focus on high-value targets,” he said. “Banking systems, hospitals, power grids, water treatment systems, schools.”
Paperknot acknowledged that as AI-powered hacking tools supercharge the search for computer vulnerabilities, there needs to be more collaboration between companies, government agencies and academic researchers to develop countermeasures.
“That old laptop you have in your basement that you don’t check regularly may not seem like a high-value target, but it can be used as a launch pad to attack these high-value targets,” he said. “Anything connected to the Internet is now at risk because of how low the cost of mounting these cyberattacks is.”
#Major #firm #insists #pausing #development #risks #mount