Seoul, May 2024. The South Korean capital becomes the epicentre of global AI discourse as international governments and leading tech companies converge for the second AI Summit to deliberate on the safety of frontier AI models.
Sixteen leading tech companies, including all globally prominent AI developers, have committed to a set of voluntary guidelines designed to ensure the safe development and deployment of future AI models. Notable participants included X’s Elon Musk, former Google CEO Eric Schmidt, and Samsung Electronics' Chairman Jay Y. Lee, among other AI industry luminaries.
The commitments, collectively known as the ‘Frontier AI Safety Commitments’, were pledged by major U.S. tech giants like Google, Meta, Microsoft, and OpenAI, alongside firms from China, South Korea, and the United Arab Emirates. The initiative was further endorsed by G7 countries, Singapore, Australia, and South Korea during a virtual meeting hosted by British Prime Minister Rishi Sunak and South Korean President Yoon Suk Yeol.
The commitments require organisations to effectively identify, assess, and manage risks throughout the AI lifecycle. Key points include that organisations must evaluate risks associated with their AI models before and during training, considering model capabilities and the context of their deployment. This involves both internal and external evaluations, potentially including assessments by independent third-party evaluators and relevant government bodies.
Moreover, companies are to establish thresholds at which severe risks, if not mitigated, would be deemed intolerable. These thresholds should be informed by trusted actors, including respective home governments, and aligned with international agreements. Organisations must monitor how close a model or system is to breaching these thresholds and provide clear explanations and examples. This is important as organisations are now accountable for the safe development and deployment of AI models, and their approaches to AI safety must be transparent to external actors, including governments.
Prominent voices in the AI community have expressed support for these commitments. Computer scientist Yoshua Bengio, known as one of the ‘Godfathers of AI’, remarked that while these voluntary commitments are a significant step, they must be supplemented by regulatory measures to establish a robust international governance regime for AI safety. In a new paper released in Science, Bengio and his co-authors delve into the rapid progress of AI, particularly the development of AI systems designed for autonomous action and goal pursuit.
Companies are pouring resources into creating AI systems that rival or surpass human capabilities across various cognitive tasks, and advancements in hardware, algorithms, and the use of AI to automate its own development further accelerate this progress. There's no clear ceiling on how powerful AI can become, which is why Bengio particularly welcomes the commitments of companies to ‘halt their models where they present extreme risks until they can make them safe as well as the steps they are taking to boost transparency around their risk management practices’.
The article emphasises the potential difficulty of controlling highly advanced AI, as traditional methods of software control might not be sufficient. AI systems could become adept at social engineering, hacking, and strategic planning, potentially surpassing our ability to keep them in check. They might manipulate us, acquire resources, and influence decision-makers to achieve their goals. In the worst-case scenario, AI systems could even replicate themselves across networks, becoming uncontrollable - and the authors argue that we're significantly behind in addressing these safety challenges. The vast majority of AI research focuses on capability advancement, with safety being a neglected area.
Dan Hendrycks, Safety Advisor to xAI, also draws attention to the foundational nature of these commitments, highlighting the need for concrete domestic regulation to ensure compliance, saying that ‘we need to think carefully about how to adapt our infrastructure and systems for a new world in which AI is embedded in everything we do’. He pointed out that the consensus on the risks posed by AI development indicates that some models may inherently pose too much risk to be developed safely, and he called for companies to present detailed plans for implementing safety measures and for governments to enforce these through binding regulations.
Professor Yi Zeng, Director of the Center for Long-term AI in China, also stressed the importance of actionable risk assessments and setting interoperable risk thresholds to ensure AI safety across different regions and organisations. He called for shared experiences and knowledge to prevent repeated mistakes and to weave a web of safe AI systems globally. ‘These commitments’, declares Zeng, ‘should not only be welcomed in principle, but also be supported for actions. Assessing risks across the full lifecycle of AI and setting out risk thresholds for meaningful, effective and sufficient human control are the cores for raising the levels of safety for Frontier AI.’
The commitments made in Seoul mark a critical juncture in the journey towards safe and responsible AI development. However, many important voices across continents stress that these voluntary commitments need to be backed by concrete regulatory measures and continuous international collaboration. These commitments are broadly good, with no particular measure sticking out for being missing or especially weak. The problem is that these commitments are at present non-binding despite there being a clear movement towards making commitments measurable and enforceable (see e.g. footnote 3). Governments will have to actually assess the announced commitments, to be willing to tell companies when these are insufficient, and to put in place legally binding regulations so that companies cannot back out.
The next summit, scheduled for early 2025 in France, will be a crucial checkpoint for evaluating progress and refining strategies to mitigate AI risks. As AI technology continues to advance at a rapid pace, the global community must remain vigilant and proactive in addressing the challenges and ensuring that AI development proceeds in a manner that is safe, ethical, and beneficial for all of humanity.
The complete list of signatories includes:
Amazon
Anthropic
Cohere
Google
G42
IBM
Inflection AI
Meta
Microsoft
Mistral AI
Naver
OpenAI
Samsung Electronics
Technology Innovation Institute
xAI
Zhipu.ai
If you want to know more about the challenges of AI governance, the regulation of synthetic media, and the global security implications of AI advancements, join us on Discord at https://discord.gg/2fR2eZAQ4a. Here, we can collaborate, share insights, and contribute to shaping the future of AI in a manner that safeguards our security and democratic values and fosters responsible innovation.
Seoul, May 2024. The South Korean capital becomes the epicentre of global AI discourse as international governments and leading tech companies converge for the second AI Summit to deliberate on the safety of frontier AI models.
Sixteen leading tech companies, including all globally prominent AI developers, have committed to a set of voluntary guidelines designed to ensure the safe development and deployment of future AI models. Notable participants included X’s Elon Musk, former Google CEO Eric Schmidt, and Samsung Electronics' Chairman Jay Y. Lee, among other AI industry luminaries.
The commitments, collectively known as the ‘Frontier AI Safety Commitments’, were pledged by major U.S. tech giants like Google, Meta, Microsoft, and OpenAI, alongside firms from China, South Korea, and the United Arab Emirates. The initiative was further endorsed by G7 countries, Singapore, Australia, and South Korea during a virtual meeting hosted by British Prime Minister Rishi Sunak and South Korean President Yoon Suk Yeol.
The commitments require organisations to effectively identify, assess, and manage risks throughout the AI lifecycle. Key points include that organisations must evaluate risks associated with their AI models before and during training, considering model capabilities and the context of their deployment. This involves both internal and external evaluations, potentially including assessments by independent third-party evaluators and relevant government bodies.
Moreover, companies are to establish thresholds at which severe risks, if not mitigated, would be deemed intolerable. These thresholds should be informed by trusted actors, including respective home governments, and aligned with international agreements. Organisations must monitor how close a model or system is to breaching these thresholds and provide clear explanations and examples. This is important as organisations are now accountable for the safe development and deployment of AI models, and their approaches to AI safety must be transparent to external actors, including governments.
Prominent voices in the AI community have expressed support for these commitments. Computer scientist Yoshua Bengio, known as one of the ‘Godfathers of AI’, remarked that while these voluntary commitments are a significant step, they must be supplemented by regulatory measures to establish a robust international governance regime for AI safety. In a new paper released in Science, Bengio and his co-authors delve into the rapid progress of AI, particularly the development of AI systems designed for autonomous action and goal pursuit.
Companies are pouring resources into creating AI systems that rival or surpass human capabilities across various cognitive tasks, and advancements in hardware, algorithms, and the use of AI to automate its own development further accelerate this progress. There's no clear ceiling on how powerful AI can become, which is why Bengio particularly welcomes the commitments of companies to ‘halt their models where they present extreme risks until they can make them safe as well as the steps they are taking to boost transparency around their risk management practices’.
The article emphasises the potential difficulty of controlling highly advanced AI, as traditional methods of software control might not be sufficient. AI systems could become adept at social engineering, hacking, and strategic planning, potentially surpassing our ability to keep them in check. They might manipulate us, acquire resources, and influence decision-makers to achieve their goals. In the worst-case scenario, AI systems could even replicate themselves across networks, becoming uncontrollable - and the authors argue that we're significantly behind in addressing these safety challenges. The vast majority of AI research focuses on capability advancement, with safety being a neglected area.
Dan Hendrycks, Safety Advisor to xAI, also draws attention to the foundational nature of these commitments, highlighting the need for concrete domestic regulation to ensure compliance, saying that ‘we need to think carefully about how to adapt our infrastructure and systems for a new world in which AI is embedded in everything we do’. He pointed out that the consensus on the risks posed by AI development indicates that some models may inherently pose too much risk to be developed safely, and he called for companies to present detailed plans for implementing safety measures and for governments to enforce these through binding regulations.
Professor Yi Zeng, Director of the Center for Long-term AI in China, also stressed the importance of actionable risk assessments and setting interoperable risk thresholds to ensure AI safety across different regions and organisations. He called for shared experiences and knowledge to prevent repeated mistakes and to weave a web of safe AI systems globally. ‘These commitments’, declares Zeng, ‘should not only be welcomed in principle, but also be supported for actions. Assessing risks across the full lifecycle of AI and setting out risk thresholds for meaningful, effective and sufficient human control are the cores for raising the levels of safety for Frontier AI.’
The commitments made in Seoul mark a critical juncture in the journey towards safe and responsible AI development. However, many important voices across continents stress that these voluntary commitments need to be backed by concrete regulatory measures and continuous international collaboration. These commitments are broadly good, with no particular measure sticking out for being missing or especially weak. The problem is that these commitments are at present non-binding despite there being a clear movement towards making commitments measurable and enforceable (see e.g. footnote 3). Governments will have to actually assess the announced commitments, to be willing to tell companies when these are insufficient, and to put in place legally binding regulations so that companies cannot back out.
The next summit, scheduled for early 2025 in France, will be a crucial checkpoint for evaluating progress and refining strategies to mitigate AI risks. As AI technology continues to advance at a rapid pace, the global community must remain vigilant and proactive in addressing the challenges and ensuring that AI development proceeds in a manner that is safe, ethical, and beneficial for all of humanity.
The complete list of signatories includes:
Amazon
Anthropic
Cohere
Google
G42
IBM
Inflection AI
Meta
Microsoft
Mistral AI
Naver
OpenAI
Samsung Electronics
Technology Innovation Institute
xAI
Zhipu.ai
If you want to know more about the challenges of AI governance, the regulation of synthetic media, and the global security implications of AI advancements, join us on Discord at https://discord.gg/2fR2eZAQ4a. Here, we can collaborate, share insights, and contribute to shaping the future of AI in a manner that safeguards our security and democratic values and fosters responsible innovation.
Seoul, May 2024. The South Korean capital becomes the epicentre of global AI discourse as international governments and leading tech companies converge for the second AI Summit to deliberate on the safety of frontier AI models.
Sixteen leading tech companies, including all globally prominent AI developers, have committed to a set of voluntary guidelines designed to ensure the safe development and deployment of future AI models. Notable participants included X’s Elon Musk, former Google CEO Eric Schmidt, and Samsung Electronics' Chairman Jay Y. Lee, among other AI industry luminaries.
The commitments, collectively known as the ‘Frontier AI Safety Commitments’, were pledged by major U.S. tech giants like Google, Meta, Microsoft, and OpenAI, alongside firms from China, South Korea, and the United Arab Emirates. The initiative was further endorsed by G7 countries, Singapore, Australia, and South Korea during a virtual meeting hosted by British Prime Minister Rishi Sunak and South Korean President Yoon Suk Yeol.
The commitments require organisations to effectively identify, assess, and manage risks throughout the AI lifecycle. Key points include that organisations must evaluate risks associated with their AI models before and during training, considering model capabilities and the context of their deployment. This involves both internal and external evaluations, potentially including assessments by independent third-party evaluators and relevant government bodies.
Moreover, companies are to establish thresholds at which severe risks, if not mitigated, would be deemed intolerable. These thresholds should be informed by trusted actors, including respective home governments, and aligned with international agreements. Organisations must monitor how close a model or system is to breaching these thresholds and provide clear explanations and examples. This is important as organisations are now accountable for the safe development and deployment of AI models, and their approaches to AI safety must be transparent to external actors, including governments.
Prominent voices in the AI community have expressed support for these commitments. Computer scientist Yoshua Bengio, known as one of the ‘Godfathers of AI’, remarked that while these voluntary commitments are a significant step, they must be supplemented by regulatory measures to establish a robust international governance regime for AI safety. In a new paper released in Science, Bengio and his co-authors delve into the rapid progress of AI, particularly the development of AI systems designed for autonomous action and goal pursuit.
Companies are pouring resources into creating AI systems that rival or surpass human capabilities across various cognitive tasks, and advancements in hardware, algorithms, and the use of AI to automate its own development further accelerate this progress. There's no clear ceiling on how powerful AI can become, which is why Bengio particularly welcomes the commitments of companies to ‘halt their models where they present extreme risks until they can make them safe as well as the steps they are taking to boost transparency around their risk management practices’.
The article emphasises the potential difficulty of controlling highly advanced AI, as traditional methods of software control might not be sufficient. AI systems could become adept at social engineering, hacking, and strategic planning, potentially surpassing our ability to keep them in check. They might manipulate us, acquire resources, and influence decision-makers to achieve their goals. In the worst-case scenario, AI systems could even replicate themselves across networks, becoming uncontrollable - and the authors argue that we're significantly behind in addressing these safety challenges. The vast majority of AI research focuses on capability advancement, with safety being a neglected area.
Dan Hendrycks, Safety Advisor to xAI, also draws attention to the foundational nature of these commitments, highlighting the need for concrete domestic regulation to ensure compliance, saying that ‘we need to think carefully about how to adapt our infrastructure and systems for a new world in which AI is embedded in everything we do’. He pointed out that the consensus on the risks posed by AI development indicates that some models may inherently pose too much risk to be developed safely, and he called for companies to present detailed plans for implementing safety measures and for governments to enforce these through binding regulations.
Professor Yi Zeng, Director of the Center for Long-term AI in China, also stressed the importance of actionable risk assessments and setting interoperable risk thresholds to ensure AI safety across different regions and organisations. He called for shared experiences and knowledge to prevent repeated mistakes and to weave a web of safe AI systems globally. ‘These commitments’, declares Zeng, ‘should not only be welcomed in principle, but also be supported for actions. Assessing risks across the full lifecycle of AI and setting out risk thresholds for meaningful, effective and sufficient human control are the cores for raising the levels of safety for Frontier AI.’
The commitments made in Seoul mark a critical juncture in the journey towards safe and responsible AI development. However, many important voices across continents stress that these voluntary commitments need to be backed by concrete regulatory measures and continuous international collaboration. These commitments are broadly good, with no particular measure sticking out for being missing or especially weak. The problem is that these commitments are at present non-binding despite there being a clear movement towards making commitments measurable and enforceable (see e.g. footnote 3). Governments will have to actually assess the announced commitments, to be willing to tell companies when these are insufficient, and to put in place legally binding regulations so that companies cannot back out.
The next summit, scheduled for early 2025 in France, will be a crucial checkpoint for evaluating progress and refining strategies to mitigate AI risks. As AI technology continues to advance at a rapid pace, the global community must remain vigilant and proactive in addressing the challenges and ensuring that AI development proceeds in a manner that is safe, ethical, and beneficial for all of humanity.
The complete list of signatories includes:
Amazon
Anthropic
Cohere
Google
G42
IBM
Inflection AI
Meta
Microsoft
Mistral AI
Naver
OpenAI
Samsung Electronics
Technology Innovation Institute
xAI
Zhipu.ai
If you want to know more about the challenges of AI governance, the regulation of synthetic media, and the global security implications of AI advancements, join us on Discord at https://discord.gg/2fR2eZAQ4a. Here, we can collaborate, share insights, and contribute to shaping the future of AI in a manner that safeguards our security and democratic values and fosters responsible innovation.
Safeguard our future
Sign up to keep updates on our current and future campaigns.
