OpenAI is developing catastrophic risk prevention methods to address the potential dangers of cutting-edge artificial intelligence technologies.
To proactively address the potentially catastrophic risks posed by cutting-edge AI technologies, OpenAI is developing a risk-preparedness approach characterized by the creation of new teams and challenges.
As OpenAI reported in October 2023, the plan is consistent with its mission to build safe general artificial intelligence (AGI) by addressing a wide range of safety risks associated with AI.
OpenAI’s fundamental belief is that cutting-edge AI models (future technologies that surpass the capabilities of the top models currently available) have the potential to bring countless benefits to humanity.
However, OpenAI is aware that these models can pose increasingly serious risks. Our goal is to manage these risks by understanding the potential dangers of cutting-edge AI systems when they are misused now and in the future, and by building a robust framework to monitor, assess, predict, and protect against their dangerous capabilities.
OpenAI is forming a new team called "Readiness" as part of its risk mitigation strategy. According to OpenAI's report, the team will be led by Aleksander Madry and focus on capability assessment, internal red teams, and cutting-edge model evaluation.
Its work will range from near-term development models to models with AGI-level capabilities. The readiness team’s mission will include tracking, assessing and predicting, as well as protecting against catastrophic risks across multiple categories, including personalized persuasion, cybersecurity, and threats of a chemical, biological, radiological and nuclear (CBRN) nature, and autonomous replication and adaptation (ARA).
Additionally, the Readiness Team’s responsibilities include developing and maintaining a Risk-Informed Development Policy (RDP). This policy will detail OpenAI’s approach to developing rigorous assessments and monitoring of cutting-edge model capabilities, creating a set of safeguard actions, and establishing a governance structure for accountability and oversight of the entire development process.
RDP is designed to expand OpenAI’s existing risk mitigation work, helping to ensure the safety and consistency of new systems before and after they are deployed.
OpenAI is also seeking to strengthen its preparedness team to prevent catastrophic misuses by launching an AI Readiness Challenge, which aims to identify less obvious potential areas of concern and build teams.
It will award $25,000 in API credits to up to 10 top submitters, publish novel ideas and entries, and seek out preparation candidates among the challenge’s top competitors.
As cutting-edge AI technologies advance, OpenAI’s move underscores the need for rigorous risk management strategies in the AI field, highlighting the importance of being prepared for the potential for catastrophic misuse of these powerful tools. #人工智能 #OpenAI
