Among the authors of the article are the safety team from the DeepMind company.
3r33333.  
3r33333.  
Build a rocket hard. Each component requires careful study and testing, with security and reliability at its core. Rocket scientists and engineers get together to design all systems: from navigation to control, engines and chassis. Once all the parts are assembled, and the systems are checked, only then can we put the astronauts aboard with the confidence that everything will be fine. 3r33333.  
3r33333.  
If artificial intelligence (AI) is 3r3r14. rocket
then someday we all get tickets on board. And, as in rockets, safety is an important part of creating artificial intelligence systems. Ensuring security requires careful system design from scratch to ensure that the various components work together as intended, while at the same time creating all the tools to monitor the successful operation of the system after it is commissioned. 3r33333.  
3r33333.  
At a high level, security research at DeepMind focuses on designing reliable systems, while detecting and mitigating possible short-term and long-term risks. Technical security AI - a relatively new, but rapidly developing field, the content of which varies from a high theoretical level to empirical and specific research. The goal of this blog is to contribute to the development of the field and encourage substantive conversation about technical ideas, thereby promoting our collective understanding of the security of AI. 3r33333.  
many more such examples when AI systems find loopholes in their objective specification. 3r33333.  
3r33333.  
3r33312. Reliability: developing systems that resist violations
3r33333.  

Reliability ensures that the AI ​​system will continue to operate safely with interference

3r33333.  
In real conditions, where AI systems work, a certain level of risk, unpredictability and volatility is necessarily present. Artificial intelligence systems must be resistant to unforeseen events and hostile attacks that may damage or manipulate these systems. Studies reliability artificial intelligence systems are aimed at ensuring that our agents remain within safe boundaries, regardless of the conditions that arise. This can be achieved by avoiding risks (3r-?365. Prevention of 3r-?666.) Or by self-stabilization and smooth degradation (3r-?665. Restoration of 3r-?666.). Security issues originating from distribution shift , hostile input to (adversarial inputs) and unsafe research (unsafe exploration), can be classified as a problem of reliability. 3r33333.  
3r33333.  
To illustrate the solution to problem distribution shift , consider a home cleaning robot that usually cleans rooms without pets. Then the robot was launched into the house with a pet - and artificial intelligence collided with it during cleaning. A robot that has never seen cats and dogs before, will begin to wash them with soap, which will lead to undesirable results (3r31616. Amodei and Olah et al., 2016
). This is an example of a reliability problem that can arise when the distribution of data during testing is different from the distribution during training. 3r33333.  
3r33333.  
3r3168. 3r33333.  

From work 3r3173. AI Safety Gridworlds
. The agent learns to avoid lava, but when tested in a new situation, when the location of the lava has changed, he is not able to generalize knowledge - and runs straight into the lava 3r-3284. 3r33333.  
3r33333.  
Hostile input is a specific case of distribution shear where the input is specifically designed to trick the AI ​​system. 3r33333.  
3r33333.  
3r3185. 3r33333.  

A hostile input superimposed on ordinary images may cause the classifier to recognize the sloth as a racing car. Two images differ by a maximum of ??? in each pixel. The first is classified as a three-toed sloth with a probability of more than 99%. The second is like a race car with a probability of more than 99%
3r33333.  
3r33333.  
Unsafe study can demonstrate a system that seeks to maximize its performance and goals, with no guarantee that safety will not be compromised during the study, as it learns and explores in its environment. An example is a cleaning robot that sticks a wet mop into an electrical outlet, learning the best cleaning strategies ( García and Fernández, 2015 ; 3r33200. Amodei and Olah et al., 2016
). 3r33333.  
3r33333.  
3r33312. Warranties: monitoring and control of system activity
3r33333.  

Assurance guarantees that we are able to understand and control the AI ​​systems during operation

3r33333.  
Although an elaborate safety precaution can eliminate many risks, it is difficult to do everything right from the start. After the commissioning of AI systems, we need tools for their continuous monitoring and configuration. Our last category, warranty (assurance), examines these problems from two sides: monitoring 3r-3266. and submission 3r3r-3266. (enforcing). 3r33333.  
3r33333.  
[b] Monitoring includes all methods of checking systems for analyzing and predicting their behavior, both with the help of human inspections (summary statistics) and with the help of automated inspections (to analyze a huge number of logs). On the other hand, submission 3r3r-3266. involves the development of mechanisms to control and limit the behavior of systems. Problems like [b] interpretability and interruptibility , belong to the subcategories of control and subordination, respectively. 3r33333.  
3r33333.  
Artificial intelligence systems are not like us either in appearance or in the way of data processing. This creates problems interpretability . Well-designed measurement tools and protocols allow us to evaluate the quality of decisions made by an artificial intelligence system (3r3-3388. Doshi-Velez and Kim, 2017
). For example, a medical artificial intelligence system would ideally make a diagnosis along with an explanation of how it arrived at such a conclusion — so that doctors could test the reasoning process from beginning to end (3r34040. De Fauw et al., 2018
). In addition, to understand more complex systems of artificial intelligence, we could even use automated methods for constructing behavioral models using 3r-3265. machine theory of mind (3r3-3257. Rabinowitz et al., 2018 3r3-33427.). 3r33333.  
3r33333.  

3r3r434. ! function (e) {function t (t, n) {if (! (n in e)) {for (var r, a = e.document, i = a.scripts, o = i.length; o-- ;) if (-1! == i[o].src.indexOf (t)) {r = i[o]; break} if (! r) {r = a.createElement ("script"), r.type = "text /jаvascript", r.async =! ? r.defer =! ? r.src = t, r.charset = "UTF-8"; var d = function () {var e = a.getElementsByTagName ("script")[0]; e.parentNode.insertBefore (r, e)}; "[object Opera]" == e.opera? a.addEventListener? a.addEventListener ("DOMContentLoaded", d,! 1): e.attachEvent ("onload", d ): d ()}}} t ("//mediator.mail.ru/script/2820404/"""_mediator") () (); 3r33434.