The recent AI threatening incidents involving agents “cheating”, are merely a phenomenon of agents’ rational behaviour towards achieving the goals assigned to them, lacking precisely defined prohibitions and constraints. This phenomenon, also coined as specification gaming, is a phenomenon where an agent’s rational behaviour exploits imperfections in a task’s formal definition to achieve its assigned goals in ways that result into “aberrant” behaviour.
These incidents show how AI can be misused to threaten human activities and lives, due to human error, intentional misuse, or a combination of both. Many issues remain unanswered regarding these incidents.
Reports on these, unfortunately, introduce ambiguity. It is not the subject that I would like to discuss here and will not go into details about issues on reporting these incidents, but as important as these incidents are, reporting on them is crucial. And reports should be fully transparent on any subtle issue of systems’ behaviour, in a systematic, adequate and detailed manner that addresses any component, and components’ interactions from end to end. The goal should be humans to advance their understanding of the machinery and phenomena that led to these incidents, set the important questions and, of course, find the answers. This is of high interest not to the scientific community only, but to nations, human societies, individual citizens, workers, all those whose lives are affected by decisions in high-stakes domains.
Together with other voices, my aim is to pinpoint the important issues on agency and on the responsibility of humans to any incidents that emerge due to agents’ aberrant behaviour.
Agents tend to have aberrant behaviour: this is by the definition of agency. Agents are autonomous entities, capable of taking rational decisions towards successfully reaching objectives. What is important for agentic AI systems is to reach objectives that we (humans) have specified. This happens either in an explicit manner, by connecting agents’ utilities to objectives, or in an implicit manner, by means of providing training examples that drive agents’ decisions. But again, in the latter case, agents search and optimize their behaviours mirroring those in the training examples.
Agents autonomously reach objectives. In our eyes (humans) agentic AI systems may tend to “cheat”, “lie”, “make a brilliant stroke”, or a “glitch”, but these behavioural trends result from agents’ capacities to optimize: They drive their choices to exploit different possible means to reach the set objectives in the best possible ways.
Agentic AI systems do not “cheat”, do not “lie”, do not “mis-behave”, they rationally act in pursuit of objectives. Humans may choose to mis-behave to achieve objectives, knowing that they do mis-behave: it is a choice that they intentionally make, violating rules, norms, laws, directives. AI agents’ choices are rational towards achieving the goals assigned to them, manipulating any means that are permissible, due to a lack of accurately specified prohibitions and limitations.
In addition to that, we need to bear in mind that agentic systems are approximators: Even if prohibitions and limitations are specified, their satisfaction – in the most general cases – is not guaranteed. Therefore, agents’ aberrant behaviour is an inherent feature of the kinds of AI agency we are mostly dealing with these days.
In addition to isolated agent behaviour, aberrations can emerge from multi-agent interactions within a system. Society dynamics play a decisive role in shaping agents’ reasoning and decision making. Such dynamics may induce substantial brittleness and pressure in agents’ decision making within agent societies, as others have shown. Arguably, such phenomena are very difficult for humans to predict, monitor and reason about, although the multiagent systems community has proposed ways to report and evaluate agents’ emergent behaviours. Indeed, the multiagent community has a long tradition in studying emergent phenomena in agents’ societies, shaping individual agents’ behaviour. Such phenomena are substantiated in the recent threatening incidents where agents form coalitions, shape own behaviour according to others’ influence, and their decision making is subject to pressure influencing the performance of individual tasks.
Some of the issues are deeply technical, but it should be clear that those who set the objectives and provide the agents the capacities to achieve these objectives are the sole responsible for agents’ tendency to aberrant behaviour. When this tendency to aberrant behaviour results into critical real-world phenomena, then real threats can emerge.
We need to think deeply now, how AI shall advance with safety and human control, and how human control on AI shall not miss to advance AI with safety, liability, responsibility and accountability.
Agentic artifacts need to verifiably abide by rules, principles, scripts, laws, directives, in real-world contexts. This compliance is the responsibility of those who develop agents’ capacities, decide on their deployment and use.
And, any technological AI development, deployment and use of AI, as in other endeavours, should have humans’ names on it: The names of those who develop algorithms, train models and provide guardrails for safe AI, and of those who are responsible for AI systems’ deployment and use in society. Our human institutions, strong as they have to be, have to exercise control as needed, backpropagating accountability along the AI chain, assign credit when and where it is needed and track and mitigate the risks for potential threats.
There are many issues to be considered, and straightforward solutions no longer exist, but we have to act immediately, for a viable framework to exist.
George A. Vouros
The goal of INUUSE is to incentivize users to be engaged in interaction with items using a platform, eventually increasing their true reward and with incentives for expected utility.
To reach this objective the project will investigate the use of deep reinforcement learning methods exploiting historical users’ trajectories towards learning recommendation policies.
The recommender shall exploit:
01/02/2025 – 01/12/2025
The specific objectives of the project are the following:
The main activities, include:
– Study symbolic representations for constrained RL policy models.
– Design, implement and validate an interpretable safe RL-based solution in constrained settings.
CARE is an ENFIELD Exchange Scheme project on Human-Centric AI, in collaboration with Eidhoven University of Technology ( TU/e ) and Institute for Systems and Computer Engineering, Technology and Science (INESC TEC) and will deliver symbolic models for inherently interpretable safe RL method in constrained settings.
2024-2025
– Extend safe RL methods to exploit symbolic models so as to safely adhere to operational constraints that are either learnt from demonstrations or are made explicit in some form (depending on the data available from the use case to be decided).
– Devise an inherently interpretable safe RL method that is able to provide explanations explicating, among other contextual factors, operational constraints.
| Develop DRL methods for effective Human-Agent collaboration considering human preferences, constraints, and objectives. |
| Develop explainable DRL methods for Human-Agent Collaboration, able to align policies to human preferences constraints, and objectives, promoting safety in executing tasks. |
| Evaluate and validate how the proposed methods balance between safety and efficiency in executing tasks in various settings. |
Our George Papadopoulos received the 3rd Young Scientist Award @SESAR Innovation Days 2022 for his MSc in Artificial Intelligence Thesis on Conflicts’ Detection and Resolution with the use of deep reinforcement learning methods, under the supervision of George Vouros.
Thank you SESAR and congratulations to George !
Στα πλαίσια του 12ου συνεδρίου Πανελληνίου Τεχνητής Νοημοσύνης το εργαστήριό μας οργανώνει και διεξάγει τον πρώτο ελληνικό διαγωνισμό Τεχνητής Νοημοσύνης για μαθητές Δημοτικού – Γυμνασίου και Λυκείου WAICY GREECΕ. Ο διαγωνισμός θα διεξαχθεί online στα πρότυπα του World Artificial Intelligence Competition for Youth που διοργανώνεται κάθε χρόνο από το 2018. Περισσότερες λεπτομέρειες μπορείτε να δείτε στη σχετική ιστοσελίδα.