As with optimization, one of our research problems is to find a formal criterion which can say whether a given system has an agent in it, and if so, where that agent is located, and other measurements of its properties.
Such a criterion could, at least in theory, tell us whether a given AI system is an agent (or contains an agent). This would be useful, because agents are dangerous. Since agents have world models and values, detecting an agent would likely involve detecting where its world model and values are represented. This could be helpful for interpretability and retargeting the search.
Since successful agents are optimizers, one way to get started on detecting agents is to detect optimization.
The agent structure problem is an idea about one way in which we might be able to detect agents.
