Any AI system will have some degree of world model. These world models will be made of abstractions. Abstractions have something to do with how we think about and make sense of the world by “abstracting away” inessential information from essential. But how exactly do they work? How are they formed? Humans tend to share many abstractions like “cat” and “pain”, but seem to have conflicting abstractions in some domains, like whether a given color is blue or green, or how “freedom” applies to a given scenario. To what degree do we have an arbitrary choice over the abstractions we use? When are our differing abstractions an error on someone’s part, or due to us having different goals?
Understanding what abstractions are and how they work is going to be a core part of safely navigating the creation of powerful AI systems, for two main reasons:
If abstractions tend to be more “objective” in the sense of being determined by the environment, then that is evidence that these tasks will be easier. If we manage to build an ML system whose world model has a sufficiently accurate representation of human values, then we can build a relatively simpler optimization architecture around it which “points” to that representation of human values as its optimization target. This is far from a sufficient plan for building safe superintelligences, but we think doing something like this could be a key stage in a more iterative and deliberative program of alignment.
An algorithmic theory of abstractions is one of our directions of research.
One agenda aiming to understand abstraction for this purpose is John Wentworth’s “natural abstractions”. For those unfamiliar, the post Natural Abstractions: Key claims, Theorems, and Critiques provides a thorough overview of John’s agenda and how it relates to broader ideas.
