This job is part of an Advanced Research + Invention Agency-funded project.
Dovetail is an agent foundations research group. We’ve recently received an ARIA grant to fund more team members. This application is for a 10-week fellowship in graduate-level mathematics, with the possibility of extension (funding permitting).1 Due to funding conditions, applicants must be based in the UK. You must be resident and eligible to work in the UK. This application will be open until the 17th May, but we will be interviewing and accepting on a rolling basis so we strongly encourage you to apply earlier.
You can read more about agent foundations and our research agenda on our website.
Alfred and Alex will be leading a group of roughly 4 to 6 other people all engaged in mathematical AI safety research. We’re open to experience levels from undergraduates who are enthusiastic about maths to post-docs who are looking for a way to transition into AI safety.2 (We also welcome applicants outside academia!)
Some group members might work together, while others might do solo projects. All group members will have regular one-on-one check-ins with us, and we’ll also hold regular group meetings. We’ll spend our time finding useful formal definitions of relevant concepts, formulating and proving theorems about them, and communicating our ideas and results. We’ll find and share relevant papers, host read-through meetings, and discuss ideas with external researchers.
Here are some of the basic parameters of the fellowship.
As with a lot of research in agent foundations, it’s quite difficult to concisely communicate what exactly we work on. Probably the best way to tell if you will be interested in our research problems is to read our research wiki or our write-ups on LessWrong, or the example problems below and then have a conversation with us about it. A running theme in our research is the nature of `world models’ in agentic systems. All our research is purely mathematical,3 rather than experimental or empirical. None of it involves machine learning per se, but the resulting theorems should apply to ML systems.
The domains of mathematics that we’ve been focusing on include: dynamical systems of all kinds, probability theory, information theory, algorithmic information theory, measure theory, ergodic theory. Things we’re interested in but less knowledgeable about include: singular learning theory, computational mechanics, abstract algebra, reinforcement learning theory.
Here are some short write-ups of projects you could work on. Click on the problem titles to be taken to a pdf of a 1-page problem write up.
These are just examples of problems which have a reasonably concise and well-defined mathematical framing. Some of our projects are less well-defined and a big part of the work is developing them to become mathematically formal. If there is something else you are excited to work on, we are also open to hearing other project proposals, provided that they are within the scope of our research program.
If you’re interested, fill out this application form! You’re also welcome to email us with any questions. If you are unsure whether to submit an application, we encourage you to err on the side of applying. After that, the rest of the application steps are;
After we have read your proposal, we should have a pretty good sense of whether we would work well together, and we’ll make a decision about whether to offer you the 10-week fellowship (or whatever else we may have negotiated). If you are successful, we expect that the time between the first interview and the job starting will be two weeks.
We’re flexible with some specifics, if you have existing obligations on your time. ↩
To give some more detail on these bounds; we expect applicants to be ready to start research at the level of graduate mathematics. We’ve met many undergraduates at this level, but they are invariably enthusiastic about mathematics and have already developed their skill level beyond what’s learned in a STEM degree. That said, there’s no essential knowledge beyond that. No one is an expert in agent foundations, so we’re really seeking what is sometimes called “mathematical maturity”. On the other side, if you have far more mathematical expertise than either of us and still want to join, we’re happy to discuss the details of your situation to see if it makes sense for both parties. ↩
More specifically, the desired results are mathematical. Many of the ideas are “pre-mathematical”, in that the first part will be to translate the ideas into the appropriate formalisms. ↩