General objectives for RL

 

We can describe the standard RL problem as a linear function, the inner product between the state-action occupancy and the reward vector. The occupancy is a distribution over the states and actions that an agent visits when following a policy and the reward defines a priority over these state-action pairs. 

 

Sometimes, a good objective function is all we need; predicting the next word in a sentence turned out to be transformative. I've been interested in deriving more general objectives of behaviour: non linear, unsupervised, convex and non convex objectives that control the distribution that an agent visits. These include:
 

• Maximum entropy exploration — visit all the states equally

• Apprenticeship Learning — visit similar states to another policy

• Diversity — visit states other policies do not

• Constraints 

• And more

 

My main result is that we can reformulate these problems as convex-concave zero sum games, and derive a non stationary intrinsic reward for solving them. The reward turns out to be very simple and general; it is the gradient of the objective w.r.t to the state occupancy.

 

My papers below study the following questions:

• In what sense are general utility RL problems different from RL problems?

• Can we solve them with similar techniques?

• Are there interesting objectives we can now easily solve using this approach?

• In particular, I worked on Quality-Diversity objectives