Inquiry: vertex corresponds to deterministic policy and representation of reward

[gunnxx]

Hello Benjamin Eysenbach,

I have two questions about the paper. I would really appreciate it if you have the time to explain it.

1. In section 5.1, it is stated that, "In general, we can obtain the set of all feasible state marginals by taking the convex hull of the state marginals of all deterministic policies." It means that each vertex of the polytope corresponds to a deterministic policy (but not all deterministic policies are vertices of the polytope). Why the vertex is a deterministic policy? I can't see any reasoning behind this.
2. About the reward function visualization, it is a vector starting from the origin pointing towards what and what determines the magnitude of this vector?

[ben-eysenbach]

Thanks for the questions!

1. Here's one proof: Let a vertex be given. We can construct a reward function for which that vertex corresponds to a reward maximizing policy. For every MDP, there exists a deterministic policy that achieves maximum returns. So, this vertex must contain a deterministic policy.
2. The vector can be written as $(r(s_0), r(s_1), r(s_2), \cdots)$.