Partially Observable Markov Decision Processes

Synonyms

Belief state Markov decision processes; Dual control; Dynamic decision networks; POMDPs

Definition

A partially observable Markov decision process (POMDP) refers to a class of sequential decision-making problems under uncertainty. This class includes problems with partially observable states and uncertain action effects. A POMDP is formally defined by a tuple $\langle \mathcal{S},\mathcal{A},\mathcal{O},T,Z,R,b_{0},h,\gamma \rangle$ where $\mathcal{S}$ is the set of states $s,\mathcal{A}$ is the set of actions $a,\mathcal{O}$ is the set of observations o, T(s, a, s^′) = Pr(s ′ | s, a) is the transition function indicating the probability of reaching s^′ when executing a in s, Z(a, s^′, o^′) = Pr(o^′ | a, s^′) is the observation function indicating the probability of observing o^′ in state s^′ after executing a, R(s, $a) \in \mathfrak{R}$ is the reward function indicating the (immediate) expected utility of executing a in s, b₀ = Pr(s₀) is the distribution over the initial...