A Markov process is defined as a tuple (S, P), where:

• $S$ is a finite set of states.
• $P$ is a state transition probability matrix, where $P_{ss'} = P[S_{t+1}=s'|S_t = s]$.

A sequence of states possesses the Markov property if and only if the probability of moving to the next state $S_{t+1}$ depends only on the present state $S_t$ and not on the previous states $S_1, S_2, \dots, S_{t-1}$. That is, for all $t$, P[S_{t+1}|S_t] = P[S_{t+1}|S_1, S_2, dots, S_t].

In reinforcement learning, a Markov process is typically time-homogeneous. This means the transition probability is independent of the time step $t$: $P[S_{t+1} = s'|S_t = s] = P[S_t = s'|S_{t-1} = s]$.