Based on Markov chain context, we say a stochastic process is Markovian if the state at time $t+1$, $S_{t+1}$ just depends on the state in the previous step, that is, $\Pr\left( S_{t+1}|S_1, S_2, \cdots, S_t \right)= \Pr\left( S_{t+1}|S_t \right)$.

Now consider a dynamic program where we need to store all historical data up to time $t$ ($\mathcal{H}_t$) as our State variable, that is, $S_t=\mathcal{H_t}$. Now, suppose that at time $t+1$ a new data like $(t+1, a_{t+1})$ is observed and the historical data is updated as $\mathcal{H}_{t+1}=\mathcal{H}_t \cup \{(t+1, a_{t+1}) \}$ and so $S_{t+1}= \mathcal{H}_{t+1}$.

In this case, we could write the state of the problem based on the state of the previous period but we still need to store all historical data. I am wondering if this problem can be considered as a Markovian model?

In the former case, suppose we extend our problem to a continuous-time case. How is it possible to write the state transition formula? Actually, new data are added based on a new Poisson process. So, the time between two new pieces of data follows an exponential distribution.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.