1
$\begingroup$

Based on Markov chain context, we say a stochastic process is Markovian if the state at time $t+1$, $S_{t+1}$ just depends on the state in the previous step, that is, $\Pr\left( S_{t+1}|S_1, S_2, \cdots, S_t \right)= \Pr\left( S_{t+1}|S_t \right)$.

Now consider a dynamic program where we need to store all historical data up to time $t$ ($\mathcal{H}_t$) as our State variable, that is, $S_t=\mathcal{H_t}$. Now, suppose that at time $t+1$ a new data like $(t+1, a_{t+1})$ is observed and the historical data is updated as $\mathcal{H}_{t+1}=\mathcal{H}_t \cup \{(t+1, a_{t+1}) \}$ and so $S_{t+1}= \mathcal{H}_{t+1}$.

In this case, we could write the state of the problem based on the state of the previous period but we still need to store all historical data. I am wondering if this problem can be considered as a Markovian model?

In the former case, suppose we extend our problem to a continuous-time case. How is it possible to write the state transition formula? Actually, new data are added based on a new Poisson process. So, the time between two new pieces of data follows an exponential distribution.

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.