Most formulations of Markov Decision Processes for stochastic inventory models I've come across assume a fixed demand distribution.

But in my case I have a time series forecast with a non stationary distribution and seasonal behavior. Is it still possible to formulate the problem as an MDP?

  • 1
    $\begingroup$ If you have non-stationary demand but use a finite-horizon MDP, then it might make sense. I struggle to envision how an infinite-horizon MDP would make sense since there is no "steady-state." $\endgroup$ Nov 5, 2019 at 13:43

1 Answer 1


You can still use the MDP modeling approach for your system with unknown or non-stationary request distributions.

I think the paper: MDP formulation and solution algorithms for inventory management with multiple suppliers and supply and demand uncertainty, is a good start point to deepen your search. Also, look at POMDPs(Partially Observable MDPs).

In the literature, there are also some applications of learning techniques and MDPs in parallel to manage the uncertainty in the system. For example:

  • Imani, Mahdi, Seyede Fatemeh Ghoreishi, and Ulisses M. Braga-Neto. "Bayesian control of large MDPs with unknown dynamics in data-poor environments." Advances in neural information processing systems. 2018.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.