Questions tagged [markov-decision-process]
For questions related to Markov decision processes (MDPs), which model decision making in time-varying and usually stochastic environments.
14
questions
1
vote
0
answers
23
views
How to check if the state of a dynamic program is Markovian or not?
Based on Markov chain context, we say a stochastic process is Markovian if the state at time $t+1$, $S_{t+1}$ just depends on the state in the previous step, that is, $\Pr\left( S_{t+1}|S_1, S_2, \...
- 2,105
2
votes
1
answer
136
views
Understanding MDP's Dual Linear Program
I'm trying to understand a proof in Puterman'05 (Markov Decision Processes: Discrete Stochastic Systems). My question is within Theorem 6.9.1 pertaining to the equivalence of solutions to the primal (...
- 23
2
votes
0
answers
29
views
MDP model for a unique system
Let's assume that we have a system where there are two exogenous factors, name $a$ and $b$, that can affect the system state. At each decision epoch either $a$ or $b$ occurs and it is not possible ...
- 21
1
vote
0
answers
98
views
Markov Chain - Stationary or Borderline distribution [closed]
I have a Markov chain problem below, where :
The problem
An urn initially contains 3 black balls and 1 red ball. The balls are indistinguishable to the touch. One ball is randomly drawn.
If this ball ...
- 61
4
votes
0
answers
44
views
Equilibria in two player, finite horizon, zero sum stochastic game
Suppose we have a two player, finite horizon, zero sum stochastic game (a.k.a. "competitive MDP"). The game is defined over a finite set of state $S$. At each decision epoch $t\in\{1,\dots,T\}$ (for ...
- 1,957
3
votes
1
answer
90
views
Queuing Theoretic Model with Memory
Consider a telephone company which receives call request at some arrival rate and serves each request with some service rate. This can be modeled using a Poisson Process. However I wish to model the ...
- 871
8
votes
1
answer
126
views
Can I use an MDP for a stochastic inventory model when my demand distribution is non-stationary?
Most formulations of Markov Decision Processes for stochastic inventory models I've come across assume a fixed demand distribution.
But in my case I have a time series forecast with a non stationary ...
- 2,009
12
votes
3
answers
1k
views
Number of aircraft to operate in an airline company
Suppose that an airline company has X planes, in general the companies keep a number of these aircraft (say 80% of X) in reserve in case of hazards. My question is how the airline companies compute ...
- 229
12
votes
1
answer
111
views
Model or State Uncertainty in Queueing Model due to uncertain arrival rate
Introduction
I am currently modelling a scenario where two queues need to be served by a single server in a non preemptive discipline. I am quite sorted on generating the optimal policy via Value or ...
- 443
10
votes
1
answer
246
views
Algorithm for workforce scheduling for call volumes
I am trying to solve a workforce scheduling and optimization problem.
Available data:
daily level forecasted call volumes,
shift schedules,
resource utilization at the aggregate level,
AHT at ...
10
votes
2
answers
3k
views
Infinite horizon versus finite horizon MDP
When can we approximate a finite horizon MDP with infinite horizon?
Can we use infinite horizon stochastic shortest path problem on a directed acyclic graph?
- 101
9
votes
0
answers
193
views
For the solution of complex gaming are we to leave behind approximation and fixed algorithms in favor of agents?
For an evolutionary extensive form game is the use of multi-agent systems a necessity or can more traditional approximate Nash equilibria and mean-field interactions suffice?
Few games can be ...
- 2,042
12
votes
1
answer
132
views
Open source Markov decision process solvers
Which sources are out there that provide reliable and fast MDP solvers? I prefer that the library is callable from C++ but other languages are fine too.
There is indeed an abundance of repositories ...
- 2,201
8
votes
1
answer
281
views
Convergence rate value iteration
Value iteration is one of the most commonly used methods to solve Markov decision processes. Its convergence rate obviously depends on the number of states and actions. However, the convergence rate ...
- 2,201