I am currently working on a resource allocation problem and I am uncertain about which field of stochastic optimization and reinforcement learning encompasses this particular problem.
The objective is to allocate a limited resource (budget) among different investment options in a way that maximizes the accumulated revenue over the optimization horizon. Each day, a decision is made regarding the allocation of resources to an investment channel (which remains constant throughout the horizon), and prior to making the decision for the next day, the reward (sales) obtained from the previous day is revealed.
To assist in this process, we have a Bayesian parametric model that approximates the rewards received on specific dates throughout the horizon. The model is nonlinear and time-dependent, although it may not represent the true underlying model, serving as an approximation based on domain expertise. The environment being modeled exhibits a low signal-to-noise ratio.
Formally, the problem can be described as follows:
Investment channels: $I = (i, j)$
Horizon: $H$
Total resource: $B$
Decision variables: $x_{i, t}$, where $i$ represents the investment channel and $t$ represents the timestep
Bayesian parametric model (reward approximator): $F(x_{x_{i, 1}, x_{j, 1}, x_{i, 2}, x_{j, 2}....x_{i, H}, x_{j, H}})$
The optimization objective can be stated as: $$\max_{x_{i, 1}, x_{j, 1}, x_{i, 2}, x_{j, 2}....x_{i, H}, x_{j, H}}\sum_{t \in H} F(x_{x_{i, 1}, x_{j, 1}, x_{i, 2}, x_{j, 2}....x_{i, H}, x_{j, H}}) \tag{1}$$
Subject to: $$\sum_{t \in H}\sum_{k \in (i, j)} x_{k, t} \leq B \tag{2}$$
Now, let's consider two different schemes to approach this problem:
Scheme 1:
1. Collect daily data on historic investments and their corresponding rewards.
2. Obtain posterior estimates of the model using MCMC sampling with the collected data.
3. Sample a single trajectory from the posterior, projecting the parameter estimates over the horizon
4. Utilize this single trajectory as the true parameter values over the horizon and optimize the allocation strategy.
5. Execute the investment.
6. Receive the reward for the current day.
7. Repeat from step 1 the next day.
Scheme 2:
1. Same as scheme 1.
2. Same as scheme 1.
3. Sample multiple (possibly a large number of) trajectories from the posterior, projecting the parameter estimates over the horizon.
4. Use these samples as potential trajectories over the horizon and optimize the allocation strategy by considering the mean of the different reward trajectories.
5. Same as scheme 1.
6. Same as scheme 1.
7. Same as scheme 1.
I am interested in understanding if scheme 1 is an example of Thompson sampling, as it appears quite similar. Additionally, does scheme 2 fall under the category of stochastic optimization, as it seems quite comparable? I am seeking literature that addresses this type of problem, which seems to be encountered in various scenarios, but I am struggling to find relevant literature. Most of the problems I come across are substantially different. Utilizing a Bayesian model to model uncertainty and sampling trajectories seems like a straightforward approach for approximating an environment in these optimization problems while accounting for uncertainty. However, I need to find relevant literature to gain a deeper understanding. I also have issues understanding what would be the policy in this case.
Furthermore, I would like to explore how these two approaches differ in terms of exploitation versus exploration. Intuitively, sampling a single trajectory as in scheme 1 suggests a greater emphasis on exploration compared to scheme 2, although one could argue the opposite as well. I am interested in reading more about this topic and exploring alternative approaches to address this challenge.
