Consider the following optimization/control problem: We aim to maximize the cumulative reward $R$ during the horizon $H$ by every day allocating a portion of total budget $B$ to our two different investment options $inv1$ and $inv2$ and the same day seeing the response/reward for that day.

Lets say we also have an maximum we can spend everyday.


$$\begin{align} \max_{inv1t, inv2t} \sum_{t=t}^{t=t+H} r(inv1t, inv2t, & independent \; variables) \\ \mbox{s.t.} \sum_{t=t}^{t=t+H} inv1t + inv2t & = B \\ inv1t + inv2t & <= maxdaily \; \forall_t \end{align}$$

Note that i did not explicitly write down an reward function since this is the key part of the question, we will consider different linear models later on.

To our hand we have daily historical data that spans for lets say 3 months back.

Algorithm to consider:

  1. Fit our reward function to the historical data by performing bayes rule.
  2. Sample a set of parameter values from the posterior.
  3. Use this sampled set as ground truth.
  4. Maximize the reward function over the whole horizon s.t to the budget constraints.
  5. Set the allocation for the current timestep.
  6. See the reward for the current timestep.
  7. Add the allocation and reward to our dataset.
  8. Repeat step 1.

In this case however the signal-to-noise ratio might be really low, so we are really keen to get good reliable estimates of the coefficients of the linear regression model. Since the environment is non-stationary we are also keen to keep exploring even though we are quite sure about our coefficients.

Lets consider a reward function:

$$r(inv1t, inv2t) = c1 * inv1t + c2 * inv2t$$

Where c1 and c2 follows some distribution.

The one will place as much budget as possible to the channel that had the highest coefficient in the sampled set of coefficients, leaving the other one with zero.

I assume that this introduces a large amount of multicollinearity.

How will the model be able to determine if the potential decrease/increase in reward came from lowering the budget for one of the channels or increase it for the other?

Lets say i could include date variables as well such as dateinmonth, day etc in the model could that help fixing this problem?

Consider e.g. the reward function:

$$r(inv1, inv2, dayofmonth) = c1 * inv1t * dayofmonth + c2 * inv2t * dayofmonth$$

where c1 and c2 follows some distribution.

Now we may not distribute the same amount of budget each day, but we introduce structural multicollinearity since we are including dayofmonth on two parts...

How is this problem typically treated, I have not found any literature on this particular subject. Also the method of sampling the coefficients and using it as ground truth seems suboptimal for guiding the exploration. Maybe it would be better to tackle it with a stochastic optimization approach generating scenario trees, but how would that be done when we have a somewhat non-time dependent process(even though including date variables)?

  • $\begingroup$ It's helpful to point out that there is a similar question here: ai.stackexchange.com/q/36475/17742 $\endgroup$
    – Rob
    Jul 27, 2022 at 22:31
  • 1
    $\begingroup$ it was mine, now removed. Lets try in this forum with some added info. $\endgroup$ Jul 27, 2022 at 23:08


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.