I think the problem you are describing is known as constrained optimization or constrained minimization, where you want to minimize a function within a certain domain defined by constraints. In your case, the constraints are given implicitly.
There are several approaches to handle constrained optimization problems, and the choice of method depends on the nature of your problem. Here are a few common methods that might be relevant to your situation:
Penalty Function Method: You can convert the constrained optimization problem into an unconstrained problem by adding a penalty term to your objective function. This penalty term penalizes violations of the constraints, and you can adjust the penalty parameter to balance between staying within the feasible domain and minimizing the objective function.
Barrier Function Method: Similar to the penalty method, you can use barrier functions to transform the constrained problem into an unconstrained one. Barrier functions become very large as you approach the boundary of the feasible region, discouraging the optimization algorithm from moving outside of it.
Interior Point Methods: These methods work directly within the feasible region by iteratively moving towards the optimum while respecting the constraints. They are particularly useful for problems with both equality and inequality constraints.
Augmented Lagrangian Method: This approach combines the ideas of penalty functions and Lagrange multipliers to solve constrained optimization problems. It involves iteratively updating Lagrange multipliers and penalty parameters.
Sequential Quadratic Programming (SQP): SQP is an iterative method that approximates the constrained problem by solving a sequence of unconstrained quadratic subproblems, taking into account the constraints at each step.
And finally, the one I think you have in mind...
- Genetic Algorithms or Simulated Annealing: These are heuristic methods that can explore the feasible region, making occasional "jumps" outside and returning while maintaining a balance between exploration and exploitation.
The choice of method depends on the specific characteristics of your problem and the computational resources available. Some problems may benefit from hybrid approaches that combine multiple techniques. You may need to experiment with different methods to find the one that works best for your particular situation.