Solving: why is this solution optimal?
As Richard explained, the objective in OR is not "fuzzy" like in ML: we assume an objective that can be evaluated by the computer. Once the problem is specified, there is not much to explain: you can prove optimality or infeasibility directly.
Many solution methods attempt to prove optimality, and it is possible to get an optimality (or infeasibility) certificate from the solution process. This is trivial for LPs, and well known for SAT and MILP problem (see this paper for more information). When using heuristics you are out of luck, but can still compare the results to bounds obtained with other methods.
Modeling: what is a good solution?
But you asked how to explain the solutions to stakeholders.
Before you solve the problem, you have to model it in a way that matches the stakeholders' expectation of a good solution. The difficulty here is to specify what a good solution is. The stakeholders have domain knowledge that you don't, and many conflicting objectives.
The goal is to specify the model with them so that they can understand the solutions: this is a matter of engineering and communication.
I suggest the answer to this question about OR industry projects for a better view of the issues at hand.
The stakeholders may ask the OR team:
We want to minimize costs and downtime at our factory
This is not explainable: how do you compare a solution with 1 hour downtime and 100k cost, and one with 10 hours downtime and 20k cost? So the OR team will lead them to more explainable specifications.
For example, by ordering the objectives:
We want less than one hour of downtime if possible, and otherwise want to minimize the cost
Or by providing a conversion factor:
We can allow some downtime, if it saves at least 20k per hour
In both cases, it is easy to see which solution is better.
In practice, the stakeholders will have an idea of what a good solution looks like, but it will not always be easy to explain: they know it when they see it!
The task of the OR practitioners is to turn this domain knowledge into a model. A good way is to look at solutions with them, and understanding how they rank them. From there, it is generally possible to come up with an ordering of the objectives, or conversion factors, that they can agree with.
Just like in ML, it builds trust in the model when you can explain it, and is crucial for tool adoption. Compared to ML, the explanation stems from the specification, and is generally easy to understand for everyone involved.