Disclaimer: I am currently working for a commercial solver company (Gurobi) and have worked before on another commercial solver (IBM CPLEX). Hence, my opinion may be biased, but still I am trying to not turn my answer into a marketing and sales pitch. For my PhD thesis I developed the academic solver SCIP, which is still actively maintained and developed by a large group of researchers, so I also know the academic side of the solver world. In any case, what I am going to say is my personal opinion and not necessarily aligned with my current or former employers or my former research group.
I agree with dhasson's points, but I would like to emphasize a certain aspect of performance (also addressed in Kuifje's answer) that I think is hard to understand in its full consequences for people that are not experts in operations research.
First, let me state that I am only discussing mixed integer programming (MIP) here, which is just a small sub-area in the field of operations research, but I think is the main topic that Skander H.'s question is about.
The main issue of performance for mixed integer programming solvers is that all known algorithms for MIP have exponential worst-case running time. While this may also be true for other areas like ML or data base operations, I think that in practice the algorithms that are applied there scale reasonably well with the size of the input data.
For MIP, the running time heavily depends on the structure of the model to be solved and on the capabilities of the solver to exploit this structure. If there is no useful structure in the model or if the solver is not able to identify and exploit the structure, then you will often see that the theoretical exponential growth in running time manifests itself in practice.
For example, there are models with just 50 binary variables and a handful of constraints that are pretty much unsolvable with today's state-of-the-art algorithms. And for such models each additional binary variable pretty much doubles the resulting runtime. You really can see the exponential complexity. On the other hand, there are models with millions of variables and constraints that can be routinely solved to optimality. This is true for both free/academic solvers as well as commercial solvers.
Fortunately, models of practical problems that are of interest in industrial applications are most often of the latter type. Nevertheless, as said before, being able to solve a model or not within reasonable time heavily depends on whether the solver at hand is able to identify and exploit the structure of the model. And this is where the main difference is between free/academic solvers and commercial solvers when it comes to performance. Commercial vendors with their teams of full-time developers and their large customer base who provide models from a diverse set of applications are just in a much better position to develop, implement, and tune algorithms to cover all these different aspects and structures that appear in real-world models. Hence, the likelihood that the solver deals nicely with your application type is larger if you select a commercial solver.
And this aspect does not show very well in the performance comparisons that you see in publicly available benchmarks. From the pictures like the one that Kuifje refered to, you may get the conclusion that free solvers are something like 5 or 10 or 20 times slower than the best commercial solvers. And then, you may think "speed is not that relevant for me, I don't mind having to wait 10 minutes instead of just 1 minute". But this is just not the right way of looking at this. Check also the number of models that could be solved to optimality: in the (relatively old) picture you can see that CBC (which is a good free solver) is able to solve only 48 out of the 87 problem instances, while the commercial ones solve 86 of them.
For your particular problem class, it could be that a free solver just behaves nicely and has the same or at least comparable performance than a commercial solver. It could even be that there is some research advancement implemented in the free/academic solver that has not yet found its way into the commercial product and thus the free solver is actually faster than the commercial product. But there are also a number of cases where the models for your problem class are intractable for one solver while they can be solved in a fraction of a second by a different solver, often a commercial one.
This is what I think is often hard to grasp. From, say, ML algorithms or data bases or operating systems or whatever type of software where you have both commercial and free (open source) alternatives, one is used to commercial products being maybe a little faster or a little better or just a little different in certain aspects. But there usually is not the difference between "it is impossible to solve my problem" versus "my problem solves very easily". But this is sometimes the case for MIP solvers.
Thus, I would recommend the following approach:
First, you consider all the various other aspects that have been mentioned above by dhasson. Is a reliable support, SLA, cloud computing, a certain API or platform support essential to you? What about the other non-standard features that some commercial solvers offer? Is there a free solver that provides what you need? If not, you are already bound to looking at commercial alternatives.
If a free solver is a valid option, then pick one based on publicly available benchmarks and try to solve your models with it. Just check if it works. Verify carefully whether the results are correct, because apart from performance there is also the aspect of numerical robustness that is often better in commercial products. Try multiple problem instances of your problem class. And if everything works well with the free solver, just stick to it for this project.
But if at some point you run into a wall, for example because the free solver is just unable to solve your problems, then please do not make the mistake of dismissing the whole technology of mixed integer programming. It could be that with the solver you tried it appears completely hopeless to solve your problem and you have the impression that you need to look at a completely different approach like implementing a heuristic. But at this point it is definitely time to check out commercial alternatives. The commercial vendors provide evaluation licenses such that you don't need to pay anything for a first try. And it can very well be that you are going to be positively surprised. It happens frequently that a commercial solver can solve certain types of models easily that appear completely hopeless with a free solver. And this is not only due to the differences in free and commercial products, it can also happen when switching from one free solver to another, or when comparing different commercial solvers. For example, it could very well be that Gurobi struggles for some problem class but CPLEX and XPRESS have no issue at all with it. So, please try different alternatives before you give up! MIP is really a cool and useful technology that is worth learning and embracing.
