Update:
Since you updated your question might as well chip in, since I've worked with COIN-OR software a lot at the code level. In my experience, a lot of the open-source optimisation codebases (e.g. CBC) succeed in being amazing solvers, but fail in being great open-source projects. The heart of any successful open-source project (and by "successful" I mean one where many people decide to contribute) is the readability and design of its codebase, not the performance of the software per se. This is because readability and good design empower contributors to believe that they can improve things a lot, if they try long enough. What you want in an open-source project is a design where people can jump right in and start doing little things. If they need to spend months/years understanding what's happening before they can get their hands dirty, they will simply pass.
From a solver developer's point of view, the foremost reason is because a lot of these projects use autotools instead of CMake as a build system. Although it might not seem like much, it is unfortunately a deal breaker. The reason is quite simple. If we take CBC as an example, it is basically one massive file of code with lots of dependencies. This is very hard to read, let alone improve, even for a person like myself who does this for a living. What I could do, if the project were built with CMake, would be to slowly refactor the code, split it into many files and classes and improve encapsulation, so that it becomes much easier to read, understand, maintain, and improve over time, not just for myself but for other people. Of course someone could argue that I could do the same thing with autotools, but the simple truth of the matter is that people just don't use autotools anymore. I would need to spend months doing that full-time, which is time I simply don't have. It would be trivial however to make tiny changes over time if the project would build with CMake.
Therefore, this would be my recommendation: migrate the projects to CMake so that many more developers are able to contribute to the open-source code bases.
This is not pure conjecture btw. Even though CBC/CLP have been around for 20 years, HIGHS already has many more contributors despite being much younger. From a developer's point of view this makes perfect sense. The code is much more readable, and the project is built with CMake. This makes a massive difference. As another fun fact, building & linking CLP/CBC to our solver took a few weeks back in the day, especially since we also had to do it for Windows. Doing the same thing for HIGHS took 2 days - 1 days to link & build, and 1 day to test.
Original answer:
As an MINLP solver developer I'd like to share my experience on just how we make our solvers faster and why that's hard to replicate in an open-source environment.
Our day-to-day work revolves around meticulously running problems over and over again, analysing just what is happening, and thinking about how we can improve things:
- We constantly find and eliminate code-level bottlenecks.
- We reason how to skip calculations that are not necessary. This is typically math-specific, i.e., if my problem has some particular structure, do I really have to execute this piece of code?
- Once we've ran out of code-level improvements, we go back to the math and try to identify and exploit special structure. Elucidating special structure often requires spending significant time to code new features into the solver.
- Sometimes we find that our solver's architecture is limiting us, i.e., we actually can't code what we want at high performance because of certain design choices in the code. If we spot that enough times, we'll take the time (potentially many months) to rearchitect the entire code-base.
- We'll meticulously run benchmarks on thousands of problems and tweak tolerances, parameters, working limits, and all sorts of little things to squeeze out more performance for a handful of problems, without regressing in performance for the remaining 10,000 ones. Doing this over many years stacks, and performance is improved by many times.
- When nothing else works, we'll spend the time to figure out new algorithms, often for only a handful of problems.
- When we break certain performance milestones, e.g., when our solver is now 100 times faster than it was 2 years ago, we go back and re-examine whether the new performance opens up new algorithmic options that were impossible a few years ago. At this point, we rinse and repeat the entire process.
This is what it takes to make a solver performant. Patience, knowledge, and meticulous work that is often unrewarding for long periods of time, until we get it to work.
There is a stark difference here between what we need to do and what ML developers need to do. In ML code, it's much more about an overall architecture that provides high throughput, and good APIs that provide users expressive freedom to create their algorithms. It is much less about manually running things and correlating performance to specific user input. This is why new ML frameworks keep popping up: once an ML framework is architected, the only way to really improve it is to design a new one. At its core, this is a difference in culture. ML users are often comfortable tweaking/designing their own algorithms, whereas solver users expect it to just work out of the box.
In light of the above, you might see how it's difficult for this process to happen in an open-source setting. Improving performance is not often not about doing something new and exciting. It's about taking a very long time to run the same thing over and over again until we figure out small tweaks that make that one problem 100 times faster without regressing in performance anywhere else. Even though I personally find that very satisfying, most people find it boring because they're not doing something "new". More importantly, in an academic setting, the bulk of this work isn't really publication-worthy - people can't really write a paper about how they spent 100 hours to figure out that changing an internal tolerance from 1.e-5 to 3.45e-5 resulted in solving 5 extra problems in the test set. In case you are wondering why this is the case, unfortunately these little things are nearly always implementation specific, i.e., they work in our own solver because of how the different systems chaotically interact with each other, but they are not transferable to any other solver.
Fun fact: just last week I figured out how to solve a problem I've been trying to solve for the last 2 years. During that time, I was working on that problem for a few minutes nearly every day, making tiny improvements once in a while. These improvements stacked, to the point where the very last improvement reduced the solving time from about 100 hours down to 700s on 8 cores. Even though I will admit to doing a little dance afterwards, I can't for the life of me tell you exactly why we can now solve this, as it's literally hundreds of little things over 2 years, let alone write a paper. This is very different from the ML community - if someone cracks an unsolved ML problem it will break the news the day after, simply because it's much more widely applicable. I do not consider that a bad thing, but you might see how ML would attract many more talented people, since the time/reward ratio is simply intrinsically better. This difference in the number of talented people working on the problem is, at the end of the day, one of the prime reasons for the differences we observe.
