38
$\begingroup$

In recent years, a huge number of scholars in AI and ML community are using Python to develop their algorithms and then publishing their papers and codes in GitHub. This provides an opportunity for others to work on that paper and develop more efficient algorithms so fast and as we are witnessing, this community is progressing rapidly. However, this is not common in OR/MS community and a student should learn how to code an algorithm from scratch which is too time-consuming. Moreover, I am wondering why there are not video course on OR, coding algorithms and etc.

I am willing to hear others' opinion and if it possible that scholars share the source of their codes (either heuristic, metaheuristic, B&B, B&P, B&C, and etc.).

Improve this question
$\endgroup$
5
  • 4
    $\begingroup$ what makes you think that OR is special in that context, and not the AI/ML community? $\endgroup$
    – Michael Feldmeier
    Commented May 31, 2019 at 13:43
  • 1
    $\begingroup$ "I am willing to hear others' opinion" - this is a bit opinion-based and that's why your question was closed. $\endgroup$
    – double-beep
    Commented May 31, 2019 at 14:31
  • 1
    $\begingroup$ The reasons are likely to all be opinions. Another opinion, and this one is mine, is that we need to create this culture in our community. The impact of our work will be greater for it. $\endgroup$
    – alerera
    Commented Jun 4, 2019 at 21:01
  • 1
    $\begingroup$ That soft-question tag is going to go in and out for a while. Can we engage on Meta on the issue: the "official SE" position is that soft-question should not be used. or.meta.stackexchange.com/questions/163/… $\endgroup$
    – Michael Trick
    Commented Jun 13, 2019 at 20:56
  • 1
    $\begingroup$ Yes, that’s fine by me. As a note, I added this before your question was asked on meta. I think the edit was just approved. $\endgroup$
    – E. Tucker
    Commented Jun 13, 2019 at 21:08

8 Answers 8

Reset to default
34
$\begingroup$

While it's true that much more O.R. code could - and probably should, when public money paid for its development - be public, this is changing and nowadays it's increasingly more common to share the sources (e.g., on GitHub) and create a corresponding referenceable artefact (e.g., on Zenodo).

I think the main reason that hinders sharing is poor code quality. Indeed, I had to work with code by colleagues and I wanted to carve my eyes out; surely, this feeling was reciprocated by colleagues who worked with my code. It is sometimes easier to just re-implement some algorithm than try to understand, debug, or extend someone else's code.

Why is, then, code quality so low? Why do we have script-like write-only one-use code, rather than more curated libraries, as is the case in the ML community? Here are a few plausible reasons:

  • Few OR scientists are computer scientists and even fewer are software engineers. We don't have this training in our curricula. Not everyone collaborates on large software projects; rather, we mostly write for ourselves. We use our own conventions and guidelines, often based in myths and second-hand factoids, rather than in truth and measurable metrics. Worse, we perpetuate these legends teaching them to our students.
  • We see code as a disposable tool, a temporary detour from the interesting scientific juice, a distraction from the publishable goodies, a necessary evil we have to cope with as quickly as possible. For many of us code is just a way to prove that some conjecture was true ("Doing X and Y should produce better results than the current literature"). We see limited future use for our code, even though we are often proven wrong in this respect.
  • Even if we saw clearly the value of reusable, maintainable, well-documented, publicly-shared code, the publish-or-perish culture dominant in modern academia gives all the wrong incentives: every hour of time I spend to increase code quality, I could have spent working on one more publication and getting tenure.

I feel the last point is the most important: sharing code or producing widely used software libraries gives almost no benefit, and a lot of costs. Until it will be considered as a service to the community and it will be taken into account for promotions and hiring, very few O.R. scientists will devote time to producing maintainable, well-structured, clear, performant, bug-free code which is worth being shared.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I really appreciate your remarks. You mentioned that "we perpetuate these legends teaching them to our students." I think that sharing codes might cause that these conventions be modified. When a person develops an efficient code and shares it, this efficient code will be the base in the community. $\endgroup$
    – Amin
    Commented Jun 1, 2019 at 17:04
  • 3
    $\begingroup$ "It is sometimes easier to just re-implement some algorithm than try to understand, debug, or extend someone else's code." Indeed. $\endgroup$
    – SecretAgentMan
    Commented Jun 12, 2019 at 13:58
24
$\begingroup$

All answers to this question are opinions. So I will add another one.

If you are a researcher and you are good at implementation, you may be able to produce a sequence of papers and results quickly in your subfield of OR. This is especially true if you focus on heuristics for very hard optimization problems. If you share your code, then you create an opening for other researchers to borrow your implementation and extend it to a related problem before you have the chance to do so yourself.

I don't like this mindset and I think we should move away from it now. It probably exists in part because the academic peer community which judges scholars values papers and citations above other measures of impact. If good metrics were available to measure the impact of your published code (reuse, application, extension, forks, etc.) then maybe this culture could change a bit.

Improve this answer
$\endgroup$
1
  • 8
    $\begingroup$ Amen. A good piece of software which works and is supported has a far greater positive impact on practice than an algorithm with some theorems in a journal. $\endgroup$
    – Mark L. Stone
    Commented Jun 4, 2019 at 22:59
15
$\begingroup$

I agree with @independentvariable. What might be added is that many researchers actually do publish (and what's maybe even more valuable: maintain) their code, if they think it is useful. Take a look at COIN-OR.

I guess one of the differences is that in OR, people tend to publish full-blown general purpose software (often fit for industrial usage) rather than a small piece of code that is just useful to replicate the results of a single paper.

Improve this answer
$\endgroup$
1
  • 4
    $\begingroup$ I think COIN-OR is a great counter example to the basis of the question. There, a number of people don't just distribute their code but many of them work to be sure they interconnect and build off of each other. $\endgroup$
    – Michael Trick
    Commented Jun 4, 2019 at 21:21
13
$\begingroup$

Good question! I would say:

  1. OR is older, way older than AI and ML
  2. OR people don't come from a programming background, and the main focus is usually having efficient maths rather than efficient coding (but I don't deny that OR people also write awesome codes :) )
  3. OR is collected around mathematics, if one is convinced with the theorem, there is no reason to implement the code and `observe it'. The fields using OR (financial engineering, industrial engineering, etc) usually share codes with the papers.
  4. It is well known that OR is interpretable while AI&ML are effective but black-box. Therefore, in an interpretable field, I don't think the codes make too much sense to share. In AI&ML, you need to convince yourself that an algorithm works very well, and you need to see the results. But in OR papers, theorems are more like "this algorithm gives a $1+\epsilon$- approximation, and the proof is here"
  5. One of my professors told me: "Codes are for computers, not for humans". I can not think of any good reason to read a code to understand the theorems. Therefore, the only reason for such a demand would be to use these codes directly.
  6. This may be my subjective opinion, but people who use GitHub actively benefit from the version tracking of this software. In AI&ML projects, the efficiency can be improved continuously and GitHub can be used actively. But, in OR one should formulate the mathematical model before coding it. Therefore, the only version is the first working code if you think about the optimization problems. This is why on Git-Hub you can not find too many OR projects.
  7. There are many options in solvers. If one shares her code, people will be like "why are you not tuning this option, why are you using for loops when you can define a vector, etc. etc.". There is peer pressure due to the fact that there are lots of programming environments and many solvers. Also in OR, the same model can be implemented in Julia-Python-Matlab-C++-R-and so on. However, in CS-related fields, usually, packages are working in specific programming languages. This is again due to the fact that AI-ML fields are centered around computer programming while OR is centered around mathematical programming.
Improve this answer
$\endgroup$
2
  • $\begingroup$ thanks for your comprehensive response. I agree with your points but there are some reasons that might justify sharing codes. There are some algorithms that provide error bound, convergence rate, etc. However, for example, when we define a valid inequality, it is not obvious how it will improve the computation time and how much improvement will be obtained and this improvement can be evaluated through numerical examples. $\endgroup$
    – Amin
    Commented Jun 1, 2019 at 16:50
  • $\begingroup$ Moreover, as I encountered, there are some problems that the system goes out of memory (when CPLEX is used). Therefore, when using commercial software is a part of our algorithms, it might be important to take into consideration how the memory is managed in our algorithm (e.g. B&P, ....) $\endgroup$
    – Amin
    Commented Jun 1, 2019 at 16:51
11
$\begingroup$

I think the five answers (up to now) give a representative sample of the different reasons for not publishing code in the OR community. That answers your question.

To provoke a bit, I would say most of them are invalid.

As an exercise, it helps to read LeVeque's Top Ten Reasons To Not Share Your Code (and why you should anyway) and imagine mathematicians defending not publishing proofs with similar arguments. Quoting from the article:

  • The proof is too ugly to show someone else
  • I didn't work out all the details
  • I didn't actually prove the theorem, my student did
  • Giving the proof to my competitors would be unfair to me
  • The proof is valuable intellectual property
  • Including proofs would make maths papers much longer
  • Referees would never agree to check proofs
  • The proof uses sophisticated mathematical machinery that most readers/referees don't know
  • My proof invokes other theorems with unpublished (proprietary) proofs
  • Readers who have access to my proof will want user support
Improve this answer
$\endgroup$
10
$\begingroup$

Probably because a lot of us are mathematicians and are ashamed of our code quality.

I have just released the code to BCP, a poorly-named (don't blame me) algorithm for solving the multi-agent pathfinding problem using branch-and-cut-and-price. It is substantially faster than the previous state-of-the-art algorithm CBSH-RM. The paper and experimental results are at my website ed-lam.com.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I think there is another "hidden" argument in you answer: often OR researchers are mathematicians, not computer scientists. You will often see papers where the convergence rate of some algorithm is proved without actually implementing and testing an algorithm. $\endgroup$
    – Michael Feldmeier
    Commented May 31, 2019 at 13:44
  • 1
    $\begingroup$ @EdwardLam This might be true in some cases while these codes even with poor quality will definitely be better than the code of a newbie OR student and practitioner. I really appreciate you if you can share your code. $\endgroup$
    – Amin
    Commented Jun 1, 2019 at 17:09
8
$\begingroup$

In my point of view, the main reasons that people overlook sharing their codes are:

  1. It is really time-consuming and challenging to get a good-quality code. It might take the same amount of time to clean up the code compared to the time that you originally spent on it. You may see inconsistencies in results after cleaning it and many other troubles.

  2. The community gives almost zero credit for your code. For example, a special value can be provided for your paper, if you submit the code. Or, the journals might publish the codes on their websites. It can have a value toward getting tenure, etc.

I also want to mention that this issue is not specific to our community. AI/ML people have the same concerns since not too many people still open-source their codes. Here is a talk from NIPS 2017, which discusses the reproducibility issues: https://aitube.io/video/38-min-joelle-pineau-reproducibility-deep-reinforcement-learning-nips2017/ At the end of this video, the author of AlphaGo argues that it will require about a year for them to clean up their code and release it. The discussions are still going on in their community...

Finally, I would like to share my experience of open-sourcing the code for one of my paper: https://github.com/OptMLGroup/VRP-RL It took almost 1 month for cleaning the code, but after publicizing the code, I got very positive feedback from both optimization and computer science communities. 87 unique visitors within the last 6 month and ~30 daily repo visits show that people are learning from my code. 28 people have ''forked'', which means that these people can continue their research from the point that I stopped. I strongly believe that open-sourcing is one of the best ways to distribute your work, and I will continue to do afterward.

Improve this answer
$\endgroup$
4
$\begingroup$

You might be surprised how much OR algorithm code is open source:

Now, the thing is that production solvers can complexify the algorithm. For example, if you look at the Local Search solver phase code in OptaPlanner, you'll see the Local Search algorithm, but much is hidden behind the decideNextStep() method, which has an single-threaded implementation that is understandable, but also a multi-threaded implementation that is much harder to read.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.