PS: I am familiar with performance variability encountered while solving large-scale MIPs. My question may not be explicitly related to it but maybe to the inefficiency of python packages.
I have developed a Python-code to solve the LP-relaxation of a large-scale airline crew pairing optimization problem (an SCP/SPP formulation) using a heuristic-implementation of the Column Generation technique. First, I generate an initial feasible solution to initialize this CG-heuristic. Subsequently, in each of the iterations of CG-heuristic,
The resulting LP is solved for dual variables using GUROBI.
These dual variables along with information of existing flight-connections are used to construct partial pricing subproblems as the full-scale network is intractable. From this, I only generate the columns/variables with negative reduced cost. To speed up this part, I have decomposed the problem into independent subprocesses and executes them in parallel (on a 16-core processor) using Multiprocessing Python's library.
My question is related to the runtime-performance of Step 2. In that, whenever I run the code on a system that has been ideal for some time, it takes multiple (at least thrice) reruns (at least till 4-5 CG-iterations) in order to get an acceptable runtime-performance. In these initial reruns, the runtime-performance until Step 1. is the same. However, when Step 2. executes, the CPUs History chart in task manager shows that the system takes a long time to start it, i.e., no process runs for some time. This takes a lot of manual efforts (stopping and re-running the code again and again). Has anyone faced a similar issue or has a clue about why this is happening? Is this because of the Multiprocessing library of Python?