Deploying OR Model as an API

Question

I want to deploy my OR model (python, pyomo) as an Rest API (using Flask) that too on Azure Kubernet. All I have came across so far are ways to do the same for ML models, which can be uploaded as pickle file and used, but the same is not possible with my model.

I am looking for resources addressing OR models for the deployment but any help is appreciated, and a template will be a bonus.

Answer 1

I have made OR models accessible through APIs on Kubernetes before. It isn't too hard, no pickling involved. In general, the steps aren't unique for an OR model API, but there might be a few things to consider (for example, an OR API might take more time than an average API to return a result).

1. First of all, you need to decide on a data exchange format. The JSON format works really well for most applications. For example, let's assume your model is going to solve a TSP problem. Your model would expect a list of points and their coordinates and maybe a few parameters, such as a time limit or the optimality gap you are willing to accept. As a result, you would expect a JSON that contains the length of the solution and possibly the order in which the points are visited. A sample input could be as follows:

   {"points": [{"name": "Geneva", "long": 46.2044, "lat": 6.1432}, {"name": "Zurich", "long": 47.3769, "lat": 8.5417}, {"name": "Lucerne", "long": 47.0502 , "lat": 8.3093}, {"name": "Bern", "long": 46.9480, "lat": 7.4474}], "time_limit": 30}
   

   The expected result might be something like this:

   {"status": "optimal", "order": ["Zurich", "Bern", "Geneva", "Luzern"], "score": 598}
   

2. You will now need to write a Flask app that can receive the JSON input, pass it on to your OR model and then return a result as a JSON. This app will need at least one route that you can send the data to. Here's what the basic structure could look like:

   from flask_api import FlaskAPI
   
   app = FlaskAPI(__name__)
   
   @app.route("/optimizer/", methods=['POST'])
   #@require_api_token # You can implement a wrapper method to authenticate requests to the API
   def optimizer():
       """
       Route to start the optimizer. Expects JSON data sent as a POST request
       """
       problem_data = request.data
       result = my_or_model(problem_data)
       if result: 
           return {'status': 'optimal', 'result': result}
       return {'status': 'infeasible'}
   
   def my_or_model(problem_data)
       # you can access the data (e.g. time_limit) as follows: problem_data['time_limit']
       # result = run your OR specific algorithm and return a result
       return 100
   
   if __name__ == "__main__":
       app.run(debug=False, port = 5000, host = "0.0.0.0")
   

   Note that I left away authentication/ licensing, but this isn't too hard to add and you will find a lot of information about it if you search for Flask api authentication.

3. Theoretically, this app is ready to be deployed, however, it is better to server Flask apps through Nginx or something similar. A very lightweight server is waitress-server, which is a python module that allows you to set up a very simple server:

   from waitress import serve
   import myflaskapp # assuming we have defined the flask app in the file myflaskapp.py
   serve(myflaskapp.app, host='0.0.0.0', port=5000)
   

4. You now have everything to run your API locally. If you run python waitress--server.py you will be able to access your API at http://localhost:5000/optimizer/ and send data to it.

5. To get this deployed on Kubernetes, all that is left is to do is to dockerize your flask app and create a Kubernetes deployment. You will find more information on this elsewhere, but the most simple Dockerfile could look as follows:

   FROM python:3.7
   RUN mkdir /app
   WORKDIR /app/
   ADD . /app/
   RUN apt-get update
   RUN pip install flask-api
   CMD ["python", "/app/waitress-server.py"]
   

6. After building the docker image (docker build . flask-image) and pushing it to a docker registry, e.g. docker hub or whatever Azure is providing, you need to create a Kubernetes deployment file. In this example I am creating a deployment that runs 4 instances of the app, but this isn't necessary:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: flask-deployment
   spec:
     selector:
       matchLabels:
         app: flask
     replicas: 4
     template:
       metadata:
         labels:
           app: flask
       spec:
         containers:
         - name: flask
           image: path to your docker image on dockerhub
           ports:
             - containerPort: 5000
           imagePullPolicy: Always
         imagePullSecrets:
         - name: regcred
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: flask-service
   spec:
     type: LoadBalancer
     ports:
     - port: 5000
       targetPort: 5000
       name: http
     selector:
       app: flask
   

7. That's pretty much it. You now just have to connect kubectl to Azure Kubernetes and you can create the deployment: `kubectl create -f deployment.yaml`

For further questions, I think that the Kubernetes Stackexchange would be the right place as this is really not specific to an OR model.

If your OR model is requiring more than just a couple of seconds you probably don't want to let the requester wait and risk the connection to time out. In this case it might be best if you can send back the result later (either to an API that is receiving the result or maybe as an email, similar to what happens when you solve a model on NEOS server.

Answer 2

I think in your case you have to separate between the data and the model. The model itself is part of the service in some way. Either you hard code it into the service or you have some sort of abstract description of the model (for example in some modeling language that the solver can read) so that you can upload new versions of the model to the service.

What the user then uploads is the data with which the model gets instantiated. This gives a problem instance that the service then solves and eventually returns a solution for.

So in your scheduling example, the user would only provide the jobs/machines/... and the service would construct a model from this based on the abstract problem representation.

Answer 3

I am unsure what exactly you want to pickle. Is the user posting data to your endpoint (the flask service)? In that case you probably need to re-run the algorithm.

In machine learning the model is usually pickled as you have already trained it. Thus the user in that case just need to provide the input for the model and it will immediately make a prediction on the input data.

As I see it you either want the user to call your model with new input, thus it needs to re-optimize. I don't think pickling anything here will help you particularly. If you just want the user to be able to access the model, then the question is why?

Answer 4

As you have noticed there are ready-made services available for ML but not for OR. You will have to roll your own.

You can get a feel for the architecture requirements by looking at the code and REST APIs of open-source projects, e.g. graphhopper, vroom. It takes time to understand source code of others, and software architectures even more so. But the blue-prints are out there and readily available.

You will find that common components for a minimal software-as-a-service architecture include:

• Data exchange format (probably JSON): separated between input/request file (input data and input parameters of your optimization problem) and output/response (solution of the optimization algorithm)
• Server: backend that handles incoming API requests (mostly POST or PUT request), calls your optimization algorithm, and sends the solution back to the client
• Client: executable programm code on the client side (downloadable programm, or for example a python package) that builds the data exchange format internally and sends it to the server. This is an optional convenience feature for REST services because the clients can also build the data exchange file themselves.
• Licensing mechanism: checks against a database if a request has a valid API key
• Database: holds a record of user data and valid licenses for the licensing mechanism
• Website: user registration and API key purchase option for creating database entries
• Documentation: can be part of the website or hosted by dedicated third party providers (there are plenty good options if your project is open-source)

Obviously licensing and database is optional if your service is free of charge and small-scale. Also, related.

Answer 5

Perhaps, you will find some software we built https://fuinn.github.io/mos-docs/ useful. Provides a REST API for interacting with optimization models.