The question I am about to ask is not a technical one but rather based on following a correct approach, which I am sure would be helpful to many.
I am currently working on a project which involves data driven optimization for a gas compressor network, where we read in plant's data for process variables such as pressure, temperature, flow etc. We in there, are building several statistical (regression) models that form part of our objective function and constraints. We are in the pilot phase, and our ultimate purpose is to deploy this tool real time in plant. At this moment. I am currently having with me a large data set that is reliable enough to build generalized models.
However, thinking it from the deployment perspective, we would not want to be in a position where we are doing an optimization based on data that is too old. In fact, my project adviser mentioned that I should have the facility to read in live data from the plant and update my model parameters based on the new data points.
Currently, what I can think of is to read in new data to new csv files via python and update my models. However, this requires me to manually run the code at fixed frequency (eg daily at 10:00 AM). I am wondering, if there is some more smarter way to approach this problem ? Ex. deploying my code on some cloud platform that is automatically able to stack new data to the existing database, run the code and update parameters, the latest values of which I can then retrieve to my local machine to run the optimization code. I would be keen to know if someone has worked on some similar problem before as I would like to learn more on this topic.