Initial discussion

[inejc]

This issue is a continuation of this Reddit question and serves as an initial discussion regarding design choices and collaboration on this project.

High-level requirements of the project:

• the ability to expose trained doddle-models as a REST API
• the ability to serve multiple models and multiple versions of the same model simultaneously (A/B testing different models/versions)
• a mechanism to group multiple inference examples into a batch before they are sent to the actual model (similar to the groupedWithin operator and this Apache Beam trigger)
• a CLI for model deployments

Technology Candidates

• Akka Streams
• fs2
• monix

Resources:

• Tensorflow Serving (a good example of what we are trying to port to Scala)

[inejc]

The first step should be to decide on the tech stack. I should say that I don't have any experience with fs2 or monix and very limited with akka, so I'm very interested in hearing opinions re that.

[SentinelCyberSecurityMunich]

👍 Sounds good i worked with akka streams and graphs so in that I can offer help would also be interessted in the latest technologies regarding ai like tensor flow so thank you

[h0ffmann]

I'm interested in contribute. I already worked just with Akka, but i'm open to use new libraries, in special if they use more FP concepts like monix or fs2.

See also
https://github.com/Azure/mmlspark/blob/master/notebooks/samples/SparkServing%20-%20Deploying%20a%20Classifier.ipynb

They doesn't have scala examples yet but is the same objective. I think we can catch same good ideas.

[ashwinbhaskar]

I would like to the reasons for the choices suggested. If it's a simple rest API, then we don't need streaming. What use cases (in detail) are you expecting the REST API to solve?

[inejc]

Sorry for the delay, I'm AFK for a few days. I'll respond with detailed examples on Monday.

[brurucy]

I vote for akka.

[inejc]

I'll try to capture the requirements by presenting a scenario in which we want to productionize a simple predictive model.

1. Model Example

Let's assume we want to serve a model that will help doctors predict the weight of a person based on their height, gender, and age. The CSV below could be a dataset used to fit such model:

height,is_female,age,weight
179,0,27,83
170,1,24,61
185,0,23,88
190,0,21,94
164,1,23,55

Once we fit the model on the data above, we can use it to make predictions for new examples without the weight information, e.g.:

height,is_female,age
171,0,25
169,1,23
183,0,23
192,0,19
158,1,29

In other words, we can think of the fitted model as a function that maps height, gender, and age to weight: weight = f([height, is_female, age]). See the fit and predict methods in the predictor typeclass to see how this is exposed in doddle-model; in our case height, gender, and age are called features and weight is called a target.

See this to realize that Features is in fact a matrix and Target a vector of Double. This kind of API allows for fitting and predicting on multiple instances (rows in the CSV files) at once. If we take a look at the first CSV file again (the one used to fit the model), we can see that all columns except the last one constitute the features matrix, i.e.:

179,0,27
170,1,24
185,0,23
190,0,21
164,1,23

and from the last column we construct the target vector, i.e.:

83
61
88
94
55

The main reason for putting multiple examples together before calling fit and predict is that in most cases, because of the nature of the underlying implementation, it is much faster to do the calculations that way as opposed to doing them on each row separately in a loop (i.e. having predict exposed as predict(model: A, x: DenseVector[Double]), where x is each of the rows from the second CSV file).

2. Model Serving

Now let's assume we want to expose the fitted (trained) model for hundreds of hospitals, meaning that thousands of different doctors will be using it every hour. Essentially, each of them will be calling a REST endpoint (GET /model/<model-name>/predict/) for a single patient that they are currently dealing with (a single vector constructed from height, gender, and age measurements). JSON request data might look something like:

{
  "instances": [
    [171,0,25]
  ]
}

In order for us to be able to serve predictions (weight) as efficiently as possible, we need to put vectors from different doctors/patients into a matrix (as rows) and only then call the predict function on the model.

Notice that serving (generating predictions) is completely stateless and can thus easily be scaled horizontally, i.e. we can serve the same model on multiple cores/machines.

Additionally, we sometimes need to serve the same model trained on different datasets (or even with different features; e.g. we add a measurement related to body fat) and then compare their live performance with A/B tests to figure out which version works better in practice. We should thus be able to specify the version when calling the REST endpoint as well (GET /model/<model-name>/version/<version-identifier>/predict/).

[inejc]

Following the example from the previous comment, some of the doctors could collect measurements from multiple patients and send more instances at once, e.g. JSON request data might look like::

{
  "instances": [
    [171,0,25],
    [169,1,23],
    [183,0,23]
  ]
}

Despite that, the serving system should still batch instances from multiple requests together. In cases where models would be served in environments with enough memory, larger batch sizes should increase performance in high-load scenarios. Batch size and maximum latency should be configurable, similar to the groupedWithin operator from Akka Streams.

[inejc]

3. Model Deployment

It should also be possible to deploy new models and versions without changing any code, i.e. via a REST API. E.g. by calling something like POST /model/create/ or POST /model/<model-name>/version/create/.

We could load serialized models from Google Cloud Storage buckets and from other locations (whatever storage we decide to support).

I think a central type for the things that could be "served" should be something like Servable (similar to how Tensorflow Serving is designed). We would then provide implementations for doddle-models, but it should also be possible for users to provide their own implementations for custom servables. E.g. a servable that uses a trained neural network (see tensorflow_scala) to generate features from raw text (embeddings) and then a trained doddle-model that makes predictions based on the embeddings.

I hope this makes things a bit clearer. Let me know if anything above doesn't make sense or more info is required. In any case, I'm looking forward to continuing this discussion.

[ashwinbhaskar]

Can I join this project?

[inejc]

@ashwinbhaskar of course, there was no work done on that yet.