...
| Deployment method | Pros | Cons |
|---|
| Docker image | - Docker images encapsulate all the runtime dependencies
- Very flexible as users can build whatever image they want
- Docker will manage all the life cycles of the "model offloading"
| - Users have to build the images themselves, write the Dockerfile, build the image and upload to a docker registry
- Users have to provide a private docker registry if they don't want to use the public dockerhub.
|
| Machine learning model file manifestsfiles manifest | - Data scientists directly work with model files. It would be nice if they can just drop their model files somewhere
- By using a data store, it opens the door for serverless computing
| - Our framework has to manage the whole life cycles of model files deployment, update, delete, etc.
|
How can the InferenceModel CRD be used?
Simple machine learning offloading to edge
Just create an instance of InferenceModel with "DeployToLayer == edge"
Joint Inference
Create three resources:
- An instance of InferenceModel to the cloud
- An instance of InferenceModel to the edge
- A pod running on the edge for serving customer traffic. It contains the logic for deciding whether or not to call cloud model serving API.