Deploying models

Backprop supports models built on Python based frameworks.

Each model runs in an isolated serverless CPU environment that is optimised for ML inference.

How it works

Backprop's deployment is a CI/CD pipeline that takes your model from file to production with zero downtime.

Your model only gets deployed if it can successfully perform your specified tasks. In case of any errors, your previous deployment will remain running.

model deployment pipeline screenshot

All models run isolated, so you can specify the exact runtime dependencies.

A valid model must support at least one task. In order to support the task, the model must follow the task's schema.

For more control, your models can support the custom task which can take any valid json as input.

Examples

There are three ways of uploading a model:

  1. Using dill and serializing a valid model.bin file. See basic-dill for a basic example.
  2. Writing a valid inference.py file. See basic-file for a basic example.
  3. Using Backprop's library to finetune and deploy a model.

For framework specific examples, check out:

  1. Text classification and other tasks in PyTorch.
  2. Image classification and other tasks in TensorFlow.
  3. Custom task in SKLearn.

This example below shows how to upload a very basic text generation model using dill.

To deploy a dill serialized model, you need to upload a zip file that contains 3 files: model.bin, requirements.txt and config.json.

model.bin

The model.bin file is a serialized callable python object that implements your tasks.

In this case, we need to do it for the text-generation task. See the task schemas.

The following script creates and serializes the model.

Deciding what task to perform is done according to the API endpoint the request is sent to. All of the task parameters are passed as a dictionary, while the task name is passed as a keyword argument.

requirements.txt

This is a classic python requirements file. In this case we have no external dependencies and it will remain empty. It must be included, even if empty.

You can specify any python packages here, most commonly your deep learning framework.

config.json

This file requires just two keys: tasks and description. Tasks is a list of strings, while description is just a string.

Uploading

Select the three files you created and make them into a zip file. Don't put them inside a folder - zip them directly. The name of the zip does not matter.

zip model files

Next, navigate to the models tab in Dashboard, click on "add a new model", give it a name, drag your zip file, and hit upload.

model upload modal

Task schemas

In order for a model to support a task, it must follow one of Backprop's task schemas.

The most flexible is the custom schema that passes the entire JSON body to your model.

The schema specifies the dictionary argument and task keyword argument that are used to call a model. The dictionary argument is the JSON request body of an API request.

Assume we are making a POST request with the following JSON body to the /text-classification endpoint:

{"text": "Some text", "labels": ["sales", "marketing"]}

The model would be called as: model(json_body, task="text-classification").

The model can access the text and labels keys of the dictionary to calculate an appropriate response.

It can return anything, as long as it is JSON serializable.

In this scenario, our model might return a dictionary of probabilities:

{"sales": 0.85, "marketing": 0.3}

The schema also defines the output JSON key where the model's output is stored. In the case of the text-classification endpoint it is probabilities.

Our final response to the API call would be:

{"probabilities":{"sales": 0.85, "marketing": 0.3}}

See the schemas for all tasks in our API docs.

Requirements

In order for a model to be a valid API it needs to follow some requirements.

A valid model must:

  • implement one or more tasks according to the task’s input schema
  • have a valid list of tasks (strings) as a tasks field
  • produce JSON serializable output
  • not have any unspecified external dependencies (requirements.txt)

There are additional requirements, depending if using dill or inference.py.

If using dill, a valid model must:

  • be recursively pickled
  • not use super() calls (dill limitation)
  • not reply on user defined dependencies (classes, functions, variables outside of the model)

If using inference.py, a valid model must:

  • define load_model and call_model functions in inference.py
  • include any required files (e.g what your load_model function uses) as part of the upload zip file