How to finetune and deploy a text generation model

Knowing how to finetune a text generation model is incredibly powerful.

A text generation model is a text in, text out system with built-in knowledge of language and problem solving. If used correctly, such a model can be used to solve any text based task.

This guide will show how to finetune a T5 model to come up with questions that can be asked on the given context.

question generation
This guide does not include a dataset. To follow along, see the notebook version of this guide.
To run it, you must have an Nvidia GPU. If you do not have one, you can use a Google Colab notebook.

Setting up Backprop

Backprop provides an open-source library that makes it easy to finetune general models for your own specific use cases.

Install backprop.

pip install backprop

Import backprop.

import backprop

Finetuning the text generation task

Finetuning a text generation model is simple. All you need is to pass in lists of input and output text.

Start the text-generation task with a finetunable model from Model Hub.

tg = backprop.TextGeneration("t5-small")

Pass in your data to the finetune method.

prompts = [
    Generate questions:
    Backprop is a serverless model platform that makes it simple for developers to use machine learning models in any application.
    Each model is exposed as a private API that deploys with zero downtime, scales automatically, and requires no maintenance, enabling to focus on building great applications.

completions = [
    1. What is Backprop?
    2. How does Backprop help developers?
    3. How does Backprop help developers focus on building great applications?

tg.finetune({"input_data": prompts, "output_data": completions},
            max_input_length=300, max_output_length=200)

Wait for your model to finish finetuning. Depending on your GPU and amount of data it may take anywhere from minutes to hours.

Test and upload your model

Try out your model, if all went well it should now be able to generate questions.

Remember to make input data similar to your finetuning data.

prompt = """
    Generate questions:
    Backprop's API is used to solve tasks and manage models. Tasks can be solved with our pre-trained and your own uploaded models.

completion = tg(prompt)
>>> """
    1. What is Backprop's API used for?
    2. What models can be used for task solving?

Once you are happy with the model, you can upload it for scalable, production ready inference. Learn more about hosting your models on Backprop.

The model can always be updated later, even as it is receiving requests. It's good to get an initial version up and improve it down the line.

Uploading is done in a single line by using your API key from the Dashboard.

We are naming the model t5-question-generation, this is used to call it later via the API.

        description="Generates questions, given context.",

In around 10-15 minutes your model is deployed and ready for inference.

You can track the progress via the dashboard.

t5-question-generation in dashboard

Using your model via the API

Once your model is deployed, it is ready for inference on Backprop's infrastructure.

You can try it out using the text generation task in Sandbox.

To integrate it as part of an app, use the API directly. See the API reference for text generation.

import requests

prompt = """
    Generate questions:
    Backprop's API is used to solve tasks and manage models. Tasks can be solved with our pre-trained and your own uploaded models.

res ="",
                    headers={"x-api-key": "YOUR_API_KEY"},
                    json={"text": prompt, "model": "t5-question-generation"})

>>> {
        "output": "1. What is Backprop's API used for? ..."