How to build a zero-shot email classifier

Some of the best models are able to solve various tasks in a zero-shot setting. It means being able to solve tasks without any examples and finetuning.

This guide will be using a zero-shot text classification model and an emotion detection model to automatically sort emails.

The end product will be able to process e-commerce customer emails, ranking them by urgency and category (shipping, complaint, etc).

This guide assumes you already know how to use tasks.
Read the tasks tutorial if you are new.

Setting up Backprop

For convenience, we will be using the Backprop open-source library.

You can do it via plain API calls. Using the library lets us write a bit less code.

Install backprop.

pip install backprop

Import backprop.

import backprop

Starting the Emotion task

This starts an Emotion task with the base model in the API. See more details in the API Reference.

emotion = backprop.Emotion(api_key="YOUR_API_KEY")

Starting the Text Classification task

Starting a Text Classification task works exactly the same way. See more details in the API Reference.

tc = backprop.TextClassification(api_key="YOUR_API_KEY")

Detecting urgency

The default model used for the emotion task is t5-base-qa-summary-emotion.

It can detect various emotions. The logic here is that if a negative emotion is detected, the email is marked as urgent.

negative_sentiments = ["annoyance", "disapproval", "disappointment", "anger", "disgust"]
def is_urgent(text):
    detected_emotions = emotion(text)
    detected_emotions = detected_emotions.split(", ")

    # Email is urgent if the customer has negative sentiment
    for em in detected_emotions:
        if em in negative_sentiments:
            return True
    
    return False

Predicting the category

As part of zero-shot classification, we have to provide a set of labels. These are the labels that will get assigned probabilities.

The logic here is to give all possible categories and predict the one with the highest probability.

categories = ["returns", "promotion", "complaint", "technical issue", "product inquiry", "shipping question", "other"]
def predict_category(text):
    probabilities = tc(text, labels=categories)
    # Gets the highest probability category
    category = max(probabilities, key=probabilities.get)

    return category

Putting it all together

Now, we can write a simple function that makes use of the urgency and category detection to sort our emails.

def process_email(text):
    email_urgency = is_urgent(text)
    email_category = predict_category(text)

    print(text)
    if email_urgency:
        print("Urgent!")
    else:
        print("Not urgent")

    print("Category:", email_category)

Let's try out some examples to see how it performs.

email1 = "Hi, how long will it take for my order to arrive? I've been waiting four days."
email2 = "How could you sell something this broken? This is the worst thing I've ever seen, and you should be ashamed to make money off it."

process_email(email1)
process_email(email2)
Hi, how long will it take for my order to arrive? I've been waiting four days.
Not urgent
Category: shipping question
------
How could you sell something this broken? This is the worst thing I've ever seen, and you should be ashamed to make money off it.
Urgent!
Category: complaint
    

Awesome. The emails get the correct category and the customer who's really disappointed in the service gets their email marked as urgent.