Respond to Incoming Twilio Phone Calls Using NestJS and TensorFlow

A full-code tutorial that utilizes NLP, BERT, and QA (Question-Answer) Annotators to automatically assist in phone call response

Published in

Heartbeat

6 min readAug 10, 2022

Phone calls remain a huge part of customer service in many businesses. They help in resolving customer queries and disputes, and businesses hire call centers to respond to these calls. Phone etiquette, the use of professional tone, and being knowledgeable about product is key in this field.

In this tutorial, you will learn how to use machine learning to respond to incoming phone calls. You will first create a simple web application server to receive Twilio calls, then use natural language processing (NLP) to understand and respond to callers. You will also learn how to integrate TensorFlow into a NestJS app with basic NLP. TensorFlow has an implementation of Bidirectional Encoder Representations from Transformers (BERT). BERT is a well-known transformer-based NLP technique that, among other things, can be used to make question and answering models.

Prerequisites

Please make sure that you have NodeJS (v14+) installed on your operating system. Follow this guide for getting started with NodeJs. You will also need the NestJS framework to build your server-side application. Install the Nest CLI using npm, which you will use to create and run your project.

npm i -g @nestjs/cli

Project Setup

You will use the Nest CLI to scaffold your project. To achieve this, run the following command then cd into the project folder:

nest new twilio-tf-answer-calls
cd twilio-tf-answer-calls/

The new project created will have a folder called src/. This is where you will write your code. You will edit the app.controller.ts and app.service.ts files.

Next, you will install dependencies via npm, which will be building the project. These dependencies include Twilio and TensorFlow’s BERT model. For more information about installing TensorFlow for NodeJS, check out this Github Repository.

npm i -s twilio @tensorflow-models/qna @tensorflow/tfjs-node @tensorflow/tfjs-converter

Finally, you will need to buy a phone number that has voice capabilities using Twilio’s console. Sign up or log in to your Twilio console, then get your free phone number or buy one. You can use it to receive or make calls from anywhere in the world.

TensorFlow Integration

Open your project in your favorite editor and open the app.service.ts file. This is the application service that hosts the BERT model provided by TensorFlow. In this section, you will perform the following tasks:

Import the necessary TensorFlow modules installed earlier on;
Add a simple knowledge base for a particular product. For this tutorial, you can grab any product details from an e-commerce site;
Initialize the BERT model provided by TensorFlow;
And add a function to provide answers to a question.

The ML process is cyclical — find a workflow that matches. Collaborate across teams, reproduce experiments, and more with a strong MLOps strategy. Check out our expert solutions for overcoming common ML team problems.

Twilio Integration

Twilio’s JavaScript SDK enables developers to integrate communication services in web applications. It works quite well with the NestJS framework. In this section, you will focus on the app.controller.ts file. It handles the creation of the API routes for our application. To integrate Twilio voice services for this use case, you will need to do the following:

Import the required dependencies to run the app’s controller.
Initialize Twilio’s SDK using your keys. You can grab your keys from Twilio’s console. Also, add the app’s service into the controller through dependency injection.
Create a route for handling receiving a voice call and greeting the caller. Initialize the process of transcribing the call to text using Twiml’s record verb. It is important to note that there are legal obligations in regards to privacy when you decide to record calls.
Create a route responsible for letting the user know that their query is being processed.
Create a route that will handle receiving the transcribed text. Pass the text to your machine learning model and post-process the results. Transcribing calls attracts extra costs; you can find the pricing here.
Create a route that responds to the caller with the appropriate answer and finish the call. It is important to note that this call only takes one query per call.

Running the app

Nest CLI provides an easy-to-use mechanism for developers to run their applications. For this tutorial, you will be running your web app server on your machine. Make sure you are at the root folder of your application before running any command in this section. Use the following npm command to run your app in development mode:

npm run start

You can access your web application server through http://localhost:3000. But you still need to integrate your API routes with Twilio’s online services via Webhooks. Webhooks provide a mechanism for Twilio to communicate with your app. First, you need to use ngrok to create a tunnel that will expose your local app online. To do this, run this command in a new terminal session while still running your local web app server:

ngrok http 3000

3000 is the port number that your local app server is running. Please make sure you have the correct number.

ngrok offers some URLs that expose your local app server online. Copy the public HTTPS link and head over to your Twilio console. Select your phone number and under A Call Comes In select Webhook and add the public URL you copied earlier.

Conclusion

In this tutorial, you were able to create a web app server using NestJS. This web app server can receive incoming calls from Twilio. Also, it can process the caller’s query about a product. Through NLP, you were able to respond with an appropriate answer to the question. The possibilities of using NLP in communications are endless. It can save costs for companies if implemented. This technology still needs oversight by people, but it could help customer care agents and representatives boost their performance.

Editor’s Note: Heartbeat is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.

Editorially independent, Heartbeat is sponsored and published by Comet, an MLOps platform that enables data scientists & ML teams to track, compare, explain, & optimize their experiments. We pay our contributors, and we don’t sell ads.

If you’d like to contribute, head on over to our call for contributors. You can also sign up to receive our weekly newsletter (Deep Learning Weekly), check out the Comet blog, join us on Slack, and follow Comet on Twitter and LinkedIn for resources, events, and much more that will help you build better ML models, faster.