Inside AT&T’s Open-Source AI Approach to Customer Service Calls

AT&T gets 40 million customer service calls annually. Some callers want to add phone lines, register new addresses, or reschedule appointments, but many have problems to report. Those calls contain valuable information, but extracting it isn’t easy.

A person listening to each of them would get a good idea of what new issues are arising and could catch small problems before they grow into big ones. But with thousands of calls coming in each day, that would be an arduous, virtually impossible task.

Transcription has been automated for a while, so AT&T used to do the sorting by hand. But employees had to read millions of summaries and put each call into one of 80 categories to be analyzed for any follow-up actions that could be taken. The ultimate goal is to prevent what consumer-oriented companies call “churn.” Essentially, they want to keep the customers from leaving.

Hien Lam, a senior data scientist at AT&T, explained the process during a presentation at Nvidia’s GTC Conference in March.

Now, with large language models, AI can ingest the summaries and categorize the calls.

The ChatGPT way

It was pretty simple. AT&T used ChatGPT to read and sort the summaries. It did a good enough job, but Lam’s team saw problems coming down the road.

“While the GPT-4 model did produce very high-quality outputs, and we were able to save 50,000 customers annually, it was very expensive,” Lam said. Plus, customers of ChatGPT sometimes have to wait for the powerful and expensive chips to run AI systems, called graphics processing units, to become available.

Sorting the calls was a daily task. “If it takes you longer than you can run overnight, then it’s not a reasonable workflow,” said Ryan Chesler, a principal data scientist at the open-source AI platform H2O.ai, who worked with Lam on the project.

So they set out to create a more flexible system that AT&T could have more control over, but that also cost less.

The open-source way

Lam teamed up with Chesler, working under the theory that if they could stitch together several open-source AI models with different “skills,” they could achieve similar results with dramatically lower cost while keeping the company’s data private.

First, they distilled GPT-4 into three smaller, open-source models. The most basic model was smart enough to sort roughly a quarter of the categories. A call that mentioned a competitor’s name, for example, was easy for a model to identify. A call with a nuanced story about a store team member required a more sophisticated model.

About half the calls could be handled by an open-source model called Danube, a small but powerful model created by H2O.ai. Lam worked with Chesler to fine-tune it to AT&T’s needs.

The most complex calls went to Meta’s Llama 70B model, which is larger and more costly to run. Open-source models are inherently cheaper, but they’re not free to run since they still require computing power. But by only using the larger models when necessary, the team kept costs down.

In fact, the open-source patchwork solution cost 35% of what AT&T was paying to use ChatGPT, with 91% relative accuracy, Lam said. It was also faster.

“Using GPT-4, it took 15 hours to process one day’s worth of summaries. In our new solution, it took a little under five hours,” Lam said.

Next, they’re looking to speed things up even more.

“Because it takes 4½ hours for a full day, we are looking to do it real time after you hang up with AT&T,” Lam said. “We could get those outputs immediately.”