Episode 1: Evaluating AI at Your Company with the PAC Framework
Join Talla's CEO and Co-Founder Rob May, and Director of Marketing and founding team member Brooke Torres, to explore the PAC Framework. This episode is a great place to get started if you are looking at AI and thinking about how to evaluate and deploy AI at your company.
PAC Framework Matrix
Rob: Welcome to the AI at work podcast where we take a look at how artificial intelligence is changing the workplace. I am Rob May, I'm the CEO and co-founder of Talla, A Boston-based AI company. I'm an active angel investor in the artificial intelligence space, with over 50 companies that I've invested in, and I'm the author of the "Inside AI" newsletter, which you can check out at inside.com/ai. I am here with Brooke Torres. Brooke, do you want to introduce yourself?
Brooke: I am Brooke Torres, and I'm our Director of Marketing at Talla. I'm a founding team member, and I'm the author and editor of several of our AI ebooks.
Rob: The purpose of this podcast is to take a look every week at AI trends and the future of AI at work. We will invite a wide range of experts in to talk to us and interview a lot of people that we know in the AI space. We're going look at how you can apply AI at work. We're going to look very broadly across the spectrum of AI. So we'll look at technology solutions, products you can deploy, infrastructure that you might want to use, workflow behavior change that might be necessary. We'll look at key news highlights that might be relevant to how you're thinking about AI, and then we'll have special sections like a section called “Ask A Data Scientist”, where we're going to take a lot of these topics that you hear about in the artificial intelligence space and try to put them in plain English for you.
Brooke: We'll also talk a lot about what we've learned at Talla over the last two years, like case studies of the most successful AI deployments we've seen, and we'll talk a lot about some of the misconceptions in this space.
Rob: Now, let's talk about some recent AI news. Today, I want to talk about two articles in particular that I found really fascinating. The first is an article about Google duplex. If you missed the demo, you should go look it up.
The CEO of Google gave a really great demo on their new duplex product, which is a voice-controlled AI agent that can actually schedule appointments for you at hair salons, restaurants, and things like that. When you watch this demo, what's really amazing is that the AI dials a number, talks to a live human on the other side, and does a very elegant job of dealing with all the different things that a human would say-- some of which you would think would be a little bit unexpected and difficult for an AI to deal with. It's really, really impressive, and this is state of the art for natural language processing technology, given where the industry is today. Very impressive work from Google.
Brooke: Rob, talk a little bit more about how advanced this is compared to where we were a couple years ago, because this is really significant and noteworthy.
Rob: What's so interesting about this piece of technology is that people worry a lot about where AI is going in the negative sense. You hear Elon Musk, Stephen Hawking, and people who have sort of sounded the alarm about future of AI, where it could go. The truth is general AI is very hard to do right now, and we have not made much progress. Google has shown that state of the art right now for natural language processing, which is just one piece of AI, is a very narrow domain. What I mean by that is they didn't try to build an agent that could talk about hundreds of contexts, hundreds of topics, hundreds of different types of interactions.
They looked at very specific controlled situations, and they can respond very well to all the variability of language that happens within those situations. Given that Google is one of the top companies in the world, state of the art for natural language processing is to constrain your domain because that is a lot of what makes this Google Duplex product successful. If you are in a space where you deal with natural language processing or you're thinking about deploying a tool that is going to do natural language processing on some aspect of your business, one of the things that you need to grapple with is the trade-off between how constrained the domain is and how effective your tool is at what you're trying to do.
The more you constrain the domain, the better it's going to be. Now you have to balance that against whether you have enough data to do it-- the more you constrain the domain, the less data you're going to have. That's the big takeaway for listeners.
Brooke: This isn't the episode to go deep on natural language processing, we'll do that later in series called ‘Ask the Data Scientist’. But, for those listening, it's worth checking out one of my favorite blog posts by Chris Moody. He’s out of Stitch Fix, and it’s a post on word factors, which is a method in natural language processing. It's called A Word is Worth a Thousand Vectors.
Actually, we’ve got other news out of Stitch Fix.
Rob: Yeah, I love that post too. So definitely check out A Word is Worth a Thousand Vectors, fascinating technology that they're doing at Stitch Fix. There was also a great article in Forbes in recent weeks about Stitch Fix and what they're doing. What is so interesting about Stitch Fix, a company that makes clothing recommendations, is that their model combines human stylists with AI algorithms to determine, based on your style preferences, your body type, and your budget, what kind of clothes might be best for you.
They ship you a box weekly, monthly, whatever interval you set up. The company's growing like wildfire. What's really interesting is so many of the AI companies that are around today are very high tech-focused. They're building infrastructure. They're building new novel applications that do crazy things with AI.
This is a little bit of an older, more stodgy industry of "hey, we're making recommendations". This has been around for a very long time, but the AI that Stitch Fix has brought to it has really shaken it up. They now apply all kinds of intelligent algorithms from not just their recommendations, which is where they started, but across their supply chain and many other pieces of their business. Really fascinating what they're doing. That's it for articles I want to cover. Do we have any other news?
Brooke: I'll share about a conference we've got coming up. I'm really excited about this as it should be a great event. We're hosting an AI blockchain conference, which you can find tickets at aiblockchainconf.com.
It'll be in New York City on June 28. And if you look at the speaker list, it's really geared towards people who are interested in the intersection of the two opportunities-- so VCs, entrepreneurs. And we've only got 300 seats but really powerhouse speakers. So definitely sign up if you're interested in joining us there.
Rob: Matt Turck from FirstMark Capital is actually keynoting that conference. And if you follow Matt at all, I think he's one of the best writers on AI-- really involved in blockchain. So that's going to be a really interesting conference. So let's get into our main topic today. We're going to talk a little bit about PAC framework.
Brooke: This is the framework that Rob came up with a few years ago. And the reason we decided to open our first episode with it is because AI is honestly a really confusing space still. There's still a lot of info out there about it, but this is the best way to get started if you are looking at AI and thinking about how to evaluate and deploy various algorithms or products at your company. So I'll interview Rob about this, and we can get started. Rob, do you want to talk about where the PAC framework came from and why it has been so valuable, as you've talked about it with different people?
Rob: A couple of years ago, I spent some time with Jason Calcanis, who's been an investor in both of my companies. He's a very popular West Coast angel investor, and Jason puts on the Launch Conference and the Scale Conference. I don't remember which one this was for, I think it was for Launch. But Jason asked me to speak about AI, and I said. “It's a broad topic, what would you specifically like?” and he said, “how about something that is very focused on people who are just getting started. What should they pay attention to?”.
So I procrastinated up until a couple days before the conference, like I normally do. And then started talking to people, saying, "I have to come up with this talk really quickly. What do you think are some of the problems people have with AI?"
What came back consistently was “hey, there's all this like crazy news about AI playing AlphaGo and AI beating people in chess and everything else. And it really distracts from people trying to figure out OK, if I run an insurance company or I run a car dealership. Or I run an accounting firm, where can I apply AI to my business and how do I get started?”
Brooke: Talk a little bit about what is it exactly? What does it stand for? What do you do with it?
Rob: PAC stands for Predict Automate and Classify. Now there are hundreds of things that you can do with AI algorithms, but when I put this together, I tried to think about, what are the most common things that AI has been very, very effective at? And these are the things that came back.
AI's very good at predicting things. Given a large data set, in many cases, an AI will be better than humans at predicting something. Automating things is another thing that AI can do very well. So you take a task or a workflow that humans used to have to do, and it was hard because you have to string together a whole bunch of different tools and a whole bunch of different rules.
One of the things that AI can do very well, particularly neural networks, is they can sort of learn these rules and automate these tasks. Classification is probably really the sweet spot for neural nets. And neural nets are what have driven the last wave of AI innovation.
Classification basically says is this A or is it B? Is this a customer who's likely to be valuable or a customer who's not? Is this is this object A or something different? So the PAC framework is the best way to get started, because it sort of simplifies down this large world of AI into a couple things that you could really do for your business.
Brooke: I think a good takeaway there too is that often, we hear this idea or misconception that you can sprinkle AI on anything, and this is really helpful to distill down to what you can do and where you should be evaluating and deploying AI in your business use case. So Rob, say I am an executive at a media or technology company. And I want to get started with this, how do I use it? Where would I get started?
Rob: Well, what I advise people to do is make a matrix. And we'll actually make a sample of this and put it on it the website. But make a three by three matrix.
Across one axis, you want to put the PAC pieces of the framework-- Predict, Automate, Classify. And then on the other side, across that axis, you want to put product, customers, and operations, which are sort of three key parts of your business. And what you want to do is in each box, you want to put things that might impact your business.
Don't think about anything yet except the potential business impact at that intersection. So as an example, you're going to have a square where 'customers' and 'predict' intersect. So you want to list things in that box that might be valuable to you.
What could we predict about customers that might be valuable? Could we predict customer churn, for example? Then let's say you want to move on to automate, and you want to look at operations. So in the intersecting box between automate and operations, you might think what operational workflows could we automate that might be really valuable to the company?
I would get started there, and I would try to put things in all nine boxes. You might not have something for every box. And after that, you have your matrix. I think you want to look at where you have data to actually do this. So, for example, if you wanted to do the customer churn prediction, you may realize you don't have the kind of data set that can do that.
You will cross off any opportunities that don't have the data. That should leave you with some boxes that still have something in them. I think you want to look at the ROI on each of those. Given the data that we have, how easy is access? Again, not knowing how successful the algorithm might be.
If you try it, just make some standard guesses of, is this something that will be really valuable to the organization or only marginally valuable? Would it lead to a comparative advantage? You should think about those points for things that have a flywheel effect. If you're looking at something like classifying something in your product, and that classification made your product much, much better for the customer use cases, then that's a really good flywheel use case where by having better product, you win more business against your competitors.
By winning more business, you improve your data set because you have more customers. That allows you to accelerate that flywheel, and that's going to become your competitive advantage in a lot of AI companies is where you build the flywheel whereby having the most data, you can train the best models, which gives you the best use cases and the best product, which allows you to win business against your competitors, which gives you more data and so on and so on. I would work through it from that perspective, and then figure out what to do next.
Brooke: OK, so you've gone through the PAC framework matrix for your business. You identified the best use cases for you, where it's going to be most valuable, and what the best opportunities are. What should people keep in mind for next steps?
Rob: The main thing is probably just to start small. I think it's really easy to get excited about AI and want to sprinkle AI all over your company. And that would be great if you could do it, but any time you're dealing with something that is fundamentally different, and I think AI is different, the kinds of people that you need might be different.
The way you deploy it, the way you have to train it in an ongoing fashion is different. The way you might think about it and the value that it provides might be different. So I would encourage you to start small.
Pick one project. Roll it out. Don't be frustrated if it takes a while for that project to become successful.
We want to take baby steps here, and we want to do something that gets the organization thinking about AI, working with AI. You're going to have some failures. You're going to have some mistakes.
You can use those to eventually learn the right way to do it, given your industry and your organization. And then you can start to extrapolate from there and apply it in other areas. I do think it's important to get started early, because when I talk about that flywheel effect, that flywheel effect doesn't just exist on the data and product side. It exists on the AI organizational side, which means the organizations that adopt AI earlier are going to understand it better.
They're going to build the tools and work processes and technical talent and business talent to roll it out. They're going to have the proper frameworks-- mental frameworks-- for thinking about how they adapt and apply AI, as these technologies mature and improve. I think that's going to lead to big benefits down the road. I wouldn't wait too long to get started.
Brooke: To summarize the discussion, PAC framework stands for Predict, Automate, Classify. If you're trying to deploy AI on your org or identify where you should be looking to leverage it, this is where you want to get started. Again, you can visit our blog to download the matrix. The expectation, like Rob is saying, is that it's important to deploy early.
The process looks a little bit different than buying or deploying additional software or tools for you, but it's really worth the investment. Rob, let's talk a little more about news from Talla. What do we have to share from our side?
Rob: One of the things we'll do on this podcast, is we will cover learnings from Talla or new product offerings from Talla that we think are interesting. If you're not familiar with the company, we're an intelligent adaptive knowledge base. There's a whole lot of things that we can do.
Typically when you write content, it's not easy to read it into machines without doing some significant annotation. We provide a knowledge base that's targeted towards dynamic content, content that changes a lot, where it's very important that you have the most up-to-date information. We build these machine learning models that look at things, like for example we try to predict if content up-to-date. And if not, who owns it?
We can reach out to them via email or chat, Slack, Microsoft teams or whatever and get them to update it. We can do the nagging for you. We can look at topic modeling to understand if you have duplicate content. We could help you organize your knowledge base. So, think about what it would be like to have a very intelligent digital system that's built around your knowledge base and kept it all up-to-date, that's what Talla provides.
I think the newest thing that's coming out that we've seen is this use case in teams around sales and support. This isn't a use case where these people expose Talla to the end user yet, although it's definitely an option if you want to do that. The more common use case we see is that you have a very complex product that has a lot of information and a lot of different configurations.
Your salespeople typically can't master it in a short time frame, so you want them to be able to ask direct questions from content and make sure they have the most updated information and get direct answers. Particularly if they're on the phone with a client or potential customer or prospect, you want to make sure that they don't have to put that person on hold for a couple of minutes while they go search through 14 articles that were returned via keyword search, or worse-- stop the call and say, "hey, we'll email you back".
This actually started last year with the credit union that we had sign up that had this use case where they had all this complex credit information, and the reps needed to know the most specific things. We were able to take a lot of their documentation about their products, about the interest rates, and about the different options that was embedded in Talla. And we were able to make inferences about that to answer direct questions from the reps when they're talking to a customer. That's very different than doing a keyword search and just returning a bunch of matches.
Much of the time, we can get the exact answer that they're looking for. The times that we can't, we either fail over to a human, which is probably what you would have done before, or we can return a generic search. There's a whole training loop through there where that gets better and better.
And so that module, the sales and support use case module that overlays with the knowledge bases, is coming out soon. If it's something you'd like to try, please reach out to us. We have a few more beta testers while we're ironing out the final bugs for that.
Brooke: I think part of what's exciting about a lot of what AI, especially natural language processing, can do is you think about your business data and where a lot of really valuable information lives. It's in documents or natural language. So it's in your knowledge base, or it's in emails or chat. What we're talking about here is something that you can take that information and make it more intelligent and valuable to everyone in your organization. I think it's good stuff happening in the industry and obviously here at Talla too.
That wraps up our episode for today. Please subscribe or share it with your friends, we are looking forward to seeing you next week. As always, if you have feedback or questions, we'd love to hear from you at firstname.lastname@example.org or tweet at us @talllainc.