Episode 2: Chatbots for Student Success
In this episode of AI at Work, host Rob May has a conversation with Drew Magliozzi, CEO of Admithub. Tune in to learn more about Admithub, including their natural language processing technology, the chatbots that they use for student success, and lessons that Drew has learned on his startup journey.
Rob May: Welcome back to the latest episode of AI at Work. I am Rob May and I am normally joined by Brooke Torres, she is busy today and was not able to make it. This podcast is about things that are happening in the artificial intelligence world, and the world of work-- how they are transforming each other, and where we're going. Today, I have Drew Magliozzi as my guest. He's the CEO of AdmitHub, where I happen to be on the board of directors. Welcome, Drew. Do you want to give us a quick overview of your background and what AdmitHub does?
Drew Magliozzi: I've been working in education technology my whole career-- first at a tutoring company, then in the nonprofit education space to help college students collaborate and learn, and now at AdmitHub. Turning the power of artificial intelligence to higher education is something I'm really passionate about-- democratizing, bringing access to information for all students; and, helping colleges dramatically improve their operational efficiency.
RM: We'll talk a little bit more about AdmitHub in a few minutes, and particularly, their natural language processing technology, the chat bots that they use for student success, and some different lessons that Drew's learned on his startup journey. We want to talk about a couple of other things to get started. First of all, I recently finished a book called Prediction Machines: The Simple Economics of Artificial Intelligence, by Ajay Agrawal, Joshua Gans, and Avi Goldfarb. This is a fantastic book if you're interested in where artificial intelligence is going. I want to read to you a couple of things that I highlighted.
DM: Oh, this is great. It's like second grade. It's after recess. I'm going to put my head down.
RM: (Laugh) The general thesis of the book is that AI is very good at predicting things. AI can do a lot more, but most of the models that we use today, what they ultimately do is predict things. What do you do, when production becomes cheaper? What happens to the world?
There are a couple of interesting quotes. One is that, "Not only are we going to start using a lot more prediction, but we're going to see it emerge in surprising new places." If you're a big company, or if you're a startup, one of the things that you want to think about, with respect to artificial intelligence, is what kinds of things can you lower the cost of doing by lowering the cost of prediction? And then, where does cheap prediction change your business in other places?
In terms of defying prediction, there's another interesting quote from the book, "Prediction is the process of filling in missing information. Prediction takes information you often have, called data, and uses it to generate information you don't have." That's a really good way to think about it, and some of the things that will tie in later to our discussion about AdmitHub.
How to know if your organization is AI ready are also touched on in the book. One of the quotes is that, "the ability to see a problem and reframe it as a prediction problem is what they're calling AI Insight." A lot of what you're going to want to do if you're deploying AI in your organization is take the problems you have and try to figure out how you restructure them as prediction problems.
My favorite quote from the book is, "Some AIs will affect the economics of a business so dramatically that they will no longer be used to simply enhance productivity in executing against the strategy, they will change the strategy itself.” It's really interesting to see if you should think about your company as a prediction company in some way, shape, or form. I guess a lot of it depends on what you do. I know you haven't read the book yet, Drew, but any thoughts on those quotes?
DM: That last one, particularly, is interesting. We like to say organizational change is a byproduct of what we do. We never lead with it. Selling to someone and telling them it's going fundamentally uproot everything about what they do on a day-to-day basis is a tough one. We see it almost every time, when we're six months or a year in, how the patterns of behavior are changing in the office once certain things become automated as they turn their attention to some of the higher order stuff that they now can pay more attention to. It's really interesting, and it's a benefit that is hard to understand until you experience.
RM: Yeah, and do you see this in your data at all, that there is an importance to adopting AI early? It does take a while to learn and understand because these deployment models, workflows, and technologies are new and different than the stuff that's come in the past. Companies that adopt it today versus companies that start to think about this in 18 or 24 months, do you think that there's a cumulative advantage that accrues from deploying it?
DM: Yeah, definitely. I mean, we see it in what we call the automation rate. Just the number of raw unstructured questions that come in, how many we answer correctly. It rises pretty dramatically. It usually starts out with a partner just over the 50%, 55% mark, when you're launching. Then, within a year, you're at between 70 and 75%. It's mostly about gathering the label training data through the process of rolling it out. It's certainly astounding and will continue to grow. Not unlike the strict numerical analytic space, it is just building the habit around working with an AI.
We like to say you're not buying our software, you're hiring it, and having an AI coworker is probably the biggest mental barrier to successful adoption. That's one of the wrinkles that I don't know if we've fully worked out. I should say that we build software as a service that's a chatbot for universities to deploy right alongside their enrollment office. The chatbot is among peers, there usually around 30 or 40 other people doing a similar job, but it obviously is not quite as good as a person for most tasks. The idea of branding it as the mascot of the university was a very deliberate one that makes people feel a warm sense of fuzziness toward their friendly neighborhood mascot. In a sense, it's a little bit like the psychology of having one of those Tamagotchi toys, I'm dating myself, when I was growing up. It's a tender affection they feel toward this virtual mascot that's helping them every day.
RM: So you deploy this at ABC college, and the students now can ask the ABC college bot questions as they're going through the enrollment process, or after they've been accepted and they're thinking about coming to campus. One of the problems that you guys are solving that I think it's really interesting and was not aware of is this problem of summer melt. Can you talk a little bit about what that is and how the chatbot helps with that?
DM: Yeah, I mean, I think it's pretty much standard for any business, you just have a leaky funnel. People apply, get admitted, and say, I'm coming. I can't wait to be a Panther! They usually say that in the spring or early summer. By the time the fall rolls around, nationwide, about 14% of college-intending students fail to enroll. At some schools, it's as bad as 30% or 40%. So, we started focusing on this problem with large state schools who usually experience melt between 16 and 20%. One, it's lost revenue for the university, and, two, it's lost opportunity for the students. So, it's like a double bottom line impact we can have. And we were able to move the needle pretty significantly, about 21% in a randomized controlled trial. It's pretty astounding the opportunity that's there. We're not going to take anyone from a hard no to a definite yes, but there are a lot of people on the fence. They just need a little bit of a nudge here and there. Half of our system is proactive, where we're doing the outreach and telling you the things you need to worry about. And the other, the yin to that yang, is reactive. When we say you have to file the FAFSA, and you ask, "what the heck is that?" we'll be able to tell you and quickly get you back on the rails. It's not a perfect conversational system, but it's pretty darn good at doing what it needs to do at the time.
RM: A lot of people listening to this podcast have probably had bad experiences with conversational systems. I always try to explain that there have been a lot of new natural language processing techniques that have come up in just the last few years; but, this technology is getting better at a faster rate. I think some of the older companies that did this haven't kept up. I think there's a lot of opportunity with new tools like yours and the stuff that we do at at Talla. Tell us a little bit about the on-boarding process. So, I'm going to buy AdmitHub, and I want to roll it out, and I write you a check. What happens next? What kind of data do you need from me? How does the machine get trained?
DM: First of all, I send you an animated GIF of Tina Fey high fiving herself to celebrate. Then, the key is no matter what, whenever we do a deal with an economic buyer, you've got to redo the deal. You've got to win hearts and minds, essentially, at all levels of the organization. This is really a touch-and-go situation, this technology-- I'm sure you guys see this as well-- where people have a great deal of fear about it. Now, it's not like Terminator or 2001: A Space Odyssey fear, yet. It's a fear of: what is my relevance?
Some people honestly worry that they'll lose their job from bringing automation to the workplace. They quickly understand, once they see it in action, that it's not a threat to them; but, to some degree, it does threaten their patterns of behavior of what they expect will happen when they walk into work in the morning. Luckily, we're taking away the overflowing email inbox that they have by managing and triaging 60% to 70% of the stuff that comes in on a daily basis. That is a pretty joyful experience for them. Then, what we really have to do is help them understand what the heck it is that we're doing, how we're doing it, and why it's a benefit to students and them. I think there's an on-boarding process that is more psychological than it is logistical. The tasks are easy. We need these questions answered. We need to know your deadlines, and what your goals and objectives are. That process stuff will happen, generally speaking, relatively easily. But, it's the psychology of on-boarding that is the pickle for almost everyone.
RM: Yeah, it's interesting that you talked about how people look at this replacing their jobs, potentially, because one of the things that came out of this book Prediction Machines that we talked about, is that when you lower the cost to prediction, and prediction becomes easier, and, therefore, there is more prediction, the economic compliments to prediction go up. So, what are the economic compliments? One is judgment. Now you have all these predictions, but even if you could predict something, okay, what do you do now that you know the likely outcome? That still requires human judgment.
Best I can tell, machines are still a long way from understanding that for a couple of reasons. They don't operate in the physical world like we do, and they don't have the ability-- we're probably quite a ways away from melding different kinds of things there. I don't think people should be afraid of the machines in the near term. I give the analogy to people: think about a construction worker who can wear an exoskeleton, and now lift 400 pounds instead of 100 pounds, and do so more safely. We are about to go through a cognitive revolution that's very similar. People are going to be able to do their jobs better because the machines are going to take away a lot of the grunt work. I think the value of human labor is going to go up.
DM: But the skills required to operate a crane, for instance, or drive a dump truck, or operate a backhoe-- sorry, my 16-month-old son is always pointing to things, and the vocab is there, but-- I mean, that's a particular skill that needs to be acquired. I think the same holds for operating an AI, being able to adequately supervise, and shepherd, and train, and grow the system, likely, is going to be its own specialized skill that not everyone will acquire, but is going to be a vitally important one.
RM: You guys are a couple years old, how have you seen the reception in your sales process from the early days-- were you getting a lot of tire kickers who weren't serious? Were you having a lot of deals fall through early on for various reasons? Or were people tremendously interested but didn't know what to do, and how has that interest in it changed? Is there more or less interest than there was two or three years ago?
DM: I've learned a lot. Lesson number one, don't say convolutional neural nets ever, that will make people flee from you. It's been incredibly lucky that we've seen our technology get better, and I should note that we own our entire stack, which is a benefit. If you're not using something off the shelf, you can control a whole lot more. The interactions between the state machine and the NLP system are key and being able to manage both is important. But, in terms of the sales process, we've seen the tide turn in our space. There are a lot more people interested. Now, actually, we have the opposite problem than we had 18 months ago, where we have too many inbound inquiries and are parsing out who's serious and who's not serious.
Actually, we've had this full revolution in how we sell, where we were afraid to say AI because it would scare folks away. Then, AI became the thing, everyone was saying, “I want the AI” and we're like, “What do you want it for?” and they're like, “I don't know. Why don't you tell me what I want it for?”. Now it’s getting back to the basics of selling. Tell me your problems. Let's talk about you. I need to understand you because this is an incredibly powerful tool, and I just need to know where to point it. We can impact whatever you need impacted in all likelihood, but I need to learn so much more about you, and then I will tailor a demo exactly to what you're looking for. The evolution is there, there's this wave of interest; but, interest doesn't usually translate to revenue, or very rarely it does. Figuring out a way to deeply understand their needs is kind the daily practice for us.
RM: We see that a lot at Talla too. People come in, and you get on a sales call, and it's like, “Okay, well, why did you ask for a demo?” And it's like, “Well, the CEO said, we need to deploy AI.” Then it’s, “Oh, great, well what do you want to deploy AI to do?” and the answer is, “I don't know. Tell us what your thing does, and we'll see if that's an AI thing that we can use”. We try to internally talk about which companies are AI-ready and which ones aren't. So I'm curious, what have you seen as some criteria to show which organizations are ready to adopt these tools?
DM: The number one criteria is having a problem that they are desperate to solve. That is always an important one. That's why summer melt was an easy entree into the industry. They hadn't solved it. It was costing them many millions of dollars. We felt like we had the ability to move the needle, and we did. So understanding, do you have an urgent need? Usually the way I like to think of what we do is-- I never really explain in terms of AI. I usually say it's behavioral economics or behavior change at scale. We are able to nudge, convince, guide, and support students in whatever they need to do, really at any time in their student journey, to impact change.
Now, “what is it that you want to change?” is the best question. If folks are able to wrap their minds around that, then we can make a difference. Then, is there a person-- and I think it's really important that we have one key collaborator on-site at the school who is going to be like the Sherpa for their new AI mascot-- who is going to care for it, tend to it's needs, and train it every day, because it's really about the flywheel effect of training that we find brings the most benefit. If you're going to use it, use it to its fullest extent, and make sure you're doing it daily.
It's like we're giving you a puppy, we often say, “Here's this puppy. It might make a mess at first, but I guarantee you, you're going to love it. By the end of the year, it'll be rolling over and playing dead. It will be the most benefit you've ever brought to your office”. The key is, it's a very different type of technology than any other product that you buy. You probably know this. For example, I buy a sports car, and I drive it off the lot, and it starts depreciating every day. The exact opposite happens when you buy AI. It is the worst it will ever be on day one. It is totally underwhelming, and every day it gets a little bit better. And by the end of the year, you've got a Batmobile. Helping people understand that trajectory is really important, and showing them examples of other people who have gone on the journey is key. No matter when you adopt it, you have to go through that sort of uncanny valley of imperfection. That's the challenge.
RM: It's really interesting that there's a training aspect of the software that doesn't exist in other kinds of software to make it better. We were in a pilot process at a large Fortune 500 company. We were walking through it, and their use case was automating some of the inquiries that come into their IT help desk. The guy said, “hey, we'd like to see the software get better faster.” And I said, “Okay, well, let's have more people spend a couple minutes of days training Talla”. They kind of complained. It's like, “Oh, but these people are busy. They don't want to train-- we can't spend five minutes a day doing this, answering questions”. I said, “Well, how long does it take to train a new IT help desk rep?” The guy said, “Oh, well it takes about two weeks… Oh, I get it! I get it.” It hit him that suddenly he had to train the software. It was replacing two weeks of training a person. So, if you have 10 people, and they can do five minutes a day, that's 50 minutes a day. The machine can learn a lot, fast, in a week.
To your point, it gets better, faster, the more that it knows. I think a lot of organizations don't understand that when they deploy this stuff. They try to evaluate it the way you would evaluate SharePoint, or Gmail, or whatever other software you have. They're like, “Okay, well, what it does today is what it's going to do”. Now, I do think the natural language stuff is hard because so much of the stuff that came before our companies was just scripted, right-- query response, regular expressions. For those of you that aren't familiar with it, what would happen is you would build out a chatbot that would basically say, I do a search, and if you match this keyword, then I return this answer. That is not the way that these modern systems work. They use convolutional neural nets. They use recurrent neural nets. They use word vectors. They use really interesting technology. What's interesting is you can do things-- to give you an IT example, I could say, "how do I find a lost Word document?" or, "how do I recover a deleted file?" I've asked roughly the same thing. They might expect the same answer. I didn't use any of the same words, but modern techniques can capture that variation in language.
DM: Yeah, it's marvelous actually. The key when you get into an organization is just getting their lingo down. What words do you use? What are the entities, the keywords, that are unique to you? That's where the learning curve always happens.
RM: This is why you guys are smart to do some of your own stuff because entity recognition tools that are open sourced are pretty bad, right?
RM: They basically look for capitalization, I think. Before we wrap up here, let's go back and talk about an article that we didn't get to that I want to cover in Bloomberg recently. Amazon is known for automation and AI, obviously, and most of what we've read has been about automation in their warehouses. This article in Bloomberg talks about how AI is moving into their headquarters. It's moving into the white collar jobs. It talks about how they have this human team, and then this data science team, and algorithms that did forecasting for what they should order. They decided to merge the teams now, and the humans don't make the forecasts anymore. They're allowed to override them, but they have to explain why. They're relying more and more on the algorithms. I found this interesting because of the discussion that we just had. This work is impacting white collar jobs now. Obviously, Amazon is very forward-thinking in the stuff that they're doing, but this is something that every company is going to have to do. Talk a little bit about the AdmitHub space. You guys have dealt with the summer melt problem. What are some of the other things that you think you might be able to help with?
DM: For sure. I mean, the key ones, obviously, around enrollments is a much easier problem. The challenge we're trying to face now is about keeping students in school and getting them to graduate. In this country, right around 50% of students graduate from college, which is abysmally low. And actually, getting some college is worse than getting none because you're also saddled with enormous debt. We think that's the ripest area for us to pay attention to, but the solution set is not simple. It's more social and emotional. It's more of “life getting in the way” than logistics. Being able to take a student on that journey effectively is going to be a challenge. There's never a silver bullet. It's always the accumulation of a lot of small wins that get to a big gain at the end.
You brought up this idea of robots replacing white collar work. I think that is the thing that threatens people, right? We had the Industrial Revolution that replaced our muscles. This as a revolution that will incrementally make obsolete our minds is astounding. Although it's rare that our technology will displace jobs where we're bringing it, we're usually doing the thing that no one is doing. It's neglected work, rather than obsolete work. There will be a lot of jobs displaced. I honestly think that the opportunity for AI in education is that if we can use the very same technology to dramatically accelerate the learning process, we'll be able to retrain folks, and get them back into the working world, doing the things that are the most productive in needed jobs out there.
RM: Yeah, I mean, you can see a day when AI is helpful predicting that. If your job is being displaced, using AI to predict what you should go do then.
DM: Yes, and helping you get the skills to do it.
RM: Exactly. That's really going to be interesting. I've seen some companies like this in my angel investing that are trying to do AI training-based programs. Right now, corporate training or even college stuff, it's not tailored to you. It's like, you take the Chemistry 101, and then you take Chemistry 102, and then you do whatever. If you could tailor both teachers, and classes, and paths through those things to the student, and how they learn, and what they know, or to the employee and what they need next, it could be really powerful in terms of the advancements that humans can make.
DM: Most assuredly. It's a little ways before we're the AI Socrates or Aristotle teaching you the stuff, but at the very least, the logistics on the administrative side, we can take care of in short order.
RM: So before we wrap up, if people hear this, and they're interested, or they know somebody who is like, hey, AdmitHub sounds cool. I'd like to check it out. What's your website? Where else can they find you?
DM: admithub.com. I also have a long neglected blog, Please Steal This Idea. Although, I have a few ideas in the hopper that could use stealing. Basically, if you steal one of my ideas and make a billion dollars, the ask is that you footnote me.
RM: There you go.
DM: I have more mediocre ideas than I have time to execute on them. I figured that for the stack overflow of opportunities lost, I would share.
RM: Thank you for tuning in, and send questions to email@example.com-- topics that you want us to talk about or address, questions that you have, comments or anything else. You can find us on SoundCloud, and we'll see you next week with a new episode. Thanks for hanging out with us today, Drew.
DM: Yeah, thanks, Rob.
Subscribe and share AI at Work with your friends! If you have feedback or questions, we'd love to hear from you at firstname.lastname@example.org or tweet at us @talllainc.