Episode 8: AI in the Age of the Customer 

Rob May and Brooke Torres with guest Steve Peltzman, Chief Business Technology Officer at Forrester Research. Tune in for a conversation on the future of A.I. in "The Age of the Customer."

Subscribe to AI at Work on iTunes or Google Play

Rob Circle Headshot   

Rob May, CEO and Co-Founder, Talla


Brooke headshot circle  

Brooke Torres, Director of Marketing, Talla 


steve peltzman b&w  

Steve Peltzman, Chief Business Technology Officer, Forrester Research 




Episode Transcription 

Brooke Torres: Hi everyone. Welcome to AI at Work. I am Brooke Torres, our director of marketing at Talla and one of your hosts for this podcast, where we talk about what's happening in the artificial intelligence space, the world of work, and how the two are transforming each other. I'm here with Rob May, who's the CEO of Talla.

Rob May: Today we're here with Steven Peltzman, who's the chief business technology officer at Forrester. Steven and I have been friends for a couple years now across a couple different companies for both of us. He has recently moved into a new role focused more on artificial intelligence, so we're going to talk to him a little bit about that and hopefully we can learn some things. So Steve, welcome.

Steve Peltzman: Hey, great to be here. Thanks.

RM: You're not an AI guy by training. That's not where you got your start. Tell us a little bit about your career and how you got into the role that you're into right now, and what is that role exactly.

SP: Oh, it's a long story. We don't have that much time, so I'll try to be short. I spent seven years doing stealth technology in the Air Force, 10 years doing CIO of Museum of Modern Art-- MoMA-- in New York, where we first met. Then the last seven at Forrester Research here in Cambridge.

RM: You sort of crafted a role, or a persona, in your industry of being a relatively forward-thinking CIO, I think. I've worked on two startups, and you've been an early adopter of a lot of early products. You guys were early in Cloud at both MoMA and Forrester, and everything else. Is that more a personality thing for you? Is that something you learned along your career path? What makes you different that you're sort of constantly pushing on the new tech trends?

SP: A little bit of it is because I'm geeky. I say that as a good thing. I like the new toys and things like that. A little bit of it is, especially at the museum, it was part of the strategy to be scrappy. Instead of going for the big expensive oracles, IBMs of the world, we found was a better path to partner with startups who were hungrier, and more willing to work with us and bend. Then it just became part of the formula.

It was kind of unique, and now what we're seeing at Forrester too is, a lot of CIOs have figured out that there's certainly a place for big, established solutions. But everyone's really interested in finding the next thing that gives you an advantage, because the other part of that is that everyone's doing Salesforce, and IBM, and Microsoft. If you do what everybody does, you have no advantage. Everyone's trying to get an advantage. It's about finding the right tech. AI, for us, was a natural outcome of trying to differentiate.

RM: That leads really well into my first question here, which is, this current wave of AI started a few years ago. Obviously, it's technology. It's been around for decades. But the deep learning, machine learning wave really started in maybe 2014, 2015, and really took off in 2016. When you're looking at something that's new, something you don't fully understand, tell us a little bit about your process to dive in and understand and figure out the opportunities at Forrester. How do you start to work that into your planning as a CIO?

SP: It wasn't like we were sitting there and said, “Oh, AI. How could we use that?” It was more serendipitous, or whatever, than that. It was that we were looking at our own value proposition and the value experience we were offering clients, and we realized over time that what we were offering was called a library card. We offered a lot of products. We offered data consulting, events, leadership awards, and peer-to-peer networks. But our core, value proposition, is research. It's kind of like you buy a seat. That seat has access to all the reports that we write, but people evolved. They're not reading anymore. What we saw was that people were just engaging less and less, and that kind of value model.

For many years we were innovating. Can we make a report better? Finally after a bunch of innovations that didn't move the needle, it struck us that we were telling everybody else, you have to reinvent your business. You have to use digital and technology to reinvent your business. We finally said, “What does that mean for us?” It was part of a journey where “Hey, what does reinventing Forrester's value proposition look like?” You're kind of immersed in it. You not only see, and hear, and read what the analysts are working on around the company. You also see, and read, and hear what all your claims are doing and the revolutions they're working to make. It was kind of like, “If this is what our clients are doing, what does that mean for us?”

We hired a couple of analysts. We got a good rate, hired a couple of analysts to engage with us, educate us about what it is, what it could do for us. We started mapping out what AI could mean. It was such a perfect match. I mean, instead of this idea of offering a library card waiting for you to come to us, we could learn everything we could possibly learn about our clients, their industries. We could get real time data. We could put that all into an AI soup and actually come out with proactive feeds to our clients, and act like the virtual research department that you just hired, and give you really hyper-personalized bits of information.

That's the vision. And with our data, instead of offering data for you to figure out, we actually changed the name of the department to Analytics. Now we can offer insights and use AI instead of people mining through this. We can use AI to correlate that real time. It was a natural thing. Then we started doing some pilots and learned a lot from that. Now we're actually maturing to the point where we acquired a company to move us faster in this space.

RM: Awesome.

BT: You mentioned working with startups earlier. A lot of venture capitalists complain about how many products are built as AI don't really have any. What are you seeing in the market in terms of the products? Obviously you've acquired something that you've demoed or explored. Do you feel like AI's living up to those promises?

SP: Yes and no. You see it living up in many places, but you also see a lot of companies who come to you with this great, promising technology, but they're overselling it. They're selling the promise of what it would be if everything worked perfectly down the line, if your processes and your technologies integrated really well, if the change management has happened. What I'm saying is, AI's but a piece of the puzzle in most of these companies. You have to figure out how to apply it. You have to figure out how to train it. You have to figure out how to incorporate that in the processes, and the technology, and the culture that you have as a company. Otherwise, it's shouting at nobody. And you still have horrible customer experience, or horrible value proposition...

RM: Do you feel like when you are adopting an AI-driven tool, the workflow behavior change is harder? We were talking before recording about getting people to do the training, getting them to understand the different stuff they have to do, getting them to understand probabilistic outputs? Or, is it the technical implementations, the integration with existing systems, and all of that? Are those similarly challenging, or is one worse than the other?

SP: I mean, we haven't really experienced the second one. It's mostly been the first one. Everybody knows that you've got to train. You've got to have great data and you've got to have great training. The training isn't something you could just pass off to a grunt worker somewhere. It's something that, in many cases, takes a lot of insight to begin with.

We had a couple of experiences with startups where they said, “Hey, let's pilot it.” So, we pilot it. We get crap results. Then we get the answer, great. You can see the promise. Pay us $150,000 and we'll work with you for the next four months to train this, and it'll be great. That's missing part of the puzzle that a lot of these companies haven't figured out. One by one, now we know what questions are to ask when we're gauging these companies. What's the training like? Tell me about that. I almost feel like there's a market for companies that help you train. Maybe there's solutions or, as we were discussing before, maybe the right solution isn't an extra effort on the side to train the AI tools or solutions, but it's something that must be incorporated into the processes of the company on an ongoing basis.

RM: It's interesting, if you look at the Talla knowledge base product, one of the challenges that we had was to really make it effective, and really take advantage of the power that makes it different from everything else that's on the market. You have to do some annotations, some training of the data, different stuff like that. A lot of our work has been around, and I think this will be the next phase of enterprise UX/UI design, how do you design your interface to capture that while people are working?

So that  it's not a separate process. It's not like, let me go do my training. It’s part of my normal workflows, how do I train stuff? That's helped us because we had deployment problems too, in the beginning. And we have a model now where we can tell people, look. This works. Just plug it into your workflows, and it just starts gathering data, and building knowledge base articles, and answering questions for people. It just grows really quickly around all your common things. You eventually catch the long tail, and it can be pretty powerful. I think that sort of onboarding and training piece is something a lot of AI companies are still falling pretty short on.

SP: Me and my folks were talking about that, coupled with, how you do these pilots. The point I was making in an earlier conversation today was, to make an analogy here, it’s great that Tesla is trying to throw cars on the road and have them autonomously drive. That's awesome. But, why didn't they start with the three-block shuttle ride from the train station to the Forrester building, which somebody does every day, where you know the road? Why isn't that the first thing that companies like that went after? Why did they go after the big challenge?

I think that the analogy holds inside of a company, maybe you don't go after the big change. Maybe you try and find a pocket, and then work that into it. Work the process, figure out how to do what you just described. I'm not really sure, but we were talking about how everyone's going at the big things, and they should be going at the smaller things.

BT: What other advice do you have for executives who are getting started in AI, other than starting with something small? Maybe you're managing expectations around data sets that you need, high-quality data, time to train, those sorts of things.

SP: Besides the stuff that we said, I think getting smart people. By hook or crook, get somebody in the company that really knows what they're talking about. I don't know what I'm talking about. Certainly, my executive team aren't experts in it. Even some of the experts aren't experts in implementation. For us, that was one of the reasons why we were really behind this. We acquired a company called GlimpzIt out of San Francisco. There were many reasons, but one of the reasons was we need AI expertise in the company, embedded. It needs to be part of who we become.

Let's start there. Not every company can go acquire a company, but you can hire people. You’ve got to make the investment, at least in a small way, of getting AI expertise in-house, rather than just dealing with companies. We covered the other areas, which is start small, try something, learn. We did two or three pilots, and learned so much through very little progress. We didn't make a whole lot of progress, but we learned a lot. And that's okay. That's progress. There was a lot of things we didn't launch from those pilots, but we got a lot of knowledge, and a lot of experience, and now we know better.

Try and fail, I guess. Hire, try, and fail.

RM: One of the things we talked about a lot in this podcast is, compared to previous technology waves, it used to be okay to be a late adopter of technology. In fact, if you were going to wait to roll out the latest version of Windows, or whatever, it was going to be higher quality then. Now you're looking at this scenario where AI is learning on your data. It's becoming more personalized and customized. The longer it's been there, the more it's learned, the more it's been trained. Do you think that companies that are slow to adopt AI are going to find themselves in a position where they can never catch up? Do you think it's cumulative like that?

SP: I do. One of the things that Forrester talks about a lot is the age of the customer. It used to be that if you just made great products, you were great. If you could just get them out, you were great. But now, people will talk about you. If you're not solving their problems in real time, if you're not creating unbelievable experiences everywhere they go, then they'll drop you in a heartbeat. Switching costs are so much lower now across the board.

There's all sorts of reasons why the age of the customer is really powerful. My boss, George Colony, founded Forrester. He says in all his stump speeches, if you're in hotels, for example, and somebody staying at your hotel has a problem with the TV, if you don't fix it right there, right then, then in surveys 24 hours later, you’re no good.

You’ve got to fix it right then, right there. That's the expectation.

With the expectations going up, you have to be able to react. The only way I think you can react that fast is with AI and help like this. And then so if you're not-- with as much change as we talk about that has to happen here with the training, and the infusion into the process, and all that stuff, and the culture, if you don't start now, you're way behind. In fact, you're already behind. I feel very behind even though we're ahead.

RM: When you look at some of the use cases where you're looking at applying AI in Forrester, and just general stuff you see in the market, or even in your personal life, what are you excited about? What is something that you wish someone would build an AI solution for that you haven't seen yet, or hasn't been done yet? Is there anything on your wish-list?

SP: You know what just popped in my head was something I tweeted about yesterday is that someone created an AI solution to find Waldo. That little hand that goes, “There's Waldo.” (laughs) So, that's not really what I want. For some reason it just popped in my head. I don't want the Waldo thing, although I'd love to actually see it work.

Every time you call up a company for help, you have to go through and start the process over again, I don't know about you, but I'm like, “Why can't they know immediately what my problem is because I've called three times before?” When I'm on vacation and I forget to tell my bank that I'm on vacation, and they give me the alert about freezing my credit card, “You're on my phone, you've seen my calendar. You can't figure that out?” I mean there's all these little moments. Before we figure out the big moments, iron those out, because those are easy. There's all these little customer experience moments in your life. You'd probably think of a hundred more where you're like, why didn't they know that? If I'm calling and they know my number, why can't they figure out the product I own immediately? That's not even AI. That's just great data management. You could infuse AI in that. We're talking about customer experience, mostly. But I think there's so many different places you could apply it.

RM: I mean it could really lead to the trend of mass personalization. You could have the Netflix-ivization, if that's a word, of everything.

SP: Long ago I talked about this in some speech, do you remember the Minority Report movie? I mean that's a bleak future, but there's aspects of that that you legitimately want. As soon as I walk into the store, I'd like to know where the clothes are of the size I have. I don't want to have to go through it each time. I guess there's going to be an ultimate trade-off for privacy that we have to resolve as a society. But proven time and time again, if you could give value, people will give up privacy.

I talked long ago about who knows all your purchases best. To me, I think that's probably your credit card company. Where's the credit card where I sort of yield, well I shouldn't say yield my privacy. How about I trust my privacy with them. They are giving me great deals because they see my buying patterns, and they just know that they can do better. They know a company, that can do better. There's a bunch of things. I think it comes down to, not Minority Report, because that's gone way too far, but with all this AI, the companies that can produce the greatest value, yet make me feel like they are respecting, protecting my privacy, are the ones that are going to win. I think privacy's got to enter into this because AI is all based on data. You’ve got to be able to trust it.

RM: I want to hit on something you said and ask another question, which was, you gave the example of going into a clothing store. Do you think there's a scenario where AI is too truthful and causes problems? So you're like, where are the medium t-shirts? And they're like, Steven, you need an XL.

SP: They're seeing that in dating apps. I just saw something that says, if you're not doing well, you're probably lying on the form. In fact, I had a friend trying to do a dating business, but it's not based on you filling out the form. It basically figures out who you really are using AI. It matches up with who you really should be matched up with, not who you think. They say they can guarantee better rates. Of course they can, because people are full of shit. Let's face it. They're full of it. I think that truth, to an extent, is what's needed sorely.

RM: So you go to this AI version of Tinder, and it's like, “don't even try to swipe right on this one.”, “Sorry, way out of your league, way out of your league. Next.”, “You swiped left, but you should seriously reconsider.”

SP: That's where they were headed with their product. Not just the real data, but these weird correlations. We started to get fascinated to your previous question with, AI to predict what you should do. So there's a company I just saw where they're turning around great revenues for their clients by figuring out their supply chain or other things that they're trying to optimize, maybe the distribution of product, or whatever, and then correlating that with every bit of reasonable set of data that's out there, even some reasonable sets. The AI is telling them how to optimize in ways that the human mind would never think of.

How is that going to be with people? Maybe it figures you should wear a smaller shirt because it knows you're going to get embarrassed and lose the weight. I don't know. It's going to give you answers that you haven't thought of, that don't make sense, I guess. We're thinking about that with customer experience. It's one thing to tell you, okay, if you're a home goods store, your biggest issue is checkout, or inventory, or whatever. But, what if we were able to tell you that, if I were you, I'd concentrate on checkout lines at these stores, but inventory at these stores, because I'm predicting that that's more important based on the health of the people in the area? Why would the health correlate with home goods? I don't know. But maybe it works.

RM: What it it's something like, if you're in an area where you have a lot of people who, they drive really far to the store? An extra four, five minutes in a checkout line isn't a big deal, whereas if you're in Manhattan, it's like, come on, come on, come on.

SP: It may uncover things like that that don't make any sense. Or maybe they do make some sense and you never would have thought of it. That part fascinates me too, the predictive nature of AI,  because then you're really squeezing revenue and growth out of something that you never thought you could.

RM: For people looking for frameworks about how to think about prediction, there's a really great book that just came out called Prediction Machines. It's written by economists about AI, and it basically looks at how effectively AI systems lower the cost of making predictions. Places where we have people right now trying to do forecasting and other models, you can offload and automate all that. Well, what does that mean? What things become less valuable now? Then, what things do we predict now that we used to not predict because the cost of prediction's so low? What compliments to prediction go up in value? Now judgment, you're going to predict a lot of things. Now you have to make judgments on them. Machines can't do that yet. The value of judgment goes up.

SP: Think about how many bloody meetings you go through where people are trying to make big decisions-- some based on data, and some not. Imagine if you plug in a system, and it's just helping you make those decisions better and faster. That alone is make or break for companies.

RM: Do you worry at all? Are you a poker player?

SP: Some. More blackjack.

RM: So you know what a bad beat is?

SP: No.

RM: You have a full house, and you bet big, as you should when you have a full house, and you lose. Every once in a while, if you play a lot of poker, you'll lose that hand. Do you worry at all that machines sometimes, because they are going to be probabilistic and predictions aren't 100% accurate, that people are going to learn the wrong lessons? Like, from that poker hand, you can't learn, “don't bet big when you have a full house.” No. Next time you have a full house, you bet big again.

SP: Well that's why you’ve still got to keep people in the loop. Maybe we won't say that in 15 years, and we'll bow down to our robot overlords, or whatever. Until then I think we're going to have to stay in the loop and figure out, “this thing needs more data”, but also have an open mind because in the end, it's going to present stuff that doesn't make sense.

RM: Of course we worry about that.

SP: I used to say something at the museum like, we're looking at technology to enhance, not supplant, the experience, which was something I said to help make everybody there understand I'm not trying to replace the art. That's what we're trying to do. We're trying to actually enhance the whole experience. I think it's the same in business. I don't think it's about supplanting. It might be about supplanting low-yield resources in some cases, in many cases. But I don't think it's about supplanting humans yet. Talk to me in 10 years. We'll see.

RM: All right, we'll have you back on the podcast in 10 years.

SP: I'll be much more eloquent then.

RM: You'll have an AI that'll help train your speech patterns and--

SP: Exactly.

RM: Steve Peltzman from Forrester. If people want to follow you on Twitter, what's your Twitter handle?

SP: @StevenPeltzman

RM: Awesome. Thanks for coming on today.

SP: Thank you. This has been a pleasure.

Subscribe to AI at Work on iTunes or Google Play and share with your network! If you have feedback or questions, we'd love to hear from you at podcast@talla.com or tweet at us @talllainc.