How the World's Largest Retailer is using Bots & AI with Kunal Desai: Chatbot Conference 2019
Hello everyone. How are you guys doing? so, I'm Kunal this I am part of Walmart laps. I'm super excited to be here today to talk about chatbots because I love chatbots the two reasons. I love chatbots. Right one. I love is first as a user. I can type in anything. I want from a chatbot the if the bot understands it would respond back if it doesn't understand it would say I'm still learning, but that's great. Unlike my wife she will always react to my question.
So, it's really good to have a bot. Hey and the second reason why I like Bots is as a product manager as a business owner if you can train the Bots correctly it can do a lot of it can solve lot of problems for your customer and make their life better and as a business owner it will help you. He's a lot of your pain. so, those are my reasons for the bots. so, today my topic is chat Journey at an omni-channel retailer so, let me talk little bit about the company.
I work at Sam's Club, which is part of Walmart as our CEO likes to say we are fifty-nine-billion-dollar startup with in Walmart. We have almost a hundred thousand Associates and we mainly do on the wholesale side. so, we are bulk retailer and we have close to 600 clubs. so, from our members, what we are is they are families a large family. They might al so, be organization small business organization from restaurants to Care Organization. so, those are our customers and that having so many stores, having so many millions of our customers and we have thousands of products.
They do have questions for us. And so, they show at a retail store. They come and Shop your eCommerce site. And these are three of the channels they contact us and this was just a set the stage. This was last year before we introduce pot so, we had mainly on the channels as calls chat and email. so, what was going on in the marketplace was retail was going for us, which is great Ecommerce was growing at around 30% which was even better. But that's a great thing from a business perspective but for us, it was a problem from because I lead customer service from a product perspective and we wanted make sure that it can't impact the cost at the customer center. so, the question was how can we manage it? so, we said, okay, let's look at some of the technology available. And this was 2018. think that was one of the first year in the Custer sir, customer service domain where voice was less than 50% from a context perspective chats and SMS messages was taking over. so, we said okay, let's look looking to bots. so, we started our journey into Bots.
We started the discovery process and we started with a very aggressive Vision. It's like we improve our customer experience by answering 90% of the questions by an automated bot. This is like, okay. so, will Embark upon this journey. so, the first step we took as part of this journey is, we had lived a live chat. so, we looked at all the data coming from live chat. We kind of started understanding this.
What is the top We typically do from a product perspective will say okay? What are the 80/20 rule what are the top issues that we have? Let's try to address those and as we started looking into that we identified that if You Buck at the top issues most of the solutions for those issues are on the site. And so, we said, okay, let's make a very simple bot. And allow whenever a member has a question will map it to some of the links on the site. so, with that. We created. We created very simple and easy to use.
What we'll do is a question comes in. Let's say password reset will direct them to the site if there's about membership. We are a membership organization. so, if membership comes in will direct them site, that's what we did one. Good thing we did as a team was for Club hours and locations. What we did bot. is we took that data and put it in the so, if the question comes up, okay, where is a is a club near Concord will give you where the location is and it will al so, give you the so, that's that was one clever thing. We did and one in order to keep the csat score good. We wanted to make sure that if there is a question coming in will direct them directly into a custom region. We don't want to be a friction board shouldn't be a friction to start with. so, those are the three main goals we started with but as we launched it what we realized is. As many people like me they would go home come into the board and answer I put in text anything under the sky and it became really difficult for our bot to understand what it is. so, that converted low confidence responses and it went directly to our customer support agent it the conversion funnel became come to the health center open the board and went into the live agent. so, nothing much was helping out there. So, we said like what should we do next? We started filling the onions and kind of understand understanding. What's the detail around it? so, we said if the board is able to understand the intent the response has been really good like the club hours and locations. We had a great feedback from the from the community. so, it was really good. But for others where the board couldn't understand the intent, we had a problem. We said, okay now bought how to make the Bots understand the intent so, that it can give a better response. Started backwards. so, like bought understands intent, we should then go identify how customer would ask those intent which are bought understands and then to make sure that customer asks those interns we said like we need to maybe guide them through that intention intent process. So, with that what we did is so, what we did is like we want to make our Bots life easy like we need to work make sure it can understand the intent. so, we started with the guided discussion, so, what happened? Is this really help the customer? They will select the orders if they like the if they were looking for orders, then they'll come to the second menu and check for order status and will show them the order status so, that that really worked for us. And for this the adoption of the feature was really good. It was almost 7 to 75 percent people started clicking those guided selection rather than answering the question at bottom. so, that worked great. We still had an option to still have a someone can offer question. Now what happened here is we change the top of the flow where how to identify the intent. so, that was great. However, the bottom of the flow which was basically navigating to the side that remain the same. so, there were still some of the questions were pretty generic. Okay, and so, that al so, again resulted in the customer support interaction. so, we again start thinking what should we do next? How can we kind of stopped at? This is one of the conversion flows where you want to block members going to the next step, right? so, if you ask question try to re solve it right there, so, we said okay, we need to understand if a member is coming to the sun coming to a board. They have a problem. They might have tried something on the site might not have worked and that's a result. They are here. so, we need to solve it for them. We Looked at talk to our customers support agent and kind of understood what they do when the question comes up. Like what is my membership? When is it due? What are they doing for that? And we went, we saw that the customer doesn’t have an awesome tool set? they have now to get the membership information how to get order status information and many other functionalities like okay, we are and we are bought we are within the Engineering Systems we can help re solve that so, with that what we did is we thought okay, let's make our bot intelligent. And what we did is we started integrating into the order management system. We started integrating into the login system and the membership system and what that helped us to do is If there is any question coming around, let's say the membership renewal part. We want say go to the side. What we'll say is. Oh, your membership is renewed on May 11th, 2020. See we started getting all the answers. I think he was mentioning earlier. We kind of started moving the automation part of it and that kind of get into the board directly al so, wearing the human had we started thinking about personalizing readings and responses. If the user was logged in then we'll give the answer right away if the user is not logged in, we Start asking questions. So, we said that okay, if you want to know the membership renewal will ask them question around what is the membership number and what is a ZIP code or something to identify that and then we'll give you that information. so, that's the other one more thing. We did was I think that works really well is people need to use their membership ID for various reasons at the club and they forget it's a 17-digit number so, they do forget which is obvious. so, what we said if We can we will ask them again authenticate them and then email them that information so, that they can have it in their in their hands at some automation. We did. so, with this there was confidence within the management to kind of say. Oh, this bot is working. It is helping the customer solve the problems and it is available 24 by 7. so, that helped us opened up it became the primary channel on the health center. We went a hundred percent life on production. But what happened is when we kept asking some of those questions, we had a set pattern will ask the membership ID will ask a zip code and if someone missed one of that information will be blocked as like what can't do anything and will say I'm still learning. It's like, okay we need to fix that. so, it's like how can we go ahead and make sure because the answer is available, but the intent is not coming through or the verification in this case was not coming through. so, like how should we solve it? so, we what we did is we went in and started working again with our agents and he said like do you come across this scenario where you are? Let's see. someone is coming from an order status. You ask them an order ID. They don't have an order ID and they you ask for a zip code and they don't have it. Like, what would you do next? It's like we have four or five questions. We asked to verify the verify the member. It's like okay, why can't bot do that, right? So, we started thinking about Out make the board behave like a human, So, in the next step what we started doing is we will if you al so, check order status will say what is your email address? If you don't have it or some then we will ask. What is your membership number? Okay, if when you don't have your other information basically the idea behind that is will keep asking you the information and we'll keep moving it one thing. I'll touch upon here is the other way that He helped us is we looked at when we started working with other agents. We looked at scenarios that we can solve on the site scenarios we solve so, we can’t solve on the and the Agents can solve it and then he's like, okay, we to figure out and go in and how we can solve problems that agents do it, so, we see there's a flow where we need to ask multiple questions to solve it and there is no good UI to kind of solve it on the side but a conversational a you I can definitely solve for it. so, we started working on That just looking at the time I'm going to skip that but that's that was a big use case. so, focus on things. That an agent can solve but it's not available on the site. Okay. so, what one thing the 80/20 rule that we typically apply from a product perspective doesn't really apply in the customer service because it's a human right? They have lots of questions. They have different types of questions. They will ask in is it's a language they will ask in different type variation. And so, we need to so, we could only solve by looking at the 80/20 rule only maybe I would say half of it a little bit over of that like we still have a long tail that. Need to solve for and for that we need to have an active learning part. so, and the active learning board is would help us aggregate. What is the missed information and with a human would come in and then try to tag those Miss skills and create new s
kills often? And I'll go with that on my seconds last slide so, so far. We have more than 250 skills. We have half a million questions. We have answered and the rate we are going will should be double this by end of the year. And but the big impact is the adoption of Bot has been around close to 21 percent that is decrease the percentage points on the other channels, which is 12 on call 5 and chart and then for person on email so, I'll leave you with this quote from our founder as we have off. Done in the past as a new trend in retailing develop. We want to be one of the first to test it try it and see it if we can do it better and that's what we'll be doing with all the Technologies at Walmart. Thank you.
Thank you. We got to move to our fireside chat. so, we got three questions first three hands up one. Two three hi-yah. Thanks for your presentation. And I like your foot that you're trying to make the bot intelligent. I have one question about the training data. so, I'm asking the number of skills that you were talking about as possible because you have some kind of an efficient method of organizing your intense, correct. so, what framework or what? Knowledge you do you use to organize them. Do you use ontology or what is it? Not so, so, what we do is on the back end for us. We have we started using Raza as a framework and then we have data scientist who built on top of it with open source model. so, we they have made built-in in-house tools that allow us to aggregate some of the missing skills, misclassified skills, and we kind of then look at those skills and start mapping it but so, it's we use Automation for the aggregation part but then we have human too kind of math do the mapping. second question Hello a few questions. so, the oh well, let me pick the most important one than so, if you skip so, what's your email address and we're checking order status. What happens like, where do you decide? Okay, this is the last time I'm going to give an option Escalade sure. so, depending on the type of the question or the information the member is looking for we have four or five guidelines. What those questions we can ask to the to the member? And so, if we exhaust those then we kind of end up going to the agent, so, that's how we kind of identify that okay. and last question two-part question. Sure. Are you running this in the cloud or on-premise? And then how do you handle the integration to the systems of record that feed the but I didn't get the second part the integration to the systems of record that feed the bot. Okay. Sure. so, the first question we are on cloud so, we are not sure that's why we are and on the second part of it. Yeah, so, that has been the integration of bought into the ecosystem. some has been one of the challenges we had so, we do integrate like from a system of record. We do integrate with membership. so, we will directly if things change, we are going to make the change of their we do integrate with other management system where we pull that information, we all we do integrate with returns. so, depending on the system we will have we have a because we sit within the environment we can go ahead and update it if required. Thank you rattling sound Runner Applause, please.