Build a Bot for Actions on Google (Video)
At SXSW 2017, as a part of our session "The History and Future of Speaking with Machines," Matt gave a talk about how to build an action on Google in under 10 minutes. He goes from selecting an API and creating a Natural Language Understanding model in API.ai, to connecting that model to an API. You can remix this project on Glitch today.
We are excited about the Actions on Google Platform for Google’s powerful Assistant. Assistant is available on the Google Home as well as in the Assistant App and Allo messaging app. We are thrilled to be building some amazing bots in the Google Assistant for our client’s products and our own.
API.ai makes getting started with building voice interfaces really easy. The platform has an easy-to-use interface along with one-click integrations to connect your agent to the Google Home, among several other platforms.
Watch the video to see how easy it is to get started!
Transcript
When I was thinking about what to do for this demo, I had a few decisions to make. One of those was what platform I wanted to build for. I decided to go with the Google Home, which we talked about a bit, earlier. I also needed to make some decisions around which API to use. For those of you unfamiliar, API; Application Programming Interface. We’ve been talking about how humans speak to machines. APIs are how machines speak to one another. That’s how different computer systems can share data with one another.
So, I decided to build for the Google Home. When you’re building for the Google Home, you actually get to use this Actions on Google framework. This was released late last year. This allows you to give the Google Assistant, which is Google’s Personal Assistant AI, new abilities – teach it how to do new things. A lot of the work of building voice interfaces, we’re translating human speech into a series of API calls, really. So, I needed to find a cool API, that could do something interesting.
I found this one here. It’s called the College Scorecard. This is a free API put out by the federal government. It gives you information on various public Universities, and it tells you lots of different data, that they’re required to report to the federal government. I thought “Okay, cool. This is handy.” It might be useful for someone applying to college, to be able to say “Okay, Google. Ask college advisor for the acceptance rate at the University of Texas, at Austin.”I set out with that goal in mind, sat down, and got something built really quickly. I thought “Hey, this is really cool! It works.” Now, what I neglected to do was just ask Google the question, “What is the admission rate at the University of Texas?”
When you do that, you’ll see that it answers it for you in this little Answer box. It knows the answer. So, that means if you just ask your Google Home “Hey, okay, Google. What is the acceptance rate at the University of Texas?” it already knows how to answer that question.So, lesson one, when building for the Google Home: if you think you’re teaching Google a new trick, make sure that it doesn’t already know how to do that trick.
I set out to find a new API, and I found this one here. It’s called An API of Ice and Fire. For those of you familiar with Game of Thrones, this is a Game of Thrones API. It’s pretty cool. It gives you lots of information about the characters.
I’m a fan of Game of Thrones, and in the show, even more so in the books, there are a lot of characters in these books. Not only that, but each character is known by like eight or nine different names, to makes things even easier.
So, I thought “Hey, wouldn’t it be cool if I could say ‘Okay, Google. Ask Westeros Directory Services for Petyr Baelish’s other names,’ and it would just give me a list of aliases.” So, now that we have an API like this, and we have a decision on what it is that we want to build, let’s see if we can get this done. We’ll put ten minutes on the clock, and let’s go.
I’m here in API.ai. This is an NLU platform that was acquired by Google last year. That’s Natural Language Understanding. I’m going to create myself an agent in API “WesterosDirectoryServices.” An agent is kind of where we house all of the NLU for the particular project that we’re working on.
So, I need to create myself an Entity. We talked about these earlier, right? These are the data that we pull out of something that’s said. If someone asks “Ask Westeros Directory Services for Arya Stark’s other names,” Google is going to send to API.ai; not that invocation, not the “Ask Westeros Directory Services,” but the “Arya Stark’s other names” portion of that.
We’re going to need to pull that character name out of the speech, right? So, I need a character Entity. I’ll create one for Petyr Baelish. That might be mis-heard by the speech detect system, as Peter with an “e,” since that’s how we spell it on Earth. “Peter Baelish.” I’m just entering synonyms; different ways that might be heard.
And then, Arya Stark, that might be heard as “Aria” with an “i.” The reason that that’s important and we can see, if I switch over to Postman, if I call this API with Arya with a “y,” it understands. If I call it with Aria with an “i,” it doesn’t know what we’re talking about.
This is called “canonical entity values,” right? So, each one of these entities has this name, and only this name. If you hear it as this, give it to me as Arya with a “y.”
Now, we’ll need an Intent. An Intent, these are all the various things in a system that a user might want to do. In this case, they’ll want to list the aliases. They might say “Arya Stark’s other names.” You can see it’s automatically already understanding. “Oh! That’s that character Arya Stark you taught me about earlier.” “Is Petyr Baelish known by any other names?”
Here, I’m just entering all the various ways someone might ask this question, and it’s going through and annotating those to show me the Entities I’ve already created. I’ll give this an Action name of “listAliases.” Actions become more important when you get into more in-depth features like dialogues, but we’ll just use this to route to the appropriate code. I can enter a response here, if I wanted. “Arya is also known as Jill or Susan.” I can save that, and now, let’s give this a shot.
“Petyr Baelish’s other names.” You can see, this understood what I said. It actually heard it as “Petyr Baelish is other names,” but API.ai was smart enough to figure out what we were going for. It figured out we wanted that listAliases action. It pulled out that character of Petyr Baelish.
Oh, no! But it’s telling us that Arya is also known as Jill or Susan. We can’t use a static response there, right? We need another API that’s then going to call out to the API of Ice and Fire, find out Arya Stark’s other names, and then generate some kind of a response that we should respond with actual speech, back to the user.
In order to do that, I’m going to have to use this feature of API.ai called Fulfillment. First, I’ll save my Intent, there. We’ll go down to Fulfillment, turn this on. It wants a URL, so we’re actually going to need to build our own API.
For that, I’m going to use this project called Gomix (note: this is now called Glitch. Gomix, if you haven’t heard of it, it’s very, very cool. It’s a live IDE, a live code editor in your browser. What’s interesting is, as you make changes to the code, it’s automatically re-deploying your project. So, not only do you get to edit your code, but you already have it deployed. It’s a really fun way to kind of try things out.
So, we put together this Actions on Google API.ai Boilerplate. After the talk, we’ll tweet out a link to that. All you have to do is click this “Remix This” button, and then, cool! I’ve got my own version, my own API sitting here, waiting for me to code on it.
Now, I’ll head over to the server.js, and start checking out the code. Okay, it looks like it’s an express app, so if you’re a JavaScript developer, this is familiar. We’re posting, okay, it looks like this was asking the weather. So, you could grab the city. “What’s the weather like in Austin?” It’s going to make a call to the Yahoo Weather API, and then it’s going to pull some data out of there, and generate a response.
I think I see what’s going on here. So, first of all, let’s do a LIST_ALIASES_ACTION, right? That’s the one for ours. That’s listAliases. So, we have a few other places now, in the code, where it’s going to tell me, if I just follow my linter.
Okay, so I need – there. That looks like that’s where we’re routing to LIST_ALIASES. In fact, I’m going to call this function listAliases, so we’ll route to this listAliases handler function. Okay, and then I need a, so that’s going to be a CHARACTER_PARAMETER. That, we just need to switch to character.
Same deal, right? We’ve got another place where we’re pulling that parameter, that entity, out, so we’ll say let character.
Here, we don’t need this URL anymore, but we do need a requestURL, and we’ll make this. We’ll grab it right from Postman. So, I can’t hard code Arya Stark in there; we’re just going to get her names back every time, so I’ll actually just pass in that character name we just grabbed.
Cool! Now, I’ve generated the proper URL. Let’s make a request to that URL, and rather than pulling out units and temperature, like we’re doing here, we just need the aliases from that response. If I look at the response, it looks like we’ve got an array, and at the first element, we need the aliases attribute. Okay, cool. I’ll say body, and at the first element I have got an attribute aliases. So, there are my aliases. We’ll take that out of the code.
And then, okay, assistant.tell. This looks like this is where we actually generate a response, so we’ll say assistant.tell(character + ‘ is also known as’), and then we’ll use the toSentence method I included for us, (aliases). Okay, so let’s pull that response out. Let’s check our linter again. It looks like we shouldn’t have any errors, and we’re properly routing down here, from LIST_ALIASES. Great!
Now, I’m going to need that live URL. So, if I just click Show Live, grab that URL, come back here, we’ll enter that as the endpoint URL. Now, I’ll head back to my Intent for LIST_ALIASES, and I’ll say rather than using that static response, I actually want to turn on this webhook fulfillment. We’ll save that, give it a moment to update, and let’s try it again.
“Petyr Baelish’s other names.” Cool! So, that came down, and it grabbed the Intent listAliases and the action listAliases. Let’s try one more time. “Arya Stark’s other names.” Cool! It took the model a moment to update, but now you can see it’s actually grabbing our response from that API that we created, right?
Let’s see if we can get this hooked up to a Google Home device. We get all these one-click integrations in API.ai. I’m going to activate the one for Actions on Google. I’m going to give it the name – let’s turn that on – I’m going to give it the name Westeros Directory Services. I can choose different voices, if I’d like things like that, but I’m just going to go ahead and authorize, and preview that.
Now, this is actually deploying this to something called the “Google Home Web Simulator.” The Web Simulator is really interesting. It’s a place where I can actually test my Google Home actions, just right here in the web. I’ll make sure that’s reloaded the proper action, and let’s give this a shot.
“Ask Westeros Directory Services for Arya Stark’s other names.”“Sure, here’s Westeros Directory Services. Arya Stark is also known as Arya Horseface, Arya Underfoot, Arry, Lumpyface, Lumpyhead, Stickboy, Weasel, Nymera, Squan, Saltb, Cat of the Canaly, Bets, The Blind Girh, The Ugly Little Girl, Mercedent and Mercye.”
Do you see what I’m talking about? Ridiculous! Why does one character need this many names?
So, yeah. I don’t know. By the way, has HBO been doing any marketing for Game of Thrones? Someone should talk to them, because I feel like they should really be here, telling people about that show.
That was 07:42, so we did it, y’all! We did it! Less than 10 minutes! We were able to go from idea to an actual machine that can answer our weird, nerdy questions, that we just came up with.
There’s a lot of stuff we could do, to go from here, right? We could use some API.ai features like Context, so we could do things to ask follow-up questions, like “Okay. What’s her mother’s name?” In that sentence, I’m going to need to understand what that pronoun “her” is referring to, right? It’s referencing the character we just asked about.
There are a lot of places we can go with this. I hope you’ll check out the Boilerplate Project, but the fact that we’ve gotten here, after this many centuries, represents not only a monumental achievement in the history of computing, but in the history of the grand conversation between humans and machines.
We can’t wait to see what you’ve built!
Thank you so much.
Voxable helps companies and products understand the conversational commerce landscape and build bots to help them automate conversations with their teams and customers. We build all things that help humans talk to machines, including chatbots for major messaging channels, Alexa Skills, and bots for the Google Assistant. If you’re interested in building a conversational strategy, holler at us!
