Extending the lambda function to get user input

What’s next?  I want to extend the lambda function so I can ask to mirror to show me a picture of something or someone.  The mirror will interpret this and get the picture for me.  But first things first.  I’ll be extending my lambda function to interpret slots.  A slot is a sort of parameter you can pass in your Alexa intent schema.  Something like “show me a picture of <abc>”.  In this example <abc> is a slot of type LITERAL.  Alexa will interpret you sentence and automatically assign what you said to the slot.  It’s up to the lambda function to extract this info and do something with it.

Below you can find my lambda function.  I added another intent at the bottom.  I called it “ShowPictureIntent”, and I will be configuring it later on in my Alexa intent schema.

So what do we see here?  First we check if there are slots in the intent, and if there are, if an attribute is present called “Content”.  This is an arbitrary chosen attribute name.  It could be anything, but I chose to call it content, because it might server another purpose in the future as well.  If there’s no such thing, it will let Alexa say that it didn’t understand me.

If it did, however, the command is issued to the dynamoDB table.  I created a new command called “SHOW-PICTURE”.  Something my Alexa interface will have to deal with, and I put the content of the slot into a new attribute, also called “Content”.

In order to test this, I have to change the test event configuration.  You can do that in the top menu.

Here you can define some JSON, and configure the correct intent, and issue a slot.

Here i’m searching for a picture of George Michael.  So here’s the result of the test :

And how the command shows up in the DynamoDB table.

Now for the skill setup.  We adjust the intent schema like so :

we define a new intent, with the same name as we’re handling in the function.  And we define a slot, with name “Content”, just as we expect it.  We want to tell the skill to expect an arbitrary string.  This can be done by assigning the type “AMAZON.LITERAL”.  For more info on the different slot types, you can see this documentation.

The only thing left to configure is the utterances.  Like so :

Notice how I created 2 different utterances, and the syntax for referencing the slot.  Because it’s a slot of type literal, we need to tell Alexa an example of what we’re expecting.  What comes before the pipe symbol is crucial here.  We want it to expect a first name and a last name.  Otherwise, if you just type “name” for example and if you ask it for “George Michael”, it will pass on “Michael” ; just the last word.  Let’s test it :

I added some some more options to be complete :

Creating a generic link between the Java Client and the mirror

And now for the final part of the journey.  Getting the commands sent by the lambda functions to the mirror.  The idea is that the mirror will read from the CommandQueue dynamoDB table, execute the command, and remove the entry from the queue.

But we first need some trigger for the mirror to start polling for the commands in the queue.  To do that I’ll be adapting the AVS java client some more.  I’ll create two new functions in the mirror interface :

They are quite straight forward.  We can call them from the AVSController java class where the overrides for AlexaSpeechStarted and AlexaSpeechFinish are defined.

Rebuild the java client, and test that thing by issuing a question to Alexa and catching it in the Debug hub :

Works great.  Time to change the debug a bit to be able to send those functions to the mirror as well.  So some extra buttons in the Alexa.html file :

Now to for some extra code in the mirror software.  We first need to do some plumbing like install the AWS SDK.  While in the Smartmirrorproject subdirectory enter

sudo npm install aws-sdk

Then copy over the aws.json file from the root of the debug hub folder to the root of the smart mirror project folder.  We’ll be using the same settings to talk to our DynamoDB instance.

 

Now for adding some code to the mirror interface module node helper. Look a the start function override below :

First we configure the DynamoDB and instantiate the docClient variable.  We add 2 cases in the switch to handle both the speech-started and speech-stopped commands.
The first one looks for commands in CommandQueue table.  It does this by creating a params object and stating the name of the table.  Next, we need a projection list.  This is a list of columns that is to be returned by the docClient.  Well, something annoying came up here.  Apparently both the “session” and “timestamp” columns seem to be reserved words.  You can’t use them as a column name if you want to query them this way.  So, some refactoring needs to be done.  I renamed them to session_id and time_stamp.
The data is being read using the “scan” method.

The second command handler deletes the entry from the command queue.  But just in case there was an error earlier, and a command has been left in the queue, all the items have been read, and they will all be deleted in the for-loop.  The delete is being done by the “delete” method.  Below you can find the slightly refactored lambda function with the new column names.

Notice how both the delete and scan object require a callback, and a reference is passed to 2 existing functions.  Below you can see their definition.

The handle function sends the actual socket notification but also pushes the delete parameters for the delete command to a local array.  The QueueClear handler just issues some logging.

In order to make this work, we will need to change the ipWhiteList setting of the smart mirror.  If not, it will not accept requests from the java client.  You will also have to make sure the Java client on the Pi sends to the correct address and port.

Ok, so let’s test it.  Below is some output of a working session where admitted – by accident – 2 entries were in the queue, but they were both executed succesfully.

What’s left is add some extra functionality to allow showing the default modules as well.

That’s done pretty fast.  Extend the lambda function :

And provide an intent schema + utterances in the skill.

And that’s it!  A milestone in our development. We got Alexa talking to our start mirror.  Time to get to more advanced and cool stuff.

Creating the hide all modules Alexa skill

Now it’s time to create a new Alexa skill, dedicated to our smart mirror.  We will be expanding the functionality of the skill gradually, but let’s start with the most basic functionality : hiding the modules on the mirror.

You might remember from my last post that you need the US-East region in AWS to be able to make lambda functions. However, if you remember the post where I created the first table in the DynamoDB setup, it was created in a different region.  You’ll need to complete those steps again in the US-East region in order for the lambda function and the DynamoDB service to communicate.  Check out this post for more info.  Note that you’ll have to change you aws.config file as well in order to connect to it using the debug hub.

 

Anyways, let’s create our function.

The first thing I did was copy the “Hello world” example on my disk to a new folder.  I opened it in Atom and replaced the “Hello world” references to SmartMirror references, like this :

That’s a good starting point. That way I was able to zip the contents  and upload it to AWS console.  The rest of the editing can be done inside the AWS console.

When configuring the function you can choose the settings below.  I created a new role from a template and chose the basic lambda execution rule.  Once saved the config looks like this :

From experience I know that using this role will cause the lambda function to give an error when accessing the DynamoDB table.  The reason is a lack of permissions.  This is easy to solve.  You have to open the IAM module.  The one you used in an earlier post to create the SmartMirror user.

Locate the role by navigating to the roles in the left hand pane, and in the Permissions tab, click “Attach Policy”.  Locate the “AmazonDynamoDBFullAccess” policy.  In fact, this is a bit overkill.  You could setup a policy to only access the CommandQueue table, but in our case this project is only intended for personal use, so i’m not worried about security right now.

Now that we’ve got the permissions thing out of the way, we can start looking at the code.

Go to the code of the function, and on top enter the following code :

That way you’ll include the AWS SDK, and instantiate a new DynamoDB interface.

In the IntentHandlers function : create a new entry for a “HideAllModulesIntent”, which we’ll be defining later on in the Alexa Skill.  The code for it is displayed below :

Small note: the code doesn’t contain the definition of the “Message” variable, just add it above these lines of code.

Notice how it uses the “putItem” method.  It takes a parameter (in the form a JSON format) and a callback function.  The parameter contains

• The tablename
• The content of the item

Notice the special syntax.  Every item is a different JSON attribute as well.  The callback function is fairly simple.  If the “err” object is not undefined, it means there’s been an error, and we need to act accordingly.  If Not, it succeeded.  Note that we call the “response.tellWithCard” function in both branches inside the callback function. You would think that’s not very optimal and put it behind the “putItem” call.  Well, been there done that.  That doesn’t work.  It turns out that lambda function returns when the “tellWithCard” function is called, and when doing that just after the “putItem” call doesn’t give the “putItem” enough time to complete its task. That means no error will be returned, but the callback function won’t be called either.  The solution is the code above.

Save the function.  You can test it with the button above.

When doing so, you can see the result :

And the log

Let’s log on to the DynamoDB console and refresh the contents of the table :

That worked like a charm.

Now let’s create the Alexa Skill.  Go to the Alexa Skills overview and create a new one.  Empty template & settings below.

I changed the invocation name of the “hello world” sample to “hello world”, so i can use “smart mirror” for this one.  If I were to choose the same one, Alexa wouldn’t know which one to choose.

The interaction mode is shown below and resembles the “hello world” example, only customised for the smart mirror.

Configure the endpoint with the ARN of the lambda function.  I’ve got no screenshot of that, but it’s quite straight forward.

You can easily test it :

And there we have it.  Remember to change the APP_ID in the lambda function code in order to let them communicate.

If you change the Session ID parameter in the code of the debug hub, you can also see the output in the debug hub :

Good luck!

 

 

 

Setting up the hello world skills kit sample

Now we’ve got the DynamoDB things out of the way (for now), I want to start experimenting with my very own custom skill for Alexa.  My goal is to create my own smart mirror custom skill.  It needs to handle all sorts of stuff, but let’s start with the basics.

Before I start creating my own, I first want to understand how it all works exactly.  That’s why I will start by creating a skill from the samples delivered by Amazon.  Let’s download the samples from GitHub into a new folder.

mkdir alexa-skills-kit-samples

git clone https://github.com/amzn/alexa-skills-kit-js.git ./alexa-skills-kit-samples

The next thing we need to do is log on to the AWS console.  It’s important to choose the correct region in the top right corner.  There are only a few regions where the lambda service is available.  Since I’ve made the DynamoDB table in another region I will probably have to redo the earlier DynamoDB steps, but those are worries for later.  I’ve chosen this region :

Navigate to the lambda service, and click “Get Started Now”.  The lambda service lets you create your very own piece of code in the cloud.  It is executed only when it is called, so you only have to pay for the little amount of time it runs.  Furthermore, every one who registers is entitled to 1 million free runs of any lambda function per month.  So, in our case that’s actually free.

Next, we are prompted to choose a template.  There are several templates, some of which are intended for Alexa lambda functions.  We won’t be using a template since we’ll be uploading the sample code, so I just chose “Blank Function”.

Next, the service asks me which application will trigger the lambda function.  Just choose “Alexa Skills Kit” and click “Next”.

Next, we need to configure some stuff, like a descent name and a runtime . In our case, we will be writing the code in NodeJS, so i’ll use this one, but we can choose other runtimes as well.

Now, we can specify the code for the lambda function . There are several ways to do so, the simplest of which is probably just typing the code in the browser . The disadvantage here is the lack of highlighting, code completion, etc.  It’s not an IDE.  Instead I will zip the files in the “src” folder of the “helloWorld” sample, and upload it.

Choose “Upload a .ZIP file” as the code entry type.  Leave the handler as it is, and make sure “Create new role from template(s)” is selected.  Specify your own rule name.

Create the function.  In the next screen you should see the ARN (Amazon Resource Name), that looks about like this :

Just copy it to the clipboard.

Now for the second part.  We need to make a new Alexa skill to call the lambda function.  In order to do that we need to navigate to the amazon developer console & log in.  In the “Alexa” tab, we can choose to create a new skill by selected the “Alexa Skills Kit”.

So now, we start the configuration of the skill.  First of all I’ll choose a custom interaction model.  The Smart Home and Flash Briefing skills are a fixed type of interaction specifically intended for those purposes.  Since we’re creating something custom, we choose the first one.  We’ll choose “English” as a language.  While I’m writing you can also choose for English UK and German.  I hope that in the future, more languages will appear here, specifically dutch as that is my mother tongue.

Anyway.  I’ll give it a fitting name “HelloWorld”, and specify an invocation name.  This is the words Alexa listens for in order for the skill to be recognized.  There are a number of limitations and constraints to what you can specify here.  Carefully read the documentation.

I don’t want the audio player functionality.

Click Next.  Now we need to specify 2 things :

• An intent schema
• Sample Utterances.

The first one is a specification in JSON format telling Alexa what the skill can do, and what parameters it can expect.  The definition of what Alexa can do in that skill is an intent.  Alexa will interpret the user’s voice and based on the sample utterances, it will deduct what the intent of the user is.  It will then execute the code for that intent.  For the “Hello World” example, Amazon has provided the schema and the utterances for us.

Just copy past them in the appropriate boxes.

Notice you can create “slots”.  These slots are kind of parameters you can pass a long.  Such as “show a picture of George Michael”.  George Michael (or any other person or thing you want to see a picture of) will be a slot of string type.  Alexa will then recognize it, and pass it along as a parameter.  For this example we’re not going to use a slot, there aren’t any specified in the schema.

In the sample utterances, you can specify how Alexa will recognize a certain intent.  The utterance starts with the name of an intent and is followed with some words Alexa needs to interpret.  Now… Alexa is pretty smart.  You don’t really need to input every single permutation of words here.  For example : if you say “say hello to the world” (with the extra “the” in there), it will probably understand as well.  Alexa uses language models to create a context in which the user is interpreted, and he or she usually speaks.  Depending on that model (which it builds on the fly) it will better recognize your intent.

The last configuration step is linking the lambda function and the skill to each other.  Choose “AWS Lambda ARN” as the endpoint type, and paste in the ARN.  No account linking is necessary here.

Not that you could be specifying different ARN’s for different languages.  Isn’t that the coolest thing?

In the next step you can test it.  Just type what you would say to alexa, end you can see how the lambda request is built up, and what the response would be.

You could go and certify this thing, but I don’t thing Amazon would be too happy having all the hello world examples on the store, so I suggest you don’t.  Even “uncertified”, the skill should be available to you if you navigate to “alexa.amazon.com”, and log in with your account.

So, the last thing we need to do is tell the lambda function what the ID of the calling application is.  That way we ensure that the function can only be called by the Alexa skill we set up.  Doing that is easy.  Just go to the AWS console, and locate the “Hello world” lambda function, and edit the code.

Change the APP_ID to match the application ID of your Alexa skill :

You can find it in the definition of the skill :

I couldn’t help but add my own personal touch…

 

Go ahead and test this.

Good luck!