Extending the lambda function to get user input

Extending the lambda function to interpret the query for a picture of someone or something.

What’s next?  I want to extend the lambda function so I can ask to mirror to show me a picture of something or someone.  The mirror will interpret this and get the picture for me.  But first things first.  I’ll be extending my lambda function to interpret slots.  A slot is a sort of parameter you can pass in your Alexa intent schema.  Something like “show me a picture of <abc>”.  In this example <abc> is a slot of type LITERAL.  Alexa will interpret you sentence and automatically assign what you said to the slot.  It’s up to the lambda function to extract this info and do something with it.

Below you can find my lambda function.  I added another intent at the bottom.  I called it “ShowPictureIntent”, and I will be configuring it later on in my Alexa intent schema.

Schermafbeelding 2017-02-10 om 20.49.04.png

So what do we see here?  First we check if there are slots in the intent, and if there are, if an attribute is present called “Content”.  This is an arbitrary chosen attribute name.  It could be anything, but I chose to call it content, because it might server another purpose in the future as well.  If there’s no such thing, it will let Alexa say that it didn’t understand me.

If it did, however, the command is issued to the dynamoDB table.  I created a new command called “SHOW-PICTURE”.  Something my Alexa interface will have to deal with, and I put the content of the slot into a new attribute, also called “Content”.

In order to test this, I have to change the test event configuration.  You can do that in the top menu.

Schermafbeelding 2017-02-10 om 20.23.40.png

Here you can define some JSON, and configure the correct intent, and issue a slot.

Schermafbeelding 2017-02-10 om 20.48.38.png

Here i’m searching for a picture of George Michael.  So here’s the result of the test :

Schermafbeelding 2017-02-10 om 20.23.30.png

And how the command shows up in the DynamoDB table.

Schermafbeelding 2017-02-10 om 20.23.21.png

Now for the skill setup.  We adjust the intent schema like so :

Schermafbeelding 2017-02-10 om 20.50.03.png

we define a new intent, with the same name as we’re handling in the function.  And we define a slot, with name “Content”, just as we expect it.  We want to tell the skill to expect an arbitrary string.  This can be done by assigning the type “AMAZON.LITERAL”.  For more info on the different slot types, you can see this documentation.

The only thing left to configure is the utterances.  Like so :

Schermafbeelding 2017-02-10 om 20.50.09.png

Notice how I created 2 different utterances, and the syntax for referencing the slot.  Because it’s a slot of type literal, we need to tell Alexa an example of what we’re expecting.  What comes before the pipe symbol is crucial here.  We want it to expect a first name and a last name.  Otherwise, if you just type “name” for example and if you ask it for “George Michael”, it will pass on “Michael” ; just the last word.  Let’s test it :

Schermafbeelding 2017-02-10 om 20.54.32.png

I added some some more options to be complete :

Schermafbeelding 2017-02-11 om 12.57.44.png

Creating a generic link between the Java Client and the mirror

Linking the Alexa skill to the fitting behaviour on the smart mirror.

And now for the final part of the journey.  Getting the commands sent by the lambda functions to the mirror.  The idea is that the mirror will read from the CommandQueue dynamoDB table, execute the command, and remove the entry from the queue.

But we first need some trigger for the mirror to start polling for the commands in the queue.  To do that I’ll be adapting the AVS java client some more.  I’ll create two new functions in the mirror interface :

Schermafbeelding 2017-02-04 om 11.17.14.png

They are quite straight forward.  We can call them from the AVSController java class where the overrides for AlexaSpeechStarted and AlexaSpeechFinish are defined.

Schermafbeelding 2017-02-04 om 11.19.03.png

Rebuild the java client, and test that thing by issuing a question to Alexa and catching it in the Debug hub :

Schermafbeelding 2017-02-05 om 09.39.18.png

Works great.  Time to change the debug a bit to be able to send those functions to the mirror as well.  So some extra buttons in the Alexa.html file :

Schermafbeelding 2017-02-05 om 09.55.50.png

Now to for some extra code in the mirror software.  We first need to do some plumbing like install the AWS SDK.  While in the Smartmirrorproject subdirectory enter

sudo npm install aws-sdk

Then copy over the aws.json file from the root of the debug hub folder to the root of the smart mirror project folder.  We’ll be using the same settings to talk to our DynamoDB instance.

 

Now for adding some code to the mirror interface module node helper. Look a the start function override below :

Schermafbeelding 2017-02-05 om 18.38.42.png

First we configure the DynamoDB and instantiate the docClient variable.  We add 2 cases in the switch to handle both the speech-started and speech-stopped commands.
The first one looks for commands in CommandQueue table.  It does this by creating a params object and stating the name of the table.  Next, we need a projection list.  This is a list of columns that is to be returned by the docClient.  Well, something annoying came up here.  Apparently both the “session” and “timestamp” columns seem to be reserved words.  You can’t use them as a column name if you want to query them this way.  So, some refactoring needs to be done.  I renamed them to session_id and time_stamp.
The data is being read using the “scan” method.

The second command handler deletes the entry from the command queue.  But just in case there was an error earlier, and a command has been left in the queue, all the items have been read, and they will all be deleted in the for-loop.  The delete is being done by the “delete” method.  Below you can find the slightly refactored lambda function with the new column names.

Schermafbeelding 2017-02-05 om 18.37.13.png

Notice how both the delete and scan object require a callback, and a reference is passed to 2 existing functions.  Below you can see their definition.

Schermafbeelding 2017-02-05 om 18.38.49.png

The handle function sends the actual socket notification but also pushes the delete parameters for the delete command to a local array.  The QueueClear handler just issues some logging.

In order to make this work, we will need to change the ipWhiteList setting of the smart mirror.  If not, it will not accept requests from the java client.  You will also have to make sure the Java client on the Pi sends to the correct address and port.

Schermafbeelding 2017-02-05 om 18.37.33.png

Ok, so let’s test it.  Below is some output of a working session where admitted – by accident – 2 entries were in the queue, but they were both executed succesfully.

Schermafbeelding 2017-02-05 om 18.35.33.png

What’s left is add some extra functionality to allow showing the default modules as well.

That’s done pretty fast.  Extend the lambda function :

Schermafbeelding 2017-02-05 om 18.41.17.png

And provide an intent schema + utterances in the skill.

schermafbeelding-2017-02-05-om-18-46-05

And that’s it!  A milestone in our development. We got Alexa talking to our start mirror.  Time to get to more advanced and cool stuff.

Creating the hide all modules Alexa skill

Creating our own custom smart mirror skill based upon the hello world example.

Now it’s time to create a new Alexa skill, dedicated to our smart mirror.  We will be expanding the functionality of the skill gradually, but let’s start with the most basic functionality : hiding the modules on the mirror.

You might remember from my last post that you need the US-East region in AWS to be able to make lambda functions. However, if you remember the post where I created the first table in the DynamoDB setup, it was created in a different region.  You’ll need to complete those steps again in the US-East region in order for the lambda function and the DynamoDB service to communicate.  Check out this post for more info.  Note that you’ll have to change you aws.config file as well in order to connect to it using the debug hub.

 

Anyways, let’s create our function.

The first thing I did was copy the “Hello world” example on my disk to a new folder.  I opened it in Atom and replaced the “Hello world” references to SmartMirror references, like this :

Schermafbeelding 2017-02-04 om 10.27.51.png

That’s a good starting point. That way I was able to zip the contents  and upload it to AWS console.  The rest of the editing can be done inside the AWS console.

When configuring the function you can choose the settings below.  I created a new role from a template and chose the basic lambda execution rule.  Once saved the config looks like this :

Schermafbeelding 2017-02-03 om 23.23.17.png

From experience I know that using this role will cause the lambda function to give an error when accessing the DynamoDB table.  The reason is a lack of permissions.  This is easy to solve.  You have to open the IAM module.  The one you used in an earlier post to create the SmartMirror user.

Locate the role by navigating to the roles in the left hand pane, and in the Permissions tab, click “Attach Policy”.  Locate the “AmazonDynamoDBFullAccess” policy.  In fact, this is a bit overkill.  You could setup a policy to only access the CommandQueue table, but in our case this project is only intended for personal use, so i’m not worried about security right now.

Schermafbeelding 2017-02-03 om 23.18.22.png

Now that we’ve got the permissions thing out of the way, we can start looking at the code.

Go to the code of the function, and on top enter the following code :

Schermafbeelding 2017-02-03 om 23.21.10.png

That way you’ll include the AWS SDK, and instantiate a new DynamoDB interface.

In the IntentHandlers function : create a new entry for a “HideAllModulesIntent”, which we’ll be defining later on in the Alexa Skill.  The code for it is displayed below :

Schermafbeelding 2017-02-04 om 10.20.16.png

Small note: the code doesn’t contain the definition of the “Message” variable, just add it above these lines of code.

Notice how it uses the “putItem” method.  It takes a parameter (in the form a JSON format) and a callback function.  The parameter contains

  • The tablename
  • The content of the item

Notice the special syntax.  Every item is a different JSON attribute as well.  The callback function is fairly simple.  If the “err” object is not undefined, it means there’s been an error, and we need to act accordingly.  If Not, it succeeded.  Note that we call the “response.tellWithCard” function in both branches inside the callback function. You would think that’s not very optimal and put it behind the “putItem” call.  Well, been there done that.  That doesn’t work.  It turns out that lambda function returns when the “tellWithCard” function is called, and when doing that just after the “putItem” call doesn’t give the “putItem” enough time to complete its task. That means no error will be returned, but the callback function won’t be called either.  The solution is the code above.

Save the function.  You can test it with the button above.

Schermafbeelding 2017-02-03 om 23.20.49.png

When doing so, you can see the result :

Schermafbeelding 2017-02-03 om 23.20.53.png

And the log

Schermafbeelding 2017-02-03 om 23.21.00.png

Let’s log on to the DynamoDB console and refresh the contents of the table :

Schermafbeelding 2017-02-04 om 10.16.25.png

That worked like a charm.

Now let’s create the Alexa Skill.  Go to the Alexa Skills overview and create a new one.  Empty template & settings below.

Schermafbeelding 2017-02-04 om 10.52.48.png

I changed the invocation name of the “hello world” sample to “hello world”, so i can use “smart mirror” for this one.  If I were to choose the same one, Alexa wouldn’t know which one to choose.

The interaction mode is shown below and resembles the “hello world” example, only customised for the smart mirror.

Schermafbeelding 2017-02-03 om 23.22.46.png

Configure the endpoint with the ARN of the lambda function.  I’ve got no screenshot of that, but it’s quite straight forward.

You can easily test it :

Schermafbeelding 2017-02-03 om 23.23.48.png

And there we have it.  Remember to change the APP_ID in the lambda function code in order to let them communicate.

If you change the Session ID parameter in the code of the debug hub, you can also see the output in the debug hub :

Schermafbeelding 2017-02-04 om 11.08.55.png

Good luck!

 

 

 

Setting up the hello world skills kit sample

Getting to know custom skills by implementing the hello world example.

Now we’ve got the DynamoDB things out of the way (for now), I want to start experimenting with my very own custom skill for Alexa.  My goal is to create my own smart mirror custom skill.  It needs to handle all sorts of stuff, but let’s start with the basics.

Before I start creating my own, I first want to understand how it all works exactly.  That’s why I will start by creating a skill from the samples delivered by Amazon.  Let’s download the samples from GitHub into a new folder.

mkdir alexa-skills-kit-samples
git clone https://github.com/amzn/alexa-skills-kit-js.git ./alexa-skills-kit-samples

The next thing we need to do is log on to the AWS console.  It’s important to choose the correct region in the top right corner.  There are only a few regions where the lambda service is available.  Since I’ve made the DynamoDB table in another region I will probably have to redo the earlier DynamoDB steps, but those are worries for later.  I’ve chosen this region :

Schermafbeelding 2017-02-02 om 20.49.57.png

Navigate to the lambda service, and click “Get Started Now”.  The lambda service lets you create your very own piece of code in the cloud.  It is executed only when it is called, so you only have to pay for the little amount of time it runs.  Furthermore, every one who registers is entitled to 1 million free runs of any lambda function per month.  So, in our case that’s actually free.

Schermafbeelding 2017-02-02 om 20.50.31.png

Next, we are prompted to choose a template.  There are several templates, some of which are intended for Alexa lambda functions.  We won’t be using a template since we’ll be uploading the sample code, so I just chose “Blank Function”.

Schermafbeelding 2017-02-02 om 20.51.52.png

Next, the service asks me which application will trigger the lambda function.  Just choose “Alexa Skills Kit” and click “Next”.

Schermafbeelding 2017-02-02 om 20.52.27.png

Next, we need to configure some stuff, like a descent name and a runtime . In our case, we will be writing the code in NodeJS, so i’ll use this one, but we can choose other runtimes as well.

Schermafbeelding 2017-02-02 om 20.53.46.png

Now, we can specify the code for the lambda function . There are several ways to do so, the simplest of which is probably just typing the code in the browser . The disadvantage here is the lack of highlighting, code completion, etc.  It’s not an IDE.  Instead I will zip the files in the “src” folder of the “helloWorld” sample, and upload it.

Schermafbeelding 2017-02-02 om 20.56.46.png

Choose “Upload a .ZIP file” as the code entry type.  Leave the handler as it is, and make sure “Create new role from template(s)” is selected.  Specify your own rule name.

Schermafbeelding 2017-02-02 om 20.58.47.png

Schermafbeelding 2017-02-02 om 21.04.23.png

Create the function.  In the next screen you should see the ARN (Amazon Resource Name), that looks about like this :

Schermafbeelding 2017-02-02 om 21.08.07.png

Just copy it to the clipboard.

Now for the second part.  We need to make a new Alexa skill to call the lambda function.  In order to do that we need to navigate to the amazon developer console & log in.  In the “Alexa” tab, we can choose to create a new skill by selected the “Alexa Skills Kit”.

Schermafbeelding 2017-02-02 om 21.11.54.png

So now, we start the configuration of the skill.  First of all I’ll choose a custom interaction model.  The Smart Home and Flash Briefing skills are a fixed type of interaction specifically intended for those purposes.  Since we’re creating something custom, we choose the first one.  We’ll choose “English” as a language.  While I’m writing you can also choose for English UK and German.  I hope that in the future, more languages will appear here, specifically dutch as that is my mother tongue.

Anyway.  I’ll give it a fitting name “HelloWorld”, and specify an invocation name.  This is the words Alexa listens for in order for the skill to be recognized.  There are a number of limitations and constraints to what you can specify here.  Carefully read the documentation.

I don’t want the audio player functionality.

Schermafbeelding 2017-02-02 om 21.14.38.png

Click Next.  Now we need to specify 2 things :

  • An intent schema
  • Sample Utterances.

The first one is a specification in JSON format telling Alexa what the skill can do, and what parameters it can expect.  The definition of what Alexa can do in that skill is an intent.  Alexa will interpret the user’s voice and based on the sample utterances, it will deduct what the intent of the user is.  It will then execute the code for that intent.  For the “Hello World” example, Amazon has provided the schema and the utterances for us.

Schermafbeelding 2017-02-02 om 21.15.49.png

Just copy past them in the appropriate boxes.

Notice you can create “slots”.  These slots are kind of parameters you can pass a long.  Such as “show a picture of George Michael”.  George Michael (or any other person or thing you want to see a picture of) will be a slot of string type.  Alexa will then recognize it, and pass it along as a parameter.  For this example we’re not going to use a slot, there aren’t any specified in the schema.

Schermafbeelding 2017-02-02 om 21.18.50.png

In the sample utterances, you can specify how Alexa will recognize a certain intent.  The utterance starts with the name of an intent and is followed with some words Alexa needs to interpret.  Now… Alexa is pretty smart.  You don’t really need to input every single permutation of words here.  For example : if you say “say hello to the world” (with the extra “the” in there), it will probably understand as well.  Alexa uses language models to create a context in which the user is interpreted, and he or she usually speaks.  Depending on that model (which it builds on the fly) it will better recognize your intent.

The last configuration step is linking the lambda function and the skill to each other.  Choose “AWS Lambda ARN” as the endpoint type, and paste in the ARN.  No account linking is necessary here.

Not that you could be specifying different ARN’s for different languages.  Isn’t that the coolest thing?

Schermafbeelding 2017-02-02 om 21.21.16.png

In the next step you can test it.  Just type what you would say to alexa, end you can see how the lambda request is built up, and what the response would be.

Schermafbeelding 2017-02-02 om 21.23.19.png

You could go and certify this thing, but I don’t thing Amazon would be too happy having all the hello world examples on the store, so I suggest you don’t.  Even “uncertified”, the skill should be available to you if you navigate to “alexa.amazon.com”, and log in with your account.

Schermafbeelding 2017-02-03 om 06.48.35.png

So, the last thing we need to do is tell the lambda function what the ID of the calling application is.  That way we ensure that the function can only be called by the Alexa skill we set up.  Doing that is easy.  Just go to the AWS console, and locate the “Hello world” lambda function, and edit the code.

Change the APP_ID to match the application ID of your Alexa skill :

Schermafbeelding 2017-02-03 om 18.42.44.png

You can find it in the definition of the skill :

Schermafbeelding 2017-02-03 om 20.28.22.png

I couldn’t help but add my own personal touch…

Schermafbeelding 2017-02-03 om 20.21.22.png

 

Go ahead and test this.

Good luck!

 

Amazon web services

Registering with Amazon Web Services to get the most out of the mirror.

After discovering our minor setback in my last post, I had to find a way to get Alexa to notify the mirror application when a command was issued (or a question was asked), without using the Java Client.

Turns out you can add your own extra “skills” to Alexa.  What Amazon calls a “Skill” is extra functionality you can write yourself, and add to the portfolio of Alexa commands you can ask the mirror.  In order to do that we need to get an account to the Amazon Web Services (AWS).

You can register for this using your developer account at the URL below.

https://aws.amazon.com

Aws offers a broad range of online services, some of which are free, and some of which are priced.

From the Aws portal site you can create a free account.  You can sign up for it at the top right of the window.

aws-create

Doing so is quite a straight forward process.  You have to supply your name, email, etc.  But you also have to enter your credit card number.

Even though we will be using the so called “Free Tier” of AWS (services that aren’t charged for) you have to enter a valid credit card number.

It’s also important you enter a correct phone number. In order to verify your identity Amazon calls you when finishing up the sign up process.  It is an automated voice controlled computer asking you to enter a pin number.  After completing the process you can access the services from the Aws Portal.

The service we’ll need to add extra skills to our mirror is called Aws Lambda.

lambda

In short: it’s just a place where you can paste some code (Javascript, perl, and even c#) that will be executed in the cloud.  This code is referred to by the custom skills that can be created with Alexa.  After all, all custom skills do is call a web service somewhere.  This web service can also be some service started on your own hardware.  However, in my case, i’m not on a fixed IP with my internet provider, so it’s kind of hard to host a web server with one or more services attached to it.  Furthermore, I have to maintain it, and make sure it’s “always up”.

Using the lambda services you get server time on one of the servers of Amazon, executing that piece of code every time you ask it to.  Simple as that.  The service is free… at least, almost free.  You get 1.000.000 free calls per month.  That’s one million.  1 with six zeroes.  That’s a whole lotta command to ask the mirror in one month.  Basically, it means: free.

You can find some more extra free features at this URL : https://aws.amazon.com/free/

Extra storage, computing time and database hosting, basically for free.  Some paying services are free up to 12 months.

A second service we will be using is the DynamoDB service.  This is a service which is also included in the “Free” tier.  The screenshot below shows the details.

schermafbeelding-2017-01-30-om-23-02-16

According to me, quite enough to handle our requests.  So basically : free.  That’s great.

My intention is fairly simple.  I will be writing the details of the commands to a table in DynamoDB, in some sort of queue.  This queue can then be polled by the mirror. Furthermore, I will be able to maintain a log of all queries, and some stats on the mirror.  These are some nice extra perks.

Someone on the Magic Mirror portal also pointed me towards a solution somebody else made using the AWS IoT service (Internet of Things), in which I would have to register the Raspberry Pi as a device to be addressed in order to execute commands I send it.  In the end it would probably be a good solution, but when browsing the pricing model I saw it was free up until 12 months.  According to not a durable solution, so I’m going for the DynamoDB solution because it provides me with more extensive features, and can plug in to my existing architecture.

Let’s get cracking!

 

 

A minor setback

Discovering that getting non-audio feedback is not supported using Alexa Voice Services.

Now it’s time to get the feedback from Alexa to our mirror. I have been browsing the the Java code of the AVS Java client, but I didn’t find something really conclusive on how to get the content of the response.  So I’ve been browsing the Amazon developer forums.

You can get access to the Amazon forums if you have an Amazon developer account.  Just surf to http://developer.amazon.com to get an account.

After some careful research I was faced with a major disappointment, and frankly a minor setback.

I found several forum posts stating that getting the feedback in the form of a card (which is the content being displayed in an app if you create a companion app for example), is not available using Alexa Voice Services.  It seems that an enhancement request has been logged to add this functionality, but the most recent post on this dates from a few days ago (we’re talking January 2017).

I’ve posted the links to the forum posts below.

https://forums.developer.amazon.com/questions/10444/how-to-get-the-non-audio-data-from-the-alexa-skill.html

https://forums.developer.amazon.com/questions/10871/avs-card-data.html

https://forums.developer.amazon.com/questions/54337/voice-service-meta-data.html

So, it’s time to get creative.  Keep you posted in my next articles.

Signalling the mirror when Alexa listens

Editing the Java Client code to send messages when Alexa listens.

In this article I’ll be showing you the steps I took to signal the mirror when Alexa starts and stops listening.  Well, actually, I’m not alerting the mirror just yet, I’m sending a message to the debug hub.

In my last post I showed you how I was able to make a connection from the Java client to the debug hub using HTTP.  That’s great, but it was a quick and dirty solution.  We’re going to make this a bit more robust. I’ll start by adding a new class the the com.amazon.alexa.avs package, and call it “MirrorInterface“.  You can see it near the bottom of the screenshot:

Schermafbeelding 2017-01-25 om 20.19.02.png

The content is pretty simple.  I’ll be using the code to send the message and encapsulate it into a private method like so :

Schermafbeelding 2017-01-25 om 20.18.27.png

Notice how I already added support for both a command and extra content.  For now I’ll be focussing on only a command.  Also, the URL is added hardcoded too.  I plan on reading this from a config file, but that’s only a “nice to have” right now.

I intentionally made it private as the public methods all refer to this one.  I created a few, each serving their own purpose :

Schermafbeelding 2017-01-25 om 20.18.08.png

Notice the “VoiceActivated” and “VoiceDeactivated” methods.  I’ll be incorporating them in the code of the Java Client.  If you look in the “AVSController” java file you will see the following method overrides : “recordingStarted” and “recordingEnded”.  They get called whenever Alexa starts to listen, and stops listening.   That’s where I’ll be adding my code :

Schermafbeelding 2017-01-25 om 20.17.29.png

And that’s it.  I issue a question to Alexa, and voila, the result in the debug hub.

Schermafbeelding 2017-01-25 om 20.17.14.png