Instructing the mirror to show a picture

Write some code to show a picture on the mirror as instructed by the debug hub.


As for the next step of the project I need to make sure that the mirror can display an image when it is asked to do so.  We need to take a few steps in order to accomplish that.

As a first step I will extend the debug hub to mimic the Alexa behaviour of adding an “SHOW-PICTURE” command to the DynamoDB command queue table.

For this, I will be extending the already existing “messaging.html” file.

But first, for good measure I’ll add another javascript file to contain a function which I would potentially need on other places as well :

Schermafbeelding 2017-02-12 om 18.22.19.png

The content is fairly simple :

Schermafbeelding 2017-02-12 om 18.23.15.png

It’s very similar to what the lambda function is doing actually.  Instantiate an AWS object and configure it.  Put an item and a payload in the command queue.

Ok, so now to extend the messaging section of the debug hub with another text area.

Schermafbeelding 2017-02-12 om 18.24.43.png

I gave the 2 text area’s an Id : command & content, and the button as well : submit.

As for the code : a simple call to the function we just created.

Schermafbeelding 2017-02-12 om 18.25.57.png

Notice how the text area values are obtained using a jQuery command.

Let’s test this.

Schermafbeelding 2017-02-12 om 18.26.45.png

Click the send button, an there’s the result :

Schermafbeelding 2017-02-12 om 18.27.16.png

The next step is to extend the magic mirror module for the Alexa interface.  In an earlier post I showed you how I used a “handleCommands” function searches for the items in the DynamoDB table and reads them to pass them on using the “sendSocketNotification” function.  I’ve added a little bit of code here.

Schermafbeelding 2017-02-12 om 18.28.58.png

Notice how I construct a JSON object and add the command and the content to it.  This is the payload of the “sendSocketNotification“.  This way, the module knows the command, but also the content it needs to display.

In the mirror module itself, this type of command is now handled by this piece of code in the socketNotificationReceived function :

Schermafbeelding 2017-02-12 om 18.31.45.png

Essentially, this is also fairly simple.  All it does is construct its own payload from the payload it’s receiving.  This is passed to a notification with as type “INFO-DISPLAY“.  This notification is to be handled by another module that takes care of the display functionality.  Notice how I deliberately didn’t call it “show picture” or anything.  Theoretically this info could be something totally different from a picture, so I’ve made it a bit more generic.

As it happens, I don’t have this module yet, so I decided to create it.  Here it is :

Schermafbeelding 2017-02-12 om 18.34.26.png

Our next little module.  Its configuration is easy :

Schermafbeelding 2017-02-12 om 18.35.01.png

Just put it in the middle of the screen.  That’s it.  No fuss (for now).

The code is pretty straight forward.  The standard stuff for a basic mirror module, but the “notificationReceived” function looks like this :

Schermafbeelding 2017-02-12 om 18.35.54.png

Here I check if the notification is “INFO-DISPLAY”.  Notice how I don’t check for the “type” attribute of the payload yet. I’m just assuming that it’s some name of someone or something I’m getting passed a long, of which I need to show the picture.

The showing of the picture is setting the “src” attribute of an image, and setting it to visible.  I’m sorry if I’m violating some copyrights here by taking an online image, but that’s just the easiest way to test this.

The “this.imagepointer” is a variable that is created in the “getDom” function of the magic mirror module.  It looks like this :

Schermafbeelding 2017-02-12 om 18.38.37.png

Just create an “img” tag and append it to the wrapper.  Reset the “src” attribute, give it some dimensions and hide it.  From here on, the “this.imagepointer” can be used in the mirror module without a problem, and that’s just what happens in the “notificationReceived” function.

I’ve got a node helper for this one, but it doesn’t do anything for now.  That’s the next step: have it get a picture from the content that it has been passed, and pass it back to the mirror module to be shown.

But that’s for the next post.  Here is the result for now :

Schermafbeelding 2017-02-12 om 18.42.55.png


Making a connection to Wolfram Alpha

Connecting to the Wolfram Alpha service to get information on a certain subject.

Now we’ve got Alexa talking to our mirror, it’s time to go one step further.  I want to ask Alexa to show a picture of someone or something on the mirror when I ask it to.

I’ll be doing this in a few steps.

  1. I want to be able to do this in the Debug hub.  Period.  No voice interaction or anything.
  2. I want to be able to ask Alexa to show a picture by sending the command on to the mirror, and let the mirror search for an image.  In this scenario Alexa won’t respond with audio (or at least nothing meaningful), but only send the command.
  3. The last scenario is let Alexa search for an image, and have it explain something to me.  In this scenario I want to be able to make a distinction between only visual feedback or both audio and visual feedback.

The scenario whereby there is only audio feedback already exists : that’s the Alexa service itself.

So let’s get cracking.  The first thing to decide is which service to use to get images.  There are a few options.  The first one that comes to mind is Google : the custom search API does something like that.  More information here.  That’s cool.  You get 100 free API calls per day.  That’s 3100 in a long month.

An alternative is Wolfram Alpha.  It is very similar, but you “only” get 2000 calls per month.  I kind of like this one because it gives you a lot of info, and days with a lot of calls can compensate for days with little calls.

Anyway.  I might make an alternative for both, and combine them.  Let’s see.

So let’s start with Wolfram Alpha.  You can get started with API development here.

It’s quite straight forward.  Just register with Wolfram Alpha as a user, and next just create a new application.

Schermafbeelding 2017-02-06 om 20.12.49.png

Just give it a name and a description.

Schermafbeelding 2017-02-06 om 20.13.38.png

And there you have it.

Schermafbeelding 2017-02-06 om 20.14.02.png

Theoretically you could make multiple APP ID’s.  And hence increase the number of calls you can make.  However, it clearly states the maximum number of apps is 20.  I could make one for the debug hub, the mirror and Alexa separately actually.  But let’s start with a single one.

Next thing to do is extend the debug hub.  An extra tab at the left hand side, with a query text area and a button.

Schermafbeelding 2017-02-06 om 20.22.49.png

Writing the code is pretty easy.  Look at the extract below.


The only thing it does is construct an URL and concatenate the value of the text area in, followed by the App ID.  Referencing the text area is done via jQuery.  The text area element is called “querystring”.  The encodeURIComponent takes care of encoding the URL.  Spaces get replaced with %20 and other specific not allowed characters in an URL get replaced as well.

The result is displayed below.  Notice how it’s a whole bunch of information.  The image is in there, but so is a lot of other information we could leverage.  But that’s for later on.

Schermafbeelding 2017-02-06 om 20.32.19.png


Small note: getting the information back from Wolfram Alpha took quite some seconds.  That might prove an issue in the future, but for now, let’s be happy with what we’ve got.


The next step will be extracting the image from this information.  Keep you posted on that one.

Creating a generic link between the Java Client and the mirror

Linking the Alexa skill to the fitting behaviour on the smart mirror.

And now for the final part of the journey.  Getting the commands sent by the lambda functions to the mirror.  The idea is that the mirror will read from the CommandQueue dynamoDB table, execute the command, and remove the entry from the queue.

But we first need some trigger for the mirror to start polling for the commands in the queue.  To do that I’ll be adapting the AVS java client some more.  I’ll create two new functions in the mirror interface :

Schermafbeelding 2017-02-04 om 11.17.14.png

They are quite straight forward.  We can call them from the AVSController java class where the overrides for AlexaSpeechStarted and AlexaSpeechFinish are defined.

Schermafbeelding 2017-02-04 om 11.19.03.png

Rebuild the java client, and test that thing by issuing a question to Alexa and catching it in the Debug hub :

Schermafbeelding 2017-02-05 om 09.39.18.png

Works great.  Time to change the debug a bit to be able to send those functions to the mirror as well.  So some extra buttons in the Alexa.html file :

Schermafbeelding 2017-02-05 om 09.55.50.png

Now to for some extra code in the mirror software.  We first need to do some plumbing like install the AWS SDK.  While in the Smartmirrorproject subdirectory enter

sudo npm install aws-sdk

Then copy over the aws.json file from the root of the debug hub folder to the root of the smart mirror project folder.  We’ll be using the same settings to talk to our DynamoDB instance.


Now for adding some code to the mirror interface module node helper. Look a the start function override below :

Schermafbeelding 2017-02-05 om 18.38.42.png

First we configure the DynamoDB and instantiate the docClient variable.  We add 2 cases in the switch to handle both the speech-started and speech-stopped commands.
The first one looks for commands in CommandQueue table.  It does this by creating a params object and stating the name of the table.  Next, we need a projection list.  This is a list of columns that is to be returned by the docClient.  Well, something annoying came up here.  Apparently both the “session” and “timestamp” columns seem to be reserved words.  You can’t use them as a column name if you want to query them this way.  So, some refactoring needs to be done.  I renamed them to session_id and time_stamp.
The data is being read using the “scan” method.

The second command handler deletes the entry from the command queue.  But just in case there was an error earlier, and a command has been left in the queue, all the items have been read, and they will all be deleted in the for-loop.  The delete is being done by the “delete” method.  Below you can find the slightly refactored lambda function with the new column names.

Schermafbeelding 2017-02-05 om 18.37.13.png

Notice how both the delete and scan object require a callback, and a reference is passed to 2 existing functions.  Below you can see their definition.

Schermafbeelding 2017-02-05 om 18.38.49.png

The handle function sends the actual socket notification but also pushes the delete parameters for the delete command to a local array.  The QueueClear handler just issues some logging.

In order to make this work, we will need to change the ipWhiteList setting of the smart mirror.  If not, it will not accept requests from the java client.  You will also have to make sure the Java client on the Pi sends to the correct address and port.

Schermafbeelding 2017-02-05 om 18.37.33.png

Ok, so let’s test it.  Below is some output of a working session where admitted – by accident – 2 entries were in the queue, but they were both executed succesfully.

Schermafbeelding 2017-02-05 om 18.35.33.png

What’s left is add some extra functionality to allow showing the default modules as well.

That’s done pretty fast.  Extend the lambda function :

Schermafbeelding 2017-02-05 om 18.41.17.png

And provide an intent schema + utterances in the skill.


And that’s it!  A milestone in our development. We got Alexa talking to our start mirror.  Time to get to more advanced and cool stuff.

Connecting to DynamoDB using the debug hub

Modify the debug hub to create a connection to the DynamoDB table.

In my last post I showed you how I created a new table in DynamoDB.  Now it’s time to access it, and then the contents to our Debug Hub.

In order to do so I will need the AWS SDK (software development kit) installed in my project.  You can find the details here :

Just install it into the debug hub project.

sudo npm install aws-sdk

So I’ve extended the index.html page of the Debug Server to allow a new item in the sidebar, pointing to a newly create html page.  I called it “Dynamo DB”.

Schermafbeelding 2017-01-31 om 19.12.42.png

I will be connecting to the service with my own credentials.  That is : the credentials I created in my last post, the “smartmirror” user.

Schermafbeelding 2017-01-31 om 20.37.46.png

To do this efficiently, I will create a new JSON file in the root of my project.  In it, I will have to specify the following things :

  • The Access Key ID.  Information I got when creating the user.
  • The secret Access Key.  Also obtained after creating the user.
  • The region.  Very important.
  • The endpoint : the URL at which the AWS service can be reached. Notice how the region is part of the endpoint as well.

If you want to know the region, you can check out the details of the table created in DynamoDB.  It has a unique ARN (amazon resource name).  In it, you can see the code of the region you’re in.

Schermafbeelding 2017-01-31 om 19.43.55.png

So now for the code.  I will start by editing the dynamodb.html file to add a button and a grid.  The button will start a function that retrieves the data from the table. The grid will display the result.  This is quite straight forward.  We did that before when creating the page for the debug server.

Schermafbeelding 2017-01-31 om 20.37.25.png

Notice how I called the button “getcommands”.

The code is visible in the screenshot below.

Schermafbeelding 2017-01-31 om 20.37.39.png

There are some things to be said about this code.  At the bottom you can see the jQuery statement linking the button to the getCommandQueue function, which will do all the dirty work.

This function first requires the “aws-sdk” module, which we installed earlier.  It then configures it using the JSON file we created.  This way we can configure the connection with one single statement instead of having to set the access keys every single time.  That wouldn’t be very secure either.

Next a new variable is created called “params”.  This is an object that will be sent along with the query, and contains all the information needed to get data from the table :

  • The table name
  • An expression showing a filter
  • A mapping to the correctie field
  • A mapping to the correct value

the condition has a syntax whereby the field is represented by a placeholder prefixed with a # symbol, and a value placeholder prefixed with a : symbol.

These placeholders are resolved in the next 2 attributes of this parameter JSON : we map the column to the “Session” attribute, and we map the value to hardcoded “1234”, which is the key I specified when creating a record in the table.

And next, we’re all good to go.  The query method is called of the docClient object which is responsible for the communication with the DynamoDB  service.  The param is passed along, and there’s also a callback function.  It has 2 parameters.  When an error is returned, the “err” parameter will contain the details of the error.  When the call is successful, the data will be in the “data” parameter.  With this “data”, a foreach loop is executed, and it adds the result to the table.  The AddRow function can be seen below.

Schermafbeelding 2017-01-31 om 21.53.17.png

That that’s it.  Below is the result of this endeavour.

Schermafbeelding 2017-01-31 om 20.35.04.png

Good Luck!

Signalling the mirror when Alexa listens

Editing the Java Client code to send messages when Alexa listens.

In this article I’ll be showing you the steps I took to signal the mirror when Alexa starts and stops listening.  Well, actually, I’m not alerting the mirror just yet, I’m sending a message to the debug hub.

In my last post I showed you how I was able to make a connection from the Java client to the debug hub using HTTP.  That’s great, but it was a quick and dirty solution.  We’re going to make this a bit more robust. I’ll start by adding a new class the the package, and call it “MirrorInterface“.  You can see it near the bottom of the screenshot:

Schermafbeelding 2017-01-25 om 20.19.02.png

The content is pretty simple.  I’ll be using the code to send the message and encapsulate it into a private method like so :

Schermafbeelding 2017-01-25 om 20.18.27.png

Notice how I already added support for both a command and extra content.  For now I’ll be focussing on only a command.  Also, the URL is added hardcoded too.  I plan on reading this from a config file, but that’s only a “nice to have” right now.

I intentionally made it private as the public methods all refer to this one.  I created a few, each serving their own purpose :

Schermafbeelding 2017-01-25 om 20.18.08.png

Notice the “VoiceActivated” and “VoiceDeactivated” methods.  I’ll be incorporating them in the code of the Java Client.  If you look in the “AVSController” java file you will see the following method overrides : “recordingStarted” and “recordingEnded”.  They get called whenever Alexa starts to listen, and stops listening.   That’s where I’ll be adding my code :

Schermafbeelding 2017-01-25 om 20.17.29.png

And that’s it.  I issue a question to Alexa, and voila, the result in the debug hub.

Schermafbeelding 2017-01-25 om 20.17.14.png

Create a connection from the Java Client

Setting up a quick test for the connection between the Java client and the debug hub.

It’s time for the next phase in the project.  So far we’ve primarily focused on the “output” side of the story : displaying stuff on the mirror.  Now we’re going to focus on the “input” : accepting voice commands from Alexa Voice Services.

In this article I’ll be attempting to send a message from the Java client to the debug hub.

I’ll be very practical on this one, nothing too fancy.  I’ll be using one of the buttons on the window of the client to start a request.

I added all of this code to the file.

First, import some packages I’ll be using in the new code.

Schermafbeelding 2017-01-23 om 00.04.59.png

After some browsing, I found out that the “createMusicButton” method is responsible for adding the music buttons to the interface.  I’ll be using one of those buttons to simulate.  Below, you can see I commented out the default handler, and added my own call to the testMethod() method.

Schermafbeelding 2017-01-23 om 00.05.53.png

This method is fairly simple.  It just calls another message to do all the dirty work.  I added this level of indirection, because ultimately I want to separate the logic for the mirror interface.  There’s just some basic error handling involved here.

Schermafbeelding 2017-01-23 om 00.05.59.png

Below, you can find the actual message responsible for sending the command.

Schermafbeelding 2017-01-23 om 00.06.05.png

In the above code extract you can see a new URL object is created pointing to the static IP address of my Mac where the debug server is running . Port 3000 is the port it is listening on.  I added the path of the routing, and specified the “GET” method.  The rest of the code is basically some plumbing code to read the response from the debug hub.  In practice, hardly anything will happen with this response, because the Java client will have no visual feedback, apart from logging to the console.

Of course, the above code is very rudimentary.  I will have to refactor the code to be a more robust package in which I can call the get method with multiple parameters in multiple circumstances, and where the connection, for example, is read from the config file.  But all in due time.

Below, you can see what happens when I click the button.


Good luck!

Showing incoming messages in the debug hub

Adapting the debug hub to show incoming messages as they arrive.

In an earlier article I showed you how I set up the debug hub to accept external requests.  I did that just to control what was coming in as information, so I could program and configure the mirror accordingly.

This article show you how I set up a method to also see the incoming messages as they arrive.

Open the debug server page, and see how I added a table at the bottom.  It already contains a header, and I gave it an Id “message-table”.

Schermafbeelding 2017-01-22 om 19.48.51.png

Notice an extra function called AddRow with a jQuery statement adding a command and a content row to the table.  You can see the result below.

Schermafbeelding 2017-01-22 om 19.49.52.png

Below is this setup in action . I send a message just using my browser, and see how it appears in the list.