Getting and showing the Wolfram alpha picture

Using the wolfram alpha API to get a picture of what was passed along from the lambda function.

Advertisements

In this post I’ll be showing you how I got the picture from Wolfram Alpha and showed it on the mirror.

The newly created MMM-infodisplay module accepts a notification called INFO-DISPLAY.  All it does is pass along the notification and the payload to the node helper using the sendSocketNotification method.

Schermafbeelding 2017-02-17 om 20.31.04.png

Inside the node helper of the MMM-infodisplay I created a new function called getValues.  Essentially, all it does is loop through the keys of a JSON object.  If the property is an object, it will recursively loop through its contents as well.  It will push the results in to an array and return it.

Schermafbeelding 2017-02-17 om 20.34.10.png

In the socketNotificationReceived function I’m handling the notifications passed a long from the module.  First, the URL is built. You can find all of the info on how to build up the URL on the Wolfram Alpha site here.

The encodeURIComponent function will take care of the characters that aren’t supported in an URL like a slash or a space.  I’m just passing on the content part of the payload.  This is set in the Lambda function in one of my earlier posts.  Other than that I’m asking to get the result in a JSON format, and only retain the part of the output containing an image.

I’m using the request module to get an URL and parse the body using a call back function.  Im parsing out the JSON from the body and use the getValues function to get the “src” attribute.  If you investigate the result of the API call well you can see that the URL is contained in this attribute.  If I find something in the returned array I return it using the “display-image” notification in the sendSocketNotification function sending the info back to the MMM-infodisplay module.

Schermafbeelding 2017-02-17 om 20.34.32.png

Here’s the code I do when receiving this notification.  Just set the source of the image to the payload and set the image to visible.

Schermafbeelding 2017-02-17 om 20.31.37.png

 

Time to test. I just uploaded a SHOW-PICTURE command with content “Michael Jackson” using the debug hub.

Schermafbeelding 2017-02-17 om 20.31.59.png

Let’s see the result :

Schermafbeelding 2017-02-17 om 20.33.39.png

Works like a charm!

Instructing the mirror to show a picture

Write some code to show a picture on the mirror as instructed by the debug hub.

As for the next step of the project I need to make sure that the mirror can display an image when it is asked to do so.  We need to take a few steps in order to accomplish that.

As a first step I will extend the debug hub to mimic the Alexa behaviour of adding an “SHOW-PICTURE” command to the DynamoDB command queue table.

For this, I will be extending the already existing “messaging.html” file.

But first, for good measure I’ll add another javascript file to contain a function which I would potentially need on other places as well :

Schermafbeelding 2017-02-12 om 18.22.19.png

The content is fairly simple :

Schermafbeelding 2017-02-12 om 18.23.15.png

It’s very similar to what the lambda function is doing actually.  Instantiate an AWS object and configure it.  Put an item and a payload in the command queue.

Ok, so now to extend the messaging section of the debug hub with another text area.

Schermafbeelding 2017-02-12 om 18.24.43.png

I gave the 2 text area’s an Id : command & content, and the button as well : submit.

As for the code : a simple call to the function we just created.

Schermafbeelding 2017-02-12 om 18.25.57.png

Notice how the text area values are obtained using a jQuery command.

Let’s test this.

Schermafbeelding 2017-02-12 om 18.26.45.png

Click the send button, an there’s the result :

Schermafbeelding 2017-02-12 om 18.27.16.png

The next step is to extend the magic mirror module for the Alexa interface.  In an earlier post I showed you how I used a “handleCommands” function searches for the items in the DynamoDB table and reads them to pass them on using the “sendSocketNotification” function.  I’ve added a little bit of code here.

Schermafbeelding 2017-02-12 om 18.28.58.png

Notice how I construct a JSON object and add the command and the content to it.  This is the payload of the “sendSocketNotification“.  This way, the module knows the command, but also the content it needs to display.

In the mirror module itself, this type of command is now handled by this piece of code in the socketNotificationReceived function :

Schermafbeelding 2017-02-12 om 18.31.45.png

Essentially, this is also fairly simple.  All it does is construct its own payload from the payload it’s receiving.  This is passed to a notification with as type “INFO-DISPLAY“.  This notification is to be handled by another module that takes care of the display functionality.  Notice how I deliberately didn’t call it “show picture” or anything.  Theoretically this info could be something totally different from a picture, so I’ve made it a bit more generic.

As it happens, I don’t have this module yet, so I decided to create it.  Here it is :

Schermafbeelding 2017-02-12 om 18.34.26.png

Our next little module.  Its configuration is easy :

Schermafbeelding 2017-02-12 om 18.35.01.png

Just put it in the middle of the screen.  That’s it.  No fuss (for now).

The code is pretty straight forward.  The standard stuff for a basic mirror module, but the “notificationReceived” function looks like this :

Schermafbeelding 2017-02-12 om 18.35.54.png

Here I check if the notification is “INFO-DISPLAY”.  Notice how I don’t check for the “type” attribute of the payload yet. I’m just assuming that it’s some name of someone or something I’m getting passed a long, of which I need to show the picture.

The showing of the picture is setting the “src” attribute of an image, and setting it to visible.  I’m sorry if I’m violating some copyrights here by taking an online image, but that’s just the easiest way to test this.

The “this.imagepointer” is a variable that is created in the “getDom” function of the magic mirror module.  It looks like this :

Schermafbeelding 2017-02-12 om 18.38.37.png

Just create an “img” tag and append it to the wrapper.  Reset the “src” attribute, give it some dimensions and hide it.  From here on, the “this.imagepointer” can be used in the mirror module without a problem, and that’s just what happens in the “notificationReceived” function.

I’ve got a node helper for this one, but it doesn’t do anything for now.  That’s the next step: have it get a picture from the content that it has been passed, and pass it back to the mirror module to be shown.

But that’s for the next post.  Here is the result for now :

Schermafbeelding 2017-02-12 om 18.42.55.png

Cheers!

Making a connection to Wolfram Alpha

Connecting to the Wolfram Alpha service to get information on a certain subject.

Now we’ve got Alexa talking to our mirror, it’s time to go one step further.  I want to ask Alexa to show a picture of someone or something on the mirror when I ask it to.

I’ll be doing this in a few steps.

  1. I want to be able to do this in the Debug hub.  Period.  No voice interaction or anything.
  2. I want to be able to ask Alexa to show a picture by sending the command on to the mirror, and let the mirror search for an image.  In this scenario Alexa won’t respond with audio (or at least nothing meaningful), but only send the command.
  3. The last scenario is let Alexa search for an image, and have it explain something to me.  In this scenario I want to be able to make a distinction between only visual feedback or both audio and visual feedback.

The scenario whereby there is only audio feedback already exists : that’s the Alexa service itself.

So let’s get cracking.  The first thing to decide is which service to use to get images.  There are a few options.  The first one that comes to mind is Google : the custom search API does something like that.  More information here.  That’s cool.  You get 100 free API calls per day.  That’s 3100 in a long month.

An alternative is Wolfram Alpha.  It is very similar, but you “only” get 2000 calls per month.  I kind of like this one because it gives you a lot of info, and days with a lot of calls can compensate for days with little calls.

Anyway.  I might make an alternative for both, and combine them.  Let’s see.

So let’s start with Wolfram Alpha.  You can get started with API development here.

It’s quite straight forward.  Just register with Wolfram Alpha as a user, and next just create a new application.

Schermafbeelding 2017-02-06 om 20.12.49.png

Just give it a name and a description.

Schermafbeelding 2017-02-06 om 20.13.38.png

And there you have it.

Schermafbeelding 2017-02-06 om 20.14.02.png

Theoretically you could make multiple APP ID’s.  And hence increase the number of calls you can make.  However, it clearly states the maximum number of apps is 20.  I could make one for the debug hub, the mirror and Alexa separately actually.  But let’s start with a single one.

Next thing to do is extend the debug hub.  An extra tab at the left hand side, with a query text area and a button.

Schermafbeelding 2017-02-06 om 20.22.49.png

Writing the code is pretty easy.  Look at the extract below.

schermafbeelding-2017-02-06-om-20-34-41.png

The only thing it does is construct an URL and concatenate the value of the text area in, followed by the App ID.  Referencing the text area is done via jQuery.  The text area element is called “querystring”.  The encodeURIComponent takes care of encoding the URL.  Spaces get replaced with %20 and other specific not allowed characters in an URL get replaced as well.

The result is displayed below.  Notice how it’s a whole bunch of information.  The image is in there, but so is a lot of other information we could leverage.  But that’s for later on.

Schermafbeelding 2017-02-06 om 20.32.19.png

 

Small note: getting the information back from Wolfram Alpha took quite some seconds.  That might prove an issue in the future, but for now, let’s be happy with what we’ve got.

 

The next step will be extracting the image from this information.  Keep you posted on that one.

Connecting to DynamoDB using the debug hub

Modify the debug hub to create a connection to the DynamoDB table.

In my last post I showed you how I created a new table in DynamoDB.  Now it’s time to access it, and then the contents to our Debug Hub.

In order to do so I will need the AWS SDK (software development kit) installed in my project.  You can find the details here :

https://www.npmjs.com/package/aws-sdk

Just install it into the debug hub project.

sudo npm install aws-sdk

So I’ve extended the index.html page of the Debug Server to allow a new item in the sidebar, pointing to a newly create html page.  I called it “Dynamo DB”.

Schermafbeelding 2017-01-31 om 19.12.42.png

I will be connecting to the service with my own credentials.  That is : the credentials I created in my last post, the “smartmirror” user.

Schermafbeelding 2017-01-31 om 20.37.46.png

To do this efficiently, I will create a new JSON file in the root of my project.  In it, I will have to specify the following things :

  • The Access Key ID.  Information I got when creating the user.
  • The secret Access Key.  Also obtained after creating the user.
  • The region.  Very important.
  • The endpoint : the URL at which the AWS service can be reached. Notice how the region is part of the endpoint as well.

If you want to know the region, you can check out the details of the table created in DynamoDB.  It has a unique ARN (amazon resource name).  In it, you can see the code of the region you’re in.

Schermafbeelding 2017-01-31 om 19.43.55.png

So now for the code.  I will start by editing the dynamodb.html file to add a button and a grid.  The button will start a function that retrieves the data from the table. The grid will display the result.  This is quite straight forward.  We did that before when creating the page for the debug server.

Schermafbeelding 2017-01-31 om 20.37.25.png

Notice how I called the button “getcommands”.

The code is visible in the screenshot below.

Schermafbeelding 2017-01-31 om 20.37.39.png

There are some things to be said about this code.  At the bottom you can see the jQuery statement linking the button to the getCommandQueue function, which will do all the dirty work.

This function first requires the “aws-sdk” module, which we installed earlier.  It then configures it using the JSON file we created.  This way we can configure the connection with one single statement instead of having to set the access keys every single time.  That wouldn’t be very secure either.

Next a new variable is created called “params”.  This is an object that will be sent along with the query, and contains all the information needed to get data from the table :

  • The table name
  • An expression showing a filter
  • A mapping to the correctie field
  • A mapping to the correct value

the condition has a syntax whereby the field is represented by a placeholder prefixed with a # symbol, and a value placeholder prefixed with a : symbol.

These placeholders are resolved in the next 2 attributes of this parameter JSON : we map the column to the “Session” attribute, and we map the value to hardcoded “1234”, which is the key I specified when creating a record in the table.

And next, we’re all good to go.  The query method is called of the docClient object which is responsible for the communication with the DynamoDB  service.  The param is passed along, and there’s also a callback function.  It has 2 parameters.  When an error is returned, the “err” parameter will contain the details of the error.  When the call is successful, the data will be in the “data” parameter.  With this “data”, a foreach loop is executed, and it adds the result to the table.  The AddRow function can be seen below.

Schermafbeelding 2017-01-31 om 21.53.17.png

That that’s it.  Below is the result of this endeavour.

Schermafbeelding 2017-01-31 om 20.35.04.png

Good Luck!

Visual feedback using matrix effect

Explaining a new MM module called the matrix effect allowing visual feedback on the mirror while alexa is listening.

Now the mirror can detect when Alexa is listening to our voice input, it’s time to provide a visual feedback on the mirror.  We could just do that using an alert or an extra module that displays a text.  Instead I want to do something fancier.

It’s time for a small side project in the form of a new magic mirror module displaying a female head in a “matrix” effect, just like in the movie with a bunch of green falling letters hinting a silhouette.  I’m saying green because I’m kind of preconditioned, but it can be any color anyway.

The idea behind this module is to create a JavaScript routine that would generate the the letters on a canvas.  But I want it to be a bit more generic, so other magic mirror owners would be able to download the module and display something else instead of a human head.   So the solution to that would be to make a configuration file, and store all the settings in the configuration file, and refer to it in the mirror’s config.js file.  So I went ahead and created the js file, the node helper and a subfolder for configuration files in json format :

Schermafbeelding 2017-01-29 om 23.18.43.png

Now for the module itself.  Below you can see an extract of the code of the javascript file.  most important is the start function here.

Schermafbeelding 2017-01-29 om 23.17.22.png

The first line in this function is crucial.  The setting for this effect are stored in the “alexa.json” file in the “effects” sub folder.  In order to be able to use the settings, the json file needs to be read.  However, access to the file system is not something you can do in JavaScript for the obvious security reasons.  That’s something that needs to be done with some server side scripting.  That’s where the node helper comes in.  Essentially, we need to tell the node helper to get the settings from a certain configuration file, which I called an animationFile in the code.  I deliberately called it an animation file because – in due time – I want to be able to animate it in multiple frames.  But for now, a single frame will do.

The name of the animation file is stored in the magic mirror’s configuration file :

Schermafbeelding 2017-01-29 om 23.22.42.png

Specifying it in the config tag means that it can be addressed from the JavaScript file using the “config” member.  This piece of information is passed on as a payload in the “MATRIX_ANIMATION_CONNECT” notification.  This notification is caught in the node helper like this :

Schermafbeelding 2017-01-29 om 23.19.02.png

Notice how, in the code extract above, the animation file is loaded using the loadAnimationFile method.  This called whenever the MATRIX_ANIMATION_CONNECT notification is received.

Never mind the test code in the screenshot.  I used it to know what the relative path to the animation file needed to be.  I’ll remove it in the actual code.  Once loaded it is being sent back using a notification using the “SEND_MATRIX_ANIMATION” constant.  This is received in the JavaScript file like this :

Schermafbeelding 2017-01-29 om 23.17.29.png

Notice how a local variable is assigned to the payload.

Another thing you can see in the JavaScript file is the initialization of an array called yPositions . We will by using this array in our JavaScript method that displays the matrix effect on the mirror.  It is initialized with a number of zeroes.  The size of the array is a calculation.  It divides the width of the canvas by the size of the font to calculate the exact number of columns there need to be in the matrix effect.  One last thing worth mentioning here is that I’ve created a custom .css file that contains some properties of the canvas.  It is referenced in the “getStyles” function, and looks like this :

Schermafbeelding 2017-01-29 om 23.18.51.png

In order to start and stop the matrix effect, the module relies on another module telling it to start and stop.  In practice this will of course be the Alexa interface module doing this.  It listens to 2 different notifications : START_MATRIXEFFECT en STOP_MATRIXEFFECT.  The first one hides all of the modules except for the matrix effect module, shows the module itself, and starts the matrix using the animation definition and the yPositions array.  The second one does the exact opposite.

Notice how “notificationReceived” is something else as “SocketNotificationReceived”.  The first one is used from communication between modules, the second one is use for communication between the JavaScript file and the node helper.

Schermafbeelding 2017-01-29 om 23.17.39.png

The screenshot below shows a very simple piece of code overriding the getDom method by creating a HTML canvas, and getting the context from it.  These objects are used to draw something on a HTML page.  You can find more information on this here.

Schermafbeelding 2017-01-29 om 23.17.47.png

The code extract below is the bulk of the module and requires some detailed explanation.

Schermafbeelding 2017-01-29 om 23.18.30.png

The runMatrix function actually starts an interval.  This is a piece of code (executed as a callback function in the call to the “setInterval” method, which is executed every x milliseconds.  These milliseconds are also a parameter in the same method, and as a matter of fact it is controlled by the “updateSpeed” parameter of the animation definition.

The function starts by getting the canvas and the context and filling the screen with a black background.  I haven’t made this variable yet, but I might do so in the future.  Inside the callback function the canvas and the context are set again because the scope of the callback function is different.  A matrix variable is created, containing the different column definitions in the animation file.  Next, the random() function is called.  It returns a random number between 0 and 1.  I use it to determine if and when an almost transparent overlay is to be put over the existing area.  Almost transparent, as you can see, the “alpha” of the rgba function component is 0.01.  This takes care of the fading letters in the matrix effect.

Next the font is set to the primary color (secondary colours are not supported yet) and the font is determined to be a certain font name with a certain width, depending on the animation file.  The next function is very important.  The map function loops through the yPositions matrix and executes a callback function for every element.  Where y is a coördinate on the y axis of the canvas and index is the index of a certain column.  The for loop determines if a letter is to be printed because it is in between the “start” and “end” tag of a section.  If it is, the text is printed on the y coördinate passed by the map function and the x coördinate based on a formula on the index.  Lastly, the yPositions array is updated so the letters drop to the bottom.  If they reach a certain depth, the yPositions array at that index is just rest.

The stopMatrix function just clears the interval allowing the loop to stop.

All this is configured in the animation definition.  It looks like this :

Schermafbeelding 2017-01-29 om 23.22.53.png

The configuration file is “prepared” to do some more exotic things like working with different colors and different animation frames already.  The code doesn’t use these properties yet.  Notice all of the settings used in the code like canvasWidth and canvasHeight, textPoints, etc.  The animation Matrix is a matrix in 3 dimensions :

  • Dimension 1 is the animation frame.  There’s only one in this example
  • Dimension 2 is the matrix contain all columns
  • Dimension 3 is an array within the column telling the mirror to either draw or not draw a letter depending on the start and stop number of pixels.  Remember, that the counting of the pixels starts from the top to the bottom.  Even though there is a “frame” attribute and a “column” attribute, the JavaScript code expects both the frames and columns to be in the correct order.  The “frame” and “column” attributes are ignored in the code, but serve as a pointer to the person creating the configuration file.  Specifying a start and end of -1 will make sure nothing is displayed in that particular column.

When running this module, the end result looks like this :

matrix-effect.gif

Granted, it still needs some work finetuning this.  And I’ll probably need to make a little tool to be able to create the animation definition file more efficient, but that’s for a later phase.

When this module is in a more finished phase I will upload it as a magic mirror module.

 

 

Creating the MMM-alexa-interface module

Creating my own module to act as a proxy for incoming alexa feedback.

So, for my next step I’ll be creating my own module, acting as the proxy for the Alexa feedback.  All it’s supposed to do is capture the commands and the content sent by Amazon, and act accordingly… based on some configuration of course.

So the first thing to do is create an extra folder, and create a .js file and a node helper.  You can find all the info about that in the Magic Mirror Builders section on creating your own module here.

The result in Atom looks like this :

Schermafbeelding 2017-01-27 om 18.54.48.png

So, below is the contents of my js file.  I’ve implemented a very simple start function.  Just updating the DOM every minute.

Schermafbeelding 2017-01-27 om 18.55.04.png

 

What’s more interesting is the socketNotificationReceived override.  This function receives all communications from the node helper of the function. The node helper is the file where you can specify your server side code.  You can communicate with the javascript file handling the front end story using sockets.  All I do in this function is handle all of the commands I’ve specified up until now.  You can se how the “hide-all-modules” and “show-default-modules” handle the showing and hiding of the modules using the MM variable.  Furthermore, I’ve implemented the “alexa-activated” and “alexa-deactivated” commands by sending a notification to the alert module I’ve installed on the mirror.  All it’s supposed to do is show an alert of type “notification”, and show a matching title and message.

Schermafbeelding 2017-01-27 om 18.55.11.png

The node helper code is fairly simple.  The start function override just starts a routing from the express framework, just like in our tests before.  All it does is send a socket notification to the front end where it is handled.

Schermafbeelding 2017-01-27 om 18.55.22.png

All we need to do now is add it to the configuration of the mirror like in the screenshot below. I’ve got no specific settings up until now, it’s also fairly simple.

Schermafbeelding 2017-01-27 om 18.54.29.png

Starting up the debug hub, and pressing the buttons.  Here’s the result.  Exactly what I was hoping for.

alexamirrorcomm.gif

Good luck!

 

 

Showing incoming messages in the debug hub

Adapting the debug hub to show incoming messages as they arrive.

In an earlier article I showed you how I set up the debug hub to accept external requests.  I did that just to control what was coming in as information, so I could program and configure the mirror accordingly.

This article show you how I set up a method to also see the incoming messages as they arrive.

Open the debug server page, and see how I added a table at the bottom.  It already contains a header, and I gave it an Id “message-table”.

Schermafbeelding 2017-01-22 om 19.48.51.png

Notice an extra function called AddRow with a jQuery statement adding a command and a content row to the table.  You can see the result below.

Schermafbeelding 2017-01-22 om 19.49.52.png

Below is this setup in action . I send a message just using my browser, and see how it appears in the list.

ezgif.com-video-to-gif.gif

Cheers!