Connecting a camera to the Pi

Testing the camera on the Raspberry Pi

Ok, so today I got my new nifty gadget.  A camera for the Pi!  So here’s the unboxing.

I got it in a neat paper bag like this one :

16832778_2225601784330794_64021547_o.jpg

This is what it looks like.

16880823_2225601770997462_1406325668_o.jpg

I opened the casing and connected it. It’s fairly easy.  Just pull up the handles, put it in, and push the handles back down.

16879954_2225601744330798_1761360833_o.jpg

Now to configure this.  In order to do this, just log on to the pi and start raspi config :

sudo raspi-config

locate the Interfacing options

Schermafbeelding 2017-02-20 om 19.30.54.png

Locate the camera option

Schermafbeelding 2017-02-20 om 19.31.12.png

Enable it

Schermafbeelding 2017-02-20 om 19.31.22.png

and reboot.

Schermafbeelding 2017-02-20 om 19.31.39.png

After the reboot sequence has finished, it’s time to test this.  There are actually 2 command line tools you can use to capture a still image and a video image.  The first one is called raspistill.  It’s got tons of options really.  You should really take a look at them.  But the easiest way to do a test is using this command :

raspistill -o me.jpg

Where the -o parameter sets the output filename.

Schermafbeelding 2017-02-20 om 19.37.04.png

So here it is :

Schermafbeelding 2017-02-20 om 19.38.53.png

The second one is raspivid

Here’s the syntax :

raspivid -o test.h264 -t 3000

in this command the .h264 file is the output. the -t parameter sets the number of seconds to record. It’s expressed in milliseconds, so 3000 means 3 seconds.

Schermafbeelding 2017-02-20 om 19.43.19.png

Good luck testing this!

Getting and showing the Wolfram alpha picture

Using the wolfram alpha API to get a picture of what was passed along from the lambda function.

In this post I’ll be showing you how I got the picture from Wolfram Alpha and showed it on the mirror.

The newly created MMM-infodisplay module accepts a notification called INFO-DISPLAY.  All it does is pass along the notification and the payload to the node helper using the sendSocketNotification method.

Schermafbeelding 2017-02-17 om 20.31.04.png

Inside the node helper of the MMM-infodisplay I created a new function called getValues.  Essentially, all it does is loop through the keys of a JSON object.  If the property is an object, it will recursively loop through its contents as well.  It will push the results in to an array and return it.

Schermafbeelding 2017-02-17 om 20.34.10.png

In the socketNotificationReceived function I’m handling the notifications passed a long from the module.  First, the URL is built. You can find all of the info on how to build up the URL on the Wolfram Alpha site here.

The encodeURIComponent function will take care of the characters that aren’t supported in an URL like a slash or a space.  I’m just passing on the content part of the payload.  This is set in the Lambda function in one of my earlier posts.  Other than that I’m asking to get the result in a JSON format, and only retain the part of the output containing an image.

I’m using the request module to get an URL and parse the body using a call back function.  Im parsing out the JSON from the body and use the getValues function to get the “src” attribute.  If you investigate the result of the API call well you can see that the URL is contained in this attribute.  If I find something in the returned array I return it using the “display-image” notification in the sendSocketNotification function sending the info back to the MMM-infodisplay module.

Schermafbeelding 2017-02-17 om 20.34.32.png

Here’s the code I do when receiving this notification.  Just set the source of the image to the payload and set the image to visible.

Schermafbeelding 2017-02-17 om 20.31.37.png

 

Time to test. I just uploaded a SHOW-PICTURE command with content “Michael Jackson” using the debug hub.

Schermafbeelding 2017-02-17 om 20.31.59.png

Let’s see the result :

Schermafbeelding 2017-02-17 om 20.33.39.png

Works like a charm!

Instructing the mirror to show a picture

Write some code to show a picture on the mirror as instructed by the debug hub.

As for the next step of the project I need to make sure that the mirror can display an image when it is asked to do so.  We need to take a few steps in order to accomplish that.

As a first step I will extend the debug hub to mimic the Alexa behaviour of adding an “SHOW-PICTURE” command to the DynamoDB command queue table.

For this, I will be extending the already existing “messaging.html” file.

But first, for good measure I’ll add another javascript file to contain a function which I would potentially need on other places as well :

Schermafbeelding 2017-02-12 om 18.22.19.png

The content is fairly simple :

Schermafbeelding 2017-02-12 om 18.23.15.png

It’s very similar to what the lambda function is doing actually.  Instantiate an AWS object and configure it.  Put an item and a payload in the command queue.

Ok, so now to extend the messaging section of the debug hub with another text area.

Schermafbeelding 2017-02-12 om 18.24.43.png

I gave the 2 text area’s an Id : command & content, and the button as well : submit.

As for the code : a simple call to the function we just created.

Schermafbeelding 2017-02-12 om 18.25.57.png

Notice how the text area values are obtained using a jQuery command.

Let’s test this.

Schermafbeelding 2017-02-12 om 18.26.45.png

Click the send button, an there’s the result :

Schermafbeelding 2017-02-12 om 18.27.16.png

The next step is to extend the magic mirror module for the Alexa interface.  In an earlier post I showed you how I used a “handleCommands” function searches for the items in the DynamoDB table and reads them to pass them on using the “sendSocketNotification” function.  I’ve added a little bit of code here.

Schermafbeelding 2017-02-12 om 18.28.58.png

Notice how I construct a JSON object and add the command and the content to it.  This is the payload of the “sendSocketNotification“.  This way, the module knows the command, but also the content it needs to display.

In the mirror module itself, this type of command is now handled by this piece of code in the socketNotificationReceived function :

Schermafbeelding 2017-02-12 om 18.31.45.png

Essentially, this is also fairly simple.  All it does is construct its own payload from the payload it’s receiving.  This is passed to a notification with as type “INFO-DISPLAY“.  This notification is to be handled by another module that takes care of the display functionality.  Notice how I deliberately didn’t call it “show picture” or anything.  Theoretically this info could be something totally different from a picture, so I’ve made it a bit more generic.

As it happens, I don’t have this module yet, so I decided to create it.  Here it is :

Schermafbeelding 2017-02-12 om 18.34.26.png

Our next little module.  Its configuration is easy :

Schermafbeelding 2017-02-12 om 18.35.01.png

Just put it in the middle of the screen.  That’s it.  No fuss (for now).

The code is pretty straight forward.  The standard stuff for a basic mirror module, but the “notificationReceived” function looks like this :

Schermafbeelding 2017-02-12 om 18.35.54.png

Here I check if the notification is “INFO-DISPLAY”.  Notice how I don’t check for the “type” attribute of the payload yet. I’m just assuming that it’s some name of someone or something I’m getting passed a long, of which I need to show the picture.

The showing of the picture is setting the “src” attribute of an image, and setting it to visible.  I’m sorry if I’m violating some copyrights here by taking an online image, but that’s just the easiest way to test this.

The “this.imagepointer” is a variable that is created in the “getDom” function of the magic mirror module.  It looks like this :

Schermafbeelding 2017-02-12 om 18.38.37.png

Just create an “img” tag and append it to the wrapper.  Reset the “src” attribute, give it some dimensions and hide it.  From here on, the “this.imagepointer” can be used in the mirror module without a problem, and that’s just what happens in the “notificationReceived” function.

I’ve got a node helper for this one, but it doesn’t do anything for now.  That’s the next step: have it get a picture from the content that it has been passed, and pass it back to the mirror module to be shown.

But that’s for the next post.  Here is the result for now :

Schermafbeelding 2017-02-12 om 18.42.55.png

Cheers!

Extending the lambda function to get user input

Extending the lambda function to interpret the query for a picture of someone or something.

What’s next?  I want to extend the lambda function so I can ask to mirror to show me a picture of something or someone.  The mirror will interpret this and get the picture for me.  But first things first.  I’ll be extending my lambda function to interpret slots.  A slot is a sort of parameter you can pass in your Alexa intent schema.  Something like “show me a picture of <abc>”.  In this example <abc> is a slot of type LITERAL.  Alexa will interpret you sentence and automatically assign what you said to the slot.  It’s up to the lambda function to extract this info and do something with it.

Below you can find my lambda function.  I added another intent at the bottom.  I called it “ShowPictureIntent”, and I will be configuring it later on in my Alexa intent schema.

Schermafbeelding 2017-02-10 om 20.49.04.png

So what do we see here?  First we check if there are slots in the intent, and if there are, if an attribute is present called “Content”.  This is an arbitrary chosen attribute name.  It could be anything, but I chose to call it content, because it might server another purpose in the future as well.  If there’s no such thing, it will let Alexa say that it didn’t understand me.

If it did, however, the command is issued to the dynamoDB table.  I created a new command called “SHOW-PICTURE”.  Something my Alexa interface will have to deal with, and I put the content of the slot into a new attribute, also called “Content”.

In order to test this, I have to change the test event configuration.  You can do that in the top menu.

Schermafbeelding 2017-02-10 om 20.23.40.png

Here you can define some JSON, and configure the correct intent, and issue a slot.

Schermafbeelding 2017-02-10 om 20.48.38.png

Here i’m searching for a picture of George Michael.  So here’s the result of the test :

Schermafbeelding 2017-02-10 om 20.23.30.png

And how the command shows up in the DynamoDB table.

Schermafbeelding 2017-02-10 om 20.23.21.png

Now for the skill setup.  We adjust the intent schema like so :

Schermafbeelding 2017-02-10 om 20.50.03.png

we define a new intent, with the same name as we’re handling in the function.  And we define a slot, with name “Content”, just as we expect it.  We want to tell the skill to expect an arbitrary string.  This can be done by assigning the type “AMAZON.LITERAL”.  For more info on the different slot types, you can see this documentation.

The only thing left to configure is the utterances.  Like so :

Schermafbeelding 2017-02-10 om 20.50.09.png

Notice how I created 2 different utterances, and the syntax for referencing the slot.  Because it’s a slot of type literal, we need to tell Alexa an example of what we’re expecting.  What comes before the pipe symbol is crucial here.  We want it to expect a first name and a last name.  Otherwise, if you just type “name” for example and if you ask it for “George Michael”, it will pass on “Michael” ; just the last word.  Let’s test it :

Schermafbeelding 2017-02-10 om 20.54.32.png

I added some some more options to be complete :

Schermafbeelding 2017-02-11 om 12.57.44.png

Making a connection to Wolfram Alpha

Connecting to the Wolfram Alpha service to get information on a certain subject.

Now we’ve got Alexa talking to our mirror, it’s time to go one step further.  I want to ask Alexa to show a picture of someone or something on the mirror when I ask it to.

I’ll be doing this in a few steps.

  1. I want to be able to do this in the Debug hub.  Period.  No voice interaction or anything.
  2. I want to be able to ask Alexa to show a picture by sending the command on to the mirror, and let the mirror search for an image.  In this scenario Alexa won’t respond with audio (or at least nothing meaningful), but only send the command.
  3. The last scenario is let Alexa search for an image, and have it explain something to me.  In this scenario I want to be able to make a distinction between only visual feedback or both audio and visual feedback.

The scenario whereby there is only audio feedback already exists : that’s the Alexa service itself.

So let’s get cracking.  The first thing to decide is which service to use to get images.  There are a few options.  The first one that comes to mind is Google : the custom search API does something like that.  More information here.  That’s cool.  You get 100 free API calls per day.  That’s 3100 in a long month.

An alternative is Wolfram Alpha.  It is very similar, but you “only” get 2000 calls per month.  I kind of like this one because it gives you a lot of info, and days with a lot of calls can compensate for days with little calls.

Anyway.  I might make an alternative for both, and combine them.  Let’s see.

So let’s start with Wolfram Alpha.  You can get started with API development here.

It’s quite straight forward.  Just register with Wolfram Alpha as a user, and next just create a new application.

Schermafbeelding 2017-02-06 om 20.12.49.png

Just give it a name and a description.

Schermafbeelding 2017-02-06 om 20.13.38.png

And there you have it.

Schermafbeelding 2017-02-06 om 20.14.02.png

Theoretically you could make multiple APP ID’s.  And hence increase the number of calls you can make.  However, it clearly states the maximum number of apps is 20.  I could make one for the debug hub, the mirror and Alexa separately actually.  But let’s start with a single one.

Next thing to do is extend the debug hub.  An extra tab at the left hand side, with a query text area and a button.

Schermafbeelding 2017-02-06 om 20.22.49.png

Writing the code is pretty easy.  Look at the extract below.

schermafbeelding-2017-02-06-om-20-34-41.png

The only thing it does is construct an URL and concatenate the value of the text area in, followed by the App ID.  Referencing the text area is done via jQuery.  The text area element is called “querystring”.  The encodeURIComponent takes care of encoding the URL.  Spaces get replaced with %20 and other specific not allowed characters in an URL get replaced as well.

The result is displayed below.  Notice how it’s a whole bunch of information.  The image is in there, but so is a lot of other information we could leverage.  But that’s for later on.

Schermafbeelding 2017-02-06 om 20.32.19.png

 

Small note: getting the information back from Wolfram Alpha took quite some seconds.  That might prove an issue in the future, but for now, let’s be happy with what we’ve got.

 

The next step will be extracting the image from this information.  Keep you posted on that one.

Creating a generic link between the Java Client and the mirror

Linking the Alexa skill to the fitting behaviour on the smart mirror.

And now for the final part of the journey.  Getting the commands sent by the lambda functions to the mirror.  The idea is that the mirror will read from the CommandQueue dynamoDB table, execute the command, and remove the entry from the queue.

But we first need some trigger for the mirror to start polling for the commands in the queue.  To do that I’ll be adapting the AVS java client some more.  I’ll create two new functions in the mirror interface :

Schermafbeelding 2017-02-04 om 11.17.14.png

They are quite straight forward.  We can call them from the AVSController java class where the overrides for AlexaSpeechStarted and AlexaSpeechFinish are defined.

Schermafbeelding 2017-02-04 om 11.19.03.png

Rebuild the java client, and test that thing by issuing a question to Alexa and catching it in the Debug hub :

Schermafbeelding 2017-02-05 om 09.39.18.png

Works great.  Time to change the debug a bit to be able to send those functions to the mirror as well.  So some extra buttons in the Alexa.html file :

Schermafbeelding 2017-02-05 om 09.55.50.png

Now to for some extra code in the mirror software.  We first need to do some plumbing like install the AWS SDK.  While in the Smartmirrorproject subdirectory enter

sudo npm install aws-sdk

Then copy over the aws.json file from the root of the debug hub folder to the root of the smart mirror project folder.  We’ll be using the same settings to talk to our DynamoDB instance.

 

Now for adding some code to the mirror interface module node helper. Look a the start function override below :

Schermafbeelding 2017-02-05 om 18.38.42.png

First we configure the DynamoDB and instantiate the docClient variable.  We add 2 cases in the switch to handle both the speech-started and speech-stopped commands.
The first one looks for commands in CommandQueue table.  It does this by creating a params object and stating the name of the table.  Next, we need a projection list.  This is a list of columns that is to be returned by the docClient.  Well, something annoying came up here.  Apparently both the “session” and “timestamp” columns seem to be reserved words.  You can’t use them as a column name if you want to query them this way.  So, some refactoring needs to be done.  I renamed them to session_id and time_stamp.
The data is being read using the “scan” method.

The second command handler deletes the entry from the command queue.  But just in case there was an error earlier, and a command has been left in the queue, all the items have been read, and they will all be deleted in the for-loop.  The delete is being done by the “delete” method.  Below you can find the slightly refactored lambda function with the new column names.

Schermafbeelding 2017-02-05 om 18.37.13.png

Notice how both the delete and scan object require a callback, and a reference is passed to 2 existing functions.  Below you can see their definition.

Schermafbeelding 2017-02-05 om 18.38.49.png

The handle function sends the actual socket notification but also pushes the delete parameters for the delete command to a local array.  The QueueClear handler just issues some logging.

In order to make this work, we will need to change the ipWhiteList setting of the smart mirror.  If not, it will not accept requests from the java client.  You will also have to make sure the Java client on the Pi sends to the correct address and port.

Schermafbeelding 2017-02-05 om 18.37.33.png

Ok, so let’s test it.  Below is some output of a working session where admitted – by accident – 2 entries were in the queue, but they were both executed succesfully.

Schermafbeelding 2017-02-05 om 18.35.33.png

What’s left is add some extra functionality to allow showing the default modules as well.

That’s done pretty fast.  Extend the lambda function :

Schermafbeelding 2017-02-05 om 18.41.17.png

And provide an intent schema + utterances in the skill.

schermafbeelding-2017-02-05-om-18-46-05

And that’s it!  A milestone in our development. We got Alexa talking to our start mirror.  Time to get to more advanced and cool stuff.

Creating the hide all modules Alexa skill

Creating our own custom smart mirror skill based upon the hello world example.

Now it’s time to create a new Alexa skill, dedicated to our smart mirror.  We will be expanding the functionality of the skill gradually, but let’s start with the most basic functionality : hiding the modules on the mirror.

You might remember from my last post that you need the US-East region in AWS to be able to make lambda functions. However, if you remember the post where I created the first table in the DynamoDB setup, it was created in a different region.  You’ll need to complete those steps again in the US-East region in order for the lambda function and the DynamoDB service to communicate.  Check out this post for more info.  Note that you’ll have to change you aws.config file as well in order to connect to it using the debug hub.

 

Anyways, let’s create our function.

The first thing I did was copy the “Hello world” example on my disk to a new folder.  I opened it in Atom and replaced the “Hello world” references to SmartMirror references, like this :

Schermafbeelding 2017-02-04 om 10.27.51.png

That’s a good starting point. That way I was able to zip the contents  and upload it to AWS console.  The rest of the editing can be done inside the AWS console.

When configuring the function you can choose the settings below.  I created a new role from a template and chose the basic lambda execution rule.  Once saved the config looks like this :

Schermafbeelding 2017-02-03 om 23.23.17.png

From experience I know that using this role will cause the lambda function to give an error when accessing the DynamoDB table.  The reason is a lack of permissions.  This is easy to solve.  You have to open the IAM module.  The one you used in an earlier post to create the SmartMirror user.

Locate the role by navigating to the roles in the left hand pane, and in the Permissions tab, click “Attach Policy”.  Locate the “AmazonDynamoDBFullAccess” policy.  In fact, this is a bit overkill.  You could setup a policy to only access the CommandQueue table, but in our case this project is only intended for personal use, so i’m not worried about security right now.

Schermafbeelding 2017-02-03 om 23.18.22.png

Now that we’ve got the permissions thing out of the way, we can start looking at the code.

Go to the code of the function, and on top enter the following code :

Schermafbeelding 2017-02-03 om 23.21.10.png

That way you’ll include the AWS SDK, and instantiate a new DynamoDB interface.

In the IntentHandlers function : create a new entry for a “HideAllModulesIntent”, which we’ll be defining later on in the Alexa Skill.  The code for it is displayed below :

Schermafbeelding 2017-02-04 om 10.20.16.png

Small note: the code doesn’t contain the definition of the “Message” variable, just add it above these lines of code.

Notice how it uses the “putItem” method.  It takes a parameter (in the form a JSON format) and a callback function.  The parameter contains

  • The tablename
  • The content of the item

Notice the special syntax.  Every item is a different JSON attribute as well.  The callback function is fairly simple.  If the “err” object is not undefined, it means there’s been an error, and we need to act accordingly.  If Not, it succeeded.  Note that we call the “response.tellWithCard” function in both branches inside the callback function. You would think that’s not very optimal and put it behind the “putItem” call.  Well, been there done that.  That doesn’t work.  It turns out that lambda function returns when the “tellWithCard” function is called, and when doing that just after the “putItem” call doesn’t give the “putItem” enough time to complete its task. That means no error will be returned, but the callback function won’t be called either.  The solution is the code above.

Save the function.  You can test it with the button above.

Schermafbeelding 2017-02-03 om 23.20.49.png

When doing so, you can see the result :

Schermafbeelding 2017-02-03 om 23.20.53.png

And the log

Schermafbeelding 2017-02-03 om 23.21.00.png

Let’s log on to the DynamoDB console and refresh the contents of the table :

Schermafbeelding 2017-02-04 om 10.16.25.png

That worked like a charm.

Now let’s create the Alexa Skill.  Go to the Alexa Skills overview and create a new one.  Empty template & settings below.

Schermafbeelding 2017-02-04 om 10.52.48.png

I changed the invocation name of the “hello world” sample to “hello world”, so i can use “smart mirror” for this one.  If I were to choose the same one, Alexa wouldn’t know which one to choose.

The interaction mode is shown below and resembles the “hello world” example, only customised for the smart mirror.

Schermafbeelding 2017-02-03 om 23.22.46.png

Configure the endpoint with the ARN of the lambda function.  I’ve got no screenshot of that, but it’s quite straight forward.

You can easily test it :

Schermafbeelding 2017-02-03 om 23.23.48.png

And there we have it.  Remember to change the APP_ID in the lambda function code in order to let them communicate.

If you change the Session ID parameter in the code of the debug hub, you can also see the output in the debug hub :

Schermafbeelding 2017-02-04 om 11.08.55.png

Good luck!