Voice Control Using HTML5

Adding voice control to Arduino projects has always been a challenge. The Arduino's modest processor isn't well suited for capturing, filtering, and deciphering audio, and additional hardware is needed to connect an audio input.

There are a handful of solutions that already exists for adding voice control, but these attempts are typically limited in their capabilities and the number of words they can recognize.

The Arduino Yún's ability to connect to the Internet and host webpages opens up a simpler way to achieve voice control by utilizing a web browser and HTML5's built in voice recognition functionalities.

This project will demonstrate how this method can be used to control the color of an RGB LED.

A webpage hosted by the Yún will allow users to speak in color commands using HTML5's Speech API. These commands will be sent to an Arduino sketch, which will interpret them and change the LED to the appropriate color.

To complete this project, you need:

  1. An Arduino Yún, connected to your local network over Ethernet or Wi-Fi.

  2. A microSD card, with the OpenWrt-Yun Linux file system expanded onto it.

  3. One common-cathode RGB light-emitting diode (LED).

  4. One 330 Ω resistor.

  5. A solderless breadboard and jumper wires.

Voice Recognition with HTML5

The Yún has the ability to act as a webserver that can host user created webpages. Any devices that are connected to the same network as the Yún can access these pages from a browser and be served with the webpage's content.

OpenWrt-Yun takes up most of the available space on the Yún's Atheros AR9331. To host your own webpages, it is best to store them on the microSD card.

When you expand the Linux file system onto the microSD card, an arduino directory and www subdirectory are created in the top-level of the card. A symbolic link, /www/sd, is also created within the Yun's main website directory, which links to this new folder /mnt/sda1/arduino/www/.

Any webpage files placed into the /arduino/www directory on the SD card can be accessed through the URL http://arduino.local/sd/.

For example, if you place an HTML file test.html into this folder, it can be accessed at http://arduino.local/sd/test.html

This project uses a sample HTML5 webpage that contains the basic functionality for voice recognition.

To install the website files onto the SD card:

  1. Login to the OpenWrt-Yun command line.

  2. Type the following command and then press Enter:

    cd /mnt/sda1/arduino/www/

  3. Type the following command and then press Enter:

    wget http://www.arduinomeetslinux.com/download/VoiceControl.zip

  4. Type the following command and then press Enter:

    unzip VoiceControl.zip

  5. Type the following command and then press Enter:

    rm VoiceControl.zip

A directory VoiceControl now exists inside /arduino/www/ and contains two sub directories and one HTML file:

css/ - Contains a cascading style sheet file that defines the webpage's formatting and layout.

img/ - Contains microphone icon images used on the webpage.

index.html - Contains the webpage conent, and the JavaScript code to handle the HTML5 Speech API processing.

On a device connected to the same network as the Yún, open a web browser and navigate to the webpage using the URL http://arduino.local/sd/VoiceControl/index.html

Test the website's voice recognition by clicking the microphone icon and speaking.

When the API determines that you've stopped speaking, your words will be translated to text and displayed in the text field. You can continue to speak and it will continue to translate until the microhpone is clicked again to turn it off.

The webpage makes use of HTML5's Speech Recognition API, which provides Javascript functionality for accessing the browser's audio stream, automatically detecting speech, and returning translated speech input as a string.

The webpage's JavaScript code creates an instance of the webkitSpeechRecognition object class. When the microphone button is clicked it calls this object's start() method, which initiates the browser's microphone and begins listening for verbal input.

When the API recognizes that the user has stopped talking, the object's onresult() method is called and has access to the translated speech. The JavaScript takes this translated speech and displays it in the page's text field.

Updating the Website Code

The website code needs to be updated so that the translated speech is not only displayed, but it is also sent to an Arduino sketch that will be running on the Yún.

The Yún's Mailbox REST API allows for message strings to be sent to a sketch by appending them to the end of the URL http://arduino.local/mailbox/. Any text that is added to the end of this URL is parsed out and can be read in by the sketch using the Bridge Library and Mailbox class.

By default, the Yún's REST API access is password protected. This must be set to Open to allow the webpage to send messages to the sketch using Mailbox. Access can be updated by logging into the main administation page at http://arduino.local, clicking on Configure, and setting the REST API access checkbox to OPEN.

To update the website file:

  1. Login to the OpenWrt-Yun command line.

  2. To edit the index.html file, type the following command and then press Enter:

    nano /mnt/sda1/arduino/www/VoiceControl/index.html

  3. Before the line that reads </head>, add the following on a new line:

    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>

  4. Under the line that reads final_span.innerHTML = final_transcript;, add the following lines:

    var url = "/mailbox/" + encodeURIComponent(final_transcript);
    $.get(url);

  5. Press Ctrl + X.

  6. Press Y and then press Enter.

The first line was added inside the page's <head> tag and gives the page access to jQuery, a Javascript library that includes simple functionality for making web requests.

The next lines of code are added into the JavaScript code block that is executed after the API has recognized and translated the user's input speech. The translated text is held in the variable final_transcript.

The text is encoded to remove all special characters, and added to the string "/mailbox/" to make up the url variable.

var url = "/mailbox/" + encodeURIComponent(final_transcript);

The next line makes an HTTP GET request to this URL string using jQuery's $.get() method.

$.get(url);

Since the website itself is being hosted by on the Yún, the full URL http://arduino.local/mailbox/+message is not nescessary, and just /mailbox/+message will work.

Making this request has the same result as if someone typed the URL into a browser. The Arduino's Mailbox REST API will parse out the message, which now contains the translated text, and make it available to the sketch.

Accepting Mailbox Messages in an Arduino Sketch

The sketch for this project needs to continually listen for incoming messages sent from the webpage, and take action when it recieves them.

Open the Arduino IDE and create a new sketch called VoiceControl. Add the following directives at the top of the file:

#include <Bridge.h>
#include <Mailbox.h>

In the sketch's setup() function, add the following code to initialize the Bridge, Mailbox, and Serial libraries:

void setup() {
  Bridge.begin();
  Mailbox.begin();
  Serial.begin(9600);
}

In the sketch's loop() function, add the following code:

void loop() {
  if (Mailbox.messageAvailable()) {
    String message;
    Mailbox.readMessage(message);
    processMessage(message);
  }
  delay(200);
}

Finally, add a function called processMessage() to your sketch using the following code:

void processMessage(String message) {
  Serial.println(message);
}

Upload this sketch to your Arduino and open the Serial monitor. To test it, again open a web browser and navigate to the webpage using the URL http://arduino.local/sd/VoiceControl/index.html

As you speak into the webpage this time your words will be displayed on the webpage, and also printed out in the Serial monitor.

With each iteration of the sketch's loop() function the messageAvailable() method of the Mailbox class is called. The method returns the integer length of any incoming messages that haven't been read yet. If there are no messages available, then it returns a zero.

When a message is avaiable it is read into a string variable usigng the readMessage() method. The sketch then passes this string into the processMessage() function where it is printed out to the Serial monitor.

Building the Circuit and Finishing the Sketch

To complete this project the sketch will be updated to control the color state of an RGB LED when specific words are passed to it. Messages such as 'red', 'blue', and 'yellow' will change the LED to their respective color, and 'Off' will turn the LED off.

A common-cathode RGB LED contains three LEDs – one red, one green, and one blue. It usually has four legs. The longest leg is usually the second one, and this is the cathode. Each LED in a common-cathode RGB LED shares the same connection to ground.

When using RGB LEDs, you need to add a current-limiting resistor to your circuit – the same as you do with normal LEDs.

To connect the RGB LED:

  1. Disconnect the Arduino from its power source.

  2. Connect the LED's common cathod pin to the Arduino's GND through a 330 Ω resistor.

  3. Connect the LED's red pin to Arduino pin D11.

  4. Connect the LED's green pin to Arduino pin D10.

  5. Connect the LED's blue pin to Arduino pin D9.

  6. Reconnect the Arduino's power supply.

Add these lines to the top of your sketch, underneath the #include statements, to define the LED digital pins:

int LED_R =  11;
int LED_G = 10;
int LED_B = 9;

Inside of the setup() function, add the following code to set the digital pins to OUTPUT:

pinMode(LED_R, OUTPUT);
pinMode(LED_G, OUTPUT);
pinMode(LED_B, OUTPUT);

Finally, replace the processMessage() function with the following:

void processMessage(String message) {
  message.toLowerCase();
  if (message == "red") {
    analogWrite(LED_R, 255);
    analogWrite(LED_G, 0);
    analogWrite(LED_B, 0);
    return;
  }
  if (message == "green") {
    analogWrite(LED_R, 0);
    analogWrite(LED_G, 255);
    analogWrite(LED_B, 0);
    return;
  }
  if (message == "blue") {
    analogWrite(LED_R, 0);
    analogWrite(LED_G, 0);
    analogWrite(LED_B, 255);
    return;
  }
  if (message == "yellow") {
    analogWrite(LED_R, 175);
    analogWrite(LED_G, 255);
    analogWrite(LED_B, 0);
    return;
  }
  if (message == "purple") {
    analogWrite(LED_R, 175);
    analogWrite(LED_G, 0);
    analogWrite(LED_B, 128);
    return;
  }
  if (message == "off") {
    analogWrite(LED_R, 0);
    analogWrite(LED_G, 0);
    analogWrite(LED_B, 0);
    return;
  }
}

String messages passed into this function will now be converted to lowercase and conditionally matched against a set of predefined strings - 'red', 'green', 'blue', 'yellow', 'purple', and 'off'.

To control the brightness and color of the LED, you can use pulse-width modulation (PWM) to set the individual LEDs in the RGB LED. Depending on how bright each LED is, your eyes see a blend of the colors.

When the input message matches one of these strings the LED pins are set to the appropriate values using analogWrite() to create the resulting color.

Upload the completed sketch to your Arduino and browse to project website on a device connected to the same network. Speak one of the defined color commands into your browser and the LED will now change to the appropriate color.

The code can easily be adapted for any new voice commands that you want the Arduino to respond to. All that is required is another conditional block added within the processMessage() function that checks against the message your looking for, and includes any lines of code to be executed if there is a match.

Source Code

VoiceControl.ino

Start Building Today

book cover Start Reading Now

Your guide to Yún developement. Arduino Meets Linux includes:

  • Over 320+ pages of content
  • OpenWrt-Yun Linux tutorials
  • Python and shell scripting
  • 7 in-depth Arduino projects

Take your Arduino projects to the next level. This book shows you how.