Raspberry Pi 2 – Speech Recognition on device

This is a lengthy post and very dry, but it provides detailed instructions for how to build and install SphinxBase and PocketSphinx and how to generate a pronunciation dictionary and a language model, all so that speech recognition can be run directly on the Raspberry Pi, without network access. Don’t expect it to be as fast as Google’s recognizer, tho …

Creating the RASPBIAN boot MicroSD

Starting with the current RASPBIAN (Debian Wheezy) image, the creation of a bootable MicroSD Card is a well understood and well documented process.

Uncompressing the zip (again, there is no better tool than The Unarchiver, if you are on a Mac) reveals the 2015-02-16-raspbian-wheezy.img

With the MicroSD (inside an SD-Card adapter – no less than 8GB) inserted into the Mac, I run the df -h command in Terminal, to find out how to address the card. Today, it showed up as /dev/disk4s1 56Mi 14Mi 42Mi 26% 512 0 100% /Volumes/boot, which means, I run something like this, to put the boot image onto the MicroSD:

… after a few minutes, once the 3.28 GB have been written onto the card, I execute:

Customizing the OS

Once booted, using the sudo raspi-config allow the customization of the OS, which means that time-zone, keyboard, and other settings are adjusted, to closely match its environment.
I usually start (PI is already connected to the internet via Ethernet Cable) with

  • updating the raspi-config
  • expanding the filesystem
  • internationalization: un-check en-GB, check en-US.UTF-8 UTF-8
  • internationalization: timezone ..
  • internationalization: keyboard: change to English US
  • setting the hostname to translator, there are too many Raspberry Pis on my home network, to leave it at the default
  • make sure SSH is enabled
  • force audio out on the 3.5mm headphone jack

Microphone

Given the sparse analog-to-digital support provided by the Raspberry Pi, the probably best and easiest way to connect a decent Mic to the device, is using a USB microphone. I happen to have an older Logitech USB Mic, which works perfectly fine with the Pi.

After a reboot and now with the microphone connected, let’s get started ..
ssh pi@translator with the default password ‘raspberry’ gets me in from everywhere on my local network
cat /proc/asound/cards
returns
0 [ALSA ]: bcm2835 - bcm2835 ALSA
bcm2835 ALSA
1 [AK5370 ]: USB-Audio - AK5370
AKM AK5370 at usb-bcm2708_usb-1.2, full speed

showing that the microphone is visible and its usb extension.
Next, I edit alsa-base.conf to load snd-usb-audio like so:
sudo nano /etc/modprobe.d/alsa-base.conf
Edit
options snd-usb-audio index=-2
to
options snd-usb-audio index=0
and after a sudo reboot, cat /proc/asound/cards
looks like this
0 [AK5370 ]: USB-Audio - AK5370
AKM AK5370 at usb-bcm2708_usb-1.2, full speed
1 [ALSA ]: bcm2835 - bcm2835 ALSA
bcm2835 ALSA

Recording – Playback – Test

Before worrying about Speech Recognition and Speech Synthesis, let’s make sure that the basic recording and audio playback works.
Again, I have an USB Microphone connected to the Pi, as well as a speaker, using the 3.5mm audio plug.

Installing build tools and required libraries

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install bison
sudo apt-get install libasound2-dev
sudo apt-get install swig
sudo apt-get install python-dev
sudo apt-get install mplayer
sudo reboot

/etc/asound.conf

sudo nano etc/asound.conf and enter something like this:

Recording

The current recording settings can be looked at with:
amixer -c 0 sget 'Mic',0
and for me that looks something like this:

alsamixer -c 0 can be used to increase the capture levels. After an increase, it looks like this:

Playback

The current playback settings can be looked at with:
amixer -c 1
alsamixer -c 0 can be used to increase the volume. After an increase,
amixer -c 1
it looks like this:

Test Recording and Playback

With the mic switched on ..
arecord -D plughw:0,0 -f cd ./test.wav .. use Control-C to stop the recording.
aplay ./test.wav

With recording and playback working, let’s get into the really cool stuff, on-device speech recognition.

Speech Recognition Toolkit

CMU Sphinx a.k.a. PocketSphinx
Currently pocket sphinx 5 pre-alpha (2015-02-15) is the most recent version. However, there are a few prerequisites that need to be installed first ..

Installing build tools and required libraries

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install bison
sudo apt-get install libasound2-dev
sudo apt-get install swig
sudo apt-get install python-dev
sudo apt-get install mplayer

Building Sphinxbase

cd ~/
wget http://sourceforge.net/projects/cmusphinx/files/sphinxbase/5prealpha/sphinxbase-5prealpha.tar.gz
tar -zxvf ./sphinxbase-5prealpha.tar.gz
cd ./sphinxbase-5prealpha
./configure --enable-fixed
make clean all
make check
sudo make install

Building PocketSphinx

cd ~/
wget http://sourceforge.net/projects/cmusphinx/files/pocketsphinx/5prealpha/pocketsphinx-5prealpha.tar.gz
tar -zxvf pocketsphinx-5prealpha.tar.gz
cd ./pocketsphinx-5prealpha
./configure
make clean all
make check
sudo make install

Creating a Language Model

Create a text file, containing a list of words/sentences we want to be recognized

For instance ..

Upload the text file here: http://www.speech.cs.cmu.edu/tools/lmtool-new.html
and then download the generated Pronunciation Dictionary and Language Model

For the the text file mentioned above, this is what the tool generates:

Pronunciation Dictionary

Language Model

Looking carefully, the Sphinx knowledge base generator provides links to the just generated files, which make sit super convenient to pull them down to the Pi. For me it generated a base set with the name 3199:
wget http://www.speech.cs.cmu.edu/tools/product/1427343814_14328/3199.dic
wget http://www.speech.cs.cmu.edu/tools/product/1427343814_14328/3199.lm

Running Speech-recognition locally on the Raspberry Pi

Finally everything is in place, SphinxBase and PocketSphinx have been building installed, a pronunciation dictionary and a language model has been created and locally stored.
During the build process, acoustic model files for the english language, were deployed here: /usr/local/share/pocketsphinx/model/en-us/en-us

.. time to try out the the recognizer:
cd ~/
export LD_LIBRARY_PATH=/usr/local/lib
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

pocketsphinx_continuous -hmm /usr/local/share/pocketsphinx/model/en-us/en-us -lm 3199.lm -dict 3199.dic -samprate 16000/8000/48000 -inmic yes

Output

READY….
Listening…

INFO: ps_lattice.c(1380): Bestpath score: -7682
INFO: ps_lattice.c(1384): Normalizer P(O) = alpha(:285:334) = -403763
INFO: ps_lattice.c(1441): Joint P(O,S) = -426231 P(S|O) = -22468
INFO: ngram_search.c(874): bestpath 0.01 CPU 0.003 xRT
INFO: ngram_search.c(877): bestpath 0.01 wall 0.002 xRT
OPEN GARAGE
READY….
Listening…

Live Demo

This video shows the recognizer running in keyword spotting mode, using the dictionary and model mentioned above:
pocketsphinx_continuous -lm 3199.lm -dict 3199.dic -keyphrase "OKAY PI" -kws_threshold 1e-20 -inmic yes
The purpose is to provide some indication of the recognition speed that can be expected, running PocketSphinx on the Raspberry Pi 2.

16 Replies to “Raspberry Pi 2 – Speech Recognition on device”

  1. HI, great tutorial, wondering if this would work on raspbian Jessie?

  2. getting Input overrun, read calls are too rare .. and poor recognition

  3. Hey
    any idea how to get the pocketsphinx-python package running on a raspberry pi 3?

    1. yes i started on raspberry pi 3

      1. it works fine on raspberry pi 3.i followed the instructions and all went smoothly.
        https://www.youtube.com/watch?v=5kp5qpwVh_8

  4. excellent guide … but you could just connect a bash command recognizes the word

  5. it would be nice to have a tutorial how to use this with GPIOs..

  6. I have given all the commands as stated but i am getting the following error;
    INFO: continuous.c(307): pocketsphinx_continuous COMPILED ON: Apr 28 2017, AT: 09:23:15

    Error opening audio device (null) for capture: Connection refused
    FATAL: “continuous.c”, line 245: Failed to open audio device
    Please tell how to rectify this

    1. Like this post: https://stackoverflow.com/questions/35867490/fatal-error-continuous-c-line-246-failed-to-open-audio-device

      pocketsphinx_continuous -hmm /usr/local/share/pocketsphinx/model/en-us/en-us -lm 5893.lm -dict 5893.dic -samprate 16000/8000/48000 -inmic yes -adcdev plughw:0,0

      Just add the -adcdev and plughw:0,0 !

  7. please help me! everything worked, but I need to run scripts in python with the recognized phrases.

    1. Try to add the two missing parameters at the end:

      pocketsphinx_continuous -hmm /usr/local/share/pocketsphinx/model/en-us/en-us -lm 5893.lm -dict 5893.dic -samprate 16000/8000/48000 -inmic yes -adcdev plughw:0,0

    2. Racha Nikhil says: Reply

      Could you tell me did u install in jessie and even if possible please respond to my reply coz i am stuck with the imstallation part

  8. please help!
    i am getting this error. what to do?
    FATAL: “continuous.c”, line 245: Failed to open audio device

  9. Hi.
    I am interesting in this for a different purpose. I would like to take a continuous stream (live microphone) and extract a phoneme string from the spoken text. The Phoneme string would then be used to manipulate servo’s controlling an animatronic mouth.
    This is an approximation – not a accurate reproduction.

    Many thanks
    Dave

  10. Please Help!
    All went well; I successfully ran pocketsphinx_continuous.
    I then tried to run a python app that had the import statements:
    from pocketsphinx.pocketsphinx import *
    from sphinxbase.sphinxbase import *

    The import failed on the first statement:
    Traceback (most recent call last):
    File “eb3.py”, line 4, in
    from pocketsphinx.pocketsphinx import *
    File “/usr/local/lib/python2.7/dist-packages/pocketsphinx/__init__.py”, line 37, in
    from pocketsphinx import *
    File “/usr/local/lib/python2.7/dist-packages/pocketsphinx/pocketsphinx.py”, line 35, in
    _pocketsphinx = swig_import_helper()
    File “/usr/local/lib/python2.7/dist-packages/pocketsphinx/pocketsphinx.py”, line 34, in swig_import_helper
    return importlib.import_module(‘_pocketsphinx’)
    File “/usr/lib/python2.7/importlib/__init__.py”, line 37, in import_module
    __import__(name)
    ImportError: No module named _pocketsphinx

    This was my second attempt to install pocketsphinx. Same error as on the first attempt.
    What am I missing?

    Thanks for your guide, It is very professionally done.
    Fred

    1. More info:
      I am attempting to install on RPI with
      RASPBIAN STRETCH OS

Leave a Reply