On the back, it has three USB -, one Firewire -, one Thunderbolt, and one Ethernet port. There is also this single bigger cable to the Laptop, combining Thunderbolt and power and keeping the desk neatly organized.

For a year or so, I have an external USB Harddrive (Western Digital 2 TerraBytes) connected to one of the display’s USB ports. I’m mainly using it for Time-Machine backups and storing application disk images etc. Since all cables disappear behind the display and not cluttering my desk, I never thought about connecting the harddrive straight to one of the Laptop’s USB ports directly. However, as it turns out, my nice and neat setup comes with quite a penalty.

With the setup described above, 15 inch Macbook Pro connected to a Thunderbolt monitor, which is connected to a 2TB WD external USB Harddrive, I picked a roughly 1GB movie file and copied it form the Laptop’s 500 MB SSD to the external harddrive. After that I reversed the process and copied the file back from the external drive, on to the internal solid state drive.
I used new names, i.e. didn’t override an existing file, and no other user applications but the Terminal were running. I used the simple time command to measure the performance like so:

time cp ./Movie1.mov /Volumes/TERA/

I ran every copy three times but recorded only the fasted time. Then I unmounted the drive and connected it straight to one of the Macbook’s USB ports, i.e., not sending the data through the Thunderbolt display. Again, I used new names, i.e. didn’t override an existing file, and no other user applications but the Terminal were running.

I know, this is not a very scientific test but you can see that writing files to the external harddrive, when it’s connected to the Thunderbolt display is quite a bit slower, takes about twice as long. When reading files back from the external drive, the penalty is not as bad.
Just for fun, I also measured the time (and datatransfer speed) it takes to copy the file on the SSD.

Prerequisites

Your computer needs to be equipped with a microphone and speakers and needs to have the latest version of Google’s Chrome browser installed, which is currently Version 27.0.1453.93.

Experiencing Speech-enabled Web Search

Now goto https://www.google.com and click on the small microphone icon, on the right side of the text input field.
The page your were looking at disappears and is replaced with a much larger red microphone and text that changes from “Speak Now” to “Listening…”

Your computer’s microphone is now “hot“. Try it out. Say something like “What will the weather be like tomorrow?”. You will hear a spoken response like “Tomorrow’s forecast for Poway is 73 degrees and partly cloudy.” The location for the forecast is determined by your current IP address, but you can also add a location, for instance by asking “What will the weather be like tomorrow in Frankfurt Germany?” Or try something like “Stock Quote for Apple” or “Where is the closest microbrewery?”.
Often (but not always) will you get a spoken response and a search result page that leads with an information card, similar to the cards displayed by the Google Now mobile app.

Here for instance is visual information that is returned while you hear “Apple closed up 0.68 percent at 445 dollars and 15 cents.”

The voice that is used for the speech synthesis may sound very familiar to users of Google’s mobile navigation apps. It is a hight quality voice, encoded as MPEG Layer 3, 1 channels, 16KHz.
In case you read this post in Google Chrome or any HTML-5 compliant browser, you =should be able to listen to an example right here:

The legends and current elites of AIML and Turing AI (not necessarily disjoint groups) met for their yearly gathering at Seed Philly, a Philadelphia tech startup incubator, located in the heart of Center City.

The Chatbots 3.3 conference was a fast flowing event with exciting flash-talk style presentations, always followed by Q&A segments. The amazing speaker lineup included several Loebner Prize winners, AIML Engine developers, VCs, Psychiatrists, Artificial Intelligence researcher, and best-selling authors.

Dr. Richard Wallace

The Highlights

Dr. Richard Wallace, Loebner Prize Winner 2000, 2001, 2004 and father of AIML, talked about the new AIML 2.0 Specification that focuses much on making AIML more succinct, while maintaining its simplicity. He stated that it currently took about 10,000 AIML categories to create a believable character. ALICE for instance has about 100,000 categories and the PROFESSOR, a bot with one of the largest AIML knowledge bases, has about 580,000 AIML categories. Writing that many categories is not only very time consuming (experienced AIML authors may be able to write one category per minute) but also requires a memory capacity, not  available on many embedded and mobile devices. AIML 2.0 therefore tries to make AIML more efficient, allowing the creation of a believable character with as little as 1/6 of the categories needed before.

Mohan Embar

Mohan Embar, 2012 Loebner Prize Winner for his bot entry Chip Vivant, gave an informal talk entitled Moonshot or Low-Hanging Fruit, sharing his motivation for creating his bot (which is not based on AIML) and encouraging other bot authors to question their own motivation and to stay focussed.

Dave Morton

He currently works on creating a more empathetic and engaging bot that is more than simply reacting. He suggested a project that would allow to program a bot with a persons’ personality, core-values, views, achievements, etc. which would be available after the persons’ death, for relatives and friends to still communicate with the deceased.

Jeremy Spiegel, MD, Cascon Bay Medical, was making the case for the use of Chatbots in Psychiatry, Dave Morton, Lead developer of the Program-O AIML-Interpreter, talked about the improvements he is currently making on the PHP/MySQL based engine, and Wolf Paulus, Inuit Inc., talked about Voice Enabling Chatbots.

Bill DeSmedt

Bill DeSmedt, system designer, Artificial Intelligence researcher, and science fiction author (Singularity (2004), Dualism (2010)) talked about his PASCAL based deductive and inductive reasoning system that can solve reasonably sophisticated murder mystery puzzles when fed with all the facts.

Dave Morton

Chris Shaw

The event ended with an invigorating talk from the entertaining, always challenging, and thought provoking Chris Shaw. His focus was more on visual expression rather than artificial intelligence, showing many of the cutting edge visual presentations of avatars, like Zoe . Chris emphasized the importance of tell a compelling story, a bots needs to be able to play a role in an interactive drama and always needs to be able to create that moment of surprise. He concluded with stressing the significance of sound quality of a natural voice over any visual presentation.

The Conclusion

Seeing the legends of AIML and Turing AI being all invigorated and out in force, confirmed the intuition that everyone at the conference seemed to have had: all the parts and pieces are finally coming together, to create an believable virtual human.

There was a spirit of optimism that many of the more experienced could only compare with the time shorty before the word-wide-web popularized the Internet.

Install Oracle JRE 7 on Debian Linux

To install Oracle’s Java Runtime with apt-get, we first need to entend the list of apt-get’s sources. Once that is done, an java-installer will actually install the Java SE Runtime Environment. Here are the steps to follow:

$ su root
# echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" > /etc/apt/sources.list.d/webupd8team-java.list
# echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" >> /etc/apt/sources.list.d/webupd8team-java.list
# apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886
# apt-get update
# apt-get install oracle-java7-installer
# java -version
java version "1.7.0_17"
Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
# exit
$

With Java now installed we move on, installing Tomcat. However, it may be beneficial to have an dedicated user for Tomcat, in case you …
BTW, Java got installed into this location: /usr/lib/jvm/java-7-oracle

sudo adduser \
  --system \
  --shell /bin/bash \
  --gecos 'Tomcat Java Servlet and JSP engine' \
  --group \
  --disabled-password \
  --home /home/tomcat \
  tomcat

Installing Tomcat7

$ cd /tmp
$ wget http://mirror.cogentco.com/pub/apache/tomcat/tomcat-7/v7.0.37/bin/apache-tomcat-7.0.37.tar.gz
$ tar xvzf ./apache-tomcat-7.0.37.tar.gz

$ sudo mkdir /usr/share/tomcat7
$ sudo mv /tmp/apache-tomcat-7.0.37 /usr/share/tomcat7

To make it easy to replace this release with future releases, we are going to create a symbolic link that we are going to use when referring to Tomcat (after removing the old link, you might have from installing a previous version):

$ sudo rm -f /usr/share/tomcat
$ sudo ln -s /usr/share/tomcat7/apache-tomcat-7.0.37 /usr/share/tomcat

Since we created a tomcat user, he should also own all these files in

$ sudo chown -R tomcat:tomcat /usr/share/tomcat7
$ sudo chmod +x /usr/share/tomcat/bin/*.sh

If Tomcat’s default HTTP port (8080) is already in use, you need to edit the server.xml configuration file, e.g.
edit /usr/share/tomcat/conf/server.xml and replace 8080 with 8000

Starting Tomcat

$ sudo /bin/su - tomcat -c /usr/share/tomcat/bin/startup.sh

Using CATALINA_BASE:   /usr/share/tomcat
Using CATALINA_HOME:   /usr/share/tomcat
Using CATALINA_TMPDIR: /usr/share/tomcat/temp
Using JRE_HOME:        /usr
Using CLASSPATH:       /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar

Stopping Tomcat

$sudo /bin/su - tomcat -c /usr/share/tomcat/bin/shutdown.sh

Using CATALINA_BASE:   /usr/share/tomcat
Using CATALINA_HOME:   /usr/share/tomcat
Using CATALINA_TMPDIR: /usr/share/tomcat/temp
Using JRE_HOME:        /usr
Using CLASSPATH:       /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar

Tomcat with Webmin

If you were to manage your debian server with WebMin, the configuration of the start/stop behavior would look like this

which created a script just like this one:

#!/bin/sh
### BEGIN INIT INFO
# Provides: Tomcat
# Required-Start: $network
# Required-Stop: $network
# Default-Start: 2 3 5
# Description: Java Servlet and JSP Engine
### END INIT INFO

case "$1" in
'start')
	/bin/su - tomcat -c /usr/share/tomcat/bin/startup.sh
	;;
'stop')
	/bin/su - tomcat -c /usr/share/tomcat/bin/shutdown.sh
	;;
*)
	echo "Usage: $0 { start | stop }"
	;;
esac
exit 0

.

A verbally responding mobile application has many obvious advantages. For instance, users don’t have to decipher tiny fonts on small displays, in fact, they don’t have to look at the display at all. Just like colors and typography contribute considerably to the look and feel of an application, so does the voice quality for a voice enabled mobile application.

There are at least three different approaches to synthesize text.

There might be a Text-To-Speech module built into the OS, or a separately installed Text-o-Speech engines can plug-in to the OS’s Text-To-Speech module.
Secondly, instead of requiring a separate install, a synthesizer and voices can be packaged and shipped with the application.
Lastly, a web-service can be used, to synthesize text. The advantage of this, would be a more predictable and consistent voice quality, comparatively independent from the hardware and operation system used on the mobile client.

Voice Synthesis Demo

Voiceware Co.,Ltd., located in Seoul, Korea, provides one of the highest speech synthesis quality solutions available. This however is highly subjective, see “Factors to assess voice quality” below.

Listener’s choice of what is a good voice is subjective without measurable units. However, a voice can still be described as:

• Bright: Higher loudness without effort (not-shouting)
• Confident: Steady Pitch and Loudness level –opposite of wavering
• Melodious: Good correlation between Pitch and loudness
• Resonant: Articulation tuned to Pitch – Ringing
• Hoarse: Presence of noise component – absence of harmonics
• Pressed: Long closed phase in vibration cycle
• Breathy: Steep spectral slope, strong fundamental
• Nasal: Heavily damped resonances

Factors to assess voice quality

Voice quality depends not only on the functioning of the voice box or the larynx but also on articulation (enunciation or pronunciation), rate of speech (too rapid or too slow), intonation, pauses etc.

• Rate of speaking: Number of words per breath, Number of words per minute
• Rhythm of Speech: Accent and Inflections (Pitch range)
• Breathing Pattern: Speech as during fast walking Vs relaxed
• Pitch Level: Habitual F0 or Pitch
• Loudness level: Deafening (too loud) Vs Whispering (too soft)
• Mood State: Sudden changes in pitch level, loudness and rhythm
• Clarity of Speech: Good articulatory habit

When perceiving a persons credibility, accent is more important than looks

More TTS – Web Services

• Google http://translate.google.com/translate_tts?tl=en&q=text, e.g.:
• http://translate.google.com/translate_tts?tl=en_uk&q=Hello+,+how+are+you+today+?
• Free text-to-speech conversion service: http://speechutil.com/
• Microsoft http://www.microsofttranslator.com/dev/ an example would be http://www.bing.com/translator/
• iSpeech Demo: http://www.ispeech.org/text.to.speech

Artist on Android

The display is the canvas and your finger becomes a paint brush.
Simply use your finger to create beautiful paintings.
Use the Menu to change color or brush styles and also to print or share your artwork via Email etc.
Publish your best work on our flickr group, straight from the app.
Using Advanced Photo Paper has produced stunning print results, worth framing.
I am totally amazed by what users have drawn with this app.

Voice Recognition

Mobile applications like Artist on Android are perfectly suited to accept voice input.
Instead of accessing the menu several times, to select a tool (like a pen or a brush), its style (like embossed or blurred), and its color, you now press the microphone button in the Actionbar and say “Lavender Blur Pen”.

The application’s vocabulary include all menu topics like “Pen, Brush, Roll, Emboss, Blur, Blend, Alpha, Erase, Reset, Save, Print, Share Publish, Explore, and Help” and also over one hundred color names. To bring up a list of all those color names, simply say “Colors”.
Speech recognition, interpretation, and processing is hard and “Artist on Android” is far from perfect. So please be gracious and generous if it not always understands. We are just getting started ..

]]> http://wolfpaulus.com/jounal/android-journal/artist-voice-recognition/feed 0 http://wolfpaulus.com/jounal/mac/installing_python_osx?utm_source=rss&utm_medium=rss&utm_campaign=installing_python_osx http://wolfpaulus.com/jounal/mac/installing_python_osx#comments Mon, 28 Jan 2013 17:00:28 +0000 Wolf Paulus http://wolfpaulus.com/?p=2071 While Python comes pre-installed on OS X, Apple doesn’t do a good job on keeping the Python runtime environment up to date. Currently, on Mac OS X 10.7.4 “Lion”, entering python -V returns Python 2.7.1. Even worse, Mac OS X 10.6 “Snow Leopard” is still on a Python 2.6 release.
While the latest Python releases are always available on http://www.python.org, updating a Mac isn’t a simple, straight forward process.

Follow along and update your Mac to Python 2.7.3, or 3.3.0 or whatever the newest 2.x and 3.x release might be, when you read this. To update your Mac to something like Python 2.7.3, I assume that

• your Mac-User account is setup as an “Administrator” account.
• your Mac already has this folder: /System/Library/Frameworks/Python.framework/Versions/

To read about how to upgrade to Python 3.3, jump to the very bottom of this post.

1. Downloading and Installing the latest Python Release

Go to python.org and pick the most recent Python 2.x release: http://python.org/download.
I picked Python 2.7.3 Mac OS X 64-bit/32-bit x86-64/i386 Installer (for Mac OS X 10.6 and later), an about 18 MB download.
Opening the DMG and executing the installer (Python.mpkg) will install the Python release into /Library/Frameworks/Python.framework, which is not next to the other, already installed Python versions and may lead to some nasty incompatibilities. Moreover, /usr/bin/python still executes python 2.7.1., i.e. some work still needs to be done to really update Python on your Mac.

2. Moving Python into the right place

If Python 2.7.x is already available on the Mac (e.g. on OS X Lion), open a terminal and enter:

sudo rm -R /System/Library/Frameworks/Python.framework/Versions/2.7

This will delete Python 2.7.
In any case, now enter this into the terminal

sudo mv /Library/Frameworks/Python.framework/Versions/2.7 /System/Library/Frameworks/Python.framework/Versions

which moves the just installed python 2.7.3 release next to the other Python releases.

3. Fixing the Group

Setting group to wheel, just like it’s done for the already installed Python Versions:

sudo chown -R root:wheel /System/Library/Frameworks/Python.framework/Versions/2.7

4. Updating the Current Link

sudo rm /System/Library/Frameworks/Python.framework/Versions/Current
sudo ln -s /System/Library/Frameworks/Python.framework/Versions/2.7 /System/Library/Frameworks/Python.framework/Versions/Current

5. Fixing /usr/bin

Since python, pythonw, etc. are not linked but actually reside in the /usr/bin directory,
/usr/bin/python still executes python 2.7.1..
Therefore, we are going to remove the old python executable files and replacing them with links into /System/Library/Frameworks/Python.framework/Versions/Current:

5.1. Removing old copies

sudo rm /usr/bin/pydoc
sudo rm /usr/bin/python
sudo rm /usr/bin/pythonw
sudo rm /usr/bin/python-config

5.2. Creating links into /System/…/Python.framework/Versions/Current

sudo ln -s /System/Library/Frameworks/Python.framework/Versions/2.7/bin/pydoc /usr/bin/pydoc
sudo ln -s /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python /usr/bin/python
sudo ln -s /System/Library/Frameworks/Python.framework/Versions/2.7/bin/pythonw /usr/bin/pythonw
sudo ln -s /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python-config /usr/bin/python-config

6. Updating .bash_profile

Use TextMate, pico, or your favorite text editor to edit the hidden ~/.bash_profile file. If you want to be able to execute python tools like idle easily, without providing the path, edit the PATH for Python like this:

# Setting PATH for Python 2.7
# The orginal version is saved in .bash_profile.pysave
PATH="/System/Library/Frameworks/Python.framework/Versions/2.7/bin:${PATH}"
export PATH

However, you can also remove those four lines altogether or fall-back to the .bash_profile.pysave file.
Try it by closing the terminal app, re-opening it, and entering python -V as well as /usr/bin/python -V.

7. Updating your IDE

The last remaining step it to tell your IDE about the new Python release that you just installed.
I hope you have discovered PyCharm, JetBrain’s IDE for Python and Django.
In PyCharm, you would open its properties and under “Project Interpreters” change the Python Interpreter, by pointing it to the /System/Library/Frameworks/Python.framework/Versions/2.7 directory.

Updating to Python 3.3

Following the same procedure, you can also update to Python 3.3. E.g, I downloaded the Python 3.3.0 Mac OS X 64-bit/32-bit x86-64/i386 Installer form here: http://python.org/download/

After running the included installer, a script like this (run with sudo) should put everything into the right place and make python 3.3 the default python version.

#!/bin/bash
rm -R /System/Library/Frameworks/Python.framework/Versions/3.3
mv /Library/Frameworks/Python.framework/Versions/3.3 /System/Library/Frameworks/Python.framework/Versions
chown -R root:wheel /System/Library/Frameworks/Python.framework/Versions/3.3

rm /System/Library/Frameworks/Python.framework/Versions/Current
ln -s /System/Library/Frameworks/Python.framework/Versions/3.3 /System/Library/Frameworks/Python.framework/Versions/Current

rm /usr/bin/pydoc
rm /usr/bin/python
rm /usr/bin/pythonw
rm /usr/bin/python-config

rm /System/Library/Frameworks/Python.framework/Versions/3.3/bin/pydoc 
rm /System/Library/Frameworks/Python.framework/Versions/3.3/bin/python 
rm /System/Library/Frameworks/Python.framework/Versions/3.3/bin/pythonw 
rm /System/Library/Frameworks/Python.framework/Versions/3.3/bin/python-config 

ln -s /System/Library/Frameworks/Python.framework/Versions/3.3/bin/pydoc3 /usr/bin/pydoc
ln -s /System/Library/Frameworks/Python.framework/Versions/3.3/bin/python3 /usr/bin/python
ln -s /System/Library/Frameworks/Python.framework/Versions/3.3/bin/pythonw3 /usr/bin/pythonw
ln -s /System/Library/Frameworks/Python.framework/Versions/3.3/bin/python3-config /usr/bin/python-config

For instance, in April 2012, Speech Recognition was first added to their iPhone (not iPad) mobile application and later to the Android app as well; allowing the user to request stock quotes, options chains, company information, or to launch the stock order, just by using voice.

“Investors are becoming more accustomed to interacting with voice-enabled technology, and we’re proud to be one of the first to offer this innovative feature to our mobile users,” said Michael Curcio, President, ETRADE Securities. “By integrating voice technology, ETRADE provides a mobile experience unlike any other – creating a state-of-the-art and convenient approach to navigation.”

ETrade uses speech recognition and speech synthesis software provided by Nuance Communications, Inc. The application is feature-packed, comes as an 11 MB download, and is not, what you would call a thin client. Only a very few of those features however, are accessible through Voice Commands.

Available Voice Commands

Stock Quotes

• “Price [symbol or name]“
• “Can I get a quote on [symbol or name]?”
• “Please give me the stock price on [symbol or name]“

Company News

• “I would like news on [symbol or name]“
• “News [symbol or name]“
• “Can I get some news on [symbol or name]?”

Company Options Chains

• “Can I get options chains on [symbol or name]“
• “Give me options on [symbol or name]“

Screen Navigation

• Take me to the dashboard”
• “Show me my accounts please”
• “I would like to see my alerts”
• “Can I see my watchlists”
• “Show me my portfolios”

Stock Order Ticket

• “Buy 50 shares of [symbol or name]“
• “Buy 75 shares of [symbol or name] at limit price 9 dollars and 50 cents”
• “Sell my holdings on [symbol or name]“
• “Sell 200 shares of [symbol or name]“
• “Sell 50 shares of [symbol or name] at limit price of 10 dollars”

Chat Bot

When it comes to interpreting voice inputs, the Etrade app is much more forgiving than Google’s. For instance, while only “Take me to the dashboard” is listed as a voice command,
“Open the dashboard” or “Show me the dashboard”, or “Can I see the dashboard”, is understood and does open the application’s dashboard view. However, the monotone voice and demeanor of the bot suggests some kind of bitterness, more than even Apple’s Siri does. This of course is hight subjective.

Besides what is listed as an available voice command, the Etrade app doesn’t seem to be very communicative. Simple questions like “How did the market close today?”,“How are you?”, or “What is your name?” all trigger the same answer “One moment please – I’m sorry I could not process that.“

Dissecting the mobile application

Once the microphone button is pressed, a voice prompt is played “How can I help you?” and the application goes into listening mode. It will keep listing until a drop in amplitude is recognized. I.e., if I didn’t say anything, it kept listening, probably until the phone runs out of battery.
When an amplitude drop is recognized, the sound recording is sent for speech recognition and acknowledged with another voice prompt: “One moment please.”
The recognition of an input like “Please give me the stock price on Apple” creates a spoken output like “Price 439.88 Change Percent Minus 2.36 Percent”. The time between the end of the request and the beginning of the response is a little over 5 seconds. However, the voice prompt “One moment please” and the fact that the app shows a view with the requested quote, even before it starts playing the synthesized response, are clever tricks to obscure the relative long delay.

Since the application doesn’t seem to include the standard Nuance Voice Kit, I am assuming the sound file is not streamed during the recording (the forever listing behavior would be another indication), but rather encoded and sent via HTTP POST once the recording has stopped.
The response is synthesized by a Web-service, instead of on the mobile device itself. While this has the advantage that no high quality voice synthesizer needs to be deployed with the app, the pre-recorded and included voice prompts sound differently, in quality and also noticeably in loudness.

Final Thoughts

ETrade has done a great first step, voice-enabling their mobile application. It’s better than Google’s Voice Commands, since it’s much less rigid. And while voice and demeanor remind us of Siri, it’s good to see trailblazers like Etrade pioneering Voice User Interfaces on mobile.

Demo

Take a look and a listen how I interacted with ETrade’s mobile Android app via Voice Commands:

• Pebble, E-Paper Watch for iPhone and Android.
• Smartwatch, Sony Android watch.
• MOTOACTV, Motorola GPS Fitness Tracker with MP3 Player.
• NEXD, The Next-gen Android watch.
• i’m Watch, i’m SpA, Smartwatch connects via Bluetooth to a phone.

i’m Watch

However, there is i’m SpA, an Italian company, maker of the I’M WATCH, an Android 1.6 powered watch that works as a Bluetooth extension for iPhone and Android phones and comes with microphone and speaker on board.

Gestures

I really wanted to like the i’m Watch, but a couple of significant shortcomings became immediately obvious, during the setup process.
There is a delay with all touch and swipe gestures, which we are just not used to anymore. A response often comes delayed and you end up pressing buttons twice. Mainly due to the small display size, gestures seem too sensitive, i.e. minuscule gestures create unexpected effects.
The notification panel, which is heavily used, is too small and therefore hard to access, when dragging it downwards to open it.

Audio Quality

Using the watch as an Bluetooth extension for a phone, e.g. to make phone calls, is virtually impossible. The audio quality for both playback and recording is terribly bad.

Internet Access

Most applications running on the Watch need to access the Internet, which means that the smart phone that connects to the watch via Bluetooth, needs to act as hotspot for Bluetooth devices. Enabling this feature on your smartphone requires to upgrade ($$) your dataplan (at least for Verizon and AT&T customers).

Software development

Software development, at least for the purpose I wanted to use the watch for, turned into something close to a nightmare. There are two Android 1.6 images available for the Watch, one has adb enabled, i.e. allows to deploy Android apps, the other one allows to mount the watch as a USB drive – it’s either or. I went with the adb enabled image for developers, but the Watch still doesn’t easily show up with “adb devices”. Instead, one needs to setup a RNDIS network device and hardcode 10.0.0.168 as its ANDROID HOST IP. adb needs to be manally killed and restarted every time the watch gets (re-)connected. Which needs to happens very often, since a DRRS-plug is the only way to tether the watch for deployment, but while connected, the DRRS-plug also shuts-off microphone and speaker.

SpeechSynthesis

While android.speech.tts.TextToSpeech was introduced with Android 1.6, there was no implementation provided on the i’m Watch. However, after a little bit of searching around, I came across com.svox.langpack.installer.apk and PicoLangInstaller.apk, which can be installed on the watch (adb install ..). This eventually helped to enable a crummy but still understandable voice output.

//
// TextToSpeech.OnInitListener listener
//

   	@Override
   	public void onInit(int i) {
       	tv.setText("onInit");
       	tts.setPitch(1.2f);
       	tts.setSpeechRate(1.1f);
       	tv.setText("Locale:" + tts.getLanguage().toString());
       	tts.speak("Hello, how are you today?", TextToSpeech.QUEUE_FLUSH, null);
   	}

Since that didn’t take too long, I was optimistic to get SpreechRecognition working as well. However, huge portions of the Android 1.6 SDK are simply missing. E.g. RecognizerIntent is not implemented and MediaRecorder will always fail when prepare() is called. Meaning, I ended up using AudioRecord for the voice recording. Something like so:

private void startRecording() {

        mRecorder = new AudioRecord(
                MediaRecorder.AudioSource.MIC,
                RECORDER_SAMPLERATE,
                RECORDER_CHANNELS,
                RECORDER_AUDIO_ENCODING,
                mBufferSize);

        mRecorder.startRecording();
        isRecording = true;

        mRecordingThread = new Thread("Recording Thread") {
            @Override
            public void run() {
                writeAudioDataToFile();
            }
        };
        mRecordingThread.start();
    }

    /**
     * <code>writeAudioDataToFile</code> writes the raw recording buffer into a temp. file.
     */
    private void writeAudioDataToFile() {
        final byte data[] = new byte[mBufferSize];
        try {
            final OutputStream os = new BufferedOutputStream(new FileOutputStream(getTempFilename()));
            while (isRecording) {
                final int k = mRecorder.read(data, 0, mBufferSize);
                if (k != AudioRecord.ERROR_INVALID_OPERATION) {
                    try {
                        os.write(data, 0, k);
                    } catch (IOException e) {
                        Log.e(LOG_LAG, e.getMessage());
                    }
                }
            }
            try {
                os.flush();
                os.close();
            } catch (IOException e) {
                Log.e(LOG_LAG, e.getMessage());
            }

        } catch (FileNotFoundException e) {
            Log.e(LOG_LAG, e.getMessage());
        }
    }

    private void stopRecording() {
        if (mRecorder != null) {
            isRecording = false;
            mRecorder.stop();
            mRecorder.release();
            mRecorder = null;
            mRecordingThread = null;
        }

        final String wavFile = getFilename();
        copyWaveFile(getTempFilename(), wavFile);
        new File(getTempFilename()).delete();
        // AsyncTask calling SpeechRecognition web service ..
        new NAPI(mHandler).execute(getHttp_Api_Url(), wavFile );
    }

Unfortunately, the sound quality was once again very poor and Nuance’s Web service for Speech Recognition could not render any results. After significantly improving the wav-file using Wiretap Studio on my Mac, Nuance would finally return a result, which wasn’t even too far off. However, even if the improvement could be done on the i’m Watch itself, the total processing time was just way too long that even when considering all optimizations, we would never get into usable range.

List of things that won’t work

• Sensors APIs require special library, which have not been released yet.
• The i’m Watch has a custom audio system. Android Audio recording/routing APIs are not working (Manufacturer has acknowledged the problem, but no ETA)
• Speech recognition APIs don’t work due to the previous point.
• NetworkManager.isConnected() always returns true (Manufacturer has acknowledged the problem, but no ETA)
• Option menus are not working (UI/UX decision: screen is too small for this) (Manufacturer promised to publish a fix on their GitHub account soon.)
• Vibration APIs are not working (no vibration motor in the i’m Watch)
• Input Method Editors are not supported (UI/UX decision: screen is too small for this) (Manufacturer promised to publish a fix for the NumPad at least on their GitHub account soon.)
• Links that open the web browser are not supported. The web browser is present, but will be removed in a future release (UI/UX decision: screen is too small for this)

Final Verdict

Requested RMA. Returned i’m Watch ..

I never really liked going to “PowerPoint Meetings”, sharing the screen content with others during a meeting, usually requires to connect a projector to the presenter’s laptop. In a lengthy process, the laptop’s screen resolution and refresh-rate needed to be manually adjusted to synchronize with the projector. What follows is a lecture style presentation, featuring slides being projected, hugely magnified onto one of the meeting room’s walls; often, lights need to be dimmed, and listeners either doze off or start checking email.

A couple years back, I wrote TiffanyScreens, which allows you to share presentations (or any screen content), without requiring a projector.
Imagine a scenario, where every participant brought a laptop computer to a meeting and watched the presentation on that laptop’s display – participants sit on a table facing each other, instead of the wall. No adjustments are necessary; images are scaled automatically on arrival, to best match the receiver’s display-capability. To support lively meetings, everyone participating, can with a single button click, turn his computer into the presenting device.

During the development, I had small engineering groups in mind, but it became more and more obvious that instructors, educators, and educational institutions would appreciate TiffanyScreens the most. Listening to users’ feedback, TiffanyScreens has been iteratively improved, refined, and optimized.

For instance, to accommodate instructors and teachers, only one peer needs to have a licensed copy of the application installed and since you never know what kind of laptop users bring, TiffanyScreens is of course a multi-platform application, running on Windows, Linux, and the Mac. In fact, most of TiffanyScreens is implemented in Java.

But almost like the boiling frog story, Apple slowly turned up the heat on Java and Java developers. OS X 10.7 Lion introduced the Mac Application Store, in which Java applications were not welcome. OS X 10.8 Mountain Lion makes it extra hard for users to install applications that are not distributed through the Mac App Store.





With Apple withdrawing most of its Java support, it’s now Oracle, providing up to date Java runtime and Java SDK of Mac OS X. However, recent Zero-Day Java security vulnerabilities make it less like that users’ machines have a Java runtime installed.

Considering all these obstacles, I could have let TiffanyScreens run its course, letting it die a slow death, but during a couple of days off work, over the Christmas break, I came up with a decisive rescue plan … putting TiffanyScreens into the Mac Application Store.

Putting TiffanyScreens into the Mac Application Store

1. Administrative Tasks

Trying to put a Java desktop app into Apple’s Mac App Store is a daunting task, no mean feat, some may even call it a total waste of time. However, TiffanyScreens, the innovative presentation and screen sharing tool, used and loved by many educational institutions worldwide, must not die …

1.1. Joining the Mac Developer Program

It all started with a $99 investment, purchasing a Mac Developer ID and getting into the Mac Developer Program.

https://developer.apple.com/programs/start/standard/

1.2. Request a OS X Signing Certificate

The next step was all about preparing my machine to sign and submit a binary to the Mac App Store approval process.
https://developer.apple.com/certificates/index.action#maccertrequest

Since TiffanyScreens does neither need the iCloud nor Apple Push Notifications, I did forgo the Mac App Development Certificate but requested and installed:

• Mac App Certificate
• Mac Installer Certificate
• Developer ID Application Certificate
• Developer ID Installer Certificate

1.3. Creating an Application Bundle ID

I created and Application Bundle ID here: https://developer.apple.com/certificates/index.action#bundle
An App ID is an identifier used by iOS and Mac OS X to recognize your app. This was pretty simple and straight forward. I just needed to enter my applications name and a bundle identifier.

• TiffanyScreens Pro
• com.carlsbadcubes.tiffany.TiffanyScreens.pro

1.4. Creating an Application on the iTunes Store

I opened https://itunesconnect.apple.com/ and in “Manage Your Applications”, selected “Add New App” followed by “Mac OS X App”
Here, the application got defined in more detail, like it will appear in the iTunes Store. Completing this step also included deciding on a primary and secondary application category, as well as uploading a couple of screen shots. I think it’s important to note that once an application has been approved, those screen shots cannot easily be changed.

This pretty much concluded the administrative tasks and now the much more interesting engineering tasks began.

2. Engineering Tasks

2.1. Java Runtime

Since I wanted to bundle TiffanyScreens with the latest Java Runtime, I downloaded and deployed JDK 7 on my development and build machine.
http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html
Installing jdk-7u11-macosx-x64.dmg, the most recent Java release for 64-bit Intel Macs, meant getting something like this when typing java -version into a terminal.

java version "1.7.0_11"
Java(TM) SE Runtime Environment (build 1.7.0_11-b21)
Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode)



In ~/.bash_profile I have JAVA_HOME declared like this:
export set JAVA_HOME=`/usr/libexec/java_home`
which in fact points here:
/System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java_home
which is an executable and returns:
/Library/Java/JavaVirtualMachines/jdk1.7.0_11.jdk/Contents/Home
and this location contains the Java Runtime (jre) folder, which will eventually be bundled with TiffanyScreens.

2.2. A Single JAR

TiffanyScreens was already packaged to be distributed as a Mac friendly DMG. What I mean with that is that TiffanyScreens was always packaged into a single Jar. 3rd party libraries like Swixml for instance get unpacked
<unzip src="${swixml.jar}" dest="${build.dest}"/>
during the build process and re-packaged together with TiffanyScreens’ classes and resources into a single jar file.

The package target for instance in my TiffanyScreeens’ Ant build.xml looks something like this:

<target name="package" depends="compile,unjarparties" description="Creates the class package">

       <!-- Filters defined specifically for updating the manifest -->
       <filter token="version.spec" value="${version.spec}"/>
       <filter token="version.impl" value="${version.impl}"/>
       <copy
           todir="${build.dest}/META-INF"
           filtering="yes">
           <fileset dir="${package.dir}/META-INF"/>
       </copy>
       <copy
           todir="${build.dest}" filtering="no">
           <fileset dir="${build.src}" includes="**/*.xml"/>
       </copy>
       <copy
           todir="${build.dest}" filtering="no">
           <fileset dir="${build.src}" includes="img/*"/>
       </copy>
       <jar jarfile="${build.dir}/${name}.jar"
            basedir="${build.dest}"
            excludes="META-INF/MANIFEST.MF"
            manifest="${build.dest}/META-INF/MANIFEST.MF"
           />
</target>		

2.3. AppBundler Ant Target

Oracle also has a document on its site that explains some of the things I wanted to do, Bundling the JRE with Tiffany. Unfortunately, the example they show is somewhat oversimplified to be really helpful. However, it has links to the Java Application Bundler project, which contains a Jar, providing an Ant target, to bundle the JRE with a Java app.

So I download the appbundler-1.0.jar and read the related documentation.

Going back and force between the documentation and my Ant build file, I ended up with an target configured like this:

    <taskdef name="bundleapp" classpath="${appbundler.jar}" classname="com.oracle.appbundler.AppBundlerTask"/>
    <target name="appbundle" depends="package" description="Create Mac OSX Application Bundle">
        <bundleapp
            outputdirectory="/Users/wolf/Work/Tiffany/build/dist"
            name="${Name}"
            displayname="${Name}"
            identifier="${mainclass}.${bid}"
            icon="res/${bid}.icns"
            shortversion="${version}"
            signature="CABA"
            copyright="2006-2013 Techcasita Productions."
            applicationCategory="public.app-category.productivity"
            mainclassname="${mainclass}"
            >
            
            <!-- JDK / JRE -->
            <runtime dir="/Library/Java/JavaVirtualMachines/jdk1.7.0_11.jdk/Contents/Home"/>

            <!-- The bundleapp task doesn't support classpathref so all
                the run classpath entries must be stated here too.
            -->
            <classpath file="./build/${name}.jar"/>
            <classpath file="./libs/libscreen.jnilib"/>

            <option value="-Xdock:name=${Name}"/>
            <option value="-Xdock:icon=Contents/Resources/${bid}.icns"/>
            <option value="-Dapple.laf.useScreenMenuBar=true"/>
            <option value="-Dcom.apple.macos.use-file-dialog-packages=true"/>
            <option value="-Dcom.apple.macos.useScreenMenuBar=true"/>
            <option value="-Dcom.apple.mrj.application.apple.menu.about.name=${Name}"/>
            <option value="-Dcom.apple.smallTabs=true"/>
            <option value="-Dcom.apple.textantialiasing=true"/>
            <option value="-Dcom.apple.antialiasing=true"/>
            <option value="-Dcom.apple.showGrowBox=false"/>
            <option value="-Dapple.awt.fullscreencapturealldisplays=false"/>
            <option value="-Dapple.awt.fullscreenusefade=true"/>
            <option value="-Djava.library.path=$APP_ROOT/Contents/Java"/>
            <option value="-Xms32M"/>
            <option value="-Xmx256M"/>
        </bundleapp>
        <delete
            file="/Users/wolf/Work/Tiffany/build/dist/${Name}.app/Contents/PlugIns/jdk1.7.0_11.jdk/Contents/Info.plist"/>
    </target>

Most of tag’s attributes should be obvious, I’m using this build file for both TiffanyScreens version (Pro and Standard), which is what the bid property is used for.

      <property name="Name" value="TiffanyScreens"/>
      <property name="bid" value=""/>
      <property name="name" value="tiffanyscreens"/>
      <property name="Short" value="Tiffany"/>
      <property name="mainclass" value="com.carlsbadcubes.tiffany.TiffanyScreens"/>
      <property name="version" value="5.0"/>

The applicationCategory attribute value should of course match with what was selected as a primary category, when the application entry was created on the iTunes Store. How this value needs to be decoded can be found here.

Since I had selected ‘Productivity’ for the primary category, I used this UTI: public.app-category.productivity

The runtime tag points to the Java_Home location, from where it will pick up the jre.
The classpath tags declare additional locations that need to be put into the bundle as well.
Finally, the option tags will create in the Mac applications Contents/info.plist file.
Once the bundling has happen, as a last thing, I deleted the Java runtime’s info.plist. The submission process seems to be very sensitive about that and only after there was only a single info.plist file left in the bundle was I able to submit the application.

2.5. Application Icon

The Ant target shown above contains two references to the application’s icon, which has to be carefully constructed.
Since the introduction of the “retina display”, Mac application that want to pass the rigorous approval process, need to have an icon file containing icons all the way to 1024×1024 pixels.
After I had created the icon artwork in five resolutions, I carefully named them like so:

• icon_16x16.png
• icon_128x128.png
• icon_256x256.png
• icon_512x512.png
• icon512x512@2x.png

A tool provided with the latest XCode release allow to combine the icons into a single icns file:
iconutil -c icns /Users/wolf/Work/Tiffany/res/pro.iconset

2.6. Signing the application bundle

Running the appbundle build target ./build/dist/TiffanyScreens Pro.app, already created a perfectly executable Mac application.

ant appbundle
..
appbundle:
[bundleapp] Creating app bundle: TiffanyScreens Std
[delete] Deleting: /Users/wolf/Work/Tiffany/build/dist/TiffanyScreens Std.app/Contents/PlugIns/jdk1.7.0_11.jdk/Contents/Info.plist

BUILD SUCCESSFUL
Total time: 4 seconds


Before uploading the application for approval, the application and all packaged libraries needed to be signed. Since TiffanyScreens has all its classes and resources in a single jar, this meant that I only needed to sign the TiffanyScreens.jar file and the one jnilib that I included, .. plus all the libraries in the Java runtime of course.

#!/bin/bash
echo "Using ant build-target 'appbundle' to create the app file"
ant appbundle

echo "Sign everything inside the app file"
codesign --entitlements ./tiffany.plist -s "3rd Party Mac Developer Application: Wolf Paulus" -v "./build/dist/TiffanyScreens Pro.app"
find "./build/dist/TiffanyScreens Pro.app/Contents" -type f \( -name "*.jar" -or -name "*.dylib" \) -exec codesign --verbose -f -s "3rd Party Mac Developer Application: Wolf Paulus" --entitlements tiffany.plist {} \;
find "./build/dist/TiffanyScreens Pro.app/Contents" -type f \( -name "*.exe" -or -name "*.jnilib" \) -exec codesign --verbose -f -s "3rd Party Mac Developer Application: Wolf Paulus" --entitlements tiffany.plist {} \;

echo "Verify all libraries have been signed..."
find "./build/dist/TiffanyScreens Pro.app/Contents" -type f \( -name "*.jar" -or -name "*.dylib" \) -exec codesign --verbose --verify {} \;
find "./build/dist/TiffanyScreens Pro.app/Contents" -type f \( -name "*.exe" -or -name "*.jnilib" \) -exec codesign --verbose --verify {} \;

echo "Create signed package"
productbuild --component "./build/dist/TiffanyScreens Pro.app" /Applications --sign "3rd Party Mac Developer Installer: Wolf Paulus" --product tiffany.plist TiffanyScreensPro.pkg

echo "Test pkg before uploading it to Apple"
sudo installer -store -pkg ./TiffanyScreensPro.pkg -target /

echo "more tests"
spctl --assess --verbose=4 --type execute TiffanyScreensPro.pkg
spctl --assess --verbose=4 --type install TiffanyScreensPro.pkg
codesign -v ./build/dist/TiffanyScreens\ Pro.app/

This bash script runs the Ant script, before doing the code signing (using both certificates: “3rd Party Mac Developer Application: Wolf Paulus” and “3rd Party Mac Developer Installer: Wolf Paulus”) and running two sanity checks, recommended before trying to submit the app.

The outcome of this is a *.pkg file, which is made for submission to the app-store approval process but not for installing on your own Mac.

2.7. Submitting for Approval

With the latest version of XCode installed, the Application Loader can be found by, control-clicking XCode and selecting “Show Package Contents”. The Application Loader in the Contents/Applications folder. Selecting “Deliver Your App” automatically picked up all the information I had provided, when creating the application entry in the iTunes Store. Application Loader runs a lot of tests, before finally accepting and uploading the pkg file.

Waiting

Now the waiting began. While iTunes Connect (https://itunesconnect.apple.com/WebObjects/iTunesConnect.woa) showed the current status of the approval process, the information provided here http://reviewtimes.shinydevelopment.com/ gave a pretty good indication of the average App Store review time.

A day after submitting Tiffany Screens Pro, I submitted TiffanyScreens Std, which obviously only took a fraction of the time. The only work that I really had to do, was to create that insanely great 1024×1024 icon.

Approved

Each application got approved after 8 days and is now available in the Mac application store. Take a look here:
http://itunes.com/mac/wolfpaulus