I wanted to post an entry documenting where Pennyworth is now and some of the next steps I am taking with respect to its development and my own research.

Pennyworth

The C4.5 decision tree implemented in the last release has proven itself to be a much better “brain” than the previous machine learner. I’ve been running Pennyworth fairly constantly for the past week and it’s picking up the context as it should and I find myself training it only on the rare occasion to correct the model when it’s seeing something new. I’m quite pleased with how this has worked out.

That said, I’m contemplating at the moment removing the “interrupt me every X seconds” training mode. This is a fairly standard practice to use in evaluating machine learners (collect a representative sample of features to label and evaluate). However, in the context of this particular machine learner and the user’s interaction with it, I think that this method is unnecessarily burdensome and doesn’t really help the learner. In any case, if a time-based training mode remains in Pennyworth, I’ll be advocating the “train it when it’s wrong” approach in the documentation (that I need to write).

Growl

Pennyworth is designed to work with the Growl notification system, and I’ve been refining both programs so that they can operate in a more harmonious manner. Pennyworth now has a “Suppress Growl notifications” preference in the development build. This is intended for those of you who become annoyed by all of the notifications Pennyworth emits on context change.

I’ve also contributed a patch to the Growl developers that adds support for choosing an appropriate notification style or sound effect through AppleScript. Combined with Pennyworth’s AppleScript-based action-taking component (I need a better name for this), Pennyworth can control Growl and its notifications. This leads us to…

Context-aware notification systems

Last summer when I began working on a context-sensing platform for MacOS X, my goal was to build a context-aware notification system that changed its behavior in order to present the best style of notification, given the user’s state. The fundamental premise underlying this idea is that not all notification styles are appropriate for all situations. For example, a low-interruption notification style is appropriate for “unrelated” information when the user is heavily focused on a specific task. (See “Attuning Notification Design to User Goals and Attention Costs” by McCrickard & Chewar for more details.)

With the modifications to Growl, I can now use Pennyworth+Growl to begin exploring this space. I implemented an example of this over the weekend that worked quite well in my own tests. My specific problem was that I don’t like getting frequent notifications when I am gaming. Due to the MacOS X display system, overlaying a notification on an OpenGL canvas flickers the display and reduces the frames drawn per second (FPS). So, I wanted to change the notification style of two applications when I’m playing Warcraft: Pennyworth and Cidney.

I wanted to eliminate Pennyworth notifications because they are a frequent offender when I temporarily switch out of WoW to check mail or the web while I’m playing. When I do one or the other, I get this rapid sequence of “Activity: E-Mail”, “Activity: Web Browsing”, “Activity: World of Warcraft” messages when I’m multitasking this way. They don’t do me any good and just end up annoying me.

Likewise, when I’m playing WoW, I’m often not interruptible because I’m playing the Battlegrounds. So, when I receive an incoming phone call and notifications from Cidney (a desktop-based virtual Caller ID program), I’d rather listen for the custom ringtones that I’ve assigned to important people rather than be interrupted with a visual notification.

Given these constraints, I put together a script that implements the following pseudocode to deal with this contextual situation:

if Activity is “Playing World of Warcraft”:
tell Growl:
Use the Null notification style for World of Warcraft
Use the Null notification style for Cidney

else if World of Warcraft is not running:
tell Growl:
Use the Smoke notification style for World of Warcraft
Use the Smoke notification style for Cidney

Basically, this script tells Growl to use an invisible notification style the first time it detects me playing World of Warcraft. When I switch activities, it checks if WoW is still running before re-enabling the notifications for the two applications. Since I will probably be coming back to Warcraft if it’s still running, it doesn’t re-enable the notifications until I quit the program.

In an ideal world, I’d be able to refine this script a bit more by having it detect when I’m within an attention/focus-heavy context within World of Warcraft. When I’m playing the battlegrounds, interruptions are not okay. However, when I’m scanning the auction house or traveling from one zone to another, interruptions are acceptable. Unfortunately, until I am able to poke within the WoW process to determine what I’m doing in-game, this fine-grained, within-application context sensing probably isn’t possible. (Though, as I type this, a birdie in my head is whispering “make an add-on”.)

The example above is just one instance of how we can use context-aware notifications within our computing environments. “World of Warcraft” may be substituted with “Writing Novel” or other attention-heavy tasks. As long as users are okay writing AppleScript to define the relationships between context and notifications, the current system will be adequate and useful. I hope that it’s clear how a user could add other specialized configurations by simply adding conditions to the if/else statement above (e.g. “else if Activity is ‘Watching Film’ then turn off all notifications”).

I’ll be packaging up the various parts of this system with the next release of Pennyworth for those interested in defining their own context-aware notification configurations and replicating the above.

In a future post, I may think about the potential of adding a machine learner that determines the right combinations of styles and contexts for the user. The script above could easily be intuited by a decision tree, given access to information such as the current notification styles. I also haven’t touched on the idea of taking action on context changes (“play some opera when I’m reading, play some trashy Euro-techno when I’m programing”). I’ll be bringing this idea up in a later post when my thoughts become a bit more organized on the topic of the types of context-aware applications that may be built in addition to notification and home automation systems.

Another new feature coming in Pennyworth is the ability to rename or delete labels you have applied to training examples:

This interface is located in the Preferences and can be used to correct misspellings, merge labels, or delete irrelevant training cases. The way that it works is that it creates an operation (delete or rename) and applies that operation to the appropriate machine learner.

In deletion cases, the examples labeled the same as the one specified are simply deleted from the training cases and the learner generates a new model. In the renaming operation, matching cases have their labels replaced with the new one and the learner generates a new model.

I’m hoping to finish testing this release in the next few days and do a proper release this weekend.

I just wanted to drop a note about some of the recent updates to Pennyworth that will be released soon. I’ve been hard at work on the software, and I think that some of the new code represents big steps in the direction I’m heading with this application.

1. C4.5 decision trees: I’ve ditched the antiquated ID3 decision tree in favor of its successor, C4.5. C4.5 is superior to ID3 because it allows the learner to deal with missing values, continuous ranges, pruning the decision trees. I implemented an initial version this weekend and it already works (much) better than ID3.

2. Decision tree visualization: Unless you’ve been running Pennyworth in Xcode, you have no idea why it makes the predictions it does. I’ve addressed this by adding an additional user interface component that allows you to inspect the program’s “mental model”:

I’m using a nested set diagram to represent the trees.

This code will be released in the next version of Pennyworth. It’s working quite well at the moment, but I still have a few bugs to squash and a few new features to implement.

Stay tuned.

Pennyworth 1.0b5 is now available for download.

This is the first release of Pennyworth, the application formerly known as Genkanban. To migrate your settings from Genkanban to Pennyworth, copy or rename the following files and folders, replacing “Genkanban” with “Pennyworth”:

 
~/Library/Appplication Support/Genkanban
~/Libary/Preferences/net.aetherial.context.Genkanban.plist
 

Instead of the normal bzip’ed tarball, the link above will fetch a compressed disk image. In light of the new AppleScript functionality, a variety of sensors have been removed from the main application and replaced with straight AppleScript files. These files are named with the “Observer” prefix in the disk image and should be copied to

 
~/Library/Appplication Support/Pennyworth/AppleScript/Sensors
 

Also included is a sample action script for Adium that will update your Adium status message with your current activity (Adium.scpt). Copy this file to the folder

 
~/Library/Appplication Support/Pennyworth/AppleScript
 

In addition to the renaming and AppleScript changes, this release also includes a number of bug fixes that should address performance problems and UI glitches. I want to thank David Symonds (creator of MarcoPolo) for some performance tips and Ricky Buchanan (of ATMac) for feature suggestions and AppleScript consulting.

Prerequisites

MacOS X 10.5 “Leopard”

Growl (Recommended): Pennyworth can be used without the Growl notification system by viewing the current predictions, but this occupies precious screen real estate and you attention. In order to receive visual notifications of updated predictions, I recommend that you install and configure the Growl notification system before running Pennyworth.

Source

There is not a source distribution available yet, but if you’d like the source code, it is available via Subversion at

 

http://context-macosx.googlecode.com/svn/trunk/Tools/Applications/Pennyworth/ 

The source requires Xcode 3 to build.

Bugs, suggestions, or comments?

Please post any feedback in the comments below or send it to context@aetherial.net. This is early release software, so there will be bugs, and the application will only improve when problems are identified and submitted.

After some discussion with my colleagues and Genkanban users, I’ve decided to rename the software “Pennyworth”. The name is based on the only butler I can recall by name: Alfred Pennyworth.

The justification for the rename is that the “Genkanban” name is clumsier than “Pennyworth” for us English speakers and (honestly) I never knew the proper pronunciation of “Genkanban”.

If you’re a current user of Genkanban, you don’t need to do anything to get the new software. I’ll be pushing out Pennyworth as part of the 1.0b5 release, so auto-updating should take care of the transition.

Genkanban 1.0b4 is now available for download.

This release fixes a number of user interface bugs, including focus issues and
case-sensitive autofilling. This release also adds a variety of new observations to the iTunes observer.

Prerequisites

MacOS X 10.5 “Leopard”

Growl (Recommended): Genkanban can be used without the Growl notification system by viewing the current predictions, but this occupies precious screen real estate and you attention. In order to receive visual notifications of updated predictions, I recommend that you install and configure the Growl notification system before running Genkanban.

Source

There is not a source distribution available yet, but if you’d like the source code, it is available via Subversion at

http://context-macosx.googlecode.com/svn/trunk/Tools/Applications/Genkanban/

The source requires Xcode 3 to build.

Bugs, suggestions, or comments?

Please post any feedback in the comments below or send it to context@aetherial.net. This is early release software, so there will be bugs, and the application will only improve when problems are identified and submitted.

Genkanban ships with a variety of default sensors, but it also includes some functionality that allows you to create your own sensors using AppleScript. The process is simple and I’ll demonstrate it by creating a sensor that detects your currently selected weblog in MarsEdit.

Before we get into the details, here are some general notes about the AppleScript sensors:

1. AppleScript sensors are standard .scpt files authored in Script Editor.

2. The AppleScript files should be placed in the folder at

~/Library/Application Support/Genkanban/AppleScript/Sensors

3. On a set duration (currently 10 seconds), Genkanban scans that folder, and executes each script located within. The predictions generated by each script last for the set duration and will disappear once that duration expires or the observation is renewed on the next pass.

4. Scripts return observations by returning a semicolon-delimited string of key-value pairs in the form

key1=value1;key2=value2;...;keyN=valueN

The keys and values may contain any character, save “=” and “;”.

5. Genkanban translates the key-value pairs as separate observations in the form

AppleScript Observer (KEY) = VALUE

That said, let’s create a sensor for MarsEdit.

1. First, we need to create a script file to return the selected weblog. In Script Editor, enter the following script:

tell application "Finder"
    repeat with p in (processes whose name is "MarsEdit")
        tell application "MarsEdit"
            return "Marsedit: Current Blog=" & name of selected weblog
        end tell
    end repeat
end tell

Wrapping the call with the call to Finder allows us to only return a value when MarsEdit is actually running. If this was not included, the sensor would launch MarsEdit (if closed) each time it ran.

2. Test the script by running it with MarsEdit open and closed. When close, it should return nothing. When open, it should return something like:

"Marsedit: Current Blog=The Context Blog"

3. Save the script file and put it in the location mentioned above.

4. If Genkanban is running, in less than ten seconds, the script will be executed and the results will be visible in the observations window:

5. Congratulate yourself on a job well done.

Some parting thoughts: At the moment, Genkanban uses a ten second interval for executing and retiring observations. I may extend this interval to longer period or make it user-configurable. There’s a tradeoff between having reliable current observations and executing the scripts too often that I’m still negotiating.

Genkanban 1.0b3 is now available for download.

This is the first public release of Genkanban and should be considered quite beta. There are likely the normal memory leaks and other issues present in early release software. That said, I have been using this version for some time and it hasn’t presented me with any major problems.

This release fixes a number of UI bugs present in prior private releases.

Prerequisites

MacOS X 10.5 “Leopard”

Growl (Recommended): Genkanban can be used without the Growl notification system by viewing the current predictions, but this occupies precious screen real estate and you attention. In order to receive visual notifications of updated predictions, I recommend that you install and configure the Growl notification system before running Genkanban.

Source

There is not a source distribution available yet, but if you’d like the source code, it is available via Subversion at

http://context-macosx.googlecode.com/svn/trunk/Tools/Applications/Genkanban/

The source requires Xcode 3 to build.

Bugs, suggestions, or comments?

Please post any feedback in the comments below or send it to context@aetherial.net. This is early release software, so there will be bugs, and the application will only improve when problems are identified and submitted.

At its core Genkanban does two things: predicts your context and takes actions on the basis of the predicted context. When first installed, Genkanban does not “know” you and cannot predict your context until it is trained. There are two methods of training Genkanban, described below. In order for Genkanban to function, you must first spend some time training it.

All of the functionality described below is accessible from the Genkanban status menu represented by a bell in menu bar:

Training method 1: Uniform sampling

Using uniform sampling of your context, Genkanban collects context labels from you at regular intervals. This is the conceptual equivalent of someone asking you every five minutes, “Am I doing it right?”

To enable this training method, open the Preferences and select the Training tab. Within this tab, you can select a training interval ranging from thirty seconds to five minutes. This is how often Genkanban will interrupt you for new context labels. You may choose to enable an audible cue that sounds with the interruption. This sound is a small bell and may be useful if you work on a busy desktop where a small window my be lost among the others.

To begin training, click the “Start Training” button. The system will begin to interrupt you periodically until you stop the training. You may adjust the interval or toggle the audible cue without stopping and restarting the training process. In addition to the button within the preferences, training may be toggled using the “Start Training” menu item under the bell icon in your status bar. Restarting training does not begin the process anew – instead it simply continues from where it left off.

(Note that as of 1.0b3, there is no “reset training” functionality. You would like to start over, please post a comment below and I’ll provide instructions for resetting training. This functionality will be included in a future revision.)

Training method 2: Correct on error

If you do not want to be interrupted periodically and would like to take responsibility for monitoring Genkanban’s training, you can use a simple “correct on error” approach to teaching Genkanban. To do this, select “Predictions…” from the bell menu in your status menu. This will bring up a translucent black heads-up display (HUD) that can be placed anywhere on your screen. This HUD displays the context Genkanban currently predicts:

If you see that Genkanban is making an incorrect prediction, you can correct it by selecting “Correct Current Prediction…” from the bell status menu. This brings up the window used to train Genkanban and you can select an existing label or enter a new one. When you click the “Set” button, your current context and environment is sent to the machine learner and this generates a new prediction based on the new evidence.

The pros and cons of the different methods

The uniform sampling training method will collect more information about you as you use the computer throughout the day. You will not have to remember to give it new information, as it will periodically interrupt you for context labels.

The “correct on error” method eliminates the interruptions at the cost of you being responsible for verifying the predictions of the system. Ideally, this method only collects labels that contradicts the existing model, so the a wider variety of contexts should be predictable using a smaller number of samples.

Please note that these training methods are not mutually exclusive and can be intermixed as desired. You can correct the system’s current prediction while using the uniform sampling method. You my choose to begin using the uniform sampling method and then switch over to the “correct on error” method.

I personally prefer to use the “correct on error” method to avoid the regular interruptions. However, I do not know which method is the optimal one for building a more reliable model. This is an open research question that I plan to answer while building this system.