Mind Project Mobile Robot Team - Spring 2007 Snapshot Guide

Purpose

This guide details the setup of the Iris snapshot that was presented at the ISU Undergraduate Research Symposium in April of 2007. It also provides a high level explanation of all of the modules included in the snapshot and some other information. For more general documentation visit the Iris.4 Developer's Site.

Prerequisites

The Spring 2007 snapshot system was a fully updated Windows XP Professional system.

Java JRE

Large amounts of the Iris software are written Java, consequently they require the Java Runtime Environment. As of Spring 2007, the most recent version of the JRE was 6. JRE 6 is guaranteed to work with the software, later versions should work as well. The JRE download is available at Sun's download page.

POSIX Threads for Win32

In order to ensure cross platform compatibility, the CCP uses a Windows implementation of the POSIX thread library in place of the standard Windows threading system. This allows the CCP to be run on both Windows and *nix platforms without a problem. The required libraries are available at Pthreads Win32. No extra libraries are required for the Java clients.

Usbor Robix

Controlling Iris's arms requires that the Robix driver/software be installed. The software is available on CD in the Mind Lab.

Logitech QuickCam

The Spring 2007 snapshot uses a Logitech QuickCam 4000. Drivers must be installed for the camera so that Iris is able to request photos. Drivers are available at Logitech's downloads page for the camera.

ProtoThinker

The Spring 2007 snapshot did not use ProtoThinker, but it is available and easily integratable. The latest version of ProtoThinker is available on the Mind Project website.

Setup

Checkout

Most of the software needed for the Spring 2007 snapshot is located in the SVN in a single folder. Checkout a copy of IRIS4/Resources/2007ResearchSymposium/PresentationSoftware.

TTT Vision System - Neural Network

The TTT Vision System used by the Neural Network currently exists in a finished, but unpolished state. In order to give symposium viewers a closer look at Iris was doing, a decision was made to run the TTT Engine out of Visual Studio so that debug information could be easily seen. The necessary Visual Studio project files are available in the SVN at IRIS4/C++/vision/ttt/TTTVisionSystem-NeuralNetwork.

Execution

Visual Studio

Start Visual Studio and open the TTT Vision System-NeuralNetwork solution file.

Software Launcher

In the Presentation Software folder checked out in the above step, start SoftwareLauncher.jar

Start Modules

  1. Launch Central Control Program
  2. Launch Mind Module
  3. Launch Robix Interface
  4. Launch Camera Interface
  5. Start TTT Vision System-NeuralNetwork in Debug Mode
  6. Launch TTT Engine

Module Descriptions

Central Control Program

The Central Control Program is the server in the Iris client/server model. It should be the first module started every time and it must be present at all times in order for the modules to exchange messages. All messages go through the CCP before reaching their intended target. More detailed information is available at the Iris Programmers Guide.

Mind Module

The Mind Module serves as the brain of the robot. Currently, the Mind Module consists of a simple command-line interface that allows a human user to issue commands to the CCP, and a second thread of execution that sits in the background and passively allows all messages the CCP sends it to proceed through the system unchanged. Future Mind Modules will do much more processing on messages.

Robix Interface

The Robix Interface acts as the gateway between Iris's modules and the Robix controlling software. The Robix Interface should be present anytime one of the modules wants to use any of the arms. More detailed information about the Robix protocol is available at the ROBIX INTERFACE Protocol Page.

Camera Interface

This module acts as the gateway between Iris's modules and the web camera. Modules send a request to the camera interface for a shot, the interface processes the requests in a queue, and then sends messages back with the picture in bitmap format as an attachment. Mind Project alum Bob Arrigo wrote a bitmap class that makes converting the attachment into an actual bitmap very easy. Detailed documentation for the interface is available at the CAMERA INTERFACE Protocol Page.

TTT Vision System-NeuralNetwork

The TTT Vision System is responsible for fielding requests from the TTT Engine for an update on the status of the board. The format of these requests is defined on the Tic Tac Toe protocols page Currently, this module is still in a beta form and it must be run out of Visual Studio, as is described above.

If a developer wants to reuse Nick Kunkes's neural network code and apply to a new problem , they are required to write a new preprocessor. The preprocessor is responsible for taking the problem information and translating it into a form that the neural network can use as input.

TTT Engine

This module is responsible for starting and managing the Tic Tac Toe algorithms, coordinating the message requests between the camera interface and visions systems and presenting the GUI to the user. Complete documentation is available at the Tic Tac Toe Engine page

SVN Organization

Our SVN repository holds all of the Iris.4 source code. This section details what is contained in each folder. Where applicable, descriptions from the actual source code will be used.

C++

cameraInterface
This is a Windows API program and requires both the DirectX SDK and DirectShow to compile and link properly. This program provides an image acquisition interface to the Iris Software Environment for most Windows webcams, digital cameras, and other such video recorders.
ccpServer
This main simply creates a CCP object and runs it.
dummyCameraInterface
This program lets the user manually send images to modules requesting images from the camera interface. This is useful when you are drawing images or using old images (that is, images not captured in real time from the real camera interface.)
dummyclient
New dummy client using the managed CCP client object.
inputGenerator
Used in neural network training. Goes through a predefined directory structure looking for training files. Used by the neural network trainer.
learningAlgorithms
Implementation of a pattern learner for Tic Tac Toe. Exists both as a CCP client and a stand alone program.
mindmodule
In the future, this will act as the central point for Iris's operation. Currently it doesn't do very much.
parallelRobixInterface
CCP client that receives requests from other modules to execute Robix arm scripts
pictureSplicer
This program is used by the vision system while it is working on the Tic Tac Toe problem. It will take a bitmap and spilt it into nine files, based on the coordinates used to train the Tic Tac Toe neural net. A more robust system needs to be able to split the board dynamically.
ptinterface
Gateway module that sits between the network and ProtoThinker, necessary because ProtoThinker in its current state cannot be made into a client directly. Currently has capability to take input meant for ProtoThinker and to transmit ProtoThinker's responses back over the network.
ptreply
Captures PT's output and sends it to PT INTERFACE. Monitors REPLY.TMP file in ProtoThinker's directory.
robixModel
DirectX powered 3d model of Iris's arms created by James Barr. An idea for the expansion of this program might be the inclusion of control routines that allow the manipulation of the arm to actually control the physical arms.
rsml
This program can be used by other programs to launch Robix scripts. This is not a CCP client.
sapistt
CCP client that uses Microsoft's SAPI to convert speech to text
sapitts
CCP client that uses Microsoft's SAPI to convert text to speech
shared
Directory with components used by a number of modules. Allows for easy upgrading when new versions are released.
vision
Collects various parts of vision system. 24bitbmptograyscale is a command line utility that will convert an existing bitmap file to a grayscale only one. Also included in the folder are the two different vision systems developed for the Tic Tac Toe problem.
visionDisplay
Shows a window containing a live feed from the web cam. Useful for debugging purposes.
visionNerualNetwork & visionSystem
These directories contain Nick Kunkes's implementation of a neural network and the preprocessor code necessary for its application to the Tic Tac Toe problem.

Java

javaBasicClient
Java version of the CCP client. The author of a new Java module should either subclass CCPClientPrimitive or include an object of class CCPClientPrimitive in their module in order to handle message passing
randomLA
This is a Tic Tac Toe algorithm that plays by selecting valid moves at random
robixInterface
CPP Robix Interface to execute macros from script files when messages are received from other CCP clients with a type of CMD_RUN_MACRO and flags of script_name,macro_name
softwareLauncher
Software launcher for presentations
superLA
Tic Tac Toe learning algorithm that observes opponents moves and learns what moves lead to wins and what moves cause losses.
tttEngine & tttEnginev2
GUI and CCP client that coordinates playing of Tic Tac Toe.

Resources

RecruitmentFlyer.psd
This is a sample flyer that can be used for recruitment activities.
2007ResearchSymposium
As discussed above, this folder contains the software necessary for showing the same setup that was present at the 2007 symposium. It also contains all of the written material presented, including a copy of the project abstract.
Articulator.iso
This is an image file of the install CD for the Lucent Articulator text to speech system. This file can be used with any virtual cd mounting software.

Working with ProtoThinker

ProtoThinker, in its current state, is not made to work with CCP clients. However, it is possible, with some extra tools, to exchange information with ProtoThinker. The key to working with ProtoThinker is accessing the various files that it writes to its home directory. For example, its possible to access the response that ProtoThinker outputs. If "Output responses to Reply.TMP" under Speech options is enabled, ProtoThinker will write its output to Reply.TMP rather than display it on screen.

However, just writing to a file is not enough to make ProtoThinker part of the Iris network. We also need to be able to act on the contents of the file. One method, which can be inefficient, is to write a program that checks the contents of the file at a regular interval. Another, likely better, solution is to enable "Run Reply.EXE" under ProtoThinker's Speech options. With this option enabled, ProtoThinker will run Reply.EXE every time it writes something to Reply.TMP. Reply.EXE can then be used to pass information to a CCP client. However, Reply.EXE should not be a CCP client itself, as its execution behavior (Run, Quit, Run, Quit) does not agree with the rest of the CCP clients, which stay on the network for comparatively longer periods of time

Lets walk through designing a text to speech system for ProtoThinker as an example of building a CCP client that communicates with ProtoThinker. We will ignore the complexities of the actual text to speech system and just assume that we have a CCP client that can accept text input and vocalize it. However, we can't assume that the TTS client is running on the same system as ProtoThinker, so we have to pass our text over the CCP network. Therefore, our task consists of two steps, getting the information out of Reply.TMP and then passing it over the network to the TTS client.

The first step in our process requires writing a Reply.EXE program and writing a ProtoThinker centered CCP client. ProtoThinker will overwrite Reply.TMP with its response every time it outputs. So, Reply.EXE only needs to open Reply.TMP and read its contents. It then needs to pass that information over to our ProtoThinker CCP client and alert the client that it has some data to process. This can be accomplished with any message passing system. On Windows platforms, the Windows API for messaging might be used. On Unix platforms, files and signals can be used to accomplish the same task.

Once our ProtoThinker associated CCP client has the text in its possession, its a trivial task to pass the information to our TTS client. A message needs to be crafted with the text to be spoken as the attachment. Once the message is made, it gets sent to the TTS client and we hear ProtoThinker's response.

This design can be used to facilitate ProtoThinker's communication with any other module, such as the Robix arms or the camera.