Kinect-Based IWB

Ever since a $3000 bounty was placed on cracking open Microsoft’s fab new gaming hardware, the motion-sensing Kinect for Xbox, hackers and tinkerers have been putting the open-source drivers to lots of interesting uses on platforms that Microsoft never envisioned.  I’ve written about interesting Kinect hacks before (and before that,) and I’ve written about my experience with the Wii-based $50 Interactive Whiteboard (IWB,) but I haven’t seen a fully-developed Kinect-based Interactive Whiteboard.

Perhaps an Interactive Whiteboard is too narrow a description.  Many of the pieces are in place (see below) to interface with a computer using Kinect.  So, as with the Wii-based IWB, any application you can use on your computer can be controlled by this hardware.  If you connect your computer to a projector, you essentially have an Interactive Whiteboard.

Is the Kinect-based experience different from a Wii-based IWB or a Smartboard?  Almost certainly.  There would be no need to touch the screen at all, but rather to gesture in front of the Kinect to interact with the projection on the screen.  Would this be an improvement?  I’m not sure.  A touch-based IWB is more analogous to traditional whiteboard that uses markers and an eraser.  So, the touchless experience would be quite different.  I need to try it myself to really wrap my head around the opportunities that this motion-sensing interface offers.

I’m not sure if anyone here at Ohio State is working with Kinect as an interface for non-Xbox applications.  But I do know that the Digital Union has a Kinect which could probably be used to see if and how things work.  If anyone else is interested in trying to pull this together, drop me a line or leave a comment.

Multitouch with Kinect

Kinect on a Mac

Multitouch Kinect

Kinect Fingertip Detection

Kinect + PC + Mario = Fun

Gestural Interfaces and 2-Year-Olds

In the video above, a dad asks his son to draw something on a new iPad, the ubiquitous Apple tablet.  The 2-year-old clearly has some facility with the device as he casually switches between apps and between tools within the drawing app.  Interestingly, (though not surprisingly for anyone with a 2-year-old,) the boy also wants to use his favorite apps including playing some pre-reading games and watching videos.  He very naturally fast-forwards through the video to his favorite part.  He also knows to change the orientation of the device to properly orient the app to a wider landscape format.

Although I like gadgets, I’m not a true early adopter.  I do carry a PDA — an iPod touch — which my 2- and 4-year-olds enjoy playing with.  It’s amazing how quickly they understand gestural interfaces, pinching, pulling and tapping their way from app to app.

While I don’t think that I need to rush right out and get my kids iPads so they don’t get left behind, (the whole point is that they’re easy to use anyway,) I do wonder about some of the interesting opportunities for learning on these devices: drawing, reading, and linking information.  Of course, they also do a lot of these things on paper which places far fewer limits on their creativity — instead of choosing from 16 colors in a paint program, they can choose from 128 crayon colors or create their own by mixing their paints.

In the end, this new technology is flashy and fun, but I’m not convinced that iPads and other tablets are essential tools that will give our kids and our students a clear learning advantage.  I sure would like one, though.

Building Blocks 2.0

If I told you we were going to play a game by stacking a bunch of smart phones and moving them around, you might get a picture in your head like the one above.  But there is actually a simpler, more fun way to go about this.

Last weekend, I discovered Scrabble Flash in the toy aisle of my local grocery store :

Each of the five game pieces is a small, location-aware blockwith a screen that displays a letter.  By rearranging the blocks, words are formed.  The blocks are all aware of each other, so they can tell you when you have them arranged to spell a word.   Several different games can be played with this remarkable little interface.  Apparently, Scrabble Flash was released in time for Christmas last year, but I didn’t notice it until now.  For about $30, I may have to pick this up for myself.

When I first saw Scrabble Flash, I thought it might be a commercial manifestation of Siftables, a similar interface designed by an MIT student that I wrote about a couple of years ago after seeing this TED talk.  It turns out that Siftables are now Sifteo:

Both Scrabble Flash and Sifteo are block-like computers that are aware of the others in their set.  Scrabble Flash is not as robust with only three games available on the monochrome display.  But it is available now and the price is reasonable.  Sifteo blocks are full-color screens that are motion sensitive and connect to a computer wirelessly, which means more games can be downloaded as they are developed.  But they won’t be available until later this year and I suspect the price will be higher than Scrabble Flash.

Is this the future of language games?  That would be a pretty bold prediction.  But clearly as we all become more accustomed to using apps on our smartphones, these kinds of “toys” will begin to feel like a very familiar technology.  Scrabble Flash is an affordable entry point, but I’m excited that Sifteo is actively seeking developers to create more games.  They already have several learning games but there is potential for many more.

Kinect Development

Microsoft recently announced plans to release a software development kit (SDK) for the Kinect.  This should allow academics and enthusiasts to find new ways to connect the motion-sensing Xbox hardware to other platforms, such as desktop and laptop computers, much more easily.  In short, there should be many more Kinect hacks to come.

I’m still not sure how this would directly apply to classroom teaching, although it stands to reason that these applications could someday replace physical interactive whiteboards in the same way that Kinect was originally designed to replace physical videogame controllers for the Xbox.

For more, see my previous post on Kinect Hacks and below for some new examples of how Kinect is being used in new and exciting ways.

Control Windows 7

The touchless multitouch is really nice.  Mice are so 2008.

3D Tetris with Face Tracking

As the user moves his head, the perspective on the screen changes to match so that the 3D perspective is constantly updated.

Kinect Lightsaber

A wooden stick becomes a lightsaber in real time.  This would save hours of  frame-by-frame editing.

Balloon Body

After Kinect scans your body, use your scroll wheel to expand or contract the surface.

Christmas Lights

Use Kinect attached to a bunch of dimmers to control Christmas lights for a very nice effect.

Flying Robot

The 3D capability of connect makes it perfect for a robot that navigates three-dimensional space.

Hacking Kinect

I never really thought much about Microsoft’s Kinect until I saw what hackers were doing with it.  A story in the New York Times outlines how a designer and senior editor at Make magazine posted a $3000 bounty for the first person to post an open-source hack of the Kinect interface.  Huzzah!  In fact, I’m still not that impressed with it — 3D drawings are cool, but will they help me teach English? — but I’m thrilled that hackers big and small are poking around under the hood.

Interestingly, Johnny Chung Lee, who became famous for his TED talk where he described hacking a Wiimote to act like an interactive whiteboard, is involved in the development of Kinect.  Microsoft were so impressed with his skills on the Wii-based IWB and other projects they hired him.  He is reportedly very happy to see hackers taking on Kinect in the way he took on Wii a couple of years ago.  If a hacker can squeeze an interactive whiteboard out of a $40 Wiimote, what will come out of the $150 Kinect system?

Will this technology help us teach ESL and EFL?  It’s not easy to see how, at least not immediately.  But prepare for a giant step forward in how we interface with computers in the next few years.  Interactive whiteboards are just the beginning.  You can always show your students this video and ask them to predict the future (in English).

Interface with the Future

I’ve been thinking about interfaces for a while. Interactive whiteboards (IWBs) probably got be thinking in this direction, and the recent announcement of the motion sensing Project Natal for the Xbox has finally pushed me to write about them.

Note the fine print: actual features and function may vary.

As computers have evolved, so have the interfaces we use to interact with them.  To give you an idea of my own personal timeline, I have no experience programming on punchcards, but I do remember the pre-GUI days.  The mouse has become ubiquitous, but what comes next?  Here are some possible answers:

Interactive Whiteboards: These are pretty heavily documented on this blog and elsewhere.  Here, the user can interact with the display, making the mouse and, to an extent, the keyboard unnecessary.

Slap widgets: Physical tools to be used with IWBs and the like, further blurring the line between the physical and the virtual.  (Watch the video!)

Siftables: An interesting miniaturization of an IWB-type interface with the added bonus that each mini screen can interact with the others.  If each Siftable had a letter or a word, how could they be used to teach English?

MIT’s Sixth Sense: This video went viral a few months ago, but keeps popping up.  Why can’t your computer interface with the physical world?  It soon may.

Wii & Natal: Nintendo has enjoyed blockbuster sales of it’s Wii gaming system which features motion-sensitive controllers.  Now, Microsoft has unveiled the same thing with Natal, but no controllers are needed.

Do it yourself interfaces: In addition to making your own IWB using a Wiimote, other examples include the Beatbearing, in which ball-bearings are used to sequence electronic drum beats, a modified Nintendo Powerglove, which upgrades late 1980s virtual reality technology, and a giant Katamari ball controller, which is a controller designed and built specifically for the game Katamari, in which players roll a ball through an environment which attracts more stuff (litter, park benches, people, their pets, etc.).  The makers of each of these interfaces have posted detailed plans and instructions online so that you can build and modify your own.

Move over visual learners, kinesthetic learners are about to have their day.  And why not?  Why should  simply watch something when we can interact with it in more natural ways?  The next generation of computer interfaces promises to expand our idea of how we relate to computers.

Siftables: Perfect for the Sandbox

A colleague of mine, Kathy, and I have been discussing ways to create a language sandbox game.  This idea would be to create blocks with words on them that somehow interact in a way that demonstrates grammaticality.

Siftables work like colored blocks, but they can interact.

Kathy’s oldschool.  She’s been experimenting with blocks of wood painted different colors which students can manipulate.  Great for kinesthetic learners!  We discussed cutting them into puzzlepiece shapes so that each block only “fits” other words according to grammatical rules.  At first, it seems like it would be possible to make nouns with square tabs that fit into square slots on verbs, and so on.  However, as complexity increases, this becomes exponentially more difficult.  Structures as complex as nouns modified by multiple adjectives would be prohibitively difficult.

What if a computer application could be developed that would replace the wood blocks with word tiles that could be manipulated with a mouse (or an interactive whiteboard!)?  Could the tiles snap together and repel each other like magnets according to grammar rules?  Could words be tagged for part-of-speech automatically within the application?  Could different categories of words (verbs, adjectives, adverbs, specifiers, etc.) be added and deleted with the check of a box?  Could students add their own tiles seemlessly into the pile?  Clearly, some intelligence would be required of the application to implement all of these features.

Siftables react to each other. Imagine a word on each one.

As I was kicking all of this around, my friend Mike at Ohio University pointed me to siftables, which seem to be the synthesis of both ideas.  Rather than try to describe these brilliant little devices, watch the TED Talks video.

Not only could these little devices fit the bill perfectly, the way they interact could inform interactions in the language sandbox I’ve been envisioning.  Until we’ve all got pockets full of siftables to pass out in class, my $50 wiimote-based interactive whiteboard will have to do.  In the meantime, I’m hoping that having students drag word tiles across the screen will work almost as well for kinesthetic learners.