DM 4144 Interface Design Advanced Studio
When talking about computer interfaces, we almost always think of sitting in front of a monitor, mouse, and keyboard and manipulating visual elements on the screen. This is an unnatural asymmetric interaction, with the human communicating using physical input, and the computer communicating using a visual response. This kind of interaction completely avoids sound, and greatly limits the tactile and visual interaction.
This course aims to break out of the WIMP (Windows Icons Mouse Pointer) world of interfaces and move into utilizing more of the realm of human senses. We start by looking at how humans interact, from speech to physical behavior, and then apply this thinking to creating engaging interfaces that use audio, video, 3D graphics, game controllers, sensors, and networking.
week 1 the six senses
- sight, hearing, taste, smell, feeling
- proprioception/kinesthesia (nope, no ESP)
- the human brain processes 160 GB/sec. in visual data alone
Reading: When will computer hardware match the human brain?
week 2 fundamentals of human interaction
- giving out vs. giving off
- 2 minute penalty for interruptions in concentrated work
week 3 tracking auditory events
- processing audio input in real-time
- dealing with noise
- realtime synthesis and effects
week 4 controlling video
- live camera input
- using movie files
- video effects and filters
- matrix operations
Assignment due: create a program that responds to auditory or visual stimulus
week 5 tracking visual events
- video tracking: grid, blob, color, movement
- environmental limitations for video tracking
- filtering out unwanted stimulus, targeting zones
Example: "The Manual Input Sessions:http://tmema.org/mis/ is a performance environment which tracks shapes made with the hands to generate audio events.
week 6 3D graphics
- core OpenGL? concepts
- displaying data in 3D
- Using 3D objects to represent interface ideas
Example: VIFE is a DJ/VJ environment where the performer manipulates 3D objects to control the music. The 3D objects in turn display their settings visually.
week 7 controllers
- tablets, joysticks, gamepads, mice, keyboards
- force feedback devices
- HID API (Human Interface Device)
Example: StickMusic
Assignment due: create a program which a user can control auditory and visual output in real-time using controllers, auditory input, and/or visual input.
week 8 sensors
- arduino with Pd firmware
- processing sensor data
- dealing with noise
Example: Audio Visual Aero Device is a immersive environment controlled by the position and movement of the user.
week 9 output to the physical world
- arduino with Pd firmware
- LEDs?
- relays to control bigger things
- motors, lights, servos, etc.
Example: YoYo Berimbau is a musical robot which responds sound, especially rhythmic events.
week 10 Mapping
- getting beyond direct mapping
- non-obvious reactions make for a more playful interface
- initial interaction should draw the user in
- learn from existing interfaces: musical instruments
Example: Bob Huott's Ski is custom built musical instrument with a novel interface.
Assignment due: analyze the mapping of an existing program, propose a more engaging mapping, and implement it. You can use your previous homeworks, projects, or some of the examples provided.
week 11 networking
- TCP, UDP, OSC, Service Discovery (DNS-SD)
- interacting across distances
- collaborative enviroments
Example: netpd allows electronic musicians to jam with each other across any network.
week 12 user testing
- do your ideas match up with your users' experience?
- work out final sticking points
- trial runs
week 13 final presentations
- each student will present their final project to the class for commentary and criticism from the class.
Readings
These are the core readings for the class, which are all required.
- Pd Manual http://crca.ucsd.edu/~msp/Pd_documentation/
- Snow's Two Cultures Revisited: Perspective from Design in Human-Computer Interfaces, by Bill Buxton http://www.billbuxton.com/2cultures.html
- Composing Interactive Music, by Todd Winkler ( MIT Press, ISBN 0-262-23193-x)
- Physical Computing http://itp.nyu.edu/~dbo3/physical/physical.html
- Computer Vision for Artists and Designers http://www.flong.com/writings/texts/essay_cvad.html
Optional Readings
This is a collection of readings on related topics for anything that you migth want to delve deeper into.
Interaction
- Buxton, W. (1990). The Natural Language of Interaction: A Perspective on Non-Verbal Dialogues. http://www.billbuxton.com/natural.html
- Buxton, W. (1990). A Three-State Model of Graphical Input. http://www.billbuxton.com/3state.html
Audio
Video
OpenGL?
Networking