Thesis Paper Draft v.01

Link Huang
Art Center MDP
Thesis 2011

Perceptual Exaptations Part 1:
Hyperreality with Physical Mutation Interface

Introduction
How can we manipulate/exploit human sensorial perceptions? The adaptations and exaptations of perception through evolution and the rapid increase in technological developments has greatly expanded long since human learned to apply their senses in a primitive way. If the interaction is intuitive, what type of new functions and controls can be introduced to existing senses? The main tasks of vision is to navigate and identify but a good example of an expatiation of the human eye function is communicating emotions through the scaling of the iris. Vision is important because it directly informs us about our environments with objects and spaces, but sometimes alternative ways to retrieve this information are researched and developed for people with various disabilities. My investigation aims to promotes new functions of our perception through the ideas of “sensory fusion”, “sensory substitution”, and “sensory hijack”. Many of the projects dealing with sensory substitution are designed for the blind and people with disabilities, when these new affordances are applied to a fully functional person, how can these activities inform new interactions embed the notion of discovery and stimulating imagination? The contents of my experiments are inspired by neurology, cognition, gestalt psychology, with apprehensions through phenomenological and biological factors. The vehicle of communicating these hybrids of ideas are executed through physical animation through tangible parts driven by motors and servos to move electronic devices which are inspired by structural language and techniques used in visual effects and motion graphics.
Motion graphics and visual effects mediates the world of fantasy and reality. For centuries, animated graphic representations and information displaying in a scripted scenario have been a tradition for storytelling, enhancing the visual and cognitive perception allowing the viewer suspend their disbelieve and immerse in the scripted fictional world. In the entertainment world this hyperreality are incorporated, from stage performance, silent films, to television and cinema. In an industrial design scheme, old clock works, automatons, mechanics to drive these animations requires mathematical calculations and the art of craft making. Visual effects are mediated in many ways, although one particular aspect of morphology that continuously incorporates a mesmerizing effect is through transformation and mutation of forms over a predetermined period of time. When one objects structurally alters itself to another representation, it instigates curiosity in visual perception and the phenomenology of understanding. If this mutational ways of animation translates and represent in tangible objects, how can it give the user the affordance as an interface? While operating this shape changing device, the user will require not only using their sight, but also touch and motor coordination. What type of algorithmic system within these mutational devices can promote understanding, discovery in conjunction to operate as an interactive physical user interface?

Historical / Context
Hyperreality comes from layers of creative imagination. A humanistic desire for a different world. A world in which varies by experiential interpretation creatively expressed in a system of simulated experiences. The reality are mediated to an extent that can no longer be distinguished from fantasy. Take Disneyland for example, as Jean Baudrillard puts it, “An imaginary effect concealing that reality no more exists outside than inside the bounds of the artificial perimeter”. (Baudrillard,1). Baudrillard defines three orders of simulation. First would be where the representation of the real world is artificially represented in different mediums such as books, painting, map, etc. The second-order simulation blurs the boundaries between reality and representation. The third order describes the simulation and it’s surroundings which he calls hyperreality. It is produced algorithmically like the computer programing code to construct virtual reality or augmented experiences. Baudrillard believes that hyperreality will dominate the way of experiencing and understanding the world we live in. (Lane, 2).
Visual perception isn’t a passive act, rather it’s an active learning experience. Human learns to see and understand the data perceived through active body motor movement to understand the world around us. It involves the integration of multi-sensorial processing ranging from our senses and our muscles to coordinate with the neurological brain function. It is a fundamental understanding in which animation needs to be constantly moving to feel a continuous flow, even if it’s a static frame, the object or text will still be drifting in slow motion. Therefore, translating these techniques into tactile devices will also require movement for a better understanding and organic spacial relationship between the object and the person.
Many motion graphics and visual effect techniques originated from theater and stage art. Some example reference such as artificial weight goes as far back as Pantomime performances from the 1700s. As entertainment advances with technology, so does the effects to enhance visual perception.
Tactility isn’t only about touching from the tip of the fingers, crafters such as sculptors and pottery artists requires the use every aspect of their hands, from the palm, nails, multiple fingers, to the back of their hands or even other parts of their limbs and torso. By using sensitive touches and feel in conjunction with a keen observation of form, to carefully perfect the craft.
Many hand made craft working skills prior to the Industrial Revolution has been lost or eliminated. Many of these detailed and elaborate hand-made skills makes objects and architecture more humanistic. Every duplication is different with it’s unique imprints left by the maker, even with decades of experience, every hand-made craft still have it’s variations. It makes each item a unique one-of-a-kind. Since the invention of the assembly line manufacturing process, every mass produced item looks identical. Sure it lowers the cost to purchase these items, but each item is cold and lifeless.
After we moved into the digital era, there is a major dehumanization with all the devices and tools. Everything is mass produced to give a digital affordance. Limiting our senses to perceiving information on a flat screen. Most tactility within the interface in interaction revolves around only the tip of our fingers. Keyboards and mouse has made humans as an extension of the machine itself, making us as part of the digital system to keep the world operational.
In the next phase of inventions will be a merge between the tactility of craft and digital affordances. Every control on the digital touch screen can be re-designed digitally with tangible objects without a screen.
Tangible user interfaces provides affordances to digital information while facilitates the advantages of human’s capability to grasp and manipulate physical objects in the real world. (Ishii, 3). In an environment full of activated devices which implies rich contents and interactions that are embedded in tangible things and inhabitable spaces will assemble a new symbiotic system with ongoing relationship human and their environment. This new ecology of things provides an evolving system that can be interpret and influenced by the interactions and decisions by people or other objects. (Allen, 4).

Process
My goal through these experiments are to translate the notion of motion graphic and visual effects animation and embed these forms and movements into physical objects, devices, and environments. These project explorations are initiated through a matrix chart that I have put together. One side of the grid being my ingredients and definition of every visual effects language applied. The other side of the grid includes eye tracking, other sensorial input sensors (breath, heat, proximity, force, bend, touch), gestalt/cognitive psychology, and neurology tests/theories. By infusing the cross section of two specific directions, the result provide informations which allows me to further develop innovative ideas to promote new functions for various human senses. To retain the value of these devices, they will be speculative tools containing the ideas of manipulation, interpretation, discovery, and stimulating imagination.
During my investigation, many contextual information comes from studying neurology and psychology of human perception. Not only does the human brain interprets human sensorial inputs through interactions with it’s environment, but it is constantly creating gestalts from incomplete information in attempt to make connections and logical sense of the abstraction. (Ramachandran, 3) This information can derive from the foveating vision, far periphery of vision, touch and feel, or any senses of the human body. Understanding the way our eyes and brain works as well as the phenomenological comprehension has greatly impacted my decisions to create devices that puts a twist to the human perception.

Prototypes
Physically Animated Motion Graphics Immersive Space is the first experiment during this investigational process. If the text and graphic elements have volume which occupies physical space, how can these elements give a similar experience inspired by television commercial and film title sequence? My approach in this investigation utilizes several physical graphic elements connected to ropes driven by multiple pulleys. This installation is presented in a dark space with illuminated graphic elements. By sitting or standing in a designated position, the viewer can experience a pre-scripted animation as these graphic elements move into the periphery in sequence.

Saccade Controlled Visual Angle, there is two versions to this project. Both controlled and view through an eye-tracking glasses that includes a hacked PS3 infrared webcam for the left eye and a LCD monitor mounted for the right eye. The LCD monitor is mounted on a bracket driven by two feather weight servos to allow a two axis rotation to follow the focus of the pupil. The main task of the eye-tracking is to control another physical device that holds a viewing camera for the LCD screen. The biggest difference between these two version is how the viewing camera was mounted.
The first iteration of this experiment is called “Extended Sight”, where the viewing camera is mounted on a small robotic bracket driven by two standard size servos to allow x and y axis rotations for the camera to look around the outside world. If the human eye sight can be physically moved to another location, what perspective can the viewer perceive?
The second version takes on the metaphor of jump cut techniques in film editing to see different angles. When a person gaze at something or someone, naturally the eye focus would constantly jump back and forth, scanning the contours and following while identifying various information of the subject. These quick eye movements are call Saccades. Although our brain interprets these saccades to be seamless, but biologically we are blind between these saccade points. The glasses mount eye tracking camera controls two semicircular acrylic armature driven by two servos that rotate in x and y axis. The main purpose of this structure is to allow the viewing camera to rotate 180 degrees around an object that can be placed on a central platform. Viewing the object in various angles. Due to the scale constraints, he maximum size that this specific prototype will allow to revolve around is 6in x 8.5in.

Super Hero Gaze Telekinesis takes on the idea of visual effects in film when the super hero has the power to control and manipulate things with their eyes. The controlling interface of this experiment has it’s similarities with the Saccade Controlled Visual Angle project. It also utilizes an eye-tracking glasses to control a flash light driven by two axis servo motors. As the user look in different directions, the flash light points at the direction the viewer is looking at. The purpose of this setup is to open up a physical box while looking at it. The box is constructed with a light sensor on the top center. As the light triggers the photocell, a servo underneath the box will pull on four strings that are attached to all four sides of the box and pulls the four sides like flaps folding downwards in a synchronized motion. This experiment investigates using the gaze of the human eye to trigger physical transformation of objects in the real world. What if everything in the world can be activated and controlled by just looking at it?

Physical Mutation Interface is an interactive morphing interface inspired by the ideas of organic mutation, morphing, and Cognitive visual angle. It is a speculative experiment that maybe suggestive for an interactive physical interface that can be implemented for consumer electronics. What if the form, function and interface can be changed by a simple gesture of wrist rotation? This stand-alone device is driven by a 3-axis accelerometer, controlled by an Arduino board, which triggers three servo motors, one for each axis of rotation. The servo arms are attached with strong skeletal armatures that rotates in their own independent directions without getting in the way of each other. This device is then wrapped in cloth to hide the mechanics within to produce a morphing look and feel from one shape to another, creating multiple and interchangeable surfaces. This interface provides new modes of usage as the panel changes and constrain functions to each display.

Reflections
The experiments I have created so far investigates two major directions. One is a perceptual expatiation of the eye, allowing control and manipulating the user’s own perception. The other is creating mutating physical interfaces, allowing the user to manipulate and influence the form and the outcomes of the device. Both has their technological limitations and the interactions are quite simple and on the surface level.
Due to the use of a motion tracker for the eye-tracking device, the location of the iris isn’t always accurate because it requires movement for the tracker to detect the location of the pupil. This creates a problem which the tracker only works a little more than half of the time while the user is trying to control the device. If I were to further develop the possibilities of eye tracking to manipulate vision, I will look into a better software for eye tracking.
The physical mutation interface is still in very rough initial stage. The interactions are very straight forward, only allows for one or two manipulation of it’s form and yet the interactions and affordances on the human receptive side is not complex enough to create productive and variety personal interpretations. The importance of the physical mutation interface is not the result and outcome but more about the process of the interactions.

Plans for Spring Term
Over the Christmas break and next semester, I plan to further investigate the idea of mutating/transforming physical objects as interfaces. How will these operate in a more in-depth matter? Giving complexity to the interactions and it’s algorithms to promote a productive interaction that provides various results among different users and create personalized interpretations. Not only investigate the interactions of these mutating interfaces between object and human, but also between objects and objects. How will an ecology of mutational objects communicate and/or influence each other?

Bibliography
1. Baudrillard, Jean, Simulacra and Simulation (The Body, In Theory: Histories of Cultural Materialism), University of Michigan Press, February 15, 1995.

2. Lane, Richard J., Jean Baudrillard, Routledge; 2nd edition, January 16, 2009.

3. Ramachandran, M.D., Blakeslee, Sandra, Phantoms In The Brain, Harper Perennial, 1999.

4. Ishii, Hiroshi, Tangible Bits: Beyond Pixels, Tangible Media Group at MIT Media Laboratory, February 18-20 2008.

5. Allen, Philip V., The New Ecology of Things, Media Design Program at Art Center College of Design; Limited edition, April 16, 2007.

Abstract V.05

Perceptual Exaptations

How can we manipulate/exploit human sensorial perceptions? The adaptations and exaptations of perception through evolution and the rapid increase in technological developments has greatly expanded long since human learned to apply their senses in a primitive way. If the interaction is intuitive, what type of new functions and controls can be introduced to existing senses? The main tasks of vision is to navigate and identify but a good example of an expatiation of the human eye function is communicating emotions through the scaling of the iris. Vision is important because it directly informs us about our environments with objects and spaces, but sometimes alternative ways to retrieve this information are researched and developed for people with various disabilities. My thesis explorations investigates and promotes new functions of our perception through the ideas of “sensory fusion”, “sensory substitution”, and “sensory hijack”. Many of the projects dealing with sensory substitution are designed for the blind and people with disabilities, when these new affordances are applied to a fully functional person, how can these activities inform new interactions embed the notion of discovery and stimulating imagination? The contents of my experiments are inspired by neurology, cognition, gestalt psychology, with apprehensions through phenomenological and biological factors. The vehicle of communicating these hybrids of ideas are executed through physical animation through tangible parts driven by motors and servos to move electronic devices which are inspired by structural language and techniques used in visual effects and motion graphics. My main focus is around exploiting visual perception, whether it is the understanding of seeing or the physical act of looking, but I am also interested in exploring other sensorial perceptions such as touch, feel, breath, sound. Visual perception isn’t a passive act, rather it’s an active learning experience. Human learns to see and understand the data perceived through active body movement to understand the world around us. My project explorations are initiated through a matrix chart that i have put together. One side of the grid being my ingredients and definition of every visual effects language applied. The other side of the grid includes eye tracking, other sensorial input sensors (breath, heat, proximity, force, bend, touch), gestalt/cognitive psychology, and neurology tests/theories. My investigations will provide informations which will allow me to further develop innovative ideas to promote new functions for existing senses. To retain the value of these devices, they will be speculative tools containing the ideas of manipulation, interpretation, discovery, and stimulating imagination.
My obsession began from studying motion graphics in my undergraduate program at Art Center with a handful of years working in the industry prior to returning to grad school. Through this experience, I have developed an interest in the way visual perception works. With the interest in gestalt and cognitive psychology, I have began to study the neurology of visual perception as well as disorders such as synthesis and various blindness (color, face, object, motion). My hobby while growing up however, revolves around auto mechanics, and craft works using materials such as metal, wood and plastic. With the hybrid of my professional background of motion graphics and my hobby in mechanics, my interests have emerged into building tangible interface, apparatus, and installations inspired by the techniques and languages of visual effects.

Literature Review Rough Draft

Literature Review Rough Draft

How far can we push the human sensorial perceptions? Visual perception for example, is an active gesture rather than a passive one. In order to understand what we see, our body relies on sensory motor (body movement) and other sensorial inputs including phenomenological interpretations to draw conclusion of information perceived.(1) Cognitive psychology is a basis for many HCI developments. Although many HCI are designed around functionality and less on visual perception, rather more focused on the ability to understand and control through cognition. Motion Graphics and Visual Effects animation are executed based on the same motor sensing theory, except it is restricted to the spacial dimensions within the screen.

Visual effects and Motion Graphics are based on the idea of active view through movement, therefore, introducing the idea of a slow multi-dimensional drift to reinforce the understanding while the audience absorbs the text and graphical information. Through movement, we also get a spacial understanding of the scripted environment. Therefore, even though screen-based visual effects and animation is displayed on a flat screen, the space within is portrayed with the possibility of a greater or infinite dimensions. There are not many physical interaction projects based on the visual effects language, but any existing projects can be interpreted in relations with some aspects of the VFX techniques and philosophies.

Spacial recognition through sensory motor is crucial to our perceptual understanding. In the 1960s, Richard Held and Alan Hein exercised a famous study about “Movement-produced stimulation in the development of visually guided behavior”. The experiment placed two kittens on either sides of a device that looks like a classic weight scale. One kitten was allowed to walk with protruding limbs, the other is held fully suspended without the affordance of mobility. The kitten that is actively exploring the environment have normal visual system compared to the kitten with only passive visual stimulation. (2)

Many researchers dealing with perceptual technologies develop ideas geared towards people with disabilities, lacking one or more sensory input to the brain. Therefore, not too many of these developments are geared to art and expression mainly due to budget, profit, and marketing aspect. In that respect I can understand but many of these researches can be very rich if applied to new ways of interaction with devices or tools for the everyday use.

Through technology, one sense can be replaced with another. In the psychology world, this is called Sensory substitution, which was coined by Paul-Y-Rita in 1969. (3) His experiment using a camera that turns pixilated image into braille ,a interpretation of dark and light pixel contrast, protruding onto the back using the sense of touch and feel to understand imagery. Another example around this idea is a project that uses similar idea except it impliments the transfer of pixelated images to a sensation on your tongue using small electricity emitters layed on a grid and placed across the top surface of one of the most sensitive location on our body.
Sensory Hijack is about stealing from one sense to apply to another. It is similar to sensory substitution, but with slight obvious differences. Couple examples I like to bring in for this example is “Finger Sight” (4) which uses sound waves or reflected laser beams to give haptic feedback on the side of fingers, allowing the blind to navigate. Another example is also developed for the blind, called “The vOIce” (5) which is a device and system that allows the blind to see with sound. The design of the pitch triggers the neurological sensors in the brain that are designed for sight. Another term to explain this phenomenon is “Artificial Synethesia”.

Taking particle animation infused with Gestalt psychology’s theory of “emergence”, and execute the idea in the physical interactive world is the BMW kinetic sculpture by ART+COM. (6) This installation brings animated and synchronized particles to the physical world. Allowing the viewer to experience the effect in a physical space while providing the ability to view from various angles. This is one of the inspirations which brings me to the interest of translating the visual effects language to the real world rather than on screen and the only from the point of view designated by the creators.

Optical illusions are a way to inform us about the way human perception works. Through an interview on TED on a neurologist/artist name Beau Lotto.(7) He works with Perception-bending projects. Some of which are installation art, and some are smart phone apps, as well as object based speculative projects. Many of which Lotto describes as synethetic experiences.

Through these examples, and more out in the world, one can see the vast variety of implementations through manipulating perception with technology. Many of which used for medical feel like there is a large gap between these technologies and HCI. Although once a similar “artificial synethesia” is implimented into a speculative project for the everyday user or an communicative art installation, it becomes a impressive experience. The way humans perceive isn’t a one to one interaction. It involves motor sensory as well as cognition to comprehend the message being communicated through these speculative objects or art installations. Human interactions should not be flat. If it is required to be on a flat surface such as a digital reader or personal computer, the interface should introduce some dimensionality for the audience to fully engage. The ultimate solution is giving a dimensional affordances to every possible Human-Computer Interaction technology.

Bibliography

1. Noë, Alva. Action in Perception. Cambridge, MA: MIT, 2004. Print.

2. R. Held and A. Hein, “Movement-produced stimulation in the development of visually guided behavior.” Journal of Comparative and Physiological Psychology 56(5): 872-876

3. Bach-y Rita P, Collins CC, Saunders F, White B, Scadden L.(1969). “Vision substitution by tactile image projection.”. Nature, 221:963–964.

4. Finger Sight
Stetten, G., Klatzky, R., Nichol, B., Galeotti, J., Rockot, K., Zawrotny, K.,
Weiser, D., Sendgikoski, N., Horvath, S., Horvath, S., 2007. Fingersight: Fingertip visual haptic sensing and control. In: Haptic, Audio and Visual Environments and Games, 2007. HAVE 2007. IEEE International Workshop on. pp. 80–83.
URL HYPERLINK “http://dx.doi.org/10.1109/HAVE.2007.4371592″ http://dx.doi.org/10.1109/HAVE.2007.4371592

5. The vOICe. Synthetic vision through auditory video representations.

http://www.seeingwithsound.com/

6. ART + COM, Kinetic Skupture. BMW Museum Munich, 2008

http://www.artcom.de/

7. Beau Lotto (Neuroscientist and artist) – Perception-bending
HYPERLINK “http://blog.ted.com/2009/10/08/beau_q_and_a/” http://blog.ted.com/2009/10/08/beau_q_and_a/

Perceptual Exaptations Manifesto

Perceptual Exaptation is:

Is a exploration to invent new uses for the human visual perception whether it is the sensation and phenomenon of seeing or the act of physically directing the eye. Playing with various physical parts of the human eye, including the Iris, retina, fovea, and blind spot to create new ways of functional productive interactions.

Is a series of investigations that involves the physical human eye, cognitive psychological concepts, Gestalt psychology, and Visual effect languages.

Is a translation of visual effects techniques to the tangible world using motors and sensors embedded in mechanical parts.

Is using the structural language of visual effects as a vehicle to create tangible apparatuses that introduces new uses for existing bodily functions.

Is open to the experimentation of extending the human perceptual functions to other locations of the body or other locations of the physical world.

Is open to the experimentations involving two or more ways of interactions involving two or more personals. Is actively interactive.

Perceptual Exaptation is not:

Is not a optical illusion

Is not a magic trick to deceive the human eye

Is not a engineering dominate project, but rather more expressive with metaphors.

Article Summary

Critique of Alva Noe’s book “Action in Perception”
- by Ned Block

This article lays out the fundamental ideas of Noe’s book Action in Perception with the writer’s thoughts and criticisms. There are several intriguing and provocative explanations. It combines neuroscience of perception and the phenomenology of experience with an appreciation of the psychology. Noe believes in the process of visual perception, dependencies on the interactions between the observer and the spacial environment includes multiple sensories not only from the eye. There is a great emphasis that vision is not passive, but rather an active act of seeing along with touch or body movement to reassure what is perceived with additional information and conformation. The act of body movement constructs the experience of visual perception.
Noe claims the idea of “enactive view”, he explains that “Perceptual experience, according to the enactive approach, is an activity of exploring the environment drawing on knowledge of sensorimotor dependencies and thought” (p228). Sensorimotor serves to explain the way sensory stimulatenactiveion varies as you move. Sensorimotor knowledge is knowing how objective appearances change as you move but it is also a matter of knowing how rather than knowing that. The mind-body informations will inevitably be a source of attention to the enactive point of view.
The enactive view means the perceptual experience depends on sensorimotor contingencies (body movement and other sensory information to hypothesize. Our tactile experience of solid objects depends on sensing its resistance when we push against it, thus showing a minor dependency on the experience on action.
Humans and other primates have two distinct visual systems, a conscious visual system that begins in the back of the brain, moving to the bottom and side(ventral system) and from the back towards the top of the brain, a much less clearly conscious (dorsal system). The conscious ventral system is slow, directs towards long term visual planning of motion and uses object-centered observations from a stereotypical view instead of using the view from the current position. The egocentric dorsal system is fast, representing the distance and orientation without memory or color vision to guide the action. For example dribbling of a basketball down the court and avoid obstacles uses the dorsal system. Another example would be walking down the sandy beach, our feed avoids stones that we don’t seem to see. The dorsal system feeds more strongly than the ventral system to the peripheral vision.
The author defends the Alva’s attack by two points he’s trying to convey. First the information perceived through the experience is the brain and doesn’t include the rest of the body. Second, although body movement output instructions that affect the perceptual experience, the experience that is understood is abstract. I disagrees with Block’s defense because I too believe the active perceptual experience requires feedback from multiple senses not just from the passive sensation of the eye.

Abstract V.04

Abstract

Tangible Visual Effects is a thesis investigation on the structural components of screen based visual effects, (ex: animatronics, multiplicity, scripted depth of field, type animation, mutation/transformation, lighting effects, augmented interfaces, etc) and seeks to explore ways in which they can be deployed to form a new language of tangible interactions. I believe the structural components of visual effect techniques can change future interaction. This can introduce and develop of new uses for informing future perceptive interactions. This would entail undertaking experiments that explore the possibility of extending the visual experience to other senses and subverting expectations of “familiar interactions” through the tools of illusion, mutation, discovery, and stimulating imagination.

References

There are certain attributes in devices when and if manipulate, creates a sense of illusion. It is a natural phenomena to dilute one’s mind just as a chameleon can change colors to deceive predators while sitting on a tree. I’m creating illusion to dilute the eye, experimenting with perception and sub-conscious to allow variations of experiences by individuals.

As Maurice Merleau-Ponty observed that “consciousness exists in the world and experience of things in the world exist in the consciousness”. Perceptive manipulation devices can alter the consciousness. A recent experiment by Ehrsson in 2007 was done by mounting a video camera on the head to allow users to perceive a variety of out-of-body experience.

Galvano-vestibular stimulation system by Fitzpatrick investigates a systems that can manipulate the sense of balance, reflexes, and proprioception. Startling sensations such as sirens, bright flashes, or even vibrations to the sense of touch can unconsciously divert our attention. George Stetten devised the concept of FingerSight in 1999. It extends human sight to fingers by detecting reflected laser light to vibrations felt on the surface of finger, allowing the device to mimic the phenomena of synethesia.

The Approach

Professionally I have resided in the world of motion based narratives in conjunction to this I have also been attracted to breaking and making things. These have informed many of my projects during my two years of masters studies, 11th Finger (extension of the body), Velit (object animism and mythology), and Healthbrush (experience design into our everyday lives). Through experimentation from making, I plan to further develop devices and interfaces, even small one function objects around perception manipulation and physical animation. I will collaborate with other researchers as well to develop hybrid of thesis concepts and tackle a broader spectrum.

Investigation Questions:

• if visual effects in film have the capability to convincingly deceive us, and draw us into a scripted narrative, what would be the implications of the application of embedded visual effects in physical world?

• would they become entertainment, or could they have a deeper effect in subverting our expectations?

• what would be the implications if our objects plays with inversions of our expectations-on our sensory perception, relationship to objects, to entertainment?

• if the frequency of light waves and audio waves can be manipulated enough to cross over in our neurological responses, how can this phenomenon be synthesize towards a comprehensible narrative of personal interpretations?

• when the physical interface takes on a greater responses to touch and controls with hands and feet over digital responses that stimulates human perception, what possibilities can arise from memory, recovery, and discovery?

 

Abstract V.03

Sight, Touch, Mind Bending Apparatus- Thesis abstract by Link Huang

There’s the saying, “eyes are the windows to your soul”. This window is a two way access for massive amounts of quick interactions between the viewed and the viewer. Although to completely rely on sight can often mislead and deliver false information to the viewer, tricking the mind to think one thing is bigger than another, one color looks different than the compared just because its environments are different, or even unexpected reveal of objects. This occurs in real life but even more so on film and Television. My inspirations are driven by motivations from techniques used in visual effects and optical illusions in conjunction with physical mechanics. Techniques such as mechanical transformations, green screen composite, particle animations, sense of 3 dimensional space, or even holographic interfaces. Some methods are by embedding artificial sensors (such as cameras, microphones, temperature, proximity, and touch sensors) in tangible forms to allow the separation of perception and body.

These attributes are embedded in devices to manipulate perception and conscious to allow variations of experiences by individuals. As Maurice Merleau-Ponty observed that “consciousness exists in the world and experience of things in the word exist in consciousness”. Perceptive manipulation devices can alter the consciousness. A recent experiment by Ehrsson in 2007 was done by mounting a video camera on the head to allow users to perceive a variety of out-of-body experience.

Galvano-vestibular stimulation system by Fitzpatrick investigates a systems that can manipulate the sense of balance, reflexes, and proprioception. Startling sensations such as sirens, bright flashes, or even vibrations to the sense of touch can unconsciously divert our attention. George Stetten devised the concept of FingerSight in 1999. It extends human sight to fingers by detecting reflected laser light to vibrations felt on the surface of finger, allowing the device to mimic the phenomena of synethesia.

My professional background comes from storyboarding to animation in the broadcast motion graphics and film industry, it is in conjunction with my personal hobby of various material craft and mechanical works has led me towards my obsession. Example projects I worked on include the 11th Finger, Velit, and Healthbrush. 11th Finger is a hat-like device worn on the head with a finger on the top. Driven by two servo motors and read-outs from a infrared camera that senses the direction of the left eye worn by the user and points the finger in the same direction. Velit was a project to investigate object animism and this autonomous wishing machine takes on a life of it’s own. This creature reacts left and right to the movement of the person standing in front and opens its mouth at a intimate distance when the person’s face comes close, revealing a dandelion looking wishing star created with several tiny LEDs and an audio sensor. The user can also interacts with it by feeding it with some wishes and a breath of air, blowing into the star. Feeding Velit with a wish excites the creature and changes it’s LED tail pattern. When there is no interaction, this creature doze off and falls asleep. The third project is Healthbrush. This is a concept demo of a toothbrush that will analyze the user’s blood from the average daily routine and give the user’s health statistics, assessments, and suggestions.

The thesis investigation I am proposing plays with the language of visual effects as they are applied in an immersive film experience (ex:multiplicity, duplication) into tangible real world things. The goal is to experiment with the possibility to manipulate perception, extending the visual experience of special effects (with theories like gestalt and cognitive psychology) to other senses and in effect subverting expectations of “familiar interactions”, reintroducing the possibility for discovery, and stimulating imagination.

Investigation Questions:

• if visual effects in film have the capability to convincingly deceive us, and draw us into a scripted narrative, what would be the implications of the application of embedded visual effects in physical world?

• would they become entertainment, or could they have a deeper effect in subverting our expectations?

• what would be the implications if our objects subverting our expectations-on our sensory perception, relationship to objects, to entertainment?

• if the frequency of light waves and audio waves can be manipulated enough to cross over in our neurological responses, how can this phenomenon be synthesize towards a comprehensible narrative of personal interpretations?

• when the physical interface takes on a greater responses to touch and controls with hands and feet over digital responses that stimulates human perception, what possibilities can arise from memory, recovery, and discovery?

 

Abstract V.02

Thesis Workshop 09-17-2011: Thesis Intent Wk02

by Link Huang

Tangible Visual Effects

Interests for Investigation

There’s the saying, “eyes are the windows to your soul”. This window is a two way access for massive amounts of quick interactions between the viewed and the viewer. Although to completely relying on sight can often mislead and deliver false information to the viewer, tricking the mind to think one thing is bigger than another, or one color looks different than the compared just because its environments are different or even unexpected reveal of objects. This occurs in real life but even more so on film. Filmmakers often use illusional techniques to complete the visual compositions displayed for storytelling. These techniques such as mechanical transformations, green screen composited scenes, particle animations, sense of 3 dimensional space within, or even holographic interfaces. It is often technologies science fiction writers talk about in their stories, given them new affordances and functions in their fictional portray of the future. When the book is transformed into a feature film, designers are often required to think about not only the aesthetics but also the way they appear. These movement must feel natural and seamless in the way it appears and disappear. In many ways visual effect animations are illusions created to enhance the story, giving credibility for the the audience to believe everything they are watching actually exist.

My interest began collectively towards the investigation of “Hollywood and Science Fiction display of technologies in comparison to a perspective shift of the crude reality when various technologies are implemented in the real world“. In this initial stage, I had the notion of investigating glitches and its imperfections as part of the technology flaws through exploring it’s possibilities. Then as I develop my projects during the courses over the summer, I began to realize my interests lies within the visual effects aspect of these fictional technologies in the stories. My goal is to look into ways to bring these visual effects to the tangible world. I would like to creating inventive and innovative projects while staying away from a carbon copy reflection of a existing visual effects used in film.

 

Expertise and Obsession

My professional background derives from a handful of years doing motion graphics through storyboarding to animation. It is also a combination with some of my hobbies building RC vehicles, craft works exploring materials. Example projects I worked on that led me to this obsession, a few examples include the 11th Finger, Velit, and Healthbrush. 11th Finger is a hat-like device worn on the head with a finger on the top. Driven by two servo motors and read-outs from a infrared camera that senses the direction of the left eye worn by the user and points the finger in the same direction. Velit was a project to investigate object animism and this autonomous wishing machine takes on a life of it’s own. This creature reacts left and right to the movement of the person standing in front and opens its mouth at a intimate distance when the person’s face comes close, revealing a dandelion looking wishing star created with several tiny LEDs and an audio sensor. The user can also interacts with it by feeding it with some wishes and a breath of air, blowing into the star. Feeding Velit with a wish excites the creature and changes it’s LED tail pattern. When there is no interaction, this creature doze off and falls asleep. The third project is Healthbrush. This is a concept demo of a toothbrush that will analyze the user’s blood from the average daily routine and give the user’s health statistics, assessments, and suggestions. Interaction with a Function while using visual effects techniques in the tangible world became a large part of my interest to explore infuse with my expertise.

Questions

I intend to investigate questions like, “Can animistic characteristics of these devices be utilized in terms of a narrative? Can we find out what sorts of “power” fit for these narratives?”,”What would be some amazingly powerful information, if the user could only get it somehow?”, “Can the device be some kind of mounted display, something your essentially wear?”, ” “What kind of experience does the user go through while using these tangible interactions?”, “How can the sense of discovery play into the experience?”, “How can system, network, or social media implications tie in with the design?”.

 

Abstract V.01

Link Huang

Thesis Workshop 09-12-2011: Thesis Intent Wk01

 

During my thesis gateway, my interest began collectively towards the investigation of “Hollywood and Science Fiction display of technologies in comparison to a perspective shift of the crude reality when various technologies are implemented in the real world“. In this initial stage, I had the notion of investigating glitches and its imperfections as part of the technology flaws through exploring it’s possibilities. Then as I develop my projects during the courses I was taking over the summer, I began to realize my interests lies within the visual effects aspect of these fictional technologies in the stories. Looking into ways to bring these visual effects to the tangible world. Creating inventive and innovative projects while staying away from a carbon copy reflection of a existing visual effects used in film.

These include asking questions like, “what type of hardware support and leverage would it require for these servo or motors to move and create interactive interfaces?”, “How can I make these tangible interactions create personalized experience?”, “How does these interactions challenge the existing ways of using certain tools or objects for various functions?”, “What are the target audience demographics?”, “Why would anyone want this?”, “Are there possible ways to make these affordances more sustainable to the environment?”, “What kind of experience does the user go through while using these tangible interactions?”, “How can the sense of discovery play into the experience?”, “How can system, network, or social media implications tie in with the design?”.

My expertise in my professional background derives from a handful of years doing motion graphics through storyboarding to animation. It is also a combination with some of my hobbies building RC cars and wrenching on real cars. I would very much like to combine all my skills together including graphic design, flash driven digital or physical interfaces, digital electronics, gears and motors, even craft works using plastic, metal, and wood to exploring possibilities to create the most interesting interactions.