IMMERSIVE
ENVIRONMENT
ADVANCED
AR+VR
TECHNOLOGY
IBM WATSON
INTERACTIVE
EXPERIENCE
OVERVIEW
The Cognitive and Immersive Systems Lab (CISL) is a
multi-year collaboration program between Rensselaer Polytechnic Institute and IBM Research to lead the frontier of research and development in immersive cognitive systems. CISL promotes a culture of multidisciplinary research across science, engineering, art, communications, architecture, and business. Our mission is to create scientific breakthroughs and technical innovations that augment group intelligence in real world scenarios.
Project Details
Date : May - Dec 2017
Role : Data Visualization & UX Designer
Project : IBM Watson Immersive Room
Team : Cognitive & Immersive Systems Lab
Design Work
Data Visualization Research
User Interface & Experience Design
Narrative Generation
Use Case Studies
PROJECT VISION
Creating a human-scale immersive space where humans can begin to interact naturally with computers.
IMAGINE THE FUTURE
Experiencing Human-Scale Immersive Virtual Environments
CLASSROOMS &
MEETING ROOMS
automatically recognize attendee's
face and takes attendance.
COMMUNICATION
between individuals andgroup conversations are saved automatically in text bubbles.
IDEAS & PROTOTYPE PRESENTATIONS
are more scale-able & immersive to better express the applications in real-life scenarios.
VIRTUAL TRAVELLING
anywhere around the world in real-time. And experience the area in an immersive way.
IMMERSIVE & INTERACTIVE
USER EXPERIENCE
with endless possibilities.
INITIAL RESEARCH
IMMERSIVE SPACE FOR HUMAN-COMPUTER INTERACTION
The Interaction Loop for
Human-Scale Group-Computers Interaction
When considering human-computer interaction, whether in an office with laptops, in a car with a mobile phone, or on a couch with a tablet, an individual user is most often the sole focus of that interaction. However, many situations involve groups of people working together, especially in critical situations involving complex problems. These people often have diverse backgrounds and domain expertise. We can imagine scenarios in corporate board rooms where groups of executives are tasked with allocating capital, or in health centers where teams of doctors determine patient treatment plans. In most collaborative situations, people communicate using their personal devices and they
may share information on a large screen or multiple screens to disseminate information to the group. However, the ability of the group to directly interact with one another’s information is limited by restricted access to personal devices. When additional information is required and not available, that information must be researched and gathered, analysis must be performed and reports must be created for the next meeting.
Given these limited affordance, decisions in these situations are too often made without full knowledge of or insight to pertinent data. Such issues must be resolved to significantly improve the quality and productivity of group decision making. Even with increasingly intelligent computers and data-analytical algorithms, there is inherent difficulty in linking them directly into the group’s wisdom in the context of discussions or decision making in real-time.
We aim to create a Situations Room that allows increasingly powerful computers to interact naturally and directly with group discussions and augment group intelligence in real-time:
-
Summarize meetings automatically
-
Bring real-time analysis of context information to the discussion when person-directed decision making is required
-
Facilitate discussion when people have biases or misunderstand one another
Interject into discussions as appropriate; -
Generate new hypothesis
-
Use analogies to create links between seemingly uncorrelated data-sets.
THE SITUATION ROOM
Imagine being in a room to ask questions of one of the world's most powerful computers. An artificial intelligence containing more information than the largest library, it can recognize you, hear you, see what you are pointing at, and even notice if you might be perplexed or inattentive. It knows all of your earlier work and might even anticipate your questions.
Such a room is now being created at Rensselaer Polytechnic Institute under a partnership with technology giant IBM and its supercomputer Watson, which first gained worldwide attention in 2011 when it beat humans in the TV show Jeopardy.
The Situations Room serves as a bridge between groups of individuals working on a task, and the supercomputing services (our Society of Cognitive Agents) those individuals use to make critical decisions. The Situations Room integrates immersive environments—to allow human-scale interaction—and cognitive computing—a digital analysis of the context of group activities, or “mind-of-room.” As an initial step towards understand group dynamics, the interaction between humans and the Situations Room is approached as a long-term multi-round conversation loop with multiple interaction modalities.
RESEARCH QUESTION
How can humans interact with computers using natural gestures and speech?
Recognize interpersonal communication, identify conversations between humans and computers, and organize this information as input. The idea is to utilize advanced cameras, microphones, and sensors as the eyes and ears the Situations Room. Those elements would capture humans locations, movements, head position, eye focus, and body movements, speech, physical objects, digital objects, and voice in the room. At this level, cognitive computing technologies are applied to room input channels to recognize human speech, gestures, attention, emotion, grouping, and interpersonal interactions.
CONCEPT SKETCH
IDEA BRAINSTORM
HAND GESTURE COMMAND SKETCHES
USING PRE-EXISTING KNOWLEDGE IN HAND SIGNS FOR INTUITIVE CONTROL
Within the immersive space, sensors detect human activity, such as a change in the position of an occupant of the room, speech, gesture, and head movement. Absent the CISL architecture, each of the cognitive technologies acts in solitude, responding to a specific activity detected by a single type of sensor and provided to the computer for interpretation. A sensor provides an input, and the computer provides an output. The interaction between human and machine is based on a single action with a finite duration.
USER FLOW
Cognitive Walk-through
USER INTERFACE DESIGN
IMMERSIVE SCREEN UI MOCKUP
Present and communicate the analysis and cognitive viewpoints to discussion participants through creative multimodal narratives. Humans must understand what computational processes are gathered by the computers to analyze context information. What is in the data and why did the computers derive its conclusion(s)?
The most memorable way for humans to explain such insight to others is through storytelling, often in a multimodal fashion, with creative analogies, summaries, and references. Research in creative interactive multimodal storytelling will help create multimodal narratives to translate insights and discovery from large amounts of data into advanced visualization and sonification in an immersive environment and allow interaction by humans to further explore the information and thus enable human-in-the-loop analysis.
CULTURAL EDUCATION AGENT
Cyber-Enabled Learning
-
A Virtual Space for Children to Meet and Practice Chinese
-
Interactive Storytelling for Foreign Language Learning
-
Argument Interactive Narratives with Cognitive Robots
The game features a scene in which players take on the role of a restaurant customer who will converse with a waiter-bot and place orders in a restaurant. The art style will evoke the experience of casual dining at a warmly-lit Chinese restaurant. Environment art will be displayed on three large screens which are reconfigurable. Food Presentation screen is shown on the table display.
USABILITY TESTING
HEURISTIC EVALUATION
INTERFACE DISPLAY TESTING
Syncing UI Designs on a curved, large-scale immersive screens.
INTERFACE INTERACTION TESTING
Identifying any interaction defects in the interface display in the immersive lab.
USER TESTING
Receiving real-time feedback from the audience.
Dialogue Flow Design using Draw.io Diagrams
Narrative Generation
THE FUTURE
HUMAN-COMPUTER
INTERACTION
CISL PROJECT USE-CASE STUDY
The Group Meeting Re-imagined: Designing Intelligent Rooms
A group of four people walk into a room and the leader says, “Watson, bring me the last working session.” The computer recognizes and greets the group, then retrieves the materials used in the last meeting and displays them on three large screens. Settling down to work, the leader approaches one screen, and swipes his hands apart to zoom into the information on display. The participants interact with the room through computers that can understand their speech, and sensors that detect their position, record their roles and observe their attention. When the topic of discussion shifts from one screen to another, but one participant remains focused on the previous point, the computer asks a question: “What are you thinking?”
Facial Recognition
& Auto Check-In
Facial Recognition and auto check-in for group meeting and many other purposes.
Human-Machine
Group Interaction
Immersive Smart Room that interacts naturally with the users. Users mix seamlessly with tech.
Natural Control
Natural Interaction with the smart room using voice command, hand gestures, and facial expressions.
Automatically
Saved Sessions
Everything that was discussed in the meeting will be auto saved in text form to access anytime.
The room will be able to address the following questions within the discussion:
-
What is the focus of the discussion? What are its topics and knowledge domains?
-
What types of discussion activities are taking place? Validation, exploration, decisions, or task prioritizations?
-
What are the interpersonal dynamics between discussion participants? What are their relationships and trust levels?
IBM - COGNITIVE AND IMMERSIVE SYSTEMS LAB
FUTURE DEVELOPMENTS
LIMITLESS POSSIBILITIES
Immersive Human-Scale VR Environment
