IMMERSIVE

ENVIRONMENT

ADVANCED

AR+VR

TECHNOLOGY

IBM WATSON

INTERACTIVE

EXPERIENCE

OVERVIEW

The Cognitive and Immersive Systems Lab (CISL) is a

multi-year collaboration program between Rensselaer Polytechnic Institute and IBM Research to lead the frontier of research and development in immersive cognitive systems. CISL promotes a culture of multidisciplinary research across science, engineering, art, communications, architecture, and business. Our mission is to create scientific breakthroughs and technical innovations that augment group intelligence in real world scenarios.

Project Details

Date : May - Dec 2017

Role : Data Visualization & UX Designer

Project : IBM Watson Immersive Room

Team : Cognitive & Immersive Systems Lab

Design Work

Data Visualization Research

User Interface & Experience Design

Narrative Generation

Use Case Studies

PROJECT VISION

Creating a human-scale immersive space where humans can begin to interact naturally with computers.

IMAGINE THE FUTURE

Experiencing Human-Scale Immersive Virtual Environments

CLASSROOMS &

MEETING ROOMS

automatically recognize attendee's

face and takes attendance.

COMMUNICATION

between individuals andgroup conversations are saved automatically in text bubbles.

IDEAS & PROTOTYPE PRESENTATIONS

are more scale-able & immersive to better express the applications in real-life scenarios.

VIRTUAL TRAVELLING

anywhere around the world in real-time. And experience the area in an immersive way.

IMMERSIVE & INTERACTIVE

USER EXPERIENCE

with endless possibilities.

INITIAL RESEARCH

IMMERSIVE SPACE FOR HUMAN-COMPUTER INTERACTION

The Interaction Loop for

Human-Scale Group-Computers Interaction

When considering human-computer interaction, whether in an office with laptops, in a car with a mobile phone, or on a couch with a tablet, an individual user is most often the sole focus of that interaction. However, many situations involve groups of people working together, especially in critical situations involving complex problems. These people often have diverse backgrounds and domain expertise. We can imagine scenarios in corporate board rooms where groups of executives are tasked with allocating capital, or in health centers where teams of doctors determine patient treatment plans. In most collaborative situations, people communicate using their personal devices and they

may share information on a large screen or multiple screens to disseminate information to the group. However, the ability of the group to directly interact with one another’s information is limited by restricted access to personal devices. When additional information is required and not available, that information must be researched and gathered, analysis must be performed and reports must be created for the next meeting.

Given these limited affordance, decisions in these situations are too often made without full knowledge of or insight to pertinent data. Such issues must be resolved to significantly improve the quality and productivity of group decision making. Even with increasingly intelligent computers and data-analytical algorithms, there is inherent difficulty in linking them directly into the group’s wisdom in the context of discussions or decision making in real-time.

We aim to create a Situations Room that allows increasingly powerful computers to interact naturally and directly with group discussions and augment group intelligence in real-time:

  • Summarize meetings automatically

  • Bring real-time analysis of context information to the discussion when person-directed decision making is required

  • Facilitate discussion when people have biases or misunderstand one another
    Interject into discussions as appropriate;

  • Generate new hypothesis

  • Use analogies to create links between seemingly uncorrelated data-sets.

THE SITUATION ROOM

​Imagine being in a room to ask questions of one of the world's most powerful computers. An artificial intelligence containing more information than the largest library, it can recognize you, hear you, see what you are pointing at, and even notice if you might be perplexed or inattentive. It knows all of your earlier work and might even anticipate your questions.

Such a room is now being created at Rensselaer Polytechnic Institute under a partnership with technology giant IBM and its supercomputer Watson, which first gained worldwide attention in 2011 when it beat humans in the TV show Jeopardy.

The Situations Room serves as a bridge between groups of individuals working on a task, and the supercomputing services (our Society of Cognitive Agents) those individuals use to make critical decisions. The Situations Room integrates immersive environments—to allow human-scale interaction—and cognitive computing—a digital analysis of the context of group activities, or “mind-of-room.” As an initial step towards understand group dynamics, the interaction between humans and the Situations Room is approached as a long-term multi-round conversation loop with multiple interaction modalities.

RESEARCH QUESTION

How can humans interact with computers using natural gestures and speech?

Recognize interpersonal communication, identify conversations between humans and computers, and organize this information as input. The idea is to utilize advanced cameras, microphones, and sensors as the eyes and ears the Situations Room. Those elements would capture humans locations, movements, head position, eye focus, and body movements, speech, physical objects, digital objects, and voice in the room. At this level, cognitive computing technologies are applied to room input channels to recognize human speech, gestures, attention, emotion, grouping, and interpersonal interactions.

CONCEPT SKETCH

IDEA BRAINSTORM

HAND GESTURE COMMAND SKETCHES

USING PRE-EXISTING KNOWLEDGE IN HAND SIGNS FOR INTUITIVE CONTROL

Within the immersive space, sensors detect human activity, such as a change in the position of an occupant of the room, speech, gesture, and head movement. Absent the CISL architecture, each of the cognitive technologies acts in solitude, responding to a specific activity detected by a single type of sensor and provided to the computer for interpretation. A sensor provides an input, and the computer provides an output. The interaction between human and machine is based on a single action with a finite duration.

USER FLOW

Cognitive Walk-through

USER INTERFACE DESIGN

IMMERSIVE SCREEN UI MOCKUP

Present and communicate the analysis and cognitive viewpoints to discussion participants through creative multimodal narratives. Humans must understand what computational processes are gathered by the computers to analyze context information. What is in the data and why did the computers derive its conclusion(s)?​

The most memorable way for humans to explain such insight to others is through storytelling, often in a multimodal fashion, with creative analogies, summaries, and references. Research in creative interactive multimodal storytelling will help create multimodal narratives to translate insights and discovery from large amounts of data into advanced visualization and sonification in an immersive environment and allow interaction by humans to further explore the information and thus enable human-in-the-loop analysis. 

CULTURAL EDUCATION AGENT

Cyber-Enabled Learning

  • A Virtual Space for Children to Meet and Practice Chinese

  • Interactive Storytelling for Foreign Language Learning

  • Argument Interactive Narratives with Cognitive Robots

The game features a scene in which players take on the role of a restaurant customer who will converse with a waiter-bot and place orders in a restaurant. The art style will evoke the experience of casual dining at a warmly-lit Chinese restaurant.​ Environment art will be displayed on three large screens which are reconfigurable. Food Presentation screen is shown on the table display.

USABILITY TESTING

HEURISTIC EVALUATION

INTERFACE DISPLAY TESTING

Syncing UI Designs on a curved, large-scale immersive screens.

INTERFACE INTERACTION TESTING

Identifying any interaction defects in the interface display in the immersive lab.

USER TESTING

Receiving real-time feedback from the audience.

Dialogue Flow Design using Draw.io Diagrams

Narrative Generation

THE FUTURE

HUMAN-COMPUTER

INTERACTION

CISL PROJECT USE-CASE STUDY

The Group Meeting Re-imagined: Designing Intelligent Rooms

A group of four people walk into a room and the leader says, “Watson, bring me the last working session.” The computer recognizes and greets the group, then retrieves the materials used in the last meeting and displays them on three large screens. Settling down to work, the leader approaches one screen, and swipes his hands apart to zoom into the information on display. The participants interact with the room through computers that can understand their speech, and sensors that detect their position, record their roles and observe their attention. When the topic of discussion shifts from one screen to another, but one participant remains focused on the previous point, the computer asks a question: “What are you thinking?”

Facial Recognition

Auto Check-In

Facial Recognition and auto check-in for group meeting and many other purposes.

Human-Machine

Group Interaction

Immersive Smart Room that interacts naturally with the users. Users mix seamlessly with tech.

Natural Control

Natural Interaction with the smart room using voice command, hand gestures, and facial expressions.

Automatically

Saved Sessions

Everything that was discussed in the meeting will be auto saved in text form to access anytime.

The room will be able to address the following questions within the discussion:

  • What is the focus of the discussion? What are its topics and knowledge domains?

  • What types of discussion activities are taking place? Validation, exploration, decisions, or task prioritizations?

  • What are the interpersonal dynamics between discussion participants? What are their relationships and trust levels?

IBM - COGNITIVE AND IMMERSIVE SYSTEMS LAB

FUTURE DEVELOPMENTS

LIMITLESS POSSIBILITIES

Immersive Human-Scale VR Environment

LIKE MY WORK?

Let's talk about your project and how I can help you!

WILLIAM S. KIM

UI/UX+WEB DESIGN

  • will-kim social icons - email
  • will-kim social icons - linkedin
  • will-kim social icons - instagram
  • will-kim social icons - behance

© WILLIAM S. KIM 2020