Science of Interaction

Starting point: Illuminating the Path

Executive Summary

p. 9 (Executive Summary) in [Thomas and Cook, 2005]

Recommendation

Develop a new science of interactions that supports the analytical reason- ing process. This interaction science must provide a taxonomy of interaction techniques ranging from the low-level interactions to more complex inter- action techniques and must address the challenge to scale across different types of display environments and tasks.

Interaction is the fuel for analytic discourse. Although the fundamental principles of interaction have been around for more than a decade, they do not address the needs for higher-order interaction techniques, such as task-directed or hypothesis-guided discourse, to support the analysis process. A new scientific theory and practice are critical to address the complexity of homeland security needs for analysis, prevention, and response. These interaction techniques must adapt to the particular dimensions of the analytical situation, ranging from longer-term analytical assess- ments to urgent and highly stressful emergency response support tasks. These interactions must be adaptable for use in platforms ranging from the large displays in emergency management control rooms to field-deployable handheld devices in the hands of first responders. This is a high priority for initial investments.

[Thomas and Cook, 2005]

A Science of Interaction

p. 73-81 (Chapter 3) in [Thomas and Cook, 2005]

State of the Art

Levels of interaction: human time constants (p.74)

• ~100 milliseconds
  • perceptual fusion time constant
  • rate necessary to produce the perception of a smooth animation
  • perception of an immediate response
• ~1 second
  • unprepared response time
  • rate of response to simple user interactions
  • it is important to provide some kind of feedback in the 1-second timeframe
  • a 1-second transition animation can reduce user task performance time
• ~10 seconds
  • unit task time
  • time within which users expect more comples user-initiated activities to
• ~100 seconds (minutes to hours)
  • rational band

Uses of interaction (p.75)

• primary uses of interaction
  • modify data transformation (filtering)
  • modify visual mappings
  • modify view transformation (i.e., navigation)
  • NEW: human-information discourse (higher-level user dialogue with the information)
• Interactions for modifying data transformation (filtering)
  • direct manipulation
  • dynamic queries
  • brushing
  • details-on-demand
• Interactions for modifying visual mappings
  • dataflow systems
  • Pivot Tables
• Interactions for modyfing view transformation (navigation)
  • direct selection
  • selecting and highlighting
  • panning
  • zooming
  • overview and detail
• Interaction for human-information discourse
  • comparing
  • categorizing
  • extracting
  • recombining data
  • creating and testing hypotheses
  • annotating data

Nature of interactions

• 3D manipulation and navigation techniques tend to be harder to use and harder to learn (p.75)
• multimodal interfaces
• sonification

Technology Needs

Although a lot of isolated design work has been done in specific aspects of interaction science, little systematic examination of the design space has been done. As a field, we are in a transition phase in which researchers are beginning the foundational work to understand that design space. Creating a science of interaction is critical because the large-scale nature of the analytic problem and the compressed timeframe for analysis require that we identify and develop the correct interaction techniques for any given human timeframe, interaction use, or interaction environment.

[Thomas and Cook, 2005]

Basic interaction techniques

To achieve successful adoption, visual analytics software must support both basic interactions and highly sophisticated interactions that support the analytic reasoning process. Before these more sophisticated interactions can be addressed systematically, work is needed to create a scientific understanding about the basic interactions that are used to support simpler operations. This understanding will form the foundation for research into more sophisticated interactions.

Recommendation 3.3

Create a new science of interaction to support visual analytics.

The grand challenge of interaction is to develop a taxonomy to describe the design space of interaction techniques that supports the science of analytical reason- ing. We must characterize this design space and identify under-explored areas that are relevant to visual analytics. Then, R&D should be focused on expanding the repertoire of interaction techniques that can fill those gaps in the design space.

[Thomas and Cook, 2005]

Interaction techniques for human-information discourse

Existing work on interaction techniques for human-computer interaction and information visualization has focused on cognitive time bands, interaction for data manipulation, visual mapping manipulation, and navigation. The discussion on analytic discourse and sense-making in Chapter 2 makes it clear the higher-level dialogue between analyst and information, or human-information discourse, is of vital importance. This discourse involves the rational time band and higher-level uses of interaction, but neither has been sufficiently explored.

Recommendation 3.4

Expand the science of interaction to support the human-information discourse needed for analytical reasoning. In particular, identify and develop interaction techniques that support higher-level reasoning and that address the rational human timeframe.

Human beings are very skilled at analyzing complex situations using a combination of their available information and their combined knowledge and experience. However, there are inherent human tendencies that analysts must recognize and overcome. Interaction techniques must be developed that support an analytic discourse and help compensate for human limitations, including:

• Information overload in complex situations. Techniques are needed to help analysts simplify their cognitive load without compromising the analyst’s effectiveness and to help compensate for faulty memory.
• Overcoming biases. Biases affect the way data are interpreted. Biases about the reliability of different sources may lead people to discount information from sources that aren’t considered reliable. People often see what they expect to see and tend to ignore evidence that is contradictory to a preferred theory. If they form a preliminary judgment too early in the analytical process, they may hold firm to it long after the evidence invalidates it [Heuer, 1999].
• Satisficing. People settle for a “good enough” answer, sometimes stopping their analytical process before they identify critical information that would lead them to a different conclusion [Heuer, 1999].

New interaction techniques are needed to support the user in evaluating evidence, challenging assumptions, and finding alternatives. Analytical environments should support the user in identifying and understanding all relevant information to reach a solid conclusion rapidly. The tools we create need to establish a correct balance between structure and intuition.

[Thomas and Cook, 2005]

Leveraging New Media to Support Interaction

• Harnessing new display technologies
• Scaling to multiple devices and device configurations
• Multimodal interaction (p.80)

References

• [Green et al., 2008] Green T., Ribarsky W., Fisher B.: Visual Analytics for Complex Concepts Using a Human Cognition Model. In Proc. IEEE VAST, pp. 91–98, 2008.
• [Lam, 2008] Lam H.: A Framework of Interaction Costs in Information Visualization. IEEE TVCG 14(6):1149–1156, 2008.
• [Liu et al., 2008] Liu Z., Nersessian N. J., Stasko J. T.: Distributed Cognition as a Theoretical Framework for Information Visualization. IEEE TVCG 14(6):1173–1180, 2008.
• [Liu & Stasko, 2010] Zhicheng Liu; Stasko, J.T.; , Mental Models, Visual Reasoning and Interaction in Information Visualization: A Top-down Perspective, IEEE TVCG, 16(6):999-1008, 2010.
• [Pike et al., 2009] Pike W. A., Stasko J. T., Chang R., O’Connell T. A.: The Science of Interaction. Information Visualization 8(4):263–274, 2009.
• [Thomas and Cook, 2005] J.J. Thomas and K.A. Cook, eds., Illuminating the Path: The Research and Development Agenda for Visual Analytics, IEEE CS Press, 2005.
• [Yi et al., 2007] Yi J. S., Kang Y. A., Stasko J., Jacko J.: Toward a Deeper Understanding of the Role of Interaction in Information Visualization. IEEE TVCG 13(6):1224–1231, 2007.
• [Sedig & Parsons, 2013] Sedig K., Parsons P. Interaction Design for Complex Cognitive Activities with Visual Representations: A Pattern-Based Approach. AIS Transactions on Human-Computer Interaction 5(2):84-133, 2013.