Empirical Evaluation of Interactive Visualizations for Preferential Choice: Difference between revisions

From InfoVis:Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
Line 61: Line 61:
In Part B, the authors followed a more qualitative approach by '''observing subjects using the tool in a real decision-making context'''. In this second part of the study, they attempted to measure the users’ insight in the decision problem.  
In Part B, the authors followed a more qualitative approach by '''observing subjects using the tool in a real decision-making context'''. In this second part of the study, they attempted to measure the users’ insight in the decision problem.  
Once the subjects had completed Part B, they filled out a questionnaire regarding their experience with VC+ in the decision-making process. In our exploratory study we measure the amount of insight each sub ject gains from using VC+ for a particular decision-making scenario.  
Once the subjects had completed Part B, they filled out a questionnaire regarding their experience with VC+ in the decision-making process. In our exploratory study we measure the amount of insight each sub ject gains from using VC+ for a particular decision-making scenario.  
In the exploratory study the amount of insight each sub ject gains from using VC+ for a particular decision-making scenario is measured.
In the exploratory study the amount of insight each sub ject gains from using VC+ for a particular decision-making scenario is measured.  
'''Insights''' are characterized by the following:


'''Fact''': The actual finding about the data (e.g. “Samsung [cell phones] are the smallest”)  
'''Fact''': The actual finding about the data (e.g. “Samsung [cell phones] are the smallest”)  
Line 74: Line 75:


– Realization of personal preferences: users often stated that they made a realization about their preferences e.g. “it makes sense, because I really like hiking and nature”, “brand should be more important [to me]”  
– Realization of personal preferences: users often stated that they made a realization about their preferences e.g. “it makes sense, because I really like hiking and nature”, “brand should be more important [to me]”  
The '''Datasets''' the subjects were allowed to choose were: '''house rental''', '''cell phone''' and '''tourism'''.


== Figures ==
== Figures ==

Revision as of 18:33, 27 May 2009

An Empirical Evaluation of Interactive Visualizations for Preferential Choice by Jeanette Bautista and Giuseppe Carenini

UNDER CONSTRUCTION

Short Description

The authors of this paper tried to not only show the usefulness of Value Charts to support preferential choice, which is finding the best option out of a set of alternatives. Furthermore, they compared two types of Value Charts: a horizontal version against a vertical version. The outcome of this extensive user study was that Value Charts in general and in particular Vertical Value Charts (abbreviated VC+V) seemed to be very effective in supporting decision making.

Process of decision making

The process of effective preferential choice can be divided into 3 steps according to prescriptive decision theory.


Phase 1 / Model construction phase: the decision maker (abbr. DM) finds objectives, which are important to him/her. The degree of importance is also chosen.

Phase 2 / Inspection phase: DM analyzes his/her preference model as applied to a set of alternatives.

Phase 3 / Sensitivity analysis: DM is able to answer "what if" questions - such as "if we make a slight change in one or more aspects of the model, does it effect the optimal decision?"

In the development of interactive tools for preferential choice, we argue that full support for - and fluid interaction between - all three phases are essential in making good decisions.
[Bautista and Carenini, 2008]



ValueCharts+

ValueChart+is a set of interactive visualization techniques for preferential choice and is an improvement of ValueCharts. It supports the DM in the 3 phases described above. In an Additive Multiattribute Value Function the DM’s objectives are hierarchically organized. In VC+ this hierarchy is displayed as an exploded stacked-bar. The ValueChart+ follows the information-seeking mantra by Ben Shneiderman: overview first, zoom and filter, then details on demand.


Evaluation Part A: Controlled Study

In Part A, the authors took a quantitative approach by performing a controlled usability study to see how users performed the primitive tasks of the PVIT (Preferential Choice Visualization Integrated Task Model). Part A starts in the sensitivity analysis phase. The task is the following: 5 questions (based on the PVIT model) should be answered:

What are the top 3 alternatives according to total value?

For a specified alternative, which ob jective contributes to its total value the most?

What is the domain value of objective x for alternative y?

What is the best alternative when considering only objective x?

What is the best outcome for a objective x?


Mapped to the house domain, for example, we get the following tasks:

List the 3 highest valued houses.

For HouseX, which is its strongest factor according to your preferences?

How many bathrooms are there in House1?

Which is the least expensive house?

What is the best bus-distance?


The subjects had to test the ValueCharts+ in the sensitivity analysis and inspection phase — no questions directed at the experimenter were allowed. Test domain was a set of hotels in the Vancouver area. 20 subjects were tested, 10 tested the VC+H and the remaining 10 the VC+V. Each subject performed each task, writing down the answer to applicable tasks that asked a question about the data. This procedure was re-iterated 5 times. The authors looked closely at their results to find an indication of whether one version of VC+ was a better fit than the other during the decision making process.

Finally, in the second step of the analysis the authors determined, for each task, what interface the subjects performed better. VC+V performed better on all five inspection tasks and also performed better on three out of the four sensitivity analysis tasks.


Evaluation Part B: User Study

In Part B, the authors followed a more qualitative approach by observing subjects using the tool in a real decision-making context. In this second part of the study, they attempted to measure the users’ insight in the decision problem. Once the subjects had completed Part B, they filled out a questionnaire regarding their experience with VC+ in the decision-making process. In our exploratory study we measure the amount of insight each sub ject gains from using VC+ for a particular decision-making scenario. In the exploratory study the amount of insight each sub ject gains from using VC+ for a particular decision-making scenario is measured. Insights are characterized by the following:

Fact: The actual finding about the data (e.g. “Samsung [cell phones] are the smallest”)

Value: How to measure each insight? The authors determined and coded the value of each insight from 1 - 3, whereas simple observations of domain value and top ranking (e.g. “cheapest place is in East Van”) are fairly trivial, and more global observations regarding relationships and comparison (e.g. “more expensive phones have all the features”) are more valuable.

Category: Insights were grouped into several categories:

– Simple fact: an alternative rank or identification of domain value e.g. “This phone is fairly light”, “This phone is only [ranked] fourth for battery”

– Sensitivity: how a change affects the results e.g. “This house again!”, “Now this phone is third”

– Realization of personal preferences: users often stated that they made a realization about their preferences e.g. “it makes sense, because I really like hiking and nature”, “brand should be more important [to me]”


The Datasets the subjects were allowed to choose were: house rental, cell phone and tourism.



Figures

Shown above is the horizontal version of a ValueChart+ presented by Bautista et Carenini

Shown above is the vertical version of a ValueChart+ presented by Bautista et Carenini.

The vertical height of each row indicates the relative weight assigned to each objective (e.g., size is much less important than internet-access). Each column represents an alternative, thus each cell portrays an objective corresponding to an alternative (bottom-right quadrant). The amount of filled color relative to cell size depicts the alternative’s preference.


Important Citation(s)

An effective visualization will aid the Decision Maker to see things that would otherwise go unnoticed, as well as enable her to view information about her preferences in a new light.
[Bautista and Carenini, 2008]



Suitable for which data types

Evaluation(s)

References

An empirical evaluation of interactive visualizations for preferential choice