Evaluating Information Visualization Applications

Author

Riccardo Mazza

Short Description

Evaluation of user-centred tools often makes use of analytic and empirical techniques with the objective to investigate the usability, the functionality, and the effectiveness of an interactive system.

[Dix, Finley, Abowd, Beale, 1998]

There are two different types of "Human Computer Interacion Evaluation Methods":

• Analytic Methods
• Empiric Methods
  • Quantitave Studies
  • Qualitative Studies

CourseVis

[...] a novel approach of using web log data generated by course management systems (CMS) to help instructors become aware of what is happening in distance learning classes. Specifically, techniques from Information Visualization are used to graphically render complex, multidimensional student tracking data collected by CMS. A system, called CourseVis, illustrates the proposed approach.

[Mazza and Dimitrova, 2004]

Figures

The following visualizations are used to represent the collected data from CourseVis:

• Access Plot:
  

The access plot shows the students' accesses to the course. It is an mixture of a scatterplot and a histogram.
The scatterplot shows the data of students and of the course while the histogram shows the number of pages of the course accessed by all students on a special date.

Access Plot

• Discussion Plot:
  

This 3D scatterplot is about the disussions of the course. The figure shows time, topics and students.
The sphere represents a new thread and the number of messages are encoded with the color and size of the sphere.

Discussion Plot

• Behavioural Graph:
  

This graph represents the behaviours of a single student.
It shows the student's access to the content pages, the global access to the course, a progress with the schedule of the course, messages and the submission of quizzes and assignments.

Behavioural Graph

Evaluation Types

• Focus Group

• Controlled Experiment

• Semi-structured Interview

The focus group and the semi-structured interview aimed to assess the usefulness of the graphical representations generated in CourseVis, while the experimental study aimed at verifying the effectiveness and the efficiency of the representations.

[R. Mazza, 2006]

Important Citation

attitudes, feelings and beliefs ... are more likely to be revealed

via the social gathering and the interaction which being in a focus

group entails

[A. Gibbs, 1997]

formulate and answer questions [users] didn’t anticipate having before looking at the visualization

[C. Plaisant, 2004]

References

• CourseVis
• [Mazza and Dimitrova, 2004] Mazza, Ricardo; Dimitrova, Vania. Visualising student tracking data to support instructors in web-based distance education. In Proceedings of the 13th international World Wide Web conference on Alternate track papers & posters, pp. 154-161. 2004.
• [Mazza, 2006] Mazza, Ricardo. Evaluating information visualization applications with focus groups: the CourseVis experience. In Proceedings of the 2006 AVI Workshop on Beyond Time and Errors: Novel Evaluation Methods For information Visualization, pp. 1-6, 2006.