Tutorials

T1Tutorial 1
Evaluating Intelligent User Interfaces with User Experiments

Overview

Are you interested in evaluating your Intelligent User Interface with a user experiment?
Have you conducted user experiments before, but want to learn how to use state-of-the-art techniques like factor analysis and structural equation modeling?

This tutorial will teach best practices and state-of-the-art methods for designing, conducting, and evaluating user experiments in the field of Intelligent User Interfaces.

The tutorial will have something to offer for (and be accessible to) both beginners (e.g., students starting out in the field, systems researchers interested in user evaluation) and experts (seasoned HCI researchers).

Topics:

• Developing a research model: This part discusses how to turn your research questions into a body of interrelated hypotheses. It also teaches how to select the right variables to measure and system aspects to manipulate in order to test these hypotheses.
• Setting up an experiment: This part discusses how to select participants, how to select experimental conditions to test, and how to assign participants to conditions.
• Determining the sample size*: This part teaches how to determine the optimal sample size for your study using power analysis, and covers sequential analyses as an economical way to run large studies.
• Measuring perceptions and experiences: This part teaches how to measure users’ perceptions and experiences using multi-item questionnaires.
• Evaluating measurement scales*: This part teaches how to evaluate the quality of the developed questionnaires using factor analysis.
• Evaluating entire research models*: This part covers structural equation modeling, a state-of-he-art method for evaluating the results of an experiment.

The tutorial will mainly consist of lecture-style demonstrations, but there will be ample time for questions regarding your specific research. No prior knowledge of statistical evaluations or specific software is required, but participants with some knowledge of stats who have RStudio and G*Power installed on their laptops can follow along with examples in the starred (*) topics.

About the instructor

Bart Knijnenburg is an assistant professor in Human-Centered Computing at Clemson University. He is the author of one of the leading frameworks for user-centric evaluation of recommender systems, as well as a book chapter on user experiments for the upcoming 2nd edition of the Recommender Systems Handbook. He is an expert reviewer on the topic of user-research statistics for several conferences and journals, and a consultant on several academic and industry projects. He has also attempted to explain the tutorial topic to a general (non-academic) audience in a TEDx talk titled “Technopsychometrics”.

---

Chairs