Reducing biases in usibility studies through UX Research and testing

Project Context

A UX study on a major banking mobile app

As part of my Master's thesis in UX, I joined a large-scale applied research project at Tech3Lab, conducted in partnership with a major Canadian financial institution. The project's goal was to improve accessibility and the overall user experience of the bank's mobile application.

‍

For over a year, our team ran an experimental study to understand how age and digital self-efficacy influence different dimensions of user experience when completing core banking tasks.

‍

‍

Team and Collaboration

UX lab and banking partner

Throughout the project, I worked closely with the lab's research directors and assistants. My involvement started remotely during the exploratory phases, sharing and validating research work, then shifted to in-person during lab sessions with research assistants. The role required strong scientific discipline and clear, structured communication with a wide range of stakeholders: researchers, directors, research assistants, and the client's UX team.

‍

‍

Research Question

Rethinking how we segment users in UX research

In UX research, recruiting by age to get varied usage profiles is common practice. But is age actually the best segmentation variable? A review of the literature pointed to other factors that could influence how people experience digital services, including variables tied to trust, perceived competence, and relationship with technology. This led us to investigate the potential role of digital self-efficacy in explaining the differences observed between users.

‍

The research question: to what extent does digital self-efficacy moderate the relationship between age and user experience?

‍

The goal was to give the banking partner two things:

• Scientific evidence demonstrating the impact of this moderating variable
• Concrete recommendations for improving sampling practices in UX research

‍

‍

Process and Methods

100+ controlled lab sessions

We ran a controlled experimental study combining performance measures, psychophysiological data, and psychometric assessments.

‍

Transactional tasks

• Log in to the mobile app
• Add an Interac recipient
• Complete an Interac transfer

‍

Informational tasks

• Find a mortgage rate
• Compare two credit cards
• Locate the local support number

‍

‍

Key Decisions and Trade-offs

Choosing depth over breadth in the measurement approach

One of the core decisions in this project was committing to a multi-measure methodology, combining behavioral performance data, biometric readings, and psychometric scales. This added significant complexity to both data collection and analysis. A simpler approach would have been to rely on self-reported UX scores alone, which is far more common in applied research.

‍

We chose the more rigorous path because the research question required it. If we wanted to make a credible scientific argument about self-efficacy as a moderating variable, the evidence needed to hold up to scrutiny. That meant more sessions, more setup time per participant, and a much heavier analysis phase. The trade-off was real: the project took over a year. But the quality of the findings is what made the recommendations actionable rather than just interesting.

‍

Pushing back on age as the only segmentation criterion

There was also a methodological tension with standard industry practice. Recruiting by age is the default in UX research, partly because it is simple to operationalize. Questioning that assumption required building a solid literature base first, then demonstrating empirically that self-efficacy explained variance that age alone did not capture.

‍

The recommendation to the partner was not to abandon age as a variable, but to couple it with digital self-efficacy and mobile usage frequency at recruitment. That combination gives a richer, more representative sample without making the recruiting process significantly more complex.

‍

‍

Results

Hypotheses supported, actionable recommendations delivered

The hypothesis was supported: digital self-efficacy significantly moderates the relationship between age and several dimensions of user experience. Age does influence certain UX measures such as performance and cognitive load, but on its own it does not capture the real diversity of user profiles.

‍

Other key findings

• Strong correlation between mobile phone usage frequency and digital self-efficacy
• A single question about usage frequency is sufficient to reliably predict self-efficacy level

‍

Strategic recommendation to the partner

Couple age with digital self-efficacy and mobile usage frequency at the recruitment stage. This produces richer participant profiles, reduces sampling bias, and increases the reliability of future UX studies.

‍

The banking partner's UX team found the recommendations compelling enough to integrate them directly into their internal research process. The methodology was adopted, not just noted.

‍

‍

Conclusion and Learnings

Rigorous science in a business context

This project sharpened my ability to operate at the intersection of academic research and industry constraints. Working within a real partnership meant the findings had to be both scientifically sound and immediately usable by a product team.

‍

• Running a complete scientific process end-to-end in an industrial setting
• Building expertise in advanced quantitative analysis using SAS and statistical moderation models
• Conducting complex usability testing involving eye-tracking and biometric equipment
• Collaborating with diverse stakeholders across research and product functions
• Structuring and presenting findings in a way that drives real decisions, not just reports