Case Study: Microsoft Teams Meetings

Restoring Metric Integrity to the Meeting Join Flow

A funnel analysis revealed a key dropoff step. The Meeting Join team was diving headfirst into a “fix the number” strategy, without understanding why it was happening or what it really meant for our customers and business.

The hidden problem: Leadership was operating on a metric that was mischaracterizing user behavior and they didn’t know it!

What I did:

  • Uncovered the root-cause of drop-off in the funnel

  • Aligned the UX goals and plans of three different product groups: Pre-Meeting (Czech Republic), In-Meeting (USA), Devices (China).

  • Re-defined the team’s metrics and OKR

  • Repaired the organization’s foundational understanding of our customers’ and our businesses real needs

  • Identified, conceptualized, and prioritized new features into the roadmap

The Problem

Leadership was working from an incorrect hypothesis and poorly constructed number.

The team reported an alarming figure around drop-off in the meeting join funnel… leadership reacted to it and gave them the annual OKR to improve it.

The problem is that this number was deeply flawed, but no one questioned it.

The (incorrect) assumed causal-chain

There is drop-off + there are usability issues = the drop-off is due to usability issues

There were years of documented usability issues within this flow; I myself had advocated for fixing them. When this drop-off number got reported to leadership, they made a common error of assuming they were related.

It’s not like users didn’t know how to click the big purple button.

We had very little evaluative research on the join flow itself but every research session that involved any meeting research, had included joining a meeting as a by-product. I quickly pulled together research from over the years (including my own) to demonstrate that while the usability issues were real, they did not explain the dropoff.

You’re telling me that our professional users, who have multiple meetings a day… just sometimes lose-their-heads and don’t know how to join a meeting? Do they just tell their client that day, “sorry folks, can’t figure out how to join the meeting, maybe see ya’ next time”? It just didn’t add up.

The implication: fixing the usability issues (as directed and planned) wouldn’t result in hitting our annual OKR.

Did users really not know how to open the door?
Or… did they just not want to enter the room, yet?

Creating a metric that actually maps to our user’s and our business’s reality.

This drop-off metric was an engineering metric which (over the years) had warped and was now masquerading as an experiential measure; and no one was questioning it.

Revealing its flaws.

Success vs Failure Codings

With my Data Scientist, I audited the data pipeline to demonstrate how we were miscoding “successful” behaviors as “failures” (and vice versa).

  • False Failures: Is someone who starts joining a meeting a day before the meeting (in order to just test it), who then leaves and comes back and joins at the time of the meeting a “failure”? Is this really a problem for the user, or just a problem for our own internal metric (a metric that isn’t even tied to business outcomes)?

  • False Successes: Is someone who gets all the way into the meeting, realizes they have the wrong headset connected, so they exit and then come back a “success”?

Aggregate vs Unique User Reporting

I also demonstrated how the aggregate-level data was insufficient for telling the real story and understanding the data; it was glossing over every nuance for the sake of easy-reporting instead of doing what a good metric does; provide true-signals about health.

For example:

  • User A: Has 100 meetings a month. Drops-off from 5 (5%).

  • User B: Has 5 meetings a month. Drops-off from 5 (100%).

According to the aggregate-view data, there’s no distinction whatsoever between these. They’re both just +5 counts of drop-off.

The lack of this individual-level analysis also meant we weren’t analyzing by market, by meeting type, by organizational size… by anything that could be potentially helpful in diagnostics.

I argued that this demonstrates the shallowness in the reporting (the reporting that organization-wide directives and goals were being set based on).

Fixing it.

By demonstrating a few cases like the above and how we were currently miscoding these in our metric, I opened the door for a deeper look into the OKR itself and was able to communicate up as to why this was entirely essential data cleanliness, not just a researcher being nit-picky.

Along with my PM and Data Scientist partners, we re-designed the OKR entirely. Moving it from its crude model to a more nuanced approach with various behavioral buckets. Such as,

  • Re-join Success: Users who attempt to join a meeting X minutes before it is scheduled and leave— but then come back and join successfully at the meeting-start time.

  • Exit Failures: Our funnel analysis used to end after entering a meeting, meaning it was missing a key join-failure… Users who enter the meeting but then leave quickly due to settings-related issues (e.g. headset or microphone changes).

INSERT IMAGE SHOWING THE FLOW/FUNNEL… leaving and coming back.

Uncovering the underlying needs for us to improve: the social-emotional opportunities.

This isn’t an e-commerce flow but we were treating it like it was.

In a checkout flow, items left in the cart are potential lost revenue. But we don’t lose revenue when people choose not to join the meeting. In fact, its the opposite; every time users join a meeting it costs the business more due to service calls.

So, what was the real goal? This investigation inspired my discovery research into the social-emotional jobs to be done for our users which went on to re-shape our organization’s thinking around what users need out of Teams and redefining our goals.

Where previously we were exclusively talking about Microsoft Teams as a “productivity tool”, my work shifted our focus onto the real jobs-to-be-done for our users which were the often-ignored aspects of JTBD, the social-emotional jobs.

Ultimately, I identified, drafted, and prioritized new features which mapped to our user’s most important social and emotional needs within the join flow.

Such as, “meeting peek”, a feature which is now available. This feature set shows information like who is already in the room before joining.

  • Why do we ask our friends when they’re arriving at a party before we show up?

  • Why do office doors and rooms typically have internal windows?

  • What is more welcoming? A storefront that’s a solid brick wall or one with big windows allowing you to see inside?

I used these physical-world examples to drive home what I learned in my research. Users weren’t failing to join because they didn’t know how to click the button. They were not joining due to social uncertainty and this was our opportunity for providing additional value to our users, not forcing them through a funnel.

Showing the team how to make evidence-backed decisions; preventing catastrophic actions done out of urgency.

The team was rushing to just entirely remove this step in the flow; user's can’t drop off from a step that isn’t there… Sure, this might help us hit the number and the team could report up to leadership that they achieved their goal but it would be a false-victory. I conducted research to prove that this would actually hurt the user experience as this was a high-value step in the flow and most likely just move the dropoff to another step in the flow, not actually resolve it.

The team was under-pressure and wanted to act fast but… They didn’t understand why the dropoff was happening.

By outlining a holistic learning plan, I showed the team how we could get to success if we stopped panicking and started strategizing.

My milestone-based plan got the team to pause and focus first on understanding why the numbers were showing what they were before diving headfirst into risky “chase the number” solutions.

Research can often become perceived as a bottleneck, or a killer of ideas. This plan allowed all disciplines to be able to communicate progress and predictability to their leadership — even if deadlines were being delayed for this discovery and foundational restructuring of the metric to be done.

All while still like they’d be able to hit their goals in time to report to their leaders.

Image of a learning plan outline with milestones.