Case Study: Microsoft Teams
Restoring Metric Integrity & Emotional Context to the Meeting Join Flow
A meeting-join funnel analysis showed a specific step in the flow was the biggest contributor to drop-off. Leadership handed down directives to fix it within the year. The product team, now under pressure, began to panic and was preparing to dive headfirst into an aggressive “fix the number” strategy; including an expensive design overhaul and elimination of key steps in the flow. But they didn’t understand why users were dropping off, nor what the numbers were really telling us.
Leadership was operating on a product metric that mischaracterized healthy user-behavior as “failures” (and vice versa) and they didn’t know it.
My work:
Uncovered the root-cause of this drop-off behavior
Re-defined the OKR
Identified what adding value would really mean for users
Proving and convincing leadership that their hypothesis was wrong.
This was not a usability issue.
I had been advocating for UX improvements as my previous research demonstrated significant usability issues within the join funnel. The product team had inherited an assumption coming from above that the dropoff was being caused by these usability issues. But this assumption was the result of a surface-level read and an incorrect, assumed causal-chain…
There is dropoff → there are usability issues → the dropoff is from usability issues.
Did users really not know how to turn the knob and open the door?
Did they really not know how to click the big, purple button?
Investing in fixing the usability issues (while meaningful and still a priority) would not help us hit this OKR.
I pulled together research across the organization (including my own) to demonstrate that while there were usability issues in the flow and they did deserve attention, they didn’t explain the dropoff behavior. And I did so quickly without investing in additional research.
The metric itself was the problem because it was detached from real user-behaviors and the business needs. Even if we hit the goal, were we improving the experience?
Why was the guy who told us to fix the meeting join UX now telling us that it wouldn’t achieve our goals?
Why is he telling us that dropoff isn’t a problem for users or the business?
With my Data Scientist, I audited the data pipeline to demonstrate how we were miscoding “successful” behaviors as “failures” (and vice versa).
False Failures: Is someone who starts joining a meeting a day before the meeting, who then leaves and comes back and joins at the time of the meeting a “failure”? Is this really a problem for the user, or just a problem for our own internal metric?
False Successes: Is someone who joins the meeting, has the wrong headset connected and then leaves and comes back a “success”?
By demonstrating a few cases like this (above) and how we were currently miscoding these our metric, I opened the door for a deeper look into the OKR itself.
Creating a metric that actually maps to our user’s and business’s reality.
Along with my PM and Data Scientist partners, we re-designed the OKR entirely. Moving it from its crude model to a more nuanced approach with various behavioral buckets. Such as,
Re-join Success: Users who attempt to join a meeting X minutes before it is scheduled to start and leave— but come back and join successfully at the meeting-start time.
Exit Failures: Our funnel analysis used to end after entering a meeting, meaning it was missing a key join-failure… Users who enter the meeting but then leave quickly due to settings-related issues (e.g. headset or microphone changes).
This enabled us to better track the health of the experience.
Uncovering the real underlying needs for us to improve the UX: the social-emotional opportunities.
The meeting join funnel isn’t a purchase/conversion funnel like in e-commerce but we had been treating the funnel like it was. This investigation inspired my discovery research into the social-emotional jobs to be done for our users which went on to re-shape our organization’s thinking around what users need out of Teams.
Where previously we were exclusively talking about Microsoft Teams as a “productivity tool”, my work shifted our focus onto the real jobs-to-be-done for our users which were the often-ignored aspects of JTBD, the social-emotional jobs.
Ultimately, I identified, drafted, and prioritized new features which mapped to our user’s most important social and emotional needs within the join flow.
Such as, “meeting peek”, a feature which is now available. This feature set shows information like who is already in the room before joining.
Why do we ask our friends when they’re arriving at a party before we show up?
Why do office doors and rooms typically have internal windows?
What is more welcoming? A storefront that’s a solid brick wall or one with big windows allowing you to see inside?
I used these physical-world examples to drive home what I learned in my research. Users weren’t failing to join because they didn’t know how to click the button. They were not joining due to social uncertainty and this was our opportunity for providing additional value to our users, not forcing them through a funnel.
Showing the team how to make evidence-backed decisions; preventing catastrophic actions done out of urgency.
The team was rushing to just entirely remove this step in the flow; user's can’t drop off from a step that isn’t there… Sure, this might help us hit the number and the team could report up to leadership that they achieved their goal but it would be a false-victory. I conducted research to prove that this would actually hurt the user experience as this was a high-value step in the flow and most likely just move the dropoff to another step in the flow, not actually resolve it.
The team was under-pressure and wanted to act fast but… They didn’t understand why the dropoff was happening.
By outlining a holistic learning plan, I showed the team how we could get to success if we stopped panicking and started strategizing.
My milestone-based plan got the team to pause and focus first on understanding why the numbers were showing what they were before diving headfirst into risky “chase the number” solutions. All while still feeling like they’d be able to hit their goals in time to report to their leaders.
