Case Study: Microsoft Teams Meetings
Restoring Metric Integrity to the Meeting Join Flow
A funnel analysis revealed a key dropoff step. The Meeting Join team was diving headfirst into a “fix the number” strategy, without understanding why it was happening or what it really meant for our customers and business.
The hidden problem:Leadership was operating on a metric that was mischaracterizing user behavior and they didn’t know it!
What I did:
Uncovered the root-cause of drop-off in the funnel
Re-defined the team’s metrics and OKR
Repaired the organizations foundational understanding of our customers’ and our businesses real needs
Identified, conceptualized, and prioritized new features into the roadmap
Proving and convincing leadership that their hypothesis was wrong.
There were years of documented usability issues within this flow and I myself had advocated for foxing them. So, when this dropoff number got reported to leadership, they assumed an incorrect causal chain between the dropoff and these usability issues.
There is dropoff + there are usability issues = the dropoff is due to usability issues
It’s not like they didn’t know how to click the big purple button.
While we had very little research on the join flow itself, every research session that involved in-meeting research, had included the join flow as a by-product. I quickly pulled together research from over the years (including my own) to demonstrate that the usability issues, while they existed, did not explain the dropoff itself.
I quickly pulled together research from over the years (including non-join focused research to demonstrate that the usability issues that existed, didn’t
I pulled together research from over the years (including my own) which often involved (as a by-product), users joining meetings. Which demonstrated that the usability issues didn’t explain the dropoff itself. There were no documented cases of someone being unable to join the meeting due to these issues.
What I did… pull together research across my previous work and across the organization where meetings were involved to show that usability wasn’t the cause of the issues.
Fixing the usability issues would not help us hit this OKR.
There were usability issues, I myself had documented and advocated for fixing them.
I pulled together research across the organization (including my own) to demonstrate that while there were usability issues in the flow and they did deserve attention, they didn’t explain the dropoff behavior. And I did so quickly without investing in additional research.
Did users really not know how to open the door? Or did they just not want to enter the room?
Creating a metric that actually maps to our user’s and business’s reality.
The metric itself was the problem because it was detached from real user-behaviors and the business needs. Even if we hit the goal, were we improving the experience?
Why was the guy who told us to fix the meeting join UX now telling us that it wouldn’t achieve our goals?
Why is he telling us that dropoff isn’t a problem for users or the business?
With my Data Scientist, I audited the data pipeline to demonstrate how we were miscoding “successful” behaviors as “failures” (and vice versa).
False Failures: Is someone who starts joining a meeting a day before the meeting, who then leaves and comes back and joins at the time of the meeting a “failure”? Is this really a problem for the user, or just a problem for our own internal metric?
False Successes: Is someone who joins the meeting, has the wrong headset connected and then leaves and comes back a “success”?
By demonstrating a few cases like this (above) and how we were currently miscoding these our metric, I opened the door for a deeper look into the OKR itself.
Along with my PM and Data Scientist partners, we re-designed the OKR entirely. Moving it from its crude model to a more nuanced approach with various behavioral buckets. Such as,
Re-join Success: Users who attempt to join a meeting X minutes before it is scheduled to start and leave— but come back and join successfully at the meeting-start time.
Exit Failures: Our funnel analysis used to end after entering a meeting, meaning it was missing a key join-failure… Users who enter the meeting but then leave quickly due to settings-related issues (e.g. headset or microphone changes).
This enabled us to better track the health of the experience.
INSERT IMAGE SHOWING THE FLOW/FUNNEL… leaving and coming back.
Uncovering the real underlying needs for us to improve the UX: the social-emotional opportunities.
The meeting join funnel isn’t a purchase/conversion funnel like in e-commerce but we had been treating it like it was. We weren’t losing revenue from people not joining a meeting, unlike a purchase flow where items left in the cart are potential revenue left on the table. So, what was the real goal? This investigation inspired my discovery research into the social-emotional jobs to be done for our users which went on to re-shape our organization’s thinking around what users need out of Teams and redefining our goals.
Where previously we were exclusively talking about Microsoft Teams as a “productivity tool”, my work shifted our focus onto the real jobs-to-be-done for our users which were the often-ignored aspects of JTBD, the social-emotional jobs.
Ultimately, I identified, drafted, and prioritized new features which mapped to our user’s most important social and emotional needs within the join flow.
Such as, “meeting peek”, a feature which is now available. This feature set shows information like who is already in the room before joining.
Why do we ask our friends when they’re arriving at a party before we show up?
Why do office doors and rooms typically have internal windows?
What is more welcoming? A storefront that’s a solid brick wall or one with big windows allowing you to see inside?
I used these physical-world examples to drive home what I learned in my research. Users weren’t failing to join because they didn’t know how to click the button. They were not joining due to social uncertainty and this was our opportunity for providing additional value to our users, not forcing them through a funnel.
Showing the team how to make evidence-backed decisions; preventing catastrophic actions done out of urgency.
The team was rushing to just entirely remove this step in the flow; user's can’t drop off from a step that isn’t there… Sure, this might help us hit the number and the team could report up to leadership that they achieved their goal but it would be a false-victory. I conducted research to prove that this would actually hurt the user experience as this was a high-value step in the flow and most likely just move the dropoff to another step in the flow, not actually resolve it.
The team was under-pressure and wanted to act fast but… They didn’t understand why the dropoff was happening.
By outlining a holistic learning plan, I showed the team how we could get to success if we stopped panicking and started strategizing.
My milestone-based plan got the team to pause and focus first on understanding why the numbers were showing what they were before diving headfirst into risky “chase the number” solutions. All while still feeling like they’d be able to hit their goals in time to report to their leaders.
Image of a learning plan outline with milestones.