menu
close
Wayfair
Senior Product Design Lead
UX / Conversational Design / Web / Mobile
Life before the Virtual Assistant
Welcome to the Wayfair Virtual Assistant (VA) case study – a classic case of hacking user experience design to transform an underperforming AI chatbot into a customer's best friend.
Picture this: a chatbot, initially stumped by a wide range of customer inquiries, reinvented to deliver precise, tailored responses with style. We didn't stop at just making the chatbot smarter; we aimed to boost customer satisfaction and cut operational costs, making the entire process leaner and more efficient. And the best part? The impact was not confined to the chatbot. The ripple effects were felt across the organization, leading to significant operational savings and a substantial drop in escalations to customer service agents.
Leading the charge as the Senior Product Design Lead, I spearheaded the strategic redesign of Wayfair's Virtual Assistant. It was all about using a blend of quantitative and qualitative data to guide our UX problem-solving. And this wasn't a one-person show – I got to work closely with teams from operations, customer service, and sales, ensuring our redesign was a collective success. From sifting through user data to brainstorming and prototyping, I oversaw each stage of this platform-level project. Plus, I had the chance to mentor upcoming design talent on our team.
PM, Content Design, 2 Designers, 5 Engineers, Ops, Customer Service Agents
What is The Virtual Assistant?
Wayfair’s Virtual Assistant (VA) is an A.I. chatbot that helps customers answer questions and solve issues with their orders. The benefits - reduced phone, chat, and email interactions between customers and agents. It handles simple matters, while agents can solve complex problems.
The Challenge
Defining the Problem
Here's the problem. We discovered our trusty VA was struggling with a glaring issue: It was failing to accurately respond to the most frequently asked topic – Order Status ("Where is my package?"). We're talking about a whopping 30% of customers using the VA were receiving incorrect solutions. This was a significant problem affecting over 5% of our customer base globally.
Here is an example of a "Good" customer flow when interacting with the VA compared to "Bad" flows:
The fallout? An unacceptably high "Unhelpful Rate" from customers, a low customer satisfaction rate, increased operational costs, and a spike in cases needing direct intervention from customer service agents. Clearly, a surface-level tweak wouldn't cut it. This was a call to action for a full-blown, strategic intervention.
Below, you'll see that flow that a typical customer goes through when their conversation is categorized as Order Status:
Discovery + Competitive Audit
Transcript Deep Dive
To solve a problem, you first need to understand it. So we rolled up our sleeves and dove into a deep-dive analysis of over 500 customer transcripts tagged as "Order Status". Our objective was to uncover the patterns and identify the recurring issues.
The findings were illuminating. Our VA was providing generic, one-size-fits-all solutions, missing crucial information cues from the customers, and failing to offer specific actions based on customer needs.
This screenshot shows actual transcripts where the VA got confused and gave customers the wrong solution!
Competitive Audit
We didn't stop at self-examination. We looked outwards, conducting a comprehensive audit of how other eCommerce sites and delivery services handled similar inquiries with their chatbots. This was about learning from the best practices out there and infusing those insights into our redesign strategy.
Ideation + Prototyping
Armed with our insights, we stepped into the ideation phase. I crafted three distinct prototypes, each representing a unique conversational flow for the VA. Each flow was designed to present follow-up actions to the user after the initial VA prompt. This was more than just a cosmetic makeover; it was about reengineering the entire conversational structure of our VA to better cater to our customers' needs.
But there were challenges to be mindful of: the limited screen real estate for the VA, the urgency to improve the VA's accuracy without breaking the bank, and the need to present additional options in a way that wouldn't overwhelm the users.
Shows customers a complete list of sub-intent options based on their current Order Status (ex. Shipped vs. Delivered).
Shows high-level categories first. Once the user clicks on the category, they will be presented with the corresponding options for that category.
Similar to Flow #1, but before seeing all of the options, the user will be asked a Yes/No question if they want to take another action.
Prototypes
Testing + Iteration
Next up was the testing phase. We subjected our designs to rigorous user testing involving interactive prototypes, moderated usability tests, and meticulous session planning. What did we learn? Users were overwhelmingly in favor of the flow that displayed a full menu of follow-up options right upfront (flow #1).
Despite what myself and the team originally predicted, Flow #1 that surfaces a complete list of options upfront performed the best qualitatively and quantitatively.
Design System Component Redesign
The journey didn't end with testing. We went back to the drawing board to iterate on our designs, revisiting every design component, refining our copy, and customizing the options that would appear based on the status of each item.
Below, you'll see an example of how refactoring the message component saved over 140px in height. This may sound like a small change, but turns out to be huge when considered the limited screen height of a customers device.
Solution + Impact
After some rigorous iteration, I updated the designs with new flows and copy, visualizing every flow using existing and new UI, copy, and components. The final designs were then handed off to the engineering team to bring them to life.
The impact of this project exceeded our expectations. We're talking about $300k in annual savings, a 1.9 increase in CSAT (customer satisfaction rate), a nearly 50% reduction in the "Unhelpful rate," and over 30% fewer escalations to customer service.
The takeaway
The numbers speak for themselves, but what they don't tell you is the story of the countless hours spent in testing, the painstaking attention to detail in design, and the relentless commitment to improving the customer experience. That was the real win - not just creating a more efficient VA, but creating one that understood and catered to the needs of our customers in a more human way.
