Apartment Therapy: Delivering Better Content Across Devices
2016 - 2018
The Challenge, Hypothesis and Goals
Challenge: How can we deliver personalized content options to readers?
Hypothesis: By implementing and testing data driven recommendation APIs, we can deliver more opportunities for readers to read better content.
Background: We already used Parse.ly for editorial analytics, so pulling recommendations from their API was a relatively low hurdle. In addition to Parse.ly we planned to test LiftIgniter, a machine learning API for making recommendations. We had been using Parse.ly for a limited number of users and saw a very slight upward trend in related metrics, but nothing crazy.
Editorial Goal: Deliver better content recommendations on article pages.
Metrics: Time on site and page views.
Business Goal: Raise view-ability numbers for advertising impressions.
Metrics: Viewability, impressions and CTR for ads in DFP.
User Goals: Make it easier and more fun to read AT Media properties by making relevant content easier to find.
Metrics: Social sharing and sessions.
Out of the gate our cross-functional team came together to define a common-sense scope for our project. What came out of that were a few ideas on how we could make this engaging and fun for users to navigate the three categories of content we want to deliver:
- Could we make it feel magical?
- A choose your own adventure game?
- A survey of some sort?
- A simple tabbed solution?
Lots of other fun and kind of silly possibilities were discussed. We compiled research both on what had worked for us in the past and what other media outlets were doing to make related content visible. The options kind of ran the gamut or what was possible: text links, teaser rolls, simplified teasers, images, stylized images. It was kind of all over the place. Based on our research and some of the notes we took, I started playing around in Sketch trying to see what some of those ideas might look like.
After reviewing what we had in Sketch, we moved the mockups to InVision to test them out. Through some simple user testing (both in-person and through UserTesting.com’s PEEK feature) we realized that the fun ideas we had were kind of terrible for a reader. We tested the concepts with users inside our ideal demographics range: Two female and one male, between 24-34, 2 with children, one without. The overwhelming response by these users was that we had made the interface more complex than it needed to be. Some specific comments were, “I might use this once, but it’s pretty cumbersome,” and “My only problem… is that this makes me have to think a lot more than I feel like I should about what I’m going to be seeing.” It was clear that these early concepts created roadblocks for users and were not conducive to skimming headlines and clicking into articles. This qualitative data early on was a guiding light for the concepts that came after.
During our regroup with one of our Product Managers, it became more and more clear that we were starting too big. We needed to scale back and start again. What does that look like? How does that feel? We refocused and simplified our original scope:
- We have three content types we need to deliver to all readers
- We want all three to be optimized across devices
- We need an easy-to-use interface that doesn’t feel clunky
Based on these learnings it was clear that building this into some sort of game didn’t feel on-brand and didn’t meet the goals of the feature. So we stripped everything back to a significantly lower common denominator for testing. We still wanted an interactive experience. We wanted users to feel like they were in control of the experience. As such we started with 3 clickable tabs correlated to our three content categories.
In implementation we ran an initial test comparing Parse.ly to LiftIgniter apples-to-apples and saw a marked difference in engagement. LiftIgniter was clearly making more actionable recommendations. Over time we would come to traffic most of our user recommendations through LiftIgniter because of the staggering increase in engagement.
Results and Iteration
What we found after a few days of monitoring analytics numbers was that very few people (<2%) were switching through the tabbed content on larger screens and smaller screens were doing only slightly better. After talking through the results and ramifications, we decided to split the three content types out into three columns on larger screens. We kept the tabbed interface on smaller screens in hope that adoption was taking more time than expected. While tabs hide content and hiding content is almost never a good thing.
It wasn’t. We got analytics numbers back and “desktop” click through rate had gone up from 3.6% to 5.8% and “mobile” had gone down to 3.3% with 99% of clicks coming from the default active tab. Up to this point, we had been using the Parse.ly API to drive recommendations in the “Related” and “Popular” tabs. We decided to follow some suggestions from Parse.ly and LiftIgniter to test a “back to the drawing board” option. Their data suggests that a simple 4x3 grid of article teasers might increase overall interactions.
Based on Parse.ly and LiftIgniter suggestions we did, in fact, remove all tabs on smaller screens and went to a 3 row, 4 column view on larger screens. Switching to this layout yielded growth in teasers clicked across screen sizes. As time has gone on we’ve tested and implemented more changes. Moving to LiftIgniter to power most of our machine learning-driven content recommendations. Splitting recirculation into a mix of plain text links after post content and switching most of our teasers to a highly scrollable new horizontal teaser design. Click through rates seem to hover around 8%. The Apartment Therapy team continue to test, monitor and iterate to deliver the best context-optimized content recommendations.
Other Projects for Apartment Therapy
Color Search by Sherwin-Williams
My first project at AT Media was a complete overhaul of our Sherwin-Williams sponsored Color Search tool, which was a collaboration with McKinney. I came into the process as wireframes were being handed off and was able to jump in and contribute to the team in the first 2 weeks after I started. Our work did not go unnoticed - along with our partners at McKinney and our direct sales and creative services teams, our work on Color Search won an IAB Mixx award in 2016.
Helping Build an A/B Testing Framework and Using it
Because of my love for data-informed design I was tapped to work with our Lead Front End Engineer to build a new system into our - largely monolithic - codebase. Using a fancy Varnish setup, courtesy of our Back End team, we created an internal methodology to use feature flippers and traffic splitting via our CDN. We were able to test this out of the box on social sharing designs for images. Our tests Eventually produced a 20% increase m/m in social sharing to Pinterest.
Home Page Teaser Redesign
At the end of 2017 our team wanted to be more experimental and take more risks with how we were showing content. A piece of low-hanging but effective piece of fruit related directly to this was our home page teasers. Those little image and text sandwiches that give our readers hints about what they’re looking at and for. We say a 35%+ increase in clicks on teasers on the home page by updating our layout to be more scroll-friendly. I worked with our Lead PM and my Design Manager to work through a couple of iterations of a horizontal teaser. Using our A/B testing framework and Google Optimize, we tested a couple of treatments. What we landed on was simple and scrollable and was a great win in user engagement metrics.
Community Focused initiatives
My all-time favorite projects for AT Media have been the online community driven initiatives. Launching our new Save feature, refreshing our profile pages and ongoing projects to integrate more connective community features have been a joy to work on. We have a dedicated user0base that loves to connect through comments and social media. It’s been a real treat developing features to satisfy some long-standing requests and work with users to develop a roadmap of new and upcoming features.