SlideShare a Scribd company logo
1 of 59
Download to read offline
Metrics-Driven Design
                            by Joshua Porter

                            Twitter: bokardo


                                    bokardo
                                  52weeksofUX
                                  performable
                                   microcopy
                           designing for the social web
                                make them care!
                                                          by Joshua Porter

Saturday, March 12, 2011
Saturday, March 12, 2011
“Unfortunately for me, there was one
                           small problem I didn’t see back then.”



Saturday, March 12, 2011
Doug Bowman on Design at Google




      “                    Yes, it’s true that a team at Google couldn’t decide between two
                           blues, so they’re testing 41 shades between each blue to see
                           which one performs better. I had a recent debate over whether a
                           border should be 3, 4 or 5 pixels wide, and was asked to prove my
                           case. I can’t operate in an environment like that.




      Metrics-Driven Design                                                     Daring. Decisive. Conviction.   SXSW
Saturday, March 12, 2011
41 Shades of Blue Test




                                                        link color
        Gmail                                                                            Google Search




                           41 Bucket Split Test: ~2.5% of users each got a shade for 2 weeks.




Saturday, March 12, 2011
Saturday, March 12, 2011
#2200C1          #0044CC



                           Google             Bing



Saturday, March 12, 2011
“                    That blue was worth at
                           least $80 million.
                           Paul Ray, UX Manager, Bing




      Metrics-Driven Design                             Microcopy   SXSW
Saturday, March 12, 2011
Doug Bowman on Design at Google




      “                    When a company is filled with engineers, it turns to engineering to
                           solve problems. Reduce each decision to a simple logic problem.
                           Remove all subjectivity and just look at the data. Data in your favor?
                           Ok, launch it. Data shows negative effects? Back to the drawing board.
                           And that data eventually becomes a crutch for every decision,
                           paralyzing the company and preventing it from making any daring
                           design decisions.




      Metrics-Driven Design                                                      Daring. Decisive. Conviction.   SXSW
Saturday, March 12, 2011
Spectrum of Design


   Intuition-Driven                                                                               Data-Driven



           Make best-guesses                                                Every design choice is tested
           Rely on previous experience                        Takes others experience with a grain of salt
           Study what others are doing                                         Design is a logic problem
           Use best practices, principles & patterns                   Rely on data for decision-making
           Aesthetics are integral                                             Aesthetics are secondary
           Rely on our gut                                                          Never trust your gut
           Creative, visionary                                                          Cold, calculating
           Inherently risky                                                                   Risk-averse




           Doug’s words:                                                                       Assumed:

           instinctive, subjective, daring                          deliberate, objective, safe



Saturday, March 12, 2011
Imagine that your design is a mountain...




      Metrics-Driven Design               The grass is always greener on other mountainsides.   SXSW
Saturday, March 12, 2011
Imagine that your design is a mountain...




                    EXISTING DESIGN




      Metrics-Driven Design               The grass is always greener on other mountainsides.   SXSW
Saturday, March 12, 2011
Imagine that your design is a mountain...




                               Now



                    EXISTING DESIGN




      Metrics-Driven Design               The grass is always greener on other mountainsides.   SXSW
Saturday, March 12, 2011
Imagine that your design is a mountain...




                 Optimized (local maxima)

                               Now



                    EXISTING DESIGN




      Metrics-Driven Design                 The grass is always greener on other mountainsides.   SXSW
Saturday, March 12, 2011
Imagine that your design is a mountain...



           Potential
           Improvement



                 Optimized (local maxima)

                               Now




      Metrics-Driven Design                 The grass is always greener on other mountainsides.   SXSW
Saturday, March 12, 2011
Imagine that your design is a mountain...


                                        Goal




                                                                    BETTER DESIGN



                       Current Design



                    EXISTING DESIGN




      Metrics-Driven Design                    The grass is always greener on other mountainsides.   SXSW
Saturday, March 12, 2011
Imagine that your design is a mountain...


                                      Goal

           Potential
           Improvement

                                                                  BETTER DESIGN



                              Now



                    EXISTING DESIGN




      Metrics-Driven Design                  The grass is always greener on other mountainsides.   SXSW
Saturday, March 12, 2011
Optimization asks:
                           What works best in the current model?



                                    Design innovation asks:
                             What is the best possible model?




Saturday, March 12, 2011
What are Metrics?




Saturday, March 12, 2011
DEFINITION


                  Metrics are simply numbers
                that measure the effectiveness
                       of your business.



Saturday, March 12, 2011
5 Reasons why Metrics are a
                          Designer’s Best Friend




Saturday, March 12, 2011
1. Metrics reduce arguments
                          based on opinion.




Saturday, March 12, 2011
2. Metrics give you answers
                            about what really works.




Saturday, March 12, 2011
3. Metrics show you where
                           you’re strong as a designer.




Saturday, March 12, 2011
4. Metrics allow you to test
                               anything you want.




Saturday, March 12, 2011
5. Clients love metrics.




Saturday, March 12, 2011
PRINCIPLE

                  Your metrics will be as unique
                        as your business.




Saturday, March 12, 2011
Saturday, March 12, 2011
Vanity Metrics




      Metrics-Driven Design   Ego   SXSW
Saturday, March 12, 2011
The Usage Lifecycle




Saturday, March 12, 2011
Interested   Trial/beta User   Customer   Passionate
                                                             Customer




Saturday, March 12, 2011
Acquisition          Conversion      Engagement        Satisfaction




           Interested               Trial/beta User      Customer        Passionate
                                                                         Customer




Saturday, March 12, 2011
With metrics you are essentially just measuring
                how well you move people along the lifecycle!




Saturday, March 12, 2011
Acquisition Metrics




Saturday, March 12, 2011
CPA - Cost per Acquisition

                           If your CPA is higher than your LTV
                                  then you’re in trouble.




Saturday, March 12, 2011
Acquisition vs. Referral: Dropbox


                                            Ran Google Adwords campaigns
                                            to drive traffic to dropbox.com

                                            The traffic that completed the lifecycle:

                                            1) Searched on a keyword
                                            2) Visited their site
                                            3) Signed-up for service
                                            4) Became a customer

                                            cost them $233-$388 per person!
                                            (for a $99 product)




      Metrics-Driven Design                                  LTV = Lifetime value   SXSW
Saturday, March 12, 2011
Dropbox Lifecycle




                              30 days prior to April 2010, Dropbox users
                              sent 2.8 million direct referral invites.


                              Dropbox’s referral program with 2-sided
                              incentive increased sign-ups by 60%
                              permanently.




      Metrics-Driven Design                         Referral = Word of Mouth   SXSW
Saturday, March 12, 2011
Performable Acquisition Metrics



                      Comparative Metrics

                           Revenue by Channel

                       Revenue by Keyword




      Metrics-Driven Design                     Microcopy   SXSW
Saturday, March 12, 2011
Conversion Metrics




Saturday, March 12, 2011
Trials / Conversions = Conversion Rate




Saturday, March 12, 2011
Conversion Funnel Analysis



                    A                A         100%




                     B               B     60%




                     C               C   20%


      Metrics-Driven Design                           SXSW
Saturday, March 12, 2011
Sign-up Conversion Funnel



       Original Flow                     1                    2                    3                      4
                                  confirm personal info   add your friends       invite others         getting started




        Original Conversion
        Funnel
                                                                            1                            100%

         of the 100% of people who
         started the sign-up process, only                                  2                         63%
         14% made it to the getting
         started screen.
                                                                            3                   26%


                                                                            4          14%


      Metrics-Driven Design                                                                                             SXSW
Saturday, March 12, 2011
Sign-up Conversion Funnel




          New Flow
                                                  2                          4
                                             add your friends           getting started




          New Conversion Funnel                                  2                                     100%
          of the 100% of people who
          started the sign-up process, 86%
          made it to the getting started
          screen.
                                                                 4                                86%




      Metrics-Driven Design                                Removing stuff is a quick way to improve your funnel   SXSW
Saturday, March 12, 2011
Engagement Matters: Twitter




                                                     http://www.lukew.com/ff/entry.asp?1039



      Metrics-Driven Design           Speed without context is wasted.           SXSW
Saturday, March 12, 2011
Engagement Metrics




Saturday, March 12, 2011
•   Hits
                           •   Page views
                           •   Visits
                           •   Unique Visitors
                           •   Returning Visitors
                           •   Registered Users
                           •   Customers
                           •   Frequency
                           •   Time on Site
                           •   Daily Active Users




Saturday, March 12, 2011
Cohort Analysis

                                                                Engagement over time

                      Month 1     Month 2   Month 3   Month 4   Month 5   Month 6    Month 7    Month 8     Month 9      Month 10 Month 11 Month 12

      (Joined in)
                           100%      20%       19%       13%       13%       10%        12%        11%              7%           7%            7%
                                                                                                                                                         ?
      January


                           100%      21%       16%       13%       11%          9%         9%         7%            7%           7%
                                                                                                                                           ?
      February


                           100%      24%       20%       17%       15%       13%        11%        10%           10%
                                                                                                                             ?
      March


                           100%      31%       27%       24%       19%       15%        12%        12%
                                                                                                                ?
      April


                           100%      31%       27%       25%       21%       18%        16%
                                                                                                  ?
      May


                           100%      39%       28%       24%       20%       19%
                                                                                       ?
      June


                           100%      40%       33%       27%       23%
                                                                            ?
      July


                           100%      47%       41%       32%
                                                                  ?
      August


                           100%      52%       43%
                                                        ?
      September
                                                                          In this case engagement is improving nicely.
                           100%      53%
                                              ?                           Of the January cohort, only 20% were engaged in month 2.
      October                                                             Of the October cohort, 53% were engaged in month 2.

                           100%
                                    ?
      November

                           ?
      December                                                                                        http://redeye.firstround.com/2008/01/after-the-techc.html



Saturday, March 12, 2011
Prevention: Facebook Deactivation

                                                      Design changes to
                                                    the deactivation page
                                                   accounted for 1 million
                                                   members not leaving the
                                                           service.
                                                                              Julie Zhou, Facebook




                                    http://www.readwriteweb.com/archives/what_happens_when_you_deactivate_your_facebook_acc.php


      Metrics-Driven Design                                          I will miss you after my talk today.             SXSW
Saturday, March 12, 2011
Satisfaction Metrics




Saturday, March 12, 2011
Referral: Net Promoter Score




            How likely is it that you would recommend
              our company to a friend or colleague?

                0             1   2   3   4   5   6            7            8           9            10




      Metrics-Driven Design                           Do you know what your net promoter score is?        SXSW
Saturday, March 12, 2011
Net Promoter Score




            How likely is it that you would recommend
              our company to a friend or colleague?

                0             1      2       3        4   5   6            7            8           9            10

                                         Detractors                          Passives               Promoters




                                  Score = % Promoters - % Detractors


      Metrics-Driven Design                                       Do you know what your net promoter score is?        SXSW
Saturday, March 12, 2011
Mint.com & Net Promoter Score




      “                    Maybe we didn’t have a high viral coefficient
                           but we had a great net promoter score.
                           Jason Putorti, Lead Designer, Mint.com




      Metrics-Driven Design                                              SXSW
Saturday, March 12, 2011
Emergent Metrics




Saturday, March 12, 2011
Emergent Metrics: 5 Friends

                                      The magic number is 5.
                                      Once a FriendFeed user
                                      found five friends, they
                                       became active users.
                                               Bret Taylor, Friendfeed




      Metrics-Driven Design                                        SXSW
Saturday, March 12, 2011
Is there one metric that drives others?




      “                    At Blogger, we determined that our most critical
                           metric was number of posts. An increase in posts
                           meant that people were not just creating blogs, but
                           updating them, and more posts would drive more
                           readership, which would drive more users, which
                           would drive more posts.

                                                           Ev Williams
                                                           founder of Blogger & Twitter




      Metrics-Driven Design                                    Once you’re lucky, twice you’re good.   SXSW
Saturday, March 12, 2011
Principles of Metrics-Driven Design


               1. Optimize in small steps; innovate with daring leaps.
               2. No design survives contact with the user.
               3. Small improvements, taken together, yield amazing results. 
               4. Testing is empowering, reversion is cleansing.
               5. Metrics are not creative: human beings are.
               6. All team members are responsible for the user experience.
               7. If metrics aren’t actionable, they aren’t useful.
               8. Design is never done.




      Metrics-Driven Design                           Reminder: Principal = Person & Principle = Thing   SXSW
Saturday, March 12, 2011
Saturday, March 12, 2011
This Talk.



       http://bokardo.com/talks/metrics-driven-design/




Saturday, March 12, 2011
My Life.




                    http://performable.com              http://oneflightbooks.com




Saturday, March 12, 2011

More Related Content

What's hot

Validate Your Ideas Quickly with Google Design Sprint
Validate Your Ideas Quickly with Google Design SprintValidate Your Ideas Quickly with Google Design Sprint
Validate Your Ideas Quickly with Google Design SprintBorrys Hasian
 
Designing To Learn: Creating Effective MVP Experiments - Lean UX NYC 2014
Designing To Learn: Creating Effective MVP Experiments - Lean UX NYC 2014Designing To Learn: Creating Effective MVP Experiments - Lean UX NYC 2014
Designing To Learn: Creating Effective MVP Experiments - Lean UX NYC 2014Melissa Perri
 
Customer Centricity and Product Led Growth by Airbnb Product & Growth
Customer Centricity and Product Led Growth by Airbnb Product & Growth Customer Centricity and Product Led Growth by Airbnb Product & Growth
Customer Centricity and Product Led Growth by Airbnb Product & Growth Product School
 
NYT Product Discovery Activity Guide
NYT Product Discovery Activity GuideNYT Product Discovery Activity Guide
NYT Product Discovery Activity GuideAl Ming
 
Edward Tufte and Information Design for the Web
Edward Tufte and Information Design for the WebEdward Tufte and Information Design for the Web
Edward Tufte and Information Design for the Websprocketeer
 
Making jobs-to-be-done actionable / Service Design Drinks
Making jobs-to-be-done actionable / Service Design DrinksMaking jobs-to-be-done actionable / Service Design Drinks
Making jobs-to-be-done actionable / Service Design DrinksService Design Berlin
 
15 minutes on impact mapping
15 minutes on impact mapping15 minutes on impact mapping
15 minutes on impact mappingMarcus Hammarberg
 
Dropbox startup lessons learned 2011
Dropbox   startup lessons learned 2011Dropbox   startup lessons learned 2011
Dropbox startup lessons learned 2011Eric Ries
 
Data Storytelling: The only way to unlock true insight from your data
Data Storytelling: The only way to unlock true insight from your dataData Storytelling: The only way to unlock true insight from your data
Data Storytelling: The only way to unlock true insight from your dataBright North
 
IBM Design Thinking - Hen shkedi
IBM Design Thinking - Hen shkediIBM Design Thinking - Hen shkedi
IBM Design Thinking - Hen shkediHen Shkedi
 
Breakthrough Innovation with Jobs To Be Done
Breakthrough Innovation with Jobs To Be DoneBreakthrough Innovation with Jobs To Be Done
Breakthrough Innovation with Jobs To Be DoneClaudio Perrone
 
UX Maturity Models
UX Maturity Models   UX Maturity Models
UX Maturity Models Gena Drahun
 
UX & UI Design: Differentiate through design
UX & UI Design: Differentiate through designUX & UI Design: Differentiate through design
UX & UI Design: Differentiate through designMoodLabs
 
A Playbook for Achieving Product-Market Fit
A Playbook for Achieving Product-Market FitA Playbook for Achieving Product-Market Fit
A Playbook for Achieving Product-Market FitLean Startup Co.
 
Lean analytics: Five lessons beyond the basics
Lean analytics: Five lessons beyond the basicsLean analytics: Five lessons beyond the basics
Lean analytics: Five lessons beyond the basicsLean Analytics
 
Design Thinking
Design Thinking Design Thinking
Design Thinking Provectus
 

What's hot (20)

Validate Your Ideas Quickly with Google Design Sprint
Validate Your Ideas Quickly with Google Design SprintValidate Your Ideas Quickly with Google Design Sprint
Validate Your Ideas Quickly with Google Design Sprint
 
Designing To Learn: Creating Effective MVP Experiments - Lean UX NYC 2014
Designing To Learn: Creating Effective MVP Experiments - Lean UX NYC 2014Designing To Learn: Creating Effective MVP Experiments - Lean UX NYC 2014
Designing To Learn: Creating Effective MVP Experiments - Lean UX NYC 2014
 
Customer Centricity and Product Led Growth by Airbnb Product & Growth
Customer Centricity and Product Led Growth by Airbnb Product & Growth Customer Centricity and Product Led Growth by Airbnb Product & Growth
Customer Centricity and Product Led Growth by Airbnb Product & Growth
 
NYT Product Discovery Activity Guide
NYT Product Discovery Activity GuideNYT Product Discovery Activity Guide
NYT Product Discovery Activity Guide
 
IDEO - Design thinking workshop 2016
IDEO - Design thinking workshop 2016IDEO - Design thinking workshop 2016
IDEO - Design thinking workshop 2016
 
Edward Tufte and Information Design for the Web
Edward Tufte and Information Design for the WebEdward Tufte and Information Design for the Web
Edward Tufte and Information Design for the Web
 
Design Thinking
Design ThinkingDesign Thinking
Design Thinking
 
Making jobs-to-be-done actionable / Service Design Drinks
Making jobs-to-be-done actionable / Service Design DrinksMaking jobs-to-be-done actionable / Service Design Drinks
Making jobs-to-be-done actionable / Service Design Drinks
 
The Minimum Loveable Product
The Minimum Loveable ProductThe Minimum Loveable Product
The Minimum Loveable Product
 
Design Thinking for Business Innovation
Design Thinking for Business InnovationDesign Thinking for Business Innovation
Design Thinking for Business Innovation
 
15 minutes on impact mapping
15 minutes on impact mapping15 minutes on impact mapping
15 minutes on impact mapping
 
Dropbox startup lessons learned 2011
Dropbox   startup lessons learned 2011Dropbox   startup lessons learned 2011
Dropbox startup lessons learned 2011
 
Data Storytelling: The only way to unlock true insight from your data
Data Storytelling: The only way to unlock true insight from your dataData Storytelling: The only way to unlock true insight from your data
Data Storytelling: The only way to unlock true insight from your data
 
IBM Design Thinking - Hen shkedi
IBM Design Thinking - Hen shkediIBM Design Thinking - Hen shkedi
IBM Design Thinking - Hen shkedi
 
Breakthrough Innovation with Jobs To Be Done
Breakthrough Innovation with Jobs To Be DoneBreakthrough Innovation with Jobs To Be Done
Breakthrough Innovation with Jobs To Be Done
 
UX Maturity Models
UX Maturity Models   UX Maturity Models
UX Maturity Models
 
UX & UI Design: Differentiate through design
UX & UI Design: Differentiate through designUX & UI Design: Differentiate through design
UX & UI Design: Differentiate through design
 
A Playbook for Achieving Product-Market Fit
A Playbook for Achieving Product-Market FitA Playbook for Achieving Product-Market Fit
A Playbook for Achieving Product-Market Fit
 
Lean analytics: Five lessons beyond the basics
Lean analytics: Five lessons beyond the basicsLean analytics: Five lessons beyond the basics
Lean analytics: Five lessons beyond the basics
 
Design Thinking
Design Thinking Design Thinking
Design Thinking
 

Similar to Metrics Driven Design by Joshua Porter

How small changes big
How small changes bigHow small changes big
How small changes bigAdam Hassan
 
Metrics drivendesign
Metrics drivendesignMetrics drivendesign
Metrics drivendesignRoman Zykov
 
Scrum, it's all about the people
Scrum, it's all about the peopleScrum, it's all about the people
Scrum, it's all about the peopleDennis Vroegop
 
Adaptive Experiences
Adaptive ExperiencesAdaptive Experiences
Adaptive ExperiencesWINTR
 
SEF for students 2012. Product Management vs. Project Management
SEF for students 2012. Product Management vs. Project ManagementSEF for students 2012. Product Management vs. Project Management
SEF for students 2012. Product Management vs. Project ManagementAnthony Marchenko
 
What User-Centered Design is Good For
What User-Centered Design is Good ForWhat User-Centered Design is Good For
What User-Centered Design is Good ForDan Saffer
 
Midwest UX Mobile Workshop 2012
Midwest UX Mobile Workshop 2012Midwest UX Mobile Workshop 2012
Midwest UX Mobile Workshop 2012Haig Armen
 
LISA 2011 Keynote: The DevOps Transformation
LISA 2011 Keynote: The DevOps TransformationLISA 2011 Keynote: The DevOps Transformation
LISA 2011 Keynote: The DevOps Transformationbenrockwood
 
Critical Thinking forUX Designers (Workshop)
Critical Thinking forUX Designers (Workshop)Critical Thinking forUX Designers (Workshop)
Critical Thinking forUX Designers (Workshop)Stephen Anderson
 
Ux paper prototyping
Ux paper prototypingUx paper prototyping
Ux paper prototypingGrace Ng
 
Sustainable Theming with Fusion - DCCO 2011
Sustainable Theming with Fusion - DCCO 2011Sustainable Theming with Fusion - DCCO 2011
Sustainable Theming with Fusion - DCCO 2011sheenadonnelly
 
Designing for Smart Spaces and Objects
Designing for Smart Spaces and ObjectsDesigning for Smart Spaces and Objects
Designing for Smart Spaces and Objectsfrog
 
Insider's Technology Guide to Measuring and Improving Your ROI
Insider's Technology Guide to Measuring and Improving Your ROIInsider's Technology Guide to Measuring and Improving Your ROI
Insider's Technology Guide to Measuring and Improving Your ROIDan Leatherman
 
Design for Marketing Technical Services
Design for Marketing Technical ServicesDesign for Marketing Technical Services
Design for Marketing Technical ServicesJon Sandruck
 
Design thinking notes with explanation i
Design thinking notes with explanation iDesign thinking notes with explanation i
Design thinking notes with explanation iSRINURAJU3
 
Design Thinking is Killing Creativity
Design Thinking is Killing CreativityDesign Thinking is Killing Creativity
Design Thinking is Killing Creativitydesignsojourn
 
S.P.A.C.E. Exploration for Software Engineering
 S.P.A.C.E. Exploration for Software Engineering S.P.A.C.E. Exploration for Software Engineering
S.P.A.C.E. Exploration for Software EngineeringCS, NcState
 
Class 2: Setting the foundation for a successful website design
Class 2: Setting the foundation for a successful website designClass 2: Setting the foundation for a successful website design
Class 2: Setting the foundation for a successful website designDon Stanley
 

Similar to Metrics Driven Design by Joshua Porter (20)

How small changes big
How small changes bigHow small changes big
How small changes big
 
Metrics drivendesign
Metrics drivendesignMetrics drivendesign
Metrics drivendesign
 
Scrum, it's all about the people
Scrum, it's all about the peopleScrum, it's all about the people
Scrum, it's all about the people
 
Adaptive Experiences
Adaptive ExperiencesAdaptive Experiences
Adaptive Experiences
 
Promise notes
Promise notesPromise notes
Promise notes
 
SEF for students 2012. Product Management vs. Project Management
SEF for students 2012. Product Management vs. Project ManagementSEF for students 2012. Product Management vs. Project Management
SEF for students 2012. Product Management vs. Project Management
 
What User-Centered Design is Good For
What User-Centered Design is Good ForWhat User-Centered Design is Good For
What User-Centered Design is Good For
 
Midwest UX Mobile Workshop 2012
Midwest UX Mobile Workshop 2012Midwest UX Mobile Workshop 2012
Midwest UX Mobile Workshop 2012
 
LISA 2011 Keynote: The DevOps Transformation
LISA 2011 Keynote: The DevOps TransformationLISA 2011 Keynote: The DevOps Transformation
LISA 2011 Keynote: The DevOps Transformation
 
Critical Thinking forUX Designers (Workshop)
Critical Thinking forUX Designers (Workshop)Critical Thinking forUX Designers (Workshop)
Critical Thinking forUX Designers (Workshop)
 
Ux paper prototyping
Ux paper prototypingUx paper prototyping
Ux paper prototyping
 
Sustainable Theming with Fusion - DCCO 2011
Sustainable Theming with Fusion - DCCO 2011Sustainable Theming with Fusion - DCCO 2011
Sustainable Theming with Fusion - DCCO 2011
 
Designing for Smart Spaces and Objects
Designing for Smart Spaces and ObjectsDesigning for Smart Spaces and Objects
Designing for Smart Spaces and Objects
 
Insider's Technology Guide to Measuring and Improving Your ROI
Insider's Technology Guide to Measuring and Improving Your ROIInsider's Technology Guide to Measuring and Improving Your ROI
Insider's Technology Guide to Measuring and Improving Your ROI
 
Design for Marketing Technical Services
Design for Marketing Technical ServicesDesign for Marketing Technical Services
Design for Marketing Technical Services
 
13 0212 toccon - carpenter altmetrics 2
13 0212 toccon - carpenter altmetrics 213 0212 toccon - carpenter altmetrics 2
13 0212 toccon - carpenter altmetrics 2
 
Design thinking notes with explanation i
Design thinking notes with explanation iDesign thinking notes with explanation i
Design thinking notes with explanation i
 
Design Thinking is Killing Creativity
Design Thinking is Killing CreativityDesign Thinking is Killing Creativity
Design Thinking is Killing Creativity
 
S.P.A.C.E. Exploration for Software Engineering
 S.P.A.C.E. Exploration for Software Engineering S.P.A.C.E. Exploration for Software Engineering
S.P.A.C.E. Exploration for Software Engineering
 
Class 2: Setting the foundation for a successful website design
Class 2: Setting the foundation for a successful website designClass 2: Setting the foundation for a successful website design
Class 2: Setting the foundation for a successful website design
 

More from Andrew Chen

What's Next in Growth? 2016
What's Next in Growth? 2016What's Next in Growth? 2016
What's Next in Growth? 2016Andrew Chen
 
Zero to traction
Zero to tractionZero to traction
Zero to tractionAndrew Chen
 
SaaS products aren't viral
SaaS products aren't viralSaaS products aren't viral
SaaS products aren't viralAndrew Chen
 
Minimum Desirable Product
Minimum Desirable ProductMinimum Desirable Product
Minimum Desirable ProductAndrew Chen
 
Growing Renewable Audiences (presented at OATV)
Growing Renewable Audiences (presented at OATV)Growing Renewable Audiences (presented at OATV)
Growing Renewable Audiences (presented at OATV)Andrew Chen
 
CommunityNext: People not Pages
CommunityNext: People not PagesCommunityNext: People not Pages
CommunityNext: People not PagesAndrew Chen
 
Jeremy Liew - Stanford Facebook Class Presentation
Jeremy Liew - Stanford Facebook Class PresentationJeremy Liew - Stanford Facebook Class Presentation
Jeremy Liew - Stanford Facebook Class PresentationAndrew Chen
 

More from Andrew Chen (7)

What's Next in Growth? 2016
What's Next in Growth? 2016What's Next in Growth? 2016
What's Next in Growth? 2016
 
Zero to traction
Zero to tractionZero to traction
Zero to traction
 
SaaS products aren't viral
SaaS products aren't viralSaaS products aren't viral
SaaS products aren't viral
 
Minimum Desirable Product
Minimum Desirable ProductMinimum Desirable Product
Minimum Desirable Product
 
Growing Renewable Audiences (presented at OATV)
Growing Renewable Audiences (presented at OATV)Growing Renewable Audiences (presented at OATV)
Growing Renewable Audiences (presented at OATV)
 
CommunityNext: People not Pages
CommunityNext: People not PagesCommunityNext: People not Pages
CommunityNext: People not Pages
 
Jeremy Liew - Stanford Facebook Class Presentation
Jeremy Liew - Stanford Facebook Class PresentationJeremy Liew - Stanford Facebook Class Presentation
Jeremy Liew - Stanford Facebook Class Presentation
 

Recently uploaded

AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024Brian Pichman
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTopCSSGallery
 
Patch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updatePatch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updateadam112203
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0DanBrown980551
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud DataEric D. Schabell
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameKapil Thakar
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosErol GIRAUDY
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and businessFrancesco Corti
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Alkin Tezuysal
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...DianaGray10
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNeo4j
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
 
The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)IES VE
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsDianaGray10
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfInfopole1
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch TuesdayIvanti
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applicationsnooralam814309
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIVijayananda Mohire
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInThousandEyes
 

Recently uploaded (20)

AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development Companies
 
Patch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updatePatch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 update
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First Frame
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenarios
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and business
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4j
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projects
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdf
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch Tuesday
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applications
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAI
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
 

Metrics Driven Design by Joshua Porter

  • 1. Metrics-Driven Design by Joshua Porter Twitter: bokardo bokardo 52weeksofUX performable microcopy designing for the social web make them care! by Joshua Porter Saturday, March 12, 2011
  • 3. “Unfortunately for me, there was one small problem I didn’t see back then.” Saturday, March 12, 2011
  • 4. Doug Bowman on Design at Google “ Yes, it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see which one performs better. I had a recent debate over whether a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. I can’t operate in an environment like that. Metrics-Driven Design Daring. Decisive. Conviction. SXSW Saturday, March 12, 2011
  • 5. 41 Shades of Blue Test link color Gmail Google Search 41 Bucket Split Test: ~2.5% of users each got a shade for 2 weeks. Saturday, March 12, 2011
  • 7. #2200C1 #0044CC Google Bing Saturday, March 12, 2011
  • 8. That blue was worth at least $80 million. Paul Ray, UX Manager, Bing Metrics-Driven Design Microcopy SXSW Saturday, March 12, 2011
  • 9. Doug Bowman on Design at Google “ When a company is filled with engineers, it turns to engineering to solve problems. Reduce each decision to a simple logic problem. Remove all subjectivity and just look at the data. Data in your favor? Ok, launch it. Data shows negative effects? Back to the drawing board. And that data eventually becomes a crutch for every decision, paralyzing the company and preventing it from making any daring design decisions. Metrics-Driven Design Daring. Decisive. Conviction. SXSW Saturday, March 12, 2011
  • 10. Spectrum of Design Intuition-Driven Data-Driven Make best-guesses Every design choice is tested Rely on previous experience Takes others experience with a grain of salt Study what others are doing Design is a logic problem Use best practices, principles & patterns Rely on data for decision-making Aesthetics are integral Aesthetics are secondary Rely on our gut Never trust your gut Creative, visionary Cold, calculating Inherently risky Risk-averse Doug’s words: Assumed: instinctive, subjective, daring deliberate, objective, safe Saturday, March 12, 2011
  • 11. Imagine that your design is a mountain... Metrics-Driven Design The grass is always greener on other mountainsides. SXSW Saturday, March 12, 2011
  • 12. Imagine that your design is a mountain... EXISTING DESIGN Metrics-Driven Design The grass is always greener on other mountainsides. SXSW Saturday, March 12, 2011
  • 13. Imagine that your design is a mountain... Now EXISTING DESIGN Metrics-Driven Design The grass is always greener on other mountainsides. SXSW Saturday, March 12, 2011
  • 14. Imagine that your design is a mountain... Optimized (local maxima) Now EXISTING DESIGN Metrics-Driven Design The grass is always greener on other mountainsides. SXSW Saturday, March 12, 2011
  • 15. Imagine that your design is a mountain... Potential Improvement Optimized (local maxima) Now Metrics-Driven Design The grass is always greener on other mountainsides. SXSW Saturday, March 12, 2011
  • 16. Imagine that your design is a mountain... Goal BETTER DESIGN Current Design EXISTING DESIGN Metrics-Driven Design The grass is always greener on other mountainsides. SXSW Saturday, March 12, 2011
  • 17. Imagine that your design is a mountain... Goal Potential Improvement BETTER DESIGN Now EXISTING DESIGN Metrics-Driven Design The grass is always greener on other mountainsides. SXSW Saturday, March 12, 2011
  • 18. Optimization asks: What works best in the current model? Design innovation asks: What is the best possible model? Saturday, March 12, 2011
  • 19. What are Metrics? Saturday, March 12, 2011
  • 20. DEFINITION Metrics are simply numbers that measure the effectiveness of your business. Saturday, March 12, 2011
  • 21. 5 Reasons why Metrics are a Designer’s Best Friend Saturday, March 12, 2011
  • 22. 1. Metrics reduce arguments based on opinion. Saturday, March 12, 2011
  • 23. 2. Metrics give you answers about what really works. Saturday, March 12, 2011
  • 24. 3. Metrics show you where you’re strong as a designer. Saturday, March 12, 2011
  • 25. 4. Metrics allow you to test anything you want. Saturday, March 12, 2011
  • 26. 5. Clients love metrics. Saturday, March 12, 2011
  • 27. PRINCIPLE Your metrics will be as unique as your business. Saturday, March 12, 2011
  • 29. Vanity Metrics Metrics-Driven Design Ego SXSW Saturday, March 12, 2011
  • 31. Interested Trial/beta User Customer Passionate Customer Saturday, March 12, 2011
  • 32. Acquisition Conversion Engagement Satisfaction Interested Trial/beta User Customer Passionate Customer Saturday, March 12, 2011
  • 33. With metrics you are essentially just measuring how well you move people along the lifecycle! Saturday, March 12, 2011
  • 35. CPA - Cost per Acquisition If your CPA is higher than your LTV then you’re in trouble. Saturday, March 12, 2011
  • 36. Acquisition vs. Referral: Dropbox Ran Google Adwords campaigns to drive traffic to dropbox.com The traffic that completed the lifecycle: 1) Searched on a keyword 2) Visited their site 3) Signed-up for service 4) Became a customer cost them $233-$388 per person! (for a $99 product) Metrics-Driven Design LTV = Lifetime value SXSW Saturday, March 12, 2011
  • 37. Dropbox Lifecycle 30 days prior to April 2010, Dropbox users sent 2.8 million direct referral invites. Dropbox’s referral program with 2-sided incentive increased sign-ups by 60% permanently. Metrics-Driven Design Referral = Word of Mouth SXSW Saturday, March 12, 2011
  • 38. Performable Acquisition Metrics Comparative Metrics Revenue by Channel Revenue by Keyword Metrics-Driven Design Microcopy SXSW Saturday, March 12, 2011
  • 40. Trials / Conversions = Conversion Rate Saturday, March 12, 2011
  • 41. Conversion Funnel Analysis A A 100% B B 60% C C 20% Metrics-Driven Design SXSW Saturday, March 12, 2011
  • 42. Sign-up Conversion Funnel Original Flow 1 2 3 4 confirm personal info add your friends invite others getting started Original Conversion Funnel 1 100% of the 100% of people who started the sign-up process, only 2 63% 14% made it to the getting started screen. 3 26% 4 14% Metrics-Driven Design SXSW Saturday, March 12, 2011
  • 43. Sign-up Conversion Funnel New Flow 2 4 add your friends getting started New Conversion Funnel 2 100% of the 100% of people who started the sign-up process, 86% made it to the getting started screen. 4 86% Metrics-Driven Design Removing stuff is a quick way to improve your funnel SXSW Saturday, March 12, 2011
  • 44. Engagement Matters: Twitter http://www.lukew.com/ff/entry.asp?1039 Metrics-Driven Design Speed without context is wasted. SXSW Saturday, March 12, 2011
  • 46. Hits • Page views • Visits • Unique Visitors • Returning Visitors • Registered Users • Customers • Frequency • Time on Site • Daily Active Users Saturday, March 12, 2011
  • 47. Cohort Analysis Engagement over time Month 1 Month 2 Month 3 Month 4 Month 5 Month 6 Month 7 Month 8 Month 9 Month 10 Month 11 Month 12 (Joined in) 100% 20% 19% 13% 13% 10% 12% 11% 7% 7% 7% ? January 100% 21% 16% 13% 11% 9% 9% 7% 7% 7% ? February 100% 24% 20% 17% 15% 13% 11% 10% 10% ? March 100% 31% 27% 24% 19% 15% 12% 12% ? April 100% 31% 27% 25% 21% 18% 16% ? May 100% 39% 28% 24% 20% 19% ? June 100% 40% 33% 27% 23% ? July 100% 47% 41% 32% ? August 100% 52% 43% ? September In this case engagement is improving nicely. 100% 53% ? Of the January cohort, only 20% were engaged in month 2. October Of the October cohort, 53% were engaged in month 2. 100% ? November ? December http://redeye.firstround.com/2008/01/after-the-techc.html Saturday, March 12, 2011
  • 48. Prevention: Facebook Deactivation Design changes to the deactivation page accounted for 1 million members not leaving the service. Julie Zhou, Facebook http://www.readwriteweb.com/archives/what_happens_when_you_deactivate_your_facebook_acc.php Metrics-Driven Design I will miss you after my talk today. SXSW Saturday, March 12, 2011
  • 50. Referral: Net Promoter Score How likely is it that you would recommend our company to a friend or colleague? 0 1 2 3 4 5 6 7 8 9 10 Metrics-Driven Design Do you know what your net promoter score is? SXSW Saturday, March 12, 2011
  • 51. Net Promoter Score How likely is it that you would recommend our company to a friend or colleague? 0 1 2 3 4 5 6 7 8 9 10 Detractors Passives Promoters Score = % Promoters - % Detractors Metrics-Driven Design Do you know what your net promoter score is? SXSW Saturday, March 12, 2011
  • 52. Mint.com & Net Promoter Score “ Maybe we didn’t have a high viral coefficient but we had a great net promoter score. Jason Putorti, Lead Designer, Mint.com Metrics-Driven Design SXSW Saturday, March 12, 2011
  • 54. Emergent Metrics: 5 Friends The magic number is 5. Once a FriendFeed user found five friends, they became active users. Bret Taylor, Friendfeed Metrics-Driven Design SXSW Saturday, March 12, 2011
  • 55. Is there one metric that drives others? “ At Blogger, we determined that our most critical metric was number of posts. An increase in posts meant that people were not just creating blogs, but updating them, and more posts would drive more readership, which would drive more users, which would drive more posts. Ev Williams founder of Blogger & Twitter Metrics-Driven Design Once you’re lucky, twice you’re good. SXSW Saturday, March 12, 2011
  • 56. Principles of Metrics-Driven Design 1. Optimize in small steps; innovate with daring leaps. 2. No design survives contact with the user. 3. Small improvements, taken together, yield amazing results.  4. Testing is empowering, reversion is cleansing. 5. Metrics are not creative: human beings are. 6. All team members are responsible for the user experience. 7. If metrics aren’t actionable, they aren’t useful. 8. Design is never done. Metrics-Driven Design Reminder: Principal = Person & Principle = Thing SXSW Saturday, March 12, 2011
  • 58. This Talk. http://bokardo.com/talks/metrics-driven-design/ Saturday, March 12, 2011
  • 59. My Life. http://performable.com http://oneflightbooks.com Saturday, March 12, 2011