A/B/N Testing or Split-test
EXP-platform
"Data Trumps Intuition" "significant learning and return-on-investment (ROI) are seen when development teams listen to their customers, not to the Highest Paid Person’s Opinion (HiPPO)", ACM video presentation about Amazon and Microsoft controlled experiments - startling results that were very surprising. Political and economic environment can dramatically affect results, necessary to use A/B/N testing. Approximately 1% drop in revenue per 100mS increase in page load time. Become data-driven, automate, implement abort mechanism.
Newer AB testing video, lots of new information.
Use A/A tests
OEC - Overall Evaluation Criteria
#1 Customer Lifetime Value - not always simple or obvious which to choose
Return visits
Revenue
Conversion
Click-through - is something else being cannibalized?
Time on site
Search optimization
Twyman's Law - Any figure that looks interesting or different is usually wrong
Doug Hubbard's How to Measure Anything
"Data Trumps Intuition" "significant learning and return-on-investment (ROI) are seen when development teams listen to their customers, not to the Highest Paid Person’s Opinion (HiPPO)", ACM video presentation about Amazon and Microsoft controlled experiments - startling results that were very surprising. Political and economic environment can dramatically affect results, necessary to use A/B/N testing. Approximately 1% drop in revenue per 100mS increase in page load time. Become data-driven, automate, implement abort mechanism.
Newer AB testing video, lots of new information.
Use A/A tests
OEC - Overall Evaluation Criteria
#1 Customer Lifetime Value - not always simple or obvious which to choose
Return visits
Revenue
Conversion
Click-through - is something else being cannibalized?
Time on site
Search optimization
Twyman's Law - Any figure that looks interesting or different is usually wrong
Doug Hubbard's How to Measure Anything
See split-test from Lean Startup by Eric Ries
You Should Test That by Chris Goward
You Should Test That by Chris Goward
My disjointed notes
from a couple of read-throughs. Most of this is specific to my company,
and i probably should keep some of it secret. Oh Well
22: Move SEO-centric content below the fold if it's not directly related to the funnel. Use 301 redirects to cover test pages that you take offline.
29: Before and After is not conversion optimization. There are no variables being isolated. The only way to A/B test is with two pages with a distinct change.
32: We should study seasonality, specifically as it relates to new sets. Use google trends to match up search trends with conversion trends.
39: CRO (conversion rate optimization) is the intersection of persuasion marketing, scientific method and experience design. PM = communicate the value prop, build desire and urgency. right message, right people, right time, with clarity. ED = faciliatating the action. seamless, painless, enjoyable signup process. how can we make our signup process a bit more fun. "welcome to the club". SM = isolating variables, not testing too much at once, not changing the control. NEVER change the control.
57: 7 Step Conversion Testing:
1) LIFT analysis of value prop: (Relevance + Clarity) - (Anxiety + Distraction) * Urgency
2) Hypothesis
3) Funnel Experiment Map ( test design doc )
4) Design and Ad Copy
5) Run experiment with a proper tool.
6) Monitor the test to ensure it's running properly. ( do not draw conclusions from early monitoring )
7) Statistical significance -> analysis -> followup test.
63: View data at the page template level ( /category/foo , /tag/bar, /forum/baz, etc ) Run tests on site-wide layout & template changes, it gives us leverage. test sidebars / no, left vs right. In GA: Content -> Site Content -> Landing Pages, filter for these template pages.
104: Relevance - People searching switch from meaning-evaluation method to pattern-matching method. They look for a scent trail based on their search query. They process much faster, but far less. They can't afford to waste time on pages that are not clearly relevant and will click off.
105: Anxiety - What potential misgivings or fears might the user encounter, and how can we prevent or mitigate it? Brand Equity = Credibility. Make sure MTG.GG is clearly associated with QS at launch so it can build on our cred (but take care not to introduce the "finance" anxiety to the new brand ).
105: The tone of copywriting can effect external urgency. Why are people compelled to act now? Theories: they desperately want to be a part of a community, they want to make money, they're REALLY bored at work.
109: Weakness into Strength:
Value Prop: No 3rd party testimonials -> Add testimonials and "as seen on" badges (B&W not color).
Relevance -> Popular inbound keywords in headlines and header text.
Clarity -> Calls to Action above the fold in clear language.
Anxiety -> Short forms with on-page validation, objection busting on-page.
Distraction -> Focus on reducing options to the minimal needed for funnel
Urgency -> Ensure the user knows how soon they'll get access / get replies for support.
115: Value Prop - What are all the elements of my offering and C2A that could be seen as benefits / costs? People make approximations of cost-benefit ratios quickly and logically, and justify the approximation emotionally. The "fuzzy edge".
118: A tangible feature is something that satisfies a prospect's stated need. These are the facts that permit the emotional decision to occur. Descriptive characteristics (not too many adjectives though!) elements of offer, incentives. immediate benefits.
123: Intangible Benefits - Comfort Power Acceptance Freedom Control Love etc. Find and meet unspoken needs.
126: A pretty landing page design from 37signals. Small content boxes to show off and differentiate product lines, a big "hero box" with an image of the book and some credibility testimonials.
138: Price Font Size: People judge savings to be bigger if the sale price is in a smaller font than the original. Try this on the 3mo/12mo form.
139: Anchoring: In which order do people read? Should we show the 12mo first, so people can think " oh at least there's a 1 month (30 day? we should test that!) option ". Anchoring people with higher numbers makes the other numbers seem smaller by comparison.
147: Optimize for Relevance. Prospects are animals hunting for information, entertainment,etc. Animals have refined biological algorithms that evaluate feedback from the environment to confirm they're still on the scent trail and not wasting time chasing wild geese.
150: An inconsistent experience with reduces relevance creates dissonance, high cognitive load, and hurts conversions. Consider the source medium when building landing pages, and understand which kind of hunting behavior comes from which source. Satisfy the scent trail before selling benefits or asking for a sale.
151: Pretty landing page to look at. "Make the perfect movie" - hey that's what I wanna do! What do MTG.GG readers want?
152: Search engine visitors are actively looking for a solution or information to solve their needs. The most prominent message in any landing page should match the search terms closely.
153: Synonyms are invisible. ( do Commander players recognize "EDH"?) Use the language of your customer. I don't think anyone calls themselves a "spike". Find out what language people use to describe themselves, and what they're searching for. Use Google Trends to compare "magic strategy" and "competitive magic", "commander" and "edh", etc.
160: A few lines of code can detect the source of incoming visitors and customize content appropriately.
162: Test the labels used for audience segments (153).
167:How do visitors prefer to navigate? Search? Category/archive pages? Menus?
184: 2-column signup forms dramatically underperform. Stick to one column, with as few fields as possible.
186: Image captions are the most-read text on any page.
187: Images are important when selling intangible products.
195: Under-educated customers need the benefits explained up-front, in clear simple language. 15 interesting tips on this page, the ones that matter:
Highlighted box at the top of the page for C2A "johnson box".
Support claims with proof, testimonials.
Break up copy with subheadings and bullet lists.
Write copy in the active voice.
Adjectives ( and !!!s ) are over-used but the right ones can do some real damage. "Exclusive" is a real good one.
Manage expectations. Clarifying expectations reduces anxiety. How fast can users expect a support response? Is it easy to cancel? When will they get access after paying?
208: Use reminder text and on-page validation for signup forms. Reduces anxiety.
209: Move any optional fields to a post-signup page.
213: Example of how to do "legalese". DO it in a modal dialog so the page flow is not interrupted.
215: Great example of a Wordpress.com signup form. 3 fields, with on-page validation, helper text, managing expectations, urgency.
218: Check GA for 404 pages. Do something useful with them.
219: Reduce page load times. Each second is worth a TON. Allow for useful filtering of content beyond simple category/tag pages.
220: What don't our prospects know, which we presume that they do? Do they know how to sign up? How to find our stuff? Who our contributors are and why to trust them?
223: Fulfillment anxiety - will someone be there to help when something screws up? Make sure they know the answer is "yes, quickly".
226: good example of managing expectations by delineating the checkout process in a nav / status bar. The customer fears post-purchase disappointment more than anything. A guarantee works best as reference info in a secondary location. This should go into a modal with < 1 paragraph, with a link to the full 111% info.
227: Add clear "cancel any time" language in a similar or same modal.
228: Instill a sense of ownership pre-purchase to increase entitlement complex. Build a better cancellation workflow to learn more about why people cancel, when, and how.
229: What is lost by NOT subscribing? People are loss-averse.
232: The first few milliseconds of a page view are spent evaluating the visual layout above the fold. Will they stay or bounce? What will they be looking at? They want and expect visual cues about how the page is organized. The QS home page nails this.
Color emphasis can tie benefits together on a product page. CrystalReports landing page is gorgeous and worth copying / testing.
270: How can we show people that others are using this product without giving away our numbers publicly?
277: Use the thanks.php checkout-finished page to do more things. Install additional tracking, guide them to a "start here" page. Ask for information about the referral.
282: Testing at the template page level is usually about eye flow, layout clarity and the overall value prop. (less)
22: Move SEO-centric content below the fold if it's not directly related to the funnel. Use 301 redirects to cover test pages that you take offline.
29: Before and After is not conversion optimization. There are no variables being isolated. The only way to A/B test is with two pages with a distinct change.
32: We should study seasonality, specifically as it relates to new sets. Use google trends to match up search trends with conversion trends.
39: CRO (conversion rate optimization) is the intersection of persuasion marketing, scientific method and experience design. PM = communicate the value prop, build desire and urgency. right message, right people, right time, with clarity. ED = faciliatating the action. seamless, painless, enjoyable signup process. how can we make our signup process a bit more fun. "welcome to the club". SM = isolating variables, not testing too much at once, not changing the control. NEVER change the control.
57: 7 Step Conversion Testing:
1) LIFT analysis of value prop: (Relevance + Clarity) - (Anxiety + Distraction) * Urgency
2) Hypothesis
3) Funnel Experiment Map ( test design doc )
4) Design and Ad Copy
5) Run experiment with a proper tool.
6) Monitor the test to ensure it's running properly. ( do not draw conclusions from early monitoring )
7) Statistical significance -> analysis -> followup test.
63: View data at the page template level ( /category/foo , /tag/bar, /forum/baz, etc ) Run tests on site-wide layout & template changes, it gives us leverage. test sidebars / no, left vs right. In GA: Content -> Site Content -> Landing Pages, filter for these template pages.
104: Relevance - People searching switch from meaning-evaluation method to pattern-matching method. They look for a scent trail based on their search query. They process much faster, but far less. They can't afford to waste time on pages that are not clearly relevant and will click off.
105: Anxiety - What potential misgivings or fears might the user encounter, and how can we prevent or mitigate it? Brand Equity = Credibility. Make sure MTG.GG is clearly associated with QS at launch so it can build on our cred (but take care not to introduce the "finance" anxiety to the new brand ).
105: The tone of copywriting can effect external urgency. Why are people compelled to act now? Theories: they desperately want to be a part of a community, they want to make money, they're REALLY bored at work.
109: Weakness into Strength:
Value Prop: No 3rd party testimonials -> Add testimonials and "as seen on" badges (B&W not color).
Relevance -> Popular inbound keywords in headlines and header text.
Clarity -> Calls to Action above the fold in clear language.
Anxiety -> Short forms with on-page validation, objection busting on-page.
Distraction -> Focus on reducing options to the minimal needed for funnel
Urgency -> Ensure the user knows how soon they'll get access / get replies for support.
115: Value Prop - What are all the elements of my offering and C2A that could be seen as benefits / costs? People make approximations of cost-benefit ratios quickly and logically, and justify the approximation emotionally. The "fuzzy edge".
118: A tangible feature is something that satisfies a prospect's stated need. These are the facts that permit the emotional decision to occur. Descriptive characteristics (not too many adjectives though!) elements of offer, incentives. immediate benefits.
123: Intangible Benefits - Comfort Power Acceptance Freedom Control Love etc. Find and meet unspoken needs.
126: A pretty landing page design from 37signals. Small content boxes to show off and differentiate product lines, a big "hero box" with an image of the book and some credibility testimonials.
138: Price Font Size: People judge savings to be bigger if the sale price is in a smaller font than the original. Try this on the 3mo/12mo form.
139: Anchoring: In which order do people read? Should we show the 12mo first, so people can think " oh at least there's a 1 month (30 day? we should test that!) option ". Anchoring people with higher numbers makes the other numbers seem smaller by comparison.
147: Optimize for Relevance. Prospects are animals hunting for information, entertainment,etc. Animals have refined biological algorithms that evaluate feedback from the environment to confirm they're still on the scent trail and not wasting time chasing wild geese.
150: An inconsistent experience with reduces relevance creates dissonance, high cognitive load, and hurts conversions. Consider the source medium when building landing pages, and understand which kind of hunting behavior comes from which source. Satisfy the scent trail before selling benefits or asking for a sale.
151: Pretty landing page to look at. "Make the perfect movie" - hey that's what I wanna do! What do MTG.GG readers want?
152: Search engine visitors are actively looking for a solution or information to solve their needs. The most prominent message in any landing page should match the search terms closely.
153: Synonyms are invisible. ( do Commander players recognize "EDH"?) Use the language of your customer. I don't think anyone calls themselves a "spike". Find out what language people use to describe themselves, and what they're searching for. Use Google Trends to compare "magic strategy" and "competitive magic", "commander" and "edh", etc.
160: A few lines of code can detect the source of incoming visitors and customize content appropriately.
162: Test the labels used for audience segments (153).
167:How do visitors prefer to navigate? Search? Category/archive pages? Menus?
184: 2-column signup forms dramatically underperform. Stick to one column, with as few fields as possible.
186: Image captions are the most-read text on any page.
187: Images are important when selling intangible products.
195: Under-educated customers need the benefits explained up-front, in clear simple language. 15 interesting tips on this page, the ones that matter:
Highlighted box at the top of the page for C2A "johnson box".
Support claims with proof, testimonials.
Break up copy with subheadings and bullet lists.
Write copy in the active voice.
Adjectives ( and !!!s ) are over-used but the right ones can do some real damage. "Exclusive" is a real good one.
Manage expectations. Clarifying expectations reduces anxiety. How fast can users expect a support response? Is it easy to cancel? When will they get access after paying?
208: Use reminder text and on-page validation for signup forms. Reduces anxiety.
209: Move any optional fields to a post-signup page.
213: Example of how to do "legalese". DO it in a modal dialog so the page flow is not interrupted.
215: Great example of a Wordpress.com signup form. 3 fields, with on-page validation, helper text, managing expectations, urgency.
218: Check GA for 404 pages. Do something useful with them.
219: Reduce page load times. Each second is worth a TON. Allow for useful filtering of content beyond simple category/tag pages.
220: What don't our prospects know, which we presume that they do? Do they know how to sign up? How to find our stuff? Who our contributors are and why to trust them?
223: Fulfillment anxiety - will someone be there to help when something screws up? Make sure they know the answer is "yes, quickly".
226: good example of managing expectations by delineating the checkout process in a nav / status bar. The customer fears post-purchase disappointment more than anything. A guarantee works best as reference info in a secondary location. This should go into a modal with < 1 paragraph, with a link to the full 111% info.
227: Add clear "cancel any time" language in a similar or same modal.
228: Instill a sense of ownership pre-purchase to increase entitlement complex. Build a better cancellation workflow to learn more about why people cancel, when, and how.
229: What is lost by NOT subscribing? People are loss-averse.
232: The first few milliseconds of a page view are spent evaluating the visual layout above the fold. Will they stay or bounce? What will they be looking at? They want and expect visual cues about how the page is organized. The QS home page nails this.
Color emphasis can tie benefits together on a product page. CrystalReports landing page is gorgeous and worth copying / testing.
270: How can we show people that others are using this product without giving away our numbers publicly?
277: Use the thanks.php checkout-finished page to do more things. Install additional tracking, guide them to a "start here" page. Ask for information about the referral.
282: Testing at the template page level is usually about eye flow, layout clarity and the overall value prop. (less)
Comments