Professional Documents
Culture Documents
Improvements
Michael Monaghan
If you came for stuff about APIs, a few of
my best examples were shut
down/deprecated. Ill be giving another
talk or writing a how-to about it!
1.Overview
How To Run A Test
Stats 101
Pitfalls
Rules & Best Practices
TOOLS
Q&A
Overview
Who I Am
Lived abroad for a while, came back to go to Uni... still need to do that.
You send emails as part of your day-to-day business. Its a unit of work.
Youre adventurous enough to explore some tools (its tedious and unreliable to
gather data manually).
What is AB Testing?
AB Testing, aka Split Testing or bucket testing, is comparing two versions of
something to see which performs better.
Where Can I Use it?
Emails
Phone Scripts
Job Descriptions
Landing Pages
Event Invites
Interview Processes
Why should I be A/B testing?
1. It helps you realize the actual impact of proposed changes
2. It lowers risk and helps you AND OTHERS trust your data - less time arguing
on executing and more time actually doing
4. Probably cheapest way improve conversions - every email (or phone script, or
job description) is valuable, make the most of it
Key Terms
- Conversion Metric - The metric that defines what you want or are asking
people to do.
- Ie. For Emails, a possible conversion would be a reply. It could also be to click on a link.
Not HIPPO
3. Do you know who didnt even open your email (as far as you know)?
Survey Data
Not the right time to contact me, I just started
a new role
(Poor Sourcing/Research)
Analyze & Ideate - Qualitative Data
Heat Mapping
Are people reading the PDFs I send? How
much are they reading?
Form A Hypothesis
IF [variable] then [result], because [rationale]
Result (Predicted outcome) = Use data to determine size of effect (10% more
responses = X more phone screens based on prior numbers)
Strong Hypothesis = If we personalize the first sentence of the email past the
salutation, response rate will increase by 10% because candidates will see that
were doing basic research on them and thus, our outreach is likely to seem more
relevant.
Form A Hypothesis
If you dont really have any idea how big an effect your changes will have, but
youve figured out where you might need some work:
Go for Confidence
There are already enough variables we cant control, lets limit or guesswork or
shotgun approach as much as possible.
- Put constraints on your problem so its easier to solve.
Interpret Results
Interpret Results
Stats 101
Statistical Significance and Why Its Important
We want to trust our numbers.
Already enough variables in this world, lets grab hold of what we can.
Statistical Significance and Why Its Important
[Ive never done more than a bit of reading about the following]
If you want flexibility and are good at math (or know someone who is):
Sequential Experiment Design - Lets you set up points in advance where you
will decide whether or not to continue and it gives you the correct significance
levels.
Bayesian Experiment Design - You can stop your experiment at any time and
make perfectly valid inferences.
Rules & Best Practices
Document and Share your findings!
Track your results! Most basic, keep a spreadsheet detailing experiments, what
was changed and how it was changed, as well as metrics.
Test frequently.
SS at least 95%! 99% is where you SHOULD be at, if you have the volume.
http://www.yesware.com/blog/best-time-to-send-email/
Pitfalls
Pitfalls