Regression to the Mean

This is a statistical phenomenon where if a variable is extreme on its first measurement, it will tend to be closer to the average on its second measurement (and vice-versa). In A/B testing, it means an unusually high or low conversion rate observed early in a test is likely to become less extreme as more data comes in. Don't stop tests too early based on extreme initial results!