AB Testing Glossary

Lost in the CRO and UX jargon? We've got you covered! Test It's glossary covers all the A/B testing terms into plain English. From conversion funnels to statistical significance, we explain everything you need to know without the fluff.

A/A Testing: Testing two identical versions against each other.
A/B Testing: Comparing two versions of something to see which performs better.
A/B/n Testing: Testing more than two versions at once.
Above the Fold: Content visible without scrolling.
Alternative Hypothesis: The idea that your change *will* have an effect.
Analysis: Examining data to find insights and understand performance.
API (Application Programming Interface): How different software programs talk to each other.
Average Order Value (AOV): The average amount spent per order.
Average Revenue per User (ARPU): Average revenue generated per active user over a period.
Baseline: The standard performance level used for comparison.
Bayesian Testing: A statistical approach focusing on probability updates.
Below the fold: Content only visible after scrolling down.
Benchmarking: Comparing your performance against standards or competitors.
Bounce Rate: Percentage of visitors who leave after viewing only one page.
Call to Action (CTA): The button or link prompting a specific action.
Cart Abandonment Rate: Percentage of shoppers who add items to cart but don't buy.
Challenger: A variation being tested against the current best (Champion).
Champion: The current best-performing version.
Chance to win: Bayesian measure of the probability that a variation is better.
Click Maps: Visual shows where users click on a page.
Click Through Rate (CTR): Percentage of impressions that result in a click.
Client-Side Testing: Changes made in the user's browser.
Confidence Interval: The range within which the true value likely lies.
Confidence Level: How sure you are about the test results (e.g., 95%).
Control: The original version in an A/B test.
Control Group: The group seeing the original version in a test.
Conversion Funnel: The path a user takes towards a conversion.
Conversion Goal: The specific action you want users to take.
Conversion Rate: The percentage of visitors who complete a desired action.
Conversion Rate Optimization (CRO): Improving your site or app to get more conversions.
Cookies: Small data files stored in a user's browser.
Correlation: Statistical measure showing how two variables move in relation to each other.
Cost Per Acquisition (CPA): How much it costs to get one conversion/customer.
Covariance: Statistical measure of the joint variability of two variables.
Cross-Browser Compatibility: Ensuring a website works correctly on different browsers.
CSS: Code used to style the visual look of web pages.
Customer Feedback Loop: Systematically collecting and acting on user feedback.
Customer Lifetime Value (CLTV / LTV): Total predicted revenue from a single customer over time.
Customer Satisfaction (CSAT): Metric measuring how happy customers are with a specific interaction.
Data Layer: A hidden structure holding data to share with tools.
Event Tracking (Analytics): Measuring specific user interactions (clicks, downloads, etc.).
Exit Rate: Percentage of sessions that ended on a specific page.
Experiment Prioritization: Deciding which tests to run first based on potential impact.
Flicker Effect / FOOC: Briefly seeing the original page before the test version loads.
Form Analysis: Understanding how users interact with web forms.
Frequentist Testing: The common statistical approach using p-values and significance.
Funnel Drop-off Rate: Percentage of users leaving between steps in a conversion funnel.
Glossary
Heatmaps: Visual data showing user interaction (clicks, scrolls, moves).
Heuristic Analysis: Expert review using usability principles.
HTML: The standard code for creating web page structure.
Hypothesis: Your educated guess about what change will improve results.
Information Architecture (IA): Organizing and structuring website content logically.
Interaction Effects: When the effect of one change depends on another change.
JavaScript: Programming language for web page interactivity.
Landing Page: A standalone page designed for a specific campaign or goal.
Landing Page Optimization (LPO): Improving landing pages to increase conversions.
Learning Effect: When returning users perform differently because they adapt to changes.
Macro Conversion: The main, most important goal of your website.
Micro Conversion: Smaller actions leading towards the main goal.
Minimum Detectable Effect (MDE): The smallest improvement you want your test to be able to spot.
Mobile Optimization / Responsiveness: Making sure your site works well on phones and tablets.
Move Maps: Heatmap showing where users move their mouse cursor.
Multivariate Testing (MVT): Testing multiple changes on multiple elements simultaneously.
Net Promoter Score (NPS): Measures customer loyalty ("How likely are you to recommend?").
Novelty Effect: Initial user reaction just because something is new.
Null Hypothesis: The default assumption that your change has *no* effect.
P-value: Probability of seeing the results if there's no real difference.
Page Load Speed / Site Speed: How quickly your web pages load for visitors.
Page View: A single instance of a page being loaded in a browser.
Pages per Session: Average number of pages viewed during one visit.
Personalization: Tailoring website content to individual users or segments.
Post-Test Segmentation: Analyzing test results for specific user groups after the test.
Prototyping: Creating preliminary versions of a design for testing.
Qualitative Data: Non-numerical data explaining the 'why' behind user behavior.
Quantitative Data: Numerical data measuring 'what' users are doing.
Randomization: Assigning users to test groups randomly.
Regression to the Mean: Statistical tendency for extreme results to move closer to the average over time.
Return on Investment (ROI): Measures the profitability of an investment.
Returning Visitor: Someone who has visited your website before.
Revenue Per Visitor (RPV): Average revenue generated by each unique visitor.
Sample Size: The number of users included in each group of your test.
Scroll Maps: Heatmap showing how far down users scroll on a page.
Segmentation: Dividing your audience into groups based on shared traits.
Sequential Testing: Statistical method allowing tests to stop early once significance is reached.
Server-Side Testing: Test variations delivered directly from the web server.
Session: A single visit to your website by a user.
Session Replays / Recordings: Recordings of individual user sessions on your site.
Single Page Application (SPA): A website that loads content dynamically without full page reloads.
Social Proof: Using evidence of others' behavior to influence users.
Split URL Testing: Testing completely different page versions hosted on separate URLs.
Statistical Power: The probability that a test will correctly detect a real effect.
Statistical Significance: How likely it is that test results aren't just random chance.
Tag Management System: Tool to manage marketing and analytics tags without coding.
Targeting: Showing specific content or tests only to certain user segments.
Test Calendar / Roadmap: Schedule for planning and running experiments.
Test Pollution: When external factors or other tests interfere with results.
Test Results Analysis: Interpreting the data and outcomes of an experiment.
Test Validity: Ensuring test results are accurate and trustworthy.
Testing Period: The specific timeframe during which an experiment is actively running.
Time on Page: Average time users spend viewing a specific page.
Tracking Code: Snippet of code used to collect data for analytics or testing tools.
Traffic Quality: Assessment of how relevant and valuable website visitors are.
Traffic Sources: Where your website visitors are coming from.
Treatment: Another term for a variation being tested in an experiment.
Two-Tailed Test: Hypothesis test looking for a difference in *either* direction.
Type I Error (False Positive): Incorrectly concluding there's a difference when there isn't one.
Type II Error (False Negative): Failing to detect a real difference that actually exists.
Unique Visitor: An individual person visiting your website in a given period.
Urgency & Scarcity: Tactics encouraging immediate action (limited time/stock).
Usability Testing: Observing real users trying to complete tasks on your site.
User Experience (UX): The overall feeling a user has when interacting with your site/product.
User Interface (UI): The visual elements users interact with (buttons, menus, etc.).
User Interviews: Directly asking users questions about their experience.
User Journey Mapping: Visualizing the steps and emotions a user goes through.
User Surveys: Collecting feedback through questionnaires.
Value Proposition: Clear statement of the benefits you offer and why you're better.
Variance: Statistical measure of how far data points are spread out from their average.
Variation: The new version you're testing in an A/B test.
Website Analytics: Tracking, collecting, and analyzing website data.
Website Goals: The overall objectives you want your website to achieve.
Website Traffic: The number of visitors coming to your website.
Wireframing: Creating basic structural blueprints for web pages.
WYSIWYG Editor: Visual editor showing changes as they'll appear live.