Skip to Content
FeaturesA/B Testing

A/B Testing

Run experiments to optimize your website’s performance.

This feature is available on the Premium plan and above.

Overview

A/B Testing (also called split testing) lets you compare two or more versions of a page or element to see which performs better. Instead of guessing what works, you can make data-driven decisions based on actual visitor behavior.

Experiment reports appear under Experiments in the left reporting menu. Experiments are created and managed under Administration (gear icon) → Websites → A/B Tests.

How A/B Testing Works

  1. Create an experiment — Define your hypothesis and variations
  2. Split traffic — Each participating visitor is randomly assigned a variation (including the original) directly in their browser, and keeps seeing the same variation on return visits
  3. Measure results — Up to three success metrics track which variation performs better
  4. Pick a winner — Finish the experiment and implement the best-performing version

Common A/B Tests for Healthcare

Landing Pages

  • Different headlines or value propositions
  • Various hero images
  • Alternative form placements

Calls-to-Action

  • Button text (“Schedule Now” vs “Book Appointment”)
  • Button colors or sizes
  • CTA placement on page

Forms

  • Number of form fields
  • Form layout (single column vs multi-column)
  • Required vs optional fields

Content

  • Long-form vs short-form content
  • Different content structures
  • Video vs text explanations

Creating an Experiment

  1. Go to Administration (gear icon) → Websites → A/B Tests
  2. Click Create new experiment
  3. Enter the basics: a descriptive name, your hypothesis, a description, and the names of your variations
  4. After creating, open the experiment to configure the details:
    • Success metrics — Up to three, chosen from: goal conversions and revenue, ecommerce orders and order revenue, total conversions and revenue, page views, bounces, and visit length
    • Success conditions — The confidence threshold (default 95%) and the minimum detectable effect you care about
    • Target pages — Which URLs the experiment runs on (include/exclude rules)
    • Redirects — For split-URL tests, the URL each variation redirects to
    • Traffic — What percentage of visitors participate (default 100%), and optionally how traffic is split between variations
    • Schedule — Optional start and end dates
  5. Click Embed code and add the generated snippet to your site
  6. The experiment starts running as soon as the first visitor enters it

Users with write access can create and edit experiments; finishing an experiment requires admin access.

Traffic Participation vs Variation Split

The traffic percentage controls how many visitors enter the experiment at all — non-participants always see the original and are excluded from the results. Visitors who do participate are then split across the variations including the original. You can lower participation to limit exposure while a change is unproven.

Implementing Variations

Each experiment generates its own embed code. Place it in your pages before the trackPageView call so variations activate without flicker:

var _paq = window._paq = window._paq || [];
_paq.push(['AbTesting::create', {
  name: 'AppointmentCTA',
  includedTargets: [{"attribute":"url","type":"starts_with","value":"https:\/\/www.example.com\/appointments","inverted":"0"}],
  excludedTargets: [],
  variations: [
    {
      name: 'original',
      activate: function (event) {
        // usually nothing needs to be done here
      }
    },
    {
      name: 'scheduleNow',
      activate: function (event) {
        document.getElementById('cta-btn').innerText = 'Schedule Now';
      }
    }
  ]
}]);

Split-URL (Redirect) Tests

To test entirely different pages, configure a redirect URL per variation in the experiment’s Redirects section, or redirect from the variation’s activate callback:

activate: function (event) {
  event.redirect('/appointments-v2');
}

Make sure the tracking code is installed on the destination pages. If you implement redirects server-side, always use temporary (302) redirects — never 301.

Server-Side Tests

For backend-rendered variations, choose and persist the variation in your application code, then report it:

_paq.push(['AbTesting::enter', {experiment: 'AppointmentCTA', variation: 'scheduleNow'}]);

Use the variation name original for the baseline.

Tag Manager

You can also deploy the experiment snippet through Tag Manager as a custom HTML tag: set its position to the head start, give it a low priority so it fires early, and trigger it on all pages (plus history changes for single-page apps). Contact support for guidance on implementing your specific test.

Previewing Variations

QA a variation before (or during) the experiment by appending ?pk_ab_test=variationName to the page URL — useful for sharing each variation with stakeholders for approval.

Reading Test Results

Open Experiments in the reporting menu and select your experiment. For each variation you’ll see its visitors, each success metric, and the change relative to the original — with the evolution over time. The winning variation is highlighted once results are statistically reliable, and significantly losing variations are flagged too.

Statistical Significance

Results are evaluated against your configured confidence threshold (default 95%):

  • Significant — The difference is reliable, not due to chance
  • Not significant — Need more data, or the difference is smaller than your minimum detectable effect

Wait for statistical significance before declaring a winner. Ending tests too early is the most common way to reach a wrong conclusion.

Best Practices

Test One Thing at a Time

Changing multiple elements makes it impossible to know what caused the difference. Test one variable per experiment.

Run Tests Over Full Business Cycles

  • Run for several full weeks, not days — traffic behaves differently by day of week
  • Keep the variation count small; every additional variation extends the time to significance
  • There’s no limit on how many experiments you can create, but focus beats volume

Have a Clear Hypothesis

Before testing, state what you expect:

“Changing the CTA button from ‘Contact Us’ to ‘Schedule Appointment’ will increase form submissions because it’s more specific and action-oriented.”

Don’t Peek Too Often

Checking results frequently can lead to stopping tests too early. Set a schedule for reviewing results.

Document Everything

Keep records of:

  • What you tested
  • Your hypothesis
  • Results and learnings
  • Actions taken based on results

After the Test

  1. Click Finish experiment on the report page (requires admin access)
  2. Implement the winning variation permanently in your site’s code
  3. Remove the experiment snippet from your pages
  4. Optionally archive the experiment to tidy the menu — its report remains available via the “Archived” status filter

If the original wins, that’s valuable data too — you’ve confirmed your current approach and avoided a regression.

Privacy Notes

  • Variation assignment happens entirely in the visitor’s browser — no extra data leaves the page just to assign a variation
  • The assignment is remembered in the browser’s local storage (falling back to a cookie for up to 13 months) so visitors get a consistent experience
  • Safari’s tracking prevention clears that storage after 7 days of not visiting, so returning Safari visitors may occasionally be re-assigned
  • To gate experiments behind consent, call _paq.push(['AbTesting::disable']); until consent is given

Limitations

  • Tests require sufficient traffic to reach significance
  • Complex tests may need developer support to implement
  • Results apply to your specific audience and context

Next Steps

  • Goals — Set up conversion goals to measure in tests
  • Funnels — Analyze user paths that tests might improve
  • Heatmaps — Understand current behavior before testing
Last updated on
Media AnalyticsCohorts