Experiments let you test multiple variations of a prompt, such as copy, design, triggers, and actions, to find the version that performs best against your conversion goals.

Available on all Recurly Engage subscription plans

1Definition 2Key benefits 3Key details

Prerequisites

You must have Company, App Administrator, or App Member permissions in Recurly Engage.

Limitations

Only one active experiment can run per prompt at a time. Historical experiments remain accessible.
We recommend a minimum of 30 users and five conversions per variation for statistical reliability.

Definition

An experiment divides traffic among a prompt's variations, including an optional control group, and measures conversions with statistical tests to determine a winning configuration.

Key benefits

Data-driven optimization Use real user interactions to choose the best-performing variation.

Controlled testing Isolate single changes, such as title, imagery, or behavior, to understand their impact.

One-click rollout Promote the winning variation to replace the original prompt when the experiment ends.

Key details

What experiments can modify

Prompt title and message body
Call-to-action text and behaviors
Images, styling, and layout
Triggers, schedules, and actions (including 1-click workflows)

Traffic allocation

Assign any percentage of visitors to each variation and to a Control group (users who see no prompt). Control group users are still measured for conversion against your custom goal.

Statistical analysis

Experiments use a Z-test to compare variation conversion rates against the control. To detect a meaningful lift (for example, an improvement greater than 5%), each variation should see at least 30 users and five conversions. Depending on your baseline rate, that often means hundreds or thousands of users.

When statistical significance is reached, select Use This to end the experiment and update your baseline prompt to the winning variation.

NoteZ-test significance indicates superiority over the control only. It doesn't compare variations against each other. We plan to support Bayesian methods in the future.

Create and run an experiment

Select a prompt

Navigate to Prompts and select the prompt you want to experiment on.

Start a new experiment

Scroll to the Experiments section and select

New Experiment

Name the experiment

Enter a clear experiment name.

Add a control group (optional)

If you have a custom goal configured, add a Control group. It measures baseline conversions without showing a prompt.

Add a variation

Select Add variation, name it to reflect the change (for example, "New headline"), and modify the title, copy, imagery, triggers, or actions.

Configure the variation

Configure the variation details by editing directly in the prompt editor.

Allocate traffic

Allocate traffic percentages to each variation and the control, making sure they total 100%.

Start the experiment

Select Start experiment and confirm to begin dividing traffic.

Monitor the experiment

View users per variation, conversions, and conversion rates in real time for in-progress experiments.

Promote the winning variation

When a variation demonstrates statistical significance, select Use This to end the experiment and promote that variation as your new baseline.

Edit live experiments

You can make minor updates to a running experiment's variants, such as copy changes or action updates, without stopping and restarting the experiment. This is useful for minor adjustments that are unlikely to affect the experiment's core metrics.

To use this feature, enable the is live editable option when you set up a new experiment. This setting is disabled by default.

Once the experiment is live, you can edit all editable variants, including the original variant's triggers and actions, directly from the main prompt screen.

Experiment reporting

Experiment reporting lets you download a comma-separated values (CSV) file with detailed data from your experiments, so you can analyze and report on results.

Data availability: Export experiment data for a specific time frame from the Settings section of the application. The export includes data for currently running experiments and for any completed experiments that overlapped with the selected date range.
Data content: The exported CSV file includes only experiment-specific data, not general prompt data. For completed experiments, the export provides the total stats for the entire experiment run. For running experiments, the stats are scoped to the specified time frame.