We know that it can be difficult sometimes to determine when you should stop your A/B test and when you should let it run for longer to gather more valuable data from your customers to optimize your product discovery experience.

As we continue to build unique features you can use in your A/B tests, our next addition to this powerful functionality aims to help you make that critical decision about when to stop your test. The Minimum Detectable Effect (MDE) chart will show how long you should leave your test to run for in order to achieve the accuracy of results you desire.

The Minimum Detectable Effect is the smallest difference between two versions (e.g. with global rules vs. without, or one recommendations strategy vs. another) that you want to reliably identify as significant. It helps determine whether observed differences are meaningful or simply due to chance, ensuring that your test results are dependable. The lower the MDE you desire, the more data and therefore traffic you need to calculate it.

As you can see in the image of a test report, the longer the test runs for, the more accurate the results get with more data from your customers. While we usually recommend stopping a test once you have achieved 3% MDE, you can now make a more informed decision about whether you are comfortable stopping it earlier for more directional results, or leaving it to run for a greater accuracy.

This new chart will also be available when setting up a new test, so that you can see how long you can expect it to run for before starting it.

For more information about this capability, see this help article about A/B Testing. If you still have questions, connect with your Customer Success Manager or contact us through support@constructor.io.