To determine how long you need to run a landing page test for valid results, use a handy sample size calculator.
How absolutely certain do you want to be that the difference in conversion rate between page A and page B is real? That is, it was not caused by random events like a cat falling asleep on a keyboard and accidentally submitting your web form.
Let me suggest to keep it simple. There's no need to get into all the statistical theory behind things here. Because we are measuring marketing conversion rate and not death rates of a new medicine, the risk of being wrong doesn't come with intolerable consequences and is outweighed by getting answers quickly.
I use the "good enough" method. Here's my rule of thumb for setting tolerance:
(I'm x% sure the results are real.)
99% Super conservative, takes longer.
95% Good enough. Sweet Spot.
90% Fast & loose, quick results.
(How much relative change in the conversion rate matters.)
10% Small lift. Even worth it?
20% Good enough. Sweet spot.
25% Big lift.
In other words, at the end of the test you will be able to say: I'm 95% sure that there was a 20% lift in conversion rate (or there wasn't).
*Note that "lift" means a relative change. For example, a 20% lift of a 10% conversion rate is 12%. (It is not 30%. You do not add 20%).
There are a variety of sample size calculators you can use to determine how much traffic you need to each landing page. Here are three that I like:
Or you can use this quick chart for an estimate:
Keep in mind that this is traffic needed to EACH variation of your landing page. If you are testing your existing page against one other page, multiple the traffic by 2.
If your total sample size needs to be 1,000 total visitors and your daily traffic is 100 visitors, then you need to run your experiment for 1000/100=10 days.
Many A/B testing tools will do the calculations and let you know when you have statistically valid results based on your tolerance assumptions. It is good to know ahead of time how long to expect a test to take.
If you already have done a test and are wondering if the results are significant or to what level of confidence you can have in your results, you can use this A/B Testing Significance calculator.