Having heard it described as ‘unnecessary admin’ more than a few times, I can tell you more than not the importance of the test plan itself is often overlooked.
But when it comes to test planning the devil is truly in the detail.
You may be excited and have a great idea, but to execute it successfully needs careful planning where all the finer details are recorded.
Even if your idea is fairly simple, you’ll need to think about all the details of how you’ll execute it. What will your idea look like on the site? Which page or pages will it show up on? How will it work? When do visitors enter? Who qualifies? What device and browser combinations will you target? How will you track it?
If this information isn’t pulled together somewhere you could find yourself pouring effort, time and resources into a broken test which will not only ruin your chances of success but you’ll also miss out on the learnings and results you truly desire.
Why people don’t document their tests
We don’t need to as we know what we’re doing
If you’re a one person team and you’re excited about an idea you just go and do it! You know what’s happening but you might not be thinking about the whole process and the future.
You might not care about a test plan being recorded but what happens in a year when you want to look back and you have nothing to look back at?
In a larger set up, the person planning your test will likely be different to the one designing it, and the one building it - so if multiple people are involved they all need to understand the changes.
If you hire an agency to run testing they will create your test plans for you so you may feel there’s no point in having them yourselves but you should always have a back up copy.
It takes too long
A test plan looks like a simple document to fill out but the time it takes can vary - even a simple test plan can take half an hour, whilst some can take 4 hours!
An easy one could be a simple copy change on one page where there’s not much to worry about in terms of it breaking.
A longer one for example could be restructuring the way you display your products across multiple devices.
There’s so much detail
Having worked with a number of clients on their test plans over the years we often hear ‘wow there’s so much detail’ - even on the most basic of test plans!
There’s almost an expectation that just noting down what the change is/what you’re testing is enough.
Even with a small test there can be so much detail, it can feel quite daunting! But you do need to cover all the necessary details for executing the test for example, how to enter the test, browser & device segmentation and any possible integrations.
We don’t know what to document
If some of your thoughts aren’t totally crystalised for example you don’t have the test URLs yet, that missing information might stop you from completing your test plan.
And you might be unsure of who needs what! If you’re a one man team doing the plan you might struggle to think about what the dev team needs. The dev team may have some idea of what’s going to change but they might not know if the test is happening on one page or multiple pages. It makes sense for both teams to review the test plan and check if anything has been missed.
Why you always need a test plan
1) It provides an easy overview for stakeholders
What's often missed is that the test plan is a simple and effective way to highlight everything that is going on and answer some of the most asked questions.
Figures have gone down, did we test anything?
Figures have gone up, why did we make more money?
Loads of people might be interested so a test plan is crafted in a way that their questions will be answered easily. Especially with the layout we use - it’s all presented in a way you can have a quick look.
If you’re the designer and just want to see the changes it’s easy. If you’re the dev you can see all the technical aspects in one place. If you're interested in monitoring you can look at the conversions section.
Even for external stakeholders like us. It’s important for us to see what you’ve been testing before so we know what’s going on.
2) For future you and future testing
When conducting any kind of performance reviews you might just want to check back. If you notice revenue spikes you can look back at your test plans and see what might be the reason.
There are definitely tests we’ve run that aren’t dissimilar to previous ones. For example we’re running a test where we added item imagery on PDPs - that test was run similarly earlier this year, and that previous test plan came in massively handy for the current test. We’re able to look back at the previous test and see how we built it, what details we needed, how it will work and it gives an idea of conversions.
You might just want to go back to see what you’ve done. If you have a recent test that didn’t work so well/was inconclusive, going back and seeing what changes made the difference could help.
With the maturity of your CRO programme it can sometimes feel like you’re testing in the same places all the time so this allows you to look back and see what you’ve tried and areas you maybe haven’t done much on at all.
3) To break down silos and share learnings
You may test in areas that other teams have special interest in. Keeping clear documents of your testing makes your work accessible to others and allows you to build good working relationships.
This could be both positive and negative changes - if something really tanks you would want to avoid doing it elsewhere.
Your work also can affect other areas of the site - another team could be seeing negative impacts and not understand why.
Say you introduce a guest check out feature and you have a team that looks after this - and they see they’ve lost a load of sign ups which could see less returning visitors - they would have no idea why because they weren’t privy to the plan.
There is a risk of colliding with content changes that are possibly coming up when you’re testing so clear test plans are vital as to not compromise each other’s work.
In larger companies where some tests are reruns on other brands, you can share your test plans and show what you were measuring and how you ran those tests.
4) To help you pinpoint what to track
When you’re building out your idea your test plan forces you to look at all the details and things you didn’t think about. As you go through each step of your test plan you can really think about what needs to be tracked. For example, if you were changing a banner, when you see it you realise there’s a cross to close it and you now think ‘ah ok I’ll note that down too’.
In the review stage of a test after it’s live, when you’re monitoring you might not remember what exactly you named your conversions. So you can go back to the test plan and look at the conversions section and you know exactly what you were tracking!
5) To know which changes are successful - and which are unsuccessful
It’s here in the test plan that you define what success looks like. So when you are doing your analysis and reviewing your test results you can look back at your test plan and add those metrics in as they are the ones you are most interested in.
We usually add 3 main metrics in our test plans - primary, secondary and tertiary. Your primary is most often sales - do they go up or down? Some people list the change that's being made as their primary metric. Your secondary and tertiary metrics could be add to baskets and product clicks for example.
6) Because people come and go
When you have someone new join the team, having test plans is good to catch them up on previous work you’ve done. They also may come in and suggest ideas that have already been tested so at least you won’t duplicate old tests!
When someone leaves or even goes on holiday they could be half way through a test!
There’s times when someone might need to just finish a test plan, review a test or do the analysis so it needs to be documented and accessible for them to do so.
Ways to help your team with test planning
Train the team on how to complete a test plan
Initially do a walkthrough with a realistic example. Then give them a dummy idea/ or old test, give them the designs, the details and ask them to create a test plan. You have something you can compare it to once they’ve filled it in then.
Sometimes there’ll be things you haven’t thought about so it’s good to share your ideas and see what’s missing. Then ask them to do the next one that comes up that's real and you help and feedback.
Encourage them to play with the website
Playing with the website is a key part of crafting the test plan so you can see what might be impacted by your changes and what needs to be taken into consideration. You’ll miss bits in your plan if you don’t do this part.
The breakdown of the plan:
1) Understanding of the test on a conceptual level
2) An idea of how it's going to look
A key part is to play with the website itself. What happens when people scroll for example - what should disappear, stay, change? Does a new tab open and visitors leave the previous part of the journey? Check the responsiveness of the page.
Find out all the ways you might be able to break the page - that can help you think of every scenario to cover off in your plan and with your QA scenarios.
Make sure everyone has access
Part of the training should include making sure everyone who needs access has it - anyone who’s part of the process to avoid a lot of fiddling about later.
Have a platform everyone can access like a SharePoint or Confluence page - this will become increasingly important as your programme grows and you have more interested parties.
Need a test plan template? Here’s ours :)
Hopefully I’ve convinced you of the importance of documenting your tests for the success of your testing.
Now you might be wondering “so what goes into a test plan, please can you show me?”
This post here gives you some further guidance on planning your tests.
You can download our test plan template, along with some test plan examples (for ecommerce and lead gen) to help you with your test plans.
Happy test planning!
.png)
