Steps Eight to Ten of the CRO Process: Serve Optimal, Full Analysis and Recommendations
Step 8: Serve Optimal
When you are sure that your test can be concluded, there's going to be an outcome which is going to be either that the test is a loser, in which case obviously, the control version should remain in place, or another outcome could be that the test has won or indeed has done no harm and you may want to rollout the feature that you tested.
Now, this is not always possible to be done through your development team maybe because they are busy, there's a long road map and the backlog of things to do. And in a case like this most CRO platforms will actually offer you an option which is to basically serve the optimal.
So what we call the optimal is essentially the winning content of the test and by serving it we mean essentially having that optimal content published on the platform from which you run the test. Which means that it will be showing this time to all of the traffic that you are targeting without the A/B testing element.
Limitations of serving optimals
Now importantly, there are limitations to this. The test and therefore the optimal only show if users accept the right cookies. Another limitation is that you may have changes to the base code of the page which means that if your control version changed in any way during the test you may have had a broken experience and then the same goes for your optimal being served. And obviously these tend to be served for longer periods of time. So as much as you can use your CRO platform in order to serve that content, ideally what you want is to move forward into the hard coding part of it and get that done as soon as possible. Essentially use the platform that's been tested with as a stop gap until the changes can be in your base code.
Now, an example of why optimals need to be particularly maintained if you are going to serve them through the CRO platform. Let's imagine that you’ve got a jigsaw sort of diagram with the existing base code at the bottom of the test itself and the optimal. Let's add new test code on top of that.This is what is tested and potentially what has won and that gives us a new experience.
But if for any reason over time your base code changes it's going to be problematic potentially because it could create a broken experience. Now you may be lucky, maybe the control page will change and the areas that get changed do not affect the optimal that happens. But you could also be unlucky and have these areas affected and the optimal breaking very badly. And because people tend to look at Optimals less often than they do at tests, it could be a while before you realise that this is broken. So the maintenance aspect of optimals and being aware of the fact that they are not meant to be there forever through the CRO platform is very important.
Most of the issues that we see with live tests are actually caused by this structure. I think it comes down to where the process is hugely important again, for that reliability element. Without process, your own developers may not know what tests you've got live or what optimals that you've got live. And what you don't want is to receive that message saying - Oh we've updated this page and the whole thing is now a crock, what's happening?
If you've got that process in place, you know where your tests are running, you know where your optimals are running. It means that you can pre-empt any issues and make sure that before any changes are being made to base code, you understand the impact that that might have on the optimals or tests that you have running.
Step 9: Full analysis
Once the optimal is live, we're then moving into one of the final stages of the process which is the full analysis of that test.
Review all data
So in this we are looking at all of the data that was collected within that test, regardless of whether it seems relevant at the time or not. I have lost count of the number of times over the years that I have adversely affected a metric I didn't intend to have an impact on whatsoever. One of the things that we say to our clients quite frequently is if we had all of the answers we wouldn't run tests - we’d just tell you what to do and we'd be right all the time. That's never the case. And there will be times where you have had a knock on impact on a metric that you weren't necessarily meaning to and therefore reviewing the data as a whole is hugely important.
Include test details and outcomes
Within that final analysis document, you also want to detail clearly what the test was and the outcomes of it. This again, is for future reference. This document should be suitable effectively as a standalone piece. You should be able to open the document and understand all relevant information about what was tested and why and what the outcomes of that were.
Looking forwards
And then lastly, an analysis may seem like the end of a process, but for us it's not just about looking backwards, it's about looking at what has happened and determining what we could or should or might do nex toff the back of the information that we receive. Because again, having this process, this is about repeatability. How can we continue to improve and making sure that your analysis looks forwards, not just backwards, is a big way of ensuring that the cyclicality that Aline talked through earlier, that that is achievable.
So, within our full analysis document we have three main sections:
Nut and bolts
Again, you've got the nuts and bolts element. In here you are repeating the background that you have, the hypothesis that you have and the key measures of success that you had for that test.
Test summary
In the summary, again, you will show what the page looked like pre-test then the variant or different experiences that you are creating visually and then I guess a headline view of the primary, secondary and tertiary metrics that you had. Those are things that generally speaking are going to determine whether the test is deemed as successful against those metrics or not.
Metrics
But then in the remaining metrics section, we've then got a summary of performance of all metrics within that test comparing control and the experiment. As I said, there may well be elements within there that you did not foresee and they'll be hugely important to what is, I would argue, the most central element of our full analysis, which is the recommendations that we are going to make from this data, for future tests that we want to run.
You may see there's a lot of elements that come from the test planning here and you may think that there's a lot of needless repetition. But it is important to realise that the full analysis is likely to go into more hands than the test plan itself, quite often into the hands of people who had no idea that this test was running. So it is important to have these elements in there so that they know what the idea behind the test was and what we tested. There will be a lot more people who care about the outcomes of your test than the inputs of how did we get here in the first place, yes absolutely.
Step 10: Recommendations
What you want to ensure is that you have a certain thought process in place because what you do not want is for anybody to be able to question your recommendations. So the way that we look at it, from our perspective, we always start with data. So insights essentially are key, but if they are not backed up by data, you might as well not have presented them at all.
D > I > R flow
So we have what we call a D > I > R flow. So starting with the data, the first question you can ask yourself is what happened factually? Then out of this factual discovery, you're going to infer essentially why you think it happens so that's your insight part. And then when you have made that inference, you can then start thinking about a way to potentially combat the insight if it was a negative outcome or indeed perpetuate the positive impact, if it was positive. So what can we do now with that insight, is going to be recommendation part.
It is very important to go through this.The data, as I said, is key. And it is because you want to avoid bias as much as possible, especially if you had a lot of involvement in the planning of the test and maybe it was your idea. You're going to have a bit of a natural preference for what you think should have won and it's easy, even with data, to dissuade into thinking that something performed better than it did. But if you look at it from a stone cold objective data perspective, it will help and it also will help to uncover some data gems.These are the things that potentially you didn't think you would influence, but you track them anyway and all of a sudden you find that maybe there was a big trend in there that’s interesting to look at. If you do not start from the data, you wouldn't see it essentially. So again, important to start with the data.
D > I > R flow Example
Now, let's look at this with a fake real life example. Essentially we have a test background here of a test that runs on a funnel. There is higher abandonment in that funnel, particularly on a page that contains add-ons. So the test rationale is that we want to remove that page altogether from the funnel. Now, the hypothesis would be that by making the funnel essentially shorter, the visitors will have a faster experience with it and they will proceed to purchasing faster and therefore they will abandon less, which as a consequence will increase transactions and revenue. This is the basis for the test.
Now, let's imagine these results. Data wise, we found that transactions have increased but average order value has decreased. What do we think happened there? Well, we think that maybe removing the add-ons page simply made visitors less likely to buy upsells because they didn't find them where they would have normally.
Now, what can we do about that? Because essentially here we have a bit of a double sided result. We have transactions that have improved, but the amount of revenue you get out of them has decreased, which is the bit that we really want to combat.
Well, one logical recommendation could be that you realise that upsells are important and add-ons therefore are important. So maybe you should show them again, but maybe not on the page separate from the rest. Maybe you could actually show these upsell features within an existing page of your funnel, like the basket. That way the funnel remains short, but you do not lose the benefit of having the content that seems to influence average order value.
So you see, this is basically the kind of reasoning that we're looking for. We're looking at the problem from a data perspective and quite often you will find that some trends actually conflict with each other and essentially the insights and recommendations they're there to try and understand why that conflict exists and to find a solution to actually improve the situation.
So what now?
Look at what you're already doing. There's a high probability that you've already got more process than you think you do. What we find talking to people who are effectively starting afresh is that when you ask what process have you got, the immediate answer is kind of like a nervous laugh and nothing. And then when you get into how do you handle this? There is a way that they're already doing it. So a lot of the time you're selling yourself short.
Look at what you're already doing and whatever that is, start to write it down somewhere, anywhere. That gives others who have a stake in the programme to see and review what it is that you are doing and possibly even feed in to how you're doing things, explaining it from their perspective. CRO programmes always involve a diverse number of stakeholders with differing understandings of how the process works, different technical levels. Having those things written down will really help to hold those conversations with people from different areas of the business.
And lastly, whatever you document now, don't worry about it too much in terms of its quality or its detail, because it will not stay that way. I've lost count of the number of times that our process has changed over the years. I guess the core cyclicality of it has always remained, it's something we've always believed in but certain elements of that process have been adapted and improved over the years. As with anything to do with CRO, this is about progress forward it's not about perfection. Just get it written down, get it shared with people, and you will start to build the kind of progress that will give you the speed, the reliability and the repeatability of your programme.
Related resources:
Webinar: Our CRO Process in action: 10 steps for repeatable success
The CRO process: four steps to take before your test goes live
The CRO process - three steps to take during the testing phase
.png)
