People often ask me how I went from working for the government to fashion. Yes, a pretty crazy story. The truth is simple – transferrable skills. And to be honest I don’t work in fashion – I work in analytics for a fashion company.

What I want to share with the digital analytics community is the most valuable analytical technique I learned while in government. It’s called ACH – Analysis of Competing Hypotheses.

Why do I love it? Often we set out to prove something and then are persuaded by certain key pieces of evidence until we confirm what we believed in the first place (aka confirmation bias).

ACH is the opposite – you set out to disprove something. The strength of this approach is that it is difficult to allow your natural biases to influence the outcome of your analysis. Particularly, in cases where you have missing information.

ACH was first developed by Heuer in the late 70s, early 80s, while he was an analyst at the Central Intelligence Agency (CIA). ACH draws on the scientific method, cognitive psychology and decision analysis. This method became widely available for the first time when the CIA published online Heuer’s now-classic book, The Psychology of Intelligence Analysis, well worth a read for people who work in analytics.

ACH is a very methodical, time intensive process. I have very much adapted it to suit my everyday analytics work, as the full method is not required for all pieces of analysis (see below for the cheat’s version). You can read the full process here. It starts with a question I would like answered.

Moe’s Step-by-Step Guide to ACH for Digital Analytics

1. Brainstorm hypotheses – identify possible hypotheses. Use a group of analysts, product owners and/or marketers with different perspectives to brainstorm possibilities. Be MECE in your approach – mutually exclusive, collectively exhaustive (see The McKinsey Mind for more information).
2. Collect the data – consider what data you need to support and disprove each hypothesis and set about collecting it (referred to as evidence and arguments).
3. Prepare a matrix – Take your hypotheses from Step 1 and the evidence and arguments from Step 2 and put this information into a matrix. The hypotheses are listed across the top and evidence and arguments down the side. Then take the first item of evidence and ask whether it is consistent with, inconsistent with, or not applicable to each hypothesis. Use whatever notation works best for you (C/I/NA – I prefer +, – and NA because this also allows me to add ++ for very strong pieces of evidence). The important step here is to work across the matrix – analysing each piece of evidence against each hypothesis.
4. Analyse the “diagnosticity” of the evidence – identify which items are most helpful in judging the relative likelihood of the hypotheses. If a piece of evidence supports every hypothesis – then it has no diagnostic value. Reconsider or reword the hypotheses if necessary and most importantly, are there any hypotheses missing that should be added.
5. Disprove your hypotheses – you can have all the evidence in the world to back one hypothesis – but if there is one piece of evidence that disproves it – you can reject it. In Step 3, you worked across the matrix, focusing on a single item of evidence or argument and examining against each hypotheses. Now, you work down the matrix, analysing each hypothesis as a whole.
6. Sensitivity analysis – analyse how sensitive your conclusion is to a few critical items of evidence. Consider the consequences for your analysis if that evidence was wrong, misleading, or subject to a different interpretation. Here, it is important to consider risk. If your analysis is driving a $5m marketing budget, make sure you have not misinterpreted any pieces of evidence. If your analysis is a small piece of work, then this step does not require as much rigour.
7. Report your findings – the best bit!  And don’t forget to tell a story with your data…. Focus on the “so what” should we do, rather than “what does this mean”.

Here’s a basic example to get you started (keep in mind this is basic and not MECE):

Question: Why aren’t more mobile site (m.site) users moving to our iOS app?

Hypotheses (across the top):

1. M.site users are not on iOS devices
2. M.site users are unaware of the value add of app
3. There are features on the m.site which are not on the app
4. We are pushing users to m.site (via our marketing execution)
5. Users are accessing content only accessible on the m.site

Evidence (down the side):

1. % of users on iOS devices
2. # of users who use m.site now but have the app in the past
3. Usage of features not on the app
4. Channel information for m.site users
5. # of sessions where content is accessed on m.site, that is not available on app

	Users are not on iOS devices	M.site users are unaware of value on app	Features on the m.site not on the app	We are pushing users to m.site (marketing execution)	Content not on app
XX% of users are on iOS devices	–	NA	NA	NA	NA
XX% of m.site users have not used the app	NA	+	+	+	+
# of users accessing features on m.site	NA	+	–	+	+
Emails link to the m.site	NA	+	+	++	NA
# of sessions accessing content on m.site	NA	NA	+	+	–

For example, with the above matrix, I can remove hypothesis 1 and 3 because they both have evidence which disproves them.

I also have a cheat’s version for when I need the same rigour, but without taking as much time. Here I list each hypothesis down the left and then have two columns, evidence that supports and evidence that disproves across the top. This cheat’s version still allows me to disprove and therefore disregard some hypotheses.

Hypotheses	Evidence that supports	Evidence that disproves
Users are not on iOS devices
M.site users are unaware of value on app
Features on the m.site not on the app
We are pushing users to m.site (marketing execution)
Content not on app