What does being an evidence based agency really mean?

An evidence strategy

Increasingly organisations feel the need to set out their evidence ambitions within a specific strategy, closely tied to the wider organisation strategy and other related thinking and activity. This strategy acknowledges the huge value their access to evidence has, and what a force for good it can be across the eco-system that the agency is operating within. The evidence strategy typically spans a number of key areas, including:

1. A clear and agreed evidence ‘value chain’ – setting out the organisation’s evidence value chain, highlighting the overall approach to managing evidence and data and ensuring that there is a shared understanding of how this value chain supports the organisation to deliver its ambitions and mandate.

One additional benefit of crafting this value chain is that it will flush out critical interpretations on things such as the difference between ‘data’ and ‘evidence’. Most definitions centre around data being raw information with no judgment attached, with evidence being a body of facts and information showing whether a hypothesis is true or untrue. The widespread understanding and acknowledgement of this key differentiation is vital to a successful evidence-based approach.

There are also some key points of principle that should be made explicit within here, eg the importance of always asking the question “what is the perfect evidence we want and need in this situation?”, and then working back from there, rather than starting from what evidence is currently available.

2. Focus and scope – mapping each element of this value chain against the key strategic priorities. This spans both the evidence available and necessary to inform activity or policy, as well as evidence available and necessary to demonstrate impact and progress, for example:

This includes using evidence to stop doing things, as well as starting new things. There is always the risk that evidence leads to more action and more activity. Sometimes the more challenging scenario is where evidence demonstrates that current activity is not impacting in the way originally envisaged and should therefore be changed or stopped.

3. Core assumptions – establish, agree, test and validate the various key core assumptions that underpin the evidence value chain for each element of the organisation’s areas of activity. This is both vital and challenging, with typical examples including:

• “We assume that if we furnish communities with clear and compelling evidence of X, this will lead them to think, act and behave like Z”
• “We assume that if we are actively and aggressively targeting our enforcement activity at X, this will lead to Z impact on the whole eco-system we support”

4. Risk – understanding the risk appetite for each mandate area, and the impact this will have on the strength and nature of evidence required. Areas where there is low levels of risk appetite will require detailed consideration of:

• The depth of evidence required to support policy, activity and decision making, and
• The degree of testing of the base assumptions outlined above to ensure that the risk of a wrong assumption is fully minimised

The concept of ‘certainty’ can also be applied in relation to risk, ie ensuring that the level of certainty an organisation has in relation to a concept or solution succeeding being balanced with the associated risk.

This clarity of risk and the need for certainty (or not) can inform if and when an agency needs to act on a precautionary basis. Being able to respond at pace to a fast-moving situation is often where an agency is most tested. Being clear about the extent to which it can act without having the ideal evidence in place is vital in encouraging and enabling the optimum degree of agility.

5. Design principles – developing and agreeing a set of strong design principles that guide and inform the way evidence will be managed. These need to be brought to life in ways that ensure they are genuinely embedded in the fabric of the way the organisation and its people think, behave and act in relation to evidence and data. So, for example, if such design principles span areas such as ‘transparency’, ‘agility’, ‘fairness’, or cover issues around ‘source validation’ and ‘proportionality’, there must be a shared understanding of exactly what this means in the various contexts within which data and evidence is being captured and applied

As with all strategies, the process of developing them is as important as the document itself. By crafting such a strategy an organisation is ensuring that it genuinely understands what it means to be an evidence-based agency in their own unique set of circumstances, challenges and opportunities.

Having the capability to fully deliver the evidence strategy

If an organisation is to fully realise the impact and benefits set out within their evidence strategy, then it will be necessary to pay attention to a number of key capabilities, some of which are set out below:

1. Governance – there is a clear and obvious relationship between evidence and governance. The above strategy requires the mapping of data and evidence flows, and the associated design principles, against the organisation’s governance structure and regime. Part of this is establishing the need for either ‘opinion’ or ‘evidence’ based decisions. Integral for this is the need for the organisation to make timely and informed decisions through accessing the right evidence whilst retaining the need to balance this off with professional judgment and experience

2. Skills – organisations need to make some fundamental decisions around where their evidence and data literacy skills need to reside. For example, does it want to embed the ability to capture, analyse, understand, interpret and apply data and evidence as a widespread core skill, or through a more ‘centre of excellence’ It is only then that an organisation can map the necessary skills, confidence and access to tools required across the entire evidence value chain for all of the relevant stakeholder groups.

Most of these skills assessments make the clear distinction between technical skills relating to the ‘capture’ and ‘analysis’ activity, as distinction to the judgements and interpretations associated with ‘interpreting’ and ‘applying’ evidence, which as demonstrated by the visual to the right involves varying perspectives and interpretations,

3. Tools and techniques – of these, there are many. What is key to ensure that a) the right blend of tools are selected to reflect all stages of the evidence value chain, b) there is appropriate investment in ensuring that those that need them are fully equipped to apply them well, and c) the organisation continues to evolve its tool set in line with what it needs

4. Culture – in many respects establishing the optimum culture to ensure that the organisation is genuinely and authentically evidence based can be the hardest part. There are a number of key aspects to this optimum culture, including

• Respect – a culture of genuine openness to be influenced by evidence, logic and reason. Agencies are crammed with people with a huge level of commitment and connection with purpose, however this can lead to entrenched perspectives and a motivation to seek out evidence to support an existing perspective, rather than being genuinely guided and influenced by what the breadth of available evidence is suggesting
• Judgement v evidence – establishing and sustaining a culture where evidence and professional judgement and experience are both respected and embraced, and where key decisions can be made and policies evolved through accessing and applying the optimum combination of them both

Dr David Sackett an American-Canadian medical doctor and a pioneer in evidence-based medicine defined it as “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of the individual patient. It means integrating individual clinical expertise with the best available external clinical evidence from systematic research.”

• Avoidance of bias – there is both a ‘process’ and ‘cultural’ dimension to avoiding bias. From a process perspective there is the specific and proactive approach to assessing the areas where bias might exist and countering this with the appropriate depth and visibility of evidence. And there is the cultural aspect, where an organisation encourages and enables staff to be perpetually curious, to foster healthy debate and challenge, and enter into discussions or analysis with a genuine and authentic open mind.

As always, key to establishing and sustaining this kind of culture is through senior leaders acting as genuine role models, but then further reenforced through cultural levers such as internal communications, performance management and the way key processes reflect evidence requirements within them.

5. Performance management – last, but certainly not least, is the role performance management plays in ensuring the right data and evidence is being applied effectively. This operates at two key levels:

• Management information – the key performance indicators the organisation uses to track the impact it is having and the progress it is making against key strategic priorities. By ensuring that these KPIs are appropriately evidence based it will ensure that such evidence is indeed appropriately embedded within the fabric of the organisation
• Individual objectives – employee performance management systems almost universally seek to span an assessment of what impact an individual has had, as well as how they have achieved this. It is therefore possible to fully embed the aspirations an organisation has for being evidence based within the ‘how’ aspect of each individual’s objectives

One key element of performance management is the selection of the right data sources – achieving the optimum blend of data and evidence sources is both an art and a science. For example, how to ensure that when exploring the impact of existing agency policies it is possible to access and interpret large banks of quantitative data (primary evidence) alongside what is gleaned from small but highly insightful groups with genuine ‘lived experience’ (secondary evidence.