Reality check?

Auditing is considered an essential part of a safety management system. It appears as the "check" in the plan, do, check, act (PCDA) model used in BS OHSAS 18001:2007 and in the draft standard ISO 45001, and as the "A" in the Health and Safety Executive's HSG65 POPIMAR (policy, organising, planning, implementing, monitoring, auditing and review) schema. BS OHSAS 18001 and the draft ISO 45001, along with the management standards for quality and the environment, use the same definition of auditing: a "systematic, independent and documented process for obtaining audit evidence and evaluating it objectively to determine the extent to which the audit criteria are fulfilled".

Unpacking this definition reveals auditing's strengths and weaknesses. On the one hand, a systematic, independent and documented process allows organisations to measure their safety management systems consistently, with the aim of achieving continuous improvement. The use of audit evidence and independent auditors should minimise bias, and management can consider the results of an audit to gain a balanced view of what they could improve.

On the other hand, can an auditor ever be truly independent? And does fixing the audit criteria in advance mean that auditors are expected to ignore failings or non-conformances if they are not on the checklist?

Andrew Sharman, consultant on safety leadership and culture, summed up his view of audits in From Accidents to Zero (Routledge, 2016), including them in a list of traditional workplace safety measures that "have all had their day".

Auditing alone is certainly not a bulwark against disaster. In March 1999, an audit of First Great Western identified problems with trains passing signals which were set at "danger" and advised a review along a two-mile stretch of track between London Paddington station and Ladbroke Grove. A follow-up audit that September found no evidence of any action to check the signals. The following month a driver passed a signal at danger in that area, and into the path of a First Great Western express, killing 31 people and injuring more than 520 (see the Ladbroke Grove Cullen report at bit.ly/2obOWkJ).

So audits have no magic force if their results are ignored. Nor will they be useful if they are based on the wrong criteria. The Baker Panel report (2007) into the Texas City Refinery explosion in 2005 said of BP's internal audit programme: "The principal focus of the audits was on compliance and verifying that required management systems were in place to satisfy legal requirements." The report suggested BP should have been using audits "to ensure that the management systems were delivering the desired safety performance". In other words, the audits checked that BP was doing what it said it was, but not that it was safe.

Type casting

Audits can base their criteria on a recognised certification standard such as BS OHSAS 18001, guidance such as HSG65, industry advice or, as with BP at Texas City, on internal safety management standards.

All audits will provide qualitative results, identifying areas where there is evidence of non-compliance or at least insufficient evidence of compliance.

There are three types:

• A first-party or internal audit is one commissioned by the organisation, and internal or external staff can conduct it. The report will be used primarily in the organisation, though it might also be used as evidence for second- or third-party audits.

• A second-party audit is required by an external organisation, typically a customer, to check that a supplier meets stated standards. A regulator's audits also fall into this category.

• Third-party audits are commissioned by the organisation but are carried out by an accreditation body (such as BSI, LRQA or NQA).

These approaches can be combined. BS OHSAS 18001, for example, requires that organisations carry out internal (first-party) audits, and the third-party accreditation audit would look for evidence that these are completed and the actions followed through. In other cases, having a third-party accreditation might satisfy prospective customers without the need for a separate second-party audit.

The problem with audit criteria, as illustrated by the BP Texas City experience, is that auditors can look only for what the audit criteria specify. Consultant and trainer Annet van de Wetering suggested a novel approach which she describes as "appreciative auditing" (2010, bit.ly/2lgn5gI). Instead of starting with a set of criteria, she suggests that an auditor asks auditees to describe "the successful moments during which the desired result was achieved". This creates an opportunity for what she calls "co-creation", knowledge sharing about good performance against standards. Van de Wetering reports on a case study of an appreciative audit of patient care in a Dutch hospital, where the processes were seen as "fun, safe and meaningful" and departments looked forward to them.

Getting the same

Daniel Hummerdal, director of safety innovation at Australian consultancy Art of Work, refers to van de Wetering's ideas in his blog, www.safetydifferently.com. He has a list of concerns about traditional health and safety audits, but concludes that most stem from "the belief that productive, efficient and safe work comes from the precise application of standards, best practices and approved systems of work". In other words, he challenges the idea of pre-agreed audit criteria, believing they make organisations "focus on what is an accepted way of showing and fixing compliance, rather than on improving performance of what actually gets done".

He also expresses a concern about audit evidence, pointing out that "the messy details of work at the sharp end are local, contextual, and unique" with the result that audits lead to a disregard for the local specifics. He argues that a culture of innovation is needed. As risk academic James Reason pointed out in his book The Human Contribution (Ashgate, 2008), some heroic recoveries from potential disaster can be achieved through well-established techniques, but success in other cases requires novel adaptations and cognitive flexibility.

Hummerdal backs van de Wetering's approach, giving frontline workers "the trust and freedom to tell any story they wish", and using an audit to "focus on what you don't know" rather than on predefined criteria. In Safety 101 (Mind the Risk, 2016), health, safety, environment and quality specialist Carsten Busch refers to a similar idea as auditing "work-as-done" rather than "work-as-imagined".

Mentioned in reports

In Health and Safety, Environment and Quality Audits (Routledge, 2014) Stephen Asbury quotes senior figures from industry as expecting audits to give "assurance". He does recognise the danger with this attitude, warning that if the members of a management team expect assurance "this is what they are likely to be given".

Having researched many major accidents, including the Piper Alpha and Deepwater Horizon oil rig explosions, Andrew Hopkins, emeritus professor at the Australian National University, believes audit reports should be neither reassuring nor appreciative. An incident that resulted in the deaths of four miners in Australia in 1996 led Hopkins to write Lessons from Gretley (CCH Australia, 2007). Unlike van de Wetering, he argues that audits should be conducted sceptically and set out to find something wrong.

"If the task is to provide some overall assessment of how well the organisation is being managed, the chances are that the assessment will be positive," he says. "Even the most reputable and independent auditors will feel the pressure to provide a generally favourable audit report."

Hopkins suggests the problem is not with the auditors, but with leaders. He recommends that when auditors report all is well, good leaders will challenge these assurances. Asbury echoes this thought: "I believe that auditees should be concerned if there is too much good news."

Hopkins advises changing the criteria so that the onus is on auditors to look for faults. "Auditors who fail to come up with a list of significant concerns have failed their assignment," is his maxim.

Such an approach could exacerbate another of Hummerdal's concerns -- that audits damage trust, but Hopkins has an answer for this too: "If everyone starts with the assumption that there are likely to be significant problems and that it is the auditor's job to identify them, then no one need feel undermined when such problems are duly identified."

Inside and out

What are the benefits (and disadvantages) of using internal or external auditors? When second-party audits are a contract requirement you have to co-operate with whatever sort of audit the prospective client wants to use. Similarly, if an organisation aims to gain accreditation to an ISO or BS standard it will have to use an approved accreditation organisation. Where an organisation does have choices is over first-party or internal audit, whether this is a precursor for a second or third-party audit, or for the organisation's own purposes.

BS OHSAS 18001 makes it clear that an independent auditor does not have to be someone external to the organisation: "Independence can be demonstrated by the freedom from responsibility for the activity being audited", while the draft ISO 45001 is less illuminating, adding only that "an independent process includes provisions for ensuring objectivity and impartiality".

The UK Royal Society for the Prevention of Accidents (RoSPA) lists some pros and cons of using internal and external auditors in an article at bit.ly/2lYgHi4. In support of internal auditors, they will have a better understanding of the organisation, and more freedom to help develop any suggestions for action after the audit. On the other hand, external auditors will have more experience and so a better understanding of the criteria (assuming a standard is being used) as well as, according to RoSPA, being less biased.

Phil Chambers has provided health, safety, environmental and quality support to clients around the world. While working for aluminium maker Comalco -- now part of Rio Tinto -- he was involved with internal and external audits, although he prefers the term "effectiveness reviews". The internal audits involved peers visiting other sites, using criteria developed in-house to review the risk of molten aluminium explosions at sites in Australia, New Zealand and the US. Chambers says of internal audits: "There's a sharing of good practice and an awareness of bad. An internal auditor may suggest that something used at one site may be beneficial at another. Conversely, the auditor may observe something at one site and take the practice home (if it is good) or review how they do something (if it is bad). Few external auditors share experiences and practices."

Elsewhere Chambers has seen problems of lack of honesty towards external auditors. He uses the term "parallel universe" to describe the idealised management systems that some organisations present to outsiders.

In van de Wetering's audit case study, the hospital arranged for staff selected for their personal qualities to receive two days of auditor training. As with Chambers' experience with Comalco, the auditors learned a lot from the best practices in other departments during the audits "to the advantage of their own department".

One size

The success of the internal audits at Comalco was also due to the effort put into developing their own criteria. Sharman admits a particular dislike for off-the-shelf audits, explaining that organisations will suffer when they are "inflicting a programme or system upon themselves that just doesn't fit congruently with 'the way they do things'". Busch warns of over-reliance on ISO compliance, arguing that, though good organisations will more easily pass an ISO audit, possession of a certificate "is no guarantee for good safety management".

Though Sharman is sceptical about audits as the prime means of measuring safety, he does see the value of interdepartmental cross-audits. But even these, he suggests, must supplement the process of looking "for the 999 things that go right".

Management standards allow for organisations to tailor their management systems to their businesses, but there will always be a concern that a standard approach to auditing will miss those hazards or concerns that are not on the checklist. In Risk-based, Management-led, Audit-driven, Safety Management Systems (CRC Press, 2016), Ron C McKinnon reminds us that "what gets measured usually gets done", using this as a positive reason for measuring against standards and quantifying the results. However, stakeholders need to consider -- as BP came to realise -- that the highest risks may not lend themselves to easy measurement.

Organisations should be clear what the choice of audit criteria means for the organisation, and understand the pros and cons of off-the-shelf criteria, such as a standard or industry guideline versus in-house criteria defined in a safety management system.

Busch reminds the reader that auditing is not the only way to achieve the "check" part of the PDCA cycle, and lists some alternative or supplementary approaches to check, including monitoring, evaluations, exercises, tests and investigations. Traditional auditing, particularly after all the work on ISO 45001, is here to stay but it should be supplemented by problem detection, hazard hunting, effectiveness reviews and appreciative inquiries to provide a more complete check.