Print this page

Key Performance Indicators

 

Key Performance Indicators in Public Service Broadcasting
and Possible Benefits to CBA member Stations in Moving Together

by Anthony Rendell

Up until the 1980s public service broadcasters used performance indicators, without calling them that, mainly about inputs. The emphasis was on showing economy in budgets and staff numbers. Change in the public sector around the world introduced many indicators about the efficiency of internal processes. By the 1990s the emphasis was on outputs and outcomes to show effectiveness. Key performance indicators (KPIs) are seen as essential in answering the question, what are we taxpayers getting for our money?


Now, with ideas like the Balanced Scorecard, there is interest in going further. At this point, some members of the CBA see the desirability of establishing industry-wide indicators, so that any organisation in the CBA can make sensible comparisons with other members. Some members would like to explore benchmarking built on an industry-wide data base. This paper, which concentrates on KPIs, was commissioned by Elizabeth Smith, Secretary-General, as a first step. It is based on a reading of KPI documentation from the ABC, BBC and CBC, and personal experience. It has no answers, but the hope is that it will help interested members by being something to focus on as they discuss how to carry their project forward.



*
What do Public Service Broadcasters use KPIs for?



Public Service Broadcasters (PSBs) appear to use KPIs mainly for the following reasons.


These tend to change in order of importance depending on circumstances, such as imminent funding negotiations (is the ultimate KPI the size of the next funding settlement?) But overall, KPIs are seen as necessary for PSBs to show themselves and especially outsiders that they are monitoring their processes and monitoring their impact, doing good for the nation, and doing it well.


Common concerns among CBA members


KPIs are about the objectives which are most important or of most concern. Although there are obviously differences across CBA countries in things like structures, funding arrangements, government policies, social preoccupations and competition, their PSBs have many concerns in common. These probably arise not so much from their current broadcasting legislation as from deeply held feelings that Reith’s ideas were right.



Despite different societies and different arrangements, PSBs in the CBA have sufficient things in common to develop a set of common KPIs. There would be clear advantages in being able to point to world-wide industry standards and common best practice. There would be a temptation for any CBA member to use material to tell its own government how much better it is than another CBA member, but that has certainly happened before anyway.


Each would undoubtedly need some KPIs specific to its own circumstances. Each has something unique to cope with – call it the Quebec syndrome.



What is this PSB for?



Even so, PSBs have a problem right at the start in defining their basic purpose, “what are we really here for?” However the answer is worded, it implies making a difference to people’s lives, and involves intangibles which are difficult to measure.



In theory, the starting point should be the statement of the PSB’s purpose set out in the Act or Charter. In practice, these are not always concrete enough to be directly helpful. The BBC Royal Charter refers to information, education, and entertainment, and the Licence tells BBC World Service to “plan and prepare its programmes in the national interest.” Over the years, the BBC has filled in the next step by writing its own objectives (under whatever name), but these too have contained many abstractions.



The old ABC Act said the ABC was to provide “adequate and comprehensive programmes.” The present one says it must provide “innovative and comprehensive broadcasting services of a high standard”. Voice of America must “present a balanced and comprehensive projection of significant American thought and institutions.” CBC should “build mutual understanding among Canadians” and “enhance pride in Canada”. Radio Canada International is charged with “further developing international awareness of Canada and the Canadian identity”. Legislators have become increasingly keen on getting into PSB statutory documents ideas about nation-building, diversity, culture, national identity, and so on. Many of these things are hard to define, and impossible to measure quantitatively.



So PSBs start off with obligations which cannot be measured by the standard financial measurements used by commercial organisations, because they are about intangible public goods, unquantifiable, and not possible to measure directly. But they are the important things for which they exist. PSBs can no longer say “we are achieving these noble aims, and you can believe it, because we are rather good at writing flowing essays on our achievements.”



Many government departments have the same problems. Take the Navy. It is often discussed and funded as though its purpose is to maintain and operate warships, which are very quantifiable and tangible things. Actually, its purpose is deterrence, which is intangible. If that fails there will be war, and then we expect victory. So the final outcome we want from the Navy is not ships, but something better defined as peace or security.



Publicly funded broadcasters might take note that many public bodies changed their minds radically about their mission when they were privatised. State owned airlines believed their mission was to fly aeroplanes. State railways thought they existed to run trains. After privatisation, they discovered, or remembered, that their purpose was to serve passengers. PSBs frequently say their ultimate purpose is great programmes.

Some might say it is better seen as great radio and television stations, because in that definition the role of support services falls more usefully into place. But these definitions merely postpone questions about what the programmes are for. Usually PSBs say they are about information, education and entertainment. But in order to achieve what? As with so many publicly funded activities, the purpose of public service broadcasting is to make a difference to people’s lives, to enable them to do different things and be different. The final outcome is something like “enrichment of people’s lives”.




So measuring performance for PSBs is difficult from the very beginning, because their outcomes arehard to define, and are about intangibles. Theatres, cinemas and bookshops are also in the business of providing education, information and entertainment to enrich people’s lives, but they can measure their progress through sales figures.

PSBs must often use indirect indicators. For example, although it might not be possible to observe directly whether a PSB is contributing to the national identity, it is feasible to ask audiences whether they think it is, or whether they now know more about their fellow citizens, say, Inuits or Torres Strait Islanders, than before. The point of all this is that it is simply worth recognising from the start that the task is difficult, which means having to try harder, and that other public organisations have been here too (see Appendix).


Types of indicators



It would be easier if KPIs could be of much the same type about much the same type of things. In fact, one can see that PSBs have many variations to cope with.



Some KPIs used by CBA members are for external use to justify funding.


Some are for external use because of society’s current preoccupations (racial balance, gender etc).
Some are wanted for internal use only.


Some KPIs are about inputs, some about processes, outputs, products, outcomes.


Some are specific to broadcasting (about programmes) and some might apply in non-broadcasting organisations (about the use of resources).


Some are about quantifiable things, and some about things which are non-quantifiable. So some are direct measures using hard quantified data, and some are indirect indicators, using soft qualitative data.


Different things fall under the different headings of economy, efficiency, effectiveness.


Some are about programme making, others are about financial systems, engineering, marketing, staffdevelopment etc.


Some are about the desired behaviour of people working in the organisation.



This list is simply to emphasise that there is no one type of KPI which can be applied to the many different activities and objectives. It is not surprising if some confusion among the different types creeps in. It also shows that it is only too easy for planners to be tempted to think up an enormous number of KPIs, which might make for a tight ship, or for a crew drowned in detail.



Hierarchies



PSBs appear to use some similar indicators. But it is not clear whether they use the same approaches. For example:



Does each PSB have a hierarchy of KPIs? An orthodox approach (which has been known to work) is that at the top there can be a dozen KPIs which summarise the whole organisation. Feeding upwards into each are a number of KPIs at different levels, starting from a very low level. Thus a small unit might have some KPIs which contribute to departmental KPIs at the next stage up, and which ultimately feed into corporate level indicators.



Does each PSB see the Mission as the starting point? Otherwise, it might be the corporate plan, or the current annual plan, but those must assume some mission or overall aim. Again, an orthodox approach is along the following lines:


The Balanced Scorecard approach implies that there are various corporate and annual plans lying behind it, if there is to be progress under the many headings. The above can be effective, although there is room for argument about each stage. It is worth noting too that this kind of central planning has lost some support, because it is seen as too mechanical and can take more time than it is worth.



Draft high level KPIs



If one CBA member wanted to get a quick assessment of another member’s performance, what might be the KPIs?



First, by way of preamble, there needs to be a paragraph summarising the broadcaster’s statutory obligations, whether wholly publicly funded, current income, number of radio and television services, international service, number of staff.


Then KPIs – all for this year and previous five years to show trends:


If one CBA member had this information about another member, would it feel it had a good picture?

Would a CBA member feel it gave an adequate picture of itself? The Balanced Scorecard approach goes wider to cover areas which the above headings do not cover.


Some questions for PSBs



A number of questions present themselves – these are not in any order of importance.



What are the KPIs for? Sometimes the question “what is this indicator for” hides a political question, “who is it for”. Because PSBs seek public funding, they must tell a convincing story to politicians. Indicators for internal use might not be comprehensible to politicians. Also, some indictors produced for external use can seem to staff less than the whole truth. It is important to be clear why a particular indicator is being introduced.



Comparators. It is important to be able to answer the question, how well do you do compared with other organisations doing similar things? Who can PSBs compare themselves with, and in which ways? In time it might be possible for CBA members to compare themselves with each other. And with commercial competitors? Are there non-broadcasting organisations which are similar in certain ways?



Quality. PSBs claim to be in the quality business. It is part of their claim to public money. But in practice, quality of similar programmes can vary enormously across an organisation, depending on where it is made, or for which channel, or time, or because of uneven budgets. A programme might be intended to be innovative, or safely familiar. One type of indicator will not fit. Quality, in practice, can mean fitness for purpose, implying widely differing budgets and expectations of audience reaction. In support areas, it sometimes means something conservative, like free of error or operationally reliable.



The tyranny of what exists. KPIs are backward looking, about things we have been doing. The most important thing might turn out to be something we are not yet doing. There is not normally a KPI to measure resources devoted to messing around with some weird programme ideas, one of which might turn out well. A tight reporting structure discourages experiment.



New activities. PSBs develop new outputs alongside radio and television – books, CDs, T-shirts, and especially on-line services. Each new activity should have appropriate KPIs, and inventing them should test how the new product helps to fulfil the mission, as distinct from being an interesting thing to do.



Too many KPIs. There are many types of activity in broadcasting, in programme making, engineering, finance, and all the support areas. Planners are prone to enthusiasm for using as many headings as they can think of. The result can be a thick document, a nightmare of lists and figures and boxes, which not many people read.



Measurability. Broadcasting has a lot of things which are hard to measure, but it is not necessarily beneficial to make up for that by measuring everything that can be measured, although that is the temptation. The worth of a KPI system should not be judge by the length of the list.



New approaches. Do people who have experience of the Balanced Scorecard approach believe it is turning out to be an improved way of going about things?



Definitions. CBA members might use KPIs which have similar wording, but the definitions behind the words might be different. So might some of the methodology. Can CBA members agree not only to measure the same things, but also in the same way?



Vocabulary. Organisations, like a number of management writers, use several popular words in rather different ways – Mission, Vision, Objectives, Aims, Targets, Measures, Indicators. Similarly with the triad of Economy, Efficiency, Effectiveness. If CBA members are to evolve some common indicators they will need to agree among themselves how they will use the words. There is no one standard practice to follow. It is a case of Lord Melbourne’s advice to his Cabinet, “It is not much matter which we say, but mind, we must all say the same”.



Which culture. For a long time, production culture was dominant inside PSBs. More recently, as a corrective to decades of cheerful spending, accounting culture became the dominant culture. It took accounting culture to drive through the establishment of indicators. Not surprisingly, some programme makers feel the correction has gone too far.. The accountants and planners, who can expect little thanks for getting this work of the ground, have to remember the question “What is this PSB really here for?”, a useful sign to put on their office walls.



Distortion of purpose. Experience has shown how organisations can place so much importance on some indicators as to distort their purpose. Hospitals, for example, can easily raise their “number of patients treated” by freeing beds for new patients by sending people home quicker. Dumbing down increases tabloid sales, and perhaps radio and television audiences. Programme production costs can be lowered, for a while, by sacking producers. Costs can also be cut by reducing studio time, but is there an associated KPI to monitor any effects on quality? Better canteen food increases staff satisfaction (good) and as word gets around, increases queues (bad). One indicator has implications for another one, so when an annual report is pleased to announce that efficiency gains exceeded all targets (good news) it implies inexact target setting (not so good management).



A KPI figure can be improved by doing things in one area which damage the organisation as a whole. CBA members must have some very interesting experiences with this problem to share.



Direct and indirect. - Satisfactory indicators might be found by measuring a thing, or by coming at it from the other direction and measuring the effects of that thing. For example, the ABC is supposed to make programmes which are “innovative”. It might invent a measure of innovation. Or it might measure the effect on audiences, hoping to find enough people to agree with some proposition along the lines of “I’m always finding something new on the ABC”.


Presentation. Those of us who find that thinking about KPIs is a very interesting intellectual exercise should recognise that most people in broadcasting think we are odd, and that they find the subject deeply boring. The work must therefore be presented very clearly and attractively. Staff want to read something interesting and relevant. Government wants to read something short and clinching. The media want something easy. Finding a summary slogan can help: at one stage World Service was able to sum up its inputs and outputs with “£1 per listener per year”. Once all the work has been done, it seems a pity not to use it in a short “Facts and Figures About Our Performance” booklet for staff and public.



Top down, or bottom up? In practice, KPI systems are devised by senior managers or planning departments, because they see the need for them. Staff are not involved in the KPI design. But programme producers, low on the hierarchy, hold the fate of the organisation in their hands. If they are involved from the beginning, working through debates about the ultimate purpose of the organisation, and how to achieve it, they can learn the importance of KPIs and contribute. Otherwise they see performance measurement as a game invented for managers play.



Costs. Reporting achievements has limited meaning unless they are put against the costs which produced those results. It is easy to improve some KPI figure by throwing resources at it. Effectiveness cannot be judged well without reference to the costs of the resources used to achieve the objective. A programme series might win a big audience and therefore be judged a great success: if its full costs are known, its ratings might be seen as no better than they should be.



Reference points. Data must be put against something, other data, or last year, or what someone else does. Otherwise it is simply a statement of fact saying “this happens” – so?



Data. Some is available, some has to be got by planning to get it, a lot is unreliable. People who are asked to report data, especially programme people, will be casual about their figures if they do not see the point having to do this work.



Stability. PSBs find themselves under threat from time to time, and make urgent efforts to prove their value. This happens especially in the run up to the renewal of the charter or funding settlement. To broadcasting staff, this means being asked about new things in a hurry. Also, planners tend to set up a system and then tinker with it each year. The staff who have to reply to questions would appreciate a stable system.



Costs of the KPI system. PSBs want to spend their funds on broadcasting, not on maintaining bureaucratic structures and systems. How much must a PSB spend on a KPI system? How little can it spend?



Major events. PSBs hope that people turn to them at times of crisis or major events. This is worth measuring in retrospect. But as covering a particular crisis cannot be planned in advance, it does not go into a plan with a target, so paradoxically success cannot be entered on the KPI charts. It is as though what was not planned for does not matter. But of course it can be a major indicator. Is there a KPI for having editorial systems running which can cope successfully with the unexpected?



Outside check. Outsiders can be suspicious when any organisation reports that it has had yet another successful year, demonstrated by a selection of indicators. Many company annual reports provoke the desire for an independent assessment. Who could PSBs turn to for an understanding but rigorous independent assessment (given that many of us have little faith in the major management consultancy companies)? Clearly, it would be helpful for a CBA member to show that its performance is in line with those of other members.



Remembering the point. KPIs can help, first and foremost, to improve outputs and outcomes. For PSBs, this means improving the broadcasting. Any indicator which does not help this should be questioned.



Accounting publicly. It is now the habit for PSB Annual Reports to promise accountability and transparency.


Usually they contain extensive financial reports, but mostly for the past year, set against the previous year, which is not a very long trend.


There is also usually some statistical information, such as audience share, hours of broadcast, staff numbers, prizes won, and so on. These remain merely facts and figures unless figures are put against each other to show ratios of outputs/inputs, or trends over time, or achievements against targets (CBC and ABC report audience share going back five years). The information about indicators tends to be placed under the relevant functional heading throughout the report, so someone looking for an overall assessment of performance has to collect the information. The BBC World Service Annual Review (a separate document from the BBC Annual Report) has one section of half a dozen pages labelled Key Performance Indicators and Targets: someone looking for an overview of performance can find it in one place.


The Annual Report is where one would look for information about the current corporate plan. BBC summarises its Promises, and reports on progress. ABC summarises its Corporate Plan, and reports progress. In both, some progress reports have figures. CBC says its strategic plan, with indicators, is in a separate document.



CBA members will not want to adopt an identical template for KPIs in their annual reports, but they might want to consider some minimum common practice in what should be reported in their annual reports and how it should be presented. At the moment, it is not at all easy to compare like with like, and yet the annual report is the prime means of reporting publicly.



PSB journalists demand the right to know detail about private and public corporations. The public reporting of their own organisations is strong on written claims of success, but comparatively opaque about performance detail.



Hostages to fortune. Corporations of all kinds produce more documents for public consumption than ever before – Annual Reports, Visions, and so on. The language of these is usually not noted for modesty or understatement. Strong claims are made for stimulating debate, stretching boundaries, making significant contributions to something, providing important other things, broadening the range, offering programmes and services of outstanding quality, unrivalled range, and much else. If the writers of these documents were asked how they would use indicators to support their claims, they might become more cautious, and exact, and therefore more informative. There is also a multiplying number of publications about Codes of Practice of various kinds – programme making, journalism, governance, equal opportunities, complaints, and these have explicit targets, or imply them, and many could be quantified. It would be a major task to have to define indicators for what can be rather easily promised but measured only with much staff effort.



Seeing the whole. PSBs have learnt to break down their work and assign matching KPIs to the different elements. There is a need to find a way to bring it all together again so that we can see the overall picture in a brief summary. Perhaps even a new word is needed to express the idea of overall performance –“efficacy”?




At all times, the KPI system should help fulfil the PSB’s ultimate purpose, making great programmes, or enriching people’s lives, or however it is defined.. If it does not help, it is almost certainly hindering, by using up staff time and patience.

 



If in doubt about whether something might or might not be a useful performance indicator, one might go back to questions from Peter Drucker:



10 Reasons Why KPI Systems Do Not Succeed in PSBs

  1. Staff believe top management imposes a system under political pressure or merely because it is the managerial fashion to have one
  2. The real purpose of the organisation escapes definition
  3. The staff who have to contribute indicators cannot see their relevance to their working life
  4. Too many, too complicated, too incomprehensible
  5. What is measured is not necessarily important, only measurable
  6. Managers do not use the indicators to improve work, only compile them
  7. Staff forget about them, until it is time for the annual returns
  8. The available data is not reliable, or not sufficient, or too expensive to collect
  9. The results are disappointing, and are suppressed
  10. The KPI system is treated as an end in itself, not as means to an end


Appendix



In 1987 the UK House of Commons Committee of Public Accounts questioned senior civil servants about work on performance measurement, still then at an early stage. Everyone agreed it was necessary, but rather difficult in the public sector, because of the problems in defining purpose and the intangible nature of outputs.


Mr Deakins MP: In some areas like defence it may be very difficult surely to measure effectiveness at all?
Sir Peter Middleton, (Treasury): It is not difficult if there is a war on.
Mr Latham MP: A simple question, I hope. Are we getting value for money
from the Treasury? If so, how do we know and how does it show?
Sir Peter Middleton: Yes, I expect you have all read the Treasury section in this (briefing document).
Mr Latham: Summarise it for us please.
Sir Peter Middleton: I know one should not ask the Committee of Public Accounts questions but I will give a bottle of wine to anyone who can tell me what our main activity is.
Mr Latham: Preparing thick books by the look of it.



Summary: The way ahead for CBA Member stations?



Following the reasoning above – that using KPI’s is helpful in establishing the effectiveness of public service broadcasting – it could be helpful to the majority of public service broadcasters to move together, and adopt essentially the same KPI’s. Then, if they cared to share information between each other, comparisons could be made with like organisations. Essentially, this would mean organisations adopting a plan of action along the following lines:



1. Stations would summarise their statutory obligations, state how they are funded, and in what proportions, give their current income, state the number of radio and television services including any international service and website, and indicate their number of staff.



2. They would collect information on the following 13 KPI areas, and keep this up over the years to show trends:



Speech on International Benchmarking Group