The Short Answer
(1) There are many classification schemes or typologies for performance measures that have used over the years. The 4 quadrant typology is a new way to account for all the different kinds of performance measures. And it can be used to diagnose other classification schemes for performance measures.
(2) Think about the quadrants in terms of the following three questions. It is possible to use these questions as the labels for the different types of performance measures, instead of jargon words like “input, output and outcome.”
How much did we do?
(quantity of effort: upper left: least important)
How well did we do it?
(quality of effort: upper right: second most important)
Is anyone better off?
(quantity and quality of effect: lower left and right: number and percent:
lower right percent is most important)
(3) The most important distinction is between what we, the staff do, and whether anyone is better off – the difference between effort and effect. So the upper quadrants look at what we do and how well we do it. The lower quadrants look at our customers and the conditions of their well-being that our activities can affect.
(4) The second distinction is between the quantity and quality. This is the difference between how much we do andhow well we do it. The left quadrants look at quantity: how many things got done (upper left) and how many customers were better off (lower left). And the right quadrants look quality: how well things got done (upper right) and how customers (as a group) were better off.
Full Answer
(1) There are several different ways to explain the difference between these four types of performance measures:
Upper Left: Quantity of Effort: How much do we do? What functions do we perform? Here we typically count the number of clients served in total and by subcategory. Subcategories are usually based on client characteristics or geography. We also count activities in total and by subcategory.
Examples:
Total # of children served, # of children served aged 0 – 5
Total # of referrals taken, # of emergency referrals
Upper Right: Quality of Effort: How well do we do it? Here we typically count standard administrative measures of how well service is delivered (like client staff ratio, staff turnover, unit cost) and activity specific measures of service functions (like the timeliness or accuracy)
Examples:
Client staff ratio, client staff ratio for intensive service cases
Percentage of referrals acted on within 2 days, percentage of emergency referrals acted on within 24 hours
Lower Left and Lower Right: Qualitity and Quality of Effect: Is Anyone better off? In what way are our client’s or customer’s lives improved? The Left Quadrant is how many clients experienced this improvement. The Right Quadrant is the percentage of clients that experienced this improvement. Here we are counting what is most important about the program. What difference did it make for people? There are (at least) four ways in which people can be better off: Improved skills/knowledge, Improved attitude, Improved behavior, Improved circumstance. Each of these can be measured in (at least) two different ways, point in time (e.g. # and % of children with good attendance) or change from one time period to another (e.g. # and % of children whose attendance was equal to or better than last quarter).
Examples:
Number and percentage of clients with jobs
Number and percentage of clients who got jobs in the past month.
Each measure identified has two forms: a lay definition and a technical definition. The lay definition describes the data element in terms that non-experts can understand. The technical definition describes exactly how the data element is constructed. In the case of percentages or rates, it describes the numerator and denominator.
Examples:
Lay definition: High School Graduation Rate
Technical definition: Number of graduates at the end of the 12th grade divided by the total enrollment of 9th graders four years earlier.
(2) Consider the following example from a typical alcohol and drug abuse treatment program:
|
QUANTITY |
QUALITY |
|
|
EFFORT |
Number of clients served (lay definition); |
Staff vacancy rate (lay definition |
|
EFFECT |
Number of clients off alcohol and drugs at program exit (lay definition) |
Percentage of clients off alcohol and drugs at program exit (lay definition) |
(3) Why are these distinctions so important? Here’s one explanation.
The four quadrants are not equally important. The least important quadrant is the upper left where we count number of clients or number of activities. The most important is the lower right where we count the effect on peoples’ lives.
(4) Here’s another explanation.
Many programs are stuck in the upper left quadrant (“We served all these people. Aren’t we great!)
If they are not stuck there, then they are stuck in the upper right quadrant. (We served all these people, and we did it at the with the best qualified staff and the lowest unit cost in the county. Aren’t we great!) .
If they are not stuck there, then they are stuck in the lower left quadrant. (We served all these people, at a low unit cost, and we got 20 people off of alcohol and drugs. Aren’t we great!)
But 20 people off of alcohol and drugs is great if the total number you served is 25. It’s not so great if the total number served is 1,000. So until you get to the lower right quadrant and put the effect on peoples’ lives in proportion, you have not spoken to the bottom line effect of the program.
(5) For some measures it appears that they could go in either the upper right or upper left.
(a) First, don’t worry too much about where you put these kind of measures. You are going to use all the measures from the upper right and lower right in the next phase of selecting headline measures. So put it one place or the other without spending a lot of time.
(b) One example that comes up frequently is the percentage of people completing a program. This could be an measure of how well the program is delivered (UR) or it could be a measure of some benefit to the customer in terms of skills/knowledge, attitude, behavior or circumstance. Graduation rates for high school, completion of a job training program or satisfactory completion of a service plan can be viewed this way. Generally, I allow these in the lower quadrants, provided that other measures are added that describe in what way completion of the program might improve the well-being of its graduates. So, what percentage of high school graduates go on to school or work? What percentage of job training graduates get and keep jobs? After completion of the service plan, what percentage of clients stay out of the system over the next year? And so forth.
(c) Another example comes from an effort at Ohio State University in Columbus to improve a congested traffic system and an unreliable bus service (2000 RIT/USA Today Quality Cup: USA Today 5/9/2000 www.quality.nist.gov ). The Transportation and Parking Department completely overhauled the system, doubled the number of routes, bought more buses, hired more drivers, and increased parking rates as an incentive to use the buses. One benefit for customers: The length of the trip from the most remote parking lot to campus improved from 30 minutes to 7. This improvement is clearly a lower right quadrant “circumstance” benefit for the system’s customers. But what about the increase in ridership, up from 1.3 million to 3.5 million? Does the fact that more people are riding the buses mean they are better off? Not necessarily. It is possible to imagine a situation where more people ride the buses but are not better off? What if the parking rate increases made parking prohibitively expensive and people rode the buses because they had no choice? If it is possible to imagine a circumstance where the numbers get better, but customers are not better off, then the measure probably goes in the upper right quadrant, showing a possible improvement in how well services are delivered. The decrease in time to campus from 30 to 7 minutes is an unambiguous benefit to customers.
(6) When is the mere receipt of service an indication that someone is better off? And what are the implications for selecting headline performance measures?
This question addresses an important distinction between receipt and benefit. And this distinction is an important break with the past. First some examples.
Does the fact that someone receives counseling services necessarily mean they are better off?
Does the fact that someone receives education services necessarily mean they are better off?
We can easily think of situations where someone got counseling or education services and were not better off. It might be a bad counselor or teacher, or it might be a problem with the person themselves not attending class or not taking the counseling seriously. The point here is that the mere receipt of service is not a good proxy for people being better off. We must go beyond receipt to explore, “If this services was successful, in what ways would it show up in the lives of the clients, customers, recipients? When we answer this question we come to the lower quadrants…
Now consider some other services where this difference is less clear:
High school graduation, Red cross training, emergency room treatment, transportation services, etc.
In these cases, we might reasonably conclude that the customer who gets these services is better off, than if they didn’t, but not so fast. Again, can you imagine a circumstance where someone got these services and were not better off?
High school graduation: bad school or social promotion
Red Cross training: bad instructor, marginal passing grade
Emergency room treatment: understaffed service leads to preventable death
Transportation service: inconvenient service, 3 bus routes to work, but no other choice.
The reason we would consider receipt of service as a proxy for better offness is the general reputation of the service (Red Cross) or the expectation that most people receiving the service get good service and are helped, or that people would’nt use the service in the first place if they weren’t better off. But we can see that any of these assumptions can mask bad service delivery and poor help or maybe actual harm to customers.So we must go beyond mere receipt of service even in these cases.
Finally we come to the third and final set of examples:
Hospice services,
Battlefield treatment of wounded,
Disaster assistance.
Two things are true about these services. In some cases the client can’t “get better” (hospice) and the receipt of service is almost all there is. Second, the service received is so desperately needed that the absence of service causes additional harm. These are the kinds of services where receipt of service may be most persuasively argued to be a good proxy for clients being better off. But even here, there are ways to go further:
Hospice service: client and family customer satisfaction
Battlefield treatment: survival rates
Disaster assistance: persons who are saved, persons who find new housing.
Conclusion: In cases where the receipt of service is prima fascia evidence of better-offness, then upper left quadrant counts of “how many people received service” (upper left) and “what percentage of need for such service is met” (upper right) should be also considered for inclusion among headline performance measures (in step 4 of the selection process above).
(7) See the 4 Quadrants link to other performance measurement terms
(8) See the TECHNIQUE: A Five Step Process for Identifying and Selecting Performance Measures