The Short Answer

1. Start with the data you already have. Most programs collect far more data than they use. Do not start the process by collecting new data.

2. Create new data where necessary. Be creative. Use common sense sampling techniques. Some data is better than no data.

Full Answer

  Existing sources and creation of new data (link to data sources)

(1) Program people will often say, “We don’t have any good data.” And sometimes they’re right. But more often than not programs collect a great deal of data. The problem is not the availability of data but the fact that it is not used for anything. Programs collect far more data than they use. 

An interesting corollary here is that programs enter more data into management information systems than they use.  Another use of the 4 quadrants is to evaluate requests for data elements in MIS design to focus on those that are most important and most likely to be used.

(2) So the first place you look is at the data that’s already available. The technique described in the selection process  in step 4 identifies the data elements in the upper right and lower right quadrants for which there is currently good data.

(3) Where data does not exist, it is possible to create new data. And the most important thing to remember here is that this does not have to involve 100% reporting. It is possible to gather important performance data using sampling techniques. And these processes can be put in place quickly – in some cases in a matter of days, not  years.

(4) Story: Let me tell you a story about a community mental health office in a small state. I went to visit these people last year. And the conversation more or less started like this: “We don’t have any data. We can’t do this.” I asked, “If we took 10% of your caseload every month and asked just two questions, do you think you could do that?” They thought they could. And we fashioned two customer satisfaction questions, one in the upper right and one in the lower right quadrants, in the most simple, plain language we could think of:

• “Did we treat you well?” (a proxy for courtesy, timeliness and cultural competence etc.)

• “Did we help you with your problems?” (a proxy for making a difference in their life or the life of their children.)

12 words total!!

(5) The difference between one time and ongoing data collection. This is one of the most important things to think about with regard to evaluation. Often evaluations are thought of and structured as one time events. Partly this is because evaluations involve intensive data collection and while you’re at it you might as well collect a lot of data, often too much data, so much that they burden the program. Performance measurement data to be useful must be collected on a regular basis. This does not mean 100% reporting or even continuous reporting. It could be a sample every 3 months

For more information on data collection methodologies see the work of The Evaluation Forum
www.evaluationforum.com