Which Capacity Building Metrics?

So I'm working up a scorecard for a software donation program to charities and social enterprises. We care most about the capacity building impact within our grantees. When a grantee receives our intervention, the final outcome should be increased organization capacity and increased organizational effectiveness. What metrics should we use?

I must say the siren song of "number of donations" is strong in my ears, but show an executive that metric and decision making might not head in the right direction...

I've found the NPower / NTEN work around impact evaluation for Nonprofit Technology Assistance Providers (NTAPs) useful, but no simple scorecard emerges. They identify two channels to social impact for an NTAP (which is a good benchmark for a software donation program). #1 Technology Management Capacity #2 Technology Innovation.

Software donation program basically work on the technology innovation lever allowing organizations without financial resources, but with technology management capacity, improve organizations effectiveness.

Not sure it's possible for a software donation program to generate outcomes through improving technology management capacity. But by providing a well defined implementation methodology, the software donor can increase the technology management capacity of a single organization on the single task of implementing the software donation.

Michael Gilbert has a good piece on why asking the right question is critical in evaluating nonprofit technology outcomes. Metrics, key performance indicators, scorecards, etc. are only useful in so far as they ask the right questions. I'm not sure we've ever come up with the right questions for technology donation programs.

So, out to the network of smart people... any bright ideas on scorecard metrics that would point a software donation program in the right direction?