GiveWell: Evaluating Marketing Messages as if they were Impact Statements

I have followed Givewell (http://www.givewell.net/) since it was founded and have come to an epiphany. Givewell is evaluating marketing messages as if they were performance data. So naturally very few charities look effective.

Givewell was founded on the idea that if we evaluated charities in the same way we evaluate stocks... with the same rigor and analysis... donors could make better investment decisions. Unfortunately, donors make donation decisions, not investment decisions.

And charities know empirically (fundraising is a science with direct mail stats, A-B testing, etc.) that donation decisions are driven by emotion not analysis.

In the world of public companies there is a simple fact. Companies lie. That is why we have the SEC, that is why we have accounting standards, that is why stock analysts are always trying to pry out more information to evaluate the "truth".

The lies that companies tell are not about maliciousness, they are about marketing. Take a class in marketing and they tell you to focus on the benefits to the consumer. Simplify the message. The process of taking complex and nuanced facts and boiling them down into marketing messages leads to distortion and hyperbole.

Now we apply this to GiveWell and the 388 charities they evaluated and the 9 they recommend. Our sector has no equivalent of the 10-k form so Givewell looks at the marketing messages and sees distortion and hyperbole. Well of course!

Lets just remind ourselves of the difference between a 10-k and a marketing messages.

The 10-k is written by operations professionals in the finance department based on reams of real financial and operational data. Marketing messages are written by marketing staff, who generally don't have access to the same financial and operational data.

The 10-k goes out under the CEO and CFO's signatures. By penalty of law, the data needs to be correct. The marketing messages doesn't go out under the CEO/CFO signature and in all but the most egregious situations, hyperbole, distortion, and yes even lies are not sanctioned in any way.

The information Givewell has to work with is generally not produced based on internal financial and operational information (with the exemption of the 990) AND hasn't gone out under the CEO/CFO signature as true and factual under the penalty of law. The fact that most evaluated charities can't be considered "effective" is less about the charities themselves than about the data those charities make available.

I personally think that Givewell should continue doing what they do. Their methodology is sound and highlights that fact that charities generally don't provide the data required to evaluate effectiveness/ROI. Hopefully their work will encourage more charities to provide that data.

At the same time, they need to lower the intensity of the attacks on the marketing messages. They recently took SmileTrain to task (http://blog.givewell.net/?p=466) for their $250 per surgery figure. While the analysis is good and I don't disagree with them, I would like to see them acknowledge that the $250 figure is a marketing message not an operational measurement.

Again, the corporate world is littered with marketing messages that don't stand up to the facts - "Build apps five times faster, at half the cost." and a million similar statements. Let's not hold charities to a higher marketing standard than everyone else, lets highlight the fact that they don't provide the operational data required to evaluate their effectiveness.

-