GiveWell: Evaluating Marketing Messages as if they were Impact Statements

I have followed Givewell (http://www.givewell.net/) since it was founded and have come to an epiphany. Givewell is evaluating marketing messages as if they were performance data. So naturally very few charities look effective.

Givewell was founded on the idea that if we evaluated charities in the same way we evaluate stocks... with the same rigor and analysis... donors could make better investment decisions. Unfortunately, donors make donation decisions, not investment decisions.

And charities know empirically (fundraising is a science with direct mail stats, A-B testing, etc.) that donation decisions are driven by emotion not analysis.

In the world of public companies there is a simple fact. Companies lie. That is why we have the SEC, that is why we have accounting standards, that is why stock analysts are always trying to pry out more information to evaluate the "truth".

The lies that companies tell are not about maliciousness, they are about marketing. Take a class in marketing and they tell you to focus on the benefits to the consumer. Simplify the message. The process of taking complex and nuanced facts and boiling them down into marketing messages leads to distortion and hyperbole.

Now we apply this to GiveWell and the 388 charities they evaluated and the 9 they recommend. Our sector has no equivalent of the 10-k form so Givewell looks at the marketing messages and sees distortion and hyperbole. Well of course!

Lets just remind ourselves of the difference between a 10-k and a marketing messages.

The 10-k is written by operations professionals in the finance department based on reams of real financial and operational data. Marketing messages are written by marketing staff, who generally don't have access to the same financial and operational data.

The 10-k goes out under the CEO and CFO's signatures. By penalty of law, the data needs to be correct. The marketing messages doesn't go out under the CEO/CFO signature and in all but the most egregious situations, hyperbole, distortion, and yes even lies are not sanctioned in any way.

The information Givewell has to work with is generally not produced based on internal financial and operational information (with the exemption of the 990) AND hasn't gone out under the CEO/CFO signature as true and factual under the penalty of law. The fact that most evaluated charities can't be considered "effective" is less about the charities themselves than about the data those charities make available.

I personally think that Givewell should continue doing what they do. Their methodology is sound and highlights that fact that charities generally don't provide the data required to evaluate effectiveness/ROI. Hopefully their work will encourage more charities to provide that data.

At the same time, they need to lower the intensity of the attacks on the marketing messages. They recently took SmileTrain to task (http://blog.givewell.net/?p=466) for their $250 per surgery figure. While the analysis is good and I don't disagree with them, I would like to see them acknowledge that the $250 figure is a marketing message not an operational measurement.

Again, the corporate world is littered with marketing messages that don't stand up to the facts - "Build apps five times faster, at half the cost." and a million similar statements. Let's not hold charities to a higher marketing standard than everyone else, lets highlight the fact that they don't provide the operational data required to evaluate their effectiveness.

-

Crazy...

I couldn't agree more,

"Sitting in the session about Drupal made me think about how crazy it is that we have hundreds of PBS and NPR stations all working on the same problems, and there is not enough sharing going on. Why should we all spend tens of thousands of hours working on the same issues separately?"

Via Ken Rickard http://www.palantir.net/blog/drupal-and-non-profits

Who is carrying that torch now? Or better yet, perhaps no one is carrying the torch since it is more of an organic discovery process going on.

If the Whitehouse understands how software can have mission impact, why do we have so much trouble in the social sector?

http://www.washingtonpost.com/wp-dyn/content/article/2009/10/24/AR2009102401266.html

http://radar.oreilly.com/2009/10/whitehouse-switch-drupal-opensource.html

CiviCRM's 3.0 Review: Where are the weaknesses?

I wrote a post in 2006 about CiviCRM's weaknesses. With the upcoming CiviCRM 3.0 release (try it out), I thought I would revist those weaknesses and see if I could add some more.

1. Ecology. The ecology is now mature enough to support virtually any size customer.

http://civicrm.org/professional (support the active contributors!)

Community reviewed list of hosts

2. Documentation. Online docs done by the esteemed John Kenyon and just released a CiviCRM book! I think the documentation is certainly on par with anything out there.

3. Donor Management. Pledges, soft credits and user configurable LYBUNT reports, what more can anyone want? (there are some things, but the basic functionality in now all present)

According to my criteria, CiviCRM has long surpassed the sustainability tipping point:

• Lots of competant consultants are avaliable.
• Nonprofits can pay for hosted CiviCRM or an install.
• Documentation and online training is avaliable.
• People are extending CiviCRM and contributing back. Go CiviCase!

So what are the weaknesses now? Since the CiviCRM team moves very rapidly, many of the key weaknesses are getting addressed in the 3.0 release... I probably should do this post yearly.

1. Usability. This was a recent bug-a-boo that the CiviCRM team took a big swing at in the 3.0 release. The new navigation bar is pretty much to die for. The configuration checklist that makes setting up a new site a snap is underadvertised and perhaps underappreciated. Oh, and the recent items list is a basic but import feature. There is still room for improvement, but CiviCRM is certainly now on par with any competitive piece of software.
2. Reporting. The new CiviReport framework addresses the basic reporting gaps and allows the community to fill most remaining reporting gaps.
3. Donor management gaps. Things like postal mail merge are not tightly integrated into the application or into the CRM history. Prospecting and proposals have yet to be addressed directly.
4. Accounting integration. This need to be smoothed out and improved. And the community is already on it.
5. Volunteer management. Not sure this is a weakness, per se, it just hasn't been a priority.

And other things people can add to the comments.

Overall, the 3.0 release represents the end of fundimental gaps in the utility of CiviCRM for most nonprofits.

Read this and be ashamed

Every once in awhile I read something that just makes me sad.

Reading the Northern California Grantmakers blog, I came across this statement from a Grantmakers for Effective Organizations (GEO) publication.

Grantmakers, for instance, often are not aware of what it actually costs nonprofits to deliver services.

Something is seriously wrong with an industry that doesn't understand its customers. How on earth can grantmakers be OK with this situation?

Well, a gem from my experience with private foundations. In almost all cases grantees are not customers, the trustees are the customers. Foundations understand their trustees *very* well and generally (not always) the trustees don't require their organizations to understand grantees. And even if staff understands grantees, that isn't enough to change policy or trustee behavior.

In the egregious cases, the value of a grantees to a trustee derives from their ability to support the trustees ego. Does the grantee make me feel good? Does the grantee support my religious construct of "giving back." Does the grantee connect me with other powerful people? Can I chat about what the grantee does on my private plane with my social and business associates?

And finally, since the grantmakers are basically running the White House Social Innovation Fund, are we OK with people who are often unaware of what it actually costs a nonprofit to deliver services making the decisions about what the most innovative, scalable and effective programs are?

Nice little nugget to chew on there.

Inspired Contributions: Intervention for Dummies

I'm usually pretty cynical when it comes to corporate contributions to development or social change. Then I read an absolutely inspired toolkit for human centered design done by IDEO and the Gates Foundation.

http://www.ideo.com/work/item/human-centered-design-toolkit/

With more and more do-gooders floating around poor communities, they should read the toolkit and really design interventions that effectively address the problems they are trying to solve.

Social Source Four Years Later

I love it when thoughts converge.

Sonny Cloward tweeted a nice reminder of the nonprofit technology open source vs. commercial debate that has been going on for awhile [his post, my post]. He was congratulating Johanna Bates on a very nice post on her personal view of open source and nonprofits.

All this was catalized by Eben Moglen's NTC Plenary and has been covered by Holly Ross's post.

Four years latter basically nothing has changed in the discourse, but facts on the ground have seen a sea change.

The discourse is stil either/or & black/white.

The CiviCRM guys don't point out that Salesforce has a more polished functionality and better reporting tools. The Salesforce guys don't point out that CiviCRM can do membership management, events management, seamless acceptance of online donations and mass email -- all things that nonprofit users ask for every day on their message boards.

Why?

Simple marketing.

It is virtually impossible for platform providers to acknowledge one another or integrate because you are trying to get the customer to select your platform. Salesforce says select our platform and our ecology will help you out (though over the last 4 years I have seen little in the way of software innovation avaliable on the same terms as the platform donation - volunteer management tools, member management tools, event management tools, etc. CiviCRM says wait a few months and you'll get the stuff on our roadmap (better reporting, case management, improved events, etc.). I'll leave others to invent a better model of marketing a platform.

The facts on the ground, however, are finally getting really inspiring.

Salesforce has done a great job of penetrating the market through their donation program. In the last month, they finally seem to have got their act together and put the infrastructure in place to support long term, sustainable impact on nonprofit technology. And in doing so, low and behold, they have embraced open source.

Four years ago Sonny pointed out "I could easily use Salesforce as a model of proprietary/open source partnership—a corporate developer that embraces open source integration into their product." Yet it took them until today for them to actually take the lead and open source the starter pack.

The Nonprofit Starter Pack is now and open source project and they can begin the process the CiviCRM team began 5 years ago of building out nonprofit software that works for day to day users (events, memberships, etc.). By Salesforce embracing open source, they have finally, IMHO, put the critical pieces in place to transform nonprofit technology as part of a mission rather than as part of corporate philanthropy/marketing.

As an interesting side note, the traditional commercial nonprofit software providers are being assimilated into the Salesforce borg (Convio's CommonGround, recent MicroEdge announcement). Not sure of thos implications, but very interesting.

On the CiviCRM side, the juggernaugt continues to innovate and expand with features that regular nonprofit staffers need to use every day.

Four year ago Sonny took me to task and highlighted the key point: "nonprofit staff are pining away for affordable and effective apps that allow them to do their jobs"

Today, the situation is far better than it has ever been before.

Platforms, people, platforms! OpenWiser

I had a conversation years back with the WiserEarth people since it pains me to see history keep repeating itself.

Recently there was an effort to fund open APIs for the WiserPlatform which had me take another look at the software ecology behind the mission. And in that look I found a priceless quote:

Paul's vision for WiserEarth always, always included it being open-source - there was never any back-and-forth on this matter, which is why we've always been so openly confident and deadfast in stating that WiserEarth is an open source project. 

This is why folks in the sector need to find themselves some qualified technologists. Lets recap some of the highlights of executing this vision:

1. They roll their own platform because they don't want to build on existing platforms like Drupal or CiviCRM or a million other platforms I'm not personally associated with.
2. They don't make their code available to anyone (later remedied).
3. They don't build a data standard or API for www.wiserearth.org . 
4. They have to hire someone to "clean up" their code for the open source release.
5. When they release their code, significant amounts of functionality from wiserearth.org are not available.
6. They can't afford to build any APIs and have to crowdsource money to raise the money for it. 
7. Near as I can see from their developer community, no one except the folks that paid to open source their code uses their platform.

First, these guys are deserving, they run a really compelling community and they have a bunch of great ideas. But their technology execution is horribly misaligned with their mission and vision.

If Paul's vision was always to open source the platform, he either meant it in "marketing-speak"or didn't really know what he meant by "open source" - he was just using the word. This is all to common among executives and progressives with good intentions, but that doesn't make it OK. Just calling something an open source project has absolutely NO mission impact other than providing a "marketing lie" that makes people feel better about signing up at your website.

So if you want to do a social change technology project and have it be "open source," please bring in the technology strategist that knows what that means and the coders with the experience to do it correctly -- just hiring any old development firm tends to put you square in the "marketing lie" category.

Another Painful Aside

No APIs?! Talk about not having a technological clue. I suppose the logic was "wikipedia doesn't have APIs, why would we need them" (even though by 2006, MediaWiki had APIs).

This is yet another area where you need a good technologist. As an illustration: 

Back in 2005, when we had to put together a "database in the sky" (which is an apt description for so many social change web projects) for a database of every missing person on the web in the aftermath of Hurricane Katrina  the FIRST thing we did was define a data format. The next thing we did was define an API. Then we built what we wanted to build.

Believe it or Not...

Having said all that, believe it or not, you should give a couple bucks to OpenWISER. I hope that someone will nock them upside the head and make them publish their data formats (or adopt existing ones) before they build code, but it would be a very good thing for the progressive movement to actually execute this one right.

Comercializing a Community

A very interesting discussion is going on around Drupal's core development process.

As I read it, I realize the only voice of the companies that drive a lot of Drupal in that conversation is Dries. Sure, a bunch of people that are employees of the companies are contributing as individuals, but the companies themselves are not in the conversation.

Drupal is clearly being comercialized... folks noted that the Drupal homepage no longer has interesting stories, it's tipping toword site annoucements. The Drupalcons and various development sponsored by commercial interests is moving Drupal away from its roots.

I have two thoughts on this:

1. The companies are paying attention closely and communicating only through back channels. One might ask why that communication can't happen in the open.
2. The companies should be neck deep into the conversation. This is the future of their businesses.

Drupal 7 UE Redesign: Just Copy Already

So I've been observing the Drupal  7 UE redesign project and have made another tremendous discovery. There is little (probably nothing) new under the sun. So when you want to hit the 80% principle, just copy from others.

So you have some of the intial Mark Bolton concepts & Lullabot's Buzzr UI for Drupal. Then you look at other CMS's, specifically Concrete5 and CMS Box (one of 2008's best UIs according to Jakob Nielsen). And you come to the conclusion that a CMS requires, drum roll please, a header, overlay window and inline editing -- three things that are in each of these CMSs and CMS designs.

This begs the question just how much original usability testing, getting to know your user time is really required. Couldn't you  just copy what has gone before you? Or perhaps it is really good validation that the basic concepts are right on.

And a final thought. These concepts really aren't going to make Drupal unique... something else is required.Hopefully Bolton's concept of a "Tool for Site and Page Structuring" can be that unique element.

What it would take to start a CiviCRM ASP

So we tried to start a CiviCRM/Drupal based ASP to solve the constituent relationship management/website/online donation/ mass email problem that most charities face with CivicSpace.  It failed, but that does not mean that another attempt will also fail.

There are three basic approaches to doing a CiviCRM/Drupal ASP:

• Technology first
• Customer first
• Hamster first

The technology first approach is building out the infrastructure to handle a high volume, self-service ASP.... low monthly price and high customer volume. This requires either piles of money or the super-committed technical geek founder to do the work. It relies on the build it first, then find the market approch. We did that at CivicSpace and we "ran out of runway".

Customer first says lets go out and build a lot of demand. Sure it will be really labor intensive to maintain the technology infrastructure and initially the customer service will not be great, but you avoid solving the technology problem until you have the real problem of too many customers and you need to build automation technology.

Hamster first is buy a VPS, put up a cool web page, market your product and hope for the best. The technology stack (CiviCRM/Drupal)  is actually fine for this approach at the moment, but you'll face bulk mail deliverability, scalability, performance and other issues along the way.

Tech and customer both require a fair amount of capital to pay for the technology development (the ASP platform) or the marketing (making the service known in a very crowded vendor space). Hamster first could financially support a single consultant and once they have a working model, could easily be put in front of investors to attrach "expansion" capital rather than "start up" capital. 

The other trap is the set up fees. We tried to make things self service... life is just too complicated. There has to be a set up service before your customer starts paying their monthly fee. My feeling is copy success... i.e. copy PicNet who have built a similar business on Joomla. People pay a couple thou to get started and then a monthly fee. I think they cracked an important part of the code.

Idealware releases new CMS report

So Idealware released the much anticipated CMS report covering Wordpress, Drupal, Joomla and Plone. Overall it is a must read and all around general "reference for the ages."

I'll start with the nit picks and then get to the good stuff.

First, the "market analysis" fails what my ex-boss used to call the smell test. Sure the methodology is perfectly defensible, but the result is no where near reality. The 10,000 pound gorilla is Wordpress, not Joomla. Even though Joomla has a lot of traction in the traditional NPO world, I find it hard to reconcile the numbers. Plus, in most of the rest of the world the word "charity" is used instead of "nonprofit" so you might want to also inculde that keyword.

The security methodology appears to be just plain wrong. It appears that platforms with more security advisories are considered less secure. I'll hope that the actual methodology was different, but if not, it shows a fundimental misunderstanding of how open source security works. 

The starting point is that there will always be bugs and security flaws in released software. The security of a platform is measured by the significance of those flaws and the speed at which they are resolved.

There can be both good and bad reasons for a high number of announcements.

Bad

(1) Code quality is poor - more security flaws are released in the the wild

Good

(1) A larger community of people is testing and therefore identifying security vulnerabilities.

(2) The community standard for what constitutes a security vulnerability is more stringent than a comparable project.

(3) A more transparent security process. No security problem is ever fixed without the release of a security advisory.

(4) The lifespan of security issues is very short... no security issues "linger" after they have been identified.

In general, the number of security advisories is a flag to look a bit deeper. High numbers of advisories can be either good or bad, you need to dig deeper to draw a conclusion.

The good stuff is the financial model behind the report. The ad model is really a quite good one. Since charities don't have the money to actually buy the report, get the consulting shops to buy advertising.

I think they should take it one step further. There is little upside to ad sales to cover the production of a report + surplus. Idealware has a good neutral reputation. They do a good job of maintaining it.

Why not broker leads to companies? All the idealware information is "hidden" behind a registration wall. Idealware's interactions with information consumers provide an opt in for vendors to communicate with them. Those opt in leads are sold to vendors. 

This is a lot more involved than the ad model, but has a much higher upside as your volume goes up. Haven't done the numbers to see if this is really viable and don't have a solid sense of what the consulting firms would pay, but I suspect it would work.

FINALLY, the vendor community steps up

So I look at this new Blackbaud NOW product and I must say, they have their corporate strategy right on to own all of the charity software market, soup to nuts.

Blackbaud NOW is basically Groundspring/ Network for Good -- a set of services designed for very small charities -- accept online donations, keep a central contact database, send mass emails. They take around 5% of your donation and you get the service for free.

Judging from the mailing address in Indiana, this is built on etapestry's technology (PS, please spring for a web designer, guys, the etapestry site is an eye sore). I find it interesting they didn't build something on Blackbaud's Infinity platform, but hey. They also couldn't spring for an email blast tool, but I suppose that might open a can of worms for them-- their email tool is basically designed to send email to individual contacts rather than mass mails with open tracking, etc.

If I put my cynical hat on, I would say this is just an etapestry lead gen tool, but I'll give them the benefit of the doubt and hope it is the precursor to real service for small charities. And hopefully some corporate strategist at Blackbaud has figured a way to serve the bottom of the market in order to feed prospects into their higher end offerings.

It makes me a little sad since CivicSpace offered this basic package plus soo much more , but alas... we were a bit to early and under-capitalized.

And finally, yet another data point that a CiviCRM-based ASP would be a good value proposition! Come on folks, anyone?

CiviCRM Continues to Make Constituent Relationship Management Accessible to Small Groups

With the latest CiviCRM release (2.2) I am reminded that the CiviCRM team is one of the few groups actively making a product designed as a solution rather than just a tool. 

The single most important feature in 2.2 is the Simplified Configuration option for CiviMail. Email marketing is a critical engagement tool for charities and all other civic groups. But for the folks that don't have the money to use commercial services, there just aren't any integrated, simple options. The new CiviMail solves that by just connecting to a SMTP server to send mail. Got Gmail? You now have open and link tracking!

Sure there are still spam management concerns... that's what paid services like CiviSMTP are for.

And yes wouldn't it be great if there was an ASP.... [any (social) entrepreneurs out there interested?] .

And, yes, other folks out there are making strides-- the Salesforce Foundation is taking some steps in the direction of an out-of-the-bax charity experience, but that hasn't been their primary focus over the past few years. As Michelle Murrain notes, the out-of-the box functionality of CiviCRM is just better... donation pages, marketing email, relationships, smart groups and more are there and with a few clicks can be working for a small group in a couple hours. You have to (sometimes)  purchase and (always) integrate those solutions into Salesforce.

Now if we get the CiviCRM usability up a few notches we can have a horse race for meeting basic charity and civic group needs.

Operational Challenges of Scoial Enterpreneurs

I'm hosting an online discussion over at Social Edge about the operational challenges faced by social entrepreneurs. With a big focus on finding practical solutions.

http://www.socialedge.org/discussions/social-entrepreneurship/operational-challenges/

If you have a story about an operational challenge you faced or ideas on how social entrepreenurs can think about their operations and business process to maximize social impact, please drop by and leave a comment.