Personal history of the Katrina PeopleFinder Project PART I

A personal history of the Katrina PeopleFinder Project PART I

The term “social source” is something I have used for half a decade to describe collision of nonprofit technology and the open source movement. It tries to capture the idea that technology can be harnessed for a social mission by employing community development, online community, and web 2.0 strategies.

The PeopleFinder project started for me with a fairly simple social mission. Some folks wanted to put up a website that included CiviCRM, the open source nonprofit constituent relationship management system being developed by a bunch of us at the Social Source Foundation.

Being a good cause and a good test of our technology, we agreed to help install CiviCRM in www.neworleansnetwork.org. Its use would be to power a “peoplefinder” feature, like so many others on the web, to help connect evacuees with one another.

And then I thought about it for a second.
(1) Why build yet another small scale solution to a large-scale problem.
(2) We built CiviCRM to solve major, large-scale nonprofit effectiveness issues related to constituent relationship management.
(3) Open and distributed systems can scale to provide real solutions to national problems.

As this stuff was formulating in my head, I drew up the initial fields for the peoplefinder thinking that we could aggregate all the evacuee sites on the web. Never once did I think there could be a single, “master” database of evacuees. Instead I thought about ways all the evacuee sites could “talk” to one another.

So we needed a data standard with the right fields so that all these bulletin boards and online databases could interoperate. I’m not sure at what point I decided this was going to be a national solution to the problem rather than a small community based web site, but on September 1st, I observed in an email, “Seems like we could bang something usable out in a couple days, get volunteers to do data entry from discussion boards, etc. and have a pretty useful refugee matching solution.”

So then I went out to people I knew and started enlisting help. Andy Carvin, Marty Kearns and Deborah Elizabeth Finn got the first email. Kieran Lal and Zack Rosen from CivicSpace Labs were already involved and they brought in Steve Wright from the Salesforce Foundation.

And this kind of became the ethos of the PeopleFinder project. Send an email out about what needs to get done. People respond to that email and take charge of getting things done. Magically, a solution appears and you’re not quite sure what exactly happened, but you’re trilled that there is now a solution. And you move on to the next thing.

By the 2nd we had a comprehensive list of missing persons sites tagged in http://del.icio.us/tag/peoplefinder. If we wanted to aggregate the bulletin boards and databases, we would need a dynamic, living and scalable list. Del.icio.us did the trick. Note the first use of an open technology…didn’t have to buy it, could just use it to do good. This is an important theme, the technology has to be pre-positioned, accessible, and you can’t need to “ask permission” or even involve the folks that “own”/maintain the technology to use it for your purposes.

About this time, Salesforce.com Foundation committed to providing the back end database and search engine. My motives in engaging Salesforce were twofold. First, they are good guys, committed to open standards…if only they were open source ☺ Second, I felt it important to get a big corporate player involved in the hopes that they could move resources latter on in the process, though their technology is pretty cool too. (Gotta remember, I’m part of the team building the open source nonprofit CRM—I think the nonprofit sector needs a Salesforce.com class solution that meets their needs and is open source :).

The Salesforce.com Foundation and the folks in the company have good hearts, do good work, and we were blessed to have them on the team.

Also on the 2nd we put up a mailing list, katrinadev, because it’s the Internet and you can’t do a project without a mailing list, and recruited folks and did countless other tasks.

On the 2nd we also made a critical technology decision… use a distributed technology like RSS to solve the problem of 20 different evacuee databases. Rather than force everyone to go to a central database, lets make EVERY database central by syndicating evacuee data. At this point the Godfather of the PeopleFinder Interchange Format (PFIF), Ka-Ping Yee, rode in on his white horse. I’m pretty sure Zack Rosen roped him into this, but I don’t actually know for sure how Ping got involved.

Actually, I thought the spec took 24 hours longer to write than it should have, but I have no technology skills, and, as it turned out, should have just trusted Ping and Jon Plax to do a good job… cause they did a stellar one.

I’m not even sure what we actually did the 2nd… Andy Carvin was great at helping flesh out the idea and introducing us to bloggers and others that could spread the call for volunteers. Other folks “spread the meme” and people kept popping out of the woodwork to do stuff.

I think it was the 2nd that Ethan Zuckerman and Jon Lebkowsky were introduced to PeopleFinder—they become critical to the story latter on.

On the 3rd, the Salesforce.com team outlined a project plan and lined up internal resources. Kellan Elliott-McCrea connected us to some guys from Craigslist who were facing the problem of being a repository of missing persons and saw the benefits of a central database (they eventually coded part of the system we used to parse Craigslist into bite sized chunks for data entry volunteers).

We needed a website, a place for a community to self organize. I’ve been a member of the Omidyar Network (http://www.omidyar.net/home/) it started, and have always thought they could be much more than they are. They exist so that more and more people discover their own power to make good things happen. Seemed like a good fit at the time.

On the 3rd Jon Lebkowsky came into the mix from Omidyar (I think). I’ve chatted with Jon a couple times, knew he was a good guy, and basically got out of his way. At some point Jon and Ethan Zuckerman from the Berkman center at Harvard became the point people on data entry. I literally have no idea how the code got written to enable volunteers to do data entry, how the training materials for volunteers got developed, or how that whole side of things happened. I just know there are lots of amazing people that came together and made it work.

Took us about 3 hours to outgrow Omidyar’s interface and move over to http://katrinahelp.info/, a wikipedia site much better suited to the type of self organizing we were doing. Looking back at my email, I think it was Jon Lebkowsky that introduced me to Rudi Cilibrasi, the guy “in charge” of katrinahelp. I just remember trying to connect with Rudi on Skype and having the technology just not work. I ended up calling him (he lives in Europe) and having a 15 minute conversation that just lead us to trust one another… our goals and values were in alignment.

Again Rudi was providing open technology—a wikipedia site. We didn’t need his “permission” to start using the technology (though of course we got it first because that was polite). Even though we didn’t technically need his help to use the technology, he was an amazing resource because he understood deeply how his technology worked and could help others in the community use it to solve problems.

The relationship between PeopleFinder coordinating organizations (Social Source Foundation, CivicSpace Labs, and the Salesforce.com Foundation) and katrinahelp.info is part of what I call Web 2.0 Collaboration. People, technology and organizations whose default position is trust…whose first question is “how can others leverage what I’m doing” rather than “how can I protect myself from other leveraging what I am doing.”
http://socialsource.blogspot.com/2005/08/social-source-web-20-collaboration.html

This whole experience underscores the absurdity of building insular communities that “discover their own power to make good things happen.” Communities exist all over the world and in cyberspace and just need a little infrastructure to catapult them into highly effective entities. That infrastructure of communication and simple directories of what is available needs to be distributed rather than centralized.

Around 3pm on the 3rd we started data entry and started distributing a plea:
http://socialsource.blogspot.com/2005/09/katrina-gave-money-already-give-your.html

At 11:30 pm on the 3rd, I figured it might be good to actually write down an overview of what we had been doing for the past couple days.
http://socialsource.blogspot.com/2005/09/katrina-peoplefinder-project.html

By the 4th, it was pretty clear that we had expanded past the point of being coordinated. So we tried to get some folks to “officially” lead sections of the effort. That effort fell flat on its face mostly because their were people already leading the effort… they were was to busy doing things to have time to list themselves as a leader.

Around this time my role became “human router” I would look at the email stream which was getting absurdly large, and simply connect people with one another. Hey person A, talk to Person B before you do thing C.

The fourth was about details… getting the HTML data entry form from good enough to good, getting the PFIF documentation to a place that it was really useful for developers. Lots and lots of details…

By 3AM on the 5th, we had 10,000 records entered into the database and the volunteer effort was snowballing.

More to come..

Why do we need open source nonprofit CRM infrastructure?

Three distinct disaster response situations come up recently underscoring the need for a solid open source NPO/NGO CRM platform.

1. The Katrina PeopleFinder Project created a single unified database of virtually every missing and found person record on the web (640,000+ records).
2. The ShelterFinder Project created a comprehensive listing of Katrina evacuee shelters (from large Red Cross shelters to small 10 bed churches).
3. The Fluwiki is building an infrastructure to deal with the potential bird flu pandemic.

Each of these projects needs to quickly and efficiently build an application based on individuals and organizations (missing persons, shelters, flue victims). Each started with CiviCRM, but moved on to another solution because CiviCRM isn't quite ready.

The PeopleFinder experience allowed us to optimize CiviCRM, achieving a 100x efficiency improvement. What we need is your support to continue the development of CiviCRM and, most importantly, support its broad adoption in the nonprofit and non governmental sectors.

Open Source means that no one has to ask permission or buy a license to mount a disaster response. We don't have to wait for a philanthropically minded corporation like Yahoo to send 40 engineers to Houston. The nonprofit/ NGO sector can put together a response that leverages volunteer skills into a complete solution within a matter of days (as demonstrated by the PeopleFinder project).

Evacutating for a Hurricane Sucks

I live in Beaumont, TX, and the day after I arrived in Beaumont a few years back, I stayed in town through a "mandatory" evacuation. Didn't do it this time with Hurricane Rita.

Right now, Michelle and I are fine in Dallas in a hotel. The news coverage seems like the house might be fine as well, which would be quite a relief.

Thanks for all the inquiries after our health and well being.

PeopleFinder Project Follow-up/ Thank you.

In early September, I sent out an urgent call for resources for the Katrina PeopleFinder Project. I wanted to follow up, share our achievements, and thank you for your support.

We have combined virtually every missing and found person listing on the web (currently over 640,000 records) into a single searchable database at http://www.katrinalist.net/ in less than a week with an all-volunteer effort.


Goals & Achievements

The Katrina PeopleFinder Project set out to solve a single problem: an evacuee needed to search up to 40 different websites to find out where their loved ones were located and whether they were OK. We laid out three goals:

1. Create a technology specification for easily exchanging evacuee information.
2. Assemble and coordinate volunteers building technology to get all evacuee data into a central database provided by Salesforce.com Foundation.
3. Organizing a massively parallel volunteer data entry project to enter refugee data posted to online bullitin boards into a central database by hand.

We mobilized over 3,000 volunteers and accomplished these goals.

• The project started on September 1, 2005 with the Social Source Foundation, CivicSpace Labs and the Salesforce.com Foundation committing ourselves to the three basic goals.
• By September 5, we had finalized the Peoplefinder Interchange Format (PFIF), a technical standard for storing and exchanging refugee data.
• By September 6, virtually every message board post was hand-entered by volunteers into the PeopleFinder database (~100,000 records).
• By September 10, almost every missing and found person record on the web was searchable at www.katrinalist.net (~350,000 records).
• By September 19, over 620,000 records are searchable.
• Our data is being processed by IBM and the San Diego Supercomputer Center to form part of a central database of evacuees for the Red Cross and Microsoft.

On-going efforts

Two major sister projects have been spawned from the volunteer community:

ShelterFinder is creating a dynamic, comprehensive national listing of shelters (including small community based churches, etc).

PeopleFinder volunteers are coming together to make sure that the technology, procedures and relationships between institutions necessary to duplicate our achievements are ready for the next major disaster.


Learn More

For me, this entire experience is about non-profits having the capacity to leverage technology and massively parallel resources to better fulfill their missions... helping people. The same work I do at the Social Source Foundation.

It is time that we invest in pre-positioning technology, capacity, plans and knowledge so that we’re not building the bridge as we cross the river in the next disaster. I was proud to have led part of this dynamic, distributed effort. If you would like more information, please send me an email and I will connect you to the appropriate person(s).

We are at also offering an overview of the project in a Webinar sponsored by the Nonprofit Technology Enterprise Network (NTEN), HumaniNet, Aspiration, and the Innovation Funders Network. Hurricane Katrina: Innovative Information and Communication Responses is a free online event. You can register at http://www.nten.org/webinars.


Thanks from the many volunteers of the Katrina PeopleFinder Project,
http://katrinahelp.info/wiki/index.php/Katrina_PeopleFinder_Project


And a personal thank you from me,

David Geilhufe
Co-Founder
Social Source Foundation
Connect with me on Linkedin...

537,180 people as of 9/14/2005 5:51 PM

I think we now have most of the Katrina missing and found persons on the web in the database. You can search at http://www.katrinalist.net/

Now the Red Cross needs to get itself together and get a common database together. We keep hearing it will be Katrinasafe.com .... get on with it. There are survivors that need to know where their friends and family are.

We of course, built the PeopleFinder Interchange Format to avoid the need to have a common database, but unless it's widely adopted, it will not solve the problem.

Kieran Lal's Personal Story of PeopleFinder

The PeopleFinder effort has been incredible fast, distributed, disconnected and effective all at the same time.

If you've been involved, I engcourage you to write a personal story of your involvement. I'm writing mine now.

Eventually, we might actually figure out how a bunch of unpaid volunteers created the most comprehensive directory of survivors and missing persons on the web (and A LOT more including ShelterFinder) .

Keiran's story.

Search PeopleFinder Data

Paste this code into your HTML page to provide a search into the Katrina PeopleFinder Project data.

Katrina PeopleFinder Project

Enter a name, phone number, email address, city, zip or neighborhood of the person you are looking for. Powered by the community of volunteers from the Katrina People Finder Project.

346,521 records as of 9/11/2005 10:43 AM

This is our effort in numbers. On the 4th the technology went up. By the 6th, the majority of unstructured data was in the database (craigslist, nola.com, etc.) through thousands of volunteer data entry folks and "scrapes" of Gluf Coast News, IDRC, MSNBC and other structured sources were ready to go into the database, but not there yet.

Unfortunately, it took another couple of days working through technology issues for stuctured data to be bulk loaded into the PeopleFinder database and today we have the comprehensive database we wanted to have up by the 7th.

People ask me where survivors should search. I tell them (1) Yahoo People Finder (good name, I approve) because it constantly is crawling the various boards and resources. (2) Katrina Safe becuase it is destined to be the "official" repository. (3) Katrinalist.net since we have some records that Yahoo doesn't and our data is in a more structured format, allowing potentially better matches with partial information.

We are doing this becuase survivors need to find their loved ones now. Eventually, the technology instrastructure will exist, hopefully based on what we've done here, to make quick massive reaction a plan rather than a struggle that requires Yahoo to fend 40 engineers out to Houston and Microsoft to have engineering teams working around the clock in Houston, or thousands of Katrina PeopleFinder Project volunteers work straight through the holiday weekend.

246,108 Records as of 9/10/2005 5:20 PM

Includes scrapes of Gulf Coast News and IRDC familylinks data. As always our data contains a hyperlink back to the original source so survivors can evaluate the original source of information.

All data is in the PeopleFinder Data Interchange Format and can be syndicated via RSS between organizations (like shelters) and websites.

200,984 records As of 9/10/2005 10:09 AM

Katrinalist.net, the search engine into the Katrina PeopleFinder Project data, has over 200,000 evacuee records in it as of this morning as uploading data from other existing databases has begun. We know there are duplicates in there, but figure folks would rather find 6 records about their brother than no records at all.

The Katrina PeopleFinder Project has another 300,000 records qued up to enter the database.

Katrina PeopleFinder Project: Community-based information

Today was a bit of a roller coaster. PFIF uploads are functional on the server. We parsed our first inbound PFIF feed. Volunteer data entry started up again. We had a little community "crisis of conciousness" as it became aparent that katrinasafe.org would become a defacto standard.

And the community came together again, thought about it for a sec, figured we still had a niche, then redoubled its efforts. Very cool. Check out the Shelter Project, a massively parallel effort to identify shelters and gather PFIF compliant data for the search engine. Current community thinking seems to be that we may be more effective at nationally aggregating shelter data quickly, especially from small community based agencies.

Status of sites implementing PFIF feeds is here.

I have to say that I've been really, really blown away by the response and support that Microsoft has given to this project, both the company and the people. Here in Austin, we've not wanted for anything ... food, drinks, everything is brought in and, in some cases, brought to us at our desks. (Eating fajitas while coding can be somewhat challenging.)

From one of the developers of http://www.katrinasafe.com/, which seems to be on its way to being the central repository of survivor data. We will, of course, be providing date to it.

Next time I develop a survivors database, I'm gonna do it for Microsoft.

First PFIF feed loaded into PeopleFinder Project

I am proud to announce that the first PeopleFinder Interchange Format data feed has been read into the PeopleFinder database. Earthlink, a critical participant in the PeopleFinder Project provided the first feed this morning.

When you go viral and call in the calvalry, sometimes it pays to take a deep breath.

I have issued a call for full time project management in a number of areas to help out with the Katrina PeopleFinder Project. Our all volunteer leadership is pretty worn out, as you might imagine. Lots of new leadership is emerging in the community, so we might yet self-organize our way into an even bigger sucess (email list volumes are pretty staggering, so it may already be happening).

I would prefer a single organization came in to marshal resources, improve communications, help the community set goals and assemble volunteer teams to meet those goals from within the community. Pre-existing relationships go a long way-- you don't have to learn a new person's name every five minutes. It will be chaotic.

The team that will manage the official "selection" (which up to this point has been the first competant person to show up) gave me a great analogy. The first stage of the booster rocket has burnt itself out. We are in that momentary lull between the first booster rocket shutting down and the second booster rocket ignighting. The second stage is all about nourishing and supporting the community of both highly technical and regular people volunteers to self organize. From there, we break free of gravity.

The project management people are human routers, they don't develop project plans and just implement them. They over-communicate and create opportunities for the community to come together over certain goals.

• Priority 1. We need to get very specific offers from organizations. Include resumes.
• Priority 2. The selection team will look at your stuff. Hopefully you have already oriented yourself to what is happening in the community, or better yet, have been part of the community since the weekend.
• Priority 3. The selection team will train and orient the full-time volunteers. This process should take at least six hours, you'll talk to a lot of people, each with a different vision and view on what is going on. It is pretty much that complex.
• Priority 4. The project management team needs to enter the community and just start being human routers, being patient as volunteers check in and out, most not able to keep up with traffice on the email list, or even figure out what changed over the last 5 minutes.

How to help:

1. Contact me. I will forward offers to the selection team.
2. Be patient. Lots of people, lots of activity.
3. Up till now, we have been entriely self-organizing. You are welcome to just wade in. Like our data entry, the process of organization can be viral and massively parallel.

How to collect data on every shelter in America in 72 hours

OK, maybe not 72 hrs, but hey, who thought we could do 60,000 records in a day?

1. Build a database of shelters by calling on volunteers to record the location of every shelter in America (validate the addresses with USPS so you don't get duplicates).
2. Allow volunteers to "claim" shelters. Once a shelter is claimed, they are comitting to putting together an excel spreadsheet with a PFIF record for each person in the Shelter. Probably have to prequalify people and offer them a little training.
3. Volunteer uploads their excel sheet on the web (software can make sure its in the right format).
4. Another volunteer or maybe staff checks to see if the data looks accurate and in the appropriate format.
5. The excel sheet of PFIF records is uploaded into the database.
6. Those results are immediately avaliable at Katrinalist.net and syndicated to any other database on the web.

The prototype for this was posted today in our message boards. Sure wish we had some full-time project managers to implement this stuff so we don't have to waste more time chasing CNN stories so people can understand the value of massively parallel volunteer efforts and I could get some sleep. We could have the software up in 48 hrs, volunteers mobilized in another 24, and the whole process done perhaps 72 hrs latter.

Press discovers Katrina PeopleFinder Project, Now we just need 6 or 8 full time engineering and project managers

I love this Red Herring story. It is exactly as tired as I feel. Key point. Send us engineering teams (actually we more need engineering and project managers).

SF Chronicle
NPR
http://www.alternet.org/story/25177/
http://db.tidbits.com/getbits.acgi?tbart=08235
http://www.worldchanging.com/archives/003437.html
http://www.npr.org/templates/story/story.php?storyId=4834186

Thoughts on Implementing PFIF

The PeopleFinder Interchange Format has strengths and weaknesses. It's strength is that it can syndicate a missing persons request far and wide quickly and easily. It's weakness is that the more personal information you put in the PFIF record, the bigger a threat to privacy it is.

But what if you only put the minimal personal information into your RSS feed? Just the stuff required to identify that this is Bob Smith, lived on Sampson Street, in the 9th Ward, son of Doris and Kathy?

Privacy is not compromised and people who know one another can still find one another. The source URL field of a PFIF record can point back not necessarily to a record containing personal information, but a record containing the name, address and phone of the shelter where that person was located when the record was created. Privacy is safe and you can still find people.

Trusted providers can still exchange large data sets in PFIF with all kinds of personal data in them to facilitate automated matching and notification.

Lets build this stuff, people.

Search over 88,000 volunteer-entered records

www.katrinalist.net is live, allowing users to search 88,000 records hand entered by volunteers from forums accross the web.

Volunteer engineering teams are hard at work preparing additional data from a variety of sources for entry into the database. We hope to provide a robust repository of quality data.

We could use full-time engineering and project management teams. Your small corporation's full-time staff of 4 dedicated to the project for a couple weeks could accelerate our progress and increase our effectiveness. Please consider joining us.

Implement PFIF NOW

Evacuees Find No Easy Way Locate Family Members
by Joseph Shapiro
Morning Edition, September 6, 2005 · From the airport in New Orleans, tens of thousands of medical patients are airlifted to shelters in hospitals in often distant cities. That effort was coordinated by the U.S. military and government agencies. But there was almost no coordination to keep good lists of who was sent where.

The problem can only get worse. The Katrina PeopleFinder Project released the PFIF specification on Saturday to address just this issue of scattered refugees with no good lists of who went where. By Sunday, a massively parallel volunteer effort had hand-entered virtually every missing and found forum posting on the web (almost 80,000 records).

Our volunteers are tired. They worked all weekend and had to go back to their jobs. Companies should contribute an engineering team or two to the cause.

We need engineering teams

The Katrina PeopleFinder Project is an all volunteer effort.

Its crunch time and we need full-time engineering teams. Yahoo got a crawler up becuase they have full time resources dedicated to the project. The Katrina People Finder Project needs full time engineering teams.

Wouldn't you think a Silicon Valley technology company could spare an engineering team for a week to work with us in making refugee information more accessible to the displaced people of the Gulf Coast. Folks need to step up, we have the plan, the vision and the specification. It's just a matter of resources.

We need a salesforce.com administrator, project managers and engineers to implement the PFIF spec for other organizations (http://zesty.ca/pfif/1.1/#life-cycle), and engineers to do scrapes of external websites and put the information in PFIF.

We can create the largest, most useful structured data set on the web. We just need a visionary company to donate a full time engineering team or two.

Overall PeopleFinder Project Goals

The Katrina PeopleFinder Project is a massively parallel volunteer effort to solve the problem of dispirate refugee/ missing persons databases and forums.

1. Enter unstructured data on refugees from forums across the web to the highest data quality standards possible with volunteers giving a little as one hour of their time.
2. Enter data from databases across the web into the central database via the PeopleFinder Interchange Format
3. Minimize duplicate records
4. Support other organizations in implementing the PeopleFinder Interchange Format
5. Make the central database avaliable to be searched
6. Use the Salesforce API to implement innovative technology solutions to the missing persons problem

(1) Is currently implemented by dividing forums like Craigslist into "chunks" of about 25 records through software and/or volunteers. Volunteer data entry people "claim" a "chunk" and enter it into the central database. They follow instructions on how to enter the data to maximize the data quality. This effort entered over 68,000 records in less than 36 hours.

(2) Is currently implemented by software engineers either scraping or transforming existing databases into that PeopleFinder Interchange Format (PFIF) and then loading that data into the central database OR by missing persons database owners implementing the full ProjectFinder Interchange Format (PFIF) wich allows an RSS feed of refugee data to be passed from database to database.

(3) Is handled by trying to coordinate among all the different teams and volunteers so that duplication of effort is minimized. The "chunking" and record claiming process is also critical.

(4) Is handled by contacting organizations to make them aware of the PFIF format and helping them decide whether to implement it. We might also try to provide some volunteer assistance to sites in implementing PFIF.

(5) Is being handled by a search interface into our main Salesforce.com data repository.

(6) Once data is in the repository it should be processed to try to match missing people to found people, facilitate communication and just generally help refugees out in any way possible via data and technology.

This is a massively parallel volunteer effort. Please figure out a small part to play in these very large goals. Enlist some people to help, make sure other people know what you are doing and just go to it.

PeopleFinder Interchange Format (PFIF)

Some amazingly smart folks have completed the data specification that is the cornerstone of the Katrina PeopleFinder Project. Use it. Convince others to use it :)

The PeopleFinder Interchange Format (PFIF) is well described in the FAQ, even with a handy little diagram (http://zesty.ca/pfif/1.1/#life-cycle).

49,995 records As of 9/5/2005 7:52 PM

A volunteer data entry operation of the Katrina PeopleFinder Project has entered information on 49,995 people listed in message boards like Craigslist in about 24 hours. With extraordinary contributions from nonprofits, companies, regular volunteers and key partners like Salesforce.com, we have only just begun.

All our data conforms to the PeopleFinder Interchange Format (PFIF), allowing it to be combined with other structured databases that convert their data to PFIF. Data on missing persons from around the web can be combined into a large, central, searchable repository.

Go here for a link to the PFIF http://192.122.183.218/wiki/index.php/PeopleFinderTech

The next step is to add data to the central repository from structured databases like the Red Cross. They can help by transforming their data into something conforming with the PFIF spec. Their records can then be combined with ours allowing people to search for missing persons listed accross the web. Earthlink is the first organization to implement their database according to the PFIF specification. Great work guys.

The Katrina PeopleFinder Project is a distributed, community effort led by dozens of amazing volunteers and made possible by hundreds of people giving their time. Thank you all.

Why enter data when I can't search it?

We have a massively parallel volunteer effort going on. Step one was to design a data standard, the PeopleFinder Interchange Format (PFIF). Step two was to start up a volunteer effort to enter data in. Step three was to implement the PFIF for data entry of message boards, hence our move from www.neworleansnetwork.org to new servers. Step four is to enter all the data that is in databases on the web. Step five is to make it all searchable.

We are doing steps 4 and 5 right now. A wide variety of volunteers and companies are creating software to bring the existing databases into ours. We actually would prefer them all to implement PFIF so that we can just parse their feed and they only minimally change their site.

The search interface is being lead by Salesforce.com. The status of that project is publically viewable on the project wiki.

We're for real folks. Nothing nefarious going on.... just a lot of tired people on a holiday weekend away from their families for a good cause. Thanks to all of our volunteers. This wouldn't work without you.

Volunteer Data Entry works: 11,643 entries in Salesforce.com Central Repository at 7:13 AM up from about 7,000 at 3 AM

Add this to the 2,600 we did yesterday before we melted down the server and we have put about 14k forum postings into a database format with the help of volunteers. Good job folks!

Another Database? Aren't you part of the problem? NO!

The Red Cross has a missing/found persons database. We have identified another 10+ databases on the web. Web forums like craigslist are posting missing/found persons information there are probably 25 major ones of those.

We are consolodating all this information from all over the web. Hundreds of volunteers are entering craigslist and other forum entries by hand. 10,000 of them since yesterday.

We have published an open data spec, the PeopleFinder Interchange Format (PFIF), that can facilitate all the various databases syndicating information into a single database.

We are an open and community process working as quickly as possible to get information for individuals impacted by the hurricane. Please Join Us. http://socialsource.blogspot.com/

Hard core technology volunteers- hackers, Google engineers, super data heads are critical to the effort. Join the hard core techie email list by sending mail to katrinadev-subscribe@civicspace.org.

The hard core techie Wiki with the data standard is at http://192.122.183.218/wiki/index.php/PeopleFinderTech

3AM Central 9/5/2005. 10,000 records entered. Great Work!

3AM Central 9/5/2005. 10,000 records entered. Great Work!

The Katrina PeopleFinder Project is creating a central repository for all refugee records on the web.

Two main efforts are:
(1) VOLUNTEER
http://192.122.183.218/wiki/index.php/PeopleFinderVolunteer
Massively parallel volunteer data entry of refugee records into a central repository, conforming to the PeopleFinder Interchange Format (PFIF).
(2) TECH
http://192.122.183.218/wiki/index.php/PeopleFinderTech
Volunteer techies effort to publish the PeopleFinder Interchange Format (PFIF) and collect data from refugee databases across the web.

Great work!

http://socialsource.blogspot.com/

http://blog.social-source.com/ got slammed.

We are now at: http://socialsource.blogspot.com/

Thank you.

We need community leaders to step up.

We need community leaders to step up.

Self-organizing is hard. Thank you for investing your time. It will make a HUGE difference.

1. Pick an area.
2. Organize a leadership team on Omidyar Network (links below) [(3) is done already]
3. IM me so that I can publicize who the leaders for each team are and keep this blog post updated with the current information.
4. Leaders should post their contact information so that people can contact them.
5. If you can't lead, just subscribe to the email list (katrinadev-subscribe@civicspacelabs.org) and stand by while we get organized.

Keiran Lal (Kieran AT civicspacelabs org) will organize a conference call and IRC for latter to help with organizing.

To stay informed, send email to katrinadev-subscribe@civicspacelabs.org

If you are involved with the Katrina PeopleFinder Project, or want to be, it is critical that we improve our organization structure as soon as possible. Luckily, Jon Lebowsky has shown us the way. Thanks Jon! You can see how they organized the leadership team by following the Omidyar Network link for the volunteer data entry project.

Self-organizing is hard. Thank you for investing your time. It will make a HUGE difference.


The project has four main goals. Each on needs a few leaders and back-ups.

(1) DATA SPECIFICATION. Create, disseminate and support others in using the PeopleFinder Interchange Format.
Leader1: Ka-Ping Yee (ping AT zesty ca)
Leader2: (kleinpeterj at corp.earthlink.net)
Leader3: ?

(1a) Contact all existing structured data sites ask them to implement the data standard
Leader1: Katrina Data Project (?)
Leader2:

(1b) Help developers implement the data specification
Leader1:
Leader2:

(1c) Whatever else needs to be done.

Organize your leadership team here: http://www.omidyar.net/group/katrinarefugee/news/1/

(2) WRITE SOFTWARE. Write software that gets other people’s data into the PeopleFinder Interchange Format.

Leader1: ?
Leader2: ?
Leader3: ?

(2a) Assign folks to scrape specific databases
Leader1:
Leader2:

(2b) Create Salesforce.com central data repository.
Leader1: Jon Plax (jplax AT salesforce com)
Leader2:

(2c) Other items

Organize your leadership team here: http://www.omidyar.net/group/katrinarefugee/news/2/


(3) VOLUNTEER DATA ENTRY: Organizing a massively parallel volunteer data entry project to enter refugee data posted to online bulletin boards into a central database by hand.
Leader1: Jon Lebowsky (irc.freenode.net #globalvoices)
Leader2: ?
Leader3: ?

(3a) Don’t exactly know since Jon Lebowsky has it under control
Leader1:
Leader2:

(3b) Don’t exactly know since Jon Lebowsky has it under control
Leader1:
Leader2:

Your leadership team is already organized here: http://www.omidyar.net/group/katrinarefugee/news/0/


(4) Market the Katrina PeopleFinder Project and recruit volunteers.
Leader1: Andrew Hoppin (Andrew AT civicspacelabs org)
Leader2: ?
Leader3: ?

(4a) Recruit full, deep leadership teams for all segments of project
Leader1:
Leader2:

(4b) Other Items

Organize your leadership team here:
http://www.omidyar.net/group/katrinarefugee/news/3/

Gave money already? Give an hour of your time right now online. Katrina PeopleFinder Project

Refugees can search 20 web sites for lost relatives and still miss their entry on the 21st web site. There is a need to combine all the refugee data from big databases like Red Cross, large posting forums like Craigslist and many other sources on the web. The Katrina PeopleFinder Project seeks to create a single repository combining as many sources of refugee data as possible from all over the web without interrupting existing momentum.

We need help for both regular people and software engineers. Everybody is critical to building a central repository of ALL the refugee records we can find on the web. The Social Source Foundation, CivicSpace Labs and Salesforce.com Foundation are coordinating hundreds of people and organizations, including Craigslist and Earthlink.

Please consider giving us just an hour of you your time to do volunteer data entry. The PeopleFinder Project is seeking volunteers in four primary areas:

(1) Creating a technology specification for easily exchanging refugee information. A volunteer effort is working to assist online databases in implementing the specification.
Volunteer here (techies): http://www.omidyar.net/group/katrinarefugee/news/1/

(2) Coordinating volunteers that are writing software that takes information from online databases and putting it into a central database provided by Salesforce.com Foundation.
Volunteer here (software engineers): http://www.omidyar.net/group/katrinarefugee/news/2/

(3) Organizing a massively parallel volunteer data entry project to enter refugee data posted to online bullitin boards into a central database by hand.
Volunteer here (regular people): http://www.omidyar.net/group/katrinarefugee/news/0/

(4) Market the Katrina PeopleFinder Project and recruit volunteers.
Volunteer here (marketing folks): http://www.omidyar.net/group/katrinarefugee/news/3/

Katrina PeopleFinder Project

What is the Katrina PeopleFinder Project?
Refugees can search 20 web sites for lost relatives and still miss an entry on the 21st web site. There is a need to combine all the refugee data from big databases like Red Cross and large posting forums like Craigslist. The Katrina PeopleFinder Project seeks to create a single database combining as many sources as possible from all over the web without interrupting existing momentum.

How did the Katrina PeopleFinder Project start?
Community organizers from the League of Pissed off Voters, themselves refugees from New Orleans, needed a tool to help people in the New Orleans and Gulf Coast area stay connected to the communities they love. Enlisting the aid of nonprofit technology assistance providers Radical Designs, Social Source Foundation, and CivicSpace Labs, the site www.neworleansnetwork.org was created with open source technologies designed by and for nonprofits.

Realizing the PeopleFinder tool could be used to aggregate databases and information from around the web, the Salesforce.com Foundation joined the effort, combining corporate technology and resources with the power of the grassroots.

What is the Katrina PeopleFinder Project doing?
(1) Creating a technology specification for easily exchanging refugee information. A volunteer effort is working to assist online databases in implementing the specification.
Volunteer here: http://www.omidyar.net/group/katrinarefugee/news/1/
(2) Coordinating volunteers that are writing software that takes information from online databases and putting it into a central database provided by Salesforce.com Foundation.
Volunteer here: http://www.omidyar.net/group/katrinarefugee/news/2/
(3) Organizing a massively parallel volunteer data entry project to enter refugee data posted to online bullitin boards into a central database by hand.
Volunteer here: http://www.omidyar.net/group/katrinarefugee/news/0/

Who is involved?
The Social Source Foundation, CivicSpace Labs and Salesforce.com Foundation are coordinating hundreds of people and organizations, including Craigslist and Earthlink.

Contact
IM: dgeilhufe yahoo com

Links:
A blog to keep up-to-date (http://blog.social-source.com/)
Social Source Foundation (http://www.openngo.org/)
CivicSpaceLabs (http://www.civicspacelabs.org/)
Salesforce.com Foundation (http://www.salesforce.com/foundation/)
Craigslist (http://www.craigslist.org/)
Earthlink (http://katrina.earthlink.net/)

FINAL REFUGEE DATA EXCHANGE SPEC

Refugees can go to 20 different websites to find information on their loved ones. We are publishing a spec to facilitate data interchange among sites and that would allow the creation of a central database of most refugee databases on the web. We are also connecting database owners with volunteer programmers that can help implementing the spec. Special thanks to Ka-Ping Yee. Peoplefinder is a community effort lead by the Social Source Foundation, CivicSpace Labs and Salesforce.com Foundation.

The "official" data exchange format for refugee data is defined here:
http://zesty.ca/katrina/

If you run a refugee database, please publish your data via RSS in this standard. If you need volunteer programmers to help with implementation IM me at dgeilhufe AT yahoo com.

If you can volunteer to contact refugee databases (Red Cross, Gluf Coast, etc.) and help them implement the standard, please go here:
http://www.omidyar.net/group/katrinarefugee/news/1/

If can be a lead community organizer of programmers actually implementing the standard for websites, or for efforts to scrape databases, please go here:
http://www.omidyar.net/group/katrinarefugee/news/2/

Please diseminate this information far and wide.

Katrina: Gave money already? Give your time right now online.

A significant problem for refugees is that information on people is spread all over the web. If I was looking for a loved one, I could search 20 websites and still miss and entry about them. There are a number of technical solutions people are working on right now, but the old fashioned grassroots approach may be best.

Can you spare one hour today and enter missing persons data from forums accross the web into the peoplefinder database at http://www.neworleansnetwork.org/ ? We have people working on importing the major online databases on the web (Red Cross, Gulf Coast News, etc.) in addition to your data entry.

The first thing we need is community leaders that can help coordinate a massively parallel data entry effort so that thousands of volunteers can enter information from message boards across the web without too much duplication of effort.

The thing we need to resolve first is how to coordinate such a significant effort.

Go to http://www.omidyar.net/group/katrinarefugee/news/0/ where you will be coordinating the volunteer effort.

Please publicize this message far and wide.

Calling community organizers!!

We need some leaders to step in a organize some efforts. Are you up for it?
Email me dgeilhufe --AT socialsourcefoundation --DOT org

Goals we need leaders for:
(1) Organize a massively parallel volunteer data entry project.
(2) Organize programmers to populate a central database from the variety of databases on the web.

Katrina Refugee Database Data Standard

CiviCRM is being used for a refugee database at http://www.neworleansnetwork.org/

So we built a data standard that we are trying to get used in as many places as possible (to facilitate data exchange).

The current as final as we can get it dat standard is here:
http://zesty.ca/katrina/

Hurricane Katrina: Volunteer Data Entry

Information on missing persons is in hundreds of forums accross the web.

If a thousand people spend just one hour today entering information from forums into a database, hurricane victims will have a much easier time finding out the status of friends and loved ones.

We have a repository for this data at http://www.neworleansnetwork.org/ a grassroots website built with grassroots technology (CiviSpace & CiviCRM) by grassroots folks Radical Designs and League of Pissed Off Voters.

Step One:
Figure out how were going to coordinate a massively parallel volunteer data entry effort.
Sites that need to be entered by hand (we can't write software that will put their information in our database) are tagged in delicious.
http://del.icio.us/tag/peoplefinder+unstructured

Step Two:
Write clear instructions on how to participate.

Step Three:
Let as many people know about it as possible.

We are on step one. Send me an email at dgeilhufe at-- sourcialsourcefoundation dot-- org to help with the planning.

Current organizing locations/ how to help with central "distributed" database

OK, things are slowly taking shape.

First email coordination:
Our friends at civicspace labs have put up an email list:

If you are a techie/programmer, please drop me a line at david at-- socialsourcefoundation dot-- org and we'll get you on the mailing list.

Non-techies, please stand by :)

Second IRC:
irc.freenode.net
either #civicrm OR #civicspace
Coordination was going on there last night.

Screen scrapping targets

We want to get as many missing persons records into a single database as possible.

(1) Get techies to build software to "scrape" info from structured sites into a standard datamodel.

(2) Get volunteers to read forums and postings and manually enter that data into a standard datamodel.

Here are some of the places we need to have scraped/ entered into our standard datamodel. We'll of course be syndicating the unified database.

Lists of lists:
http://www.packrat-pro.com/katrina.htm
http://www.blessingsontheweb.com/

Red Cross has got their database up which is huge, so that is another good scrapping opportunity.
http://www.familylinks.icrc.org/katrina/people

The big list so far has been the Gulf Coast News site.
http://wx.gulfcoastnews.com/katrina/status.aspx

Our goal, again, is to combine all these databases into a single data model, offering a unified view of who is OK so refugees don't have to spend their time searching message boards and databases.

Ecosystems are not about design or control

Marnie Webb always triggers a good thought or two:

It’s about turning over part of yourself. In two essays (1, 2) Peter Merholz argues that it’s not about the technology. He writes: “Web 2.0 is primarily interesting from a philosophical standpoint. It’s about relinquishing control, it’s about openness, it’s about trust and authenticity. APIs, Tags, Ajax, mashups, and all that are symptoms, outputs, results of this philosophical bent.”

In a Social Source Ecosystem, openness, trust and authenticity are cultural norms. The actual code and behavior of actors are just "symptoms, outputs and results of this philisophical bent."

A Social Source Ecosystem: Catalysis

Catalysis-- an action or reaction among actors that is triggered by an outside agent--a very small amount of catalytic agent can facilitate a very large scale reaction.

Open source software becomes a catalytic agent that brings new actors into the social source ecosystem. Recently an online petition application was released using the CiviCRM API to store data. An entirely new community of users that care about online petitions will now be exposed to CiviCRM and potentially join our community and leverage our software.

One of the key factors for a successful catalyst is an environment conducive to broad cooperation across organizations, markets, commercial products, and human activities. In the software world, this means standards. TCP/IP, HTML and XML provide the basic technical standards, but for a Social Source Ecosystem, standard representations of actions, online donations, contact records, etc. Become just as important.

In the end, group forming networks are probably the most relevant mechanism for large catalytic impacts in the Social Source Ecosystem.







The implication is that the Social Source Foundation cannot be a central clearing house, but instead focuses on the basic rules and tools that allow any individual or group to immediately use CiviCRM for their purposes, creating a new group around their specific needs.

Social Source Socialist?

Oh, how I love it when corporations make arguements that can be applied in different ways. In the fundraising software space, there is a debate about people filing business process patents for specific online fundraising methods. The most noise comes from software vendors that will lose business if a single vendor is the only one able to offer a specific business process.

One arguement against business process patents is:

Shouldn't technology enable us to do more and to do it more effectively? Restricting our use of fundraising tools limits the number of people we can engage, the volume of donations we will receive, and ultimately, the universe of people we can help.

So I gotta do it:

Shouldn't technology enable nonprofits to do more and to do it more effectively? Restricting nonprofit use of fundraising tools (through expensive proprietary software licenses) limits the number of people nonprofits can engage, the volume of donations nonprofits will receive, and ultimately, the universe of people nonprofits can help.

This is not an arguement to make things free (the social source ecosystem depends on customer revenue), just an arguement to radically reduce barriers to adoption. ;)

A Social Source Ecosystem: Group Selection

In one of my early presentations on the potential of open source software for nonprofits, I had a slide called “Sounds like Socialism.” The slide tried to address a perception that a utopian view of sharing software and innovations simply couldn’t work. In our market-driven capitalist society, if you don't buy it, it can't be valuable.

The concept of group selection injects a decidedly cut-throat capitalism aspect into the concept of Social Source. The Institute for the Future notes,

“…groups work best when their members provide benefits to one another, but many of these prosocial behaviors do no survive through natural selection.” Individuals who effectively compete with other individuals succeed in evolution; those that cooperate are less successful.

How then do we conceive of a Social Source ecosystem? If it’s not one group of individuals and organizations sharing software in a utopian collaboration of nonprofits, what is it?

Group selection is the concept that individuals are not the only ones subject to natural selection—natural selection also operates at the group level—groups of individuals. Cooperation within a group can be a very important asset when competing against other groups.

This has implication for the Social Source ecosystem competing with the commercial software ecosystem, but also for the internal organization of the Social Source ecosystem.

Applying this to the Social Source ecosystem suggests there is no monolithic groups or single leader. Instead, some core principles (open source licenses) simply guide the entire system. Conflict and competition at a wider scale in the Social Source ecosystem will encourage local cooperation in order to compete in the wider group.

In the case of technology like CiviCRM, there is a strong incentive to support the creation of multiple donor database solutions, multiple volunteer management solutions, etc. The competition among those solutions will actually make those solutions stronger.

This leads both to competition among groups sharing volunteer management code for their specific solution, but also supports much broader cooperation as multiple volunteer management solutions share innovations at the CiviCRM level.

Things are getting curiouser and curiouser as I think through how a Social Source ecosystem might work.

Social Source Symbiosis

The Institute for the Future's second concept for a new literacy of cooperation is symbiosis.

"...a mutually beneficial relationship that can evolve between different organism's in a system."

Symbiosis is not an instant thing, one doesn't issue a press release and announce a symbiotic relationship with customers, partners or vendors. Over time, reciprocal actions build relationships, yet reciprocity is a hard problem.

In a Social Source world, symbiosis is created with the winning strategy for a game of "tit for tat."

1. Be nice - don't defect at the first opportunity.
2. Retaliate - defect if others do.
3. Forgive - switch to cooperation when your opponent does.
4. Be clear - always react in the same way to your opponent's behavior.

In a Social Source world, you create open source software and share it with the world (be nice). If vendors and consultants choose to use the software without contributing back to the community, you withhold engineering support, priority bug fixes, and custom feature implementation (retaliate). When a vendor or consultant changes their minds and starts contributing back to the community, actively support their success (forgive). And all the while, communicate what a Social Source value system and ecosystem look like for others can behave as is expected (be clear).

Already, CiviCRM is becoming both endosymbiotic and exosymbiotic in the Social Source ecosystem. Endosymbiotic means one organism is literally inside of another. This is the relationship between CivicSpace and CiviCRM with CivicSpace's 0.8.2 release. Neither piece of software is entirely "whole" without the other.

CiviCRM is also exosymbiotic with content management systems like Drupal and Mambo. The pieces of software reciprocate (track a common set of users), but are seemingly distinct and can operate entirely separately. Ultimately, as CiviCRM becomes the basis for donor management, advocacy, and case management applications, a network of symbiotic relationships will evolve.

One interesting lesson from biology is that parasitism drives rapid evolution. In a Social Source ecosystem, not everyone needs to or even should cooperate and collaborate. Individuals and organizations that adopt software by "defecting" in the game of tit for tat by not contributing code and innovations back into to the community.

Social Source Sync

Synchrony: the process by which patterned behavior is created among many individuals without conscious control.

In a Social Source ecosystem there are a lot of actors: developers (building software), hosters (providing software as a service to customers), integrators (modifying software for customer needs), customers (using the software), etc.

Traditional proprietary software vendor models try to coordinate those actors under a single vendor. Developers have to use the SDK (software development kit) approved by the intellectual property owner. Hosters have to pay the intellectual property owner royalties. Integrators are only available from "partner programs" organized by the intellectual property owner. At least some portion of the customer spend to use the software goes to the intellectual property owner.

Even in the nonprofit sector this command and control model is prevalent.

The rational for this command and control model is that the resulting system has rules, standards and is predictable... if no one is in control, that would be too risky.

The concept of Synchrony challenges this analysis. If you have a system of actors that is communicating and engaging in some type of rhythmic give and take, those actors will, over time sync up with one another.

In a Social Source world, software developers, integrators, hosters and customers all communicate with one another, ask one another to meet their need, and contribute innovations back and forth. Over time, this rhythmic give and take yields coordinated cooperative action.

...the combination of strong and weak links can create unexpected and spontaneous outbreaks of coordinated behavior across decentralized networks.

The partnership between the Social Source Foundation and CivicSpace labs to build and deploy CiviCRM is a good example of strong links in the network. Each organization also has many weaker links in the nonprofit technology sphere, the political sphere and the open source community.

This emergent Social Source ecosystem has already begun rhythmic oscillation... PicNet and CivicActions are starting to use the technology for customers and are increasingly communicating with the partners with the strong links (CivicSpace Labs and Social Source Foundation). Over time, if enough actors join the system, communicate and exchange innovations, unexpected coordinated behavior should start breaking out across the network.

We believe these yet to be discovered opportunities for coordination will create major positive changes in the field of social purpose technology for non-profits and NGOs.

Small pieces loosely joined part II

In my previous post I talked about how venture capitalists see the next generation of technology investments. Others share a common view of the future:

"Connective and pervasive technologies are enabling new forms of human and machine interactions and relationships; they will present business [& nonprofit] institutions with a host of new possibilities for organizing people, processes, relationships and knowledge." (Institute for the Future)

In a Social Source world, the participants (vendors, consultants, customers) define new possibilities for organizing people, processes, relationships and knowledge with the goal or improving their own organization, and as a by-product, make their innovations available to a broader community, accelerating the adoption of their innovation. This is a function of open source licenses and open, modular architectures based on standards.

The key here is that organizations pursue their own interests. The Social Source ecosystem and licensing structure, which enable organizations to pursue their own interests better, faster, cheaper, makes it easy and automatic to share innovations with no action on the part of the "customer." The customer just needs to pursue their own interests.

The second notable issue is the critical nature of CRM to this vision of a new world-- organizing people, processes, relationships and knowledge.

This is why our fist piece of software is CiviCRM, a constituent relationship management engine. It is all about providing a common, open framework for organizing people and relationships. We fully expect other software will use CiviCRM as a common CRM repository.

Ultimately, this makes interoperation and compatibility between a CiviCRM-based knowledge management system and perhaps a CiviCRM-based donor database easier and more powerful.

The Big Idea

The Institute for the Future observes that we are in the middle of a collision of different forces:

1. "Companies in emerging high-tech industries have learned that working with competitors can build markets and help avoid costly standards wars."
2. "The open source movement has shown that world-class software can be built without corporate oversight or market incentives."
3. "Outsourcing has turned competitors into common customers of design firms and contract manufacturers."

Put these ideas together and you begin to see how far behind the curve the nonprofit technology sector is. Sure, where NPO technology and the broader technology market intersect, we see these themes -- computers, operating systems, groupware -- but in the nonprofit-specific technology sector, we see a market lagging behind. I'm talking about things like donor databases, advocacy platforms, membership management, etc.

Leading firms don't work with one another to build markets, but focus on building proprietary software platforms that seek to be all things to all people (Kintera and Blackbaud illustrate this point) or focus on a small little niche (membership management & case management software). Nonprofits still see open source as "iffy,"built by volunteers and not a viable option compared with proprietary commercial solutions. And finally, the nonprofit technology market makes little use of contract manufacturers/ design firms, instead vendors tend to follow a vertical integration strategy, providing customers with all the services related to their market -- installation, training, support, customization, etc.

A Social Source world actively embraces these themes and put them to work for the customer.

Social Source encourages competitors to work together and build standards that enable customers to switch from vendor to vendor without barriers. Imagine buying advocacy software from Kintera and deciding that you would prefer a consultant and technology support staff based in your city so you can get face to face treatment. Perhaps GetActive has staff in your city. Could you imagine switching from Kintera to GetActive with little pain and trouble? This starts to become possible if the two companies used the same open source technologies & software and agreed to the same standards. If both companies used the open source CiviCRM constituent relationship management core, the migration of basic CRM data would be relatively simple.

In a Social Source world, the software options are not created by a nonprofit staffer with little time and no formal software engineering training. Instead, very robust core software is built by qualified software engineers with the support of a broad open source community. Intermediaries like the Social Source Foundation and Aspiration provide governance, guidance and community building for the open source community.

Social Source means that when customers hire vendors, the project contributes to the whole community through the magic of open source. Competing vendors have an incentive to share innovations and employ the same software engineering organizations to create solutions to their customer needs. Doing so, they can deliver the functionality customers need at a low cost to the vendor.

But this Social Source vision requires a new ecosystem of integrators, hosters and developers to evolve. It requires customers and consultants and intermediaries to start understanding the dynamics that shape the new, new economy. And it requires imaginative and inventive people, funders and intermediaries to take the lead.

"Apps on top of apps on top of apps": small pieces loosely joined

From the pages of Business 2.0

Fred Wilson was a VS at Flatiron Partners during the bubble. He has recently started Union Square Ventures after asking the hard question, "Is profitable VC investment in technology innovation dead?"

His answer: "The reality is that core technology investing- everything from chips to enterprise software to communications equipment, all the stuff that big companies buy- has been on the wane for five to 10 years. So what's the next wave? The next wave is what we're calling applied technology. The Internet is a computing platform built on top of the core technology. Applied technology is what gets built on top of that: It's Web services."

CiviCRM is a great example of this. It is both built on top of core technologies (Internet, PHP, MySQL, Apache), but it is designed for others to build on top of it. Apps on top of apps on top of apps.

While the commercial world tries to figure out how to extract maximum profits at each layer, the Social Source approach is to build the core layers and let nonprofits pay consultant to fill in the blanks. As those blanks are filled in with contributed software, the entire sector gets better, more efficient and more affordable tools.

Effective, affordable and ubiquitous nonprofit technology

Social Source is a vision and practice of software development that specifically meets customer needs. Social Source changes the institutional and systemic incentives that govern the behavior of actors within nonprofit software development.

It starts with a simple idea:
Software customers need effective, affordable and easy solutions.

Commercial vendors in the proprietary world say they provide effective, affordable and easy solutions, but their definitions of those terms may not match the customers'.

**Effective**
Vendors tend to provide effective solutions for the problem that drives the purchasing decision. I buy effective fundraising software today, but in five years my needs have expanded and when it hasn't been upgraded because the vendor needs to maintain their profit margins and cannot afford to expand R&D, it might no longer be so effective.

Social Source uses open source development methodology to deliver effective solutions across the life cycle. CiviCRM, the Social Source Foundation's new nonprofit-centric contact and relationship management solution, can remain effective over the entire software life-cycle IF an social source community of volunteer and paid developers, consultants, vendors and users embrace the platform.

Commercial vendors only provide an effective solution if it is profitable. Blackbaud's Raiser's Edge was recently criticized by the transgender community because their demographic profile only included male and female. With an open source solution like CiviCRM, if the maintainers of the software won't change their software, you can hire someone or do it yourself without violating licensing agreements.

**Affordable**
Affordable to a vendor means the highest possible price that the market will bear. We call this in economics profit maximization. Customers clearly get value... they are willing to pay for it.

With open source software, there is a natural competion among vendors. You can only buy Kintera Sphere from Kintera corporation. With an open source solution like CiviCRM, you can buy the solution from any developer/integrator/consultant that understands how the software works.

The dual upside/downside is that this provides a large variation in vendor quality from college students on summer break to major corporate systems integrators. The customer will need to choose the best providers, but they can choose from a wide variety of price points and service levels so that the customer's specific needs are met. If you want Kintera Sphere, you pretty much only get Kintera Sphere at Kintera's price point and service level.

We call this the ecosystem around a piece of social source software.

**Easy**
Proprietary vendors tend to make using software easy. Getting, installing, modifying, customizing, understanding, etc. are all too often difficult in order to generate additional fee income that supports profit margins.

Social Source vendors have incentives to deliver "easy" in all aspects of their services. As they compete in the marketplace, they can't afford make any stage of the process difficult... the competition will acquire customers by making what was once hard, easy.

So how does this Social Source idea work all this magic? We'll take a deeper look in the next few days.

Building a Social Source Paper

Seems like all good ideas have a white paper... Network Centric Advocacy, Movement as a Network, Three Pillars of Social Source.

But I don't have that kind of attention span. So I thought I'd try to build a white paper from component ideas. A blog seems like a reasonable tool.

Have you ever read something that resonated so deeply with your own ideas it gave you new energy? The paper "Toword a New Literacy of Cooperation in Business" from the Institute for the Future, did it for me.

Plus it provides an organizing principal for my Social Source white paper.

Now every white paper needs a title, so here are some I came up with:

Social Source: An opportunity for nonprofits to be on the cutting edge of global change.
Social Source: Visioning a new model of nonprofit technology.
Social Source: Catalyzing a new model of social purpose technology.

I'll tell you what the paper is about tomorrow.

Open Source and "White Label" Applications

Long ago I made the arguement that all these nonprofits that build custom applications really need to get together and form a consortia to lower the costs of application development. Yes, there are many logistical hurdles, but two nonprofits with the same needs can get a custom application by commissioning the work as a consortia than each building a seperate custom application.

As CiviCRM begins to take shape (we released 0.1 and are working on 0.2), it is becoming easier and easier for nonprofits to band together and build "white label" applications on top of CiviCRM. Mark Sherman at the Progressive Technology Project has a problem... his members, grassroots organizing groups, have a common database need which they have all solved in different ways. By banding together they can commission a grassroots organizing platform that will serve their collective needs?

This works really well when the underlying technology is open source. No one owns the technology and everyone owns the technology. As a practical matter, it means a group of nonprofits is not locked into a single vendor and can be assured that they have the rights to do whatever they want with the technology that is produced.

What other unbuilt applications are out there that NPOs could collaborate on?

Both/And - Open Source and Commercial Worlds

Sonny Cloward takes me to task (nicely) for my tendency to create a black and white distinction between open source and commercial/proprietary models in the nonprofit sector.

And to be honest, I am in a continuous struggle between marketing speak (simple statements that get core ideas across) and a more sophisticated discussion. The black/white stuff is more marketing speak than anything else-- the real world is more complicated.

So let me try to lay out some core principles:
(1) Amortization. The cost of software should be amortized across as many possible users of that software. If I have 1,000 customers, they each pay 1/1000th of the development costs.
(2) Affordability. Technology should be available and affordable to the broadest range of the nonprofit sector as possible.
(3) Customer Control. Decisions about software functionality should be as close to the customer as practical. If I need the thing to do "A", I or a consultant I hire can make it do "A".

Some more general core principles include those from the National Strategy for Nonprofit Technology:

• Technology Transparency
  
• Open Systems
  
• Fair Exchange
  
• Fair Compensation
  

OK, so leading with the principles, it is important to realize "open source" meets only one of the principles- open systems. By itself, it probably creates more problems than it solves.

The current implementation of Salesforce.com for nonprofits meets a whole lot more of these principles than open source alone. In fact, it was for that reason that Paul Hagen and I went to Suzanne DiBianca, the ED of the Salesforce.com Foundation, a couple years back to accelerate its deployment and suitability for NPOs.

The thing I call Social Source brings all these principles together into an ecosystem that serves customer, rather than vendor needs. It is the ecosystem of consultants, integrators, developers, and customers that create the benefit for the nonprofit sector. I can certainly do this in a proprietary model if I can convince the community to give up their intellectual property rights.

Open source is the foundation of the social source approach because it puts everyone developers, integrators, hosters and customers in an equal power position. Everyone controls the technology and no one controls the technology. Innovations are automatically shared (primarily through the community, but reinforced by the license).

If tomorrow Oracle made a hostile bid for Salesforce.com, the likelihood of them maintaining their services for nonprofits would be in serious doubt. This is not a reflection on the good hearted people at the company, it is simply the reality of U.S. corporation owning intellectual property--when ownership changes, the new owner calls the shots.

Again, let me be clear, if I were advising a client today I would encourage them to go to a trustworthy vendor (and there are lots of them out there) and purchase a solution. It will work for them and there are currently no good alternatives.

If I'm talking to a Foundation about how to improve the way the nonprofit sector uses technology, I'm going to make the case that we need a new model- social source.

Open source is a necessary, but not sufficient condition for this new model. We still need vendors, economic transactions, NTAPs, collaboration, educated customers, support and training options. But when you build your model on open source, control devolves to the folks with the need...the customer. This is not such a bad thing.

Further reading on Social Source:
Jonathan Peizer
Gideon Rosenblatt

Death of Mission-based Software Developers?

Groundspring's announcement that they are getting out of the software development business marks a sad day nonprofit technology. Combined with the recent firing of Microedge's CEO, there are some interesting cracks appearing in the proprietary commercial model of nonprofit technology. Lucy Bernholz makes a great observation:


"Looking over the landscape of markets, vendors and products, I can tell you what I think will happen next. And it won't involve the customers (read: foundations, nonprofits, donors) getting what they want or need UNLESS they act faster than they've ever acted, in ways they haven't before, and with an eye to the motivations of market forces that are virtually foreign to them."

She's still stuck in the paradigm of proprietary commercial software vendors, but she well articulates the need for mission-based software developers to serve the desires and needs of nonprofits. The commercial world hasn't and probably won't be able to meet those needs.

We think the future is a network of commercial and nonprofit entities that drive an open source community that develops software. We think nonprofits need to:

1. Act faster than they have ever acted before in adopting alien technologies and business models (open source).
2. Act in ways they have yet to understand (partners in open source communities that develop software).
3. Keep an eye on market and social forces that are foreign to them (open source ecosystems).

It's a radical thought, but one with significant potential to meet nonprofit technology needs in an affordable, ubiquitous, effective,and understandable (to nonprofits) way.

Where Have The Values Gone?

I think I sat through the seminal NTC experience-- the introduction to fundraising session with Cheryl Gipson (Groundspring), Allan Pressel (CharityFinders), and Cathy Packard (Ctr for Nonprofit Magmt, Dallas).

You had someone whose mission is to help nonprofits with technology (Cheryl), someone whose mission is to sell nonprofits technology (Alan), and someone whose mission is to provide effective consulting (Cathy). Basically, in any session where a guys says "I'm not here to give a sales pitch" and then proceeds to demo their product, you are pretty safe in assuming his primary purpose for being in the room is to generate sales leads.

So why is NTEN hellbent on making NTC a tradeshow rather than a national Nonprofit Technology Conference? It is great for vendors, since they can mask their product demos under good titles like "Introduction to online Fundraising". Its not so good for a nonprofit seeking vendor neutral information on nonprofit technology issues.

Why aren't the vendors locked in the exhibition hall? Why do we call it a science fair instead of an exhibition hall?

Tooo many questions. For me the answer comes down to history.

When the Nonprofit Technology Enterprise Network (NTEN) was conceived, this was the vision (quoted from the National Strategy for Nonprofit Technology-NSNT).

While our original commitment was to analyze and map needs and opportunities, this process has taught us that effecting a solution requires a fundamental breakthrough that represents a collective change in behavior for the entire sector. We need a "big bang" to change fundamental assumptions about how the sector moves forward a new way of thinking and of working together that shatters old assumptions and creates a new sense of possibility. There are four core principles we believe can fuel this "big bang": Technology Transparency, Open Systems, Fair Exchange, and Fair Compensation.

• Technology Transparency is the idea that information technology should be a tool whose suitability, benefit, and ease of use makes it employment second nature (like the telephone).
• Open Systems is an approach to technology innovation that emphasizes continuous contribution by many authors, with the results owned by no one, and by everyone.
• Fair Exchange is the principle that those who receive the benefit of another’s technology should in some fashion reciprocate, propelling still more forward movement.
• Fair Compensation is the idea that those who bring their time and talents to the cause of empowering nonprofits with technology deserve due recognition, financial and otherwise.

We believe that if all players commit to these principles and to working with those who also commit to the principles nonprofits will realize the vision to use technology well, funders will have the confidence to support such work, and technology assistance providers will be most effective and creative.

It is telling to me that NTEN has chosen to move away from these core principles, this vision for a healthy nonprofit sector, and to a corporate model of vendors, customers, sponsorships and "schwag".

Perhaps that is part of the reason that nonprofits have yet to use technology well, funders don't support the work, and technology assistance providers have been transformed into vendors.

NTEN without a voice

OK, how silly is it to get to a conference for a little evangelism on CiviCRM and lose your voice? I had a sore throat on Wed and spent just a little too much time talking. I think, perhaps, that the universe is giving me a message-- talk less, listen more. So when you see me at the NTEN NTC, please feel free to carry the conversation ;)

Vision, Action and Inclusion

So I like to talk about a Social Source Ecosystem. The idea that there can be lots of developers, consultants, intermediaries, customers, and users all centered upon some mission-focused, nonprofit-specific software.

For an ecosystem to work their needs to be nourishment and symbiotic relationships among the different players. Since we live in capitalism, money often becomes the logical nourishment. But capitalism is not so good at creating non-monetary symbiotic relationships. These are the keys to making real social source ecosystems work. Relationships between players where no money changes hands, but enough value changes hands that those players become inextricably linked to one another's successes and even failures.

The ecosystem comes about because their is a vision of what can be and an evangelism that shows the different players that there is value in participating. The ecosystem must also be accessible to all-- be inclusive of all the players.

So once you all the players out there and they are all start taking action, conflict arises.
To gain nourishment in the ecosystem, players have to begin taking action. They sometimes overwhelm other players in the ecosystem, they sometimes cooperate with others in the ecosystem, but eventually the system finds a balance.

So the conflict becomes:
Vision requires collaboration to help others see what you see
Inclusion is best supported in collaborative cooperative environments
Action often requires autonomy
and the general nature of an Ecosystem is that there are winners and losers.

Now put that all together and it becomes very difficult for a single person, a single organization or a single entity to catalyze a Social Source Ecosystem.

Real Partnerships

Most of the ideas around Social Source revolve around open, cooperative partnerships with key players in the nonprofit technology space. These partnerships are very much along the lines of the very simple kindergarten lesson: "Share".

Institutionally, it is critical (in a very practical way) that there be common, concrete interests. I like to think that if both organizations were going to independently invest their own resources in in doing a project, then that project is a good practical candidate for collaboration. But if you are a real go-getter, the fact you are willing to invest your own resources from the start tends to mean it is more efficient to do the project yourself rather than incur the overhead of a partnership.

How do you surmount this catch-22 and make close collaboration part of your organization DNA?

The Values of Openness

Sometimes I feel like the people get confused between the value of openness and the values of openness.

In this day of Venture Capitalists investing in Open Source "plays" (Draper Fisher Juvetson investing $2m in SugarCRM), there is perceived value in openness. John Roberts, the CEO of SugarCRM has a great quote: "We're going to destroy a $6 billion [CRM software] market and turn into a $1 billion market." Left unsaid is 'Us getting a slice of a $1 billion dollar market is a lot better than not getting anything in a $6 billion dollar market.'

SugarCRM recognizes the value of openness. The monetary value.

Now a little outfit called TigerCRM comes along and in the grand tradition of open source software, forks the SugarCRM code per the terms of the SugarCRM license. They take a copy of the code and run with it.

An engineer at SugarCRM got angry because their work was essentially copied and used.

Somewhere along the line, they didn't understand the values of openness.

To me, the values of openness are pretty straight forward:

• Share. Because in Kindergarten they taught me I would have a lot more fun with my toys if I played with them with others. I think they were right.
• Collaborate. Two heads are better than one. Very true in software development.
• Focus. Do what you do. Don't worry about what others do.

If you are in Open Source for the customers, for the business model, or for the money, you're probably stuck on the value of openness.

Take a step back, breath, and decide if you also believe in the values of openness.

Trains, Stations and Communication

I like to compare CiviCRM to a train leaving the station.

It has mass, it has momentum, it is on a track and its basic direction is set.

But even among trains, there are significant differences. When you travel on a TGV out of Paris, if you're not on the train when it leaves the station, your not going to get to your destination. Period.

If you travel on the stereo-typical India National Railways local, it looks something like this:



That is kind of how we envision CiviCRM. It moves slowly, people jump on and off, as long as you don't mind riding on the roof for awhile, you are welcome to join us.

http://objectledge.org/confluence/display/CRM/How+to+Participate

Lest people mistake the intent of my last post, we have no intention of leaving anybody behind. But the train is moving.

The CiviCRM train starts to leave the station

So we put out a public announcement today to most of the relevant lists about the CiviCRM documentation. CiviCRM is designed to be a GPL contact and relationship "engine" to handle NPO/NGO relationships for other NPO/NGO software applications.

For all the folks that got 50 copies of the announcement, sorry. I think that underscores the fragmentary nature of the NPO/NGO software development space that we all subscribe to the same 20 email lists to stay connected.

Anyway, pretty exciting stuff:

Please review our documentation. We need all the help we can get!

[apologies for somewhat excessive cross posting]



CiviCRM is seeking input on the design documents for

our open-source (GPL) relationship-management

application for the NPO/NGO sector. Four full-time

developers will start coding CiviCRM to the design

documents on 2/14/05. Get your input in by then!!



~~~~~~~~~~~~~~~~~

WHAT IS CiviCRM?

~~~~~~~~~~~~~~~~~

CiviCRM is a LAMP-based, open-source project to create

relationship management software for the nonprofit and

nongovernmental sectors. CiviCRM stores information on

the universe of people associated with a nonprofit

organization and on their interactions (emails,

donations, petitions, events, etc.).



http://objectledge.org/confluence/display/CRM/CRM+Project+One+Pager





~~~~~~~~~~~~~~~~~

INPUT ON WHAT?

~~~~~~~~~~~~~~~~~

CiviCRM manages contacts and relationships for other

software applications. Initially CiviCRM will manage

contacts and relationships for the Drupal/ CivicSpace

platform (www.civicspacelabs.org).



For other software developers to use CiviCRM, they

need to know the data model and the public API. The

data model defines what is available (individuals,

households, organizations, relationships, groups,

actions) and the API defines how developers access and

manipulate that data.



To review documents, follow the hyper link, scroll to

the bottom of the page, and click the “add a comment”

hyper link.



Feature overview (high level):

http://objectledge.org/confluence/pages/viewpage.action?pageId=592

Data model:

http://objectledge.org/confluence/display/CRM/Data+Model

Public API:

http://objectledge.org/confluence/display/CRM/CRM+v1.0+Public+APIs





~~~~~~~~~~~~~~~~~~~~~~~~~~~

HEY, I’M NOT A CODE-HEAD!

~~~~~~~~~~~~~~~~~~~~~~~~~~~

If the documents above make your head hurt, you can

still make a valuable contribution! Write a user

narrative for us about how a NPO/NGO would store,

group and manipulate information about individuals,

households and organizations.



User narratives are plain English stories about what a

NPO/NGO would do with CiviCRM.



To contribute a user narrative, follow the link below,

read the examples, scroll to the bottom and click the

“add a comment” hyper link.

http://objectledge.org/confluence/display/CRM/User+Narratives





~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

THIS IS COOL, HOW CAN I HELP IN MY COPIOUS SPARE TIME?

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

CiviCRM is an open, community process. We encourage

broad participation. The more you contribute time and

effort, the more impact you will have on CiviCRM.



Find out how to participate here:

http://objectledge.org/confluence/display/CRM/How+to+Participate



Alternatively, drop me, David Geilhufe, an email at

dgeilhufe AT-yahoo DOT-com. We’ll find a good way for

you to contribute and get all your questions answered.



Please distribute this announcement to anyone that you

think needs to know about CiviCRM



David Geilhufe

CiviCRM Team Member

dgeilhufe AT-yahoo DOT-com

Standards, simplicity, power and flexibility

So we're building this CRM "engine" for nonprofit and nongovernmental organizations, CiviCRM.

As we design the data model, we are presented with the standard trade-offs. Who are we building for? The software developer that will build a software application using CiviCRM as a repository for their contact and relationship information? The nonprofit or non-governmental organization that will store their contacts and relationships in CiviCRM?

I'm coming quickly to the conclusion that power and flexibility are FAR less important than standards and simplicity.

Let me give you an example. A software developer builds a program like Advokit, which tracks the fact that "George" (fname for contact_ID=1) is a "precinct captain" (relationship_type) for "Democrats of Peoria" (name for contact_ID=2) using CiviCRM.

Another software developer writes a program that allows a nonprofit to track the hours worked for both volunteers and employees. In their model, they basically enable their functionality for any record in CiviCRM with relation_type=volunteer.

Democrats of Peoria is currently using Advokit and now wants to add hours tracking.

If CiviCRM focused on power and flexibility, it would be up to the user to do some stuff to make everything work. They would specify that "George" is a "volunteer" for "Democrats of Peoria" in addition to being a "precinct captain", allowing George's hours to be tracked. If "George" latter becomes a paid "canvasser" for "Democrats of Peoria", then the NPO would have to remember to go back and updates all of George's relationship_types; in this case changing him from a "volunteer" to an "employee".

If CiviCRM focused on standards and simplicity, the nonprofit should have to do very little to make the two systems work seamlessly together.

CiviCRM would implement certain "reserved" relationship_types. One would be volunteer, another would be employee. Every custom relationship_type would have a child relationship to a reserved relationship.

The Advokit guys would create the custom relationship_type "precinct captain" as a child of the reserved relationship_type "volunteer". George would start out automatically as both a "precinct captain" and a "volunteer". When George becomes a "canvasser" he automatically becomes an "employee" because the custom relationship_type "canvasser is a child of "employee".

At this point folks go, "What is this guy smoking? Have we got to the standards and simplicity yet?"

The standards are complex: we enforce a taxonomy on people developing software that uses CiviCRM as an engine for contacts and relationships.

The simplicity comes in from the prospective of the nonprofit using the system.

Democrats of Peoria implements Advokit making George a precinct captain. A week latter, they implement hours tracking. The hours tracking software works seamlessly (as a "precinct captain", George is already a "volunteer" courtesy of the software developer working within the standards). When the nonprofit re-classifies George as a canvasser, hours tracking works seamlessly-- no updates are required in the hours tracking software (because "canvasser" is a child of "employee" courtesy of the software developer working within the standards).

Isn't it better for us all to be 100% customer focused, no matter what the technical implications?

Create a living Nonprofit Tech Taxonomy

Marnie Webb had a good idea to stop talking about something and actually try a little action... imagine that.

To quote Marnie:
"So, more precisely how does this work? Sign up for a del.icio.us account. Put the bookmarklets in your link bar. And then, when you are on a site that has some link to nonprofit technology -- and you decide whether you think it does or not, there are no rules -- bookmark it. Apply the tag "nptech". That's going to allow us find everyone's bookmarks. After that, use whatever tag you'd like."

Marnie's entire coherent logic for the thing is here: http://ext337.blogspot.com/2005/01/time-to-get-serious-about-nonprofit.html

The original thread that started this discussion is here:
http://www.omidyar.net/group/compumentor/news/11/

When you make the decision to "Just Do It"

For years I've been selling a vision of a nonprofit sector powered by open-source, mission-focused, enterprise software and an organization of sufficient scale to drive that vision. Over the holidays I finally found the folks that are taking the plunge. (I too decided to get wet)

Large things don't start with large ideas. Large things start with small actions. Over two weeks, two very small actions started building the foundation for a vision--something I wasn't able to accomplish over a year of trying.

One is the CRM project. This project is working to create a LAMP-based core to CRM information for mission-focused nonprofit and nongovernmental applications (fundraising, advocacy, etc.). It is amazing what happens when folks decide to roll up their sleeves and just get the work done.

The other is a business plan for an organization to seed, grow, and extend the ecosystem for these nonprofit, open-source software projects.

The CRM project is a first technical step to a set of products (software engineered to be useful to end users, not just developers) that have the potential to change the face of nonprofit technology. CivicSpace Labs is on the front end of this effort with many other folks starting to jump on board as the train starts moving out of the station. Unlike the TGV in Europe, people don't need to board this train at the station.... much like Indian Railways, people can hope on the train at any point, many of whom will sit on the roof in the open air. The train goes slow, but it gets to its destination eventually.