The Ignorance of Nick Carr

Thanks to Otti for pointing out this article by Nick Carr, entitled the Ignorance of Crowds. It’s a look back at Eric Raymond’s seminal paper, the 10-yr old “The Cathedral and the Bazaar”, in which he makes the case for peer production, focussing on the Linux operating system as his example.

Peer productions and open source have come a long way in the ten years since the paper was written, and Nick compares the chaos of wikipedia to show how in a different environment, peer production and democracy don’t give the quality results exemplified in the Linux OS.

Nick’s argument focusses around the difference between the high-level of control exerted by Linux Torvalds over additions to the Linux OS and the lack of such control in the case of wikipedia. He points out that Jimmy Wales, founder of wikipedia, is introducing an editorial team which, in effect, will provide a Linus-like top-down control.

By coincidence, I am currently reading “The Wisdom of Crowds“, by James Surowiecki, a thorough treatment of the different ways large numbers of people contribute to solving a problem. A particular case study stands out as illuminating this continuum between the cathedral and the bazaar.

The SARS outbreak in 2003 and the subsequent global search for, and rapid discovery of, the viral cause of the disease was an example of a fantastic collaborative research project. The virtual team, dispersed in labs across the globe, pinpointed the cause after only a month of looking.

What I really want to draw out are the specific conditions and structure of the environment in which this research was conducted, and how they vary from the way peer production is portrayed in Nick’s paper. The fact that much of today’s scientific research is carried out in this way shows this is not just a one-off chance success.

Many self-governed teams vs. a democracy of non-governed individuals. The key here is that the two extremes of the cathedral and the bazaar represent organisational structures that are less than optimal. One is total top-down control on a small team of experts, the other a lack of control in a sea of people. The balance which the SARS case illustrates is that each lab, with its own control structures, motivations, reward frameworks and methods of working, form islands of organisation which combine at the same level. There is then open sharing of data and opinion between the groups, but no attempt for one group to co-ordinate the approac of another.

Diverse range of approaches vs. no agreed approach – the picture conjured up by wikipedia is of one of a mass of lone agents pushing data into the system in an unorganized way. The SARS collaboration demonstrates that a diverse range of internally agreedapproaches to solve a problem can provide an efficient way of reducing a complex problem space very quickly, especially when a number of people are required to work in tandem to complete a particular task. This diversity is to be constrasted also with the cathedral approach, where you are really only attacking the problem from one angle.

Reward and recognition – one very key theme in modern scientific collaboration is the dichotomy between the recognition obtained by publishing your work, and the reward one gets from being the first to discover, or to commercialise the discovery. This is very much about the culture that has been fostered in science, so that even if it feels that the selfish, or rather self-interested course of action, would be to withhold from sharing research, the scientific community as a whole benefits from sharing. This fact has been baked into what it means to be a research scientist.

To summarise, I agree with Nick that it is the mixing of the two approaches – cathedral and bazaar – that makes for the optimal environment for collaborative working, but I think that it is possible to identify specific criteria that can be applied to guide those setting up these environments.