The Long Tail of Knowledge

The force behind social computing [tweetmeme source=”Marcio_saito” only_single=false]

The Free/Open Source Software movement has demonstrated that community-based development can generate software that is at least comparable to what is produced by organized and well-paid professional developers in a company. Volunteers of Wikipedia have beaten the army of experts at Encyclopedia Britannica in producing uptodate and comprehensive reference information.

How did it happen? How could “regular people” outperform professional experts?

An even better question: If communities of volunteers can generate such good results, why did we get into creating companies to organize professionals experts to solve complex problems in the first place?

To understand that, let’s look at how knowledge distributes among a population. The picture above shows a Long Tail distribution. It refers to a statistical property of a distribution where the “tail” of a distribution is larger than in its “head”. This concept was made popular by Chris Anderson in a Wired Magazine article that applied it to the retail business.

If we agree that knowledge distributes in a long tail configuration, the group at the head of the curve (the “experts”) accumulates personal knowledge that is much higher than the average individual. But the total knowledge held by the experts is still relatively small compared to the knowledge held by the population.

The written language is suitable for unidirectional transmission of knowledge. Someone spends time studying something, then writes it down.  Others reads it. There is no interaction between the writer and the readers.

Because we have been using primarily books and documents to accumulate knowledge, the voice of the expert became the voice of knowledge. We have built a segmented-knowledge society where each of us is an expert in something (be it tighening a bolt, writing software, doing tax returns, or defining strategy). Collective intelligence does not have channels of expression.

The result is hierarchical organizations with focus on personal acccountability. We can see that in companies today. Management is primarily concerned with decomposing goals into tasks that can then be assigned to specialized professionals. Specialization and segmentation is a less than ideal, but efficient way to cope with the high cost of collaboration.

Collaboration is impossible when the communication medium is the printed word. Noise and cost of collaboration grows exponentially when it happens through rich face-to-face interaction. That is why the world is the way it is. To solve complex problems, we need to segment and specialize.

But if there were technologies that lower the cost of collaboration, there is a point where the long tail of knowledge can be tapped to produce concrete results.

The digital medium, embodied by the Internet is starting to do exactly that. Collaboration is still noisy (have you tried to use Twitter?), but in some domains, the long tail of knowledge is now able to express itself in ways that are competitive. As technology advances and people adapt to it, segmentation will cease to be the most effective way of solving many problems. We call the organization adapted to the digital medium “Click Company“.

Rather than decomposing goals into tasks, leaders will be synthesizing results from interactions.

Businesses in domains of knowledge more intrinsically associated to the communication medium (encyclopedias, newspapers, software development, music, etc) are the first to feel the effects of that transition. But that shift is affecting every other area of knowledge.

How is it going to affect your business?