Crowdsourcing: Knowledge Has a Long Tail

Why the Crowds will beat the Experts 

The history of human civilization is tightly connected with our ability to use language to organize groups of people to tackle complex problems and projects that are beyond the abilities of a single person.

Today, most organizations in our society (governments, armies, companies) structure themselves in a Hierarchical Pyramid, with knowledge and power concentrated at the top and segmented visibility of smaller decomposed tasks distributed along its base.

In this post, we argue this will change, that we will start effectively using less structured and specialized organizations (let’s call it Unstructured Crowd) to solve complex knowledge problems through Crowdsourcing Technology.

Crowdsourcing is not a cheap way to outsource work to volunteers. When done right, it is an entire new form of knowledge production with the potential to revolutionize our approach to solving simple and complex problems.

The Long Tail of Knowledge

The anonymous Open Source Software Community (producing the software that runs the Internet) and the volunteers of Wikipedia (have you consulted the Encyclopedia Britannica lately?) show that in at least some knowledge domains, Unstructured Crowds of regular people can effectively compete with Hierarchical Pyramids of professional experts.

How does that happen?

To understand, let’s look at how knowledge distributes among a population. The picture above shows a Long Tail distribution. “Long Tail” refers to a statistical property of a distribution where its “tail” is larger than in its “head”. This concept was made popular by Chris Anderson in a Wired Magazine article that applied it to the retail business.

If we agree that knowledge exits in a long tail distribution, the group at the head of the curve (“experts”) accumulates personal knowledge that is individually higher than the average person. But the total knowledge held by the experts is still relatively small compared to the knowledge held by the broader population.

The Medium is the Message

Differently from rich face-to-face interaction, the printed word imposes an uni-directional form of communication, where someone who knows something communicates information to someone who knows less, with little opportunity for real-time interaction.

Because we have used primarily books, documents, e-mail messages to accumulate and transfer knowledge, the voice of the expert became the voice of knowledge. We have built a segmented-knowledge society where each of us is specialized on a narrow domain (be it tightening a bolt, writing software, doing tax returns, or defining strategy). In this environment, collective intelligence does not have channels of expression.

But if there were technologies that lower the friction and cost of collaboration and co-creation, there is a point where the long tail of knowledge is tapped to produce concrete results. Digital technologies, the Internet and Social Media are starting to do just that.

The emerging Internet-based social medium emulates some of the characteristics of rich direct interaction. It is real-time (not linear), it links peer-t0-peer (not hierarchically), and it is interactive. It is creating the low-friction conditions for true co-creation to emerge.

So, if that is true, why don’t we see the effects of co-creation emerge in the most popular social (Twitter, Facebook, Google+) and Q&A websites (like Quora)? It is a new medium, but it takes some time for people to fully utilize it and the old model based on individual ego and segmented knowledge to fade.

Also, we are already using medium interactivity, but not yet created the mechanisms behind the interaction to allow for true co-creation and collective intelligence to emerge and be expressed. That back-end infrastructure is still emerging in the form of new Crowdsourcing Technology.

Conclusion

In domains being affected earlier by he digital medium, (e.g. software development and publishing), the long tail of knowledge held by the Unstructured Crowd is now able to express itself in ways that are competitive with Hierarchical Pyramids.

As Crowdsourcing technologies evolves and the adoption and application of those technologies spread over other domains, more and more complex problems can be tackled using a new form of human organization.

We will be able to solve problems not by analytically decomposing big problems into smaller ones, but by presenting complex problem to the collective intelligence and let it holistically express the solution.

—

Marcio Saito’s (@Marcio_Saito) interest in Collaboration and Co-Creation originates in his early involvement with the Open Source Software community in the early 90’s. He writes about Social Media and Collective Intelligence and is a co-founder and advisor to Ledface, a startup using Crowdsourcing to create a new kind of Intelligence.