Crowdsourcing: Knowledge Has a Long Tail

Why the Crowds will beat the Experts 

The history of human civilization is tightly connected with our ability to use language to organize groups of people to tackle complex problems and projects that are beyond the abilities of a single person.

Today, most organizations in our society (governments, armies, companies) structure themselves in a Hierarchical Pyramid, with knowledge and power concentrated at the top and segmented visibility of smaller decomposed tasks distributed along its base.

In this post, we argue this will change, that we will start effectively using less structured and specialized organizations (let’s call it Unstructured Crowd) to solve complex knowledge problems through Crowdsourcing Technology.

Crowdsourcing is not a cheap way to outsource work to volunteers. When done right, it is an entire new form of knowledge production with the potential to revolutionize our approach to solving simple and complex problems.

The Long Tail of Knowledge

The anonymous Open Source Software Community (producing the software that runs the Internet) and the volunteers of Wikipedia (have you consulted the Encyclopedia Britannica lately?) show that in at least some knowledge domains, Unstructured Crowds of regular people can effectively compete with Hierarchical Pyramids of professional experts.

How does that happen?

To understand, let’s look at how knowledge distributes among a population. The picture above shows a Long Tail distribution. “Long Tail” refers to a statistical property of a distribution where its “tail” is larger than in its “head”. This concept was made popular by Chris Anderson in a Wired Magazine article that applied it to the retail business.

If we agree that knowledge exits in a long tail distribution, the group at the head of the curve (“experts”) accumulates personal knowledge that is individually higher than the average person. But the total knowledge held by the experts is still relatively small compared to the knowledge held by the broader population.

The Medium is the Message

Differently from rich face-to-face interaction, the printed word imposes an uni-directional form of communication, where someone who knows something communicates information to someone who knows less, with little opportunity for real-time interaction.

Because we have used primarily books, documents, e-mail messages to accumulate and transfer knowledge, the voice of the expert became the voice of knowledge. We have built a segmented-knowledge society where each of us is specialized on a narrow domain (be it tightening a bolt, writing software, doing tax returns, or defining strategy). In this environment, collective intelligence does not have channels of expression.

But if there were technologies that lower the friction and cost of collaboration and co-creation, there is a point where the long tail of knowledge is tapped to produce concrete results. Digital technologies, the Internet and Social Media are starting to do just that.

The emerging Internet-based social medium emulates some of the characteristics of rich direct interaction. It is real-time (not linear), it links peer-t0-peer (not hierarchically), and it is interactive. It is creating the low-friction conditions for true co-creation to emerge.

So, if that is true, why don’t we see the effects of co-creation emerge in the most popular social (Twitter, Facebook, Google+) and Q&A websites (like Quora)? It is a new medium, but it takes some time for people to fully utilize it and the old model based on individual ego and segmented knowledge to fade.

Also, we are already using medium interactivity, but not yet created the mechanisms behind the interaction to allow for true co-creation and collective intelligence to emerge and be expressed. That back-end infrastructure is still emerging in the form of new Crowdsourcing Technology.


In domains being affected earlier by he digital medium, (e.g. software development and publishing), the long tail of knowledge held by the Unstructured Crowd is now able to express itself in ways that are competitive with Hierarchical Pyramids.

As Crowdsourcing technologies evolves and the adoption and application of those technologies spread over other domains, more and more complex problems can be tackled using a new form of human organization.

We will be able to solve problems not by analytically decomposing big problems into smaller ones, but by presenting complex problem to the collective intelligence and let it holistically express the solution.

Marcio Saito’s (@Marcio_Saito) interest in Collaboration and Co-Creation originates in his early involvement with the Open Source Software community in the early 90’s. He writes about Social Media and Collective Intelligence and is a co-founder and advisor to Ledface, a startup using Crowdsourcing to create a new kind of Intelligence.

5 thoughts on “Crowdsourcing: Knowledge Has a Long Tail

  1. Provocative as always, Marcio. I’m gonna side a bit with Kelly. In many respects, “crowd sourcing” is a misnomer. By “pointing the horse towards water”, as Kelly put it, you are providing guidance and directions from the “experts” as to where the crowd should focus. You have in essence created a community. A subtle difference, perhaps, but notable. The Open Source Community is just that, a community. It may be fluid, but there is leadership and guidance.

  2. I’m very appreciative of your perspective in this piece, Marcio. And I mostly agree with your thoughts by a large margin. I suspect you already know where we diverge in thinking based on past discussions/debates. 😉

    In essence, we’re both vocal advocates of crowd-sourcing knowledge, co-creation, and knowledge development. It’s this bit that illustrates the ‘PushMe-PullYou’ syndrome in the Structured versus Unstructured discussions:

    “We will be able to solve problems not by analytically decomposing big problems into smaller ones, but by presenting complex problem to the collective intelligence and let it holistically express the solution.”

    You’re mostly right, but history has shown, (especially in the tech industry,) that if we provide very basic frameworks and methodologies for the crowds to chew, taste, spit out bones and cartilage, then tweak – the results are usually better. It’s like a tasting menu for creative crowd thinking. I believe it is the job of the strategist chefs to pull the initial ingredients together as an offering. Open-sourcing is indeed effective, but if we look at examples like Linux, Apache, and Agile then there is a singular commonality: however flawed or imperfect, each had a basic ‘structured’ framework and methodology that the unstructured crowds enriched. There was a starting point (springboard) to work with.

    I’d never argue against co-creation & crowd-sourcing value, because those are shared ideals. I would argue that we’re not really emulating the connectivity fore-runners who’ve shaped our current world if we don’t acknowledge that they gave the open community something to improve upon in the first place. (Kick me if you’ve already heard this…) We can’t just revamp everything and toss all traditional data & knowledge-work structures aside just because we have the ability to utilize unstructured data more efficiently and intelligently. We need to keep the core data/structure/framework methodologies in place, while enriching & enhancing them with ‘unstructured crowd’ inputs.

    Better yet, we should be asking ourselves, “What would a new framework look like that we could throw out to the crowds to improve upon?”

    I think that would look like:
    * a narrative that weaves old & new (structured & unstructured data/ideas/dev)
    * a starting (probably flawed & imperfect) framework to pick apart & improve upon
    * pretty sure knowledge workers could teach us everything we really need to know about intelligent analysis & measurement, but the 80% who know have to adopt it have no idea how or where to begin.
    * offering them a loosely-structured starting point is a form of empowering them.

    It’s not counter-productive to point the horse towards the water – and then drop the reins. Just hang on for the rest of the ride and enjoy the trip.

    1. Great comments. Change in execution is almost always evolutionary. Change in ideas and frameworks are not. When discussing the ideas, sometimes we need to be extreme to create the contrast, but you are right in that pointing to water is not enough :o).

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s