In the summer of 2009, Stanley McChrystal, the US and NATO Force Commander in Afghanistan, was taking part in a briefing in Kabul. The briefing was to discuss the complexity of US military strategy. A layout showing the problem structure was being projected onto the big screen with nearly a hundred bullet points for various groups and interests, processes and resources, and hundreds of arrows for connections between them. «When we understand that slide, we’ll have won the war!» — remarked General McChrystal. The audience erupted in laughter.
It is no laughing matter though. No-one endeavours to single-handedly build a skyscraper or to win a war (other than Hollywood supermen). So why in the area of intelligent task-solving we expect some genius to single-handedly solve any mega-problem? There are many reasons behind this misunderstanding. It is partly due to bestsellers popularising the history of inventions, our personal «eureka» moments, and even simply our mindset programmed to link our intellect with the human metaphysical soul.
In the meantime, intelligence as an ability to solve complicated tasks is inherent in animals, computer systems, and human groups. Certainly, it is individual human intelligence that successfully solves many everyday tasks but there are also many problems that are beyond individual human intelligence. No-one would fully comprehend all details of a jet aircraft or a city, a microchip or an entire country. This can only be done by collective intelligence, which means that in this case, it is stronger than individual intelligence.
Collective intelligence is not a literary metaphor but rather a real event that is significantly different from a simple human group.
The same employees are capable of designing a unique airplane together on their computers one minute and having a pointless argument at a meeting the next. Why such difference? It all depends on the way of combining individual intellects into a common «supermind».
Collective intelligence is created when there are certain protocols for interaction between group members. These enable to set and divide tasks, to identify priorities, to request and obtain further information, to consider and evaluate opinions, to compare alternatives, and, in the end, to formulate the final outcome.
Up until recently, collective intelligence was created only naturally — either through evolution or around charismatic leaders.
However, the outcomes of evolution are often inefficient and have low responsiveness to reforms (one example is bureaucracy within nations and corporations). Creative groups formed around outstanding personalities are often unstable and beyond scalable.
Only lately social engineering and information technology came close to artificially developing powerful collective intelligence that is predetermined to exceed capacities of individuals or even efficient small groups, at least in solving certain classes of tasks.
Despite the expectations of sci-fi writers, there are still no smart computers but certain elements of artificial intelligence are widely used in technology. For instance, a trained neural network is capable of recognising images without any clear algorithm. Though it is the entire network that has intelligence in this case. We cannot say that any separate element (neuron) within this network performs a certain task for the network but we can determine its importance for a specific solution. So it means that the intelligence of such a neural network is determined not by its elements but by the architecture of the connections between them.
This is also largely true for humans connected in networks called social. Any individual’s life goes on within his/her network of relatives, friends, and colleagues. Certain norms operate within such networks, ideas circulate, leaders stand out, and decisions are made. The most impressive example of collective intelligence based on a social network is academia with its system of degrees, a mutual recognition between scientists, peer-reviewed articles and cross-referencing. Publications are the main outcome of scientists’ work and their reputation amongst their peers is their main accomplishment. The main influential works and their authors clearly stand out with their citing ratings. Such a system maintains a high level of competence between experts in various fields. It was this system that has ensured the technological progress of recent centuries.
Still, the scientific network operates slowly. It takes many months to write and publish articles. It takes even longer to build-up reputations. Conventional scientific and bureaucratic mechanisms of searching for ideas and identifying smart people are too inept for many tasks of modern business. Information technology opens a path to creating new collective intelligence — faster and involving more people. We only need to programme the suitable architecture of the social network. To find this architecture is to answer the most important technological challenge of this decade.
People and texts
In 1971, Ray Tomlinson, a BBN employee, sent the first e-mail on the ARPANET. In 1989, Tim Berners-Lee, a scientist at CERN, was first to connect files on different computers through hyperlinks. E-mail and the World Wide Web, the two most important Internet sub-systems, represent two approaches for searching a new platform for collective intelligence. E-mail connects people and forms a social net, and the World Wide Web connects files and forms a semantic net.
Very soon «protointelligent» actions started emerging within these networks.
E-mails distributed Internet worms and chain letters while websites witnessed distributed denial of service attacks and so-called Slashdot effect manifested in a large number of visitors suddenly attracted by fabricated news-links. Similar to the rhythms of the brain visible on an encephalogram, these actions signified that the network environment is ready to become a carrier of complicated collective events.
The «Social» line of development led to distribution lists, news feeds, forums, and blogs. Then finally, social networks emerged becoming the embodiment of Marshall McLuhan’s Global village. Social networks also epitomised Stanley Milgram’s concept of Six Degrees of Separation which means that everyone is on average approximately six steps away from any other person on Earth. Competing semantic networks started battling for the actuality and credibility of information. Their battles were refereed by search engines, ranking sites and cross-referencing pages — just like scientific articles.
At the turn of the century, social and semantic networks started rapidly joining together with Internet projects which joined efforts by many previously unacquainted individuals for solving their own tasks. To define this approach, in 2006 Wired magazine introduced the term «crowdsourcing». But the phenomenon of crowdsourcing itself emerged a few years previously.
From individualism to cooperation
In 2000, GoldCorp, a gold producing company headquartered in Canada, introduced an unprecedented initiative by providing open access to its database that included 50 years worth of data on the company’s geological exploration. It offered an award of $575,000 for anyone to «find gold on the maps». Participants from 50 countries using rather unexpected ideas in the end determined 110 areas for prospecting drilling, and 80% of them proved productive.
Reserves and knowledge discovered during the initiative increased company turnover 90 times within several years.
GoldCorp is not alone. Established in 2001, InnoCentive started offering awards ranging from several thousand to a million dollars for anyone to solve scientific and technical tasks for their clients — amongst which are dozens of Fortune-500 companies, such as Procter & Gamble that itself has a research and development department with over 7,000 staff. However, InnoCentive accounts for over 160,000 «solvers» and often someone will step-up with ways to solve a new problem immediately. From the day of founding the project, the solvers were able to deal with nearly 400 complicated inventive tasks.
The mirrored approach includes attracting a «crowd» not for solving but for identifying problems.
No-one hardly ever reads complaints in the suggestion book. Even more useful is a visitor’s book on a website as it can be accessed by other visitors. If we enable these visitors to evaluate previous suggestions, the company will gain a tool for identifying important suggestions and increasing customer satisfaction. At first we would certainly need to attract visitors to the site, to learn how to «combine» similar suggestions, to deal with counter tampering, and of course, make employees treat client creativity seriously. This approach is called idea management. For instance, Starbucks, a global coffeehouse chain, used this method to collect nearly 25,000 suggestions through its website, and authors of some ideas already had an opportunity to evaluate the effects of their ideas being implemented.
Nevertheless, neither InnoCentive nor idea management models can be considered fully-fledged collective intelligence because each participant here works alone. Their interaction becomes more prominent on prediction markets — another crowdsourcing model that emerged from sweepstakes and market gambling. Here bets for future events are traded on a special stock market. Those that believe in a prediction, purchase bets, and those who don’t — sell. Their incentives are discussed on forums. If the prediction was accurate, organisers pay bet holders the nominal price. These funds grant the organiser access during all trading to monitoring how market price varies as community expectations shift. In many cases the accuracy of such materialistic collective intelligence is significantly higher than any predictions made by experts.
Wikipedia acts as another example of even more productive interaction between members. Wikipedia was established the same year as InnoCentive. Today it is probably the most dramatic example of crowdsourcing and the largest semantic network emerged within a single project.
Collective correction process combined with a complicated self-improving procedure for conflict resolution enabled to create millions of texts reflecting widely accepted perceptions of reality. However, Wikipedia’s disadvantage is in the actual anonymity of hundreds of thousands of contributors. Since contributors get neither any material remuneration nor even reputational recognition, many take delight in promoting their own views. The majority of Wikipedia’s collective resources are directed at reducing such information noise.
Simple incorrect solutions
Who would have thought that «people on the street» would discover gold or write an encyclopaedia? An unexpected efficiency of crowdsourcing led to steep expectations. But amorphous «crowds» are not the most reliable help. Starbucks is drowning under suggestions to reduce its prices. InnoCentive customers have to analyse hundreds of crazy ideas in their search for a brilliant one. «Correction wars» rock through the world of Wikipedia, and its informed authors have to bicker with laymen instead of writing new entries. The «crowd» is influenced by trends and informal leaders. Often the crowd prefers a «simple, logical, and obviously incorrect solution». In addition, there are also rather active provocateurs sent by competitors or just idle Internet surfers who interfere with work and offer off-tangent «virus» ideas.
To efficiently combine its intellectual forces, the «crowd» needs an internal structure.
Strictly speaking, there is nothing new in this — this is how science operates, and even any viable company differentiates between its employees based on their experience, qualifications, and success rate. However, offline regalia cannot be directly applied to the online world — the differences in their type of operation are too wide. This makes crowdsourcing projects experiment with so-called Karma that is accumulated through points given by colleagues. However, these points are often subjective so «Karma-tampering» is often possible. Also, its operation mechanism often gets out of hand and stops serving its purpose. That’s why organisers have to regularly change their algorithms for Karma calculation, and this reduces the system’s credibility.
It is obvious today that a technology aimed at ensuring a high-quality self-organising «crowd» that would become a powerful collective intelligence can be simple. It should include many processes and parameters; if possible, use objective criteria; and ensure efficient abuse protection. In general, the main principles of future complicated collective intelligence systems are obvious. A new generation of similar products will enter the market within the next year or two. One of them is Witology that combines a number of famous and new mechanisms creating an environment for producing and developing collective intelligence on the Internet.
Breath of «large technology»
Witology is based on a hybrid social and semantic network targeted at both community members and texts (documents and ideas) used by such communities.
As they start working on a certain problem, community members suggest their own ideas, and review and discuss others. The system tracks who reads whose texts and who establishes friendly relations with whom. Using these data, the system launches a multi-stage process for selecting the most interesting ideas and bright individuals.
To evaluate texts, a certain mechanism similar to prediction markets is used. Community members bet on ideas under discussion attempting to select the most valuable. As their currency, they use something similar to Karma, namely member reputation earned by his/her work. The market can simultaneously trade completely opposite ideas (moreover, that is what often happens) while their authors and proponents simultaneously promote their arguments in a semantic network in their attempts to raise their ratings. By recognising decent ideas at early stages, the member increases his/her reputation as a creative thinker.
There are other types of reputation within the system which depend, for instance, on how important are the texts read by the member, what is the reputation of his/her friends, and how important are the texts they read and write. In order to avoid any false accounting, the system takes into account the individual’s behaviour when working with the text: is the pattern of his/her actions reminiscent of a real review? So no reputation can be built through merely reading important texts. Anyway, this is just one example of how the system of reputations operates. In reality, in order to ensure that a community solves the required task, a team leader accountable to the client for the community’s work can easily introduce new approaches and rules. This keeps any chaos under control that simultaneously acts in social networks as a source of bright ideas and information noise — in which these ideas drown.
The general principle of work is to first attract new people and ideas into the system and then launch an evolutionary selection process amongst them. In the end, the client achieves two outcomes at once: effective ideas that survived when encountering competitors and a group of smart individuals who understand the gist of the problem well and know how to solve it. They will not all necessarily be recognised experts. Many members of collective intelligence are valuable not with their special knowledge but also with their communication skills and circle of contacts, their ability to correctly present sensitive issues or concisely and brightly formulate their ideas.
Different groups of individuals from the same large community will be allocated to different tasks. Upon finishing each project, members maintain their earned reputation and start working on new projects based on that reputation. A community that values people not based on any paper confirming their qualifications but instead on their participation in real projects — would that not be a dream come true for any recruitment department? It should also be noted that Witology targets (at least, at first) not just any outstanding major Internet projects but major corporations. Their large human resources are often underutilised, and their administrative system is often simply unable to identify any individual with a distinct vision for a specific problem working in a remote division. Any decisions made by the management, especially by any newly appointed management, often fail to take into account important tacit knowledge and habits of its personnel and are greeted with inexplicable resistance. In order to solve problems like this, we can use collective intelligence of employees that can also act as an additional social ladder to launch careers of the brightest individuals.
Perhaps, the most important conclusion we can draw from our review of the Witology system would be that the collective mind is facing «the end of its beginning time».
Previously, when amazing social and information effects were only first discovered, all pioneers of the industry would improvise and launch outstanding projects like Wikipedia or Twitter. While now in the area of collective intelligence we can feel the breath of a new «major technology». Similar to programming or car manufacturing, success in this area will soon be determined by the right connection of many previously tried and tested methods of social engineering combined with innovative solutions to ensure a clear competitive advantage.