Kiffets Voice

Just as iTunes changed the unit of music consumption from albums to individual tunes, online news technology has changed the unit of news consumption from editions of newspapers and magazines to individual articles. This creates a branding challenge for mass media publishers. Is it also a big opportunity for mass customization?

Mass Customization

Like mass production and mass transportation, mass media is about reaching a large market with a few products.  Mass customization is about reaching a large market with customized or “personalized” products.

Mass customization has its own movement and international conference. In the context of products and design, mass customization provides product families based on family catalogs of selectable parts. Historically the challenges have been to assure that the selected parts satisfy the user’s needs and work well together. Configuration systems came into prominence in the late 1990s to reduce the errors and costs when people ordered customized computers, automobiles, or even kitchens. If you shop online for a computer at HP or Dell, a configuration system makes sure that the parts are compatible and figures out the price. In these industries, mass customization grew the market by creating a better fit of products to customers.

Lessons About Mass Customization

Consider customization in ice cream and frozen yogurt shops. Back in the day you would mainly find only chocolate, vanilla, or strawberry ice cream. In 1953, Baskin Robbins increased choice and became known for its “31 flavors.” In 1988, Cold Stone Creamery took customization up a level by allowing users to choose nuts, candies, and other foods to mix with the flavors of ice cream. Cold Stone’s name comes from the cold granite stone used to mix in the “mix-ins”.

Ice cream shops offer a test case about the appeal of customization. Cold Stone advertises that it uses the finest ingredients to create a personalized ice cream experience. How many ways can you customize your ice cream treat? Just over 11.5 million, according to their web pages. Anecdotally, very young kids often pick ice cream by its color such as the blue cotton candy ice cream. Preteens want their favorite gummy bears or Reese’s Pieces mix-ins. Adults often choose berry, nut, and chocolate mix-ins.

For customers not inclined to experiment on their own, Cold Stone offers signature creations, such as “All Lovin’ No Oven” or “Birthday Cake Remix” or “Chocolate Devotion”. These are tried and true combinations of ice creams and mix-ins. Across all age groups many Cold Stone customers choose a signature creation and do not further personalize their ice cream. This may reflect the difficulty of overwhelming choice. As social creatures, we often find it easier to rely on someone else’s expertise to sort through the choices. When we are newbies this helps us to avoid bad combinations. When many combinations seem good enough, there is little incentive to spend much time making choices.  It is easier to trust someone who has done the homework already.

• Lesson 1.  Detailed choice is not for everyone.

Signature creations are highlighted,  “signed” and special.  In the language of personalized news, Cold Stone’s signature creations are their curated ice creams. This shows how curation can trump detailed personalization. Done well, curation can provide outstanding product fit within market segments. Cold Stone is now the sixth largest maker of ice cream in the United States.

Mass media seems to have a crisis of too many choices. Cable and satellite television now deliver hundreds of channels, fragmenting the audience for television networks. For many consumers, television is a wasteland too big to explore. Digital apps, the new opportunity for publishers to get it right on mobile devices,  have entered a similar phase. There was an initial burst of excitement about apps because they offered new levels of interactivity and mobility. However, now that developers have populated that space with over 140,000 apps for the iPhone, the resulting glut challenges a consumer’s ability to pay attention and an app builder’s prospects for profitability.

• Lesson 2: Mass customization can trigger a phase change in markets.

Technology-enabled changes in distribution have been reshaping the music industry for several years. The idea of creating customized party tapes had been around for some time. In 2001 Apple’s iPod and iTunes store enabled consumers to buy and enjoy music by the tune rather than the album. Apple emphasized the simplicity of such customization with its famous “Rip, mix, burn” advertisements and by play lists for the iPod.

By 2010, iTunes achieved a 26.7% market share for all music sold in the U.S. This made it the largest account, larger than WalMart and Best Buy, which displaced mom-and-pop music stores and warehouse music stores like Tower Records a few years earlier.

• Lesson 3. Social networks are poised to amplify mass customization.

As social beings, our personal tastes are seldom singular.  Social music sites enable music lovers to follow each other. By creating and sharing a list, a music lover becomes a curator.  To this point, version 10 of iTunes now includes Ping as a music-centered social network.  This addresses more of the user experience — extending it to music discovery. When consumers depend on music services for recommendations and discovery, they are much less inclined to switch services.

The idea of social networks for music is not new. For example, in a 2007 paper I described the untapped opportunity to use social networks to drive distribution, discovery, and sales of music.  MySpace and subscription-oriented companies have been exploring this space. Beyond music, the mass customization community recognizes that social media can enable community interaction on product variations, as suggested by this post.

My Take on Mass Customization in Media

For all of the products we have considered — from cars and kitchens to music and ice cream — mass customization is about personalizing products for consumers. Most people purchase cars and kitchens only occasionally. Music and ice cream are purchased more often. There is an interesting conundrum at the core of the user experience. Consumers want products that fit their needs. At the same time, they don’t want to be overwhelmed by too many choices.

The fragmentation of markets — such as in television channels and online publications — reflects that people are choosing products that fit their long tail interests. As in the ice cream example, choice can be overwhelming. This is where curation comes in — with curators competing in each of the market segments corresponding to our various personal interests. Each of us has a few special interests. For simplicity, an ideal product for personalized news would provide us with a set of curated channels satisfying our particular interests. Curators (possibly augmented by AI machinery and social input) make the choices that shape the channels. Consumers need only choose good curators that cover their particular interests.  Ultimately I think that each of us will follow a few curators. Birds-of-a-feather social networks of people with common interests may form around the curated channels, enriching content with commentary and drawing on crowd wisdom for discovery.

Just adding social networks to media is not enough. I have experimented a bit with FlipBoard, which enables Facebook and Twitter accounts to specify news channels of presumed personal interest. My observations may be of interest. My Facebook friends are diverse and have very different tastes in deciding what to report. I did not think that my wall would provide a compelling source of news for me to read. When FlipBoard enabled my Facebook and Twitter collection, I eagerly tested it. I was surprised. The presentation was as elegant as a glossy magazine. My Facebook wall had never looked so good with its compelling layout and design aesthetics. I stayed for twenty minutes but I never went back. Ultimately the content itself was not satisfying.

My Twitter-based FlipBoard channel works better. That’s closer to a curated channel but with me as its curator. Our Kiffets experience suggests that most users would like to find a channel curated by someone else. As in my earlier post, I also want a curated news channel to give me overviews and to organize the abundance of information.

Digital platforms continue to change the landscape of competition for digital media. There is no going back. At PARC we have been experimenting with augmented curation, news information services and presentation design principles. We invite news organizations and news consumers to work with us in exploring the future and to experiment with these ideas using our Kiffets system.

In the next few years, what percentage of people will personalize their news?

When I ask this question of news publishers, their answer is invariably “almost no one”. When I ask it of techies or beta users of our Kiffets social indexing system (a self-selected group to be sure!) they roll their eyes and say “almost everyone”. Why do we have such opposite expectations about the future of personalized news?

Often the hard part is asking the right questions. This post explores a deeper level of questions about personalized news.

News and Personalization Today

How many people personalize news now? Absent direct statistics, a proxy question is how many people use RSS readers since RSS users select their own feeds. A 2008 Forrester report pegs RSS usage at 11%. But, much has changed since 2008. More people are reading news online now than read newspapers, according to a BBC article published last March. The news industry is rapidly evolving, driven by declining revenue and changes in technology, as detailed in this recent Pew report. Many of the shifts enable personalization — as seen in the rise of Google News, Twitter, aggregators and mobile apps. Compared to using a single news provider, flipping across multiple providers is a hassle — whether they are accessed as multiple web sites, multiple mobile apps, multiple podcasts, or multiple radio stations. But will people take the time to set up their personal information diets?

The Arguments

Here are the main arguments that I hear from the nay-sayers — those who doubt that enough people want personalized news.

• People are lazy. People just want the news delivered. On the web they seldom take the time to vote on stories or even customize their home pages. Even if they would like personalized news, they don’t want to work hard at it.
• People don’t care. News is entertainment. Only a minority of people are concerned with how efficiently they manage their news reading time.

Here are the arguments that I hear from yea-sayers — those who already feel compelled to personalize their news.

• Mainstream news doesn’t cover my needs. There are particular topics I want to follow. The particular topics vary by the person. It may be a particular blogger, or news from a kid’s college, or news on a hobby, or professional news, or medical news.
• Mainstream news is boring. It covers the same stories several times a day on areas I don’t care about. The particular boring subjects vary by the person, whether they are not interested in traffic reports, disasters, political coverage, sports news, or whatever.

What Would Increase News Personalization?

A leader in mainstream news publishing recently told me that fewer than three percent of people enter customization information on mainstream news sites. This is consistent with anecdotal accounts of how often people customize home pages on web sites. In information foraging, there is often a perceived cost of the effort of doing something versus the expected benefit. If the benefit is not clear and immediate, people don’t bother.

Seeking deeper insights about the future of personalization, I have floated different scenarios with people. The most compelling scenarios are about virality. Suppose a person is interested in a topic that is not covered in mainstream news. From a friend or colleague they discover a specialized source that covers the topic very well. For example, there is a growing interest in the Bay Area about ecological matters such as solar energy, green building practices, and electric cars. On Kiffets there is a curated channel called Sustainable Living on this topic that enthusiasts are passing to their friends.

Adoption happens when users prefer the service for satisfying an information need. Sharing happens when they like it so well that they share it with others. Virality happens when each person shares it with enough others across social networks. Virality accelerates if they recommend the service to more people. People will make recommendations more broadly if the system satisfies many of their important information needs. There is a network effect in the middle of this process whereby personalized news gets better as more people build channels, find channels, and share them.

Systems that work on network effects take time to grow but they gain advantages over their competition as they get established. A social media example is Wikipedia. As it became an authoritative source, it appeared higher in search results, bringing in more users and attracting more contributors. Ed Chi and colleagues have reported on how its policies are now slowing its growth.

Subtle effects of policy and usability can limit rates of adoption and sharing. Because search engine rankings are designed to favor original content (like Wikipedia) and discount aggregation, growth of a personalized news site depends more on social networking and sharing. There are several places where design and usability factors matter:

• Indicating preferences. Personal news systems vary in their personalization models. Some systems determine user interests implicitly. They monitor a user’s reading habits and provide recommendations for articles or organized channels that would support their interests. Implicit recommendation is easy for users, but if personalization is opaque, they may feel powerless to control what news they get. If personalization requires that users make explicit choices, it may seem like too much work. Does the system surprise and delight users with new and interesting content?
• Finding channels. Most users would prefer finding a good channel on a topic of interest to building one.  Does the system provide good guidance about the suitability of a channel? How difficult is it for a user to determine whether a channel is good? Does the content reflect certain perspective or biases? What is the right mix between on-topic content and content that is related but not precisely on topic?
• Building channels. The set of possible topics on the long tail is open-ended. If a user wants a channel on a topic not yet covered, how difficult is it to build one? How helpful is the system in identifying good sources? Does the system get better at recommending sources as more channels get built and more users read articles? How difficult is it to tune a topic to get better results? Can curators make a subject area more attractive and compelling by organizing related topics? Can a curator share the work load by collaborating with a few colleagues and get suggestions from other people who follow a channel?
• Contributing to channels. Our approach to collecting articles uses dedicated curators for channels. In contrast, collaborative filtering approaches identify groups of people with similar interests but may seem to organize information haphazardly. Social news channels draw on crowd wisdom to find interesting articles and rank them by popularity but can require long lead times to start specialized subject areas. (For a comparison of these approaches see my earlier post.) These approaches have different abilities to satisfy users by staying on topic, providing high-quality articles, bringing out important new trends, following personal preferences, and so on.
• Sharing channels. How easy is it for users to share channels? Does this system help identify colleagues or friends with similar interests? Does it foster connections with social networks? When users share articles, does the system pass along relevant information about topics and channels as well?

In summary, personalization depends on several parts of a system and virality can be blocked at many places. Shifting from mass media to “mass customization” requires getting many things to work well together. Several parts of personalized news systems get better as the number of users increases.

Finding the Right Questions — Fast

Returning to the issue of finding the right questions, we are in the middle of it. News publishers are fighting for their businesses. They are joined by techies and entrepreneurs in trying to invent the future of news.

The next set of questions includes the economics of personalized news. There are questions like “how much will they pay for it?” In an advertising revenue model, there is another set of questions.

Kiffets is being developed at PARC. It is now in open beta release on the Web. Try it out and watch the following video overview. Let us know what you think. (Oh, and pass it along to your friends!)

I have been obsessed by curation and technology for curation. This post is about my experience curating a topically-organized channel using the Kiffets social indexing system that we are developing at PARC.

The word “curate” derives from Latin roots and means “to care for”. A web curator takes care of the subject matter, giving attention and providing context and organization for articles. A web news curator differs from a traditional news editor or wire editor by selecting stories from many sources, rather than working with in-house reporters or a news wire.

In the Trenches

At any time I curate about a dozen channels on Kiffets, supported by its machine-learning, classification, and collection capabilities.

Curation begins with my personal interests. I define a channel — choosing sources and creating topics. I refine the channel as I use it. When it is good enough I share it with others. This act of sharing transforms me from a user to a curator.  For a simple single-topic channel this may take just a few minutes. For an interesting channel that I feel like sharing, this is a more extended effort.

As an example of my adventures in curation, I started a channel on the Future of Journalism as we began thinking about engaging news organizations as partners and clients. I was fascinated by the turmoil and change in the news industry.

A few people — such as Jeff Jarvis, Clay Shirky, and Kevin Kelly — were appearing at conferences and speaking about how the web was bringing fundamental changes to the news industry. I chose their writings and related feeds as initial sources and began looking at the articles that came in. My initial topic structure reflected the charged issues that were rising — such as the loss of advertising revenues for classified ads (such as to Craig’s List), the rise of hyper-local news and citizen journalism, and the rise of new digital distribution platforms.

New themes arose and I continued to learn.  For example, I discovered schools of journalism doing experiments across the country. As different kinds of related stories came in, I added new topics and sources.

• Story: Google was criticized for its use of news without payment to publishers. (Topic: aggregation.)
• Story: Court cases reflected issues of distribution, copyright, and fair use. (Topic: copyright.)
• Story: Online ad revenues were in steep decline as numerous sites competed for advertising dollars. Failing newspapers appeared in dead lists. (Topic: advertising models.)

Over time I sensed a shift in the topics that interested me.

• Story: Amazon’s Kindle appeared and was changing  how books were sold. It  inspired competition. Amazon began offering portable newspaper subscriptions. (Topic: mobile delivery.)
• Story: Subscription model were being tried on tablets. (Topics: subscription, ereaders.)

I discovered bloggers and designers who were engaged in the new digital platforms. I felt a shift in tone from worry about the uncertainty in the future of journalism to enthusiasm for creating practices that worked.

Reflecting this shift in tone, I started a second channel on “New Media Practices”. When the iPad appeared, news organizations experimented with apps. There was a flurry of excitement about ad revenues on the iPad. The magazine-style ads were more acceptable to readers and more profitable than ads on the web. Issues about competition arose as Apple changed its rules for advertising networks on the iPad. Later I began tracking the rise of content farms, topical analytics, and the relationship to citizen journalism.

The overlap between my two channels became too high. I reworked the topics and pulled the channels back together. At this writing, the top-level organization reflects major categories for Business Models, Practices, and Structural Change in journalism. I am considering categories for legal issues and ethics of journalism.  (Feel free to subscribe to the channel).

Questions about the Future of News

I am mainly a technologist and not a journalist. Nonetheless, I now feel involved in the open questions for the future of journalism. My adventures in curation reflect the original roots of the word — “to care for”.

Some of my current interests:

• DIY Curation. Will other people build channels for themselves and then share their channels with others? Will do-it-yourself curation catch on? (Currently about one user in three on Kiffets builds a new channel. One in four of those builds a multi-topic channel.)
• Personalization. How satisfied are people with the news coverage of mass media? How can personalization become a more potent force in media consumption?
• Convergence. I have noticed that the apps from media companies mostly reflect the medium of their non-digital news business. To a large degree, the NPR app is audio, the ABC app is video, and the New York Times app is text. How far will the boundaries between television, radio, and print dissolve as these news media coexist and co-evolve on the same digital platforms?
• Social meets curation. How can we best combine the power of social news and social networks with curation?

At PARC we have been experimenting with augmented curation, news information services and design principles. We invite news organizations and news consumers to experiment with these ideas using our Kiffets system. The video below demonstrates using Kiffets to build a simple channel.

Google’s Eric Schmidt recently observed (Guardian, Techonomy) that the Internet is disruptive because it replaces information scarcity with information abundance. People expect to access this abundance of information easily.

Traditional design rules don’t apply

What is now scarce in our busy world is reader attention, not “column inches” of news print. So digital media are breaking some of the traditional rules and assumptions for presenting news, such as:

• Only main subjects (topical channels) need be presented to satisfy readers.
• Only a few stories need to be presented.
• It’s too difficult, expensive, and manually cumbersome to organize thousands of stories.

The new design principles

With information abundance, news consumers want systems that help them to manage their reading attention. It’s not practical to turn thousands of pages or see them all at once. So what are the new design principles for publishers to compete in the Age of Abundance — when a digital newspaper can contain thousands of articles on hundreds of topics?

1. Prioritize channels to focus information. Each of us can follow only a few subject areas, yet these areas are not the same across every person. Prioritization helps information consumers find channels in their primary areas of interest — mainstream and specialized.

2. Bring in the experts; organize. It is easier to find, catch up, keep up, discover, navigate across, and learn information if it is well organized. Expert curators provide the best organizations, particularly if tunable with feedback.

3. Help users forage efficiently. Based on the principles of information foraging, readers shift their attention based on whether they have a minute or two to browse/skim the news, or the mindspace to read in detail. Efficient scanning can be enabled by providing “top” or “hottest” stories they can skim across their channels and within each level of topics in a channel. Efficient navigation can be provided by enabling readers to drill down on interesting topics to get the next level of topics and more articles.

4. Provide the sideways perspective. Like a tree with its branches of stories, each channel has its topics and subtopics, and collectively all the trees form a forest of organized information. If we are browsing topics in one channel, how can we discover related information in another one?  Exploring related topics is more than just seeing “related stories”; it is about jumping to different points of view and different specializations. Ideally, connections between topics of interest should be discovered automatically and presented to readers.

Thriving with Information Abundance

Abundance is here to stay. No matter how the economics sort out for publishers’ subscription models, paywalls, content farms, aggregation, and advertising, consumers will not be satisfied with artificial scarcity. As Steve Rosenbaum argues in his analysis of media consumption, publishers who limit readers to their own narrow reporting are losing audiences.

There’s an opportunity to create a new generation of news/information delivery systems — optimized for Information Abundance – on all kinds of digital devices.

Try out the Kiffets system, where we have been experimenting with these principles. The tool is a prototype, intended to be refined with publishers, but you’ll get the idea; the video below demonstrates how a personalized news system can help news consumers manage their attention over abundant news.

Recently a PARC colleague (call him “Jake”) told me that he used to read several papers before heading to work. After Jake finished reading his papers, he felt that he was prepared for his day. At some point he stopped this practice because he came to believe that his sense of preparation was an illusion. Does reading the news get us ready for the day?

What We  Read

Jake’s story is similar to that of other people who rely on online sources including Google News and Yahoo! News. Readers like Jake may turn to the Wall Street Journal for business news, the Washington Post for national and political news, and perhaps a paper like the San Jose Mercury News for local Bay Area news. The editors of each paper select and prioritize articles — determining which articles are featured and how much space they get.

Obviously Jake benefits from the work of the editors who “advise” him on which articles are important from among the thousands that are published every day. The problem is that the advice that Jake gets from the editors is not adequately tuned to his needs. Information that is timely and important for Jake arises every day but does not appear in the mainstream news. Here are some examples:

Company news. Jake interacts on a regular basis with a few large companies that are clients. Although news about these companies is important to Jake, it may not rise to significance for the editors of mainstream papers.

Professional news. Jake is interested in developments in science and technology relating to various PARC projects. This news is scattered across many technical sources.

Medical news. Jake wants to track news in certain medical areas that are relevant to himself and his family.

Other interests. Jake has a variety of personal interests that he follows.

The Question: Who should tell us what to read?

As Jake goes through his day, he meets with a variety of people.  Each of us has a different set of people who we interact with and who give our lives context. A birds-eye view of Jake shows that he has interests in common with many different people — but the shared interests are different for different people. Jake feels ready when he is reasonably informed on the topics that matter to the people he meets.

Relying on mainstream editors or social news sites does not take this into account. Similarly, social news sites that give us crowd-selected stories or stories from our friends also miss the mark. The more contributors and friends Jake has, the greater the number of articles Jake would get. However, in greater numbers the articles seem more random because they are unorganized and less aligned with Jake’s core interests.

Adjusting our news priorities

Imagine that Jake has an information delivery system with three knobs on it. Each metaphorical knob is for adjusting how much of that news Jake needs to get ready for his day. The first knob controls mainstream news. The second knob is for professional interests. The third knob is for Jake’s personal interests. Article selection for all three channels could come from a combination of influences: curation, social networks, and Jake himself via previous reading habits. The knobs enable Jake to address how much mainstream, professional, and personal interest he gets in preparing for his day.

Some questions:

• What features do we want in a perfect news reader? (See Lawrence Lee’s post on the quest for a perfect news reader.)
• How can we best leverage the expertise and participation of others  to help us pick what to read?  Which others?

We are exploring these ideas in terms of curated channels in our Kiffets news service. You can see a demo in the following video overview.

In their competition for readership and advertising revenue, online news publishers need to differentiate themselves through curation. Read more about the why and the how here — including our step towards a solution for news companies, the Kiffets Social Indexing Engine, which is based on a research project at PARC that I’ve been leading.

Behind the scenes in research

This project (as with many things at PARC!) has roots in a trajectory of evolving expertise — spanning early collaborative filtering and later information visualization and sensemaking systems, to social computing today.

After 9/11, PARC developed and deployed information systems for U.S. government intelligence analysts — whom we dubbed “the jet pilots of sensemaking” because their information-seeking and collaboration challenges were so intense. One output of this research is my colleague Peter Pirolli’s book on Information Foraging, which has been referred to as “the ultimate source“. In our research we developed a set of “cognitive amplifiers” for analysts based on artificial intelligence (AI) and collaboration technology.

Intelligence analysts’ situations then are not that different from people’s information needs today: too much too fast or too little too late. So insights from our analyst research have guided development of the Kiffets system, which personalizes news along people’s content needs and passions. As in our systems for intelligence analysts,  the AI and collaboration technology serves as a cognitive amplifier that enables scaling the sheer amount of  information that needs to be collected, filtered, and organized.

Within the intelligence community some analysts act as curators or guides in sharing their expertise. They help others to catch up and keep up in their areas of expertise. Now in the open information on the Web, curation is emerging as a growth opportunity for information consumers and publishers.

The nuances of curation

In a recent graphical analysis of online media consumption, the conclusion was that winning media sites “all show a mix of large collections of content, mixing high quality created content with contributed, and curated content”.

The success of some news sites that curate  has created a lot of buzz – although different people mean different things by “curation”. The Nieman Lab pointed out important differences between AGGREGATORS (collectors), INDEXERS (auto-clusterers), and CURATORS (manual selection, organization, and commentary).

But, but… these distinctions, though useful, represent trade-offs: at one extreme, minimal human labor; at the other, time-consuming, unscalable, human effort. Guess which approach is most expensive for publishers. Guess which approach is most convenient and cost-effective. They won’t be the same.

So why not design a system that combines these approaches, recognizing the mundane, repetitive steps that can be amplified and the opportunities that can benefit from large scale? Additionally, news companies already know they can compete against the pack by: (1) deepening coverage (investigative reporting or well-written opinion pieces) and (2) extending coverage into long tail areas? What they can’t do is scale these approaches because there are too many niche areas with too small audiences. Kiffets addresses all of the above by drawing on three sources of power for filtering and organizing information: the hard work of the few (curators), the light work of the many (crowd wisdom), and the tireless work of the machines (AI).

Open Beta

To deliver the best of this information organized for its users, the Kiffets Social Indexing Engine combines the expertise of human curators with artificial intelligence technology and social input. Kiffets users subscribe to specialized channels on subjects that they care about, curated by people that they trust. The curators (users, media companies’ in-house editors, etc.) tell Kiffets how to select and organize information on a subject. Then Kiffets collects, classifies, and delivers the information automatically to its users according to their interests.

Kiffets is now in open beta release on the Web. Try it out or you can watch the following video overview.

Online news is a crowded field, and personalized news is becoming the Holy Grail for news publishers facing decreased revenues and outdated business models.

The challenge in personalizing the news: matching what people want with what they get. I believe that effectively personalizing the news on the long tail requires three approaches with their own unique sources of “power”: curation, search, and social participation. Relying solely on one of these engages its power and delivers its benefits, but also limits the effectiveness.

The benefits of advertising on the long tail

Editor’s note: For context about the Long Tail concept and its applications, you can watch the first part of Chris Anderson’s 2008 PARC Forum talk (delivered independently from this post), visit his blog, and/or read his original article.

The benefits of advertising on the long tail

To news consumers, the appeal of personalized news is that they can keep up on the news that they care about, better manage their reading  time, and address their information overload. For online news producers, the appeal is increased consumer satisfaction and potentially greater revenues.

Advertising revenues for online mainstream news are limited, because general ads delivered to general audiences yield weak results to advertisers. For example, in one mainstream media outlet I was presented with ads for a play opening on Broadway, medical treatments for back pain, and real estate — none of which are relevant to me. Pricing for such ads is dropping as the number of sites competing to display generic ads is rapidly expanding.

Now, consider an online news site for amateur and professional woodworkers. Such sites can run highly targeted advertisements for woodworking tools, supplies, outlets, and training. Producers and distributors of these products do not generally advertise on general news pages, though such ads are run profitably by search companies in conjunction with user-supplied queries. In general, the more specific the interest, the more precisely the ads can be targeted.

To get the revenue advantage, a personal news system must not only target the news but also target the ads. The promise of personalized news: when the news is focused by topic, the personal news system gains a precise model of the reader’s immediate interest. Similar to web search engines, the personal news system can use this model to guide ad targeting without employing behavioral targeting.

Classic approaches to personalizing news on the long tail

News alert systems

News alert  systems ask users to provide a query that indicates what kinds of news they want. This approach treats personalization as search.

Advantages. Search is powered by computers collecting and identifying relevant information. These systems search for news no matter how specialized the topic is down the long tail.

Disadvantages. The very precision of queries inherently limits their potential for surprise and discovery. In struggling to get just the right query, news consumers potentially miss articles that express things with different words. Furthermore, news consumers want to find out about what’s happening — without anticipating and specifying what the breaking news will be. Finally, in pursuit of complete coverage, search systems consider a wide range of sources, which vary in their quality and authority.

Publisher curation

Traditional news systems serve topics at the head of the long tail well because they have good sources of articles and good curators choosing them. Their editors judge which articles are important. Major papers enable a degree of personalization by offering specialized feeds in particular areas — for example, technology, sports, business, China, and so on.

Advantages. The power of curation removes the requirement of users specifying exactly what they want. In effect, the news consumer says “give me the news that the editor says is important”. This approach addresses both source quality and news discovery.

Disadvantages. Publisher curation does not scale to the long tail. Individual publishers lack both the topical coverage and expertise needed to curate the long tail. And even if they have it, Chris Anderson notes:

…the gatekeepers “got it wrong every time.” Every month, Anderson…picks which story will be on the cover of Wired, and every single month some other story ends up being the most read.

Some traditional publishers also treat “personalization” as merely a selection from their feeds. Although this approach is straightforward, it fails to provide consumers with topical coverage deep in the tail — and falls short in mining potential advertising revenues from it.

Social approaches to personalizing news

The very idea of using social approaches to personalize news may sound like a contradiction in terms, since “personal” refers to one person and “social” refers to many people.

However, for my specialized or tail topics (”biofuels” for example), I have networks of professional colleagues and friends with a shared interest and desire to discuss them. Although general or head articles (”earthquake in Indonesia”) are of general interest, they don’t usually represent areas where I have enduring engagement or special expertise. Although head articles occasionally trigger social discussion when they are very important, articles on my specialized topics much more frequently lead to social interactions.

Social news

Social news sites such as Reddit or Digg enable groups of people to submit articles. The articles are assigned priorities according to reader votes.  (Social bookmarking sites such as delicious are near cousins to social news sites. Their primary purpose is to organize a personal set of browser bookmarks to webpages, and their secondary purpose is to share the bookmarks.)

Advantages. This approach relies on social participation both for collecting and ranking articles, and with enough participants can address topics on the long tail.

Disadvantages. However, by relying strictly on social participation — especially when that participation is in the form of voting — there is a risk that the loudest or biggest voice will drive curation, leading to heightening or suppressing controversial positions. Furthermore, curation helps when it imposes a coherent point of view and way of organizing that information — but large committees do not usually excel at such nuanced decisions. Consequently, the presentation of news on social news sites often appears rather haphazard as articles are listed by popularity but without topical coherence. Finally, social news by itself misses the opportunity for automatic and systematic collection of news. There is often a challenge getting an adequate stream of articles in narrow topics, especially when the participating groups are small and getting established.

Collaborative filtering

The idea that “birds of a feather” are useful for recommending particular news (and movies, books, music) is reflected in an approach called collaborative filtering first proposed by my colleagues at PARC in 1992. Collaborative filtering collects data about user preferences, matches users to established groups of people with similar interests, and makes recommendations based on articles preferred by those groups. Findory and DailyMe are examples of early and current news systems, respectively, that use collaborative filtering to deliver personalized news.

Advantages. Collaborative filtering addresses the problem of scale to the extent that like-minded groups of people can be found for topics on the long tail. If a person is matched to a group, new articles and topics of interest to the group can (in principle) be delivered to the members.

Disadvantages. In practice, affinity groups need to be explicitly identified. In most cases, collaborative filtering systems require that people specify their interests so they can be matched with others. Since people typically have several news interests, each interest has to be separately described. By itself, collaborative filtering provides no powerful means for organizing the information for a group.

Social indexing

Social indexing delivers personalized news by enabling people to subscribe to news organized by curators with whom they share an interest. In an online setting, when a person cannot find an index that matches their interests, they can start one and share it with their friends.

Social indexing engages all three sources of power: the power of search (”the tireless work of machines”), curation (”the hard work of the few”), and social participation (”the light work of the many”).  News consumers can personalize their information diets by selecting the indexes and curators they follow.

Social indexing could be a game-changer for scaling curated personal news to the long tail. There are some near cousins to social indexing, such as socialmedian, which combines automatic collection with voting in social networks.  Other sites such as Twine employ semantic models of concepts developed by experts to relate and help organize articles. Although semantic models are like curators in that they recognize conceptual relationships, they typically fail to recognize how different topical organizations serve different human purposes.

Social indexing is in its early stages. We are currently in invitational beta for the early-stage Kiffets system being developed and incubated at PARC. New indexes are built as fast as our beta-users build them. The system itself also changes every few days. Some of the questions we ask ourselves as we evolve the system include: How do we help people with limited time find the news they want to follow and give them a sense that they are not missing anything important? How can the system engage the power of both personal choice and curation in deciding what news to emphasize? And many more.

Acknowledgments. Special thanks to Lance Good, Sanjay Mittal, Barbara Stefik, Prateek Sarkar, and Lawrence Lee for engaging conversations about these themes on social indexing while I was preparing this post.

Curation and the Future of News

Changing economics and new technologies are reshaping the future of online news. Industry observers call out changes in

• reporting such as hyperlocal news and citizen journalism;
• content such as the mixing of multimedia content with traditional text material; and
• distribution such as news aggregation and the ongoing diverse choices in business models, revenue streams and mobile devices.

In addition there is growing interest by information consumers in taking more control of their media time. They want personalized news for their specialized interests. The problem? News on specialized topics is often hard to find and is scattered across many sources. Mainstream publishing organizations do not cover topics deep on the long tail because they lack both the editorial resources and the expertise.

Can social media provide the means for curating the long tail?

Online Curating v. Publishing

In traditional news publishing, the role of curating is typically combined with publishing. For example, people seek out the Wall Street Journal for authoritative coverage of business news, or Technology Review for timely reporting on emerging technology.

There is a great need for curators of online information to help us find interesting and quality content, but the requirements and roles differ from traditional publishing.

• Deeply specialized topics. The long tail of specialized information that people read vastly exceeds the editorial expertise and capacity of traditional publishing organizations. For example, a city newspaper is unlikely to have a regular column on “polymer structures” or “muscle cars”.
• Density and aggregation. Reporting of long tail topics is sparse in major publications. For example, articles on  “faith-based neighborhood partnerships” appear in many publications, but are not comprehensively or frequently covered in any one publication. Curators would need to aggregate information about such topics as abortion, gay rights, and women in the ministry from many sources.
• Orientation. Information consumers on the web range from people seeking an introduction (”newbies”) to dedicated followers. A good curator presents the information within a structure that guides users in understanding what they need to know and what matters. For example, a newbie to the future of journalism could use a guide to the major topics, such as citizen journalism and mobile platforms.

Why People Curate

Curating requires:

• Expertise. Good curators need mastery of a subject area in order to identify authoritative sources of information and to establish a meaningful and useful organization of the material. They also need expertise to modify the selection of material and to reorganize it incrementally as the subject evolves. For example, recently the news on health care reform has evolved to cover the dynamics of the town hall meetings.
• Effort. Curating also requires dedicated effort to collect material and to arrange it into a useful organization. Dynamic areas need ongoing maintenance. For example, candidate gaffes and controversies tend to arise as topics during the course of election campaigns.

Curators tend to be passionate about their subject areas. For example, in a health area they may be dedicated mainstream or alternative medicine specialists. In technology areas they may be graduate students organizing materials for reading groups on the latest developments in their fields.

Curators can have several motivations:

• Intrinsic interest. They may be sufficiently interested to curate for their own purposes. For example, curating may help them keep up in a field or they may organize material as a step in understanding a new subject area. For example, when a family member develops a medical condition such as Crohn’s disease they may want to quickly learn about causes, treatments, and the best doctors.
• Shared interest. They may have a shared interest with friends or colleagues. Curating creates social capital. For example, some friends could start an investment group and share information on investment strategies.
• Commercial interest. They may also have a commercial stake in a subject area and benefit from curating. For example, a chiropractor could curate articles on relevant conditions and treatments, and offer them to clients on a web site.

Curating a Social Index

Many online journalists and bloggers already serve as curators by collecting links and adding commentary. However, I believe that a new form of curation is needed that helps curators work more efficiently. These ideas are informed by the experiences of early users of Kiffets, a social indexing system that we are developing at PARC. Kiffets recently started its beta release.  At the time of this post, it has about 200 users.  It has over 350 curated indexes on over 6000 topics.

Social media draw on three sources of power:

• Light work of the many. Properly harnessed, the light or casual “work” of many users can yield the “wisdom of the crowd”. This refers to the casual “work” of the many users of a social media system. The work may be incidental to using the system — such as determining popularity of articles and indexes by measuring how people use them. There may also be direct but lightweight actions, such as users commenting or voting on articles.
• Hard work of the few. Typically, the “few” contribute expertise that benefits others. In Wikipedia, this is the work of the people who write and edit articles. In a social index, this is the work of the curators of indexes.
• Tireless work of the machines. This is the automatic information processing work of computers in support of the users, including web crawling, article collection, and machine learning.

Kiffets shares a similar distribution of user roles as other social media. Most users are just consumers of information who provide simple feedback to the system. A smaller percentage of users creates single-topic indexes, which are the easiest to create. However, a small but growing number of users have graduated to more advanced curator roles, creating multiple-topic indexes and sharing them with friends. Since we expect this style of viral sharing to drive adoption of social indexing. Several features reduce the effort needed by curators in creating and sharing indexes. These facilities shift some of the required effort from the “hard work of the few” to the other two sources of power.

• Evergreen collection. Articles are collected regularly from designated web sources and automatically classified into the topics for the indexes. In contrast with social news sites such as Reddit, this reduces the burden of curators or other users in finding and submitting articles.
• Topic training. Curators define a tree of topics and provide articles as positive (on-topic) and negative (off-topic) examples. Given these examples, machine learning algorithms create computational models that automatically classify articles by topic. In contrast with tagging sites such as Delicious, this reduces the burden of identifying categories for articles and provides consistent categorization.
• Inline maintenance. Curators can mark articles as off-topic (on article reading pages) or submit new articles and their sources (using a Kiff It! book marklet) while browsing the web.

A social index aims to support information consumers in their personal news diets, even in highly specialized areas.

Curation is key to this.

A social approach can succeed because even our special interests are not entirely unique. By providing the means to leverage the work and activities of “birds of a feather,” we may make it easier for all of us to stay informed on what matters to us.

Personalization, Discovery and Sharing

Once music lovers could “Rip, Mix, and Burn” their own CDs they took control of their listening experiences. Music delivery became personalized as they included just the tunes they wanted. Consumer expectations grew, along with online services for:

• Easy discovery. With Pandora and other online services, users can easily discover new music.
• Easy sharing. Spotify is a recent online system for networking and sharing music with friends.

The economics and consumer expectations have changed for all digital media (see “Shirky: Problem is filter failure, not info overload“). Digital media consumers now demand more flexibility in getting what they want, stretching beyond traditional broadcasting models which pre-package content into channels.

As personalization, discovery, and sharing come to the news, it needs to be different from “playlists”. People

• don’t want to read the same articles over and over, like they play the same tunes;
• consume news in an information diet that represents their unique combination of interests; and
• have limited time, and don’t want to miss something important.

Here’s a news reading strategy Jim Pitkow (my former PARC colleague and CEO of Attributor) shared with me:

5-20-60 Rule. I typically take time to read twenty articles a day and to scan sixty. When I’m in a hurry I only have time to read five articles according to my interests.

Social Indexing to Remix the News

In exploring these ideas, I draw on the experiences of Kiffets users, who keep up on news in their personal interest areas several times day on their desktops and mobile phones. Kiffets is an online system for social indexing that we are developing at PARC and that recently started beta-release.

A social indexing system:

• delivers information via curated indexes
• collects articles for indexes from sources such as RSS feeds
• organizes the articles automatically by topic in each index
• includes not only mainstream news but editorial commentary, blogs, etc.

Kiffets serves up my information diet in terms of the indexes that I subscribe to.  For example, the Information Media index covers articles from about 120 RSS feeds and sorts them into topics about technology, markets, and trends. The Jokes index gathers jokes from the web, organized into dozens of categories such as occupation jokes, animal jokes, and situation jokes.

Over time I can add and delete indexes as my interests shift. Each index in my information diet (the My Indexes list, below left) expresses my interest in following stories in particular subject areas.

Five top articles selected from my indexes.

Kiffets delivers stories similar to Pitkow’s 5-20-60 rule:

• 5. My five “top stories” are drawn from those received in my indexes. If I had indexes about Middle East Politics, Motorcycles, and Jazz, then the top stories would be drawn from those indexes.
• 20. Readers scrolling down their My Kiffets see a few top articles from hot topics in each of their indexes  in order to satisfy the goal of not missing anything important.
• 60. Readers can scan more articles by looking at selected indexes. The USA index (see below) is a broad index. Its hierarchy of topics is shown on the left. The topic trees, navigation links and search enable users to explore information by topic.

Exploring an index in depth.

Articles are located in a conceptual map to the subject area to guide the curious reader. For example, several articles above are shown under the topic USA / Economy and Trade / Economic Indicators / Foreclosure. The topic hierarchy can be expanded to show additional economic indicators such as Bankruptcies, Consumer Prices, Durable Goods, and nine others. Other conceptual maps are provided by other indexes. In this case, Kiffets shows that there is a related topic in a competing index US News / Economy / Housing Crisis.

It’s the Curator, not the Channel

Kiffets currently has around 300 indexes in various subject areas. Some indexes are created quickly and have a simple topic structure. Other indexes are maintained by curators who are passionate about a subject area. The topic structure conveys the curator’s perspective and what they think is important.

Users are encouraged to become curators. They can create new indexes and share them with friends and colleagues. Sharing an index is like giving someone a magazine subscription that will continue to deliver articles every day.

Becoming More of a News Hound

My experience with Kiffets has made me more of a news hound than before, because it:

• Is more engaging. I now get good coverage of stories in subject areas I care about and enjoy that are beyond what publishers provide because the topics are much further down the long tail of popularity. I like the Science and Politics index for stories on sometimes controversial science news such as discoveries and ethics of stem cell research. My step son follows product developments at Apple and music news from bands like Phish. A colleague follows news about cricket,  Bollywood movies and business.
• Provides more efficient foraging. The news I care about comes from many information providers. Before Kiffets I would have to visit many sites or RSS feeds to cover my interests. Even when publishers provide focused feeds (e.g. “High Tech”), they are seldom a match to my more specialized interests. For example, “High Tech” may cover too many areas for me to follow or may focus too much on new gadgets for my tastes. Kiffets enables multiple information foraging strategies  — such as the 5-20-60 rule — supporting me when I have different amounts of time available.
• Enables mobile access. I can grab a few free minutes any time to check the news on my mobile phone.

Acknowledgments

The ideas in this post grew from many years of interaction with the members of PARC’s Human Information Interaction and Augmented Social Cognition researchers. Special thanks to my colleagues Lance Good, Sanjay Mittal, Lawrence Lee, Barbara Stefik, Priti Mittal, and Ryan Viglizzo. Special thanks also to the patient beta-users of Kiffets.