The Shatzkin Files


Finding your next book, or, the discovery problem


A big flap has arisen this week — which I believe I would have been equally aware of had I been home in New York rather than in London — because the giant UK books-and-stationery retailer WH Smith has apparently found inappropriate ebooks being recommended through the kids books portions of the Kobo-managed ebook offering they host. This has sparked a lot of conversation about how recommendations — indeed how curation — is managed in the online environment. In this case, the discussion is about the specifics of this problem and how metadata might have been wrong, gamed, misunderstood. This has resulted in Smith’s turning off their whole web site, which contains the Kobo-offered ebooks, while the problem is “fixed”. It’s a mess that points to how far we are from solving core challenges of selling books in a virtual environment.

Online bookselling has a long way to go before it can deliver even what it intends to deliver in response to a search or to prompt a next sale. Of course, there are two additional and larger problems that come first: knowing what the right suggestion(s) would be and being able to make enough of them to match the book shopping experiences online sales must replace.

Analysis offered by Russ Grandinetti of Amazon at our Publishers Launch Frankfurt conference last week suggested that the US and UK are on the verge of transacting more than 50% of the book business online, with other markets in Europe and Asia not more than two or three years behind. (This may understate the real state of affairs; in a meeting I just had in London I was told that one of the biggest UK publishers says that 60 percent of their sales of print, ebooks, and audio are through Amazon!) Online sales of books were probably in the neighborhood of 10%, or less, for most publishers a decade ago. That shift is why retail shelf space has diminished so much, with major chains having sunk in both of the big English-speaking markets (and in smaller ones as well).

When most books were bought in physical locations, it was axiomatic that a book displayed in a store had an exponentially greater chance of selling than one that wasn’t, despite wholesale supply in the US from Ingram and Baker & Taylor that could get almost any book to almost any store in 24-48 hours. It had to be seen in the store to be bought. Competent commercial trade publishers knew there was very little point to pushing a book through marketing efforts if inventory wasn’t in place at retail, because seeing the book at the time you might buy it was a more powerful trigger for purchase than any other. Indeed, all the other stimuli (reviews, suggestions from friends, conversation at the office) tended to be acted upon only when the presence in the store was in proximity to the suggestion or recommendation. (And that’s why recommendations from clerks in the store were the most powerful recommendations of all: hence the concept of “hand-selling”.)

One problem with the change to online buying from the discovery perspective is that the funnel for each shopper keeps getting narrower. It isn’t hard for somebody in a bookstore to look at hundreds of books in a few minutes. It’s nearly impossible online. This either requires the consumer to spend more time shopping to see the same number of titles they used to see in a store, or to make a decision having seen fewer. And the concern is that the decision that gets made having seen fewer can be not to buy anything at all. (Or, particularly in the case of tablet users, to buy something other than books.)

Of course, in theory, being able to present a personally-curated batch of suggestions for each customer could be far more precisely targeted than what a store can do, and, in that case, fewer titles shown might do the same job. But we are a long way from that. And, for reasons I hope this piece will make clear, personally-curated choices would actually be far more likely to be delivered by Google than by Amazon (although they would raise a host of what would be considered big privacy concerns to a lot of customers by doing it). And that’s not a reflection on the quality of anybody’s programmers, and certainly not of their commitment to their customers.

The technology that hopes to help you “pick your next book” is referred to as a “recommendation engine”. I’ve never been on the inside of such an effort but the thinking behind them seems to center around analyzing what books you’ve bought and what you’ve searched for and, from that, figuring out what you might read next. This might be based on analysis of the content itself (e.g. Pandora recommending music of similar style and quality) and/or collaborative filtering models — leveraging user inputs (purchase history, ratings, and reviews) to make recommendations for other similar users (“people who bought x also bought y”). It all recalls for me the experience of being told when I met a great bookseller, the late Joel Turner, at the 1978 American Booksellers convention in Atlanta, that “if a customer walks up to my cash register with five books, I can always sell him a sixth”.

Of course, over time, a bookseller can fill out that knowledge with even more data as they see more and more purchases and get to know their customers, and perhaps their families. But, in fact, using books bought as a guide to recommendations is an incomplete data set. It can also be a misleading one since people buy books for people other than themselves.

Another way to look at it came from my friend, Andrew Rhomberg.  Based on his experience with start-up Jellybooks, he formulated five major book discovery paths: serendipitous, social, distributed, data-driven and incentivized.

The point is that most people get their ideas about what to read next from many sources: people they talk to, reviews, news reports, business interactions. Some people say they get book recommendations from their friends; others (like me) say they don’t often read the same things their friends or relatives read. I suspect that online communities of readers tend to work best for people who do a lot of reading in genres and not nearly as well for people who mix fiction and non-fiction, entertainment and learning. And some people gravitate to what’s popular, so bestseller lists work best for them. It is clear that getting on a bestseller list fuels a book’s sales.

And books are bought for motivations other than “to read”, so it might also be important to know that a customer’s son is having a birthday, that a customer’s cousin is getting married, that a customer is shopping for a new home or looking for a new job or starting on a new hobby or spending money on an old one.

Few, if any, of these things would be apparent to even the most diligent hand-selling bookstore personnel. Bits and pieces of it might be detectable by the super-merchant Amazon (but not likely to any other).

This is one devilishly complex problem. There are countless potential inputs to the “next book purchase” decision and they are processed by each different individual in a highly personalized way. If you think it through, it seems obvious that most recommendations to most people wouldn’t work. Which takes us back to the need to make a lot of them, which a bookstore display does much better than online pages that show 10 or 20 books at one time.

In the long run, it would seem to me that Google is the entity best-positioned to address this challenge if they can somehow combine the knowledge of what you searched for (which they know), with what you read online (which they could know if you use Chrome for your browser), and the topics and book titles that have appeared in your emails (which they could know if you use Gmail) and the things you ‘like’ and talk about online (if you use Google+). Knowing your travel plans and patterns would be helpful too.

Of course, unless you use Google Play for ebook purchase and consumption, they’d be missing the two most important bits of data — what you bought and how voraciously you read it and they still wouldn’t know your print book purchases (unless they crawl your email receipts for that as well) – which Amazon is building on without all the other information. What you’d really want to do is to correlate the book buying and consumption information from the past with the behavioral data contemporary to it. With it all combined, perhaps you could filter recommendations so that the 20 or 50 you could show on line would have the commercial power of the hundreds or thousands you could see in the same amount of time in a store.

At the moment, both Amazon and Google are trying to see a pattern through one nearsighted eye.

But is this all really part of a larger problem for publishers? Is online discovery really affecting the sales patterns for books? It would appear so. One of the global ebook sellers told me during Frankfurt that their online sales are far more concentrated than publishers’ sales tended to be, with a tiny fraction of titles (under 5%) making up a huge percentage of total sales (nearly 70%). (I am assuming here that this retailer’s data is typical; of course, it may not be.) If memory serves, at the turn of the century Barnes & Noble stores saw only about 5% of their sales coming from “bestsellers” and, I believe (relying on memory of detail, which I admit is not my most powerful mental muscle) backlist outsold new titles. Publishers really live on the midlist. We know the long tail is taking an increasing share of sales and it would appear the head is too. Those sales come out of the midlist. It is pretty hard to run a profitable publisher without a profitable midlist.

And that would suggest that the increasing concentration of sales, which is likely the result of our hobbled ability to present choices in the digital sales environment, is a problem that publishers will want to address.

  Back to blog

  • Karen Myers

    Google is many things, but it is NOT a consumer-oriented retail company. It is very unlikely to be any sort of dominant partner in book discovery (though its search engine might be used by the real winners). These are issues that are won by the right corporate cultures, not the right technology.

    • http://idealog.com/blog Mike Shatzkin

      Karen, if you read me carefully you’ll see I said that Google is *best-positioned
      *to address this challenge. I agree that they aren’t about to do it. But someday they may, and the statement I made will remain true whether they do it or not. Or, put another way, this is not likely to be done well by anybody for a long time.

      Mike

  • Rachel Van Riel

    Maybe we need a variety of discovery tools rather than looking for a comprehensive solution? Always basing on what you’ve read before can be narrow. Try whichbook.net for a different way of opening up reading choices.

    • http://idealog.com/blog Mike Shatzkin

      We’ve always had a “variety” of discovery tools and we always will. What is new is the narrowing funnel based on disappearing bookstores so it appears now that discovery avenues are closing faster than new ones are being opened.

      Mike

      • Rachel Van Riel

        The narrowing was already happening in bookstores. The major ‘literary’ bookchain in the UK reported selling the largest part of turnover from just 3% of titles some years back. They now have a new owner and are going back to a local decision handselling model as their usp in relation to supermarkets and Amazon. Independent bookstores rely on their uniqueness to survive but I didn’t expect to see this at the chain level – welcome and interesting to watch.

  • Glenn Yeffeth

    Great post as always, but I was surprised by your observation “One of the global ebook sellers told me during Frankfurt that their
    online sales are far more concentrated than publishers’ sales tended to
    be.” It seems more likely that online sellers, which have all books, would sell more backlist than physical bookstores.

    Looking at data for all of our books, I can share that looking at our sales at BN, 16% of our sales were from books pubbed 2008 and earlier. At Amazon, it’s 36%.

    Whatever the challenges for publishers of the increasing online share of sales, discovery isn’t one of them. And the rise of social media makes broad discovery much more accessible to clever publishers.

    • http://idealog.com/blog Mike Shatzkin

      The “head” of the sales curve is all books from Big 5 publishers. And the numbers we talk about are industry-wide. YMMV. A six-foot tall man drowns walking across a river that’s an average of 3 feet deep. In other words, the big titles on the bestseller list are selling astronomically-higher numbers than your best.

      Mike

      • Glenn Yeffeth

        I get this and you may be correct. But I hope you get the opportunity to discuss this with a big 5 publisher at some point who could review their data and confirm your assumption.

        Can you see my point that on its surface this doesn’t ring true? Amazon pretty much owns the long-tail, and its promotion is fairly broad and targeted, while the bookstores tend to just stack up the bestsellers and promote them. It’s not clear why on-line selling would be more concentrated.

        I’m a big fan of your thinking, but it seems to me you should get some confirmatory data before you build too big a theoretical edifice on this assumption.

      • http://idealog.com/blog Mike Shatzkin

        Points well taken.

        Mike

  • Pingback: The Discovery Problem (via @MikeShatzkin) | Literarium – The Blog

  • Thelma Benison

    Your percentages are very startling in regards to best sellers. Thank You for sharing!! I am very happy to say that my local Barnes & Noble is still in business and when I browse for my next book, I always head for my favorite genre first which happens to be historical fiction and from there it’s subject: which century or which historical figure am I in the mood to read about? Social thinking and best sellers do not come into play at all unless the best seller is within my genre. I would place incentivized first on Rhonberg’s discovery paths. I say people have an incentive in buying their next book based upon what they are in the mood for in terms of genre.

    • http://idealog.com/blog Mike Shatzkin

      Remember my percentages, though startling, are from one global ebook seller. I believe there is additional concentration happening, but I don’t have enough data points to confirm that it is as extreme as those numbers suggest.

      Mike

  • John Fleming

    Mike, perhaps the fundamental issue here the publishing industry coming to grips with the fact that the channel (especially retail) is now all about fulfillment rather than promotion. While challenging, that is also a huge opportunity. Rather than being locked in to the rigged game of fighting for book store shelf space and then depending on the consumer to sort out the cacophony of choices, adept authors and publishers can now speak to their audiences directly in meaningful contexts and use the channel to drive instant, seamless conversions.

    • http://idealog.com/blog Mike Shatzkin

      I agree. Publishers need to take charge of the marketing and create the “funnels” necessary — mostly with the biggest retailers — to turn awareness into sales.

      Mike

      • John Fleming

        Think they need to understand how to coop with retailers, but ultimately they have to understand how to show up where the readers are to drive awareness and demand.

  • Pingback: Writer Unboxed » Said the Online Retailer to the Entrepreneurial Author

  • EricWelch

    I wonder if there aren’t a couple of other factors at work with regard to the midlist/backlist. Given the huge increase in the number of published books, the percentage of the total any physical store can give to books diminishes every year. Most bookstores purchase through distributors like Ingram and B&T who tend to focus on the major publishers because of discounts. Online sellers like Amazon have endless shelves. Advantage Amazon and online sellers.

    Secondly, the huge advantage online sellers have in the used book market using services like Amazon’s seller market. When people can purchase from a huge selection of used books at ridiculously low prices, it seems that would have a substantial effect on midlist and backlist that might have formerly been sold through physical stores. Again, advantage Amazon and online sellers.

    • http://idealog.com/blog Mike Shatzkin

      I am not sure I buy that Ingram constricts supply (they have a lot of titles — I think millions — available through print-on-demand as well as a robust stock of current titles across all publishers), but the point about used books is definitely well-taken. There is absolutely no industry visibility into that but one has to assume intuitively that Amazon is doing a very large trade in them. And, of course, they’d affect backlist (although copies of new books start showing up in the marketplace very quickly, particularly those that sell well.)

      Mike

      • MitchR

        An author once told me he took a tour of a distributor’s office/warehouse (I don’t remember if it was Ingram) and said he saw their process for choosing which books to distribute. Here’s how it worked:

        The guy in charge would sit at a table with stacks of book covers. He’d go through the covers one by one and whatever cover caught his eye would go into the acquisition pile. The rest would be discarded.

        No reading of the book was involved. It all came down to one thing: cover art. So I would say, yes, distributors constrict supply in a fairly disgusting manner.

        Online distributors, however, rarely restrict supply unless it is dangerous or offensive. If you’re looking for even the rarest of books, you’re likely to be able to find it online.

      • http://idealog.com/blog Mike Shatzkin

        It’s a wonderful apocryphal story, but, frankly, it just sounds like pure bullshit. I can assure you it wasn’t Ingram. The whole story doesn’t make any sense and anybody who had actually ever bought or sold a book with a large-scale professional wholesaler or retailer in the past 40 years would know that. The author undoubtedly wrote fiction.

        Mike

  • Pingback: Scoprire i libri è davvero difficile?

  • WGold

    “It isn’t hard for somebody in a bookstore to look at hundreds of books in a few minutes. It’s nearly impossible online. This either requires the consumer to spend more time shopping to see the same number of titles they used to see in a store, or to make a decision having seen fewer.”

    I’m not sure where you’re getting your information from, but the above statement is completely erroneous.

    I view and buy three times as many books online than I ever did when I went to bookstores. And I practically grew up in bookstores. I’m not sure what method you’re using to view books online, but I have no trouble at all finding them and can go through hundreds in a short time.

    In fact, I find the experience very similar to shopping brick and mortar except that the inventory is much more expansive online. I’m no longer limited to what what the bookstore deems worthy of stocking and can find books they wouldn’t normally carry.

    I’m not sure I understand this need for curated product. Most readers are perfectly capable of searching and finding exactly the books they want and discovering content that appeals to them. Since the advent of ereaders, I have discovered more new authors than I ever would have while browsing the shelves of my local B&N.

    • http://idealog.com/blog Mike Shatzkin

      Actually, in order to write the piece I actually did a bit of searching on AMZN and BN.com to see how many items showed up on the first page of responses. It was in the range of 25-40. And the layout was incredibly wasteful of screen real estate in both cases.

      And if you searched it again a month from now, many of the books on the first page would be the same.

      If you are searching for a *particular* book, no doubt you’ll find it online and quickly and often not at all in a store. But if you’re shopping/browsing, I’m comfortable that I’ve represented the situation right.

      Of course, you may have some magic tricks for using bookstore sites that the rest of us don’t know.

      On the other hand, the discovery problem is really a *publishers’* problem (getting *my *book discovered) more than it is *the audience’s* problem because, as you say, it isn’t hard to find *something* to read. But I really doubt that you can see the same number of books in as short a time online as you can in a store.

      Mike

      • WGold

        I think it’s the other way around. You apparently have some deficiencies in using bookstore sites. This is not meant as a diss—some people do. But it isn’t all that hard if you try, and I think most people who buy their books online, know how to navigate their favorite sites quite well. And maybe that’s the difference. There’s a learning curve involved.

        Of course, with online bookstores you don’t have to get in your car, drive several miles, then spend time wandering through the bookstore aisles, finding spines with titles that intrigue you and pulling them out to take a look. With online bookstores, all the books are face out and have a much better chance of catching your eye.

        As for it being a publisher’s problem, getting a book discovered has *always* been a problem for publishers. And that holds true in brick and mortar stores as well. Yes, they can pay for placement, but it doesn’t come cheap and such luxuries are usually only reserved for big publishers and star authors. If a reader were to stick only to the front racks, he’d never find the real gold.

      • http://idealog.com/blog Mike Shatzkin

        I know what you’re saying is largely true from the other end, since I’m just beginning to learn from people who know how to present your book so that it is processed online in a way that will make it appear prominently and frequently to people who are likely to buy it. And in other contexts (probably somewhere in The Shatzkin Files) I’ve argued that readers figure out how to find their next book and discovery is the publisher’s problem..But there’s no learning curve required to see many titles much faster in a bookstore. And, of course, publishers have bought space there. But it was never all about that. Having a more effective sales force and more efficient shipping also had an impact. Now it is all about digital marketing, both on the store’s end and on the publisher’s. And I think the science is in its infancy.

        Mike

  • Pingback: Reading about eReading this week October 21, 2013 | Allegany County Library System Director's Notes

  • progretarian.

    Good article! The narrowing funnel is a real issue for a lot of people and the most obvious weakness of Amazon’s “Customers also bought” feature.

    I believe middle sized and specialized book stores stand a chance to deal with these things via expertise/great service.

    On top of that an fully integrated GoodReads will solve most issues for Amazon there on the road to full market dominance. The data is available.

    • http://idealog.com/blog Mike Shatzkin

      We agree that the solution for potential survivors is “vertical”, which is what you’re saying about bookstores. It is also true of publishers.

      Mike

  • Pingback: » The OutRamp Guide to Self-Publishing: Episode #2 - The OutRamp

  • Pingback: Google might be your best bet for book recommendations • Liquid State

  • Mirjam Buettner

    Another important reason why people buy books is because they need a specific title (and edition) for their studies or job. In this case, recommending similar literature is often useless. If I bought a biology book for that one “science” requirement at College, it doesn’t mean I’m likely to keep buying similar biology books. I read lots of fiction and I’m interested in finding books similar to
    the ones I have enjoyed so far, but my recommendations are clouded with
    professional literature that are quite pointless, because in that area
    I’ll find what I need to read by other means (mostly references from
    other books and articles). Therefore, it would be good if some distinction could be made between books customers read for fun and books they have to read for studies/work.

    • http://idealog.com/blog Mike Shatzkin

      Yes, there is further confusion when recommendations are based on books bought as gifts. We’ve noticed that happening even when the gift was obvious: that is, shipped to somebody else!

      Nothing works every time no matter what, but these are things (both your concern and mine) that would seem fixable.

      Mike

  • Pingback: Andy on Why Bookstores Matter | AMACOM Books Blog

  • Pingback: Andy Ambraziejus on “Why Bookstores Matter”