Archive for October, 2008

Numerati Generation Gap: Nate Silver & Dan Rather

Fun interview by Dan Rather of fivethirtyeight’s Nate Silver:

Some interesting things to note:

  • it’s fun to look at Dan Rather’s bemused near-smirk. You can just hear him thinking “you dork, why don’t you stick to baseball stats”
  • the number of times Rather refers to complex statistical methods for either the baseball work or the fivethirtyeight work
  • the psychohistory line about “any one game doesn’t matter” but when you hit a critical mass of data, in polls or stats, you can “find nuggets of wisdom”
  • There’s a weird thing going on in this discussion about stats and polling where some very simple math is being turned into high science. If you spend a little time looking at Baseball Prospectus, it’s all algebra. There may be some underlying techniques in the crunching of the numbers, like regression, but the formulas are pretty simple. fivethirtyeight is largely a question of weighting polls, based on some historical data. It’s just not that complicated. Silver’s dissection of the GWU/Battleground poll is barely even a dissection — he just looked at the methodology and saw that they over-indexed older voters! I’m starting to find it frightening how innumerate people are . . . or is it how illogical they are given that it’s middle school math level?

    More publishing cool/smartness, from Neil Gaiman

    From Neil Gaiman’s Journal: The Graveyard Book Tour

    The Proper US tour starts in New York on Tuesday the 30th. At each US stop, I’m going to read a chapter of The Graveyard Book. I’m going to read them in order. Other things will happen too (we’re hoping for some exclusive Coraline footage, for example), and there will be a Q&A and maybe other things. The stops are going to be filmed. Each Chapter that gets read will also be put online by Harpers very soon after it’s read (depends mainly on how quickly the footage can be edited and put up online). So you can follow the tour around, and get the book a chapter at a time for free…

    All sorts of forward-lookingness from an publishers who are already doing trailers, author blogs, digital early releases, crowd-sourced promotions and tours. I just wish I liked Gaiman’s novels half as much as I do his comics.

    Barnacles, Butterflies, and . . . Buffoons?

    I’m reading Numerati, a fun read about the rising importance of data and modelling (and a healthy antidote to some of the extremes of Super Crunchers. In general, the book has a better, less fetishistic tone, one that acknowledges the power of what’s going on, but keeps it real:

    The only folks who can make sense of the data are crack mathematicians, computer scientists, and engineers. They know how to turn the bits of our lives into symbols . . . [he has a nice jag about using index cards to keep track of dietary patterns, and how inefficient that would be. It’s a bit of humanizing text, but I don’t feel like typing it.] The key to this process is to find similarities and patterns. We humans do this instinctively, it’s how we figured out, long ago, which plants to eat and how to talk. But while some of us were focusing on more specfic challenges, others were thinking more symbolically. I picture early humans sitting around a fire. Some, naturally, are jousting for the biggest piece of mate or busy with mating rituals. But off to the side, a select few are toying with stones thinking “if each of these pebbles represent one mammoth, then this rock . . . “

    Somehow, those paleo-ners playing with the stones instead of mating or eating meat managed to survive long enough to pass on their genes until, millions of years later, they could become Hari Seldons of the 21st century.

    The key thread of the first fourth of the book (which is where I am, according to the impossible to count progress dots on my Kindle), is how people are trying to turn data points into meaningful models of people. The first test cases are supermarkets, where discount programs and smart carts are being deployed to gather data points about people. One of the first things that emerges is that there are customers who do too good a job of taking advantage of sales and promotions. These people, called “barnacles” by the numerati and marketers who really never intended for people to take advantage of sales, are the people who watch the movies they rent on Netflix, rather than let them sit on the coffee table collecting dust, or the people who actually go to the gym and try to live up to their New Years Resolution or lower their blood pressure. These barnacles should be “fired” by retailers, as they drag down profits.

    On the other side, you have “butterflies”: “customers who drop in at the store on occasion, spend good money, and then flit away, sometimes for months or years on end.” Since they’re unreliable, it a waste of time to lavish courteous, much less fawning, treatment of them.

    I suppose that means that the most desirable customers are buffoons . . . those who don’t scrutinize, price-seek or use the products and services they buy and those who are easily ensnared in a seller’s field of gravity.

    It’s kind of fun to watch marketing lurch between respecting the customer’s individuality and trying to model them into flippable switches.

    « Previous Page