kip/bot/blog

Technology Review ran an article about blogosphere and social network traffic visualizations which featured pretty and interesting pictures as well as insights into what’s worth measuring in social networks. (The full article isn’t yet available to non-subscribers in its full format.)  The picture below visualizes a number of things including, apparently, the relative ego size/socialness of political junkies and designers.

The two regions are held together by popular blogs with ties to both subject areas. The size of the ­circle representing a given blog is proportional to the number of other blogs linked to it. Hurst notes an apparent difference in culture between the two regions: pink lines, which represent reciprocal links, are much denser among the political blogs than they are among blogs focused on technology.

Came across a line in Omnivore’s Dilemma that captures some of my frustration with super-crunching and marketing models:

To reduce [a complex agricultural system under discussion in the book] represented the scientific method at its worst. Complex qualities are reduced to simple quantities; biology gives way to chemistry . . . that method can only deal with one or two variables at a time. The problem is that once science has reduced a complex phenomenon to a couple of variables, however important they may be, the natural tendency is to overlook everything else, to assume that what you can measure is all there is, or at least all that really matters. When we mistake what we can know for all there is to know, a healthy appreciation of one’s ignorance in the face of a mystery . . . gives way to the hubris that we can treat nature as a machine.

I love the idea of biology giving way to chemistry: systemic thinking giving way to engineering problems. How often do designers struggle against models of people that focus on two factors to the exclusion of everything else, that reduce people to the actions we want them to take?

Stick a pin in it and it dies.

Two articles, once again from Todd Walker, highlight how research (or research-driven techniques) needs to be (re)-grounded in the needs of design.

The first, Design Meets Research from AIGA,  has a useful survey of leading testing techniques and provides some pros and cons about each of them.  In the middle of the piece is a paragraph that summarizes the key problem most designers have with research:

There is a group of brand consultants and cultural anthropologists alike that believe now that it is not the actual research itself that is the problem. It is rather about how research is often misused, what type of design concepts and stimulus are tested, and how data is analyzed that is most often at fault. When used correctly, research shouldn’t stifle creativity but rather offer designers stronger inspiration and focus.

They remind designers that there’s a critical interpretation phase that comes between research and design.  No one would disagree with that statement, but where it gets tricky is how people define interpretation and who participates in it.  In more than one work environment, interpretation meant a summary of major findings, was conducted by the strategy group or account lead, and somehow straight-lined to design recommendations.  (”Only 49% of respondents viewed element x favorably -> Replace element x or remove it.”)

The article hits some other high points:  know what you’re testing for; remember that testing is ultimately about better understanding a customer (heightening designer empathy with the audience) and not about having customers do design; ethnographic activities are still the best things for designers to do no matter what; research is an art not a science; interpretation is a joint activity between design and research.

The other article, Personas and the Role of Design Documentation has similar themes, but is more focused on personas.   Specifically, it focuses on the way in which most people go through personas as a deliverable that needs to be done, not as a tool with a purpose and communication goal.  Key point for the writer:

Personas are not documents, and they are not the result of a step-by-step method that automagically pops out convenient facsimiles of your users. Personas are actually the designer’s focused act of empathetic imagination, grounded in first-hand user knowledge.

The best part of the article is a distillation of lessons from Alan Cooper’s ‘origin of personas’ story (mythic in its grandeur, but true):

1. Cooper based his persona on a real person he’d actually met, talked with, and observed.
This was essential. He didn’t read about “Kathy” from a market survey, or from a persona document that a previous designer (or a separate “researcher” on a team) had written. He worked from primary experience, rather than re-using a some kind of user description from a different project.

2. Cooper didn’t start with a “method”—or especially not a “methodology”!
His approach was an intuitive act of design. It wasn’t a scientific gathering of requirements and coolly transposing them into a grid of capabilities. It came from the passionate need of a designer to really understand the user—putting on the skin of another person.

3. The persona wasn’t a document. Rather, it was the activity of empathetic role-play.
Cooper was telling himself a story, and embodying that story as he told it. The persona was in the designer, not on paper. If Cooper created a document, it would’ve been a description of the persona, not the persona itself. Most of us, however, tend to think of the document—the paper or slide with the smiling picture and smattering of personal detail—as the persona, as if creating the document is the whole point.

4. Cooper was doing this in his “spare time,” away from the system, away from the cubicle.
His slow computer was serendipitous—it unwittingly gave him the excuse to wander, breathe and ruminate. Hardly the model of corporate efficiency. Getting away from the office and the computer screen were essential to arriving at his design insights. Yet, how often do you see design methods that tell you to get away from the office, walk around outside and talk to yourself?

5. His persona gained clarity by focusing on a particular person—”Kathy”.
I wonder how much more effective our personas would be if we started with a single, actual person as the model, and were rigorous about adding other characteristics—sticking only to things we’d really observed from our users. Starting with a composite, it’s too easy to cherry-pick bits and pieces from them to make a Frankenstein Persona that better fits our preconceptions.

There are, of course, challenges embodied in these lessons.  Grounding a persona in one person could lead to endless ratholes about which one person, and number wonks will immediately jump all over the “method”/”methodology” point.  But the key point is that personas are ways of creating empathy with the user, of getting us (our team and clients and other stakeholders) out of our own heads and into someone else’s, of creating conversations with potential customers and users.

Terrific TNR article referred to me by Todd Walker describes how the Obama team uses data and wonky policy techniques in a way that seems relevant for many of us in an increasingly number-rich, -doused, -drenched, -dictated world.

The article starts with a description of the influence of neo-classical refiner Richard Thaler:

Behaviorists like Thaler believed that the perfectly rational, utterly self-interested maximizers of economists’ imaginations had little in common with actual human beings, who frequently err when making simple calculations, who have trouble with self-control, who often act out of altruism or spite.

But what’s really interesting is how Thaler and his fellow behaviorists responded to this fairly critical insight. Though rational self-interest was the central tenet of neoclassical (i.e., modern) economics, they didn’t take a wrecking ball to the field and replace it with some equally sweeping theory of human behavior. Instead, they labored to bring economics closer in line with how the world actually works, one small adjustment at a time. “‘Discovery commences with the awareness of anomaly,’” Thaler wrote in the introduction to The Winner’s Curse, quoting the philosopher Thomas Kuhn. “I hope to accomplish that first step–awareness of anomaly. Perhaps at that point we can start to see the development of the new, improved version of economic theory.”

One of my biggest gripes with data and marketing models (funnels) is that people tend to approach them as rules to live by. When faced with an anomaly, there are two responses: 1) wave it off as an anomaly; or 2) try to force the anomaly into the ‘model’. It’s a bit like the retrograde motion of planets: when the observational data pointed to non-circular motion of the planets, retrenching astronomers created these weird circle-within-circle movements that had no plausible explanation, but preserved the pretty circles. A third approach would be to evolve the model, soften its hard edges, add some dynamics to it.

The divide in economics between numbers and working models is becoming a chasm. What’s great about Thaler’s approach is that it functions somewhere between the wrecking ball of a new model, but avoids retrograde techniques. The thinking embraces the anomaly and allows for a punctuated equilibrous burst in the development of the model. “Like their intellectual godfather Thaler, the Obama wonks aren’t particularly interested in tearing down existing paradigms, just adjusting and extending them when they become outdated. (Thaler urges his students to master the same traditional, mathematical models their colleagues do if they want to be taken seriously.)”

Another nice passage highlights that there is still something along the lines of expertise and judgement that can live well with numbers:

The second difference is that the Obama hands tend to feel less hemmed in by establishment opinion. As one Obama adviser puts it, “Democrats want to be just a little bit different from Republicans, but not so different that they get attacked for being weak.” Like Hamilton, the Obamanauts generally reject this calculus–not because they favor some radical alternative, but because clinging to received foreign policy wisdom can preclude highly practical courses of action.

Of course, here they’re talking about foreign policy, which is not numbers-based. But the idea of “practical courses of action” — things which just make sense or feel right, pass the sniff test, resonate with a highly trained neerve ending have a place in their discussions, agenda, and plans.

It also allows for leadership without ignoring the polls, or innovation without ignoring the data.

Great post at Adobe about a neglected area of design:  the holistic experience of getting to good content. I use soft-edged words in that description — “getting” rather than finding, “good” rather than right — to highlight that the experiences we craft need to allow for semi-directed, imperfectly-focused user behavior.  Too often, we’re looking for the right answers rather than the right systems, we discuss user needs when they’re actually wants, or tasks that need to be completed when maybe it’s the equivalent of window shopping they’re doing.

Browsing, searching, and asking might all take place within a single attempt to find information. Finding routes are often quite circuitous, iterative, and surprising. There certainly are simple, straightforward instances of finding—say, looking up a colleague’s phone number in a staff directory. But wandering through and learning about information—with pauses to search, browse, and ask along the way—is how many of us find information and learn about both the complex (for example, determining the most appropriate health plan our employer offers) and the seemingly simple (like choosing a plumber).

As a designer who works in agency environments, I often get caught between the marketing attempt to direct a behavior (applying funnels or merchandising logic to discovery scenarios).  The language of this post does a nice job of describing the user’s state(s) of mind and avoids putting too fine a point on what they’re doing.

With all of its twists and turns, finding can be lovely and life-changing. Even when we fail to find—and we often do—we still learn. Finding is arguably at the center of all user experiences. …  Unfortunately, most of the systems we design don’t really support finding. We might do a bang-up job with searching, browsing, or asking. But we’ve failed at integrating them well; therefore our designs fail at helping users to shift effortlessly between these different aspects of finding, and instead impose harsh interruptions on the process.

And then a topic near and dear to my heart:  the need for designers to broaden what they think of as in their purview:

But there is another, less-obvious form of complacency common to so many designers: they don’t design for holistic experiences—like integrated finding—because they don’t speak data. Designers haven’t paid much attention to the terabytes of user data being logged right under their noses. Fortunately, that’s changing.

Fun set of comments attached to a blog post about Stephen Kosslyn’s psychological tips for Presentations highlights the tension around number-crunching and expertise.

For those that missed the twitter, there are some cog-sci principles reduced to four memorable (or at least re-memberable) principles: 1) Goldilocks — show the amount of information that is “just right”; 2) Rudolph — like the red nose, guide the user to the most salient point; 3) Rule of Four — people have cognitive difficulties dealing with more than four visual ideas; 4) Birds of a Feather — group similar things to smooth out the narrative. (This is 3rd or 4th hand, TED had something about it as did other blogs and this one.)

These are grounded in cog science and describe the kinds of things that are “brain compliant”. What’s funny, though, is the reaction of some designers:

So, wait….cognitive science is just figuring this stuff out? These are the sorts of things graphic designers and advertising students learn as freshmen.

I learned this stuff when I took graphic design, especially in typography class.

I don’t know this is kind of common sense. If you need a cognitive scientist to tell you to change a couple colors or stray from having 20 things on screen then I doubt you have anything worth bringing to a presentation.

Amen. Tufte explained much of this 6 years ago. Kosslyn does add valuable insight and data. PowerPoint keeps evolving. Presenters keep devolving.

Yeah this Design 101 (hello, hierarchy of information!), tarted up in science drag.

Picking sports talent, good wines, winning screenplays, even predicting Supreme Court decisions. These all turn out to be things that computers can do better than we can. We don’t have experts, we have intuitivists. Experts are just sad little thin-slicers who think their years of study and keen ability to see deep into a matter counts for something when all they are, really, are inherently-biased, easily distracted, overall inferior protein machines.

Not quite the thesis of the book Super Crunchers, but certainly a valid take-away for someone who’s still depressed about Deep Blue beating Kasparov and who couldn’t finish “I am a Strange Loop” for fear that he may never get out of bed again if he did.

Which isn’t fair to the book or its author. Ian Ayres who is actually quiet open about what works and what doesn’t in the field of super-crunching. (Basically, anything that takes a lot of data, runs it through a model and makes assessments is super-crunching. That said, some of the work he describes is simply super-sifting — querying millions to find the thousands who match certain criteria, independent of regression or any algorithm.)

The premise is simple and kind of fun:

By sifting through aggregations of data, the regression technique can uncover levers of causation that are hidden to casual and even expert observation. And even when experts feel that a particular factor is an important determinant of some outcome, the regression technique literally can price it out.

Ouch, experts, that’s gotta hurt. (BTW, regression and random A/B testing are the key tools.) But, as I say, author Ian Ayres isn’t boosterish about it. He has a great passage:

the power of corporate tera mining creepily suggests the opening lines of Psalm 139:

you have searched me and you know me
you know when I sit and when I rise; you perceive my thoughts from afar.
you discern my going out and my lying down; you are familiar with all my ways.

Where it gets interesting for a designer is in the chapter on random testing. There, designers are put under the same scrutiny as baseball scouts and other experts.

Here, we get an example from Offermatica, who, contracted by Monster, tested elements on an employees home page. No surprises here, it was the usual testing: launch a bunch of ideas to random audiences and see what flies. Below are the two candidates that finally got tested.

So some quotes from the Offermatica guy:

I go to meetings where have all these people sitting around a table claiming authority. You’ve got the analytic guy who has the amulet of historical certainty. You’ve got the branding guy who has this mystical certainty about what makes the brand stronger. … What’s missing is the customer’s voice.”

From Ayres:

Offermatica not only has to do battle with the in-house analytic guy who crunches numbers on historical data, it also has to take on “usability experts” who run-hyper-controlled experiments in university laboratories. [They] are sure of certain axioms that have been established in the lab — things like “people look at the upper left hand corner first” or “people look at red more than blue”. … [The CEO of Offermatica] responds, ‘’In the real world an ad is competing against so many other inputs. There’s no such thing as a controlled experiment. They cling to a sandcastle of truth in a tsunami of other information.”

Interestingly, Ayres challenges the reader, “if you have a good graphic eye”, to see which box you think tested better. He personally “finds the curved icons of the lower box to be more appealing” which is what Monster thought. Guess what? They were both wrong! (I’m happy to say that my sandcastle of truth said the clutter of curves, the crowded text and the fuzzy arrow (vacancy here!) were junk that most users would ignore.)

Is design expertise a sandcastle of truth? Is chess a puzzle that computers have solved? Is the soul a lie we tell ourselves so that we get out of bed? Should I just stick to comix?