Monday, November 19, 2012

Too Big to Measure

image source
This is a true story. My association Board chair and I were chatting not long ago, talking about the mission and big-picture objectives of our association, and I mentioned, literally without giving it a second thought, that we were incapable of measuring progress on some of the things we had set for ourselves.

It was only the look of shock that passed over his face that made me step back and think about what I had just said.

Maybe I was thinking about Shelly Alcorn's recent post, Is Your Mission Bigger Than What You Can Measure?, where she argues that associations had better be focused on things that are difficult to measure. If you can measure it, she seems to say, you're not aiming high enough.

Or maybe I was thinking about Seth Godin's recent post, Avoiding the False Proxy Trap, where he cautions against settling for something that's easy to measure as a proxy for what you're truly trying to achieve. Tactics that you employ to move the proxy needle will likely have very little to do with affecting change for the bigger picture.

Thought-provoking stuff. But what was more likely on my mind was an email discussion I had recently had with Jamie Notter in response to his post, Taking High Performance Seriously. Reacting to a much-circulated slide show from Netflix on their "high performance" culture and values, Jamie insightfully suggested that achieving high performance was more difficult than the slide show lets on--and that at least part of that difficulty was related to determining what to measure.

So if you want to get serious about “high performance,” then I say go for it. Let’s raise the bar. But get ready for the hard work of clarity. Get ready to spend some time (involving everyone) in determining what to measure (and how). And please don’t default to what we already know (hours worked, or overall organizational performance). We need more sophistication than that if we’re going to do this right (I was happy to see that Netflix says hours worked is “not relevant”). We can’t assume our people are like cogs in a machine, where we KNOW they accomplish more if they spin on their axis for 10 hours than they do if they spin for 8. We can’t oversimplify it and say if the company does well, then everyone is performing well. Let’s roll up our sleeves and experiment with some new metrics and try to learn enough from the experiments that we can create the clarity that would truly drive a “high-performance” culture.

I thought it was great advice. Something I wanted to learn from. So I reached out to Jamie, asking him for a reference to an organization who had done exactly what he was advocating, who had done or was doing the hard work of engaging everyone to determine what to measure and how. I thought he would know of at least a handful off the top of his head.

Turns out he didn't know any. Not one.

It's an interesting conundrum. Pledge yourself to something that can be measured and achieve things of lesser significance. Or pledge yourself to something of great significance and give up on the idea of finding a metric that truly tracks your progress.

Does any one know of a third choice?


  1. I think you've painted a bit of a false dichotomy while raising a very important question. And certainly some have long argued that not everything that maters can be measured in the manner that many would like.

    So I would echo what Jamie advocates in terms of getting clarity around purpose and performance. That's the precursor to then asking "how would we know how well we are doing on what we just agreed we want to accomplish?" While the responses may not be immediately obvious, I tend to believe thoughtful discussion would fail to yield any possible indicators for progress.

    And you shouldn't be too surprised when those of us in the consulting ranks can't give examples for some of what needs to be done. We're often brought in for one-off engagements that don't necessarily correlate to the ongoing and deeper efforts that a question like yours requires. Turning to other execs might yield a different response.

    1. It's certainly a forced dichotomy, Jeffrey, but I'm not ready to accept the idea that it's a false one. One undertone I wasn't explicit about is the distinction between what can be measured and what a given organization can measure with the resources it has at its disposal. That's another limiting factor that has to be added to the kind of thoughtful discussion you advocate. What are we capable of measuring that would let us know how well we are doing? That is often the chasm that associations have a difficult time crossing.

  2. While some initiatives are quite difficult to measure, I must advocate that there are ways to measure the effectiveness of anything. Yes, everything. It might take some innovation, imagination, the involvement of several and elbow grease, but the effects can be measured.

    One example off the top of my head: Elling Hamso who is measuring Event ROI. I know his methodology can be also be used for many many other things that you thought you couldn't measure...

    Judy Gould, MMR

    1. Thanks, Judy. I'm going to respectfully disagree with you. In the abstract, perhaps, everything can be measured. But in the practical, resource-limited day-to-day, it just isn't true.

  3. Eric,
    Great stuff. We have all struggled with this as well. There is always opinion surveys to measure the big things. For instance, if you decide on a goal of making your industry more attractive to graduating college students, you could do a benchmark survey of college juniors and redo that survey in a year or two to see if you moved the needle.

    Of course, that takes resources, but if it is worth doing, it should be worth measuring.

    1. Thanks, Dave. But how do you know it was your efforts that moved the needle?

    2. You can never be sure, but if the needle doesn't move, you know your efforts are NOT working. The attempt to measure, I would argue, has benefits of its own. If staff and members are involved with an attempt to get valid measurement, it will help them focus on the task/goal. The opposite is also true.

    3. Great point, Dave! The attempt to measure is an engagement and education exercise. Love it!