Posted by: rayseghers | November 5, 2009

Norms: What Do They Really Tell You?

Like a good high school debater, I have represented both sides of the normative comparisons issue.   So, what do they tell you?  Short answer:  A lot, but maybe not as much as you think.

And, as you might suspect, there are many, many subtopics, such as:

  • Should they be client-based or study-based?
  • How should they be grouped – individual, unit, or organization?
  • How far back should they go?
  • How finely can the data be cut?
  • How should the comparisons be displayed to the client?
  • etc., etc., etc.

Don’t worry.  I am not going to cover these in this post – maybe later.

Of course, the primary purpose of using norm comparisons is to tell the user whether their results are good or bad.  Norms try to account for differences in the question itself and for differences in the group of people being surveyed.

Let’s take a simple example. 

A supervisor receives his/her survey results.  For question 1, the mean score is 3.45.  Is this good, bad, or ugly?  Well, first of all, we need to know a little more about the response scale.  We find that it is a 5-point scale where 1 is the least favorable response and 5 is the most favorable response.  So now we know that the score of 3.45 is above the mid-point (3.00 on a 1-5 scale).  It could be a lot worse, but is that a good score?

OK, so this example is not that simple after all.  We still need to know more about the response scale.  Is it a linearly increasing scale (e.g., Very Little Extent, Little Extent, Some Extent, Great Extent, Very Great Extent) or is it a center-balanced scale (e.g., Well Below Expectations, Below Expectations, Meets Expectations, Above Expectations, Well Above Expectations).  For a linearly increasing scale a score of 3.45 may not be that good, since the mid-point score of 3 would probably be considered only so-so.   On center-balanced scale, however, a score of 3.45 would probably be viewed more positively since a mid-point score of 3 is considered to be a good score.  Of course, norm comparisons would quantify this for us.  {Please note that I am assuming that our norm comparisons would have asked the same question and used the same response scale.  Also note that this seemingly obvious caveat is often violated.}

We haven’t even looked at the question yet.  This is where norms can be most valuable.  All survey questions are not created equal.  Some questions ask about issues and behaviors that are relatively easy to accomplish and so they generate fairly high scores.  Other questions ask about tougher issues and tend to generate much lower scores.  We can’t just look at the numbers.  Again, norm comparisons would quantify this for us.

So, one purpose of norms is to account for differences in the question and in the response scale.

But what about our group of people?  Does it matter if they are engineers or accountants or managers or production-line workers?  Most people would certainly argue that it does matter.  So, to determine if our score of 3.45 is good we need to have a relevant frame of reference – an apples-to-apples comparison.  A large data bank that can be cut by industry, region, job level, etc. can be very useful in helping us evaluate our survey results.  If, in our example, the relevant subgroup in the data base has a score of 3.90, then we know that our score of 3.45 is below par.  Now we know that this is probably an issue that we want to work on.

Why do we care what other people said?  On the one hand, we don’t.  Our group is unique.  On the other hand if other groups can post a score of 3.90 why can’t we?  Norm comparisons give us valuable information and help us to interpret our results, but ultimately, the decision is up to us.  Just because a survey score is low does not necessarily mean that it is a problem for our group.  Remember the discussion in an earlier blog post about statistical significance and meaningfulness?  Well, this is the same idea.  Norms provide the statistical information but we must provide the values that determine meaningfulness.

So, take two norm comparisons and call your survey consultant in the morning.

http://www.segherssurveyconsulting.com

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Categories

%d bloggers like this: