Quick but Valid Member Surveys Done In-house
(c) Copyright 2012 JP Harrison.
Taking the pulse of the membership
is crucial. Until a few years ago, a
consultant or a research department was needed to do exhaustive studies on
mail-in paper surveys to periodically find out what it is the members really
want and other descriptive data about the members. Not anymore.
With the advent of the survey
websites and services, almost anyone with a few IT skills and a set of
instructions can launch a survey (see below).
This may sound a little too easy, but if done with diligence and proofed
carefully, such surveys can be valid (warning:
you have to stick with the rules to claim fairly that a survey’s results
are statistically valid). We’ll see why
statistical validity is important. It may
sound onerous, but most associations can perform a statistically valid survey
of the membership quickly and easily, and most of all, cheaply.
Why then do we need the experts? We don’t necessarily need them for the short
annual survey sent to the membership for a few targeted questions. We would, however, need those with strong
data analysis skills for advanced research, drill downs, and simulations and experiments
using data (such as risk analysis, decision trees, regression analysis, or
other complex comparing and benchmarking of groups). With more and more science and business graduates
getting fairly advanced statistical studies in school, in-house data analysis
will become the norm even in small associations.
What’s statistically valid and what’s
not, and why do we care?
I once worked with an association of liberal arts scholars. I did a decent study of the opinions about their annual meeting, with a careful treatment of the data which indicated what the attendees liked and their preferences for the next year. These were valid data upon which we could make decisions. What did this particular conference committee of scholars do with my well-researched data? They skipped over the loud and clear numeric results (they were liberal arts academics, after all) and went right for the handful of wild open-ended comments which came back in the surveys. Unfortunately, they acted on the comments, which were in fact in opposition to what the data said. It didn’t take long for problems to follow since the committee based their decisions for next year on the scanty anecdotal comments. Much of their time later on had to be spent undoing the damage.
We’ll save you the detail here, but
survey statistics is all about predicting what the whole population of your
group (that is, your organization’s entire membership) would say based on what
a small representative sampling of your group says. There are conventions and parameters which
describe how sure you can be that the sample reflects the population. You’d be amazed at how reliable a small,
properly-taken sample can be, especially if the population is large (it may be
counterintuitive at first, but the larger the population the easier it is to
draw conclusions in general).
One key sampling matter is
randomness. If there is a general
session in your conference of 500 persons, and you let them voluntarily fill
out and hand in surveys you might get 200 turned in on a good day. These will by and large be the 200 people who
loved it or hated it enough to fill out the survey and hand it in. This is how a lot of conference organizers gather their surveys. They
collect the voluntary occasional survey and use the results as if they’re
deeply meaningful. They’re probably not
deeply meaningful- unless, of course, almost every participant turns in a survey. It may
be better than nothing, but those types of survey results lack rigor. However, if you randomly picked 200 people
from the audience and enticed them to fill out a survey (good luck), you would
have fairly representative results. We
would say, in statistics parlance, that you could be 95% confident that your
data from the random group would be ± (plus or minus) 5% within the
population’s answer (or some other percentage, depending on the number of responses collected).
Now if the entire membership of your
association is say 10,000, and if you randomly select and get responses from
370 persons, then you have statistically valid data (check Chart 1). The data could be described as being CL 95%,
MOE ± 5%. This means Confidence Level 95% and Margin of Error of 5%. The
footnote in the polls done at election time just says MOE at 3% or 5% (3%
obviously being preferable), and the Confidence Level is understood to be
95% unless stated otherwise.
This should be enough theory to help
you handle the statistics of a quick survey; the next issue is the design of
the instrument. There is a very easy to
use book on designing surveys: Designing & Conducting Survey Research--a
Comprehensive Guide by Rea and Parker (sold by ASAE) which tells you
everything you want to know about designing questionnaires (the survey instrument). If you’re designing a survey to use year
after year with only minor adjustments, then reading the chapters on how to
construct a good question would pay off.
One tried and true approach is to
have easy-to-rank questions on a scale of 1 to 5 (generally called a Likert
scale) with one being “poor” and 5 being “excellent”. If you steer clear of labeling the middle
values (2, 3, and 4) with any adjectives, then you stand a better chance of
getting a useful and easy to understand score on the question. You will get a useful mean, say 3.7 or 4.2 on
a scale of 5 for each question. This can be a meaningful way to prioritize
services or to see how you’re doing relative to last year. Be sure to have a Not Applicable option to
help keep your data clean (if not applicable is applicable to the question,
that is). Here’s an example:
Question #1
Please rate the
quality of the Association’s offerings on a scale of 1 to 5, with 5 being
excellent:
Poor Excellent
1 2 3 4 5 Not applicable
a. Website □ □ □ □ □ □b. Magazine □ □ □ □ □ □
etc..
Question #2
Please rate your overall satisfaction with the Annual Meeting on a scale of 1 to 5, with 5 being very satisfied:
Very Dissatisfied Very Satisfied
1 2 3 4 5 Not applicable
□ □ □ □ □ □
It is vitally important to have several staff and/or members review the questionnaire before it goes out. Remember, it’s going to be used repeatedly for longitudinal results’ sake, so get the questions right. Results can be unreliable if any scales are not fair or are confusing. For instance, confusion will ensue if one question has the scale as 1=poor and 5=excellent and another question flips the scale to have 1 being excellent (or the “best”) and 5 being the “worst.” Ask only a few questions (try to have it take no more than 5 minutes), and explain in the cover email the purpose and the importance of answering this random sampling. Drawings for free gifts for participating, by the way, have made little difference in getting responses to the surveys I’ve run. Instead, explaining the nature of the survey, how quick it is, and that the person has been chosen at random to represent the whole membership has been the best approach to getting responses.
The
Nitty Gritty Instructions on Conducting the Survey
1) Find the number of responses you need to get a valid response for your population. Use the number of members (total members or paying members, or other population you want to find out about) and the following chart to figure out how many responses are needed for 95% confidence at 5% sampling error (good enough, if you want to be real impressive go for 3% error, like the Gallup Polls you see on TV). You may have to interpolate (gulp) to get the exact number of needed responses if your population is in the middle of two intervals (say it’s 8,000, instead of the 5,000 or 10,000 which are listed on the chart) or you could go for the higher number’s (in this case 10,000) sample size if you believe you may hurt yourself from too much interpolation. Interpolation sounds a lot worse than it is; it’s basically drawing a line between two points and using that line to see where the value you want would be (get someone who’s good at math or that staff person who brags on their MBA to earn their keep here).
[see chart 1 at end of article]
2) Figure out how many surveys to send
to get that number of responses. To sample for the
whole 10,000 membership, we’ll need to get 370 responses according to the chart. We’ll need to send surveys to about triple
that number at least if we normally have about a 30% response rate (a good rule
of thumb).
3) Get the needed number of random members. We’ll need
to send out probably 1,000 surveys to get 370 responses, so we need 1,000 random members to get our 370. Use an online randomizer to get
1,000 random numbers between 1 and 10,000.
10,000 because that’s our population.
My favorite randomizer is at http://www.randomizer.org/form.htm
and is very easy to use.
4) Match up the 1,000 random numbers to
1,000 members. You’ll
need a list in Excel of all your member ID numbers. Drop those 1,000 random numbers in a
spreadsheet next to your member IDs to get your 1,000 random members.
5) Check for a fair sampling.
Most of the time the sampling will be fine. To double check, take a look at the 1,000
picked to make sure they are in line with the general demographic profile. If a particular demographic, say age, is
important in the membership population, then make sure the average age of the
sample group is similar to the average age of the total membership. Check other important parameters if desired to
make sure the sample doesn’t look more skewed than the population.
6) Get the draft survey ready.
Pull the draft survey from Excel or Word into the online survey software
( SurveyMonkey, etc.), being sure to use the feature to run a test
beforehand.
7) Get the cover email ready.
Prepare a short one paragraph cover email to introduce the survey. This email from staff and will contain the
link to the online survey and an appeal for their response to help the
association get a meaningful random sampling. Be sure to give a deadline for
responses. If you have to send out a
reminder, by the way, it will have to go to everyone surveyed (if you’re
guaranteeing that the survey is anonymous) since you won’t be tracking who’s
responded and who hasn’t. You should
also mention in the cover email that since the survey is anonymous, there can’t
be any follow-up to requests made by the respondents in the comment section
unless they give their name, of course. If you send the survey out directly from a software (like SurveyMonkey), you can often--depending on the settings-- send reminders to only those email addresses (or IPs) who have not yet responded.
8) Launch the survey.
Once the survey is all lined up in SurveyMonkey and has been tested, use
the list from step four and launch the survey to the emails corresponding to
the random members. Hope your response
rate is the minimum needed. If not,
randomly survey some additional members to get to the number of responses
needed.
9) Examine the results and close the survey. The online software does a decent job of
presenting the results after the survey response period is done. There are some filter tools and such if you
want to more closely analyze the data. Examine
the results and export them to Excel for compilation. You will need to calculate the mean for any
Likert scale questions on the spreadsheet once exported, but it’s a breeze to
cut and paste the formula if you’ve kept the question format consistent. If the same questions were asked before, then
longitudinal (year to year) results can be drawn.
10) Interpret the results – it would be a good idea to bounce
the interpretation off others to double check.
There should be a footnote with the response rate, the sampling error
and the 95% confidence interval as part of the results. Voila, if you’re diligent, then the experts
don’t need to be brought in.
Make sure there is a consistent survey
process from your staff, so surveys don’t become too frequent or appear sloppy. There’s nothing that shoots down your
credibility more than tossing poorly thought out questions to the
membership. It is best to have someone
on staff responsible for reviewing all surveys before they are launched. Remember to explain that a random survey
differs from the usual participant’s satisfaction survey because it provides
much more reliable results. I’ve even
run surveys which also surveyed the Board members as a separate group in the
same survey. We compared the Board’s
results to the membership’s results.
Surprise! There were some
important deviations. Showing this to
the Board was very helpful to them as they formulated policy on particular
issues. After all, what we’re chasing
here is simply an efficient and reliable way to find out what the members
really want.
Chart 1 (note: there are a lot of these type charts found in an Internet search of "sample size charts"; this is a compilation, owing much to researchadvisors.com)
Population Size
|
± 3% Sampling Error
|
± 5% Sampling Error
|
100
|
92
|
80
|
250
|
203
|
152
|
500
|
341
|
217
|
750
|
441
|
254
|
1,000
|
516
|
278
|
2,500
|
748
|
333
|
5,000
|
880
|
357
|
10,000
|
964
|
370
|
25,000
|
1,023
|
378
|
50,000
|
1,045
|
381
|
100,000
|
1,056
|
383
|
1,000,000
|
1,066
|
384
|
Number needed at 95% Confidence Interval
No comments:
Post a Comment