Why most surveys fail
Three reasons, and they compound.
The questions are leading. Asking how much you enjoy the weekly session invites a number, not a view. The people who never attend will skip the question entirely, which is the answer you needed.
Only your fans reply. A survey sent to everyone is answered by the members who are most engaged, which is exactly the group least likely to tell you what is wrong.
Nothing visible happens afterwards. Members learn that the survey is a ritual, and reply rates fall every year until it is not worth running.
What to ask
Eight questions is plenty. Ten is the ceiling. The shape that has worked for me and for the sites I have advised:
- What did you join to get done? Open. The single most useful question on the form, and it tells you whether the site still matches why people arrive.
- Have you done it? Yes, partly, not yet. Three options, no scale.
- What is the last thing you used here? Open. Compare the answers to what you think your best material is. The gap is instructive.
- What have you not used at all? Open, and the question people are most polite about. Ask it anyway.
- If the membership disappeared tomorrow, what would you miss? Open. This is the question that tells you what you are actually selling.
- What nearly made you leave? Open, optional. The most valuable answers on the whole form come from here.
- How likely are you to still be a member in a year? Three options: probably, unsure, probably not. Followed by an open why.
- Anything else. Always keep it. A surprising share of the real information arrives in this box.
Notice how few of these are ratings. A rating gives you a number you can chart and almost nothing you can act on. Open answers are slower to read and worth the time.
From practice
The year I added what nearly made you leave, eleven people described the same thing: a change to the publishing day that I had announced once, in one place, eighteen months earlier. It had been quietly costing me members ever since and not one person had ever complained about it directly.
Who to ask
Everyone, but count the groups separately. At minimum, split the answers into members of under three months, members of one to three years, and long standing members. Those three groups want different things and averaging them produces a picture that describes nobody.
Then do the harder part. Send the same short survey to people who left in the last year. Reply rates are low and the answers are worth ten times as much, because these are the views you never hear otherwise. Two questions is enough: what did you join to get done, and what made you leave. It pairs directly with the cancellation work in reducing member churn.
A survey that only reaches current members measures the people who stayed. The information you need is mostly held by the people who did not.
Running it
- Once a year, same month. Predictability makes it a habit rather than an interruption.
- Open for two weeks, one reminder. More than one reminder annoys people and changes nothing.
- Say how long it takes and be accurate. Five minutes means five minutes.
- Anonymous by default, with an optional name box. Some people will want a reply, most want to be frank.
- No incentive. A prize draw brings in people answering for the prize.
Reading the answers
Read every one before you count anything. Sort them into themes by hand. It takes an afternoon and it is the part that produces the insight, whereas the counting only produces the chart.
Two traps. The loudest request is rarely the most common need, and a request for a feature is usually a description of a problem, so look for the problem underneath it. And a single articulate complaint can dominate your thinking for a month; check whether anyone else said the same thing before acting on it.
Closing the loop
This is what makes next year's survey work. Within a fortnight, publish a short note: here is what you told us, here are the three things we are changing, here is one thing we are not changing and why.
The thing you are not changing matters. It shows the answers were read rather than filtered, and it prevents the same request arriving every year. Whatever you promise, keep it small enough that it is done before the next survey. Nothing kills reply rates like a list of last year's unkept intentions.
What I would do first
- Write eight questions, at most two of them ratings.
- Include what nearly made you leave.
- Send a two question version to people who left this year.
- Read every answer by hand and sort into themes before counting.
- Publish what you are changing, and one thing you are not, within a fortnight.
