The Flawed Logic With Post-Conference Surveys

Published by Steve Manicor | Topics: Conferences

  • Pin It
  • Pin It

Have you ever attended a short educational event only to receive a 45-question survey a few weeks after the event is over?

As an attendee, what’s that experience like? I don’t know about you, but I am usually back in the trenches at work and can’t remember if I felt inspired after the keynote session or what I thought of the third breakout session I attended on Day 2.

A typical conference survey may include:

  • The food
  • The general session
  • Each session
  • Each speaker
  • Each handout
  • Registration
  • Location
  • And, and, and….

.

Sure, it’s great that conference organizers want to know my opinion, but sometimes I think the logic is flawed when they send out these surveys:

Flawed Logic One – Sending out surveys way after the event
Attendees have a hard time remembering all the details (two weeks later), so the responses you get might be just guesses or empty.

Flawed Logic Two – Ask lots of questions
We’ve tried longer surveys here at Omnipress when we’ve benchmarked important trends. But we learned our lesson and shortened them up when we discovered that long surveys tend to be aborted leaving you with incomplete information and less respondents.

Flawed Logic Three – Lots of feedback will give us lots of answers
A long survey takes forever to create (especially if you have to have committee members approve), and it takes even longer (and perhaps the services of a statistician) to slice and dice the responses.

Conferences – What Really Matters

So, here’s the big question: What information are you really looking for? And, more importantly, isn’t there only one question you really want to know the answer to?

<pause and think about it>

If you did your job and delivered a memorable experience for your attendees, your big question should revolve around whether they are highly likely to return and tell others. This is why I admire the Net Promoter score – a straightforward loyalty metric that holds companies accountable for how they treat customers (event organizers should think of attendees, speakers, sponsors and exhibitors as customers). The opportunity here is to get to the point about how your customers really feel about your event in a single question.

On a 1 to 10 rating scale: “How likely is it that you would recommend our event to a friend or colleague?”

Based on their responses, customers are categorized into one of three groups:

  • Promoters (9-10 rating) – are loyal and advocating
  • Passives (7-8 rating) – are somewhat loyal, but have mixed feelings
  • Detractors (0-6 rating) – are not loyal and possibly negative

The percentage of Detractors is then subtracted from the percentage of Promoters to obtain a Net Promoter score. A score of 75% or above is considered quite high.

Now I am not going to advocate you don’t ask other questions because asking just one question doesn’t really give you any insights on how to improve your event. My marketing sense tells me you need to ask a few other open-ended questions regarding the core reasons why people attend meetings (networking and learning).

  • What educational sessions did you like the best?
  • How could we improve our education?
  • What did you like the most regarding the networking opportunities?
  • How could we improve the networking experience?
  • What did you like most about the overall event?
  • What could we do to make this event better?

I know you can’t easily quantify open-ended questions, but you can group the responses into buckets to see what floats to the top. Unlike radio or check-box style questions which can influence your respondents, the open-ended question allows them to share what they think.

Here’s an example: In the restaurant industry, Noodles & Co solicits feedback using the Net Promoter approach and just a few questions. It allows the customer to place value on the topics that are most important to them. And then, allows the customer to share their opinion for each topic.

Oh, and back to the timing of this survey. Two weeks after the event is two weeks too late! This survey should be sitting in your customer’s inbox while the event experience is fresh in their minds. Or better yet, capture attendees’ feedback before they leave for the airport and get back into work mode.

Let me ask…

  • Do you think you the concept behind Net Promoter plus a handful of open-ended questions would work for your event?
  • What reservations do you have about it?
  • How else would you measure loyalty?

I am curious to what challenges you have with your current feedback process.

 

About Steve Manicor

Steve is Omnipress' Director of Business Development. He has over five years serving the meetings and training industry. He leads our product/service leadership and development teams. ...read more



Tags: , , ,

Comments

1.

October 11th, 2010Christopher Uschan says:

You might find Jeff Hurt’s article, “Do You Want Satisfied Conference Attendees Or Loyal Attendees?” interesting as it relates to loyalty.

Twitter: chrisuschan


2.

October 11th, 2010Heidi Thorne says:

Yes, fewer, open-ended perfectly-timed questions are the way to go. And, yes, it must be solicited BEFORE they leave the event. It’s easy to ignore email, even relevant email, upon return to the office. There’s too much to do and it will be ignored.

Time to complete surveys must be included in the event’s activities. Don’t run up to last 3 minutes and then hope to get thoughtful feedback.

Thanks for another great post!

Twitter: heidithorne


3.

October 11th, 2010Tony Veroeven says:

One minor disagreement:

Quote: “Now I am not going to advocate you don’t ask other questions because asking just one question doesn’t really give you any insights on how to improve your event.”

The only thing I would say is that the number score and an open field that asks “What could we do to make this event better?” is all the insight you’ll need.

Whether it’s food, education, or temperature…the attendee will tell you what’s top of mind–what’s most important to them.

Make it easy on them, and you.

Twitter: tonyveroeven


4.

October 11th, 2010Jeff Hurt says:

Chris:

Great post that get people thinking about the purpose of surveys and evaluation.

Regarding your first flaw: Sending out surveys after the event.
If no one remembers anything about the event two weeks later, was the event worth it? Did learning occur? Was it a purple-cow experience that was unforgettable?

Your words about forgetting about what happened two weeks earlier reveal a bigger problem with some conferences. They are unforgettable. If we don’t remember the value of the event, then I question the ROI of attending in the first place.

As for when and how to send evaluations, the educator in me has a different perspective.
1. Each presenter and education session should be evaluated immediately following that presentation. 2. An overall conference evaluation should occur at the end of the final event at the conference.
3. Another overall conference evaluation with a set of different questions should occur two weeks later. It should contain questions like, “Were you able to apply anything you learned at this conference?” If yes, please share how.”
4. A final overall conference evaluation with another set of different questions should occur 60 days later.

That will give the organizers real data if attendees moved from awareness to action and if they applied anything they learned or bought new products/services because of their attendance. Conference organizers should keep those survey questions the same from year to year with minor tweaks. Then they can compare conference evaluations, speakers and presentations from year-to-year. They can monitor if their attendee satisfaction and loyalty are improving or decreasing.
Jeff Hurt recently posted..How The Billboard Effect Influences Your Tradeshow

Twitter: JeffHurt


5.

October 11th, 2010Christopher Uschan says:

Jeff —

Good point on the not remembering 2 weeks after… if the event was good, you’d remember!

I like the different sets of questions at different intervals post event too.

Thanks for the interesting perspective (as always)!

Twitter: chrisuschan


6.

October 11th, 2010Philippa Gamse says:

As a speaker, one thing that I very rarely hear about is the impact that my content had on the audience. It’s one thing for them to rate my presentation skills on the day (which is important, of course), but I’d love to know whether my program persuaded them to try some alternative approaches when they got back to the office, or whether they learned something that made a real difference to their bottom line.

In contrast to your examples where it’s hard to remember how you felt 2 weeks after the conference, these are questions which can be asked a few weeks (or even a few months) later. It might be difficult to get a lot of responses, but it might be very interesting to try.

Twitter: pgamse


7.

October 11th, 2010Adam Ramshaw says:

Chris,

Good post – one that I agree with, mostly agree actually, and disagree with a little.

First on the agree side: Net Promoter Score. NPS is a good metric (not the only one but a good one) because it is related to revenue growth. Increase NPS and you will probably increase revenue growth.

If your readers would like a little more information we have put together a Net Promoter Score Introduction (http://www.genroe.com/whitepapers/net-promoter-score-nps-an-introduction) .

The disagree side revolves around having just a few qualitative questions.
We have done quite a bit of work on post conference surveys. Our approach has been to identify the key drivers of customer loyalty and measure performance against them. This has come from qualitative responses combined with NPS.

We then ask a short series of tightly focused attribute questions about these key drivers of loyalty to determine if the conference has delivered on those areas. After having run the process for some time we can tell very accurately what drives customer loyalty at conferences (it does vary by conference type) and exactly where an event has fallen short or exceeded.

This list of attribute questions started out much longer but through analysis we determined the areas that were key to attendee loyalty and narrowed it down to a much shorter list.

At the end of the survey we do use some qualitative feedback questions to understand how to improve in the long term and for that specific event.

This provides the information that you are really looking for:
1. What did attendees think of the conference
2. Which key drivers were delivered well and which were delivered less well.
3. How to improve.

Adam Ramshaw
Adam Ramshaw recently posted..The 19 point call centre review you can do today

Twitter: genroe


8.

October 12th, 2010Beth Ziesenis says:

So the other day I ordered a chicken fresco burrito at Taco Bell and came home with a beefy bean burrito. My receipt had a feedback number, so I called to express my disappointment.

They asked one magic question: “Based on your experience today, would you recommend this restaurant to friends and family?”

Now sure I was a little bummed about the burrito error, but I didn’t think it was a reason to stop visiting the restaurant. The black/white question threw me off and made me feel a little cheated. I wanted to say, “Hey, Taco Bell folks, people carefully choose their chicken burritos from the light menu because they’re getting married, you know. Your carelessness just sabotaged someone’s diet!” But the lack of depth in their question made me feel like all they wanted to know was if I planned to give them money again and definitely left the impression that they didn’t really care about the feedback. They did give me an opportunity to leave a message for the manager or the employees. But then I just felt whiny.

So my Taco Bell experience taught me that although we might really just care about one question, we may want to give attendees a chance to communicate their way, not just ours.

PS — I ate the beef burrito.
Beth Ziesenis recently posted..Webinar Recording- 5 Great Graphics Tools

Twitter: Avenuez


9.

October 12th, 2010Christopher Uschan says:

Philippa – Your point about knowing if your session had impact is kind of what Jeff is talking about above… asking a question 6 mo later. Not sure how that could be done (on a larger event), but a smaller meeting, very possible.

Adam – Based on your response, I’d say, “you get it!” Do you think you could you share the extra questions you might ask?

Beth – Shame on you for eating at Taco Bell! Over the years, I have given some local restaurants and FF chains a 1-year hiatus based on dissatisfaction… Then told all my friends how I don’t like that place. Bad thing for the restaurants is they didn’t even give me a survey. That said, I agree that you need a place for people to share straight comments.

Thanks all for commenting.

Twitter: chrisuschan


10.

October 12th, 2010Adam Dorrell says:

Chris
just like to add my small contribution. We think Net Promoter is a great way for speakers to rate their audience. I’d just finished a presentation same subject! Hope you like our free tool to measure presenter score and collect feedback – Recommendi.

http://www.slideshare.net/Recommendi/recommendi-forpresentations11-oct2010

Twitter: recommendi


11.

October 12th, 2010Adam Ramshaw says:

Chris,

Thanks for the “you get it” reference. My goal is always to help my client improve their business. In this case their customer loyalty has been increasing for a while based on taking action from our analysis.

Unfortunately, as you might guess, due to client confidentiality I don’t feel that I can make the extra questions public. I hope you understand. Always happy to chat in general though.

Adam Ramshaw
Adam Ramshaw recently posted..The 19 point call centre review you can do today

Twitter: genroe


12.

[…] Engage 365 Community Manager and Omnipress Marketer Chris Uschan recently wrote about how he forgets most of what is shared at a conference two weeks later. […]


13.

October 14th, 2010Chris Atkinson says:

Chris,
I agree with the other readers, that surveys are most effective immediately after an attendee leaves a session or exhibit. Digital surveys and information capture can help streamline the process too.

StudioPMG
An trade show marketing agency
Free report:
How Immersive Convention Marketing is Increasing Marketing Results


14.

October 14th, 2010Adrian Segar says:

Here’s what we’re doing at EventCamp East Coast (November 12-13, Philadelphia) to maximize getting prompt conference feedback.

We’ll print the evaluation questions in a booklet (which also contains additional event information) that will be given to attendees when they arrive on site. This booklet will give attendees the opportunity (which we’ll promote) of writing down their impressions during or right after each session. We will also duplicate these questions in an online survey, also available at the start of the event, so attendees can report their experience as soon as they want.

What I always tell participants several times during my conferences is a) how much we appreciate their feedback; b) that it _will_ be read; and c) that it _will_ be acted on to make their future events better.

We will send a couple of polite follow up emails to attendees who have not completed the survey during the week that follows the event, repeating the points made above.

I use a combination of rating scales, spaces for session comments, and various open-ended questions in such surveys. Different attendees will answer different questions; people will always skip some questions, so I’d rather ask a few too many questions than a limited set.

Typically, I’ll see about a 60% response rate using the above approach.
Adrian Segar recently posted..How 1984 turned out like 1884

Twitter: ASegar


15.

October 15th, 2010Christopher Uschan says:

Adrian – I should just sent you a blue ribbon right now.

Essentially, the message to the attendees is that you care about what they say and how they feel about “their” event. By starting right from day one pointing this out and providing the necessary feedback channels is a great way to enforce this.

Thank you for sharing how it works… and I look forward to hearing how this goes with EventCamp East Coast.

c

Twitter: chrisuschan


16.

October 8th, 2011Conference Calls says:

Conference Calls…

[…]The Flawed Logic With Post-Conference Surveys – The Big Ideas Blog from Omnipress[…]…


17.

May 2nd, 2013Sildenafil Oral Jelly says:

This Posting is very nice!Thank you for the info.


18.

July 30th, 2013Syers says:

I am extremely impressed along with your writing talents and also with the structure in your blog. Is this a paid subject or did you customize it yourself? Either way keep up the nice quality writing, it’s rare to peer a great blog like this one nowadays..|


Leave a Reply







CommentLuv badge