My Survey Process
In graduate school, my professor Dr. Jason Siegel (see Acknowledgements) recommended a book to anyone writing surveys (pictured below). At first glance the book, published in 1996, looked like it might contain outdated information. After reading and using the book to write my own surveys, I realized these principles were ahead of their time and definitely still applicable today. I would say the top three things I learned were 1) think about all the ways you could confuse your respondents and plan to avoid them, 2) use visual elements to show the structure of your survey and make it look professional, and 3) use your understanding of cognitive processes to order the elements of your survey.
Research Plan
This step helps me think about what my goals really are and how I will execute. In academia, this was called a proposal. In industry, it’s been called a research plan. Either way, I think the key elements are as follows:
1- Understand the background of the topic you are creating a survey for. This could be a review of prior literature, or it could be talking to your stakeholders to understand what we know now and what we hope to know after the survey. At this point, also be thinking about who your stakeholders are and what input you will need from them.
2- Use this information to set goals. What do you want to understand about the respondents? What information, feedback, or behavioral patterns do you want to extract from them?
3- Set some outcomes. Pretend for a second that you have your survey results: what are you going to do with them? If a goal was to understand a certain behavior pattern, the outcome might be the creation of personas informed by research.
4- Now is when you start thinking about the execution of the survey. Make sure you understand who you are targeting, how many respondents you hope to reach, how you will create and distribute the survey, and how much time on average the survey will take.
5- Set a schedule or timeline for your survey. You might need to set meetings to talk to your stakeholders about their goals. You might need to set up a session to have someone review and test a draft of your survey. You might need need to leave some time to make any corrections or updates. You’ll also want to have a date in mind for when the survey will go live and when you will close it.
6- Start executing on your schedule. Set up meetings, start drafting your survey, start obtaining any resources or information you need. Refer back to the book I talked about above.
See my sample research plan below. I learned a great deal about research plans from my manager at ServiceNow, Teena Singh (see Acknowledgements).
My Sample Research Plan
Background Information
Stakeholder X says that there is a set of behaviors that we are interested in knowing more about. We know what the behaviors are, but we don’t know who engages in them and how often. We have a new tool that we think will help with achieving all of these behaviors and we want to survey a particular group of people to find out if they think it will be useful and why.
Goals
Understand who engages in a set of behaviors
Understand frequency of engagement in a set of behaviors
Get feedback on proposed tool to help with a set of behaviors
Outcomes
Create personas of who does what behaviors
Document 3 most frequent behaviors
Determine useful aspects of the proposed tool
Determine areas of improvement for the proposed tool
Execution
30 completed surveys over a period of 1 week
Reach these 10 companies
Build survey on Qualtrics
Keep the survey under 10 minutes
Consult stakeholders X, Y, and Z along the way
Obtain key screenshots and descriptive text
Timeline
Monday: schedule a goal-setting session with stakeholders X, Y, and Z
Monday: ask someone what screenshots and descriptive text are best to use
Wednesday: review principles of survey methodology in Thinking About Answers, draft of survey completed by EOD
Thursday: have a colleague test the survey, make any necessary changes
Friday: schedule session with stakeholders X, Y, and Z to review the survey draft, make any necessary changes
Following Monday: launch survey
Following Friday: close survey
Survey Creation Tools and Analyses
My first survey creation tool was Qualtrics. This tool is powerful and robust and is great for creating surveys as well as experiments. I used this tool for my Master’s thesis, for my Survey Research Methods course, and for every research project in-between. I usually use RStudio for doing any analyses with my Qualtrics data.
When I arrived in industry, the tool that was available was Decipher. I would say Decipher is less powerful and robust in that you have less control over the design of the survey and coding in non-standard elements. However, the research in industry is fast-paced and Decipher is quick and easy to use. It also provides a great analysis tool with instant results and visualizations.
Survey Building in Qualtrics with ANOVA in RStudio
The following video will give a glimpse into my process for building questions and organizing survey flow in Qualtrics. It also includes a preview of how I analyze data, with clips of ANOVA analyses from my Master’ thesis.