How to achieve high response rates in online surveys

One of the most effective ways of enhancing the learning experience and teaching offer at universities is to ask the students for feedback. UK higher education (HE) providers use a range of formal and informal mechanisms to hear the ‘student voice’ and understand the student experience. Most typically, HE providers survey their students at the end of each teaching period, and increasingly mid-module. The data and information obtained from this feedback are embedded within institutional quality enhancement processes.

Student module and course evaluation surveys are now almost universally administered and managed online. A review of how HE providers approached course and module surveys found that the most significant developments for HE providers in implementing an online evaluation system were:

The review also found that questions typically asked students about teaching, assessment and feedback, academic support, learning resources and their overall satisfaction (evasys, 2016). 

Dommeyer et al (2004) identified some common features to online surveying practice. Typically, online evaluation would involve the following:

The many benefits of using online student module evaluation is recognised in the published literature (Dommeyer et al., 2004; Salmon et al. 2004; Watt et al. 2002). Watt et al. (2002) state that ‘using web-based evaluation questionnaires can bypass many of the bottlenecks in the evaluation system (e.g. data entry and administration). Another benefit of online evaluation is that it removes the need to administer surveys in class (Dommeyer et al. 2004) thus creating efficiencies (e.g. staff time and paper).

However, Nuttly (2008) identified that one of the most significant and pervasive challenges in an online student module evaluation system is that response rates are low. Research shows that response rates are generally lower when an online instrument is used, as compared to an in-class, paper-based instrument (Nulty, 2008; Anderson, Cain, & Bird, 2005; Ernst, 2006; Kulik, 2009; Benton, et al, 2010). This has raised concerns around how far valid and reliable conclusions about teaching effectiveness can be drawn from the data (Dommeyer, Baum et al. 2002).  The nature of online evaluation is that it depends on student cooperation, unlike paper-based evaluations where surveys can be administered to a captive audience. Throughout the literature, low response rates are cited as the key disadvantage of an online evaluation system.

This paper considers a variety issues surrounding response rates in online course evaluation. What are the implications of low response rates in online course or module evaluation? What is an adequate response rate, that is, what response rate can be considered large enough for the survey data to provide meaningful evidence for assurance and enhancement purposes? What practical strategies and advice are available to help boost response rates?

So what are the reasons for non-response and what are the implications of low response rates? Non-response can be due to several reasons. There can be due to a lack of motivation as the students are not completing them in class but in their own time out of class. As these evaluations are most commonly administered at the end of term, students do not necessarily feel they benefit from any improvements that may be introduced to the module as a result of their feedback. At other times, students may believe that only the teacher will see their feedback or that their views will not be taken seriously (Chapman and Joines, 2017). Students may also choose not to respond due to survey fatigue, that is they feel they are asked to complete too many surveys.

A study by Anderson et al (2006) found that most students did not respond to an online evaluation because of the following four main reasons: the students were disengaged (that is, they forgot or were too busy); they had technology problems; they perceived no benefit;  and lastly, ‘other’ reasons.

Concerns around low response rates is whether those who have participated in a survey are representative of the entire population. In other words, if respondents and non-respondents have very different views, then the results from the survey would not reflect the opinion of the population as a whole. For this reason, higher response rates are generally more desirable in order to minimise the potential effect of non-response bias.

In course and module evaluation, low response rates may affect the accuracy of data. Data from these surveys are regularly used within a programme’s quality management process. So if respondents have different views from non-respondents, it is possible that the feedback provide could influence academic staff to respond in a way that may be different if they had received feedback from all students. At the department or school level, summative judgements may be made on a teacher’s performance based on erroneous data. Low response rates impacts the credibility of the data and may have some real implications on decision-making.

Liu and Armatas (2016) assert that without adequate response rates, the benefits of implementing online surveys including efficiencies in survey administration (survey distribution, collection and analysis), data management, as well as rich open text comments are not worthwhile to achieve.

Achieving a higher response rate means that the results that are collected are likely to be more representative and there is greater confidence that the student feedback is meaningful in order to drive improvements in teaching and learning (Brennan & Williams, 2004).

In Raising response rates, the HEA (2016) identified that:

The more students that take part in the survey, the more meaningful the data. The purpose of raising response rates is to make the survey more effective for enhancement across the institution. A high response gives greater confidence to results and makes it possible to deliver results at levels relevant to staff delivering teaching and learning.

However, the publication does not offer ‘explicit targets on what response rates institutions need to achieve’ but offers the following ‘general guidance’:

• the response rate to UKES 2015 was 15%;
• 15% is a low response rate for an online survey;
• 25% is an average response rate for an online survey; the response rate to PTES 2015 was 29%;
• 35% is a good response rate for an online survey;
• the response rate to PRES 2015 was 41%;
• 45% is an excellent response rate for an online survey.

According to Nuttly (2008), the best reported response rates for online surveys (47%) are only adequate for class sizes above 750 students. So what happens if the class size is less than 750 students? Using Dillman’s formula (2000), Nuttly calculated how many respondents are required and therefore, also the required response rate, table 1.

Table 1:  Required response rates by class size summarised from Nulty (2008), based on a formula by Dilman (2000)

Nuttly, however, was insistent that his recommended response rates are only a guide of what ‘in a theoretically ideal world’ would be considered adequate. He stressed that even if the response rates are achieved, ‘great care is need to be sure that results for a survey are representative of the whole group of students enrolled’.

So what are the strategies for raising online response rates? This has been pretty well-documented in the published literature (Chapman and Joines, 2017; HEA, 2016; Naidu et. al., 2014). With over 20 years experience in providing online evaluation solutions, the following section summarises the best of what we know and what we have found. But before that, let’s go back to basics with survey design. Rutherford (2016) has advised that:

Research has shown that surveys should take 5 minutes or less to complete. Although 6 – 10 minutes is acceptable, those that take longer than 11 minutes will likely result in lower response rates. On average, respondents can complete 5 closed-ended questions per minute and 2 open-ended questions per minute.

Staff engagement in the process is a key element in achieving higher student engagement. Students are more likely to complete the surveys when their lecturers and tutors encourage them to. Therefore it is important that staff also recognise the benefit of online evaluation and take the time and trouble to promote them to their students. Various ways to increase staff engagement include:

Linked to engagement is communication. Promoting online evaluation throughout the university in the lead up to and during the live survey period will raise awareness and encourage responses. This can include:

Students are more likely to complete the survey if they know their feedback is important to their tutors, department and university, and that their feedback is acted upon. Closing the feedback loop will make students feel a part of an effective, value-added process and this in turn will drive higher response rates for future evaluations. This can be done by:

1. sharing the results of previous years and how feedback has influenced decisions and actions e.g. through a link to a web page with more information; and
2. using of the evasys Closing the Loop function (through e.g. the evasys Instructor Portal), where the Module Leader can record reflections on students’ responses and generate a Student Report to students (to which they can then respond) – either sent via email or embedded in the VLE.

This ‘you said we did’ feedback at the granular module level creates a virtuous loop of subsequently improved responses to future surveys, as students can see that their voice is being listened to and acted on.

So where does this leave us with response rates in online evaluation? No feedback mechanism is perfect but it is better to obtain feedback than not at all. The strong emphasis placed on student voice and continual quality management in recent years as a result of both higher education policy and marketisation means that student surveys are probably here to stay for the foreseeable future. The benefits of online evaluation far outweighs its disadvantages. Beyond the savings in class time, administrative burden and the use of paper, online evaluation offers students the opportunity to access the surveys when they want and from where they want, and hopefully this will also them the space to provide more considered responses.

When it comes to what is adequate when it comes to response rates, the short answer is: it depends.

1. Contextualisation is important: large class sizes will require a lower response rate than small classes (see table 1).
2. Triangulate with other data sources: making high-level decisions on the outcomes of a single, or a small number of responses would be problematic, particularly in the absence of ‘local’ module-level knowledge. So ensure that decision-making is informed by multiple sources of evidence and metrics including survey outcomes over time, NSS ratings and other module information such as progression rates and grade outcomes.
3. Data literacy: underlying the first two points is the need for dedicated staff who are able to make sense of the data and feed it back into the system so that it can support decision-making.

High survey response rates are always sought and valued as they are illustrative of more engaged students and more credible and insightful data. We hope that we have demonstrated in this paper that these can be achieved through a combination of cultural and technological interventions.