Intended for healthcare professionals

Education And Debate Hands-on guide to questionnaire research

Selecting, designing, and developing your questionnaire

BMJ 2004; 328 doi: https://doi.org/10.1136/bmj.328.7451.1312 (Published 27 May 2004) Cite this as: BMJ 2004;328:1312
  1. Petra M Boynton, lecturer in health services research (p.boynton{at}pcps.ucl.ac.uk)1,
  2. Trisha Greenhalgh, professor of primary health care1
  1. 1Department of Primary Care and Population Sciences, University College London, Archway Campus, London N19 5LW
  1. Correspondence to: P M Boynton
  • Accepted 17 March 2004

Anybody can write down a list of questions and photocopy it, but producing worthwhile and generalisable data from questionnaires needs careful planning and imaginative design

The great popularity with questionnaires is they provide a “quick fix” for research methodology. No single method has been so abused.1

Questionnaires offer an objective means of collecting information about people's knowledge, beliefs, attitudes, and behaviour.2 3 Do our patients like our opening hours? What do teenagers think of a local antidrugs campaign and has it changed their attitudes? Why don't doctors use computers to their maximum potential? Questionnaires can be used as the sole research instrument (such as in a cross sectional survey) or within clinical trials or epidemiological studies.

Randomised trials are subject to strict reporting criteria,4 but there is no comparable framework for questionnaire research. Hence, despite a wealthof detailed guidance in the specialist literature,13> 5 w1-w8 elementary methodological errors are common.1 Inappropriate instruments and lack of rigour inevitably lead to poor quality data, misleading conclusions, and woolly recommendations.w8 In this series we aim to present a practical guide that willenable research teams to do questionnaire research that is well designed, well managed, and non-discriminatory and which contributes to a generalisable evidence base. We start with selecting and designing the questionnaire.


Embedded Image

What information are you trying to collect?

You and your co-researchers may have different assumptions about precisely what information you would like your study to generate. A formal scoping exercise will ensure that you clarify goals and if necessary reach an agreed compromise. It will also flag up potential practical problems—for example, how long the questionnaire will be and how it might be administered.

As a rule of thumb, if you are not familiar enough with the research area or with a particular population subgroup to predict the range of possible responses, and especially if such detailsare not available in the literature, you should first use a qualitative approach (such as focus groups) to explore the territory and map key areas for further study.6

Is a questionnaire appropriate?

People often decide to use a questionnaire for research questions that need a different method. Sometimes, a questionnaire will be appropriate only if used within a mixed methodology study—for example, to extend and quantify the findings of an initial exploratory phase. Table A on bmj.com gives some real examples where questionnaires were used inappropriately.1

Box 1: Pitfalls of designing your own questionnaire

Natasha, a practice nurse, learns that staff at a local police station have a high incidence of health problems, which she believes are related to stress at work. She wants to test the relation between stress and health in these staff to inform the design of advice services. Natasha designs her own questionnaire. Had she completed a thorough literature search for validated measures, she would have found several high quality questionnaires that measure stress in public sector workers.8 Natasha's hard work produces only a second rate study that she is unable to get published.

Research participants must be able to give meaningful answers (with help from a professional interviewer if necessary). Particular physical, mental, social, and linguistic needs are coveredin the third article of this series.7

Could you use an existing instrument?

Using a previously validated and published questionnaire will save you time and resources; you will be able to compare your own findings with those from other studies, you need only give outline details of the instrument when you write up your work, and you may find it easier to get published (box 1).

Increasingly, health services research uses standard questionnaires designed for producing data that can be compared across studies. For example, clinical trials routinely include measures of patients' knowledge about a disease,9 satisfaction withservices,10 or health related quality of life.1113w3 w9 The validity (see below) of this approach depends on whether the type and range of closed responses reflects the full range of perceptions and feelings that people in all the different potential sampling frames might hold. Importantly, health status and quality of life instruments lose their validity when used beyond the context in which they were developed. 12 14 15 w3 w10-12

If there is no “off the peg” questionnaire available, you will have to construct your own. Using one or more standard instruments alongside a short bespoke questionnaire could save you the need to develop and validate a long list of new items.

Is the questionnaire valid and reliable?

A valid questionnaire measures what it claims to measure. In reality, many fail to do this. For example, a self completion questionnaire that seeks to measure people's food intake may be invalid because it measures what they say they have eaten, not what they have actually eaten.16 Similarly, responses on questionnaires that ask general practitioners how they manage particular clinical conditions differ significantly from actual clinical practice.w13 an instrument developed in a different time, country, or cultural context may not be a valid measure in the group you are studying. For example, the item “I often attend gay parties” may have been a valid measure of a person's sociability level in the 1950s, but the wording has a very different connotation today.

Reliable questionnaires yield consistent results from repeated samples and different researchers over time. Differences in results come from differences between participants, not from inconsistencies in how the items are understood or how different observers interpret the responses. Astandardised questionnaire is one that is written and administered so all participants are askedthe precisely the same questions in an identical format and responses recorded in a uniform manner. Standardising a measure increases its reliability.

Just because a questionnaire has been piloted on a few of your colleagues, used in previous studies, or published in a peer reviewed journal does not mean it is either valid or reliable. The detailed techniques for achieving validity, reliability, and standardisation are beyond the scope of this series. If you plan to develop or modify a questionnaire yourself, you must consult a specialist text on these issues.2 3

How should you present your questions?

Questionnaire items may be open or closed ended and be presented in various formats (figure). Table B on bmj.com examines the pros and cons of the two approaches. Two words that are often used inappropriately in closed question stems are frequently and regularly. A poorly designed item might read, “I frequently engage in exercise,” and offer a Likert scale giving responses from “strongly agree” through to “strongly disagree.” But “frequently” implies frequency, so a frequency based rating scale (with options such as at leastonce a day, twice a week, and so on) would be more appropriate. “Regularly,” on the other hand,implies a pattern. One person can regularly engage in exercise once a month whereas another person can regularly do so four times a week. Other weasel words to avoid in question stems include commonly, usually, many, some, and hardly ever.17 w14

Figure1

Examples of formats for presenting questionnaire items

Box 2: A closed ended design that produced misleading information

Customer: I'd like to discontinue my mobile phone rental please.

Company employee: That's fine, sir, but I need to complete a form for our records on why you've made that decision. Is it (a) you have moved to another network;(b) you've upgraded within our network; or (c) you can't afford the payments?

Customer: It isn't any of those. I've just decided I don't want to own a mobile phone any more. It's more hassle than it's worth.

Company employee: [after a pause] In that case, sir,I'll have to put you down as “can't afford the payments.”

Closed ended designs enable researchers to produce aggregated data quickly, but the range of possible answers is set by the researchers not respondents, and the richness of potential responses is lower. Closed ended items often cause frustration, usually because researchers have not considered all potential responses (box 2).18

Ticking a particular box, or even saying yes, no, or maybe can make respondents want to explain their answer, and such free text annotations may add richly to the quantitative data. You should consider inserting a free text box at the end of the questionnaire (or even after particularitems or sections). Note that participants need instructions (perhaps with examples) on how to complete free text items in the same way as they do for closed questions.

If you plan to use open ended questions or invite free text comments, you must plan in advance how you will analyse these data (drawing on the skills of a qualitative researcher if necessary).19 You must also build into the study design adequate time, skills, and resources for this analysis; otherwise you will waste participants' and researchers' time. If you do not have the time or expertise to analyse free text responses, do not invite any.

Some respondents (known as yea sayers) tend to agree with statements rather than disagree. For this reason, do not present your items so that strongly agree always links to the same broad attitude. For example, on a patient satisfaction scale, if one question is “my GP generally triesto help me out,” another question should be phrased in the negative, such as “the receptionists are usually impolite.”

Apart from questions, what else should you include?

A common error by people designing questionnaires for the first time is simply to hand out a list of the questions they want answered. Table C on bmj.com gives a checklist of other things to consider. It is particularly important to provide an introductory letter or information sheet for participants to take away after completing the questionnaire.

What should the questionnaire look like?

Researchers rarely spend sufficient time on the physical layout of their questionnaire, believing that the science lies in the content of the questions and not in such details as the font size or colour. Yet empirical studies have repeatedly shown that low response rates are often dueto participants being unable to read or follow the questionnaire (box 3).3 w6 In general, questions should be short and to the point (around 12 words or less), but for issues of a sensitive and personal nature, short questions can be perceived as abrupt and threatening, and longer sentences are preferred.w6

How should you select your sample?

Different sampling techniques will affect the questions you ask and how you administer your questionnaire (see table D on bmj.com). For more detailed advice on sampling, see Bowling20 and Sapsford.3

If you are collecting quantitative data with a view to testing a hypothesis or assessing the prevalence of a disease or problem (for example, about intergroup differences in particular attitudes or health status), seek statistical advice on the minimum sample size.3

What approvals do you need before you start?

Unlike other methods, questionnaires require relatively little specialist equipment or materials, which means that inexperienced and unsupported researchers sometimes embark on questionnaire surveys without completing the necessary formalities. In the United Kingdom, a research study on NHS patients or staff must be:

  • Formally approved by the relevant person in an organisation that is registered with the Department of Health as a research sponsor (typically, a research trust, university or college)21

  • Consistent with data protection law and logged on the organisation's data protection files (see next article in series)19

  • Accordant with research governance frameworks21

  • Approved by the appropriate research ethics committee (see below).

Box 3: Don't let layout let you down

Meena, a general practice tutor, wanted to study her fellow general practitioners' attitudes to a new training scheme in her primary care trust. She constructed a series of questions, but when they were written down, they covered 10 pages, which Meena thought looked off putting. She reduced the font and spacing of her questionnaire, and printed it double sided, until it was onlyfour sides in length. But many of her colleagues refused to complete it, telling her they found it too hard to read and work through. She returned the questionnaire to its original 10 page format, which made it easier and quicker to complete, and her response rate increased greatly.

Summary points

Questionnaire studies often fail to produce high quality generalisable data

When possible, use previously validated questionnaires

Questions must be phrased appropriately for the target audience and information required

Good explanations and design will improve response rates

In addition, if your questionnaire study is part of a formal academic course (for example, a dissertation), you must follow any additional regulations such as gaining written approval from your supervisor.

A study is unethical if it is scientifically unsound, causes undue offence or trauma, breaches confidentiality, or wastes people's time or money. Written approval from a local or multicentre NHS research ethics committee (more information at www.corec.org.uk) is essential but does not in itself make a study ethical. Those working innon-NHS institutions or undertaking research outside the NHS may need to submit an additional (non-NHS) ethical committee application to their own institution orresearch sponsor.

The committee will require details of the study design, copies of your questionnaire, and anyaccompanying information or covering letters. If the questionnaire is likely to cause distress, you should include a clear plan for providing support to both participants and researchers. Remember that just because you do not find a question offensive or distressing does not mean it willnot upset others.6

Conclusion

As we have shown above, designing a questionnaire study that produces usable data is not as easy as it might seem. Awareness of the pitfalls is essential both when planning research and appraising published studies. Table E on bmj.com gives a critical appraisal checklist for evaluating questionnaire studies. In the following two articles we will discuss how to select a sample, pilot and administer a questionnaire, and analyse data and approaches for groups that are hard toresearch.

This is the first in a series of three articles on questionnaire research

Acknowledgments

Susan Catt supplied additional references and feedback. We also thank Alicia O'Cathain, Jill Russell, Geoff Wong, Marcia Rigby, Sara Shaw, Fraser MacFarlane, and Will Callaghan for feedbackon earlier versions. Numerous research students and conference delegates provided methodologicalquestions and case examples of real life questionnaire research, which provided the inspiration and raw material for this series. We also thank the hundreds of research participants who over the years have contributed data and given feedback to our students and ourselves about the design,layout, and accessibility of instruments.

Footnotes

  • Contributors and sources PMB and TG have taught research methods in a primary care setting for the past 13 years, specialising in practical approaches and using the experiences and concerns of researchers and participants as the basis of learning. This series of papers arose directlyfrom questions asked about real questionnaire studies. To address these questions we explored a wide range of sources from the psychological and health services research literature.

  • Embedded Image

    References w1-w17, further illustrative examples, and checklists are on bmj.com

  • Competing interests None declared.

References

View Abstract