Home | Feature | Do questionnaieres affect findings?

Do questionnaieres affect findings?

17/08/2013 01:17:00 BY MPHIYAKHE M. NTULI

Font size:

After receiving numerous requests from the readers, on how to design a research questionnaire, I found it proper to discuss some of their concerns. Thanks for the constant feedback and suggestions.

The heart of a survey is its questionnaire (Krosnick and Presser, 2010). In fact your research or findings are as good as your questionnaire or the instrument used to collect the data. According to the National EMSC Data Analysis Resource Center, the manner in which a question is asked can greatly influence the results of a survey.

For example, in the early 1980s an NBC/Wall Street Journal poll asked, “Do you favour cutting programmes such as Social Security, Medicare, Medicaid, and farm subsidies to reduce the budget deficit?” The results: 23 per cent of the respondents reported favouring cuts in programmes while 66 per cent opposed; 11 per cent reported no opinion. Then the question was reworded to ask, “Do you favour cutting government entitlements to reduce the budget deficit?”

The results were dramatically different. Now 61 per cent of the respondents favoured cuts in programmes; 25 per cent opposed; 14 per cent reported no opinion. This example highlights the need for careful consideration in developing your survey instrument.

RESEARCH OF THE WEEK

Topic: A Step-by-Step Guide to Developing Effective Questionnaires and Survey Procedures for Programme Evaluation and Research.
Author: Keith G Diem, Ph.D., Programme Leader in Educational Design (2002).

INTRODUCTION

Surveys can be an effective means to collect data needed for research and evaluation. However, the method is often misused and abused. The challenge is to design a survey that accomplishes its purpose and avoids the following common errors:
a) Sampling error (How representative is the group being surveyed?)
b) Frame error (How accurate is the list from which respondents are drawn?)
c) Selection error (Does everyone have an equal chance of being selected to respond?)
d) Measurement error (Is the questionnaire valid and reliable?)
e) Non-response Error (How is the generalisability of findings jeopardised because of subjects who did not reply?)

DISCUSSION

1. Determine the purpose: What do I need to know? Why do I need to know it? What will happen as a result of this questionnaire? Can I get the information from existing sources instead of conducting a survey? It’s a good idea to start with research questions or objectives.
2. Decide what you are measuring: Attitude, knowledge, skills, goals, intentions, aspirations, behaviours and practices, Perceptions of knowledge, skills, or behaviour. Of course, it’s possible that you might measure more than one. But the questions will be clearly different based on the information you are trying to gather.
3. Who should be asked?
*    What is the appropriate population (group of people/subjects) to be studied or questioned?
*    Should a census or sampling be used? There are a variety of ways samples can be taken: Simple random (pull names from a hat); systematic random (i.e. every fifth name); stratified random (separate samples for each subgroup); and cluster sampling (treating intact groups that cannot be broken up, such as classrooms, as subjects to be sampled).
*    For whom do the results apply? This is an indication of the external validity of a study.

4. Consider the audience: age, education level, familiarity with tests and questionnaires, Cultural bias/language barrier and other demographics. Field test your instrument. Validate your instrument.

5. Choose an appropriate data collection method: Mailed, telephone, personal (face-to-face) interview or Web-based.

6. Choose a collection procedure: anonymous versus confidential.

7. Choose measurement scale and scoring: Use scales that provide the information needed and are appropriate for respondents. Some choices are:
*    Fixed-response: Yes-no, true false, multiple choice, rating scale/continuum (such as a likert-type scale), agree disagree or Rank ordering.
*    Open-ended (narrative response)

8. Title the questionnaire.

9. Start with non-threatening questions: This will make sure the respondent is not intimidated. Make the first questions relevant to the title/purpose and easy to answer.

10. Include simple instructions.
11. Use plain language: Be direct. Use simplest language necessary. Avoid jargon and acronyms. Include definitions if needed.

12. Be brief.

13. Put most important questions up front: Respondents may get fatigued or hurried by later questions. Include questions about demographic information at the end so questionnaire is focused on topic at hand.

14. Make sure questions match the measurement scale selected, and answer categories are precise.

15. Ask only one question at a time: Avoid ‘double-barrelled’ questions that confuse the respondent into not knowing how to answer.

16. Avoid ‘loaded’ questions: Minimise ‘bias’ in questions by using experts and field testing. Example of ‘loaded’ or biased questions: Do you agree that our programme deserves more funding than the greedy politicians currently provide?

17. Arrange in a logical order: Group similar questions together (such as by topic or scoring method).

18. Minimise open-ended questions: ‘Essay’ questions make the survey look more like a test at school and gives impression the form is a lot of work. They’re also difficult to score and summarise.

19. Provide space to tell more.

20. Make sure it looks professional! Proof read! Consider using a ‘booklet’ format so it stands out from just “paper.” Use quality reproduction.

21. Use a cover letter

22. Indicate what to do when questionnaire is complete.

23. Thank respondents!

24. Check reliability: Pilot test your instrument. Examples of reliability measures are: Test-retest (determines repeatability). Internal consistency measures (such as cronbach’s alpha reliability coefficient or split-half analysis) determine how well items contained in the questionnaire measure the ‘same thing’.