- Check to see if the Findings already exist.
- Ensure that your Sample is Random
- Ask only the most important questions
- Encourage answers you can use
- Predict the Results
- Start Backwards
- Record Everything
This type of research can be very useful in demonstrating characteristics of a given population. Getting an accurate, first-hand representative sample of needs, opinions or qualities can be the best way to make a case for any initiative. Primary Statistical Research, especially sound statistical research is very difficult and expensive and should not be entered into lightly. Establishing a representative sample, deciding on an appropriate methodology (survey, interviews, mass mailings) and collecting data are very complex tasks. Tabulating findings, processing and analyzing data collected, controlling or accounting for bias, inconsistency and inaccuracy are also very challenging. This all represents a profession in itself and one that commands a very high price.
Findings based on statistical research are often discredited based on faults in their methods, in which case the work will have been an enormous waste of time.
If you are planning on conducting a survey or statistical research, we have a few suggestions:
1. Check to see if the findings already exist.
Often there are larger bodies that already have the particular statistics you are looking for or have similar statistics that could be applied to your situation or area of concern. Larger surveys like those conducted by Statistics Canada, are carefully conducted in such a way that conclusions about the population you are concerned with might be extrapolated. Sometimes, custom tabulation may be possible for a reasonable fee. A review of likely locations of such findings should be the first step for any project requiring this type of data. Government ministries, web sites, and offices are good places to start looking.
2. Ensure that your Sample is Random
“Random” is often confused with “hap-hazard.” To achieve a defensible representative sampling you may pay close attention to selecting the respondents in a truly random fashion. What this means is two things:
- The respondents were selected without bias, and
- That the entire population had and equal opportunity or chance to respond to your survey.
What this means in practical terms is that you do nothing in your methods that favours certain segments of the population in question, nor do you put any barriers in place against the participation of others. For example: a phone survey of parents in a community conducted between 9 and 5 would skew the respondent pool away from working parents. Another example would be to do client interviews within the offices of an agency that offered both “walk-in” and “in-home” programming. This would skew the results in favour of the clients using the walk-in services and exclude those who don’t usually come in to the office. Other concerns can be language barriers, literacy, access to a phone or transportation, privacy concerns existing relationships between the interviewer and the respondent, timing cycles, etc.
Suffice to say that establishing a “random” sampling is anything but a haphazard process and any unaccounted for introduction a bias will weaken the results. Conversely, a truly random sampling is a very costly and time-consuming investment, one often beyond the capability of one student, working alone.
3. Ask only the most important questions
Each question on a survey represents a large investment in time in terms of asking the question, recording the responses, tabulating the results, drawing conclusions and establishing findings. Always return to your Key Research Questions to assess whether each particular question contributes directly to the big picture results you are after. Furthermore, surveys that are too long are seldom completely thoughtfully or correctly or completed at all. An incomplete or improperly completed survey weakens your findings by reducing your sample and bringing your methods of delivery into question. A survey that takes 10 minutes or longer to answer is a long survey indeed.
4. Encourage Answers you can use
It is tempting to allow for the most accurate answers on an individual basis. Questions like “describe your experience with our agency” asked of 300 respondents will result in 300 unique responses. Although this might be interesting reading, it is difficult to produce a finding. There can be no clear statistically representative conclusions drawn and any qualitative summaries a researcher can make will be fraught with personal bias. Questions like “which of the following best describes your experience with our agency” and providing several options can result in strong findings like “75% of respondents found our agency lacking in one-on-one counselling” or “only 10% of respondents said they would not use our services again.” If your sample is carefully constructed, these conclusions can be applied to the entire population being studied.
5. Predict the Results
This might sound wrong when you are striving for unbiased findings, but all research is done with a starting set of assumptions about the population and about the answers you will likely receive on your survey. Making predictions serves two purposes:
- It brings your opinions out in the open and makes the efforts you have made to account for this bias clear and evident.
- It gives you an opportunity to re-think some of the questions you are asking in terms of how they will be perceived by the respondents and how useful the findings will likely be.
6. Start Backwards
Answering the questions “who is going to use this information, how will they use it, and how will they judge its validity” can go a long way in making decisions about the design of your research. Determine who will ultimately be reviewing the results and consult with them as to what they would like to see. If it is a funding agent; determine what criteria they judge by, if it is a board of directors; determine what decisions they will be making with your findings. Find out if you can have anonymous respondents, find out how much personal data will be required (profile of respondents, age, gender, etc.). Find out if statistical information is what you really need. If you are dealing with a small population, percentages may be meaningless anyway. If after all, you are looking at some new ideas of just a quick bit of feedback, other mechanisms like “focus groups” or “informational interviews” might be a more useful and efficient use of your time.
7. Record Everything
Make notes explaining why you make each decision, who you want to include in your sample, who you will exclude and why. Why you choose the methods you use, who you consulted with, where did the idea of this research come from and why, who is conducting the research, when and where. These are questions that will be asked by people who know about research and if the answers are not readily available, your findings will be suspect. Making detailed notes to justify each decision you make is always a good idea anyway, in that it will also help you make more thoughtful decisions.