"Online Surveys: the Glitz and the Substance" Research Notes, Magnet Communications, 2001
< < Back to Press Room
Approach Online Surveys with Caution - A Historical Perspective
One of the more infamous examples of market research failing to accurately predict behavior is the Truman-Dewey presidential election of 1948. The Gallup, Roper, and Crossley polls all predicted an easy Dewey victory, confirming the views of virtually every media pundit. As you probably know (even if you weren't even a twinkle in 1948) Harry Truman won that election, triumphantly holding up the newspaper whose front page proclaimed in 150 point type: DEWEY ELECTED. This incident damaged the credibility of market research for decades - some people still hold it out as a reason not to believe in polls.
One reason cited for the inaccuracy of the polls in 1948 is that they conducted their interviews by telephone, but a lot of people didn't have telephones then - maybe 40% or 50% at that time. As you might expect, telephones were most prevalent among affluent urbanites. The people being interviewed were not only more affluent, but more likely to be Republicans, and consequently, more likely to be voting for Dewey. The net result is that the survey sample was not representative of American voters; rather, it was representative of American voters who had telephones!
Why am I telling you this bit of ancient history? Because the analogy is nearly perfect with Internet surveys, which have been around for several years, beginning at a time when there were not very many Internet users - even at work. In 1995 Internet users comprised about 15% of the population, and online surveys were being hawked as the best thing sinceÉ.telephone surveys! But who were they reaching? Upscale consumers: economically and educationally.
Sampling Concerns and Solutions
Obviously, obtaining a representative sample in an online survey - as in any survey - is an important issue, and a controversial one as well. This is particularly true if the subject is broad and the desired audience heterogeneous. Imagine doing an online survey about soft drinks or vacation destinations when you are reaching only the top 15% of Americans. Do you want to craft messages or determine product positioning from those data? Fortunately, the past few years have seen an unprecedented growth in Internet access, and more to the point, this growth has brought Americans of all socio-economic levels into the category. At least 50% of American households are online, and one estimate puts Internet access at 70%, including those who access it at work. Many believe we have reached critical mass in terms of producing a representative sampling of Americans in an online survey. But even if you want to interview online users, specifically, the issue of representativeness must still be addressed. In fact, MSNBC.com and a research team from Yale, Stanford and Columbia Universities announced that they would conduct an in-depth study regarding the 2000 General Election, where they would be comparing telephone interviewing and online interviewing to better understand the difference between these two survey techniques.
As recently as five years ago, the methods for online sampling were primitive, at best, yielding highly questionable samples. Procedures such as "snowballing," where two people would each send a survey to two other people, and so forth, may have produced significant numbers of completed questionnaires but any inference of representation (of any population) were highly speculative.
Some people suggested elaborate weighting designs, projecting the obtained data to known proportions in the population. Statistically, this might offer some comfort, but, depending on the actual respondents, it can give a great deal of weight to a handful of people in one group and very little weight to a lot of people in another group.
By now, there are far better solutions to these thorny problems, some coming from new companies with new techniques, and others from well-known research companies that combined their understanding of survey sampling with the new electronic technology. For the most part, they rely on panels of Internet-enabled consumers who agree to participate in surveys - much the same as certain companies have panels of households that participate in mail and telephone surveys. Demographic data are collected from these participants (for themselves and their households), which are then used to construct panels which demographically reflect the U.S. Not only can these companies then conduct surveys - quickly and relatively cheaply - among a balanced sample of Americans, but they can extract from the panels sub-groups that meet your specific need: teenagers, men between the ages of 18 and 59 who watch 4 hours or more of sporting events each week, or people who suffer from arthritis (for example).
When to Consider an Online Survey
Before you decide to conduct a survey online there are some key questions you should ask.
1. Does the topic of the survey have to do with a product or service offered online?
- Buying online
- Receiving information online - financial, health, or anything else
- Playing games online
- Conducting business online
2. Is the population you want to reach consistent with the typical online population (skewing young, skewing middle class to affluent)?
3. Do you want to reach a niche group, which is normally expensive to do you telephone and slow to do via mail panels?
4. Do you want to conduct an employee survey, where the questionnaire can be delivered through the company's system?
5. Are most of the questions multiple choice?
6. Are there complicated grids that must be filled out?
From a content perspective, online surveys should have mostly (if not exclusively) closed ended questions. Moreover, complicated grids (that is, asking several questions about a list of items) may very well yield incomplete or incorrect questionnaires.
Carrying Out an Online Survey
Working with a company that conducts online surveys, as opposed to trying to "do it yourself" has several advantages.
- They have the software to quickly develop a questionnaire file that will be disseminated, usually around 3 to 4 days, depending on the length and complexity of the questionnaire.
- The applications have become more and more sophisticated, allowing for graphic images to be tested along with attitudes and behavior.
- Many companies have panels of individuals, as mentioned above, which make the sampling procedure easier, quicker, and much more reliable.
- Their methods of distribution and data collection are also faster, cheaper, and more reliable.
- They can produce data very quickly, as can most telephone interviewing companies that use computerized systems.
Online surveys can increase the rate at which we can find and interview certain kinds of population groups that would be more difficult (and therefore more expensive) to interview by telephone. This technique is yet another tool in our research tool box.
< < Back to Press Room
|
