In the July 1991 issue of SAM we ran an Issues piece by Jim Spring, partner in the research service of Leisure Trends/Gallup (LT/G), that discussed the implications of their finding that roughly five million adult Americans had skied in the 1990-91 season. The finding contrasted starkly with the numbers from the sports participation studies from National Sporting Goods Association (NSGA), which counted 11.3 million (over the age of seven who had skied two or more times in calendar year 1990), and with American Sports Data (ASD), which counted 14.2 million who skied once or more.
Spring suggested in his commentary that the huge difference might be because his methodology, in which the respondent is asked to recall yesterday’s activities, is so immediate that it “measures actual behavior, or reality, while the other [which calls for the respondent to recall activity for the previous year] taps into attitude, which may reflect wishful thinking.”
The implications of Spring’s research, and of his conclusions, sent some shock waves through the world of marketing statistics. The LT/G approach, obviously posed a threat to some of the most widely used research services, and went way beyond the tiny little ski industry arena. Ironically, the threat touches on Gallup itself, which finds itself with its name on the Leisure Trends/Gallup work, while it sells volumes of research using the more conventional “last 12 months” recall that both NSGA and ASD use.
The difference in findings is not just a blip or a skew; it produces profound impacts. As Spring puts it, “A ski area’s fight for market share is based partly on the size of the total population of skiers, thus understanding the total size of the market is a significant factor. Secondly, the USIA was using the participation numbers as the foundation for growing the market. Growing a market ten percent off a base of 14 million yields 1.4 million new skiers, but if the base is half that size, the results will be only half those anticipated. The lower incidence of skiers in the general public dictates different strategies for marketing to them.”
Over the years, as recorded in these pages, the ski industry has struggled to reconcile the larger skier numbers found by the likes of Nielsen, NSGA and others with the industry’s own data on skier visits. (See “Our statistical disarray” in September 1982 SAM, for instance.) The number of skiers these studies found, when multiplied by the number of times they claimed to have skied, produced skier visit totals at odds with what the industry knew it was getting. “Phantom skiers” they came to be known as by some marketers.
The problem still remains: The 11.4 million NSGA skiers multiplied by the average (mean) number of days they said they skied produces over 95 million skier visits; the ASD 14.2 million mathematically generate 96.7 million skier visits. Those two are close, but out of the park by a factor of two compared to the 47 million skier visits counted by the industry. (That 2:1 ratio seems fairly constant, by the way.)
Obviously, some of the appeal of the new LT/G numbers is that they fit rather more comfortably with the industry numbers and with what many industry marketers sense by instinct.
On the other hand, could the methodology that provides vast amounts of data that presumably guides the marketing decisions of countless firms in countless fields, could that all be wrong? It seemed improbable.
To help readers understand the research issues, here is a brief description of the contrasting methodologies.
Jim Spring describes the LT/G methodology as follows: “We call 1,000 households a month, calling each night of the month, 12 months a year, or 12,000 interviews a year. The telephone field work, done by the Gallup Organization, is random and balanced to the total population of the United States. The line of questioning does not single out sports activities but inquires about all leisure activities so that we can uncover the real and all competitive issues which pull and tug on people’s time and pocket books.
“We ask up front in the interview a series of questions which uncover what Americans did the previous day. This does not require a long memory and guards against wishful thinking.
“At the end of last ski season, using this methodology, we arrived at the fact that approximately 5 million adults (over the age of 18) skied between December 1, 1990 and the end of April, 1991.”
The NSGA and ASD methodologies are similar, their differences not affecting the discussion at hand. Each uses an existing consumer panel — NSGA using one controlled by NPD, and ASD using an organization called NFO. Such consumer panels consist of tens of thousands of names selected (“pre-recruited”) to reflect statistical balance as a national cross-section. These people have indicated their willingness to respond to consumer research. NSGA and ASD mail questionnaires to a balanced sample of their respective panels and ask questions about the previous year’s sports and recreational activities of themselves and other members of their households.
We asked Jim Spring to state the case for Leisure Trends/Gallup, and we asked Harvey Lauer, president of American Sports Data and Tom Doyle, NSGA’s director of research and information, to rebut. We present all three, essentially unedited. We invite SAM readers to address the issue: Who has counted what — and why the difference?
Leisure Trends: Methodology Explained
by Jim Spring
The Leisure Trends’ figure which states that 5 million skiers over the age of 18 skied during the season 1990/91 has been challenged because it is a new figure and because it is lower than previously published figures.
Its critics challenge the methodology employed, stating it does not conform to conventional statistical measuring standards. More than likely they are correct. It was not our intent to copy what had been done previously, and it is not our intent to disavow the methods used by other research companies. It is our practice to take a different look at consumer behavior within the much broader perspective of time spent at all leisure, not just sports activities. Using this different methodology, we came up with a different set of figures. We are not contentious about them, but believe they should be made public.
That said and done, the following is an example of the logic and methods we used to arrive at our figures.
There are two premises which need to be accepted in understanding the Leisure Trends methods. The first is a definition of time. In the example which follows I maintain that a ski year is five months. Since all the work we do is monthly (we aggregate monthly, call each night), I cannot begin a ski season on December 15th or end it on April 15th. Thus I have to consider that the 151 days which begin December 1 and end on April 30 are the days when it is possible for the most Americans to ski. It is not a full 12 months.
The second premise is not so easy. In 1990, the NSGA and ASD studies found skier participation respectively at 11 million and 14 million. In the former case this results in an incidence of skiing of 5.1%, while the 14 million of ASD yields an incidence of 6.4 percent.
However, Leisure Trends finds an incidence of skiers of 2.5%. 4,738,230 skiers divided by 189M (over 18 population) equals 2.5% incidence. This is tricky because we are dealing only with people who are 18 years of age or over. Since the population is probably different, total incidence numbers are going to be different. Leisure Trends does not account for people aged under 18.
(Other research suggests there were over two million under-18 skiers during the 1990/91 season, which could yield a total estimate of somewhere between 7 and 8 million Americans who skied.)
The challenge rests on whether or not the methods we use are legitimate, so the following I offer as a response:
| Month | Incidence of Skiers found in daily calls (%) | Population 18+ | Days per month | No. of 18+ who skied |
|---|---|---|---|---|
| December | .00024 | 189M | 31 | 1,406,160 |
| January | .00016 | 189M | 31 | 937,440 |
| February | .00020 | 189M | 28 | 1,058,400 |
| March | .00017 | 189M | 31 | 996,030 |
| April | .00006 | 189M | 30 | 340,200 |
| Total | 4,738,230 |
I think my case rests on the following:
- The probability of interviewing the same person is statistically almost nil. This is because of randomness.
- It is suggested that the methodology uses averaging. What we do is not averaging and then adding; rather, we take the incidence of skiing over a period of time — in this instance a month — and weight it by the number of days in a month. Thus we have the number of people who told us they went skiing on an average day in a month. We then take the sum of the months to arrive at the number of people who skied. This is not perfect, but at least provides the information on a monthly break.
- We define the ski season as 5 months, not 12.
- We do not try to come up with the number of days an individual skis. Thus the person who skis only one-half day or the person who skis fifty days is included.
Methodology and Mythology in Ski Research
by Harvey Lauer
An article in SAM (July 1991) by Jim Spring of Leisure Trends/Gallup (LT/G) informed us that there are far fewer skiers than previously believed. We were told that a new methodology revealed the existence of only about five million adults who skied in the past year — a much lower figure than the 11 million adults reported by both the NSGA and my own firm, American Sports Data, Inc. (ASD).
We were further informed that the discrepancy between the 5 million (which must be correct because it’s based on daily surveys of what people did the day before) and the higher number (based on 12-month recall) is the result of “wishful thinking” on the part of respondents whose memories of what they did six months or a year ago are fuzzy at best.
Sounds good so far — until we take a closer look at the methodology and logic employed by LT/G in arriving at their 5 million skiers. In a nutshell, LT/G has committed a logical error of epic proportions, the rough equivalent of which might be the addition of 2 plus 2, and claiming an answer of 22! A simplified illustration will help explain what happened.
Let’s say that the U.S. population contains only 100 people, and that a year has only 3 days. Let’s further suppose that on the first day, only 1 individual skied. On the second day, 2 people went skiing, and on day three, 3 people participated in the activity. The daily incidences of ski participation are therefore 1%, 2%, and 3% respectively. The aggregate “annual” incidence would be 2%.
But this does not mean that 2% of the population skied at least once during the year. We don’t know how many people skied at least once, because our total of 6 hypothetical ski visits could have been accomplished by as few as 3 people, or as many as 6! Anywhere from 3% to 6% of this population skied at least once during the year. The 2% number (or any similarly derived projection) will always understate the participant population, and this is precisely why the 5 million projection is necessarily low and meaningless.
By conducting nightly surveys (that ask people about what they did on one particular day) it is impossible to determine what percentage of the population skied at least once during the entire year. The colossal blunder of LT/G is the equivalent of saying that only 2% of the population skied at least once, when in fact, the number must be between 3% and 6%.
Now let’s see exactly how the Leisure Trend data was misapplied. The firm conducts an ongoing telephone survey every night of the year, except for Christmas. Over a 364-day period, some 12,000 respondents have reported, among other things, which sports/leisure activities they participated in the day before. While this is a highly accurate method of determining the total number of skier visits over a 12-month period, once again it is impossible — through any type of daily or monthly aggregation — to determine from “yesterday” recall data, how many people skied at least once during the last 12 months.
What LT/G has done in effect is take a roughly 150-day period, December-April, and add up all of the people who claimed to have skied “yesterday.” It’s really a bit more complicated, but in essence, they’ve divided this total number of skier days into the total number of adults interviewed, and have come up with something like 2.5% (the equivalent of my hypothetical 2% figure), which they innocently believed was the percentage of people in the U.S. who skied at least once. By multiplying the 2.5% by some 190 million (U.S. adult population) the mythical 5-million number is born.
The fallacy here of course is that the 2.5% merely represents the percentage of all days available that the average person skis — not the percentage who skied at least once.
But we have not come to bury Caesar, but to praise him — where praise is due. In and of itself, the methodology (not the statistical logic) employed by LT/G is an excellent one. “Yesterday” recall methodology can portray with a high degree of accuracy the percentage of the population that participated in a given activity the day before, and in this sense it’s probably the closest we’ll ever come to a truly “objective” measurement of what people really do. If used properly, this method has the potential of enhancing our knowledge and understanding of participation behavior in sports and other activities. Unfortunately, in this isolated instance, its results were misinterpreted.
What about the 12-month recall method? To paraphrase Churchill’s comment on democracy, it’s the very worst of all methods, but it’s the best one I’ve seen yet. More precisely, it is the only way to measure the number of people who participated at least once in an activity over the course of a year. The only alternative is to survey the same respondents for shorter recall periods, but on a more frequent basis, but this is cost-prohibitive and logistically impractical. If we had the luxury of 1,000 interviews each night with the same people over an entire year, then and only then would Providential Truth finally be revealed.
It is no secret that 12-month recall methodology is fraught with pitfalls such as “telescoping” (tendency to report a ski trip 15 months ago within the 12-month period), innocent memory distortion, social desirability response sets (deliberately over-reporting various status-enhancing activities), to name a few. However, it’s been my experience that to whatever degree such response distortion occurs, it affects the number of times per year, and not so much whether they’ve performed the activity at all. The tendency to overstate frequency of participation may go a long way toward explaining the discrepancy in objectively recorded annual skier visits and estimates based on 12-month recall.
On the other hand, it seems to me that when compared with other activities, accurate measurements of ski participation may be easier to obtain. To begin with, when compared with other activities such as running, tennis, aerobics, etc. skiing is a much more distinctive experience. It’s not done 150 times a year, the season is neatly confined to certain months, and the logistics, effort, and time spent all combine to make it a far more memorable event than a 5-mile run or even a round of golf. For these reasons, I believe that telescoping is far less of a problem in accurately measuring this activity.
One of the things I remember Dan Yankelovich telling us many years ago was that “people rarely lie in surveys.” In the rare instance when response distortion does occur, I think that it takes the form of exaggerating the number of occasions, or consciously claiming participation in the current recall period when it really occurred previously. “Wishful thinking,” or the complete fabrication of an imaginary ski experience, is far more rare.
Having said all that, let me say this: 12-month recall is not without major limitations, and if the Lord came down from Heaven and assured me that there are in fact only 8 or 9 million adult skiers, I wouldn’t argue. But if He (She) said 5 million, I would argue no less stridently and risk eternal damnation.
The Conservative NSGA Numbers
by Tom Doyle
I would like to start with clarifying comments on NSGA Sports Participation data as it relates to Jim Spring’s article.
- For 1990, the NSGA Sports Participation study reported 11.4 million skiers seven years of age or older who said they skied more than once during the past year. Since 1985 (the first year of the NSGA study), ski participation has ranged from 9.4 million (1985) to a high of 12.4 million in 1988.
- In NSGA participation studies, the incidence of skiing has ranged from 4.4% to a high of 5.6% in 1988. In 1990, the incidence was 5.1%. Incidence is based on the population surveyed for the NSGA study, i.e., seven years of age or older, not the total U.S. population.
- In the NSGA Sports Participation study, the 18 to 24 age group represents 23.9% of all people who skied in 1990.
Having clarified the NSGA data, I would like to make some overall comments on the issue that Jim raises. The base number (even if it is a range) has a tremendous impact on the marketing decisions that everyone in the ski industry makes.
First, the NSGA sports participation numbers are considered conservative. By some good fortune, there were four sports participation studies done in 1985, including one by the Gallup organization. Given differences in definitions of the sport, the age groups surveyed and the methodologies, the NSGA numbers on sports participation were the most conservative.
Second, research prepared by other associations using other methodologies has generated numbers that are larger than the numbers generated in the NSGA sports participation study. For 1990, the National Bowling Council estimates the 71 million bowlers; NSGA, 40.1 million. The National Golf Foundation estimates 27.8 million golfers; NSGA, 23.0 million. The American Fishing Tackle & Manufacturer Association estimates 62.5 million fisher persons; NSGA, 46.9 million. Again, in the total picture, NSGA sports participation numbers appear quite conservative.
Third, Jim says that, because it asks what Americans did the previous day only, his methodology does not require a long memory. This is certainly true, but does not mean that other methodologies that require memories of longer time periods involve “wishful thinking.” Given a definite time frame, most people are able to recall memorable events that take place within that time frame. I consider skiing a memorable event; I believe Jim does also.
Although the event (skiing) itself is memorable, it is much easier for someone who skied to overestimate the number of days skied. A skier might count the traveling days as part of the number of days he or she skied, etc. An overreporting of the number of days skied would account for much of the discrepancy between the total number of skier days reported by Dr. Kottke and the NSGA data.
