GWS 2022: On the reliability of data

By Jon Freitag

The Great Wargaming Survey (GWS) accepts voluntary responses from any participant willing to complete the survey. No random sampling mechanism is in place. Sampling is thus by convenience. Since a respondent must have knowledge of the survey, access to the questionnaire to participate, and an interest in wargaming, the final dataset is naturally constrained to this population. For each annual survey, all respondents submitting a survey (excluding those having data quality issues) are included in a cross-sectional study. Cross-sectional studies collect data from many subjects at a single point in time.

The relaxation of random sampling methods may introduce bias. Having a sample not been drawn randomly, a sample may not fully represent the population as a whole. Inferences may therefore be misleading. Understandably, the possibility of sampling bias has been a concern raised by more than one reader over the years of publishing aggregated survey results.

Is the concern for sampling bias justified? Does a sample size hovering around 10,000 annually mitigate bias? Can survey analysis provide reasonable, and useful, results in the face of potential sampling bias?

What if the same target population is sampled at regular intervals? Does repeated sampling improve confidence in the underlying method? At each interval (for the GWS the interval is a year), the survey collects a different sample of the target population. This is a repeated cross-sectional study. Repeated cross-sectional studies can be used for analyzing population changes over time even though the samples, by themselves, are not identical.

To gain confidence in the methods used and results presented, each survey ought to be drawing similar samples from the target population. To assess this hypothesis, a few of the demographic attributes are selected for the years 2016-2022.

Prior survey response by year

Every year, respondents are asked if they have taken part in the GWS before. In a repeated survey, the earliest years ought to see fewer repeat respondents than in later years as the probability of having previously completed the survey increases. What is interesting is that the percentage of repeat respondents stabilizes at about 50% since 2020. About half of each sample is generated by respondents who say they have not participated in the survey before. Surprising result?

Years spent wargaming by year

When capturing the number of years spent wargaming, besides 2016 and 2017, the percentage of respondents in each duration bin is relatively stable. Wargamers gaming 31 years or more remains locked in the upper 30% range. 

Respondent location by year

When examining respondent location over time, again, the time series remains stable.

Age group by year

Examining the time series by age shows an interesting, but not unexpected, result. Given that prior analyses suggest that the population of wargamers is getting older in the aggregate, samples drawn from a similar population ought to produce this tendency. The chart below exhibits the tendency of older cohorts to increase over time.

Primary interest by year

Finally, consider a respondent's primary interest in wargaming. Over time, we see a shift from Fantasy/Sci-Fi to both Mixed, and Historicals. Given the tendency for non-historical wargamers to gravitate away from preferring purely non-historical wargaming as they age, the underlying samples seem drawn from similar populations.

While GWS samples may not be drawn randomly, the hypothesis that these data are drawn from similar populations across time should not be rejected outright. Can results from several years of the GWS be relied upon for useful inferences on the population of wargamers? The answer is up to the individual. I know my position.


Hello Don, thank you for taking the time to make a comment.

Many wargamers hold a bias or preference toward their favored gaming genre whether it is historical, fantasy/sci-fi, or somewhere in between. The focus of this post is to address perceived survey sampling bias and not an individual’s bias. Hopefully the arguments presented in the post allow for a better glimpse into the validity and usefulness of the survey as a tool for understanding the hobby.

Jonathan Freitag

Aren’t we all just a bit biased? As a historical gamer that dabbles in fantasy or scifi, (I’ll play ANYTHING). I am always looking to bring more to the historical side. But the lines get blurry between “fantasy” and “history”, when you get those that insist ALL gaming is a fantasy, i.e. not “real”, merely an idea of what happened.

Don MacIntyre

Leave a comment

Related Posts