GWS 2025: On the reliability of data

Data from the 2025 Great Wargaming Survey (GWS) has arrived and the annual ritual of teasing inferences from these data begins! Following Excel file conversion and recoding variables for more efficient data analysis, the process of addressing some of the survey results kicks off. First off, a word about data reliability.

Questions about data collection methods and reliability often arise, so I frequently provide select meta analyses to address these concerns. The GWS relies on voluntary responses from anyone who completes the survey. Without random sampling, responses are based upon convenience sampling. In convenience sampling, respondents must know about the survey, have access to it, and be interested in wargaming, which naturally limits the dataset to this group. Each annual survey includes all valid submissions in a cross-sectional study, capturing data from many participants at a single point in time. Non-random sampling can introduce bias, meaning the sample might not fully represent the broader wargaming population. Any significant bias could potentially lead to misleading inferences.

While the possibility of sampling bias has been a recurring concern from readers over the years, is this concern justified? Does collecting a large sample size per year help mitigate this risk of bias introduction? How large of a sample is required? Can reasonable insights still be drawn despite potential issues? What about sampling the same target population repeatedly but drawing different samples? Annual survey analysis like the GWS rely on repeated cross-sectional studies, allowing analysis of changes over time even with varying samples. To build confidence, each survey should draw similar samples from the population.

A quick initial review reveals one immediate surprise: participation dropped significantly over the last three years. After 9,282 responses in 2023, the 2024 survey garnered only 5,995 responses. In 2025, total responses dropped yet again to 5,005 responses. Reasons for the continued fall off in responses are unclear but reports suggest that only 78% of respondents beginning the survey completed it. Many of these dropouts were due to failure to click "Submit" once finished. Could this participation reduction skew upcoming analyses for the 2025 reporting cycle? Speculation aside, data might offer clues on bias. As the first in the 2025 reporting series, this post examines data reliability and compares select demographics from 2023, 2024, and 2025.

With a decrease in responses in both 2024 and 2025, the percent of those repeat survey respondents falls from about two-thirds in 2023 to roughly 60% in 2024 and 2025. Still, overall percentages are similar across all three years.

The mix of respondents' home countries remains relatively stable from 2023 to 2025. Fewer respondents originated in USA/Canada and Australia/New Zealand in 2025 to the benefit of UK/Ireland respondents but no significant change in the overall distribution.

When examining age demographics, a shift in age cohort emerges. In 2025, the 61+ age groups saw a marked uptick in responses. Outside of a heavier weighting to the older cohorts, 2023 and 2025 distributions look similar. For age distributions, 2024 looks like the anomaly. As past surveys demonstrate, older cohorts tend more toward historical wargaming. Will the survey confirm this inference?

Highest education level attained remains consistent across all three years. No significant difference with respect to education.

With an age group shift to the 61+ age cohorts as noted above in Age Groups, we might expect a corresponding shift toward historical wargaming as primary interest. These data suggest that may be the case. This is exactly the result that the breakdown of primary interest shows. Fantasy/Sci-Fi primary interest grouping lost ground in 2025 to both Historical and Mixed classifications. Historicals gained about four percentage points from 2024 to 2025. Fantasy/Sci-Fi lost five-and-a half percentage points from 2024 to 2025. Why this shift and what effect (if any) will this hold for following analyses?

Finally, when examining responses to how long a respondent has been wargaming, the results are not surprising given the age and primary interest shifts seen above. The percentage of gamers having been in the hobby for 31 years or more increased by 7.5 percentage points from 2024 to 2025. Notice that 2025 results follow the distribution pattern seen in 2023 more so than the pattern seen in 2024. Again, was 2024 an anomaly?

Despite a non-scientific approach to survey sampling, the GWS demonstrates again that sampling tends to remain consistent year after year. I return to the topic of data reliability frequently to help mitigate concerns about bias (and usefulness) of these surveys. Even with a move toward older gamers, cross-sectional, repeated sampling, in this case, seems to produce a stable sample from which to investigate inferences. With fewer responses again this year, will established trends hold? Some may and some may not. Watch as results and analysis of the 2025 survey unfold.

There are a few new questions this year and I look forward to exploring these results. I would enjoy seeing your thoughts on why fewer gamers completed the survey and/or the demographic attributes of the GWS Class of 2025.