It’s important for democracy for voters to be politically engaged and knowledgeable, but opinion polling has consistently found that women have less political knowledge than men. In new research, Patrick Kraft finds that this gender gap in political sophistication is down to the survey methods used, which focus on factual knowledge, rather than any actual differences in sophistication. He writes that by using a new measure where survey respondents discuss their political preferences in open-ended survey responses which are more relevant to them, and using computational text analysis, any evidence for systematic differences between the political sophistication of women and men disappears.

Survey researchers and pundits alike frequently lament the fact that citizens know too little about politics. Furthermore, public opinion polling consistently suggests that women appear to be less knowledgeable about politics than men. But how can we measure political sophistication in the first place? And more importantly, how does our measurement approach impact our conclusions about citizen competence about politics? The conventional approach to measure political sophistication in surveys focuses on multiple-choice questions assessing people’s ability to recall information about political institutions and officeholders. (For instance, the American National Election Study routinely asks questions such as, “Do you happen to know which party currently has the most members in the U.S. Senate?”) These types of factual knowledge questions—which have been used in scores of surveys for academic research and polling—have important limitations. For instance, determining a comprehensive set of knowledge questions is far from trivial since it involves strong assumptions about what information is necessary for people to be politically competent. Furthermore, knowledge questions vary in difficulty across demographic segments and can therefore introduce systematic measurement error when comparing groups of respondents.

One important consequence of these measurement issues is the apparent gender gap in political sophistication. Traditionally, women tend to score lower on factual knowledge assessments than men. While this may partly stem from genuine differences in political interest due to gendered socialization, a substantial portion of this gender gap can be attributed to the measurement issues associated with conventional knowledge metrics. For instance, men’s higher willingness to guess instead of admitting that they don’t know an answer inflates their aggregate knowledge levels compared to women. Furthermore, research indicates that gender differences lessen when focusing on gender-relevant political knowledge or in contexts with balanced gender representation.

Introducing Discursive Sophistication

To overcome these limitations, I propose an alternative measure called discursive sophistication. Instead of focusing on factual knowledge questions, discursive sophistication evaluates how people discuss their political preferences in open-ended survey responses using computational text analysis. Specifically, I develop a framework to assess whether beliefs and attitudes in a given political domain are expressed in a more elaborate manner—a question that is not directly discernible from off-the-shelf factual knowledge items. This approach holds two key advantages over conventional metrics: first, it captures the extent to which political beliefs are grounded in elaborate reasoning, and second, it can pinpoint competence in specific domains through targeted open-ended queries. The resulting measurement therefore more closely matches our theoretical conceptualization of political sophistication.

I validate the new measure across diverse surveys encompassing multiple languages, revealing its effectiveness in predicting various indicators of civic competence and engagement. While closely related to traditional metrics, the two measures are distinct. For example, both discursive sophistication and factual knowledge independently predict political engagement and efficacy, implying their complementary nature. Importantly, this new measure disrupts the conventional gender gap narrative. While women might lag men in factual political knowledge, there’s no discrepancy in the complexity of their expressed political attitudes across various surveys (see Figure 1).

Figure 1 – The gender gap in political sophistication

Note: The figures display distributions of political sophistication using open-ended or conventional measures comparing women and men (including 95% confidence intervals around the means). Gender differences are statistically significant at * p< 0.05, ** p< 0.01, and *** p< 0.001. 

Photo by Towfiqu barbhuiya on Unsplash 

Explaining the (Lack of a) Gender Gap 

Intriguingly, there is evidence that this divergence between both measures can be attributed to women focusing on different issues in open-ended responses. Figure 2 shows gender differences in topics raised in the open-ended responses themselves. The results indicate that the gender gap in conventional knowledge metrics is at least partly driven by the fact that the issues women care about are not represented in standard item batteries. For discursive sophistication on the other hand, any evidence for systematic differences between women and men disappears since open-ended questions allow respondents to focus on specific considerations that are most relevant to them.

Figure 2 – Gender Differences in Topic Proportions in Open-Ended Responses

Note: Figure is based on the structural topic model used to compute discursive sophistication (Including 95% Confidence Intervals). Coefficients indicate the difference in predicted topic prevalence among women and men; positive values indicate higher prevalence among women. Labels are based on the five most frequent and exclusive (FREX) terms associated with each topic. 

Looking Ahead: Political Sophistication Reimagined

My findings suggest that while conventional knowledge measures and the novel open-ended approach share commonalities, they offer distinct insights into political sophistication. This distinction is vital, as both metrics independently influence political engagement and efficacy. Notably, the gender gap typically observed in factual knowledge disappears when scrutinized through the lens of discursive sophistication. The absence of gender differences in this measure could stem from the opportunities open-ended responses offer for women to focus on distinct considerations.

Past survey research has almost exclusively relied on conventional knowledge metrics to assess political sophistication. My study calls for a broader approach, highlighting the potential of open-ended responses to help us better understand the structure of people’s political belief systems. In other words, the focus of political research needs to shift from a mere emphasis on factual knowledge to a deeper understanding of how individuals think about their political preferences. Discursive sophistication provides such an alternative that can help us gain new insights into how people navigate the political landscape. This method is easily adaptable to diverse decision-making contexts through tailored open-ended questions. Moreover, researchers and policymakers alike can harness a user-friendly software package in the R programming environment to implement this framework.

• This article is based on the paper, ’Women Also Know Stuff: Challenging the Gender Gap in Political Sophistication’, in American Political Science Review. 
• Please read our comments policy before commenting. 
• Note: This article gives the views of the author, and not the position of USAPP – American Politics and Policy, nor the London School of Economics. 
• Shortened URL for this post: https://bit.ly/3PzjMSX