Counting the Uncountable - The Limits of Quantification in the Social Sciences: Why the human world resists the logic of numbers
1. Construct Validity: Are We Measuring the Right Thing?
One of the earliest and most pointed insights into this problem came from Lee J. Cronbach, who introduced the concept of construct validity in 1955. Ironically, this is the same Cronbach who developed “Cronbach’s Alpha,” a standard measure of reliability. He emphasized that the crucial question is not how reliably a test measures, but whether it measures what it claims to measure at all.
In both psychology and the social sciences, many research objects are abstract constructs: “intelligence,” “social trust,” “democratic satisfaction,” or “prejudice.” Such terms have no direct material counterpart. They cannot be measured like temperature or weight. Instead, indicators are constructed—such as questions in a survey or reaction times in an experiment. Yet whether these indicators truly capture the construct is highly uncertain.
Example: Intelligence tests measure specific skills like pattern recognition or verbal comprehension. But whether they truly capture “intelligence” as a whole is questionable. Similarly, in sociology, “social capital” may be operationalized via membership in clubs or associations. Reducing complex constructs to quantifiable aspects obscures their actual complexity.
Cronbach called this the “needle’s eye of construct validity”: all empirical research in the human sciences must pass through this narrow point, and many approaches fail right here.
2. The Generalizability Crisis: When Results Depend on Context
Even if we assume a construct is correctly measured, the next hurdle remains: generalizability. A study may produce valid results in a specific context—but do these results hold beyond that context? Here, the social sciences face a systematic challenge.
In the natural sciences, universal laws are often possible because the objects studied (e.g., chemical reactions) behave consistently regardless of context. In the social sciences, however, people are embedded in social, cultural, and historical contexts that cannot be fully controlled.
This gives rise to what is called the generalizability crisis: results cannot easily be applied to other groups, cultures, or time periods. A psychological experiment conducted at an American university tells us very little about people in other countries or social milieus. Likewise, a sociological survey from the 1980s may yield entirely different results today due to changing societal conditions.
A classic example is the many social psychology studies based on WEIRD populations (Western, Educated, Industrialized, Rich, Democratic). These studies are often presented as universal but, in reality, only describe a very specific, culturally influenced segment of humanity. The social sciences’ claim to generate universal insights is fundamentally undermined here.
3. RCTs in the Social Sciences: Illusory Precision through Experiments
Randomized Controlled Trials (RCTs) are considered the “gold standard” of empirical research. In medicine, they are invaluable because clearly defined interventions and control groups are possible. For several decades, RCTs have increasingly been applied in the social sciences, for example in development economics or education. Yet here the methodological limitations become especially evident.
The basic principle of RCTs sounds deceptively simple: participants are randomly assigned to an intervention or control group, and differences are measured. This is supposed to allow causal relationships to be clearly identified. But in the social sciences, this approach only works to a limited extent:
- Control is impossible: Humans cannot be isolated like chemical substances in a lab. Their behavior is influenced by countless contextual factors that an RCT cannot capture.
- External validity is weak: Even if an RCT shows that a particular intervention works in a rural school in Kenya, this does not mean it will work in a German city.
- Ethical boundaries: Many of the most pressing social questions—poverty, violence, trauma—cannot be randomly assigned ethically.
Economist Esther Duflo and her team gained worldwide attention with RCTs aimed at poverty reduction. Yet critics such as Angus Deaton and Nancy Cartwright have pointed out that these studies often deliver only “illusory precision”: they show small, context-specific effects that cannot be generalized. RCTs suggest precision where, in reality, context-dependence dominates.
Conclusion: Thinking Beyond Empiricism
The three examples—construct validity, the generalizability crisis, and RCTs—highlight a common underlying problem: the social cannot be fully quantified. Of course, we can count people, sum survey responses, or measure reaction times. But social phenomena themselves—trust, values, motives, culture—are not material objects that can be neatly represented by numbers.
For the past century, the social sciences have heavily modeled themselves on the natural sciences, hoping to achieve scientific rigor and societal recognition through quantification. Yet the generalizability and reproducibility crises show that this path has clear limits. On the replication crisis in another article.
This does not mean that qualitative methods are automatically superior. They also have significant limitations and remain embedded in the empirical paradigm. Perhaps the way forward lies in methods that do not reduce the social to measurability, but instead take its ambiguity and context-dependence seriously.
The central question remains: Should the social sciences continue to be forced to speak the language of numbers—or dare we imagine a science that takes the social seriously, beyond empiricism?