People are surveyed, data are collected, results are analyzed. In the end, we are presented with a number, an average, or a conclusion about “the population.”
Yet one question is asked surprisingly rarely:
How does this “population” enter the data in the first place?
Because long before any analysis begins, a selection has already taken place.
And that selection is often more invisible than we realize.
1. The Invisible Decision Behind Every Study
In Empirical Social Research, there is a simple but fundamental challenge:
Researchers can only make claims about what they have actually observed.
That sounds trivial, but it lies at the heart of every empirical conclusion.
Two well-known concepts describe this problem:
- Sampling Bias
- Selection Bias
In simple terms, both refer to the fact that data are not gathered from reality in a neutral way. Instead, they emerge through specific conditions that influence who becomes part of a study and who does not.
These conditions often remain hidden in plain sight.
2. Location Shapes the Data Before the First Question Is Asked
To survey people, researchers first have to reach them.
That means identifying places or situations where participants are accessible.
Public squares, shopping districts, transportation hubs, university campuses, and other highly frequented locations are common choices.
This seems practical and entirely reasonable. If many people pass through a location, it becomes easier to recruit participants.
But this is precisely where an important issue begins.
Not everyone is there.
And perhaps more importantly, not everyone can or wants to be there.
As a result, the location itself becomes a filter.
Not a technical filter, but a social one.
3. What We Do Not See Is Still Missing From the Data
Once we adopt this perspective, a crucial insight emerges:
Every sample contains two groups:
- those who are likely to appear
- those who are systematically less likely to appear
Among the frequently overlooked groups are:
- individuals with limited mobility
- people who avoid public spaces
- highly withdrawn or socially isolated lifestyles
- institutionalized populations, such as those in care facilities or religious communities
The key issue is not that these groups are necessarily rare.
The issue is that certain methods of data collection are less capable of reaching them.
And when a group is less likely to be reached, it is less likely to be represented.
4. Visibility Is Easily Mistaken for Reality
When researchers primarily capture what is easy to access, a particular picture of reality begins to emerge.
Not necessarily a false picture.
But a distorted one.
Visibility has its own logic:
What is frequently seen appears typical.
What is rarely seen appears marginal.
This is not primarily a moral problem. It is an epistemological one.
The concern is not that researchers deliberately measure the wrong thing.
The concern is that what is measured can easily come to stand for the whole.
5. An Often Overlooked Dimension: Everyday Life Is Not Neutral
A particularly interesting question arises when we examine who is most likely to be present in common survey environments.
Many of these locations are not merely public spaces. They are spaces structured around everyday activity:
shopping, commuting, errands, consumption, and social interaction.
This introduces an often invisible variable into the sample:
proximity to everyday consumption contexts.
This does not mean that everyone present is highly consumption-oriented.
Nor does it imply that consumption is the defining characteristic of the individuals being surveyed.
 |
| Picture: Einar Storsul on Unsplash |
Rather, it means that some lifestyles are more likely to intersect with these environments than others.
And this creates a subtle shift.
Certain ways of living become more visible—and therefore more measurable—than others.
6. When Frequency Begins to Look Like Normality
One of the most common assumptions in the interpretation of empirical findings is the following:
What appears frequently in the data is representative of the broader population.
But this assumption is not automatically justified.
Frequency is shaped not only by how people are distributed throughout society, but also by how accessible they are to researchers.
Or more simply:
What is easy to measure can quickly begin to look like what is generally true.
This is not necessarily a flaw of any individual study.
It is a structural feature of many forms of data collection.
7. A Known Problem—But One That Is Difficult to Eliminate
These concerns are well documented within research methodology.
One prominent example is the discussion surrounding WEIRD populations.
A substantial portion of psychological and behavioral research has historically relied on participants from Western, Educated, Industrialized, Rich, and Democratic societies.
The criticism is not that such studies are useless.
Rather, it is that conclusions drawn from highly specific groups are often generalized more broadly than the data justify.
The same principle applies on a smaller scale to many everyday forms of participant recruitment.
The important lesson is that research always operates through particular channels of access.
And those channels are never entirely neutral.
8. Between Methodological Ideals and Practical Constraints
In theory, researchers have many tools available to reduce sampling distortions:
- random sampling procedures
- stratified sampling methods
- statistical weighting techniques
- transparent reporting of limitations
In practice, however, these approaches require substantial resources.
Time, funding, infrastructure, and logistical support are often limited.
As a result, especially in exploratory projects or smaller studies, more pragmatic solutions are frequently adopted.
There is nothing inherently wrong with this.
The problem emerges only when the limitations of these approaches disappear from view.
9. A Quiet Assumption: We Often Define “Ordinary Life” Too Narrowly
Many sampling decisions rest on an implicit belief:
What appears ordinary is what appears visible and accessible.
But everyday life is not a single, uniform experience.
It is distributed across different social environments, lifestyles, and circumstances.
Some people spend their days in crowded public spaces.
Others do not.
Some are highly visible within the routines that researchers commonly encounter.
Others remain largely outside those routines.
This means that what appears normal may sometimes be little more than what is easiest to reach.
10. Closing Thought: Reality Begins Before the Analysis
Perhaps the most important lesson is this:
Empirical research does not begin with statistical analysis.
It begins much earlier—with decisions about what will become visible in the first place.
That is where some of the most subtle forms of bias emerge.
Not because anyone is acting in bad faith.
Not because researchers are careless.
But because certain assumptions feel so obvious that they rarely attract attention.
And yet those assumptions shape the data long before the first result is calculated.
Reality, in other words, is not only analyzed.
To some extent, it is already being selected before the analysis even begins.
Inspired by Kanwar Singh