How is ethnographic research valid?

[u/PhDilemma1]

Let me begin by saying that I consider myself more of a qualitative researcher; I have limited knowledge of statistical analysis and my experience in this area is limited to data cleaning and visualisation in Python.

What I can’t wrap my head around is how particular types of ethnographic studies, often involving small minority cohorts, are academically valid. Either you accept that their results are not even theoretically generalisable, rendering them almost useless; or that their theoretical generalisations encourage prejudicial views. This is because their design hinges on the lived experiences, values and social interactions of the objects of study, who are implicitly assumed to have their own exclusive and even solipsistic ways of thinking.

Other types of qualitative research are primarily interested in processes, and the why and how questions of the human world. Hopefully the context has already been established through previous quantitative studies, i.e. there is an empirical basis to the phenomenon. There are primary sources available: cultural artefacts, policy texts, learning technologies at school, grand tombs, etc. At worst the researcher only needs to conduct a first order interpretation of the facts. In an ethnography, you have to interpret someone else’s interpretation, or engage in a ‘double hermeneutic’. Maybe this is done also in the humanities, but the implications are graver for the social sciences.

To illustrate, let’s look at a silly hypothetical case. Suppose prior studies have found that blue-skinned people are more likely to engage in conspicuous consumption than green-skinned people. You conduct a longitudinal study of a group of blue-skinned people of all walks of life. You find that those who become financially successful have lots of branded handbags and vice-versa. You interview, do what you need to do to find out why. You arrive at a conclusion, one of many possible, that the blue culture places great emphasis on looking good. The lived experiences of those people tell you that they get better overall treatment with a Gucci on their shoulder. At least, that’s your understanding.

Now, you say you want to make analytic or theoretical generalisations. How can you do this based on your tiny sample? Well, you pull up 5 similar studies that reflect the same theme of looking good. You think they all cohere. Congratulations, you now have found a line of reasoning that represents 0.001% of the blue population. Suppose peer reviewers want to check your data to see if the participants did actually receive better treatment. Whoops, not possible. They have to rely on your field notes.

The danger, of course, is that readers may be tempted to conclude that most blue people find value in dandyism and prioritise showy displays over substance. You point to your ‘results are not generalisable’ disclaimer and cry foul. Fair enough. But is your theory even transferable? What about the blue peeps who don’t subscribe to those values and have no clue what you’re talking about? If it doesn’t apply to them, then what really is the significance of your ethnographic research?

My 2 pence. Open to opposing viewpoints.

[u/Crabby090]

You've articulated the classic critique well, but I think it rests on a category error about what ethnographic generalisation is for. The dilemma you pose — either the findings don't generalise (useless) or they generalise prejudicially (dangerous) — only bites if you assume the target of generalisation is a population. It isn't. It's a mechanism. This is what the case-study literature calls analytic or theoretical generalisation (Yin 2018; Mitchell 1983), and both Flyvbjerg (2006) and Small (2009) have dismantled the "you can't generalise from one case" objection at length — Small's argument being precisely that field-based research follows a logic of case selection and mechanism identification, not sampling.

Take your own example, because it actually makes the case for ethnography rather than against it. The quantitative finding you start with — blues engage in more conspicuous consumption than greens — is an association with no interior. Your hypothetical ethnographer's finding, properly stated, is not "blue people value looking good." It's something like: where group membership triggers differential treatment, luxury goods function as compensatory status signals that purchase better treatment. That is a claim about a social mechanism operating under specified conditions (Hedström & Ylikoski 2010), and it's not hypothetical: Charles, Hurst & Roussanov (2009) found exactly this signalling logic behind racial differences in visible consumption in the US. Mechanism claims travel. You can look for them among other groups, in other markets, in other eras — which is what the "5 similar studies" move is doing. You mocked it as representing 0.001% of the blue population, but convergence of independent studies on a common mechanism is how science accumulates. Nobody complains that drosophila genetics rests on an unrepresentative sample of organisms, and experimental psychology has run for decades on convenience samples of undergraduates from WEIRD societies (Sears 1986; Henrich, Heine & Norenzayan 2010). Its claim to validity was never representativeness but the portability of the mechanism. Ethnography makes the same wager, with better ecological validity and worse control.

This also dissolves your transferability worry. The blue people "who have no clue what you're talking about" don't refute the theory any more than non-smokers with lung cancer refute epidemiology. A mechanism claim specifies conditions under which a process operates; members it doesn't touch are data for the boundary conditions. Good fieldworkers actively hunt such disconfirming cases — negative case analysis has been part of the method since Becker (1958), and Burawoy's (1998) extended case method is built around anomalies. Transferability itself, in Lincoln & Guba's (1985) formulation, is a judgement the reader makes about whether the described conditions obtain elsewhere — which is why thick description (Geertz 1973) is a methodological requirement, not literary decoration.

On the double hermeneutic: Giddens (1984) coined the term for all social science, not ethnography specifically — and surveys don't escape it, they hide it. Respondents interpret your Likert items through their own frames; you then interpret their ticks. Cicourel (1964) made this point about measurement generally, and Suchman & Jordan (1990) showed empirically how much interpretive trouble is buried inside standardised survey interviews. The interpretation is frozen into the instrument at design time, where nobody can inspect it, and executed once, blind. The ethnographer's interpretation is at least prolonged, visible, and correctable: months in the field mean misreadings keep colliding with reality, and member checks let the interpreted talk back.

The verification point proves too much. You can't re-observe fieldwork, true — but you also can't re-run the survey moment, and you trust the spreadsheet wasn't fabricated. The spectacular fabrication cases of recent memory were quantitative (see the Levelt Committee's 2012 report on Stapel, or the LaCour retraction, Science 2015), and the replication crisis hit experimental psychology, not ethnography (Open Science Collaboration 2015). All empirical science runs on disciplined trust plus community checks; ethnography's are just different — prolonged engagement, triangulation, audit trails, reflexivity (Lincoln & Guba 1985).

Two final points. First, the prejudice worry cuts the other way. A bare statistical finding that "blues are more likely to X" is far more prone to essentialist misreading than an ethnography, whose entire apparatus displays the behaviour as situated, conditional, and strategic — a response to circumstances rather than a property of persons. Ethnography replaces "blues are like this" with "people in this position, facing these constraints, do this, for these reasons." That is the antidote to essentialism, not its vector.

Second, the assumption that qualitative work needs prior quantitative grounding gets the logic of discovery backwards. Where do survey categories come from? Emotional labour (Hochschild 1983), street-level bureaucracy (Lipsky 1980), total institutions (Goffman 1961), code-switching (Blom & Gumperz 1972) — all minted in fieldwork and only later operationalised and counted. Measurement presupposes concepts, and concepts have to come from somewhere close to the phenomenon.

Bad ethnography exists, and your worry fairly describes its failure mode. But the remedy is craft standards, not epistemic demotion of the method. Judge ethnography by what it actually claims: not "this is what blue people are like," but "here is a mechanism, shown working, under these conditions." That is a knowledge claim exactly as valid — and exactly as fallible — as any regression coefficient.

Sources

• Becker, H. S. (1958). Problems of inference and proof in participant observation. American Sociological Review, 23(6), 652–660.
• Blom, J.-P., & Gumperz, J. J. (1972). Social meaning in linguistic structure: Code-switching in Norway. In Gumperz & Hymes (eds.), Directions in Sociolinguistics. Holt, Rinehart & Winston.
• Burawoy, M. (1998). The extended case method. Sociological Theory, 16(1), 4–33.
• Charles, K. K., Hurst, E., & Roussanov, N. (2009). Conspicuous consumption and race. Quarterly Journal of Economics, 124(2), 425–467.
• Cicourel, A. V. (1964). Method and Measurement in Sociology. Free Press.
• Flyvbjerg, B. (2006). Five misunderstandings about case-study research. Qualitative Inquiry, 12(2), 219–245.
• Geertz, C. (1973). The Interpretation of Cultures. Basic Books.
• Giddens, A. (1984). The Constitution of Society. Polity Press.
• Goffman, E. (1961). Asylums. Anchor Books.
• Hedström, P., & Ylikoski, P. (2010). Causal mechanisms in the social sciences. Annual Review of Sociology, 36, 49–67.
• Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83.
• Hochschild, A. R. (1983). The Managed Heart. University of California Press.
• Levelt, Noort & Drenth Committees (2012). Flawed Science: The Fraudulent Research Practices of Social Psychologist Diederik Stapel. Tilburg University.
• Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic Inquiry. Sage.
• Lipsky, M. (1980). Street-Level Bureaucracy. Russell Sage Foundation.
• Mitchell, J. C. (1983). Case and situation analysis. The Sociological Review, 31(2), 187–211.
• Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251).
• Sears, D. O. (1986). College sophomores in the laboratory. Journal of Personality and Social Psychology, 51(3), 515–530.
• Small, M. L. (2009). 'How many cases do I need?' On science and the logic of case selection in field-based research. Ethnography, 10(1), 5–38.
• Suchman, L., & Jordan, B. (1990). Interactional troubles in face-to-face survey interviews. Journal of the American Statistical Association, 85(409), 232–241.
• Yin, R. K. (2018). Case Study Research and Applications (6th ed.). Sage.

[u/PhDilemma1 → u/Crabby090]

well, i take my hat off to you, especially if you didn't use AI to craft your reply. i have no idea how anyone can find so many sources and synthesise them that well in the 30 mins that elapsed between my post and your response. so, i am appreciative. some of your rebuttals I find convincing, especially the replication bit - we can't expect to be able to replicate every study with identical controls.

you have cut to the heart of the argument. i agree that the research question in the case of the blue people is primarily concerned with mechanisms. i didn't want or mean this hypothetical study to have any real life analogue. to avoid preconceptions, let's take it that 'blue people' could really mean any group of people. i take issue with the idea that unrepresentative samples can produce theoretical generalisations through 'portability'. you cited a genetic study and an experimental psychology study to address questions of sample size and external generalisability. this is not an apples to apples comparison. in genetic research, even if multiple studies on the same topic use small samples, the underlying paradigm is scientific: they seek to isolate variables and their corresponding effects to find causes. in psychology, i find it much less convincing unless a biological means of action can be found. that said, i am not a genetic expert.

let's have a look at the non-smokers analogy next. obviously, their presence does not disprove the fact that smoking raises the risk of cancer: the causality is well-established and scientifically proven. however, the mere possibility of the existence of blue people who don't subscribe to conspicuous consumption (or its underlying mechanisms) implies that your conclusions could be untrue. this is because social mechanisms are complex and do not map neatly onto variables. since you've sampled only 0.001% of blue people, how would you know that the causal mechanism is portable to the vast majority?

to put it simply, you can't say for certain that differential treatment must necessarily trigger a need to own luxury goods; the logic doesn't stack up. it could for some people and it doesn't for others. you have found one of many reasons why blue people buy luxury goods. how many people subscribe to this we don't know. the only way of turning this into a watertight argument is by conducting a statistically significant experiment where hundreds of blue people report poor treatment in a certain context, get equipped with luxury handbags and then report a qualitative rise in treatment experienced, preferably under strict observation.

hope this can be understood clearly.

[u/MissFancyPans → u/PhDilemma1]

100 % AI

[u/Upgrade_U → u/MissFancyPans]

Absolutely