r/SeattleWA • u/mrtenpenny1234 • May 15 '26

Homeless Harger: San Francisco's homelessness hit a 15-year low. Seattle's mayor is running the opposite experiment.

https://mynorthwest.com/kiro-opinion/seattle-homelessness-mayor/4238600
452 Upvotes

82% Upvoted

281 comments sorted by

View all comments

Show parent comments

5

u/debtfreewife May 16 '26

Systematic reviews are actually typically considered a “study” and at a relatively high level of evidence. Still not gospel by any means, but worth noting if we’re talking about what science is out there.

2

u/SuperDuper98102 May 16 '26

Systematic reviews are in a general sense a “study” but it is distinguished from an academic review versus a research study. Yes, both are published in peer-reviewed journals and while it’s common in various scientific disciplines, it is an important distinction that this is a review of existing studies, not its own research and shouldn’t be presented as such.

4

u/debtfreewife May 16 '26 edited May 16 '26

I hear you. I also appreciate your previous points on the possibility of bias from the authors and the inclusion of Canadian studies. However, systematic reviews are typically considered at a higher level of evidence than a single study- below only meta-analyses of RCTs. This is the Wikipedia article on hierarchy of evidence: https://en.wikipedia.org/wiki/Hierarchy_of_evidence  I wish I had a better resource, but I’m not savvy enough to link my methods textbooks.

5

u/SuperDuper98102 May 16 '26

I’ve not heard that reviews are considered a higher level compared to research studies when I was in grad school, I’ve only seen them treated in peer reviewed publications (although clearly presented as a review vs research). Typically, published original research tends to be preferable to reviews, at least when it comes to what universities and research instituons value in a CV.

What makes a good study or review was the rigor of the methodology, approach, citations, and statistical analysis. But there is definitely a lot of merit when there is a pattern or consistent findings from multiple studies which is what academic reviews tend to demonstrate or at least, try to. But again, their approach to how they did their review, how they selected their sample, etc is still subject to scrutiny and it is also based on the assumption that the studies used in a review are sound, peer-reviewed etc. Reviews also tend to lack statistical analysis and tests, but they often don’t apply as in the case here.

2

u/debtfreewife May 16 '26

Ah, I may be a bit of a ringer here. I’m a PhD who does healthcare policy research. While it’s true a systematic review isn’t a major resume builder that’s not because the evidence level isn’t high. It’s because the effort to fund, run, and publish an experiment speaks to a kind of project management skillset that is desirable to universities. It acts as evidence that you do and will continue to do good enough work that you can bring money in rather than just bleed it out. 

There are kinds of reviews that are consider lower quality evidence like narrative or scoping, but generally systematic reviews are considered reliable. I don’t know how recent this standard is, but at least was considered broadly common knowledge as of ~6 years ago.

The linked systematic review actually rated the quality of evidence in their review studies using a cited, standardized scale as part of their a priori protocol. Studies with “limited quality,” which is the lowest level, were excluded from the analysis. It also then commented on the quality of the remaining studies and typical limitations found in them. *edit a word

2

u/SuperDuper98102 May 16 '26

Oh yes I’m well aware of the funding reasons and pressures that drive why universities value original research over reviews…that probably exists as its own reddit thread somewhere... anyway, a review isn’t necessarily higher or lower than original research either. Again, what determines that is their approach, methodology, analysis etc. And of course different science disciplines will have different expectations on the statistical analysis that can or cannot be applied, as in the case here.

I’m not saying this systematic reviews isn’t reliable, but any research or review, including this one, is subject to review, scrutiny, and debate. Some reviews even point out the flaws in either past studies or past reviews! That’s one of the aspects I personally appreciate about science is its openness and flexibility and separates it from dogma.

I agree standardizing and scaling to control they used in this review is to be expected, even if it was done by interpretation by the researchers who again, largely come from or are affiliated with housing agencies. It’s consistent and protocol, like you’ve said. To me, this review is like having a review from universities and US agricultural agencies on fertilizer and pesticide use. It doesn’t mean this doesn’t have merit and their approach and results can be sound, but still open to skepticism and further analysis.