They barely even begin to address the problem

Last month the journal Science published an article titled “Fraud, so much fraud”. It discusses their exposé of Eliezer Masliah, an Alzheimer's scientist and US government official who has been accused of systematically faking over 25 years worth of research. Such scandals are now common, like this one at a big cancer research center and this one at another Alzheimers lab.

The problem was discussed on Hacker News, where people exchanged ideas for how to improve the situation. In these debates someone will usually suggest an intuitive sounding fix: the government should fund scientists to repeat other people’s experiments, just like it funds any other kind of study. For example,

“We can solve this at the grant level. Stipulate that for every new paper a group publishes from a grant, that group must also publish a replication of an existing finding.”

This sounds like it should work; the science crisis is usually called the replication crisis, after all. It stands to reason that having an independent team do the same experiment and get the same results should prove the absence of problems with that study.

This idea is reasonable. Unfortunately, science isn’t.

In the past eight years or so I’ve read a lot of junk-tier scientific papers. It wasn’t intentional — there’s just so much out there that if you dig into claims found in mass media you’re going to encounter pseudoscience very quickly. I’ve written on this blog about the so-called “questionable research practices” in medicine, epidemiology (twice), “twitter bot studies” (also twice), archaeology and PCR testing. The weird selection of fields just reflects my personal interests — I don’t actually go looking for this stuff.

Of the bad claims I’ve seen, almost none would have been prevented by a replication study. Some would have been exacerbated. This is a very counter-intuitive claim. How can replication not help, let alone make things worse? Well, there’s a bunch of reasons. Don’t get me wrong: replication isn’t entirely useless, but it can only help rarely and doing more of it could actually pour fuel on the fire.

Replicating wrongness = more wrongness

The first and biggest problem with replication is that it only helps in one very specific scenario. You need to have a paper which:

1. Makes an important, interesting claim that would be Big If True.
2. Has a logical, detailed, and scientific methodology clearly derived from the claim.
3. Is free of other obvious flaws.

This is a low bar. Some fields are better at clearing it than others. Computer science usually produces papers that pole-vault over it, thank goodness. But in many other fields the median paper doesn’t even reach the point where replicating it would make sense. Instead these papers:

1. Make trivial claims e.g. the average man would like to be more muscular or people choose their clothes based on how warm it is (“It is evident that further studies are needed in this field” 🤮).
2. Have nonsensical methodologies like defining anyone who tweets five times after midnight as a “bot”, or which involve ad-hoc criteria presented “just so” instead of a description of how they were derived.
3. Are full of obvious errors that render replication pointless. This COVID paper has 19 authors yet the very first sentence is a false claim about public statistics.

Replicating the definition of replicating

The second problem is that people don’t agree on what the word replication means. It might seem obvious: surely it’s “we did the same things and got the same results”? But when careers are on the line, people find ways to disagree.

One case arises when scientists make a claim of the form “we did this thing and that other thing changed in a big way”. Someone tries to replicate this and they find the other thing changed but only by a bit, or they discover the effect fades with time (this is especially common in education studies for some reason).

Is this a successful replication? Scientists are often tempted to argue it is — after all, the same effect appeared and the natural world is inherently noisy. You’ll never get exactly the same numbers. But often the reason a paper was interesting was its claim of a big and important effect. If it shrinks to the point nobody would care, it’s not really informative to give the paper the “it replicates” stamp of approval. Doing that would just incentivise the exaggeration of real but small effects.

Another frequent case is where a study fails to replicate but the replicators changed some aspect of the method they felt was unimportant. The original authors point to the difference and say that’s why it didn’t work.

But the most common replicability problem I’ve seen is that the methodology isn’t actually derived from the hypothesis. The study design itself isn’t replicable. Examples litter the field of misinformation research. Academics in this field like proving claims of the form, “conservatives believe more conspiracy theories than liberals”. But what exactly is a conspiracy theory? Invariably these studies define conspiracy theory as anything in an ad hoc list of items. There’s no explanation of how the list was created and such lists don’t include things believed by liberals, so the whole thing becomes a circular firing squad: conservatives believe conspiracy theories because conspiracy theories are things believed by conservatives. It’s just a form of scientific fraud, so a replication that followed the written methodology would simply perpetuate it and a replication that didn’t would be rejected by the original researchers as illegitimate.

The eternal sunshine of the spotless mind

A less common but more damning situation occurs when scientists pretend something replicated even though it didn’t.

In 2020 Neil Ferguson shocked the world with a paper claiming COVID would spiral out of control unless there was an immediate and draconian lockdown. The prediction of viral spread wasn’t from a model formally written down in some published paper: it turned out to exist only in a single unpublished program. He helpfully explained that he never revealed the code before because it was “all in my head, completely undocumented. Nobody would be able to use it”.

Perhaps another explanation is that the program simply did not work. I wrote an article laying out the gruesome details. Bugs created unintentional non-determinisms: running the simulation on a different computer would calculate totally different results for identical scenarios. It was like if two people opened the same spreadsheet in Excel and saw totally different numbers.

This case is interesting because it’s a slam-dunk failure to replicate. There is quite simply no wriggle room here: two people solving the same equations with the same inputs should get the same results. If they don’t then something has gone horribly wrong.

It should have been a massive scandal. It wasn’t because Imperial College London published a press release saying a replication study was done and then sent it to some pet journalists, who promptly reported that the concerns were a false alarm. The supposed replication starts by saying “I was able to reproduce the results from Report 9”, but then the very same paragraph admits that every number the author got was different, some by up to 25%. He even admitted that one reason for the differences was that “the CovidSim codebase is now deterministic” i.e. he didn’t try and replicate the version he was supposed to be replicating because that version was inherently non-replicable to begin with.

In fields like public health the whole concept of replication seems to be barely understood. Circular logic runs rampant. It’s common to conflate replication and validation by claiming that a model is valid if its outputs roughly match the output of other models. Things taken for granted in machine learning, like withholding data to create a test set, are considered controversial in public health (see the section “Comparisons with external data” in that last link).

Pushing on wet string

The Ferguson case reveals the core problem with funding replication studies. The idea takes as axiomatic that bad scientists are rare and so the chance of a replicator being an honest scientist is high. Yet in reality there are no consequences for anything, ever and so the dishonest rise to the top. You just assert you followed the rules and everyone believes you!

A fascinating, bizarre, stupid and frustrating example of this problem is the recently published paper, “High replicability of newly discovered social-behavioural findings is achievable”. They claimed they fixed the replication crisis by using better methods! Big If True. The paper has 17 authors, some of whom are involved with the science reform movement or have even accused other scientists of fraud, so this sounded very promising. Surely it will be a seminal paper worth rea….

Oh. Why was it retracted? Ah, y’know, because they did every single unscientific thing that’s known to create non-replicable papers. They p-hacked the study by coming up with a different hypothesis after collecting their data, a practice that’s supposed to be stopped by pre-registering their plans, which they did … but then they “mis-stated” the contents of their pre-registration. Of course nobody at the journal checked.

So this is a study arguing for rigorous methods that breaks the rules they’re supposedly fighting for. And yet somehow, it goes downhill from there. If you have a weak constitution stop reading now, because the original study they planned was an attempt to test the existence of paranormal forces that screw with scientist’s experiments. Yes, Virginia. These wannabe Mulders hypothesized that maybe social science has poor replicability not because lots of them suck at science but because the act of studying a phenomenon somehow makes it disappear.

Perhaps you think this part is so absurd I must be making it up, so here’s an explanation by some of the authors:

Other presenters offered unconventional explanations [for observing weaker evidence in replications], such as the act of observation making effects decline over time, possibilities that Schooler believed worthy of empirical investigation . . . and others have dismissed as inconsistent with the present understanding of physics.

The researchers helpfully explain that they knew this was stupid but went along with it to get money from Schooler:

While the unconventional explanations were not considered plausible (or even possible) to most of the team, they agreed on an approach that included tests of those possibilities

The guy who discovered all of this wrote a long blog post describing how hard it was to bring this “utterly batshit supernatural framing” to light, and how everyone seemed to minimize the problems or pretend it’s all an accident (which he does not agree it was). Andrew Gelman also has a long writeup of this disaster.

Conclusion

Any proposal for science reform that involves paying for replication studies needs to first address at least these problems:

1. Many papers don’t make sense to replicate because they’re pointless, can be proven wrong just by reading them or because the derivation of the methodology is itself non-replicable.
2. Some fields use weak, nonsensical or circular definitions of “replicate”.
3. Some academics are willing to claim things replicate even if they don’t in order to preserve the influence of their field.
4. Scientists who make a big deal out of their high standards might actually not have any.

What’s at the root of all this? At some point we must look to the incentives as the cause. Science exists in a strange Soviet-esque system in which philanthropists and taxpayers allocate funds into a planned economy. Perhaps the odd inability to get rid of paranormal research is unsurprising in that context, as the Soviets were also big into parapsychology, funding it to the tune of half a billion dollars per year.

Without practical technological/commercial goals to ground scientific exploration it appears that the latter can easily enter a downwards spiral of ever lower standards, until eventually they just disappear entirely. There’s no One Weird Trick to fix bad incentives; only changing them can. Although it’s tempting to look around for ways to let governments continue spraying universities with cash, it’s worth remembering that we’ve only been mass funding academic science since the end of World War 2. Some of the most productive and innovative periods of history predate the expansion of universities — the Industrial Revolution was powered almost entirely by individual inventors and companies protected by patents. Is it time to revisit this model?

Replication studies can’t fix science was originally published in Mike’s blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

How will the hurricanes change the election? Vast swaths of Appalachia are still recovering from Hurricane Helene. Now, Hurricane Milton is expected to make landfall, either near Tampa Bay or closer to Sarasota, and Florida has experienced massive displacement, with nine counties ordering mandatory evacuations. (The Waffle House index roughly tracks with the official orders, providing additional impetus for residents to believe this storm will be severe.)

So how will this already-devastating, about-to-get-worse hurricane season affect the election that's scheduled in less than a month?

Both North Carolina and Georgia were ravaged by Helene. They're also both critical swing states with large rural populations. Local election officials are attempting to convey confidence, but struggling with the logistics of recovery.

"Early indications are that key election equipment such as ballots and voting machines were largely unaffected by the storm, avoiding a major last-minute logistical nightmare," reports Politico. "But the mounting to-do list is daunting." For residents who can't return to their homes, for example, how will they receive their absentee ballots by the relevant deadlines? Will the absentee ballots actually arrive at the temporary addresses? And how do citizens predict where exactly they'll be in just a few weeks?

"Roughly one-fifth of North Carolina voters live in areas battered by Helene, according to state voter registration statistics," reports Politico. "Fourteen county election offices in the state were closed for the near-term as of Tuesday, according to Karen Brinson Bell, the executive director of the North Carolina State Board of Elections." Polling places might not have power or running water restored in time for Election Day.

Back in 2022, following that hurricane season (in particular, the devastation caused by Ian), Florida Gov. Ron DeSantis (R) let three Florida counties—Charlotte, Lee, and Sarasota (all Republican, covering some 1 million voters)—move or consolidate their polling places, extend the number of days available for early voting, and have more permissive rules for where they could send absentee ballots.

It's possible that the same type of thing will happen again, but the Florida Legislature has been sensitive to relaxed rules on voting in the wake of allegations of voter fraud following Donald Trump's electoral loss in 2020. It would be bad if attempts to mitigate the effects of Milton or Helene led to real or perceived voter fraud, or the circulation of baseless conspiracy theories surrounding the election.

But it's not just the mechanics of voting: The election could also be altered by FEMA incompetence or even the perception of FEMA incompetence. As noted in yesterday's Roundup, former President Donald Trump has falsely claimed that the Federal Emergency Management Agency ran out of money due to spending it all on housing illegal immigrants. But Twitter CEO/tech bigwig Elon Musk has been circulating these same types of easily disproven theories as well as claiming that a SpaceX engineer told him that FEMA "actively blocked" donations in the aftermath of Helene and is "seizing goods…and locking them away to state they are their own." (Credit where credit is due: At the same time he is spreading such theories, Musk is also actively connecting hurricane victims with Starlink, which restores their internet.)

It's of course very possible that the agency is attempting to take credit for the efforts of private citizens and organizations, but there's not very solid evidence at present to indicate that's happening in any sort of widespread manner. Other accounts—like this one, by The Dispatch's Kevin D. Williamson—contradict that narrative and show private organizations filling gaps but not being sabotaged by the agency.

Scenes from New York: My new pet peeve is the New York Times articles that make excuses for rent stabilization and rent control policies which make it harder for young families to afford apartments in this crazy city.

QUICK HITS

• A high number of journalists have been killed in the last year while covering the war in Gaza. "The rate, about five a week, is the highest since the CPJ [Committee to Protect Journalists] began keeping global records over 30 years ago," reported The Washington Post back in February. But what percentage are actually, in essence, propagandists, or affiliated with a terrorist group (like Hamas)? Jewish Insider reports that "one-third of the Palestinian journalists listed by the Committee to Protect Journalists as being killed in the war in Gaza were employed by terrorist groups," including Al-Aqsa Voice Radio, Al-Quds Al-Youm, and Quds News Network, which are reportedly affiliated with Hamas. "Another two worked for Palestinian Islamic Jihad outlets Kan'an and Mithaq Media Foundation." A CPJ spokeswoman responded to this by saying: "We make our determination by researching whether a journalist's past coverage meets our news and public affairs criteria, rather than on who owns or controls their outlets."
• Footage from protests in New York City commemorating the one-year anniversary of Hamas' brutal October 7 slaughter of Israelis:

I don't care if you're bringing donor organs to kids on a transplant list, if your concurrent goal is to wipe others off the face of the earth, your mission is corrupt. If you think I cherry-picked this video, know I stood in one spot and saw many such one-sided spewings of hate pic.twitter.com/DmfdmXK1eA

— Nancy Rommelmann (@NancyRomm) October 8, 2024

• Tony Dokoupil literally did his job and asked Ta-Nehisi Coates challenging questions about his new book. This led to a workplace meltdown and, apparently, this:

I understand that, in the real world, with mortgages and children and a job on the line, this can be difficult for good people to do. I honestly do. But the correct answer in this situation is still, "I literally asked a guy questions on television, go fuck yourself." https://t.co/VufWtvSuzS

— Charles C. W. Cooke (@charlescwcooke) October 8, 2024

• "In the last two decades, [Kamala] Harris has lived in two places—San Francisco and Washington, D.C.—and, as her public record shows, she has tried to ban the private ownership of handguns in both locations," writes Charles C.W. Cooke at National Review. "A couple of weeks ago, Harris told Oprah Winfrey that if someone broke into her house, 'they're getting shot' with her Glock. But, if that is true, it's only true because she has been defeated in both of her attempts to make that impossible."
• Kind of a brilliant business strategy:

One reason why Costco is so popular among working and lower middle class, especially families and the elderly. The membership program turns the store into a sanctuary from miscreants: pic.twitter.com/nrEztyPWI3

— Rob Henderson (@robkhenderson) June 13, 2024

The post The Hurricane Election appeared first on Reason.com.

Democrats and Republicans might not seem to agree on much these days. But when it comes to trade, they chant "China trade bad, tariffs good" in stereo, even as the data show the exact opposite.

For 15 years, I was responsible for hundreds of millions of dollars in Chinese imports entering the United States. I invented two enduring products—the "Backpack Beach Chair" and the "Magna Cart" portable hand truck—that can be found in Costco and other stores to this day. 

Let's use the Magna Cart to illustrate how America made billions while China made crumbs. The Chinese factory charged me $10 for a cart that cost them $9 to manufacture. U.S. retailers bought it from me for $15, then sold it to consumers for $30.

To recap: The factory made $1, I made $5, and retailers made $15, minus freight and U.S. tariffs.

The freight costs went to shipping lines, U.S. railroads, truckers, warehouses, and America's highest-paid union workers—longshoremen at the Port of Los Angeles. As for those tariffs: Do the Chinese actually pay them, as former President Donald Trump claims? That would be illegal, as U.S. Customs charges tariffs only to the "importer of record," which must be a U.S. entity. The monies collected go directly to Uncle Sam and retailers add them to their cost of goods, as with any other expense.

So each Magna Cart created $21 in profits, of which 95 percent went into American pockets. Selling 5 million carts meant a $100 million gain to the U.S. economy. Yet the official trade statistics framed that as a $75 million addition to the trade deficit.

After being fed a daily diet of such misinformation, is it any wonder Americans aren't so warm on trade with China?

Wouldn't American profits be even higher if these things were made in the U.S.A? That's a big no, because many products simply wouldn't exist. My original plan had been to manufacture in the United States. Then I saw the factory quotes, and I realized my babies would have to retail for more than $100. Thanks to China, tens of millions of Americans can now carry their chairs and gear to the beach with ease, and move heavy loads without tweaking their backs for under $40. (It used to be $30. Sigh.)

So why can't we move all that manufacturing to other low-wage countries? Because only China has the massive workforce (800 million strong), the infrastructure, and the natural resources to supply 380 million Americans (plus 7.6 billion others globally) with every gizmo and gadget imaginable.

The nearly $500 billion that America imports annually from China enriches our economy by trillions. The math is so simple, you'd think even politicians could understand it.

The post I Did Business With China, and America Won appeared first on Reason.com.

Hurricane Helene's catastrophic damage and flooding to the Southeastern states may affect the country's medical supply chain.

Hospitals nationwide are bracing for a possible shortage of essential intravenous fluids after the cataclysmic storm inundated a vital manufacturing plant in North Carolina.

The plant is Baxter International's North Cove manufacturing facility in Marion, which is about 35 miles northeast of Asheville. Helene unleashed unprecedented amounts of rain throughout the western part of the state, killing dozens and ravaging numerous communities, homes, and other structures, including the plant.

Read full article

Comments

Click here to go see the bonus panel!

Hovertext:
I have never once had a popular platonism joke but I WILL NOT STOP

The U.S. Food and Drug Administration (FDA) approved a novel antipsychotic schizophrenia drug late Thursday, marking the first new treatment in decades for the mental health condition that affects 1.8 percent of the U.S. adult population. 

It is a major victory for market-driven medication development. Chris Boerner, board chair and chief executive officer at Bristol Myers Squibb (BMS)—the biopharmaceutical company that created the drug—called the FDA decision a "landmark approval" and "one that has the potential to change the treatment paradigm." 

Up until now, antipsychotic drugs, which address symptoms like hallucinations and delusions, worked by blocking access to dopamine receptors, according to The New York Times. This mechanism, however, contributed to common side effects like weight gain, social withdrawal, cognitive impairment, and lack of motivation. 

Cobenfy, the newly approved drug, indirectly influences dopamine levels. Unlike the standard approach of targeting dopamine receptors used by other schizophrenia drugs, Cobenfy targets the cholinergic receptors. The hope is that the new medication can provide similar effects on hallucinations and delusions as the current drugs on the market, while reducing side effects.

Gordon Lavigne, CEO of the nonprofit advocacy organization Schizophrenia and Psychosis Action Alliance, told Reason that the personal-social effects of Cobenfy "would be difficult to measure," but that "you can reasonably assume that someone is going to have more of an ability to engage in daily life, be social, and be part of the fabric of [their] family."

The cost of the Cobenfy is set to be in line with most other antipsychotic medications on the market. Adam Lenkowsky, chief commercialization officer for BMS, told the Times that its wholesale cost would be $1,850 a month.

This marks yet another free market win in the healthcare space over government-funded research. Since 2002, the National Institute of Mental Health (NIMH) has allocated more than $20 billion to the research of the neuroscience and genetics of diseases like schizophrenia. Yet little benefit has actually come to patients—even by the admission of the person who oversaw the explosion in funding. "I should have been able to help us bend the curves for death and disability," said Thomas Insel, who led the NIMH for 13 years. "But I didn't."

He's not alone. "As a schizophrenia researcher, I'm embarrassed to say we've spent literally billions of American taxpayer dollars on genetics, looking to understand what causes schizophrenia to help us develop new drugs for schizophrenia, and we have not been successful," Mark Weiser, the chief of the psychiatry division at the Sheba Medical Center in Tel Aviv, told the Times.

The post In a Victory for the Free Market, FDA Approves New Schizophrenia Drug appeared first on Reason.com.