Fellow old-school physics blogger Sabine Hossenfelder has a new book out, Lost in Math: How Beauty Leads Physics Astray (currently Amazon's #1 best seller in Epistemology...). As you can guess from that title, this is largely an argument about the problems with using "beauty" in the form of mathematical elegance as a guide to developing theories about the world in the absence of data.
Most of this has to do with particle physics, and Ethan gives a good overview of the many problems of that field in his review. It touches on other subjects as well, though, including the field of "quantum foundations."
There's a bit of unsurprising nomenclatural confusion that sometimes comes up here, because "fundamental particle physics" sounds like it should be more closely related to "quantum foundations" than it is. As is often the case in naming of academic subfields and other literary genres, the two are far more distinct than their common Latin root word would suggest. With "fundamental particle physics," the emphasis is really on "particle" (albeit in the fuzzy it's-also-a-field sense of "particle" that quantum physics demands): they're mostly concerned with the nature and interaction of things that can't be subdivided into other things. "Quantum foundations," on the other hand, puts the emphasis on "quantum": it's the subfield concerned with interpretations of quantum mechanics, and what happens when you make measurements and all that fun stuff.
Hossenfelder's book includes a large-ish number of interviews with physicists working in the areas of interest, including, well, me. My role in the book is basically to make that distinction, and I'm reasonably pleased with the way that came out-- the interview that appears in the book matches my recollection of what I said, and even better, I still basically agree with what it says I said. I'm also very flattered to be in there between Steven Weinberg and Frank Wilczek, both of whom are vastly more important to physics than I am.
There was one bit edited for length, though, that seemed worth expanding on in a blog post; thus, this.
So, if "fundamental" and "foundations" are very different fields, why are they both in the book? It's not actually a flaw in the book that they're combined, it's a very sensible fit, because both areas share a kind of essentially aesthetic character. That is, if you read or listen to particle theorists talking about physics, you hear a lot of discussion of the "beauty" and "elegance" of particular models that have been invented to explain this or that, and the "ugliness" of the Standard Model, the most successful and least loved theory in the history of human civilization. Similarly aesthetic language also pops up a lot in arguments over interpretations of quantum physics: proponents of the Many-Worlds interpretations will praise its elegance and mathematical economy, opponents express revulsion at the idea of all those "extra universes," which seems needlessly baroque.
In both fields, then, there's a tendency to base arguments around what are essentially aesthetic concerns. I think there's a huge difference between them, though, in that the issues with quantum foundations are really aesthetic in an essential way, while particle theory faces large quantitative issues.
Quantum foundations is concerned with interpreting quantum mechanics, and to oversimplify very slightly, mostly involves discussing what "really" happens in the process of measuring the outcomes of an experiment. Interpretations differ about what they say happens in the moment when you go from an indeterminate wavefunction with a quantum system spread over many possible states to a single definite observed result. Has the wavefunction undergone a real physical change? Has the pre-existing state of a Bohmian particle simply been revealed to us? Or has the wavefunction of the universe simply grown a tiny bit more complicated to encompass all the possible results? Those are interpretational questions, and they're largely aesthetic in nature.
What you don't have in quantum foundations is any kind of quantitative disagreement. The means by which you calculate the expected outcomes of a particular experiment may be different from one interpretation to another, but what those expectations are is the same for all of them. We have well-established mathematical tools for calculating the probability of any particular outcome to any particular experiment, and they give the same results to some absurd precision. The difference between them is in how you think about what happened on the way from the start of the experiment to the end. Which is important for framing the questions you ask, but doesn't change the answers you get.
Fundamental particle physics, on the other hand, has enormous quantitative issues. When we look out at the distant universe, we see an accelerating expansion, that can readily be explained, conceptually, as a result of the quantum properties of empty space. Quantitatively, though, when people use known physics to estimate the size of that effect, they get an answer that's off by a staggering amount-- some claims put it at 120 orders of magnitude.
Similar problems crop up in other areas-- when we look at the universe, we mostly see ordinary matter, not antimatter, and we know the physical properties that are needed to create an excess of matter over antimatter. Known physics, though, can't come close to explaining the size of the excess that we see.
Both of those problems are quantitative in nature-- physicists think they know the right way to do the calculation, and it doesn't work. Those are real issues that are completely distinct from the "ugliness" of needing to specify a whole bunch of particle masses and coupling constants in the Standard Model, which is more purely an aesthetic issue. I'm not much moved by complaints about numbers of input parameters or the "fine-tuning" of their values, but I find "we think we know how to calculate these things, but the answers don't come out right" a much more convincing argument of the need for new approaches to theoretical physics.
So, that's the real split between fundamental particle physics and quantum foundations. Particle physics has experiments where there's a clear quantitative disagreement between theory and experiment and they know in principle what experiments to do to distinguish between theories, while in quantum foundations, nobody knows (yet) how to do an experiment that would give you a disagreement between two different interpretations. The turn to aesthetic criteria, then, comes from a very different place in the two subfields: the difference between quantum interpretations is aesthetic in an essential way, while in fundamental particles the issue is more that nobody has yet been able to do the experiments that would resolve the real quantitative issues.
The other bit I would add to what's in my comments in the book is a historical anecdote that was mentioned in my original conversation, but edited for length. It's the background behind my last comment in the book,though, which was "maybe the math is just ugly and someone needs to grind through it."
This was, in part, a reference to the story of quantum electrodynamics, which has to do with a lot of problems that came up regarding the properties of the electron. When physicists in the 1930's tried to calculate some really basic properties of the electron by quantum means, they got nonsensical answers: among them that the mass of the electron ought to be infinite, due to its interaction with its own electric field. For a while, they hoped to dodge this issue, but improved experiments after WWII made clear that it had to be confronted.
One possible approach to solving this was noted already in the 1930's: we never actually see an electron that isn't interacting with its own field, so all experimental measurements are really differences between an electron interacting with its own field and an electron interacting with its own field plus some other field from something else. It might be possible to get a sensible answer for that difference, even if it doesn't make sense to talk about either of the individual situations by itself.
Calculating that difference involves subtracting one infinite quantity from another and getting a finite answer, though, which is a tricky business mathematically, and only works under very stringent conditions. Some of the physicists involved, including Niels Bohr, thought that this would be resolved in some other way. Bohr even hoped that it would require a radical break with known physics to produce an elegant solution.
It turned out, though, that all it required was a bunch of ugly math. Julian Schwinger ground through a bunch of calculations and showed that, in fact, the mathematical conditions needed to subtract infinity from infinity and get a sensible answer for the energy of an electron in a magnetic field were met. Richard Feynman found a more user-friendly way to do the same trick, and Freeman Dyson showed that the two methods were in fact the same thing, now falling under the general heading of "renormalization."
Some key figures in physics in the 1930's found this disappointing and aesthetically unsatisfying. Paul Dirac, whose relativistic model of the electron kicked the whole business off, never liked renormalization, and I believe he went to his grave thinking it ugly. But it works, and remains at the center of our best models for how to calculate things quantum-mechanically, which many modern physicists find beautifully elegant.
So, that's the reference I was implicitly making, and it fits well with the point of Hossenfelder's book. A lot of the effort in fundamental particle theory has gone toward trying to find elegant and radical ways of doing calculations that don't come out right, and seem to require an ugly amount of fine-tuning. It may yet turn out that some experiment will come along that shows they're on the right track, but it might also turn out that the math is just really difficult and ugly, and some future Schwinger needs to grind through it.
