For Persi Diaconis’ Next Magic Trick …

A mathematician who has analyzed card shuffling for decades is tackling one final nemesis: “smooshing.”

[No Caption]

Skip Sterling for Quanta Magazine

Persi Diaconis shuffled and cut the deck of cards I’d brought for him, while I promised not to reveal his secrets. “I’m not going to give you the chance,” he retorted. In an empty conference room at the Joint Mathematics Meetings in San Antonio, Texas, this January, he casually tossed the cards into four piles in a seemingly random motion — yet when he checked, each pile magically had an ace on top. “Of course, it’s easy to get confused when there are a lot of cards, so let me just take four,” he said, scooping up the aces. He swiveled the four-card pile in his hands — always keeping it in the same flat plane — and sometimes the aces were faceup, sometimes facedown, even though they couldn’t possibly have flipped over.

Diaconis’ career as a professional magician began more than five decades ago, when he ran away from home at age 14 to go on the road with the sleight-of-hand virtuoso Dai Vernon. But unlike most magicians, Diaconis eventually found his way into academia, lured by an even more powerful siren song: mathematics. At 24, he started taking college classes to try to learn how to calculate the probabilities behind various gambling games. A few years later he was admitted to Harvard University’s graduate statistics program on the strength of a recommendation letter from the famed mathematics writer Martin Gardner that said, more or less, “This kid invented two of the best ten card tricks in the last decade, so you should give him a chance.”

Now a professor of mathematics and statistics at Stanford University, Diaconis has employed his intuition about cards, which he calls “the poetry of magic,” in a wide range of settings. Once, for example, he helped decode messages passed between inmates at a California state prison by using small random “shuffles” to gradually improve a decryption key. He has also analyzed Bose-Einstein condensation — in which a collection of ultra-cold atoms coalesces into a single “superatom” — by envisioning the atoms as rows of cards moving around. This makes them “friendly,” said Diaconis, whose speech still carries the inflections of his native New York City. “We all have our own basic images that we translate things into, and for me cards were where I started.”

In 1992, Diaconis famously proved — along with the mathematician Dave Bayer of Columbia University — that it takes about seven ordinary riffle shuffles to randomize a deck. Over the years, Diaconis and his students and colleagues have successfully analyzed the effectiveness of almost every type of shuffle people use in ordinary life.

All except one: “smooshing.”

This toddler-level technique involves spreading the cards out on a table, swishing them around with your hands, and then gathering them up. Smooshing is used in poker tournaments and in baccarat games in Monte Carlo, but no one actually knows how long you need to smoosh a deck to randomize it. “Smooshing is a completely different mechanism from the other shuffles, and my usual techniques don’t fit into that,” Diaconis said. The problem has tantalized him for decades.

Now he is on a quest to solve it. He has carried out preliminary experiments suggesting that one minute of ordinary smooshing may be enough for all practical purposes, and he is now analyzing a mathematical model of smooshing in an attempt to prove that assertion.

Diaconis’ previous card-shuffling work has helped to shed light on numerical approximation algorithms known as Markov chain Monte Carlo methods, ubiquitous in scientific simulation, which employ processes akin to shuffling to generate random examples of phenomena that are too hard to calculate completely. Diaconis believes that a mathematical analysis of smooshing will likewise have ramifications that go far beyond card shuffling. “Smooshing is close to a whole raft of practical life problems,” he said. It has more in common with a swirling fluid than with, say, a riffle shuffle; it’s reminiscent, for example, of the mechanics underlying the motion of large garbage patches in the ocean, during which swirling currents stir a large collection of objects.


Garbage accumulation locations in the North Pacific Ocean.

“The smooshing problem is a way of boiling down the details of mixing to their essence,” said Jean-Luc Thiffeault, a professor of applied mathematics at the University of Wisconsin, Madison, who studies fluid mixing.

Fluid-flow problems are notoriously hard to solve. The most famous such problem, which concerns the Navier-Stokes equations of fluid flow, is so difficult that it has a million-dollar bounty on its head. “The mathematics of any model for spatial mixing is in pretty bad shape,” Diaconis said.

Diaconis hopes that the union of fluid-flow techniques and card-shuffling math might point a way forward. “My kind of math — combinatorics, probability — is at right angles to the kind of math the Navier-Stokes people do,” he said. “If you bring fresh tools in, it might do some good in a bunch of these classical problems.”

Going Random

It might seem that no amount of smooshing can be definitively determined to be enough. After all, no matter how long you’ve smooshed the cards, wouldn’t more smooshing be even better? From a practical standpoint, probably not. Diaconis and Thiffeault both suspect that there is a particular moment in smooshing — a “cutoff,” as mathematicians call it — at which the deck transitions from highly ordered to highly unpredictable. After this point, more smooshing will confer only inconsequentially tiny increments of additional randomness.

The cutoff phenomenon, which occurs in a variety of situations in math and physics, owes its discovery to an earlier shuffling analysis by Diaconis and Mehrdad Shahshahani. In 1981 the pair was trying to understand a simple shuffle in which you just swap two randomly chosen cards. If you do many such shuffles, for a long time the deck will be far from random. But after roughly 100 shuffles it will suddenly transition to nearly random.

Since that discovery, the cutoff has been identified in many Markov Chain Monte Carlo algorithms, and recently it has even been discovered in the behavior of atomic spins in the Ising model, which describes the process by which materials become permanent magnets. “The idea of the cutoff has been very influential,” said Yuval Peres, a mathematician at Microsoft Research in Redmond, Wash.

All the card shuffling methods that have been successfully analyzed have cutoffs, and Diaconis conjectures that smooshing will too. “I’d bet $100 to $1 that smooshing has a cutoff,” Diaconis said.

10 Smooshing Tests

Diaconis is drawn to problems he can get his hands on. When he got curious about how shaving the side of a die would affect its odds, he didn’t hesitate to toss shaved dice 10,000 times (with help from his students). And when he wondered whether coin tossing is really unbiased, he filmed coin tosses using a special digital camera that could shoot 1,000 frames per second — and discovered, disconcertingly, that coin tosses are slightly biased toward the side of the coin that started out faceup.

So to get a feel for how much smooshing is needed to produce a random deck, Diaconis grabbed a deck and started smooshing. Together with his collaborators, the Stanford biostatistician Marc Coram and Lauren Bandklayder, now a graduate student at Northwestern University, he carried out 100 smooshes each in lengths of 15 seconds, 30 seconds, and one minute.

Next, he had to figure out how random the decks had become. The ideal way to do this would be to check whether each possible deck arrangement appears equally often among the smooshed decks. But this approach is utterly impractical: The number of arrangements of a deck of cards is 52 factorial — the product of the first 52 numbers — which approaches the estimated number of atoms in the Milky Way galaxy. “If everyone had been shuffling decks of cards every second since the start of the Earth, you couldn’t touch 52 factorial,” said Ron Graham, a mathematician at the University of California, San Diego. In fact, any time you shuffle a deck to the point of randomness, you have probably created an arrangement that has never existed before.

Since a direct experimental test of randomness isn’t feasible, Diaconis and his collaborators subjected their smooshed decks to a battery of 10 statistical tests designed to detect nonrandomness. One test looked at whether the top card of the deck had moved to every possible position equally often in the 100 smooshed decks. Another looked at how often pairs of adjacent cards — the seven and eight of spades, for example — remained adjacent after the shuffle.

Erica Klarreich for Quanta Magazine

Persi Diaconis demonstrating the card shuffling technique called “smooshing” at the Joint Mathematics Meetings in San Antonio, Texas, in January.

Of the 10 tests, Diaconis suspected that smooshing might have the hardest time passing the adjacent-pairs test, since cards that start out together might get swept along together by the hand motions. And indeed, the 15-second smooshes failed the adjacent-pairs test spectacularly, often having as many as 10 pairs still adjacent after the smoosh — more than enough hidden order for a smart gambler to exploit. “If you know that, say, 10 percent of the cards are still going to be next to the cards they were next to before, that’s a tremendous advantage if you’re playing blackjack,” Graham said.

Diaconis expected the 30-second and one-minute smooshes to fail the adjacent-pairs test too, but to his surprise, they aced all 10 tests. “I thought this was a lousy method of shuffling,” he said. “I have new respect for it.”

The experiments don’t prove that 30 seconds is enough smooshing to randomize a deck. They only establish that 30-second smooshes are not as egregiously nonrandom as 15-second smooshes. With a sample size of only 100 smooshes, “you can only detect very strong departures from randomness,” Diaconis said. It seems likely that the cutoff occurs sometime before one minute, since 30-second smooshes already seem to do pretty well. But, he said, “we’d be on more solid ground in discriminating between 30 seconds and one minute if we had 10,000 smooshes.” That’s far more than his group can carry out, so Diaconis is thinking about organizing a “national smoosh” in high-school or junior-high math classes.

Even more than additional data, however, Diaconis wants a proof. After all, ad hoc statistical tests are never a conclusive way to show that a shuffle is random. “It’s perfectly possible that some clever person will say, ‘Why didn’t you try this test?’ and it turns out to all be wrong,” he said. “I want to be able to say, ‘It doesn’t work after a minute and here’s why,’ or ‘It works after a minute and here’s a proof.’”

Theoretical Smooshing

When Diaconis returned to college after a decade as a professional magician, his first three grades in advanced calculus were C, C and D. “I didn’t know you were supposed to study,” he said. His teacher told him that he should write down the proofs and practice them as if they were French verbs. “I said, ‘Oh, you’re allowed to do that?’” Diaconis said. “I thought you were just supposed to see it.”

When it came to smooshing, instead of just trying to “see it,” Diaconis devoured the literature on fluid mixing. “When we started talking about the connections between cards and fluid mixing, he read the whole 200 pages of my Ph.D. thesis,” said Emmanuelle Gouillart, a researcher who studies glass melting at Saint-Gobain, a glass and construction materials company founded in Paris in 1665. “I was really impressed.”

While Diaconis grew more conversant in fluid mechanics, Gouillart benefited from his unique insight into card shuffling. “It turned out that we were studying very similar systems, but with different descriptions and different tools,” Gouillart said. The collaboration led her to develop a better way to measure correlations between neighboring particles in the fluids she studies.

Diaconis, meanwhile, has developed a mathematical model for what he calls “the sound of one hand smooshing.” In his model, the cards are represented by points scattered in a square, and the “hand” is a small disk that moves around the square while rotating the points under it by random angles. (It would be easy, Diaconis noted, to extend this to a two-handed smooshing model, simply by adding a second disk.)

Diaconis has been able to show — not just for a 52-card deck but for any number of points — that if you run this smooshing model forever, the arrangement of points will eventually become random. This might seem obvious, but some shuffling methods fail to randomize a deck no matter how long you shuffle, and Diaconis worried that smooshing might be one of them. After all, he reasoned, some cards might get stuck at the edges of the table, in much the same way that, when you mix cake batter, a little flour inevitably gets stranded at the edges of the bowl and never mixes in. But by drawing on 50 years of mathematics on the behavior of random flows, Diaconis proved that if you smoosh long enough, even cards at the edge will get mixed in.

His theoretical result says that the smooshing model will eventually mix the cards, but doesn’t say how long it will take. The model does provide a framework for relating the size of the deck to the amount of mixing time needed, but pinning down this relationship precisely requires ideas from a mathematical field still in its infancy, called the quantitative theory of differential equations. “Most studies of differential equations focus on what happens if you run the equation for a long time,” Diaconis said. “People are just now starting to study how the equation behaves if you run it for, say, a tenth of a second. So I have some careful work to do.”

Diaconis is optimistic that the work will lead him not just to an answer to the smooshing question, but to deeper discoveries. “The other shuffles have led to very rich mathematical consequences, and maybe this one will too,” he said.

Diaconis shares his magical secrets with only a select inner circle, but he dreams of laying the secrets of smooshing bare. “Smooshing is something that people use thousands of times a day, and mathematicians should be able to say something about it.”

View Reader Comments (12)

Leave a Comment

Reader CommentsLeave a Comment

  • We frequently play “Hand and Foot” which, for four persons, requires the use of five decks. I have used the smooshing technique for shuffling between hands.

    A question to ponder: If 30 seconds is good enough for one deck, how long do you have to do it for five decks (some of which may already be somewhat random)?

  • I wonder how Diaconis quantifies how much smooshing happens per second in order to be able to give results based on smoosh duration. Obviously if we perform exactly the same smooshing motion but you move your hands twice as fast as me, you’ll have accomplished the same amount of smooshing in half the duration. How does he capture the amount of smooshing we would both have done in this situation, without referring to a duration?

  • Douglas — that’s a great question. At this point, I think the only thing we can safely say is that you’d have to smoosh five decks longer than one deck. One of Diaconis’ goals, as I understand it, is to come up with a smooshing theory that would handle any deck size. At baccarat games at Monte Carlo, by the way, they smoosh eight decks together.

    Fred — that’s true, a fast smoosher presumably doesn’t have to smoosh as long. I think, though, that there’s probably not *that* much variation in how quickly people smoosh — there’s probably a fairly natural smooshing tempo that feels about right to most people. Part of the work of understanding smooshing will be to determine to what extent that is indeed the case. When Diaconis studied riffle shuffling, he examined how real people shuffled to make sure his mathematical model captured that fairly well. That will be an important step in studying smooshing too.

  • Fred,
    The same question came to mind while reading the article. How come the editor won’t even mention it? At leat comment on it? Is like making and apple pie with no apples and calling it an apple pie

  • Erica, Douglas—Multi-deck seems like a fascinating question. In fact, there’s two different interesting problems to explore here: The effect of the size of a single deck on the time required to shuffle, and the other is the effect of shuffling multiple decks. They really are different questions!

    Allow me to highlight the distinction with a very simple example.

    First, consider a deck with 4 unique cards. There are 4! = 24 unique permutations of those 4 cards. You need to shuffle long enough to be able to reach all 24 permutations from your starting permutation with roughly equal probability.

    Now consider shuffling a combined deck constructed of 2 decks of 2 unique cards. It’s the same total number of cards, but it has far fewer unique permutations. If I number the cards 0 and 1, you can see there are only 6 unique permutations: {0,0,1,1}, {0,1,0,1}, {1,0,0,1}, {0,1,1,0}, {1,0,1,0}, and {1,1,0,0}.

    It stands to reason that you reach the smooshing cutoff point for this second configuration much more quickly than the first. At least, my intuition is that you would. For random-exchange, it seems pretty clear that the cutoff point arrives sooner. Construct a graph where each of the nodes is a configuration of the deck, and an edge connects two nodes if they differ by one swap. The diameter of the 4-unique-card graph (representing the longest distance between two configurations) is much larger than for the 2×2 graph. When you consider random-swap shuffling is just a random walk on the graph we just constructed, it becomes obvious. I’m not sure what an equivalent model would be for smooshing, though.

    I imagine the same distinction would hold for a 5 x 52 configuration vs. a deck with 260 unique cards. (Or 8 x 52 vs. 416 unique cards.)

  • Note: In my previous comment, I mentioned the diameters of the two graphs differed greatly. For this small example, the difference in diameter isn’t huge, but the smaller graph still makes a big difference. (I just wrote up a program to exhaustively check.)

    In the {0,1,2,3} case, the diameter is 3, and most permutations require 2 swaps. You cannot shuffle {0,1,2,3} into {1,2,3,0} in fewer than 3 swaps. In fact, here’s the minimum distance table from {0,1,2,3} to all other permutations:

    0,1,2,3: 0
    0,1,3,2: 1
    0,2,1,3: 1
    0,2,3,1: 2
    0,3,1,2: 2
    0,3,2,1: 1
    1,0,2,3: 1
    1,0,3,2: 2
    1,2,0,3: 2
    1,2,3,0: 3
    1,3,0,2: 3
    1,3,2,0: 2
    2,0,1,3: 2
    2,0,3,1: 3
    2,1,0,3: 1
    2,1,3,0: 2
    2,3,0,1: 2
    2,3,1,0: 3
    3,0,1,2: 3
    3,0,2,1: 2
    3,1,0,2: 2
    3,1,2,0: 1
    3,2,0,1: 3
    3,2,1,0: 2

    In the {0,0,1,1} case, the diameter is 2. You cannot shuffle { 0,0,1,1 } to { 1,1,0,0 } in fewer than 2 swaps. You can reach _all_ other permutations in 1 swap.

    0,0,1,1: 0
    0,1,0,1: 1
    0,1,1,0: 1
    1,0,0,1: 1
    1,0,1,0: 1
    1,1,0,0: 2

  • This kind of shuffling is used in dominoes! Western domino games leave the tiles spread on the table, but the Chinese domino games stack the tiles in a “woodpile” which delivers the tiles to players in a predetermined order, like a card deal. If you have any interest, lo0k at and the section on tile games.

  • Is there anyway to figure this on a deck that does not start sequentially, like most decks we have around the house. Ths is fascinating and made me wonder about the efficiency and effectiveness of “52 pickup” Also made me curious about methods of shuffling that do not create randomness now matter how much you do them. Hard to conceive.

  • If he desired to be famous, Prof. Diaconis would focus his research on mah jongg instead of (or in addition to) cards. The principal method of shuffling mahjongg tiles is “smooshing” them in a circular motion while keeping the tiles face down. There are 144 tiles (or 136, because some games exclude the 8 flower tiles).

    For comity among friends and family, it’s very important that the walls at the start of the game be random and that players truly believe them to be. More than pocket money can be at stake!

    Mah jongg is played by millions and is especially popular among the Chinese diaspora. It’s played in homes, gaming parlors, and, inexplicably, by Jewish retirees in Florida.

    Please, Professor Diaconis, tell us how long we have to smoosh and what motions (e.g., counter-clockwise, figure-eight, wide diameter, etc.) randomize most effectively. A desperate world looks to you for guidance.

  • Isn’t there a deep difference between human-done smooshing and smooshing done by a robotic arm that is informed by a really good random-number generator? The first one is an experiment in psychology (are human smooshing patterns correlated across seconds or minutes, leading to specific outcomes), while the second is an experiment in statistics and possibly permutation group theory.

Comments are closed.