Where can you read about the experience of mathematics? Certainly not in textbooks. They teach you topics. Algebra. Differential geometry. Category theory. Generally speaking, they are 100 percent divorced from the human experience; they are only about finding answers.

Another common genre of math book is the biography of a famous mathematician. But it’s almost impossible to reproduce the experience of creating new mathematics in this way, because the famous mathematician’s most famous results are already known. You can’t vicariously experience the stumbles, the frustrations, and the sudden insights, when you already know how the story ends. 

Then there’s another common type of math book, which I would call the smorgasbord. It presents a sampling of different topics, with a new topic every chapter, and perhaps some historical anecdotes around each one. I’ve read a lot of pop math books in this genre, and there’s nothing wrong with them, but they don’t tell a unified, coherent story. My vision of a book about the mathematical experience would describe the nearly obsessive focus that you need to live with a problem for months or perhaps years at a time, knowing all the while that it may not even have a solution. It would talk about all the stages in solving such a problem, some of them serendipitous and others deeply planned.

And even if you find a book that is literally a biography of a problem (for example, a book about Fermat’s Last Theorem), you’ll run into one more challenge, which is that the level of abstraction of frontier mathematics makes it nearly impossible for our proverbial “general public” to grasp. 

So here are my desiderata: A book that describes the whole experience of doing math, including the mistakes and the dead ends. A book that engrosses the reader like a mystery. A book that takes the reader to places they’ve never been before, like a travelogue. A book with surprising twists and turns, like a Shakespearean play. A book that is unified, that tells the story of one problem instead of flitting about from topic to topic. The problem should be deep enough and subtle enough that it requires input from different areas of mathematics — because I do want to convey the mysterious unity of mathematics, the way that any sufficiently hard problem tends to overflow the boundaries that we try to construct around it. Only a genuinely hard problem will do. And yet — and yet! — I want this problem to be so conceptually simple and so universal that a person with no particular mathematical training can understand what it’s about.

A problem like that comes along only once in a lifetime. Or maybe once in 2000 years.

Master Sun’s problem came to me. I didn’t go looking for it. A friend of mine sent it to me, and I got obsessed by it. I worked on it for a full year, and sorta kinda solved it. Then a few months later, an unexpected collaborator helped me solve it completely. We wrote it up and published it in a journal, and that was that.

But there was so much more to tell that I couldn’t put into the journal article. The whole human story, reaching back to ancient China and leading up to today. The story of my friend, who rediscovered the problem, looked around for 30 years for someone who would take it seriously, and finally got to see his “bucket list” problem solved before his untimely death at age 59. 

Now, ten years later, Master Sun’s Problem has finally been published, not in abbreviated journal-article form but as the book that it always wanted to be. Not only that, in this watershed year of 2026, a whole new set of reasons for the book has appeared. For the first time, artificial intelligence has surpassed human intelligence at solving math problems. OpenAI released a whole catalogue of 372 previously open math problems that its experimental AI had either solved or made major progress towards solving. Mathematicians are openly questioning what is the purpose of having humans do mathematics, if computers can do it better.

I’ve found it particularly interesting that top-level mathematicians are saying something they previously did not say openly — namely, that the true value of mathematics to them lies not in the solution of a problem, but in the experience of solving it. Here are some sample quotes I’ve collected (the last from a personal interview, the rest from the Internet):

“As a mathematician, I care not just about the end results, but also the journey.” – Zhiwei Yun (professor, MIT)

“If I’m doing a piece of math, I probably think of it as more akin to storytelling.” – Akshay Venkatesh (professor and Fields Medalist, IAS)

“We’re all frustration addicts. We just want to bang our heads against problems we don’t yet know how to solve.” – Garvy (a recent PhD who posts on Substack)

“I love the process of problem solving, banging our heads against the wall and getting frustrated and all of a sudden the satisfaction of seeing the idea that works.” – Kevin Ford (professor, Illinois)

What do mathematicians themselves value about mathematics? The journey. The story. The struggle. And the satisfaction. All of these are just as important as finding the answer. You cannot experience any of these things if you feed the problem to an AI and simply copy its answer. These are exactly the mathematical experiences that Master Sun’s Problem is all about.

At first, when the news about OpenAI and its wholesale harvesting of math problems came out, I was depressed. It seemed to me that my book, just days away from publication, had turned into an instant museum piece, a relic of the way we used to solve math problems before AI came along. But now I’m not depressed any more. The trauma of AI has forced us to re-evaluate our priorities, or to state them for the first time, and they were exactly the same as my priorities in writing Master Sun’s Problem. So I no longer consider my book to be a relic of bygone days. Instead, I consider it the first book of the new AI era in mathematics. In this era, the journey, the story, the struggle, and the satisfaction are what we have to preserve at all costs.

Mathematicians have been facing an unprecedented summer of reckoning, as artificial intelligence has proved itself capable of solving major unsolved problems. Over the last three years, we have clung to an increasingly tenuous belief that there are certain kinds of problems (ones that require “creative thinking”) that humans can still do better than AI’s. The solution of the Navier-Stokes problem, in my opinion, removes the last illusion of human superiority. We have now entered the age when AI can now solve any math problem as well as humans. (There are some caveats to that statement. I would say that they are still not as good at explaining their solutions; they are roughly equal to the best of us in finding solutions; and they are WAY WAY faster than us.)

My Washington Post op-ed looks at these developments from the point of view of a chess player. In the world of chess, we already had our moment of reckoning long ago, in 1997, when Deep Blue beat Garry Kasparov in a six-game match. But surprisingly, that wasn’t the end of chess. In fact, it has kicked off a golden era. You might think that grandmasters would have become irrelevant. Think again! In the age of streaming video, they have become media stars. Chess has never been so popular, and I argue in the Washington Post that computers with superhuman abilities have been a large part of that boom. They make top-level chess more accessible. They democratize it. I see this as a possible path for mathematics, assuming that mathematicians can adjust to the new landscape, and not fall into the “woe-is-me” trap.

AI has been on my mind a lot lately. Two weeks ago, I wrote an article for Science News about another math problem (the small prime gaps problem) in which AI had set a new “world-record.” The previous record had been set by a human three days before, who rushed to post her solution on the Internet before OpenAI could post their improvement (which was then rumored but hadn’t been announced yet). Julia Stadlmann will almost certainly be the last human to ever hold this record.

The prime gaps story actually broke five days before the Navier-Stokes story, but the latter story was much bigger news. I fear that Julia’s accomplishment got lost in the shuffle to some extent, but I appreciate the fact that Science News still saw some value in my article.

AI also puts a new spin on my book, Master Sun’s Problem, which will be published next month. At first, I thought that my book might just end up being a requiem for math as it used to exist, pre-2026. But upon further reflection, I realized that my book speaks to an important divide in the math community. For some people, and for some mathematicians, math is about solutions. That’s what you learn in math class, right? The teacher asks you questions, and you’re supposed to find the solution.

But I emphasize a different point of view in my book: for me — and for many mathematicians — mathematics is about the journey. It is what you learn on the way to the solution that matters the most.

If you are in the camp that believes the solution is the only thing that matters, then I can certainly see why you would be worried by the latest AI developments. If the computer can find answers more efficiently than humans, then why would we need human mathematicians any more? On the other hand, if you believe that the value of math lies in the struggle and in the deeply personal experience of coming to understand something that was previously a mystery, then mathematics will continue to be a source of joy and delight, and mathematicians will continue to have a purpose.

Today is publication day for a very unusual book that I contributed to. The Best Science Stories and How They Work is a compilation of eleven articles about science, selected by longtime science editor Siri Carpenter. Siri is also the co-founder of The Open Notebook, a project that provides resources and advice to science writers and would-be science writers. 

What makes this collection unique is that Siri also asked eleven writers who did not write the original pieces to comment on them. What makes the story work? What decisions or choices did the writer make? How did they explain the science in a way that is understandable and that grabs the reader’s attention, without compromising accuracy? What were some unusual or surprising features of the article? The commenters based their annotations on interviews with the original authors as well as their own experiences as science writers.

Siri asked me to comment on an article written by Jordana Cepelewicz for Quanta Magazine, called He Dropped Out to Become a Poet. Now He’s Won a Fields Medal. The article is a profile of June Huh, one of the four mathematicians who won the Fields Medal, the most prestigious award in mathematics, in 2022. There are so many interesting things to talk about in Jordana’s article, starting with the title. Jordana captures the reader’s attention even before the body of the article begins: This is an article about a high-school dropout who has become an immense success. She’s telling an evergreen story of redemption, while at the same time confronting several stereotypes around mathematics… and dropouts.

This was a unique and fun assignment for me, because it gave me a chance to really think about the craft of writing and telling a story. Plus, it introduced me to a remarkable person, June Huh. A year later, I had the chance to interview June myself, and it was fascinating to see how the same person could appear differently to different writers. Jordana definitely revealed aspects of his personality and his story that I would never have suspected. It is a tribute to her skill as an interviewer. There is no question that her reporting has influenced the world’s perception of June Huh. After a lecture by June, I stood in the line of people waiting to ask questions of him. One common theme was, “What are the similarities between math and poetry?” even though June had not mentioned poetry at all in his lecture. Each time I heard that question, I thought, “Yes, there’s another person who read Jordana’s article.”

As I worked on my commentary, and later on my own article about June, I felt as if I had entered a hall of mirrors. At its center was June, reflected in Jordana’s interview and in my interview. Then there was Jordana’s story, reflected in my commentary. And me, reflecting on my story and her story and wondering what I could learn from them. Who is the story about, anyway? Is it about June or Jordana or me, or maybe the people reading the article and drawing their own conclusions?

I hope that this gives you at least a taste of the delights of this collection. I will certainly order this book at the first opportunity, because I want to read the other ten articles. I want to learn about some cool science, and I want to see how my fellow writers explained that science. Then, I want to see how their fellow writers tackled this odd hall-of-mirrors challenge of writing about somebody else’s article. The book will definitely benefit anyone who writes about science as a profession — a small group, I know — but I think it will also be fascinating for anyone who is interested in science or the intersection of media and science.

Helmut brings up an issue that I wondered about as I was writing the book: Do I have any material in my blog that will be of interest to European readers? The U.S. chess culture is quite different from the European culture. The people I wrote about in my blog will be mostly unfamiliar to European readers. The rating system is different: when you see a USCF rating, you should generally subtract 50 to 100 points to get an equivalent FIDE (international) rating. The tournaments are run differently. In Europe, the standard is to play one round a day. In America, most tournaments (especially at the amateur level) are completed in one weekend, usually with two or three games a day. And on the master level, chess players receive more respect in Europe than America, or so I hear. A major chess tournament is a source of civic pride in the city where it is held, just as if it were a major tennis tournament or marathon race. (For just one example, see Wijk aan Zee.) 

But one thing that unites all of our chess cultures is the power of friendship. I devoted several chapters in my book to the chess parties that were organized by my friend, Mike Splane. These monthly parties, usually with 10 to 15 attendees, allowed us to approach chess in a different way. Instead of playing games, we would talk about chess and analyze games. Instead of competing, we would try to support each other in our attempts to learn and improve. (Occasionally, to be sure, this support might take the form of vigorous disagreement over a particular position. That’s all right! That’s how you learn!) When you can share ideas in an atmosphere of respect, where nobody is on an ego trip, you can learn and at the same time you can have fun while learning. That is why the answer to the question in the title of the book is: yes, and yes!

I’m glad to see that this model of chess parties appealed to Helmut, and it even reminded him of a similar “study group” in his past. I would love to see Mike Splane’s chess parties become a model to players all over the world. 

I’d like to announce the newest addition to the Mackenzie book collection! What’s Happening in the Mathematical Sciences (WHIMS) is an ongoing series published by the American Mathematical Society, about… well, just what the title says: current and recent events in mathematics. This is the eleventh volume in the series, six of which I have written or co-written. The nine topics I chose for this volume were:

• The mathematics of gerrymandering
• Mathematical models of dieting, and why it’s so hard to keep the weight off
• The detection of gravitational waves, and how math made it possible
• Topological data analysis, a step beyond simple clustering
• The Cap Set Conjecture, or how the game of Set met its match
• Mathematical models of bike-sharing and ride-sharing networks
• Expander graphs and their applications
• Quantum computers, and specifically the search for “golden gates”
• The simplest equation in math (x + y = 1) and how it has led to new progress on the asymptotic Fermat’s Last Theorem

As you can see, I tried to cover a broad spectrum of applied and pure mathematics. The chapters also have a range of difficulty, from topics a high-school student can easily understand to some that are at the level of a college math major. However, even with the more difficult topics I try always to show the “big picture” of why it matters, and I try to stick to Einstein’s dictum, to make it “as simple as possible, but not simpler.” The book has many color illustrations, which I think are part of the series’ appeal.

WHIMS volume 11 does not appear to be available yet on Amazon, but it is available through the American Mathematical Society’s website, $25 for non-members and $20 for members. 

This book is a bit of a personal landmark for me, as it is my tenth published book. I never expected to get to double digits! The significance of this landmark dawned on me recently, when I decided to compile a list of all the book published by graduates of the Science Communication Program at UC Santa Cruz. It was quite an eye-opening list. At the end I tallied up a “leaderboard” of the most prolific alumni book writers, and I was quite surprised by who came in #4.

1. Emily Sohn — 57 (!!) books published
2. Mike Wooldridge — 37
3. George Wuerthner — 29
4. Dana Mackenzie — 10

While I have no chance of ever catching the overachievers who are ahead of me, it made me feel pretty good to realize that out of 350+ graduates in the program’s 38-year-history, only three have published more books than me. By the way, I strongly recommend Emily Sohn’s books! She writes pictorial books or cartoon books on science for youngsters. I have to think that some of those 57 books may have changed lives, by getting kids excited about science at an early age.

Finally, here’s one more look at my latest book. All of my books have to pass the sniff test!

Being a writer, especially of books, is a strange business. You have to spend a very long time, months to years, working on a project with almost no encouragement or validation. It’s just you and your editor, and your co-author if you have one, as I did. You have to believe very strongly in what you are doing, because for the longest time that belief is the only thing that sustains you.

Then comes publication day, and suddenly the validation is very visible and very public. It’s like emerging from a cave into the dazzling sunlight. It may not have always been this way, but nowadays we have Amazon, which gives a writer immediate, hour-by-hour feedback on how the book is selling. Not actual sales figures but a ranking, which to a ratings junkie like me is like an opiate. (I used to religiously listen to American Top 40 every week as a teenager.)

A year ago today I got online at Amazon and checked my ranking. #5027. I was actually kind of disappointed, because it had been in the 4000’s the previous day due to pre-publication sales, and so it didn’t look as if publication had actually made much difference.

But then the fun began! By noon we were up to #3076, and my editor e-mailed congratulations. By 1 pm, #2179. By 5 pm, when I came home from the Aptos Library chess club, my wife had taped to the door a note saying “#1296!”. And then at 8 pm, we cracked the top 1000, at #952, something I had hoped and dreamed of but never really thought would actually happen.

Although that day was thrilling, for the most part that isn’t what the book business is about. What really matters is the next 364 days… and the 365 days after that… and so on. Thank goodness, the book business is the ultimate long-tail business. The value of a book is not measured by how many copies it sells on the first day. Thomas Kuhn’s The Structure of Scientific Revolutions sold less than 1000 copies in its first year.

Over the last year, the reviews have been pretty good, the sales have been pretty steady, and Judea and I keep seeing encouraging signs that our message has been heard. For example, just a week ago he sent me an article about the head of artificial intelligence at Toyota, who basically quoted straight from our Chapter 1. We had written about something called the ladder of causation. The Toyota guy renamed it as “the ladder of autonomy,” which I think is an even better name within the context of AI. This is great! When people are taking your ideas and re-thinking them, it really shows that you are getting somewhere.

Of course this success is 99 percent due to Judea. I have no illusions about that. I am just enjoying being along for the ride.

Anyway, this morning, just for old time’s sake, I got onto Amazon and checked our book’s ranking. #6546. It won’t be zooming up into the 3000’s or 2000’s or 1000’s today. But after a whole year it’s still #1 in the “Discrete Mathematics” category and therefore gets the little “#1 Bestseller” ribbon when you visit its Amazon webpage. That’s totally not shabby. I don’t think that very many books are #1 in any category after a year, and I hope it means that people think our book has lasting value.

Amazon also does provide actual sales numbers to authors, courtesy of Nielsen BookScan. Those numbers update once a week instead of once an hour. I know that people reading this blog have been curious about it in the past, so here is a graph of the first year of retail sales for The Book of Why, week by week. This isn’t actually all sales — for example, textbook and book club sales are not tabulated — but it’s close enough to give an author an idea of how the book is doing.

A few annotations:

1. New York Times bestseller territory is about 3000+ or maybe even 4000+ copies per week. As you can see, we never came within a country mile of that level of popularity. Some dreams are just plain unrealistic.
2. A few of the jumps have identifiable causes. The one-week jump in early June was most likely due to the New York Times review. The really dramatic jump in December was surely attributable to our selection as a Top Science Book of 2018 by Science Friday. Unlike the New York Times review, that honor seems to have boosted our sales for several weeks.
3. Also, the valley in October-November had an identifiable cause — our book was out of stock at Amazon due to supply chain issues.
4. For the last few weeks, sales have been pretty steady at 150 to 200 per week. The publisher (Basic Books) is very happy with this.

Who knows what the next year holds in store? For all the people who have sent their compliments, criticisms and questions to Judea and me (mostly to Judea), we have appreciated hearing from every one of you.

Coming to the U.K. on May 2!

Last week I got a package in the mail from England: ten copies of the paperback edition of The Book of Why! The occasion brought to mind that goofy old Beatles song, Paperback Writer. In fact, I’ve been waiting quite a long time to be a “paperback writer.”

My first book, The Big Splat, never sold enough to make it to paperback. My editor at Wiley told me that to be even considered for a paperback, it would have to sell 8000 copies. Ten years later it sputtered out and died at around 7500 copies sold. 

My second book, The Universe in Zero Words (now renamed The Story of Mathematics in 24 Equations) did make it to paperback, but under somewhat odd circumstances that make it hard for me to think of it as a true success. I’ll spare you the full details, but Elwin Street (the London publisher) took back the rights from Princeton University Press so that they could publish a paperback in the U.K. The book now has no U.S. publisher. It was more bad news than good news, in my opinion, because I really liked publishing with Princeton University Press.

And that brings us to The Book of Why, which has been all good news, all the time! Penguin UK, the British publisher, is releasing a paperback version on May 2. Note that this edition cannot be sold in the U.S. If you go to Amazon and try to buy it, and give them a U.S. address, I don’t think they will sell it to you.

So I am now going to be a paperback writer twice over in the U.K., but I am still not a paperback writer in the U.S.! According to our editor at Basic Books, the U.S. publisher of The Book of Why, their paperback version is scheduled for release in February 2020. I probably should not say too much about the reasons, but if any of you are wondering whether the sales are weak, the answer is no — in fact, it’s quite the opposite. The book continues to sell very well, and the editor is very happy with it. But as I said, the decision on when to go to paperback is a business decision, and the publishing business has changed in some obvious and less obvious ways. It’s no longer automatic that a successful book will come out in paperback after a year in hardback.

Which means, for my friends and readers in the U.S., if you’ve been saying, “I’ll wait for the paperback,” you probably shouldn’t. February 2020 is a long time away. I would recommend either buying the hardback, or taking a trip to England (a very good idea anyway!) and buying the Penguin paperback.

P.S. One good thing about buying the paperback is that you are getting the definitive, corrected version. If you are in the U.S. and you purchased the hardback, you should check the copyright page and see what the number of the printing is. If it is the fifth printing or later, it has been updated with very minor corrections and revisions. If you have a fourth printing or earlier, it is the original version, and we can send you a list of errata upon request.

In the article, a group of 47 editors of research journals in the fields of respiratory, sleep, and critical care have compiled “best practices” recommendations for researchers who want to make causal inferences from observational data. That’s a mouthful, so let me explain. Our book, The Book of Why, is all about when you can and cannot answer causal queries on the basis of observational data (*). In short, we argue that if you can formulate an explicit causal model, represented in a simple dot-and-arrow diagram, then there are simple mathematical tests to determine what causal questions you can answer. Questions like “Does this drug affect the likelihood of heart attack?” or “Will working a midnight shift increase my risk of sleep disorders?” In other words, the kinds of questions that patients ask doctors all the time.

The official policy of many journals, including the Journal of the American Medical Association (stated in 2017 in an editor’s blog post) is that authors of medical articles are not allowed to answer such questions. “If it isn’t [a randomized trial], and is a report of an observational study… then all cause-and-effect language must be replaced.” (Emphasis added.)

So you can see the strength of the taboo that Judea and I are challenging. That is why the joint statement of 47 editors from a variety of medical journals is so important. It is a public admission that the guidance of the American Medical Association is outmoded and outdated. We do have the right to use cause-and-effect language, provided that we state our assumptions publicly and transparently.

It would be wrong, of course, for Judea and me to claim credit for this change in attitudes. It has been evolving for some years. However, I think it is very intriguing that Table 2 of the article lists The Book of Why at the top of the suggested resources for people wanting to learn more about causal inference. In fact, there are three books recommended: The Book of Why, Judea’s much more technical book Causality, and a forthcoming (2019) book by Jamie Robins and Miguel Hernan called Causal Inference.

Further, the article says,

We urge authors to consider using causal models when testing causal associations. The scientific, mathematical, and theoretical underpinnings of causal inference, developed by Judea Pearl, James Robins, Miguel Hernan, and others, have evolved sufficiently to permit the everyday use of causal models.

Allowing for the usual opacity of scientific writing, this is just about as ringing an endorsement of Judea’s methods as you could possibly ask for. I encourage anyone who is interested in doing causal analysis of observational studies to take a look at the full article, read our book, and connect with Judea on Twitter.

(*) Here “observational data” is intended to contrast with data gathered from a randomized controlled trial, which has long been the gold standard for establishing causal relationships. (In fact, knee-jerk acceptance of randomized controlled trials is almost as bad as knee-jerk rejection of observational studies. Both of them rely on assumptions that must be scrutinized. But that is a discussion for another time.)

Science Friday, probably the nation’s most-listened-to radio show for science news, has named The Book of Why as one of its best science books of the year for 2018. It was one of eighteen books chosen. Here they are in alphabetical order. (That way, The Book of Why is near the top instead of at the bottom, the way it is on the Science Friday web page!) I’ll leave out the subtitles except when it’s necessary to understand what the book is about.

• Bad Blood: Secrets and Lies in a Silicon Valley Startup, by John Carreyrou
• The Book of Why, by Judea Pearl and Dana Mackenzie
• Broad Band: The Untold Story of the Women Who Made the Internet, by Claire Evans
• Chesapeake Requiem, by Earl Swift
• The Dialogues: Conversations about the Nature of the Universe, by Clifford Johnson
• The Dinosaur Artist, by Paige Williams
• Dispatches from Planet 3, by Marcia Bartusiak
• Heart: A History, by Sandeep Jauhar
• How to Change Your Mind, by Michael Pollan
• In Search of the Canary Tree, by Lauren Oakes
• Losing the Nobel Prize, by Brian Keating
• The Mystery of the Exploding Teeth, by Thomas Morris
• Nine Pints, by Rose George
• The Rise and Fall of the Dinosaurs, by Steve Brusatte
• Rising: Dispatches from the New American Shore, by Elizabeth Rush
• She Has Her Mother’s Laugh, by Carl Zimmer
• Spying on Whales, by Nick Pyenson
• What the Eyes Don’t See, by Mona Hanna-Attisha

Now I have to admit: I haven’t read any of these books, except for the one that I (co-)wrote. Shame on me! I do have In Search of the Canary Tree on my Amazon wish list, because I read a glowing review of it this week in Science. I am impressed that Science Friday says that She Has Her Mother’s Laugh is “clearly Zimmer’s best book,” because he has written a lot of books. Broad Band and Nine Pints (a book about blood) split my vote for the best title. In general, blood had a pretty good year (Nine Pints, Heart, and metaphorically speaking, Bad Blood). Of course, climate change had a good year; Chesapeake Requiem, Rising, and In Search of the Canary Tree all have to do with that topic. And dinosaurs are another evergreen topic (The Dinosaur Artist, The Rise and Fall of the Dinosaurs).

As for me, I’m just happy to be sitting at the same banquet table as these folks. It gives me the same spine-tingling, “Wow, is this happening?” feeling that I had the week our book came out. It’s possible that Judea and I will make it onto some other year-end best-of lists (I certainly hope so!), but this is the first one, so it’s special to me. And it’s a pretty great list to be on.