Witness Australia’s very own Peter Dutton licking the arse of a foreign despot, by plagiarising Trump’s threats against diversity and inclusion programs and plagiarising Trump’s gleefully reckless gutting of the government workforce. The timing really emphasises the rhetorical cheating.

These destructive policies help their respective nations in much the same way as would stabbing ourselves in the face with a fork. Diversity is widely known to promote efficiency and efficacy, and (lest I contradict fascist thought) is just morally good. There are better and worse ways to promote diversity (and sometimes we make mistakes in how we go about it), but there can be no doubt about the importance of the goal. To oppose diversity and inclusion is to oppose fairness, egalitarianism and human dignity.

And those “wasteful” government workers? They stick together the whole bloody country. If you want to have a nation, in any meaningful way, you need a strong public service to act on behalf of it. And we aren’t just talking about frontline services to the public (as important as that is). The public service must have the resources to analyse, strategise, plan, and organise things behind the scenes. If it lacks the workforce for this, then it actually ends up spending even more money outsourcing those functions to enormous for-profit consultancies. And that’s just a recipe for waste and corruption.

Dutton’s alignment to Trump is hardly surprising, at first glance. Dutton is stunningly racist. He represents the regressive barbarism of Australian politics that would rather bludgeon someone in the streets than to stoop to critical self-examination. He knows how to rile people up, to build a monster out of fear, to make an enemy of good-will itself, so that his politics can appear to be the answer.

But Dutton is also congenitally incapable of generating his own ideas. He has to import them. Trump himself is probably no more able to generate ideas, but does serve as the key centre of mass around which the new fascist movement has coalesced. And (in crayon) the key insiders of that movement feed Trump all the ideas he needs.

Whatever the dubious provenance of Trump’s ideas, trying to ape Trump doesn’t really work on a couple of levels. First, he’s utterly, inhumanly incompetent. In term one, even his supporters began to see this once he actually had to do something himself, aside from blaming others. His political instincts work for him in an election campaign when he has a foil, a powerful opponent to belittle and tear into, but he wasn’t able to adapt to actually being in charge. And so, being the understudy of a fatuous clown probably isn’t good politics for anyone else in the long run.

Second, you certainly can’t pretend to be like Trump when you’re actually interacting with him, as the Australian government surely must on a regular basis. Trump doesn’t want to be a franchise. He wants to be the uncontested king, the gravitational mass at the centre of the entire universe. There can only be one of him; if there were two, they’d fight each other to the death.

Thus (while Trump is still alive), other political leaders face two choices: they must either resist and retain some dignity, or thoroughly debase themselves before Trump with everyone watching. There’s no in-between, or going-around.

So what’s Prime Minister Dutton going to do when it comes to a trade war between the US and Australia? What’s Prime Minister Dutton going to do when Trump, perpetually aggrieved, demands Australia pay some insane tribute to the US, for whatever hilariously spurious reason, and holds the ANZUS Treaty to random?

It was bad enough watching John Howard’s obsequiousness towards US president George W Bush, back in the early 2000s. But Howard’s intellect towers over Dutton’s, and W is a moral giant next to Trump. And Dutton is already debasing himself in public.

In searching for the root cause of universities’ broken governance, consider democracy, or rather the lack thereof.

Australia’s public universities are created and supported by democratically-elected governments, but are also highly autonomous, and need to be so to protect their credibility as truth-seeking institutions.

But in the absence of democracy within universities, there are almost no checks on the quality of management decision making. Universities exist in an accountability vacuum, and so they frequently fall into dysfunction and autocracy, with arbitrarily-enforced, underfunded policies and self-serving power structures.

Universities commit wage theft, while paying their senior executives more money than state premiers (or even the Prime Minister), because of the sheer concentration of power in the hands of those senior executives, unrestrained by any democratic process.

University leaders often fail to listen, because they just don’t have to. Universities suffer from their own internal enshittification, brought on by vague or narrowly-defined performance objectives, which incentivise leaders to cannibalise parts of the university that actually work. Everyday systems and processes routinely break down or limp along with grinding inefficiency. Little is done to fix them, or build resilience, because leaders are not held responsible for any but the worst disasters (and even then, one wonders how much of an impact these events really have on an executive’s career trajectory).

This all perpetually endangers the quality of teaching and research.

Democratic control of universities would have a major, positive impact on university governance. Such control could come about via a combination of mechanisms:

• Elections of senior executives—the Vice Chancellor and immediate subordinates—by popular vote of the entire body of staff. (Some positions within a university are already voted on, but generally not the key decision makers.)
• Referendums for any substantial management decision, including policy and workforce changes. (Many things won’t be significant enough to vote on, but decisions with far-reaching consequences ought to have popular support.)
• Referendums for any petition acquiring a threshold number of signatures from staff.

Elections may be the most obvious democratic reform, but referendums and petitions may be more decisive. Referendums would force whomever is in charge to build and defer to consensus. Petitions leading to referendums would ensure that systemic problems cannot be ignored.

On reflection, it is extraordinary that we do not already do this. Australia prides itself on its democratic processes, and conducting elections and referendums within universities is logistically trivial compared to federal, state and local governments. University staff are highly informed and highly engaged, certainly within the context of the institution.

The economic and societal power wielded by a typical public university easily rivals that of a typical local council, and we would not accept undemocratic local government.

Even publicly-traded, for-profit corporations are held accountable at annual general meetings to their shareholders, who have voting rights. Modern universities are often disparagingly compared to corporations. Yet, the for-profit corporations are currently more democratic than public universities, which are statutory organisations ostensibly here to serve the public good.

Perhaps it’s time to have the courage of our democratic convictions.

Well, the part on diversity is important. Let’s make that clear to begin with. There are significant imbalances in the tech industry regarding gender, First Nations representation, culture, age, disability, neurodiversity and geographic location. And we need to work on all these things, because participation in tech needs to be representative of the people whose lives it affects, which is essentially everyone.

That’s the highlight of the report. And then there’s the bit on Generative AI, of which, you may recall, I’m not a massive fan.

The report proclaims Australia has a “generative AI gap”, compared to other Asia-Pacific nations. Specifically, we have less “AI talent”, we use less generative AI, we invest less in generative AI, and we don’t think our workplaces are set up for it. The report warns that “Additional action to encourage greater development of workforce skills related to AI will be required for Australia to close the generative AI gap.”

Here’s a graph from page 35 of the report:

My problem, in a nutshell, is that I don’t see the problem. This is good news! Well, selfishly parochial good news, and purely in a relative sense. Australia is wasting less time, energy, and reputation on what is essentially a technological cult.

This section of the report really is outrageously credulous. I don’t take issue with the figures cited (obtained through various surveys), but the interpretation of the data is brazenly beholden to motivated reasoning.

There is no reflection in the report on whether generative AI is, on balance, a force for good or not. Nor are there any citations to that effect. The report simply presumes it to be, even while presenting a chart-oid on all the things wrong with it (page 38):

A lot of these issues are absolutely fundamental to the technology, or to human interaction with the technology. Yet, the report stares them down and just burbles on about using “AI tools responsibly”, establishing “systems to manage AI risks” and “pragmatic approaches”.

Did I just have a stroke?

The report identifies, right there, a range of reasons not to use generative AI, and its conclusion is “hold my beer”? Can you imagine if we talked about gun control this way in Australia? Obviously we know that assault weapons dramatically increase the risk of mass shootings, so our challenge, given that we largely don’t have assault weapons, is to introduce them while “managing the risk”. Right. Yes. Thanks for your time.

My take on this chart is to question why those numbers are so low, especially for “factual errors and hallucinations” and “bias”. These problems are uncontroverted fact, not just opinion, and if only 35%-38% of respondents have encountered them, that’s the gap we need to close. We need to educate everyone in how bad the technology is.

I also fail to see where the report addresses “A lack of accountability”, other than off-handedly mentioning it in the chart, as if it’s just something some weirdos inexplicably kept bringing up in the survey. Just a mystery of the universe, eh?

We could really use some accountability in the tech industry, of all places. There isn’t much of it to begin with, and given the enormous potential for generative AI to screw things up, it’s only going to become more important. If we’re not serious about accountability, then we’re not serious about ethics, and then what the hell are we doing?

So, ACS/Deloitte, I’m not super-confident you really have your finger on the pulse here. You didn’t use ChatGPT, did you?

This isn’t a dig at artificial intelligence more broadly. AI covers much more ground than just chatbots like ChatGPT, and image generators like Stable Diffusion or Midjourney. Other AI applications, such as voice recognition or face recognition, have existed for a long time, and sometimes warrant their own ethical discussions, but are nonetheless often quite effective.

Where generative AI is concerned, the clincher is that, even if you disagree with the sceptics—even if you personally regard generative AI up as a worthy innovation—it’s already too late. It’s had its chance to convince the world of its usefulness, and the world has been mildly diverted, at best. Generative AI has bred a generation of informed cynics, who have seen what it does, and actively value its absence from their lives.

Obviously, there’s no shortage of frantic activity in support of generative AI, especially from the big tech giants like OpenAI, Microsoft and Google, dumping unearthly piles of cash to turn their products inside out, and Nvidia, collecting unearthly piles of cash from the sale of the hardware upon which AI models run. There’s an immense bandwagon upon which ride a great many people, many well-meaning, all doing various different things with generative AI. You may be forgiven for thinking this represents positive vindication of the technology, but we still haven’t escaped the hype.

And the hype is something to behold. If you hear about “Artificial General Intelligence” (AGI) in the context of generative AI, then someone is trying to sell you something. I’m interested in AGI, because I want to sell copies of my book. See how that works? (But seriously, buy my book.) AGI, the term for an artificial system able to reason independently about anything (hence, intelligent robots and the like), doesn’t exist. And its hypothetical future existence is unrelated to extant generative AI. Many of the ambitious promises and warnings around generative AI—it will perform scientific research, or it will replace or augment humans in other ways—are things that actually require full-blown AGI. Which nobody knows how to create.

What you see, in all this investment of time, money and prophesying, is not the realisation of a dream, nor the maturation of a proven technology, but a colossal bet. The sheer size of this gamble is the most astounding thing about generative AI, over and above any feature of the technology itself. It’s all the more remarkable, and concerning, precisely because the technology has so little proven utility.

Not for nothing, the largest and most expensive models are the most promising (the closest to being useful), and these exist at the whim of large tech companies. They’re betting that chatbots and image generators will ultimately produce revenue that at least matches the billions of dollars poured into them. Because if they don’t… well, generative AI is not a public utility. The best models could still be taken away from us, or lobotomised, or made prohibitively expensive, when shareholders start questioning their profitability. At some point, the bill will come due.

The models have some uses. Chatbots (backed by large language models, or LLMs) can assist with brain-storming, low-quality sales pitches, and subverting poorly-conceived busywork tasks. But every task that demands some level of accuracy or logic, where someone must actually be informed, is crippled by generative AI’s euphemistically-labelled “hallucinations”, or rather its bullshit. “Bullshit” consists not just of lies, but any assertions made without regard to their truth or falsity, and since LLMs make assertions without any capacity to discern truth, everything they output (or, at least, everything purporting to be informative) is technically bullshit. This isn’t notably something that we need any more of, and it leads to individual outcomes that range from embarrassing (“eat at least one small rock per day”) to career-limiting (try using ChatGPT for your legal research), and perhaps even life threatening in certain situations. There are also significant risks to scientific peer review, education, medical advice, and presumably any number of other fields.

“Hallucination” (bullshit) is not a glitch, but a permanent facet of the technology, reflecting LLMs’ fundamental disconnect from reality. While they do form sophisticated, abstract connections between concepts that appear in their training data, these are ultimately just patterns with no inherent meaning. When LLMs get things right, they do so by accident, where some elements of truth present in their training data have become superficially encoded in the statistical relationships between words. Nothing binds LLMs to tell the truth, ever. They just adapt the word patterns we humans generate. They have no notion of the objective reality in which we or they exist (and they don’t even understand other rule-based systems like mathematics). LLM are not wired up to the actual human experience of the world, and so they are not a truth-seeking apparatus.

Given that LLMs are guaranteed to bullshit, it’s worrying that so many people continue to suggest they be used for serious work requiring accurate information. One makes the other impossible. Yet, there’s still some impetus to just bulldoze right on through anyway, reinforcing the observation that we humans often have trouble with reality ourselves.

Such unreflective optimism, in combination with generative AI’s pervasive bullshit, breeds cynicism en masse very quickly. Even while generative AI inspires some, and to some extent even because of this, it opens itself to ridicule by many others. This isn’t just an intellectual contest. Gen-AI cynicism manifests as a social and market force. (Nobody will be reading your website if they think it’s been AI-generated. No humans, anyway, and probably no AGIs either, among those programmed to have any self-respect.) This will embolden those seeking legal and even political redress. Since it’s already perceived that generative AI investors are damaging the public good, by diluting human-generated information (and indeed humans) with auto-generated nonsense (“AI slop”), people are already pushing back. Lawyers, courts and politicians will act, whether in well-conceived ways or not, putting yet more pressure on the already-dubious profitability of generative AI services.

Nonetheless, some people retain an unshakeable faith in the inevitable march of progress. History has thrown up plenty of technological revolutions before, and, so you may say, the Gartner hype cycle shows that, once the initial “peak of inflated expectations” is overcome, we’re on our way to the “plateau of productivity” in which the mature technology is fully embedded in our lives. Just like (let me check my notes) human cloning, flying cars, fusion power, self-driving cars, the metaverse, hyperloop, cloud seeding and blockchain. Who among us can imagine a life without those, right?

Yes, the Gartner hype cycle is itself pseudoscientific nonsense, having the predictive power of a con artist. It’s hard to be charitable here, because it appears to suggest that anything and everything that becomes the subject of hype will succeed in the end. Just no. That’s not a thing. Some ideas just don’t work out, no matter how excited everyone was about them.

Hype lets people believe in inevitability, even destiny. You may think you can see the future unfolding in front of you, and that any hurdles an invention faces will be overcome by overwhelming narrative force. We have a term for this: “science fiction”. I know. I write it (and you can read it!). But real technological progress isn’t on a schedule that proceeds smoothly from now until Star Trek. It certainly isn’t a number that you can just extrapolate from data analysis. It’s an infinitely complex, qualitative state of affairs that twists and turns unpredictably in fits and starts. It often reflects a bit of the crazy in our social and economic systems, but must answer to basic logic.

Generative AI is mildly diverting, but ultimately shit, because that’s what’s possible. It’s been on a path of incremental improvement (impressive in its own way, in a relative sense), but its qualitative problems cannot truly be addressed without an entirely different approach. Moreover, training data is running out, and what has already been used may become legally problematic on copyright grounds. And increased training data has been the main contributing factor towards the improvements we’ve seen.

It is telling (if perhaps obvious to some) that, while generative AI promises to generate unlimited content, while also requiring near-unlimited training data, it cannot use its own output for training. The service it provides isn’t good enough for its own purposes. Feeding AI-generated content (“synthetic data”) back into the model as training data rapidly degrades the output quality. So, generative AI actually depends on the availability of ever-expanding amounts of high-quality human-generated content. That is, generative AI’s success ultimately relies, in some sense, on generative AI’s failure.

One might do well to consider this problem, in the cold light of day, before making heroic assumptions about our new “age of AI”.

Israel has bulldozed Gazan farms and cemeteries, restricted delivery of food and medicine, systematically attacked water infrastructure, bombed 80% of the territory’s schools and all its universities, and reduced almost all hospitals to ruins. It has forced around 1.7 million Gazans from their homes into arbitrarily-designed “humanitarian” zones with no actual provisions for their safety or basic needs, or for their future return. The IDF has killed over 200 aid workers in Gaza since the war began, at least some apparently targetted by precision weapons, but elsewhere has used hundreds of bombs so powerful they essentially cannot avoid killing civilians en masse. There are allegations of torture by Israeli forces against Palestinians and the discovery of mass graves.

Israel and its defenders^[1]One can defend Israel at different levels, without defending specific actions of its government or military. One can and should defend Jewish people (alongside all people), regardless of anything … Continue reading respond to these reports about as well as any proud nation would: by diverting the blame and accusing critics themselves of prejudice and hypocrisy. They demand that critics instead turn their attention to the brutality of Israel’s enemies. Supporters of Palestinians in the West have fallen into their own well-worn rut, in the form of flag waving, chants and the occupation of space for visibility, far from the war. The media and the authorities see the protests as a spectacle, and respond with corporate paternalism: either they are to be tolerated for the sake of the appearance of free speech, or they are to be shut down for the sake of the appearance of safety. And the protests themselves become the story, because they’re right here, and the… other thing is over there.

Our collective reaction to genocide has trouble keeping the focus on the thing itself. Even when we’re trying to talk about it, we can’t for long before we drift and stumble into meta-discussions about people and groups who aren’t directly involved, and the rules and expectations around them. Even genocide—even the worst type of crime we have a name for—becomes little more than an entry point into the merry-go-round of familiar, daily politics, for those of us who are comfortably far enough away from it.

There’s lots of context, but the context gets in the way too. The Palestinians should have a state, yes, and be in control of their own borders and institutions^[2]All people must be able to claim some national citizenship, as a necessary—if not always sufficient—part of guaranteeing their fundamental rights. A Palestinian state seems the most … Continue reading. But this is an argument that has gone on for decades. It doesn’t rise to the challenge of conveying the full horror of the immediate humanitarian crisis. Phrases like “intifada” and “from the river to the sea” are abstract, in-group rallying cries that mean nothing to the uninitiated. We should be talking about events on the ground. Mass starvation is beyond flag waving.

Many of Israel’s defenders seem preoccupied by the task of identifying antisemites and “Hamas supporters”, labels that are occasionally justified, but which are also thrown around with absurd carelessness and disingenuity. There certainly does exist visceral fear and anger after the atrocity of 7 October 2023, in which Hamas militants killed more than a thousand Israelis, predominantly civilians, and wounded thousands more, and of course took even more as hostages. I find the sheer inhumanity of that attack difficult to comprehend.

And yet, the brutality of the Israeli Defence Force’s response outweighs it by orders of magnitude, and has not been visited narrowly upon just Hamas itself, but on the broad swathe of already-struggling humanity in Gaza: millions of people who had nothing to do with the attack on Israel. There is no moral calculus that can justify this. The attempts to justify it ignore or grossly distort what the IDF is actually doing to Gaza and Gazans, focusing only on the trigger event, as if, because Hamas tore down the rules of morality, the IDF’s own actions require no examination whatsoever. But morality never disappears, no matter the violation. There is no justice in bulldozing, bombing, herding and starving the neighbours of terrorists. Hamas’s use of hostages, and whatever other dirty tactics may be at hand, may well prove its malevolence, but that culpability doesn’t somehow flow out into the people around them. You don’t get moral carte blanch for any reason, no matter how deep your wounds.

Israel cannot claim to be acting in self defence. It can defend itself, but this isn’t it. The term “self defence” has benign connotations because it specifically refers to doing no more harm than your immediate safety demands. The idea that it also encompasses the annihilation of one’s enemies is Orwellian, and reflects unconstrained power, not any fundamental right^[3]What of the Allies’ absolute victory in World War II against the Axis powers? But the Axis presented a true, existential threat to many Allied nations, and even then the Allies don’t … Continue reading. Actual self defence is proportionate, cognisant of shared rights, and accountable.

Nor does Israel get to claim it had no choice. There were choices made since long before the Hamas attack, and choices made long after it. Politicians have always exercised choice, and a military force with weapons as numerous and deadly as the IDF’s always has choice. Many people might have been spared, had some of those choices been different.

I know of nobody who could possibly support the attack by Hamas, though if Hamas can commit such crimes in the first place, I must acknowledge that we live in a world where some people do hold such disturbing and deranged points of view. But only slightly less deranged is the presumption that every critic of Israel supports terrorism until proven otherwise. This view is cult-like in its misreading of human nature. Who presides over such ideological accounting? Who decides what level of proof is acceptable? This is not how rational discussion works.

It is okay to point out genocide. It’s okay for that to be the only thing you want to talk about. You shouldn’t need to be seen to be morally or ideologically pure to do so. You don’t need to align yourself with any nationalistic agenda to do so. It is there.

The recent statement by ICC Prosecutor Karim A.A. Khan (on both the Hamas attack and Israel’s response) is worth reviewing.

(As is often taken for granted in software development, this wouldn’t be possible without the use of a considerable volume of other people’s work, most notably the Python-Markdown engine.)

The documentation explains the objective motivation for Lamarkdown, but at a personal level, it’s been a journey. Near the start of my academic career, I threw myself upon the mercy of LaTeX in order to produce my thesis. This included learning the diagramming package PGF/Tikz, whose own manual alone currently extends to over 1300 pages (because, you see, it was vitally important that the snippets of text in my figures were rendered in precisely the same font and font-size as the rest of the text).

After my thesis submission, I never quite connected with the research side of academic life. But I used LaTeX for virtually everything in my teaching. I wrote LaTeX unit outlines, lecture slides, practical worksheets, tutor notes, assignments, exams, and miscellaneous documentation. For a time, I had a script to auto-generate personalised assignment feedback files for students, using LaTeX. I built up a personal library of LaTeX style files and agonised over how to create all sorts of reusable abstractions.

(The key attraction of LaTeX, for me, is its transparency, which comes about from the simplicity of its medium: plain text. Everything you do in LaTeX is visible in plain text, and there are no dark hiding places where things magically happen beyond your control. If something goes wrong, it was you that did it, and you can surely fix it with the proper understanding. And you know that this is how it’s going to be.)

For 15 years or so, I thought I’d nailed it, at least in principle.

And then, one day, there was an unfamiliar notification in my university’s learning management system, where we present all our course material to students. My lecture slides and practical worksheets were getting a rating of 0% for accessibility. The university had deployed a little system that provided automated feedback to lecturers on the accessibility of their uploaded files, ultimately to help make life a bit easier for students with limited vision, and/or using screen readers.

I had been convinced that LaTeX could do anything. But it could not do this, not realistically. From what I’ve understood (from seemingly-authoritative comments), fixing TeX/LaTeX so that it generates truly, consistently accessible documents would be a major architectural revision. That sort of thing is difficult and expensive at the best of times, but during its 4 decades of life, LaTeX has accumulated an astonishing number of 3rd-party packages, many of which would make fundamental assumptions about what LaTeX does and does not do.

What LaTeX does not do (and what all its packages don’t do either) is keep track of the semantic structure of a document. This is needed for screen readers, and perhaps (so I speculate) for other potential modes of document viewing. As I understand it, the assumption pervading LaTeX has always been that you’re making a series of page-sized vector images to be printed. All that matters is how the output looks, not what any of it means, even structurally. (I suspect it was only a happy accident, owing to the efficiency of storing text, that LaTeX output is even searchable.)

I am still learning about accessibility, but it seems clear enough that Markdown and HTML provide a much more promising path to get there, compared to LaTeX and PDF.

It’s just that Markdown hasn’t conventionally been used for the preparation of documents in the manner of LaTeX, and so there are some gaps to fill, and that’s what Lamarkdown is for.

Along the way, it’s also struck me how much verbosity LaTeX inflicts on us over Markdown, and also how much of an anachronism the Portable Document Format (PDF) is in today’s computing world. PDF is a poor cousin to HTML for nearly every purpose, other than actually printing something out (and even then, only if you really care about minute details). PDF viewers do the best they can, but (to me) viewing a document as a PDF seems like an extraneous layer of complexity that is also bizarrely inflexible. Why isn’t the document simply in the one universal format? HTML, unless you really stuff it up, is faster to load, adjusts automatically to different screen dimensions, font sizes, and even theme colours, and is much more easily subject to automated processing.

I am still ambivalent about the value of pages in PDF documents. These can serve a practical purpose beyond being printed, as a way for readers to tally their progress or identify parts of the document. By contrast and by default, HTML documents are a continuous vertical expanse of content. But there are ways to impose pagination on HTML if it comes to that.

So we’ll see how Lamarkdown goes. I’ve already used it to create a range of new course material, including presentations (by combining it with RevealJS). There’s plenty of room for improvement. It cannot yet produce EPUB files (the HTML-based format underpinning most e-books), and this would be an eminently logical thing for such a tool to do.

But it’s out there now.

Addendum: I was going to call it “Lamarquedown”, but ultimately that seemed a bit too much like a highly rarefied cry for aid on a 19th century battlefield, and not quite enough like a document preparation system.

Melvin M. Vopson got himself in The Conversation, opining on the old Simulation Hypothesis. I can’t say where the idea truly originated, but Nick Bostrom laid down an argument for it in 2003 (so I’m quite late, as usual). I’ll call it “SH” for my purposes here. I’m also no physicist, but I know some things. The SH is philosophically provocative, but mathematically problematic and scientifically meaningless.

It’s one thing to ask whether our universe is, in some sense, embedded in some higher-order hyper-reality. This may be mathematically implied by certain physical theories, like eternal inflation and string theory (to my non-physicist, popular-science-informed mind, at least). But these theories do not involve simulation. They propose a single set of fundamental laws operating seamlessly across the entire scope of reality. Those laws don’t stop at the boundary between what we can observe and what we cannot, but rather seek to explain that apparent boundary.

The SH, by contrast, sets a hard boundary between the inside and the outside of the simulation. It gives us nothing from which to extrapolate from one to the other, making a mockery of Occam’s Razor.

Maths says go home

SH proponents may assume that a universal simulation would run on some computing device sitting in a place with essentially the same physical laws as the thing it’s simulating. This appears to break information theory. We’re asking that the computer—one small object in a vast universe—be able to simulate an entire, roughly-equivalent universe, which potentially includes other, roughly-equivalent computers. We’re asking for infinite regress, for the computer to be able to simulate everything including itself, simulating everything including itself, simulating everything including itself, ad infinitum. This computer would require infinite storage, and this seems less like a problem for science and engineering, and more like a problem for Futurama’s Professor Farnsworth.

This isn’t just a question of more R&D. A data storage device fundamentally cannot store more data than is needed to describe its own (finite) physical existence, because that’s all it has to work with. Data compression doesn’t change this. Compression merely reduces redundancy, and once redundancy has been eliminated, that’s it. It can’t magically encode arbitrarily large amounts of data into arbitrarily small amounts of data.

You may object that there doesn’t have to be a nested simulation (or that the simulations aren’t necessarily infinitely nested), but the point is that, if the two levels of reality are comparable, then the computer must have this capability, even if unused. If the computer was not able to simulate itself infinitely nested, then that inability must be explained by fundamentally different physical laws.

But—can the computer execute some devious short-cut when a simulated civilisation tries to create an equivalent computer within the simulation? Say it tweaks simulated events to stop such a simulated computer being created, or it fakes the operation of it. Perhaps, but this asks us to believe that the computer has unique cosmic properties. If the computer can contain enough information to store an entire universe, then everything else (of comparable mass) ought to contain similar amounts of information too. The laws of physics apply everywhere. We should expect comparable information density in other non-computing phenomena: clouds, ant colonies, the magnetic field lines of stars, etc. And then we’re back to the problem of storing infinite information.

The emperor’s new reality

So we can throw away the idea that any “parent universe” has the same physical laws as our (ostensibly simulated) universe. But beyond that, we have no reason to believe it would even remotely resemble our universe. It would need a certain level of complexity to support a universe simulation, but beyond that, it could have any physical laws and structures whatsoever. Extrapolation is impossible. It may have infinite dimensionality, including multiple or even infinite time dimensions, or it may even have zero dimensionality, with its dynamics tied up in other abstract structures. It may contain uncomputable dynamics able to solve halting problems, so that the “computer” may not even be a Turing Machine—the mathematical basis of all computers we’re capable of creating. We have no basis for any assumptions about it at all. The apparatus on which our simulated universe runs would itself operate according to rules that are completely unknown to us.

And while this may all feel philosophically enthralling, blowing our minds with infinite possibilities, it obliterates any possible investigation. Our chances of developing a testable theory of universe simulation, without any leads, are zero. We can’t possibly know what to look for. To Occam’s Razor, a hypothesis of “infinite possibilities” is no hypothesis at all. It is the scientific equivalent of gibberish.

Occam’s Razor, of course, demands that we choose the simplest theory that explains the evidence. It’s not enough that a theory merely fits the evidence; it has to be the simplest version of itself that fits the evidence. Newton’s First Law, for instance, emphatically doesn’t contain an extra proviso that an object’s path through space will spontaneously trace out the shape of a rabbit when nobody’s looking. Objects could be doing that—you don’t know—but it’s of no use to us to entertain such speculation.

Our one hypothetical chance would be if the “parent universe” intruded into the “simulation” in some way. That is, say we found a way of accessing the computer’s camera or microphone (or equivalent) from within our simulation. We could then build up a body of direct observational evidence of the parent universe. We wouldn’t know what we’d found, at first (given that the parent universe’s laws could literally be anything), but we might eventually work it out. And unless and until such a thing comes to our attention, we have nothing whatsoever to go on.

Almost there, just an entire universe to go

To reflect on the actual state of the art for a moment: simulating even the tiniest parts of our universe already taxes our computational designs to their limits. We struggle to create an accurate simulation of more than two or three elementary particles at a time, let alone the 10⁸⁰ (a number so large it can’t even be easily named). The complexity of quantum chromodynamics (QCD) within a single proton or neutron—not even a whole atom—demands a supercomputer to generate even an approximate prediction. The differential equations governing the laws of physics, as we know them, are known to be unsolvable in the general case.

The only way we can simulate anything at all, in practice, is by throwing out most of the detail. Given that we nonetheless see all that detail around us, right down to the untold trillions of subatomic particles interacting to form structures and substances of bewildering complexity, this clearly cannot be anything like the approach to simulation that we know.

I gather that SH proponents appeal to the inevitable march of technological progress to suggest that advanced aliens would have obviously been able to solve these challenges. This argument rests on an unearned faith in the inevitability of future discoveries, as though it’s axiomatic that we inhabit the Star Trek universe, and all that remains is a tidy schedule of progress from here to the warp drive, transporters and replicators of the Enterprise—the things we’ve been told to expect by storytellers.

You can’t know what a discovery “will be” before it’s been made.

NPS has become a commonplace tool for gauging public opinion of a product or service. But is it the right tool to gauge student satisfaction with university teaching practice?

This has been proposed, and I entirely understand why, and believe it to be entirely in good faith, and concede that there are studies that appear to support it at some level.

But I can’t use NPS myself, because a deep analysis of it leads me to confusion and despair. I feel one cannot provide an honest answer to an NPS question, because the different layers of meaning are contradictory. You can answer truthfully to the literal meaning of the question, or you can answer in a way that results in your opinion being accurately reflected in the final analysis. You need different answers to do this!

A key detail: NPS categorises respondents as “promoters” (those who answer 9 or 10), “neutrals” (7 or 8) and “detractors” (6 or below). An actual net promoter score is the percentage of promoters, minus the percentage of detractors (ignoring the neutrals).

Thus, we don’t ask for what we actually want to know (assuming respondents wouldn’t directly admit to being “promoters”, “neutrals” or “detractors”), and the mapping seems wildly at odds with the literal wording of the question.

Given the question “How likely would you be to recommend TheThing (on a scale from 0-10)?”, an honest answer of 5/10 seems to imply a 50% probability of at least one recommendation of TheThing to some other person. Thus, if N independent people answered 5/10, then the expected number of people making recommendations would be N/2, which is most certainly a positive number, and seemingly a decent outcome. Even if 10 honest people all answered 1/10, logically, on average, one of them will make a recommendation, which is still good on balance.

So where do the “detractors” come in? We haven’t actually asked anything about what negative actions a respondent may take.

As a respondent, I might find TheThing quite praiseworthy, while at the same time estimating that there’s only a 2/10 chance I’ll get around to recommending it to someone. I’m just not the sort of person who goes around recommending things. I have a lot on my mind, and I may simply not remember TheThing later on. And during a conversation, I don’t want to pester people by talking about things I don’t think they’d be interested in. Nonetheless, my 2/10 chance logically means TheThing is worthy of recommendation, in my view. I don’t feel I’ve declared myself to be a detractor, which would require the chance of recommendation to be 0/10.

I realise NPS is calibrated along other lines. It’s common knowledge that, in a “rating out of 10”, low numbers mean condemnation. People who “rate” something 2/10 or even 5/10 tend to dislike it and perhaps recommend against it. In one sense, adopting this interpretation for NPS is pragmatic, but in another sense it feels dishonest. The actual NPS question doesn’t ask respondents to simply “rate” something out of 10. It specifically asks for a “likelihood” of “recommendation”, which makes it a prediction (albeit a loose one), and a prediction only of positive outcomes. It could be influenced by factors beyond merit, and there is no “detracting” answer, technically. (A detractor would give a recommendation probability of 0/10, but so would someone who simply intends to stay out of the discussion.)

I also find it notable that NPS asks for and then throws away a significant part of the information provided. It asks you to choose between 11 possible responses, when it only needs and uses 3 categories. This just seems rude.

I could be accused of pedantry, perhaps. However, I am ultimately trying to make a point about the use of NPS in universities. Universities are full of people essentially trained (or being trained) in pedantry, for whom the precise meaning of both words and probability values is rather important. Moreover, 5/10 in a university setting is widely recognised as a “pass” mark, and 6/10 a “credit” (answers that would mark you out as an NPS “detractor”), which introduces yet another layer of confounding interpretation.

Thus, I doubt the meaningfulness of NPS in universities. Since I find the conflicting meanings personally irritating (and I avoid answering NPS questions myself whenever possible), I can’t in good conscience inflict NPS on others.

The Indigenous Voice to Parliament is not an end in itself. It is not just some shiny thing we must value for its own sake. It is an instrument, a light to guide us. It is a beginning, not an end. Its purpose is to help the government, representing all of us, understand how to do its job in respect to First Nations peoples. There is much work to come afterwards that will benefit from the Voice.

The opponent we face is our own cynicism, sometimes even nihilism. We are almost always cynical towards politicians. We are often cynical towards those of other ethnicities, beliefs, and much else. We are sometimes cynical of the very attempt at doing good, believing it to be a kind of banal theatre. I have been guilty of cynicism many times. It is a powerful adversary. It claims much anecdotal evidence (there are plenty of failures to point at), and it even perversely seems safe. Why believe in anything, when you can hate everything and never be disappointed?

But do we want to be this cynical? Do we want to hate? These things are self-destructive and fate-sealing. If we are too cynical to believe we can act in the interests of our own fellow citizens, then our nation is lost.

To be proud of our nation, we cannot be so cynical of our own efforts to improve it. We don’t always get it right, and we never get it perfect, but inaction is the very opposite of achievement. Inaction is not safe, nor wise. Inaction is justice denied, lost opportunity. It is a failure to overcome, and even an exacerbation of, that which we know to be wrong. A nation’s greatness does not just happen while we sit around; it must be achieved and kept alive through great works and moments of change.

In regards to the Voice, what is the purpose of demanding “more detail” and “practical outcomes” at this point? It is already the culmination of comprehensive dialogues and consensus-building between Indigenous Australians. The whole point of the Voice itself is to provide more detail, to deepen the conversation between a government formed out of the British Empire, and the descendants of those whose land we all now share. These two sides have often lacked a shared understanding. Institutionally, we are ignorant, and the Voice is designed to help address that. Practical outcomes follow from better-informed people and institutions. This is hardly contentious. We will certainly have a much better chance of “getting it right” with the Voice than without it.

The Voice is a simple, clear act of truth-telling, called for by the Uluru Statement from the Heart, adopted by the 2017 First Nations National Constitutional Convention. It is not the invention of any political party. It is a historic opportunity to make our nation stronger, for almost-embarrassingly little cost.

If nothing else, consider the pragmatism of the situation. Even if you’re not convinced the Voice will “work” (and so allowing for a great deal of cynicism), at worst the referendum would be a symbolic vote of confidence in reconciliation and the status of First Nations. So which act of symbolism will it be? Of what use is voting “no”, practical or symbolic?

I think the vote will be more than symbolic, though, because listening is never just symbolic. It will be an education.

This post is a critique mainly of Shadows of the Mind, and only of Part I^[1]Part II discusses the new physical theories Penrose thinks will be necessary to explain human consciousness, but that isn’t my real concern., since it’s there that Penrose sets out his arguments against strong AI in full detail. Much has been said already—this being nearly three decades after the fact—but the topic is important to me as an author of AI-focused science fiction.

Briefly, though, The Emperor’s New Mind contains a wealth of insights on mathematics, computing, physics, cosmology and the human brain. Penrose intends these to form the background to his argument, but they stand on their own as wonderful introductions to a range of complex scientific concepts. I learnt a lot!

However, when Penrose gets around to talking about artificial intelligence (mainly in Shadows), not everything seems to stack up.

Chapter 2: “The Gödelian case”

Here, Penrose sets up a mathematical reductio ad absurdum (or, really, proof-by-contradiction). He aims to prove that you cannot encapsulate all human mathematical reasoning in algorithmic form, if it’s assumed to be an entirely sound algorithm.

A few background notes on this:

• Most people (of those who’ve thought about it) assume or proclaim that the entire universe is “computable” (or, equivalently, “algorithmic”). This means that any part of it—such as the human brain—could, in principle, be simulated on a sufficiently powerful computer. This view seems to be true for our understanding of physics so far, but we’re far from understanding all of physics. It’s possible that the deeper aspects of physics, beyond current understanding, may be non-computable (non-algorithmic). If so, they would be impossible to simulate accurately on anything resembling a conventional computer (or even a quantum computer as currently envisaged), no matter how powerful it was.
• All computation can be represented by the abstract notion of a Turing machine, which has input, output, internal state, and (once started) can either stop after some time, or keep going forever.
• Penrose focuses on mathematical reasoning, because it’s a more constrained problem^[2]He’s a mathematician after all., and if he can show that AIs cannot compete with human minds here, then they can’t compete with human minds in general.

Penrose begins by defining the algorithm:

Suppose, then, that we have some computational procedure [algorithm] A which, when it terminates, provides us with a demonstration that a computation such as C(n) actually does not ever stop. We are going to try to imagine that A encapsulates all the procedures available to human mathematicians for convincingly demonstrating that computations do not stop.

(p. 73)

That is, our hypothetical algorithm A takes as input another algorithm, and tries to determine whether that other algorithm stops or not. This is known to be impossible in the general case^[3]It’s called the “halting problem”. Algorithm X cannot find out for all cases whether some other algorithm Y stops or not. The reason is that some algorithm Y might actually invoke a … Continue reading, but Penrose is only asking that (1) A stops only when it can be sure that its input does not stop, and (2) A keeps going forever in all other cases.

Penrose then applies A to itself (taking itself as input), and arrives at the semi-paradoxical result that, if A stops, then A does not stop. Therefore, A does not stop.

Thus, we know something that A is unable to ascertain. It follows that A cannot encapsulate our understanding

(p. 75)

That is, we humans can see that algorithm A does not stop, but A cannot tell us this itself, because it’s too busy not stopping. And if we can see something that the algorithm cannot, then the algorithm cannot be in possession of all human reasoning after all.

(Note: this is Penrose’s “Gödelian” argument, in reference to Gödel’s theorems, but those are typically expressed quite differently, in terms of formal systems of mathematical logic. An algorithm isn’t such a system, in itself, though Penrose draws a link between the two (p. 92).)

Now, however, I bring up the three main problems I have with this line of reasoning (even allowing the assumption that the algorithm is sound).

1. Reporting is not knowing

We might envisage that A actually does internally conclude that it will not stop, when given itself as input. This knowledge might be encoded in A’s internal state (remembering that A is a Turing machine), but A just won’t be able to communicate it via the approved mechanism, without creating a contradiction.

In light of the broader discussion around AI “awareness”, it seems unfortunate to overlook the role of a Turing machine’s internal state.

In fact, A could even output the result—“A does not stop”—that it’s been accused of not being able to ascertain, without creating a contradiction. The only technicality is that a Turing machine’s output isn’t supposed to be considered “official” until the Turing machine has stopped, but this feels like a very legalistic viewpoint when we know that almost all real software outputs information all the time without needing to stop.

2. Why the non-stopping requirement?

It seems arbitrary, and even self-contradictory, that A should have to keep going forever if its input algorithm stops. Why? This doesn’t seem to be necessary for encapsulating human mathematical reasoning. No actual human mathematical procedure runs forever, and certainly not where the actual task is provably finite. If the input algorithm stops, A can definitely find out in finite time and then stop itself (or it can give up and stop even earlier).

A more plausible version of such an all-encompassing procedure always stops in finite time, returning “true”, “false” or “undecidable” (where, we’ll say, the latter occurs if/when the algorithm “gives up”, for whatever reason). When this algorithm is given itself as input, it would simply return “true” or “undecidable”, without any contradiction! (In fact, it never really needs to return “undecidable” in this case, because it can always just compare its input with its own code, at which point it can safely return “true”.)

Penrose is right in that A cannot encapsulate our understanding, but only because he’s smuggled in an extra factor into the definition of A.

Penrose anticipates this objection (or something like it), and tries to address it as part of a Q&A section, but I can’t make sense of his response. He says:

The argument is supposed to be applied merely to insights that allow us to come to the conclusion that computations do not stop, not to those insights that allow us to conclude the opposite.

(p. 79)

But we could conclude, from the very contradiction Penrose sets up, that such sets of insights cannot be separated from one another. It’s hard to see the utility of separating them, in any case, since ultimately we are concerned with the sum total of human reasoning. Whether human reasoning can be broken down into disparate sets of insights isn’t really the point.

And I think Penrose needs to explain why he’s introducing the non-stopping requirement. There has to be a reason for it, not just the equivalent of “don’t worry about it”.

He then tries to explain:

…think of A in the following way: try to include both types of insight in A, but in those circumstances when the conclusion is that the computation C_q(n) [the input algorithm] does stop, deliberately put A into a loop (i.e. make A just repeat some operation over and over again endlessly).

(p. 79)

This is true as far as it goes, but it means Penrose is aware of two things:

1. This isn’t a natural way to write software, and certainly not the only way; and
2. It absolutely is possible (in line with my first objection above) for the algorithm to “know” the correct conclusion, even if it cannot report it.

I feel that Penrose is explicitly instructing us in how to cover up the very result that he’s trying to prove cannot be obtained!

He then says:

The argument has the form reductio ad absurdum, starting from the assumption that we use a knowably sound algorithm for ascertaining mathematical truth, and then deriving a contradiction.

(p. 79)

Yes, but that isn’t the only assumption! The contradiction could alternatively be taken to refute the non-stopping requirement.

3. Humans can screw this up too

We can imagine a mirror-image of the situation Penrose sets up, only applying to humans rather than computers. Suppose a scientist is asked to analyse their own brain, or the exact informational content of it, with access to all conceivable tools, information, help (including from the entire human race) and whatever other external resources might exist. The scientist is then asked to give a “thumbs up” in the case where their brain doesn’t make a response at all (or at least in some subset of cases, but never inaccurately), and to give no response in all other cases.

The scientist can do no such thing, for the same reason A is forced to not stop. We aren’t restricting the scientist’s investigative tools in any way. If the scientist’s own brain has an internal truth-seeking apparatus that transcends mere algorithmic logic, then the scientist is free to bring this to bear. The problem (as a matter of principle, putting aside the complexity) is that the task itself is paradoxical for whichever entity is told to carry it out.

And in this revised scenario, at the same time that the human scientist is unable to produce an answer, an algorithm could produce the correct answer (even a very simple one, just by observing that the task is paradoxical). And so now it’s the algorithm that enjoys apparently greater understanding than the human brain!

Have I understood this?

I’m not a mathematician, and I must admit to having a constant nagging feeling that I’ve missed some crucial aspect of Penrose’s reasoning or mathematical background. Of others who have responded to Shadows of the Mind, I can’t find any who dispute the argument in Chapter 2 (even if they dispute the overall case).

To put a foolhardy, non-mathematician’s foot forward, people seem to be thinking about this problem in terms of the “formal systems” of Gödel’s incompleteness theorems, and not in the terms Penrose has actually laid out. If you assume that a hypothetical AI mathematician has a sound formal system for proving mathematical theorems, then Gödel’s theorems say that certain truths are out of reach for the AI, whereas they would be within reach for a human mathematician, given they know which formal system is being used.

I’m certainly not going up against Gödel, but I also question the premise of an entity having “a” formal system in the first place. An algorithm isn’t a formal system (in the sense that Gödel means it), and a single algorithm for proving mathematical propositions may be capable of operating according to many, perhaps infinitely many, different formal systems. Thus, an AI mathematician may be able to see the truth of a “Gödel sentence” by switching between formal systems (as a human mathematician can).

Chapter 3: “The case for non-computability in mathematical thought”

In this chapter, Penrose looks at “unsound” algorithms, and tries to demonstrate that they too (like the hypothetical algorithm from Chapter 2) are untenable.

This is the part that most other people take issue with, because “unsoundness” is a gigantic loophole in Gödelian reasoning. Penrose’s approach, it seems to me, is to insist that mathematical reasoning can’t really be unsound at all.

He begins with this categorisation:

We must distinguish clearly between three distinct standpoints with regard to the knowability of a putative algorithmic procedure F underlying mathematical understanding, whether sound or not. For F must be:

I. consciously knowable, where its role as the actual algorithm underlying mathematical understanding is also knowable;

II. consciously knowable, but its role as the actual algorithm underlying mathematical understanding is unconscious and not knowable;

III. unconscious and not knowable.

(p. 130)

Penrose proceeds by arguing that I is impossible, II is highly implausible, and III can be reduced to one of the other two, and hence is also ruled out (p. 144). Disproving any one of these contentions would end the argument, and in fact I think each of the three cases is fatally flawed in its own separate way.

Case I

Penrose says:

Strangely enough, however, unsoundness does not help at all for a known formal system F which, as asserted in I, is actually known—and thus believed—by any mathematician to underlie his or her mathematical reasoning! For such a belief entails a (mistaken) belief in F’s soundness. (It would be an unreasonable mathematical standpoint that allows for a disbelief in the very basis of its own unassailable belief system!)

(p. 131)

Penrose appears to say that mathematicians cannot have knowably unsound algorithmic reasoning, for no other reason than that they are sure it’s sound. To me this seems an obviously and fatally flawed argument. A given person might (a) be unaware of the algorithm governing their reasoning, or in denial about it, (b) be in denial about the implications of the algorithm, or (c) not hold 100% confidence in their own power of reason. All of these seem to be logically permitted under case I.

I would think that most people (even mathematicians!) accept, in general, that their mental faculties are not 100% reliable, and so wouldn’t really have a problem if this was somehow formally proven. This doesn’t make our reasoning irretrievably broken though; we can still make progress, with sufficient review and revision. Penrose’s discussion seems to conflate imperfection with complete unworkability, assuming that any level of “unsoundness” is absolute fatal.

If there is some sense of “perfection” in mathematical reasoning (and admittedly I say this as a non-mathematician), surely it’s a perfection that we try to glimpse through our own imperfect cognition, not a perfection in our own selves.

I think Penrose simply cannot accept fallibility in human mathematical reasoning, almost as an article of faith. He isn’t prepared for the “unassailability” of human mathematical reasoning to be, well, assailed, even within the hypothetical discussion here.

Case II

But is it really plausible that our unassailable mathematical beliefs might rest on an unsound system—so unsound, indeed, that “1=2” is in principle part of those beliefs? Surely, if our mathematical reasoning cannot be trusted, then none of our reasoning about the workings of the world can be trusted.

(p. 138)

To answer Penrose’s question, yes, I do find it plausible that “1=2” is hiding away in one grimy corner of the sum total of our reasoning capabilities! But it (and things like it) can be “tuned down” to an infinitesimal weight with training. We learn how to perform consistent mathematical reasoning, aided by the consistency (where it exists) of the world around us. But we are (loosely speaking) only approaching sound reading in the limit.

Regarding trust in our reasoning, in the uncertain, stochastic real-world, this trust is a matter of degrees, not a binary “yes” or “no”. We don’t instantly turn to nihilism just because of the possibility of an error. We just do the best we can!

Penrose goes on to argue that, although mathematicians obviously do commit errors, these are invariably “correctable” and therefore not evidence of fundamentally unsound reasoning capabilities.

This falls foul of Occam’s Razor. Error-correction is a clear and obvious consequence of an imperfect system, particularly in light of the process of natural selection by which our brains formed. If our faculties are ultimately completely sound, why do errors occur at all? You would have to suppose, for instance, that we have two sets of mental faculties, one imperfect set that lets us perform temporary working and leads to errors, and another perfect set that lets us make conclusions that are (to use Penrose’s oft-repeated phrase) “unassailably true”.

It seems unnecessary to entertain this when there’s a simpler workable theory at hand. Basically I think Penrose is asking us to dismiss clear evidence that his assumptions are simply wrong.

Case III

Penrose devotes numerous whole sections to this final point, on the possibility of an unconscious and unknowable algorithm. His broader argument already seems a lost cause at this point, but it’s here that he gets around to discussing a evolutionary/machine-learning (“bottom-up”) approach for reproducing human reasoning, which is something I’m specifically interested in.

Referring to mathematical thinking in the context of human evolution, Penrose at first says:

It would be very reasonable to suppose that the selective advantages that our ancestors enjoyed were qualities that were valuable for all these things [subsequent achievements], and, as an incidental feature, turned out, much later, to be just what was needed for the carrying out of mathematical reasoning. This, indeed, is more or less what I believe myself.

(p. 148)

But Penrose is convinced that this doesn’t apply to a potential algorithmic version of human reasoning. He argues that, if human reasoning was algorithmic, then from the moment it first evolved, it must have had built into it (somehow) all the conclusions subsequently reached throughout all of mathematics:

This putative, unknowable, or incomprehensible algorithm would have to have, coded within itself, a power to do all this, yet we are being asked to believe that it arose solely by a natural selection geared to the circumstances in which our remote ancestors struggled for survival. A particular ability to do obscure mathematics can have had no direct selective advantage for its possessor, and I would argue that there can be no reason for such an algorithm to have arisen.

(p. 149)

This is an extravagantly passionate splitting of hairs. Why does non-computability make any difference here? Why is computable determinism any more difficult to swallow, philosophically, than non-computable determinism?

I also think Penrose is missing that the “human algorithm” can gain information from its environment^[4]Technically, a Turing machine is the wrong way to model this, but in general it’s a trivial task.. A lot of scientific and mathematical progress has followed from real-world experimentation. The algorithm did not need to incorporate all future human achievements purely within its own code. What the algorithm gave us, presumably, was the general ability to conduct abstract reasoning, and not specifically all the individual theorems mathematicians have arrived at, stored up and just waiting to be unzipped. Since we can reason abstractly, we can create models—abstract concepts grounded in aspects of the physical world. We use these for making things, predicting nature, organising society, etc. And, once such immediately-practical modelling proves its worth, it’s not surprising (to me) that it would eventually lead to people tinkering with pure abstraction.

Later, Penrose comes to another issue. If we suppose an AI can reproduce the mathematical understanding of humanity (and perhaps beyond), then we can’t actually know this unless the AI uses our language:

Now, for the interpretation of the formal system Q(M) [the system of mathematics underlying the robot’s reasoning], it needs to be clear that, as the robot develops, the imprimatur ‘*’ [used by the robot to label what it considers “unassailably true” theorems] actually does mean—and will continue to mean—that the thing that is being asserted is indeed to be taken as unassailably established. Without input from the human teachers (in some form) we cannot be sure that the robot will not develop for itself some different language in which ‘*’ has some entirely other meaning, if it has any meaning at all. To ensure that the robot’s language is consistent with our own specifications in the definition of Q(M), we must make sure that, as part of the robot’s training (say, by the human teacher), the meaning that is to be attached to ‘*’ is indeed what we intend it to be. Likewise, we must make sure that the actual notation that the robot uses … is the same as (or explicitly translatable into) the notation that we use ourselves.

(p. 161)

Penrose’s requirement here seems practically impossible to fulfil^[5]It could even be impossible in principle, though I’m not properly equipped to determine that.. First, machine learning techniques are not able to follow absolute rules regarding the very things they’re supposed to be learning. Neural networks and genetic algorithms can only really learn approximations of rules.

Second, how would we encode the meaning of the phrase “unassailably true” into an AI? This seems an insurmountably complicated notion either to learn or to explicitly program in. It’s not even clear to me as a human that “unassailably true” has any truly objective meaning.

And more broadly, how could we possibly ensure the AI uses our language (including mathematical notations) to express itself? We could just let the AI explore the world completely on its own terms, and thus come up with its own systems for representing mathematical theorems (assuming it gets to that point). It might learn our notations by reading Wikipedia, or we might train it explicitly. However, we’d have to hobble it in quite strange and counterproductive ways to ensure that it only uses our notations, given that we’re already assuming it to be capable of arbitrary abstract reasoning.

I think this is important, because Penrose is trying to show that any algorithm underlying mathematical reasoning can actually be known. But to do this, he needs to set specific requirements on the AIs that embody these algorithms, requirements that are almost certainly impossible to guarantee. In a sense, Penrose needs the AIs’ hypothetical cooperation in order to prove that they cannot exist.

Computational intelligence?

Hopefully it’s clear that I’m not trying to prove that AGI (strong AI) is definitely possible. I just don’t think that Penrose has successfully argued against it. I’m a very late entrant to this discussion, but I wanted to engage directly with the argument myself.

It remains to be seen whether AGI can be achieved practically. But there don’t seem to be any fundamental, theoretical roadblocks to it, and I can get on with writing sci-fi about it!