Houyhnhnm Computing: Houyhnhnm Computing

Chapter 11: A Work-Horse and its Tools

Ngnghm — Mon, 10 Aug 2020 21:01:33 UT

When my friends and I discuss the technological choices that humans make, we often call it “Yahoo Computing” between each other. However, I am careful never to use that word in front of Ngnghm (whose name I pronounce “Ann”) by fear that she would readily view us as Yahoos indeed, as we all yearn to qualify as Houyhnhnms in her eyes (which we pronounce “Hunams”). I have never heard Ann call any human “Yahoo”, either, not even when we describe the most irrational human behaviors; but I strongly suspect that it might be out of politeness, or to avoid triggering an adverse reaction if she told to our face how she really feels about humans.

One day we were discussing how a lot of extremely costly mistakes, some of them deadly, were made in human computing. Ann was particularly interested in these failures. She explained that failure patterns are often a great way to understand underlying structures that are not otherwise directly observable. What patterns were there in those failures? What do they teach us about how humans make decisions? She was interested as an anthropologist. I was more interested as a practitioner: if we can identify some defect in the way humans tend to make some decisions about computing, some kind of myopia, then can we devise systematic ways to correct them? Did Houyhnhnms have similar failings, and how did they address them?

Bad Tools

I was discussing with friends, sometimes laughing, sometimes crying, about some notable mistakes that people made while operating software, that each time were leading to catastrophic results: deleting a lot of data, destroying billions of dollars worth of assets, crashing an airplane and killing people, etc. As we were laughing at the users misusing the software, Ann stopped us, and pointed out this piece of Houyhnhnm wisdom, that isn’t specific to software: When a casual user mistake causes a tool to fail catastrophically, fools blame the user who operated the tool; wise men blame the toolsmiths who built the tool.

If a worker is maimed because he recklessly operated some heavy machinery, the worker is to blame. But if a constant stream of workers end up maimed due to the not-so-uncommon co-occurrence of several routine mistakes—then the machinery is badly designed, and those who keep manufacturing, selling, buying and managing the operation of this unmodified machinery are fully responsible for all the misery that ensues.

This rule obviously applies to software engineering: When a casual user mistake causes some software to fail catastrophically, fools blame the user who operated the software; wise men blame the programmers who built the software.

In the famous incident wherein people died due to bugs in the Therac–25 control software, the programmers who wrote that software, their managers, and their executives, are more to blame than the medical personnel who operated the deadly radiation device. Less deadly but cumulatively no less costly, if a whole lot of users, like my mother, lost hours of work for failing to “save” a document, or like my father, lost years of archives to broken disks, or like many a novice Unix weenie, lost all his files to a typo or thinko that made an rm command overly broad, — then the professional authors and promoters of the applications, operating systems and shells that make these errors likely are more to blame than the amateur users who hurt themselves and others.

Now, as often in programming, we can take this rule one step further (and then one more, etc.): when a casual programmer mistake causes some catastrophic software failure, fools blame the programmer; wise men blame the programming language designer.

If programs written in a Segfault-Oriented Language such as C or C++ produce a constant stream of crashes, data corruption, security breaches, device malfunction, etc., then the authors and promoters of the C and C++ programming language are much more at fault for all those failures than the individual programmers who use these languages, especially so the novice programmers. And if more experienced programmers in these languages resort to otherwise bad, expensive or limiting practices to defend against the threat of the previous catastrophic failure, the language authors are also to blame for the costs incurred.

Of course, the rule applies not only to programming languages, but to operating systems, libraries, frameworks, applications, database servers, and any piece of infrastructure used but not implemented by the programmers whose fatal mistakes partake in a pattern that is common for that piece of infrastructure. The expert authors of software infrastructure are responsible for the structural defects of their software, wherein their users (who are programmers) will constantly make the same class of mistakes.

Bad Workmen

Now, at some point my (human) friend objected with another piece of wisdom: A bad workman blames his tools.

Shouldn’t a competent user take into account the potential dangers of his tool and still use it safely? Shouldn’t a good programmer do just as good a job in any programming language? No. Not. at. all. That’s not at all what the proverb means—though the proverb is valid and applies to the situation indeed.

Certainly, a good workman will (on average at least) get better results than a bad workman, even if provided sub-par tools. But most importantly, the good workman will evaluate whether the tools available to him are the most appropriate to complete the job, and if they aren’t, will remedy the situation before he even starts working on the job. If somehow his customer, principal, boss or foreman requires him to use bad tools, he will object and offer his expertise on how the job should better be done instead. He will make it clear that the bad choice of tools will yield lesser results at greater cost and significant risk compared to a better choice of tools. He will charge the customer triple for the privilege of telling him how to do his job. He will take extra precautions to palliate the issues with the badly chosen tool, which will increase delays and costs. In the end, he may even refuse to do the job if he judges the tools imposed as wholly inadequate, and he will warn his more junior workmen of the dangers he foresees if they accept the job.

That is, bad tools would still be bad even in the hands of a good workman. But a good workman is able to recognize them, and would not be caught dead with the bad tools, except maybe in a life-or-death emergency where there is somehow no other choice. If a workman makes a bad choice of tools, then he is ipso facto not a good workman. At the very least, this is one important measure of his lacking in workmanship, though he may be good or even great at other parts of the job.

Thus, whenever a catastrophe happens that could have been avoided through the use of better tools, inasmuch as the tool user had a say about which tool to use or not use, then he is also culpable — not because his casual mistake caused the catastrophe, but because his choice of a bad tool made the catastrophe likely or inevitable. And this culpability only increases, both (a) the more the user was an expert in the field, even if he did not make the call, and also (b) the more he had a say in the decision, even if not an expert.

Indeed, the more empowered you are in making a decision, the more you are responsible for making it a good one; at that point it is your responsibility to seek from others whatever necessary expertise you don’t possess personally. If a child under your care is sick, and you fail to consult a competent doctor who would have prescribed a proper treatment, but instead poison him dead with a “cure” you plain made up, then you are fully guilty of murder, and your lack of medical education is no excuse. Same if the “cure” was prescribed by a quack doctor whom you chose to trust against due diligence (that you may have, criminally, neglected). The argument applies just as well to the failure of a software project.

A novice at the bottom of the programming hierarchy, who lacks either knowledge or power, has little praise or blame to receive in the positive or negative outcomes of the software he partook in building. But a seasoned programmer who, through the use of a bad programming language, partakes in a catastrophe, is largely to blame for having accepted to write in said programming language. Most of all, the entire hierarchy of technical leaders and business managers who made the decision are to blame, culminating with whatever CEO or bureaucrat had executive powers.

The Blame Game

As my human friend and I were trying to pin down blame on one person to try to exonerate the other, Ann was astonished. She noticed that we were playing the “Blame Game” as if blame were additive: that is, we were supposing that blame is distributed amongst participants, such that the sum of all shares of the blame add up to 100%. Actually, Houyhnhnms well understand that blame is subadditive, or, in a layman’s terms, blame overlaps: in a joint decision between the two of us, where either of our strong objection could have had a 80% chance of averting the bad outcome, then we are each 80% for the outcome, though our joint blame is only 100%, which is less than the sum 160%. More generally, each participant or set of participants is assigned an amount of blame corresponding to the probability that a good decision of theirs could have avoided the bad outcome; then, whenever you partition a set of participants into subsets (possibly reduced to a singleton), the amount of blame assigned to the whole set is less than the sum of the amounts of blame of the parts. Looking at it in the other direction, the sum of the amounts of blame of the parts will be more than the amount of blame of the whole (160% vs 100% in the example above).

This blame game applies as well to all decisions in all aspects of life. But let us narrow it down to the outcome of choosing bad (software) tools.

In a corporate catastrophe, the CEO or other ultimate executive is fully responsible at 100% for the catastrophe, as well as for everything that happens in his company. But that does not exonerate in the least any of the workers down the chain of command. The CTO would still be responsible at 99% or so for a software catastrophe, the VP of the relevant branch would be responsible 98%, the head of products at 96%, the product manager at 94%, the team lead at 92%, senior developers at 90%, and so on, with even the most junior permanent developer having more than 50% blame, and interns and novices having over 25% blame, many shareholders at more than 10% each, and millions of casual users at more than 1% each. Anyone who could have acted to avert the catastrophe, be it by speaking out or walking away, is to blame. This blame increases with how much their action would have been likely to influence the outcome. The sum of all those blames is well over 100%. The ones’ greater culpability is no excuse for the others’ lesser culpability.

You, who are interested enough to keep reading these writings about programming, possess some software expertise or executive clout, and will be responsible, partly or fully, for any successes, failures and catastrophes that will result from the choices you either make, or accept to enact. To what degree you’re responsible will depend on how much you did know or should have known as you were or weren’t empowered in making or implementing the decisions. Ignorance will be no excuse for giving bad or evil orders. Impotence will be no excuse for carrying them out.

Your skill and your power won’t be put forward as excuses to dismiss your glory if you are wildly successful; quite the contrary, this glory will rightfully reflect well on your skill and power. Your skill and power, or lack thereof, won’t be valid excuses to dismiss your responsibility in the catastrophes you cause. Quite the contrary, the shame will rightfully reflect on your skill and power.

Expertise and Meta-Expertise

Now that we understand that making a good choice of tools is an important responsibility, let us examine how to exercise it. What tools should you be using for your next project? When can you tell the tools used for the current project are inadequate after all and need to be changed? Of course there is no one-size-fits-all answer to these questions. A good solution may depend on many criteria such as:

What are the goals and constraints of the project?
What resources will you have to complete the project?
What are the costs and risks involved?
How long do you have to find a solution? How long will the solution have to last? How will the costs and benefits evolve over the relevant timeline?
Can some off-the-shelf product help? How well do they fit the problem now? How will this fit itself evolve over the life-time of the desired solution?
How proficient are you or your team at using various potential tools? How proficient can you become within the deadlines considered? How will this proficiency and its effects evolve over the relevant timeline?
How easily can you find outside help in the amount required, at the skill levels required, within the allotted budget and the allowed risk envelope?

Yet, there are infinitely many potential such questions, and an arbitrarily high level of detail, whereas you will have a small finite amount of information based on which to make your decisions. You will have to restrict your attention to a few salient questions, that you will have to answer through a judgment call, because getting sufficient objective information, even when possible, will be unaffordable. And so, ultimately, no one can make your decisions for you. Not I, not anyone. You will have to consider the issue, to select the relevant questions, to estimate the answers, to construct the most likely scenarios, and to weigh the expected costs and benefits, to assign probabilities to various events and values to outcomes, to establish useful models, yet to avoid following them as a substitute for observing reality (or at best you’ll run into Goodhart’s Law), etc. Doing a good job at identifying the most relevant questions and the best answers to them is what expertise is about.

But often, you will personally lack some or all of the suitable expertise. Then your job will be to consult with proper experts to fill the gaps. And finding experts on topics where you aren’t yourself an expert is its own separate meta-expertise.

Titles and diplomas and low or high job positions can help you identify the best experts. But they can also hide quacks, frauds and social climbers, who prey on those who are too keen at deferring to authority — or just not-so-good and over-confident experts whose actual specialty is only so close to the topic you’re interested in, yet who may fail to disqualify themselves from giving you advice, when they instead should admit incompetence and refer you to better experts. One way to identify experts can be indeed to ask for referrals from multiple experts on close topics, until you find people highly recommended by multiple independent sources; and this can be enough in case the expertise is itself considered uncontroversial, or you somehow trust the meta-expertise of the referrers themselves.

For important decisions on specialized topics where the expertise ceases to be obvious and uncontroversial, or when you can’t defer your meta-expertise to other experts, you will want to follow some meta-principles that will help you identify actual experts while rejecting frauds who disguise as experts. One technique is to ask the potential expert to explain the issues at stake to you and other meta-experts, and to counter the objections that you may have gleaned from various sources. Lacking the expertise yourself, you may not be able to fully judge their expert knowledge and opinion on the topic; yet, their ability to cogently explain the issues and address objections can show that they have thought about the topic a lot and know what they are talking about, whereas their inability to explain, or worse, their demanding that you blindly trust them, may be symptoms that they are not expert enough, or at all. Thus, you can interview multiple candidate experts and each time bounce ideas from relevant technical writings such as the discussion below, or from other candidates, to see how well they address these issues, and can argue the pros and cons of available options.

Time-Preference for Better Tools

As we were exploring how Humans and Houyhnhnms choose tools in particular, Ann and I came to notice that one aspect that Humans often neglected, but that Houyhnhnms seemed to care about a lot, was to look at their choice of tools through time,

In an emergency, you use the best tool at hand, even if the best tool at hand is only a piece of cut stone. But if as a professional technologist, you find after twenty years of practice that your best tool at hand is still cut stone, and what more, that you are now a virtuoso at using it—then you might not be such a great professional technologist. Unless of course you live in a small paleolithic society with little slack left for invention, where cut stone actually is the best and best imaginable technology for the entire duration of your life.

Thus, for an urgent, throw-away, next-day software project, circumstantial considerations shall rightfully dominate the tactical choices made, at which point the programmer in charge should and will “just” use the language at hand for the project: a language with which he is the most familiar, a language that already solves most of the issues out of the box, and/or a language already used by the rest of the team, that is already known to be properly deployed in a larger production setting. There is little reflection to be had in such a choice, only a snap decision made as a tactical call in an emergency, where you already have all the information you can afford to have, completely based on local circumstances, valid for a day only. It’s too late for anyone else to help you in these cases. The die is cast. You made your bed, and now you lie in it. The decision was taken with what austrian economists would call High Time-Preference: you care about what you can get done right now, with no regard for consequences you will experience tomorrow, much less any long-term consequence.

At the other extreme, consider a software project developed for the foreseeable future, with no emergency: a piece of automation on which you are betting your company’s future, one that will still exist in some form or other in ten, twenty years, maybe more, on which you and your successors shall be working for all that time and beyond. This project won’t be written once, run once, then thrown away, but will have to be continuously deployed, maintained, updated, each part of it rewritten many times over its life time:

to respond to the changing demands of an ever changing world,
to interface with a moving set of external systems,
to keep satisfying a growing number of users in ever renewed ways,
to withhold attacks by ever more determined and more sophisticated enemies,
to satisfy ever higher standards, whether imposed by market pressure or government regulations,
to fix issues that didn’t use to matter much but now have become important,
to evolve in unforeseen ways, as new technological possibilities and constraints are discovered, etc.

With this larger horizon, you will have Low Time-Preference, i.e. the desire to maximize long-term value, even at the cost of some large (but affordable) short-term inconvenience. You won’t have all the information to plan the whole project for its entire duration. You will have to discover all the particulars as time unfolds, and to continuously be capable of acquiring this information and ready to process it. Today, however, you must make strategic decisions, that will affect the chain of future choices; along the way, you will regularly have to make new such strategic decisions or revise old ones. As you do, you will use the best information you have here and now, that can help you in all this future that you don’t know, yet of which you can expect that it will follow some predictable patterns. This best information is called general principles, and these principles are largely independent from whatever technology is “at hand”.

And then I realized that most humans tend to have High Time-Preference, even in the long-run choice of evolving technologies, whereas Houyhnhnms tend to adopt the principles of Low Time-Preference, and embrace the fact that technologies especially will evolve over the long run, so that you must consider the arc of their future evolution, rather than only their current situation.

Yet, in the long run, the compounded consequences of the general principles you follow, glorious or tragic, will far dominate any short-term costs and benefits of using the closest resources at hand versus a somewhat less obvious and less comfortable solution. The heuristics will be quite different. Unless of course, you somehow arrange for the closest resources at hand at the time you need to pick them to be precisely those that will provide you with the best returns in the long run; but that will require careful thought and preparation—a long-term strategy.

We thus discussed software strategy with Ann, and she inquired about the life arcs of the technologies we were considering. But that is another story…

Chapter 10: Houyhnhnms vs Martians

Ngnghm — Sun, 12 Jun 2016 00:34:38 UT

What did Ngnghm (which I pronounce “Ann”) think of Urbit? Some elements in Ann’s descriptions of Houyhnhnm computing (which I pronounce “Hunam computing”) were remindful of the famous Martian system software stack Urbit: both computing worlds were alien to Human Computing; both had Orthogonal Persistence; and both relied heavily on pure deterministic computations to minimize the amount of data to log in the persistence journal (as contrasted for instance with the amount of data to manipulate to compute and display answers to end-users). What else did Houyhnhnm computing have in common with Martian software? How did it crucially differ? How did they equally or differently resemble Human systems or differ from them? Ann took a long look at Urbit; while she concluded that indeed the three approaches were quite distinct, she also helped me identify the principles underlying their mutual differences and commonalities.

Urbit: The Martian Model

Martians have developed a peculiar operating system, Urbit (docs), the Terran port of which seems to be semi-usable since 2015. At the formal base of it is a pure functional applicative virtual machine, called Nock. On top of it, a pure functional applicative programming language, called Hoon, with an unusual terse syntax and a very barebones static type inferencer. On top of that, an Operating System, call Arvo, that on each server of the network runs by applying the current state of the system to the next event received. The networking layer Ames implements a secure P2P protocol, while the underlying C runtime system, u3, makes it all run on top of a regular Linux machine.

The data model of Nock is that everything is a noun, which can be either a non-negative integer or a pair of nouns. Since the language is pure and applicative (and otherwise without cycle-creating primitives), there can be no cycle in this binary tree of integers. Since the only equality test is extensional, identical subtrees can be merged and the notional tree can be implemented as a Directed Acyclic Graph (DAG).

On top of those, the execution model of Nock is to interpret some of these trees as programs in a variant of combinatory logic, with additional primitives for literals, peano integers, structural equality, and a primitive for tree access indexed by integers. The inefficiency of a naive implementation would be hopeless. However, just like the tree can be optimized into a DAG, the evaluation can be optimized by recognizing that some programs implement known functions, then using a special fast implementation of an equivalent program (which Martians call a jet, by contrast with JIT) rather than interpreting the original programs by following the definitional rules. Recognizing such programs in general could be hard, but in practice Urbit only needs recognize specific instances of such programs — those generated by Hoon and/or present in the standard library.

Therefore, it is the C runtime system u3 that specifies the operational semantics of programs, whereas Nock only specifies their denotational semantics as arbitrary recursive functions. By recognizing and efficiently implementing specific Nock programs and subprograms, u3, like any efficient implementation of the JVM or of any other standardized virtual machine, can decompile VM programs (in this case Nock programs) into an AST and recompile them into machine code using the usual compilation techniques. At that point, like every VM, Nock is just a standardized though extremely awkward representation of programming language semantics (usually all the more awkward since such VM standards are often decided early on, at the point when the least is known about what makes a good representation). Where Urbit distinguishes itself from other VM-based systems, however, is that the semantics of its virtual machine Nock is forever fixed, totally defined, deterministic, and therefore future-proof.

Hoon is a pure functional applicative programming language. Its syntax is terse, where the core syntax is specified using non-alphanumeric characters and digraphs thereof (or equivalent for letter keywords). The syntax allows to write expressions as one liners using parentheses, but it is colloquial to break functions onto many lines where indentation is meaningful; as contrasted with other indentation-sensitive languages, however, the indentation rules are cleverly designed to prevent extraneous indentation to the right as you nest expressions, by deindenting the last, tail position in a function call. Whereas Nock is trivially typed (some would say untyped or dynamically typed), Hoon has a static type system, although quite a primitive one, with a type inferencer that requires more type hints than a language with e.g. Hindley-Milner type inference (such as ML), yet less than one without type inference (such as Java).

Arvo is the operating system of Urbit. The Urbit model is that the state of the system (a noun) encodes a function that will be applied to the next communication event received by the system. If the processing of the event terminates, then the event is transactionally appended to the event journal making it persistent. The value returned specifies the next state of the system and any messages to be sent to the world. Arvo is just the initial state of the system, a universal function that depending on the next event, may do anything, but in particular provides a standard library including anything from basic arithmetics to virtualization of the entire system. The core of Arvo is typically preserved when processing a message, even as the state of the system changes to reflect the computations controlled by the user; as long as this core keeps running (as it should), Arvo remains the operating system of Urbit; but users who insist may upgrade and replace Arvo with a new version, or with another system of their own creation, if they dare.

The events fed into Urbit are generated by the C runtime system u3, to represent console input, incoming network messages, etc. Conversely the messages generated by Urbit are translated by the implementation into console output, outgoing network messages, etc. If processing an event results in an error, if it is interrupted by the impatient user, or if it times out after a minute (for network messages), then u3 just drops the event and doesn’t include it in the event journal. (Of course, if an adversarial network message can time out an Urbit machine for a minute or even a second, that’s probably already a denial of service vulnerability; on the other hand, if the owner, being remote, can’t get his long-running computations going, that’s probably another problem.) A stack trace is generated by u3 when an error occurs, and injected as an event into Arvo in place of the triggering event, that is not persisted. Users can at runtime toggle a flag in the interactive shell Dojo so that it will or won’t display these stack traces.

The networking layer Ames is conceptually a global broadcast network, where network messages are conceptually visible by all other nodes. However, a message is typically addressed to a specific node, using a public key for which only this node has the private key; and other nodes will drop messages they cannot decrypt. Therefore, the C runtime will optimize the sending of a message to route it directly to its destined recipient, as registered on the network. A node in the network is identified by its address, or plot, that can be 8-bit (“galaxy”), 16-bit (“star”), 32-bit (“planet”), 64-bit (“moon”) or 128-bit (“comet”). A comet has for 128-bit address the cryptographic digest of its public key, making it self-authenticating. A moon has its public key signed by the corresponding planet; a planet has its public key signed by the corresponding star, a star has its public key signed by the corresponding galaxy, a galaxy has its public key included in Arvo itself, in a hierarchical system rooted in whoever manages the base Operating System. All communications are thus authenticated by construction. Galaxies, stars, planets and moons are scarce entities, thus constituting “digital real estate” (hence the name plot), that the Urbit curators intend to sell to fund technological development.

One of Urbit’s innovations is to invent mappings from octet to pronounceable three-letter syllables, so that you can pronounce 8-, 16-, 32-, 64- or 128-bit addresses, making them memorable, though not meaningful. So that names with the same address prefix shall not sound the same, a simple bijective mangling function is applied to an address before to extract its pronunciation. This deemphasizes the signing authority behind an identity: the reputation of a person shouldn’t too easily wash onto another just because they used the same registrar; and it’s easier to avoid a “hash collision” in people’s minds by having vaguely related but notably different identities have notably different names. This constitutes an interesting take on Zooko’s Triangle. Actually, care was taken so that the syllables would not be too meaningful (and especially not offensive) in any human language that the author knew of. Non-alphanumerical characters are also given three-letter syllable names, though this time the names were chosen so that there were simple mnemonic rules to remember them (for instance, “wut” for the question mark “?”); this makes it easier to read and learn digraphs (though you might also name them after the corresponding keywords).

Houyhnhnms vs Martians

Most importantly, the Martian’s Urbit is actually available for humans to experiment with (as of May 2016, its authors describe its status as post-alpha and pre-beta). By contrast, no implementation of Houyhnhnm Computing system is available to humans (at the same date), though the ideas may be older. This alone make Urbit superior in one, non-negligible, way. Yet, we will hereon examine it in all the other ways.

Superficially, both Martian and Houyhnhnm Computing provide Orthogonal Persistence. But the way they do it is very different. Martians provide a single mechanism for persistence at a very low-level in their system, separately on each virtual machine in their network. But Houyhnhnms recognize that there is no one size fits all in matter of Persistence: for performance reasons, the highest level of abstraction is desired for the persistence journal; at the same time, transient or loosely-persisted caches are useful for extra indices; and for robustness, a number of replicas are required, with a continuum of potential synchronization policies. Therefore, Houyhnhnms provide a general framework for first-class computations, based on which users may select what to persist under what modalities.

One could imagine ways that Urbit could be modified so its persistence policies would become configurable. For instance, the underlying C runtime u3 could be sensitive to special side-effects, such as messages sent to a magic comet, and modify its evaluation and persistence strategies based on specified configuration. That would mean, however, that most of the interesting work would actually happen inside u3, and not over Nock. What would Nock’s purpose then be? It could remain as an awkward but standardized and future-proof way to represent code and data. However, unless great care is taken, using formal proofs and/or extensive testing, so that the semantics of the Nock code generated indeed implements the actual computations, while indeed being implemented by the underlying system, then at the first bug introduced or “shortcut” taken, the entire Nock VM becomes a sham.

Now, assuming Nock isn’t a complete sham, it remains an obligatory intermediate representation between the computations desired by users and the machine implementations provided by the system. Because Nock is never exactly what the user wants or what the machine provides, this intermediate representation always introduces an impedance mismatch, that is all the more costly as the desired computing interactions are remote from the Nock model.

In an extreme case, one could imagine that u3 would be configured using a Houyhnhnm first-class computation framework. Users would develop their computations at the level of abstraction they desired; and they would dynamically configure u3 to use the desired tower of first-class implementations. At this point, any encoding in terms of Nock could be altogether short-circuited at runtime; and any impedance mismatch introduced by Nock is thus worked around. But then, Nock is purely a hurdle and not at all an asset: all the semantics that users care about is expressed in the Houyhnhnm Computing system; any Nock code generated is just for show, obfuscating the real high-level or low-level computations without bringing anything; and Nock is either a sham, or an expensive tax on the computation framework.

Future-proofing the wrong thing

Both Martians and Houyhnhnms rely heavily on pure deterministic computations to minimize the amount of data to log in the persistence journal to describe issues (as contrasted for instance with the amount of data to manipulate to compute and display answers to end-users). But Martians rely on Nock, and to a lesser extent, Hoon, Arvo, Ames, etc., having a constant deterministic semantics, cast in stone for all users at all time; Houyhnhnms frown at the notion: they consider that constraint as unnecessary as it is onerous. Martians justify the constraint as making it possible to have robust, future-proof persistence. Houyhnhnms contend that this constant semantics doesn’t actually make for robust persistence, and that on the contrary, it prevents future improvements and fixes while encouraging bad practice. Also, Houyhnhms claim that requiring the function to be the same for everyone introduces an extraordinary coordination problem where none existed, without helping any of the real coordination problems that users actually have.

A global consensus on deterministic computation semantics only matters if you want to replay and verify other random people’s computations, i.e. for crypto-currencies with “smart contracts” like Ethereum; but that’s not at all what Urbit is about, and such computation replay in a hostile environment indeed has issues of its own (such as misincentives, or resource abuse) that Urbit doesn’t even try to address. If you only want to replay your own computations (or those of friends), you don’t need a global consensus on a deterministic function; you only need to know what you’re talking about, and write it down.

Houyhnhnms always consider first the interactions that are supposed to be supported by computing activities. In the case of Persistence, Houyhnhnms are each interested in persisting their own code and data. There is no global entity interested in simultaneously looking at the persistence logs of everyone; there is no “collective” will, no magically coordinated knowledge. Each individual Houyhnhnm wants to ensure the persistence of their own data and that data only, or of that entrusted to them personally; and even if they want more, that’s both the only thing they must do and the only thing they can do. Now, they each want the most adequate technology for their purpose, taking costs and benefits into account. If they somehow had to coordinate together to find a common solution, the coordination would be extraordinarily costly and would take a lot of time; they would have to settle on some old technology devised when people knew least, and could never agree on improvements. And if the technology were frozen in time at the beginning, as in Urbit, nothing short of retroactive agreement using a time machine could improve it. If on the contrary each individual is allowed to choose his own persistence solution, then those who can devise improved solutions can use them without having to convince anyone; they can also compete to have their improvements adopted, whereas users compete to not be left behind, until they all adopt the improvements that make sense. In the end, in matters of persistence as of build systems, allowing for divergence creates an incentive towards convergence, reaching better solutions, through competition.

Urbit incorrectly formulates the problem as being a social problem requiring a central solution, when it is actually a technical problem for which a decentralized social arrangement is much better. Persistence doesn’t require anyone to agree with other people on a low-level protocol; it only requires each person to maintain compatibility with their own previous data. To decode the data they persisted, users don’t need a one deterministic function forever, much less one they agree on with everyone else: what they need is to remember the old code and data, and to be able to express the new code (generator) in terms of the old one (to upgrade the code) and able to interpret the old data schema in terms of the new data schema (to upgrade the data). Indeed, even the Urbit whitepaper acknowledges that as far as data above the provided abstraction matters, such schema changes happen (see section 2.0.3 Arvo).

Where Martians get it just as wrong as Humans is in believing that solving one issue (e.g. persistence) at the system level is enough. But onerous local “persistence” of low-level data can actually be counter-productive when what users require is distributed persistence of high-level data at some level of service involving enough replicas yet low-enough latency: local persistence costs a lot, and for no actual benefit to distributed persistence may cause a large increase in latency. The entire point of computing is to support user programs, and solving an issue for some underlying system at a lower-level of abstraction without solving it at the higher-level that the user cares about is actually no solution at all. It can sometimes be part of a solution, but only if (1) the desired property can also be expressed in a composable way so that higher layers of software may benefit from it, and (2) the lower layers don’t impose specific policy choices that will be detrimental to the higher layers of software. And this is what Houyhnhnm systems uniquely enable that Human and Martian systems can’t express because it goes against their paradigm.

Neglect for the Meta-level

The mistake shared by Martians and Humans is to share the approach of neglecting the importance of metaprogramming.

For Humans, this is often out of ignorance and of fear of the unknown: Humans are not usually trained in metaprogramming they don’t understand the importance of it, or its proper usage; they don’t know how to define and use Domain Specific Languages (DSLs). Though their job consists in building machines, they “enjoy” the job security that comes from breaking machines that would replace their current jobs: Mechanized modernity for me, protectionist luddyism for thee.

For Martians, unhappily, there is a conscious decision to eschew metaprogramming. One recent Urbit presentation explicitly declares that DSLs are considered harmful; the rationale given is that the base programming language should have low cognitive overload on entry-level programmers. (Though there again, the very same Urbit authors who claim their programmers shouldn’t do metaprogramming themselves spend most of their time at the meta-level — base-level for thee, meta-level for me.) To Martians, making the system deliberately simpler and less sophisticated makes it easier for people to understand and adopt it. Martians with Hoon commit the same error as the Humans systematically committed with COBOL, or to a lesser degree with Java: they designed languages that superficially allow any random layman (for COBOL) or professional (for Java) or enthusiast (for Hoon) to understand each of the steps of the program, by making those steps very simple, minute and detailed.

But the price for this clarity at the micro-level is to make programs harder to follow at the macro-level. The abstractions that are denied expression are precisely those that would allow to concisely and precisely express the ideas for the actual high-level problem at hand. Every issue therefore become mired with a mass of needless concerns, extraneous details, and administrative overhead, that simultaneously slow down programmers with make-work and blur his understanding of the difficult high-level issues that matter to the user. The concepts that underlie these issues cannot be expressed explicitly, yet programmers need to confront them and possess the knowledge of them implicitly to grasp, develop and debug the high-level program. Instead of having a DSL that automatically handles the high-level concepts, programmers have to manually compile and decompile them as “design patterns”; they must manually track and enforce consistency in the manual compilation, and restore it after every change; there are more, not fewer, things to know: both the DSL and its current manual compilation strategy; and there are more things to keep in mind: both the abstract program and the details of its concrete representation. Therefore, the rejection of abstraction in general, and metaprogramming in particular, prevents unimpeded clear thinking where it is the most sorely needed; it makes the easy harder and the hard nearly impossible, all for the benefit of giving random neophytes a false sense of comfort.

The same mistake goes for all languages that wholly reject syntactic abstraction, or provide a version thereof that is very awkward (like C++ templates or Java compile-time annotations) and/or very limited (such as C macros). It also applies to all programmers and coding styles that frown upon syntactic abstraction (maybe after being bitten by the bad implementations thereof such as above). If you don’t build DSLs, your general purpose language has all the downsides of Turing-equivalence with none of the upsides.

Note however that even though Urbit officially rejects abstraction, Hoon is at its core a functional programming language. Therefore, unlike Humans stuck with COBOL or Java, Martian programmers using Hoon can, if they so choose, leverage this core to develop their own set of high-level composable abstractions; and for that they can reuse or get inspired by all the work done in more advanced functional languages such as Haskell or Lisp. But of course, if that’s the route chosen for further development, in the end, the programmers might better directly adopt Haskell or Lisp and make it persistent rather than use Urbit. If the Urbit persistence model is exactly what they need, they could implement a Hoon backend for their favorite language; if not, they can probably more easily reimplement persistence on their platform based on the Urbit experience than try to evolve Urbit to suit their needs.

Finally, in their common rejection of metaprogramming, both the Human and Martian computing approaches lack first-class notions of meta-levels at runtime. Therefore, all their software is built and distributed as a fixed semantic tower on top of a provided common virtual machine. It’s just that the virtual machine is very different between the Humans and Martians: the Martian VM is oriented towards persistence and determinism, the Human VM is just a low-level portability layer for families of cheap human hardware. As we explained in our chapter 4 and subsequent chapters, this makes for rigid, brittle and expensive development processes.

Impedance Mismatch

One way that Martian is worse than Human as well as Houyhnhnm systems though is that it introduce a virtual machine that makes sense neither at a high-level nor at a low-level, but only introduces an impedance mismatch.

Houyhnhnms clearly understand that the ultimate purpose of computer systems is to support some kind of interaction with some sentient users (be it via a console, via a robot, via a wider institutional process involving other sentient beings, etc.). In other words, the computer system is an enabler, a means, and the computing system is the goal, i.e. the user interactions involving applications. If some computer system makes it harder (than others; than it can; than it used to) to write, use or maintain such applications, then it is (comparatively) failing at its goal.

Humans clearly understand that the ultimate starting point for building the computer software is whatever cost efficient computer hardware is available. At the bottom of the software stack are thin portable abstractions over the hardware, that together constitute the operating system. Every layer you pile on top is costly and goes against the bottom line. If it’s a good intermediate abstraction in the cheapest path from the low-level hardware to the desired high-level application, then it’s part of the cost of doing business. Otherwise it’s just useless overhead.

Unhappily Martians seem to miss both points of view. The Nock virtual machine is justified neither by sophisticated high-level concepts that allow to easily compose and decompose high-level applications, nor by efficient low-level concepts that allow to cost-effectively build software as layers on top of existing hardware. It sits in the middle; and not as a flexible and adaptable piece of scaffolding that helps connect the top to the bottom; but as a fixed detour you have to make along the way, as a bottleneck in your semantic tower, a floor the plan of which was designed by aliens yet compulsorily included in your architecture, that everything underneath has to support and everything above has to rest upon.

Thus, if you want your high-level programs to deal with some low-level concept that isn’t expressible in Nock (hint: it probably won’t be), then you’re in big trouble. One class of issues that Nock itself makes unexpressible yet that any programmer developing non-trivial programs has to care for is resource management: the programmer has no control over how much time or memory operations really take. Yet resources such as speed and memory matter, a lot: “Speed has always been important otherwise one wouldn’t need the computer.” — Seymour Cray. There is a resource model in Urbit, but it’s all defined and hidden in u3, out of sight and out of control of the Martian programmer (unless we lift the lid on u3, at which point Urbiters leave Martian computing to go back to all too Human computing — and certainly not Houyhnhnm computing). At best, you have to consider evaluation of Nock programs as happening in a big fat ugly Monad whereby programs compute functions that chain state implicitly managed by u3.

Of course, you could write a resource-aware language as a slow interpreter on top of Nock, then reimplement it efficiently under u3 as “jets”. Sure you could. That’s exactly what a Houyhnhnm would do if forced to use Urbit. But of course, every time you make a change to your design, you must implement things twice, where you used to do it only once on Human or Houyhnhnm systems: you must implement your logic once as a slow interpreter in Nock; and you must implement it a second time in the Human system in which u3 jets are written. And how do you ensure the equivalence between those two implementations? You can fail to, or lie, and then Urbit is all a sham; or you can spend a lot of time doing it, at which point you wasted a lot of effort, but didn’t win anything as compared to implementing the human code without going through Urbit. What did the detour through Nock buy you? Nothing. Maybe the persistence — but only if persistence with the exact modalities offered by u3 are what you want. If you aim at a different tradeoff between latency, coherency, replication, etc., you lose. And even if perchance you aimed at the exact very same tradeoff, you might be better off duplicating the general persistence design of u3 without keeping any of Nock and Urbit above it.

Oh, if only you had an advanced metaprogramming infrastructure capable of manipulating arbitrary program semantics in a formally correct way! You might then automatically generate both the Nock code in Monadic style and the supporting u3 code for your software, and be confident they are equivalent. And if furthermore your metaprogramming infrastructure could also dynamically replace at runtime an inefficient implementation by a more efficient one that was shown to be equivalent, and for arbitrary programs defined by the users rather than a fixed list of “jets” hardwired in the system, then you could short-circuit any inefficiency and directly call the low-level implementation you generated without ever going through any of the Urbit code. But then, you’d have been using a Houyhnhnm system all along, and Urbit would have been a terrible impediment that you had to deal with and eventually managed to do away with and make irrelevant, at the cost of a non-trivial effort.

Computing Ownership

Martian computing is presented as a technical solution to a social problem, that of allowing individuals to reclaim sovereignty on their computations. That’s a lofty goal, and it would certainly be incorrect to retort that technology can’t change the structure of society. Gunpowder did. The Internet did. But Urbit is not the solution, because it doesn’t address any of the actually difficult issues with ownership and sovereignty; I have discussed some of these issues in a previous speech: Who Controls Your Computer? (And How to make sure it’s you) The only valuable contribution of Urbit in this space is its naming scheme with its clever take on Zooko’s triangle — which is extremely valuable, but a tiny part of Urbit (happily, that also makes it easy to duplicate in your own designs, if you wish). The rest, in the end, is mostly a waste of time as far as ownership goes (but resurrecting the idea of orthogonal persistence is still independently cool, though its Urbit implementation is ultimately backwards).

It could be argued that the Nock VM makes it easier to verify computations, and thus to ascertain that nobody is tampering with your computations (though of course these verifications can’t protect against leakage of information at lower levels of the system). Certainly, Urbit makes this possible, where random Human systems can’t do it. But if Humans wanted to verify computations they could do it much more easily than by using Urbit, using much lighter weight tools. Also, the apparent simplicity of Nock only hides the ridiculous complexity of the layers below (u3) or above (Arvo, Ames). To really verify the computation log, you’d also have to check that packets injected by u3 are consistent with your model of what u3 should be doing, which is extremely complex; and to make sense of the packets, you have to handle all the complexity that was moved into the higher layers of the system. Once again, introducing an intermediate virtual machine that doesn’t naturally appear when factoring an application creates an impedance mismatch and a semantic overhead, for no overall gain.

Not Invented Here

Martian computing comes with its own meta-language for sentient beings to describe computing notions. Since Martians are not Humans, it is completely understandable that the (meta)language they speak is completely different from a Human language, and that there is not exact one-to-one correspondence between Martian and Human concepts. That’s a given.

Still, those who bring Martian technology to Earth fail their public every time they use esoteric terms that make it harder for Humans to understand Martian computing. The excuse given for using esoteric terms is that using terms familiar to Human programmers would come with the wrong connotations, and would lead Humans to an incorrect conceptual map that doesn’t fit the delineations relevant to Martians. But that’s a cop out. Beginners will start with an incorrect map anyway, and experts will have a correct map anyway, whichever terms are chosen. Using familiar terms would speed up learning and would crucially make it easier to pin point the similarities as well as dissimilarities in the two approaches, as you reuse a familiar term then explain how the usage differs.

As someone who tries to translate alien ideas into Human language, I can relate to the difficulty of explaining ideas to people whose paradigm makes it unexpressible. This difficulty was beautifully evidenced and argued by Richard P. Gabriel in his article The Structure of a Programming Language Revolution. But the Urbit authors are not trying to be understood—they are trying their best not to be. That’s a shame, because whatever good and bad ideas exist in their paradigm deserve to be debated, which first requires that they should be understood. Instead they lock themselves into their own autistic planet.

There is a natural tradeoff when designing computing systems, whereby a program can be easy to write, be easy to read, be fast to run, and can even be two of these, but not three. Or at least, there is a “triangle” of a tradeoff (as with Zooko’s triangle), and you can only improve a dimension so much before the other dimensions suffer. But Urbit seems to fail in all these dimensions. Its alien grammar, vocabulary, primitives, paradigm, etc., make it both hard to read and hard to write; and its forced abstraction makes programs slower to run.

If that abstraction came “naturally” when factoring some programs, then it could make writing these programs easier; but the Urbit VM looks very little like what either Humans or machines use for anything, and offers no “killer app” that can’t be implemented more simply. Its applicative functional machine with no cycles exchanging messages is reminiscent of the Erlang VM; but then it’s not obvious what advantages Nock brings for the applications that currently use the Erlang VM, and all too obvious what it costs. It would be much easier to make an Erlang VM persistent or to teach Erlang Ames-style authentication than to teach u3 to do anything useful.

Yet, by having deliberately cut themselves from the rest of the world in so many ways, Urbit programmers find themselves forced to reinvent the world from scratch without being able to reuse much of other people’s code, except at a very high cost both in terms of implementation effort (doing things both in Nock and in u3) and integrity (ensuring the two things are equivalent, or cheating). For instance, the Urbit authors wrote a markdown processor in Hoon, and have a “jet” recognizing it and replacing it by some common Markdown library in C; however the two pieces of code are not bug compatible, so it’s all a lie.

Urbit as a demo

Urbit has none of the support for modular design necessary for programming “in the large”. But the superficial simplicity of Nock makes it suitable as a cool demo of orthogonally persistent system.

Of course, the demo only “works” by sweeping under the rug the difficult issues, to be solved by u3, the metasystem of Urbit; and unlike Nock, u3, where most of the interesting things happen, remains informal in its all-important side-effects, and not actually bound to behave as a faithful implementation of the parts specified by the Nock machine. In other words, the pretense of having fully formalized the state of the system and its state function, and of putting the end-user in control of it, is ultimately a sham, a corruption. The power remains in the opaque and totally unspecified centralized implementation of the metaprogram that implements Nock and issues real-world side-effects.

There is no one-size fits all way to handle all the issues with connection to real-world devices, and with policies that resolve tradeoffs regarding persistence, privacy, latency, efficiency, safety, etc. A centralized implementation for the metaprogram that handles them is not a universal solution. Only a general purpose platform for people to build their own metaprograms can enable them to each solve the issues to their satisfaction. And once you have this platform, you don’t need any of the Urbit operating system, because you already have a Houyhnhnm computing system.

Houyhnhnms have no ill feelings towards either Martians or Humans. They hope that Urbit will be a great success, and demonstrate a lot of cool things and inspire people to adopt orthogonal persistence. However, Houyhnhnms believe that Urbit won’t be able to outgrow being a cool demo unless it embraces a more general purpose metaprogramming architecture.

Chapter 9: Build Systems and Modularity

Ngnghm — Tue, 26 Apr 2016 08:05:06 UT

In my various professional endeavors, I had to deal a lot with build systems: programs like Unix Make, Common Lisp’s ASDF, or Google’s Bazel, but also package managers like rpm, dpkg or Nix, with which developers describe how to build executable software from source files. As the builds grew larger and more complex and had to fit a wider diversity of configurations, I particularly had to deal with configuration scripts to configure the builds, configuration script generation systems, build extensions to abstract over build complexity, and build extension languages to write these build extensions. Since the experience had left me confused, frustrated, and yearning for a better solution, I asked Ngnghm (or “Ann” as I call her) how Houyhnhnms (or “Hunams” as I call them) dealt with these issues. Could they somehow keep their builds always simple, or did they have some elegant solution to deal with large complex builds?

Once again, Ann wasn’t sure what I meant, and I had to explain her at length the kind of situations I had to deal with and the kind of actions I took, before Ann could map them to processes and interactions that happened in Houyhnhnm computing systems. And her conclusion was that while Houyhnhnms computing systems certainly could express large builds, they didn’t possess a “build system” separate and distinguished from their normal development system; rather their “build system” was simply to use their regular development system at the meta-level, while respecting certain common constraints usually enforced on meta-programs.

Division of labor

From what Ann understood, the fundamental interaction supported by what I called a build system was division of labor while developing software: The entire point of it all is that large software endeavors can be broken down in smaller pieces, such that each piece is small enough to fit in a mindful, and can be hacked into shape by a sentient developer. Thus, a complex process way too large to be tackled by any single sentient being in a single programming session, has been reduced to a number of processes simple enough to be addressed by one or more sentients in a large number of programming sessions. Hence, the reach of what sentient beings can achieve through automation has been extended.

Also note this division of labor takes place in a larger process of developing software: unlike many Humans, Houyhnhnms do not think of software as a solution to a “problem”, that comes into existence by a single act of creation ex nihilo; they see developing software as an interactive process of incremental evolution, that addresses on-going “issues” that sentients experience. Sentient developers will thus continually modify, grow and shrink existing software, in ways not completely random yet mostly not predictable — at least, not predictable in advance by those same sentients, who can’t have written the software before they have written it, and have written it as soon as they have written it.

A build system is thus just a part or aspect of a larger interaction. Therefore, a good build system will integrate smoothly with the rest of this interaction; and a better build system will be one that further simplifies the overall interaction, rather than one that displaces complexity from what is somehow counted as “part of the build” to other unaccounted parts of the overall software development process (such as e.g. “configuration”, or “distribution”).

Modularity

The smaller pieces into which software is broken are typically called modules. A notable unit of modularity is often the source file, which groups together related software definitions (we’ll leave aside for now the question of what a file is or should be). Source files can sometimes be subdivided into smaller modules (every definition, every syntactic entity, can be viewed as a software module); and source files can often be grouped into ever larger modules: directories, libraries, components, systems, projects, repositories, distributions, etc. The names and specifics vary depending on the programming languages and software communities that deal with those modules; but generally, a module can be composed of submodules and be part of larger supermodules.

For a given division of software in modules to lead to effective division of labor, modules should be such that most changes to a module should not necessitate changes outside the module, and vice versa. Thus, you should be able to use a module without understanding and having in mind its innards, and you should be able to modify a module without understanding and having in mind its users. In other words, the inside and outside of a module are separated, by some interface, whether it is partially formalized or left wholly informal, that is much smaller and simpler than the complete contents of the module itself, also called its implementation. As long as module developers make no “backward-incompatible” changes to a module’s interface, they shouldn’t have to worry about breaking things for the module users; and as long as module users stick to the properties promised by the module’s interface, they shouldn’t have to worry about module developers breaking things for them.

Of course, sometimes, informal interfaces or erroneous modules lead to divergent expectations between users and developers, with a painful reconciliation or lack thereof. Code may be moved from a module to another; modules may be extended or reduced, created or deleted, split or fused, used no longer or used anew, maintained or abandoned, forked or merged, adapted to new contexts or made into counter-examples. The division of code into modules is not static, cast in stone; it is itself a dynamic aspect of the software development process.

There are four quite distinct interactions to be had with any given module: authoring the module, using it (and its submodules) from another module, integrating it together with other modules into a complete application, or interacting as a non-technical end-user with a complete system that includes the module. In each interaction the sentient being interacting with the system has one of four distinct roles:

Authors write and modify the code (“authors” here is meant in a broad sense, including maintainers and contributors).
Users refer to the code by name while abstracting over its exact contents (“users” here is meant in a narrow sense, including only programmers of modules that use the referred module, not end-users).
Integrators assemble a collection of modules into an overall application, set of applications, virtual machine image, or other deliverable (“integrators” here is meant in a broad sense, including developers who put together their development environment).
End-Users use a software assembly while remaining blissfully unaware of the complex techniques and many modules that had to be mobilized to make their experience possible.

Note that for the purpose of his own applications, as well as for his personal testing needs, a same person may (or may not) be an author of some modules, a user of other another set of modules an integrator of various sets of modules into one or several different assemblies (including his own personal development environments), and an end-user of many applications made of as many sets of modules. It is also possible to be none of these, or one and not the other: A person may delegate his authoring, using, integrating and end-using of software to other more qualified or daring people.

Importantly, one author’s or integrator’s choice of which sets of which modules to use in which specific version does not and cannot bind anyone else’s. Some misguided build systems confuse authors and integrators, and force authors to specify exact versions of other modules their modules will use; but actual integrators will have to deal with security emergencies, bug fixes, missing features and module updates; they cannot wait for every author of every module in every integration to partake in a successful and timely consensus on which coherent set of all modules to use. If anything, any consensus on a coherent set of modules should happen after the set is defined then successfully tested (as with Quicklisp), and not as a prerequisite to even trying to test modules together (as used to be the case with Hackage).

In any case, understanding the distinction between these four roles is essential when designing module systems, build systems, module naming conventions, versioning conventions and version constraints specifications, or any software supposed to deal with modularity: if it fails to serve one or more of the roles, or requires a person having a role to specify information that only people with other roles may know, then it is a deeply dysfunctional design.

Pure Functional Reactive Programming

Given this context, a good build system at heart is a Pure Functional Reactive Programming (FRP) language: its input signals are source files in the version control system and intermediate outputs, and its output signals are intermediate or final build artifacts. Computations from inputs to outputs constitute a build graph: a directed acyclic graph where individual nodes are called actions, and arcs are called dependencies. The signals are called artifacts, and, by extension, the inputs to the action that generate one of them are also called its dependencies.

Actions in a good build system happen without side-effects: no action may interfere with another action, even less so with event sources outside the declared inputs. Actions are thus reproducible. Thence it follows that they can be parallelized and distributed, and their results can be cached and shared. A good build system is thus integrated with the version-control system that manages the changes in source files and the deployment systems that controls the changes in running artifacts. By analogy with content-addressed storage where the name for a file is the digest of its contents, the cache of a good build system can then be said to be source-addressed: the name of a file is a digest of source code sufficient to rebuild the cached value.

For the sake of reproducibility, a good build system must therefore be hermetic: when designating and caching a computation, the system takes into account all inputs necessary and sufficient to reproduce the computation; no source file outside of source-control should be used, even less so an opaque binary file, or worst of all, an external service beyond the control of the people responsible for the build. Thus, when caching results from previous builds, there won’t be false positives whereby some relevant hidden input has changed but the build system fails to notice.

Ideally, all computations should also be deterministic: repeating the same computation on two different computers at different times should yield equivalent result. Ideally that result should be bit for bit identical; any noise that could cause some discrepancy should be eliminated before it happens or normalized away after it does: this noise notably includes timestamps, PRNGs (unless with a controlled deterministic initial state), race conditions, address-based hashing, etc. To make this easier, all (or most) metaprograms should be written in a language where all computations are deterministic by construction. For instance, concurrency if allowed should only be offered through convergent abstractions that guarantee that the final result doesn’t depend on the order of concurrent effects.

Demanding Quality

Computing power is limited, and it doesn’t make sense to rebuild further artifacts from defective pieces known to fail their tests; therefore, computation of artifacts generally follows a pull model where computations happen lazily when demanded by some client reading an output signal, rather than a push model where computations happen eagerly everytime an input signal changes: the model is thus Reactive Demand Programming.

Now, quality assurance processes will pull in new changes as often as affordable; and when they find errors they will automatically use a binary search to locate the initial failure (unless and until issues are fixed). A good build system includes testing, and supports the release cycle of individual modules as well as their integration into larger module aggregates and ultimately entire running production systems.

Because of those cycles are out of sync, the source control system must enable developers to create branches for individual modules, assemble them into branches for larger modules, for entire subsystems and applications, for the complete system. Of course, inasmuch as user feedback from (publicly or privately) released software is required to get a feature exactly right, the length of the OODA loop determining how fast quality can improve in a software development process is the duration from feature request or bug report to user report after use of the released feature, not the distance between two releases. Closer releases can pipeline multiple changes and reduce latency due to the release process itself, but don’t as such make the overall feedback loop shorter. In other words, the release process introduces latency and granularity in the overall development loop that adds up to other factors; the delays it contributes can be reduced, but they will remain positive, and at some point improving the release process as such cannot help much and other parts of the development loop are where slowness needs to be addressed.

Dynamic, higher-order, staged evaluation

By examining the kinds of interactions that a build system is meant to address we can identify some of the features it will sport as a Reactive Programming system and as a programming system in general.

The build graph is the result from evaluating build files, and on many build systems, also from examining source files. These files themselves are signals that change with time; and their build recipes and mutual relationships also change accordingly. Yet the names of the inputs and outputs that the builders care about are often stable across these changes. Therefore, considering the build as a FRP system, it is one with a dynamic flow graph that changes depending on the inputs.

Now, building software happens at many scales, from small programs to entire OS distributions. When the build gets very large and complex, it itself has to be broken down into bits. A bad build system will only handle part of the build and introduce some impedance mismatch with the other build systems necessarily introduced to handle the other parts of the build that it is incapable to handle itself. A good build system will scale along the entire range of possible builds and offer higher order reactive programming where the build information itself in its full generality can be computed as the result of previous build actions. In particular the build system can be “extended” with the full power of a general purpose programming language, and for simplicity and robustness might as well be completely implemented in that same language.

Now, intermediate as well as final build outputs are often programs that get evaluated at a later time, in a different environment that the build system needs to be able to describe: for these programs may need to refer to programming language modules, to entities bound to programming language identifiers or to filenames, where the module names, identifiers and file names themselves might be computed build outputs. Therefore, a build system in its full generality may have to deal with first-class namespaces and environments, to serve as seeds of evaluation in first-class virtual machines. This means that a good build system supports a general form of staged evaluation. And not only can it manipulate quoted programs for later stages of evaluation, but it can also actually evaluate them, each in their own isolated virtualized environment (to preserve purity, determinism, hermeticity, reproducibility, etc.).

Yet, a good build system will automatically handle the usual case for tracking the meaning of identifiers and filenames across these stages of evaluation with minimal administrative overhead on the part of the build developers. In other words, a good build system will manage hygiene in dealing with identifiers across stages of evaluation, notably including when a program is to refer to files created in a different (earlier or later) stage of evaluation! Simple text-substitution engines are not appropriate, and lead to aliasing, complex yet fragile developer-intensive context maintenance, or manual namespace management with various unexamined and unenforced limitations.

Building In The Large

Humans often start growing their build system in the small, so it initially is only designed to work (at a time) only on one module, in one company, out of one source repository. They thus tend not to realize the nature of the larger build of software; they cope with the complexities of a larger build separately in each module by having it use some kind of configuration mechanism: a ./configure script, sometimes itself generated by tools like autoconf, that may use ad hoc techniques to probe the environment for various bits of meta-information. However, these solutions of course utterly fail as systems get built with hundreds or thousands of such individual modules, where each build-time configuration item contributes to a combinatorial explosion of configurations and superlinear increase in the amount of work for each developer, integrator, system administrator or end-user who has to deal with this complexity.

Humans then create completely separate tools for those larger builds: they call these larger builds “software distributions”, and these tools “package managers”. The first modern package managers, like rpm and dpkg, pick a single compile-time configuration and try to guide the end-users through a restricted number of runtime configuration knobs while leaving advanced system administrators able to use each “package”’s full configuration language. But administrators who manage large installations with many machines still have to use tools on top of that to actually deal with configuration, all the while being susceptible to discrepancies manually introduced in each machine’s configuration.

More advanced package managers, like Nix, its variant Guix, or its extension Disnix or NixOps, lets administrators direct the entire build and configuration of one or many machines from one master configuration file, that can import code from other files, all of which can all be kept under source control. Systems like that are probably the way of the future, but the current incarnations still introduce a gap between how people build software in the small and how they build it in the large, with a high price to pay to cross that gap.

Houyhnhnms understand that their build systems have to scale, and can be kept much simpler by adopting the correct paradigm early on: in this case, FRP, etc. Humans have a collection of build systems that don’t interoperate well, that each cost a lot of effort to build from scratch yet ends up under powered in terms of robustness, debuggability and extensibility. Houyhnhnms grow one build system as an extension to their platform, and with much fewer efforts achieve a unified system that inherits from the rest of the platform its robustness, debuggability and extensibility, for free.

Global Namespace

When you start to build in the large, you realize that the names people give to their modules constitute a Global Namespace, or rather, a collection of global namespaces, one per build system: indeed, the whole point of module names is that authors, users and integrators can refer to the same thing without being part of the same project, without one-to-one coordination, but precisely picking modules written largely by other people whom you don’t know, and who don’t know you. Global namespaces enable division of labor on a large scale, where there is no local context for names. Each namespace corresponds to a community that uses that namespace and has its own rules to avoid or resolve any conflicts in naming.

Thus, for instance, when Humans build Java software in the small, they deal with the hierarchical namespace of Java packages; and when they build it in the large, they also deal with the namespace of maven jar files. In Common Lisp, they first deal with the namespace of symbols and packages, then with that of hierarchical modules and files within a system, and finally with the global namespace of ASDF systems. In C, there is the namespace of symbols, and the namespace of libraries you may link against. But in the larger, beyond all these languages’ respective build systems, there is the namespace of packages managed by the “operating system distribution” (whether via rpm, dpkg, nix or otherwise), and the namespace of filesystem paths on each given machine. Note how all these many namespaces often overlap somewhat, with more or less complex partial mappings or hierarchical inclusions between them.

The name of a module carries intent that is supposed to remain as its content varies with time or with configuration. Humans, who like to see things even where there aren’t, tend to look at intent as a platonic ideal state of what the module “should” be doing; but Houyhnhnms, who prefer to see processes, see intent as a Schelling point where the plans of sentient beings meet with the fewest coordination issues, based on which they can divide their own and each other’s labor.

Note that a name, which denotes a fixed intent, may refer to varying content. Indeed, the entire point of having a name is to abstract away from those changes that necessarily occur to adapt to various contingencies as the context changes. Even if a module ever reaches its “perfect” ideal, final, state, no one may ever be fully certain when this has actually happened, for an unexpected future change in its wider usage context may make it imperfect again and it may still have to change due to “bitrot” (the Houyhnhnm name for which would better translate to “fitrot”: the bits themselves don’t rot, though it makes for an amusing paradoxical expression, it is the fitness of those bits that degrades as the context evolves).

Not only will content vary with time, an intent may deliberately name some “virtual” module to be determined from context (such as the choice of a C compiler between gcc, clang or tcc, etc.). In this and other cases, there may be mutually incompatible modules, that cannot be present in a same build at the same time (for instance, glibc, uclibc, musl and klibc are mutually exclusive in a same executable, and so are libgif and libungif). And yet, a “same” larger build may recursively include multiple virtualized system images that are each built while binding some common names to different contents: for instance, as part of a same installation, a boot disk might be generated using the lightweight uclibc whereas the main image would use the full-fledged glibc.

A good build system makes it easy to manage its global namespaces. To remain simple, it will not unnecessarily multiply namespaces; instead it will leverage existing namespaces and their communities, starting with the namespace of identifiers in the FRP language; it will thus hierarchically include other namespaces into its main namespace, and in particular it will adequately map its namespaces to the filesystem or source control namespaces, etc.

Out of DLL Hell

When building in the large, you have to integrate together many modules that each evolve at their own pace. Unhappily, they do not always work well together. Actually, most versions of most modules may not even work well by themselves: they do not behave as they are intended to.

One naive approach to development is to let each module author be his own integrator, and have to release his software with a set of other modules at exact versions known to work together with it. Not only is it more work for each author to release their software, it also leads to multiple copies of the same modules being present on each machine, in tens of subtly different versions. Precious space resources are wasted; important security bug fixes are not propagated in a timely fashion; sometimes some software uses the wrong version of a module; or multiple subtly incompatible versions get to share the same data and corrupt it or do the wrong thing based on it. This is called DLL hell.

Proprietary software, such as Windows or macOS, encourages this hell, because they make any coordination impossible: each author is also an integrator and distributor — a vendor. And vendors have to deal with all the active versions of the operating system, but can’t rely on the end-user either having or not having installed any other software from anyone else. A few vendors might coordinate with each other, but it would be an overall liability where the modest benefits in terms of sharing space would be dwarfed by the costs in terms of having to significantly complexify your release process to synchronize with others, without saving on the overall costs of being a vendor or of being able to promise much additional reliability to users who install any software from a vendor outside the cartel.

Free software, by decoupling the roles of author and integrator, make it possible to solve DLL hell. Authors just don’t have to worry about integration, whereas integrators can indeed gather software from all authors and beat it into shape as required to make it work with the rest of the system. Integrators can also manage the basic safety of the system, and even those remaining proprietary software vendors have less to worry about as most of the system is well-managed.

Houyhnhnms understand that software is better built not just from source code, but from source control. Indeed they reject the Human focus on a static artifact being build from source that can be audited, and instead insist on focusing on the dynamic process of continually building software; and that process includes importing changes, making local changes, merging changes, sending some improvements upstream, and auditing the changes, etc.

They thus realize that whereas a module name denotes a global intent, the value it will be bound to reflects some local context, which is characterized by the set of branches or tags that the integrator follows. Within these branches, each new version committed says “use me, not any previous version”; but then branches are subject to filtering at the levels of various modules and their supermodules: a module that doesn’t pass its test doesn’t get promoted to the certified branch; if a module does pass its tests, then supermodules containing that module can in turn be tested and hopefully certified, etc. Now note that, to solve the DLL hell, modules present in several supermodules must all be chosen at the same version; therefore, all tests must happen based on a coherent snapshot of all modules.

This approach can be seen as a generalization of Google’s official strategy of “building from HEAD”, where what Google calls “HEAD” would be the collection of branches for modules that pass their unit tests. In this more general approach, “HEAD” is just one step in a larger network of branches, where some development branches feed into HEAD when they pass their narrow unit tests, and HEAD feeds into more widely tested integration branches. The testing and vetting process can be fully automated, tests at each level being assumed sufficient to assess the quality of the wider module; actually, from the point of view of the process, manual tests can also be considered part of the automation, just a slow, unreliable part implemented in wetware: from a programmer’s point of view, the user is a peripheral that types when you issue a read request. (P. Williams).

Code Instrumentation

To assess the quality of your tests, an important tool is code coverage: code is instrumented to track which parts are exercised; then after running all tests, you can determine that some parts of the code weren’t tested, and improve your tests to cover more of your code, or to remove or replace redundant tests that slow down the release process or over-constrain the codebase. Some parts of the code might be supposed not to be tested, such as cases that only exist because the type system can’t express that it’s provably impossible, or redundant protections against internal errors and security vulnerabilities; a good development system will let developers express such assumption, and it will, conversely, raise a flag if those parts of the system are exercised during tests.

Sometimes, proofs are used instead of tests; they make it possible to verify a property of the code as applies to an infinite set of possible inputs, rather than just on a small finite number of input situations. Coverage can also be used in the context of proofs, using variants of relevance logic.

Interestingly, a variant of this coverage instrumentation can be used to automatically track which dependencies are used by an action (as vestasys used to do). In other words, dependency tracking is a form of code coverage at the meta-level for the build actions. A developer can thus “just” build his code interactively, and automatically extract from the session log a build script properly annotated with the dependencies actually used. Assuming the developer is using a deterministic dialect (as he should when building software), the instrumentation and tracking can even be done after the fact, with the system redoing parts of the computation in an instrumented context when it is asked to extract a build script.

Instrumenting code on demand also offers solution for debugging. When a build or test error is found, the system can automatically re-run the failing action with a variant of the failing code generated with higher instrumentation settings, possibly omniscient debugging, enabled shortly before the failure. The developer can then easily track down the chain of causes of the failure in his code. Now, omniscient debugging might be too slow or too big for some tests; then the developer may have to start with instrumentation at some coarse granularity, and explicitly zoom in and out to determine with more precision the location of the bug. There again, using deterministic programming languages means that bugs are inherently reproducible, and tracking them can be semi-automated. Separating code and debug information can also make caching more useful, since code once stripped of debugging information is likely to be more stable than with it, and thus a lot of code won’t have to be re-tested just because a line of comment was added.

Finally, “hot-patching” is a form of code instrumentation that is essential to fix critical issues in modules that one doesn’t maintain, or even that one maintains, but have different release cycles than the other modules or integrations that use them. Thus, one will not have to do emergency releases, or worse, forks and uses of forks and branches, followed by complete builds from scratch of entire software distributions to issue emergency fixes to security alerts or blocking problems. While hot-patching is rightfully regarded as a very bad permanent solution, it is wonderful as a readily available venue for temporary solutions: Hot-patching effectively decouples the release cycle of multiple pieces of software—a necessity for large systems. Developers need never be blocked by slow compilation, a release cycle (their own or someone else’s)—or worse, by the difficulty of searching for a perfect solution or negotiating a mutually acceptable one. Just do it! Ecosystems without hot-patching just cannot scale—or end up reinventing it in ugly low-level ways with coarser granularity without language support: at the very worst, special system upgrade tools will reboot the entire machine after upgrading libraries; these tools must first wait for the entire distribution to rebuild, or introduce subtle occasional breakage due to library mismatches; sometimes installations and configurations end up stuck in bad states, and devices get “bricked” after an internal or external system service becomes unavailable.

The Elusive Formalization of Modularity

The entire point of a “module” and its “interface” is to isolate module usage from module authoring, so that users need not know or understand the implementation details, and authors may indeed change those details without users having to know or change their code. This property of modules was once dubbed “information hiding” by some humans, an atrocious name that evokes someone preventing someone else from knowing, when no such thing happens, and the software might be all open source. Modules do not solve a problem of information to show or hide, but of responsibilities to negotiate, of incentives to align. The problem they solve is not logical, but social.

(Interestingly, the social aspect is valid even when there is a single programmer, with no other collaborator, albeit one with a limited mind: to build programs larger than fit in his mind at once, the programmer must still negotiate between the many occurrences of his single “self” across time, each being able to hold but limited amount of information in active memory, so that each module fits with the interfaces it depends on in a single mindful.)

Functional programmers sometimes try to identify modularity with functional abstraction, with linguistic notions of modules (whether first-class or not) as records with existential types, which can indeed internalize the notion of modularity. Object-Oriented programmers may “just” identify “modules” with “classes”, that do as well. But modularity happens with or without internal notions of module and interface in a language; and sometimes modularity happens by working around such internal notions, when they don’t fit the social reality. For instance, in the ubiquitous language C in which most human software “interfaces” are written, there is no internal entity for either an interface or a module; there are “header file” handled externally by a preprocessor, subject to various linguistic and extra-linguistic conventions, and have no internal representation and no language-enforced one-to-one mapping to either individual files, or “libraries”, or the recent notion of “namespace”. Even in OCaml where every file is a “module”, or in Java where every file is a “class”, a “library” is an informal collection of such modules, that has no internal entity; the internal abstraction mechanism is defeated by exporting identifiers for testing, debugging or instrumentation purposes; and inasmuch as a file may conform to an internal “interface” entity, that entity is used once and only once, for that file, and provides no actual meaningful “abstraction”.

Attempts to identify modularity with the use of internal language entities miss the point that modularity is first and foremost meta-linguistic. In any language, the actual “interface” is the semi-formal datum of whatever “identifiers” (or “handles” of any kind) are defined and made visible by a “module”, together with the types or shapes through which these identifiers can be used, and a lot of informal documentation, tests and examples that explain how to use the functionality inside. Beyond any notion of module “internal” to the language, there will also be external instructions for how to download, install, import, use, deploy and configure the “module” outside of the language itself, which may further depend on which “system”, “platform” or “distribution” the developer uses. Internal notions of “modules”, while they might be useful, are never either sufficient nor necessary for actual modularity.

Most of the time, within a given “program”, “application”, “system”, “deployment”, “configuration”, or whatever unit of development a given developer works in, there will be a single module implementing each interface at stake, whether internal or external to the language. Any “abstraction” achieved through modularity, wherein a given interface is actually implemented multiple times by different modules, seldom if ever happens within a program, and instead happens across “programs”: different “programs”, different “versions” of the same “program”, different “deployments” of a same “application”, different “configurations”, used by different people, or by the same developer in different roles at different times, etc. In any given internal state of a program as seen by a language processor or evaluator, the modularity is devoid of such abstraction; modularity takes place externally to the language, between the many states of many programs.

Attempts by some researchers to “measure” the utility or impact of modularity by examining snapshots of source trees of software projects, are thus doomed to bring nonsensical results. The relative utility or disutility of modularizations (ways to organize software into modules) relates to all those variations that happen across source trees in time (as the software evolves) and in space (multiple different uses of the software by same or different people). On the cost side, has effort been saved through division of labor? Has there been much sharing of code, and did it cost less than for each developer to make his own variant? On the benefit side, has modularization enabled software that was not possible or affordable before? Have developers been able to specialize in their tasks and go further and deeper in topics they could not have explored as much? Have new configurations of software been made possible? Has the division in modules inspired new collaborations and created synergies, or have they shut down creativity, diverted energy, and introduced friction?

Answers about the costs and benefits of modularization, as well as of any software development techniques, require economic reasoning about opportunity costs, comparing one universe to alternate potential universes where different techniques are used. And it is a fallacy to claim any accounting of what happened within a single universe can yield any conclusion whatsoever about that economic reality. Also, it is not usually possible to run repeatable experiments. Even “natural experiments” where different teams use different techniques involve different people with different abilities and thousands of confounding factors; if a same team develops the “same” software twice (which few can afford), the two variants are still different software with many different choices, and even the team learns as it develops and doesn’t actually stay the same.

Yet lack of measurable experiments doesn’t mean that informed guesses are impossible. Indeed, many developers can quite predict beforehand that a particular factorization will or won’t, or agree after the fact that it did or didn’t. But those guesses are not objective, and only stay relevant because those who make them have skin in the game. When developers disagree—they may part ways, fork the code (or rewrite it from scratch), and each use different modules and interfaces. Software development has an intrinsically entrepreneurial aspect as well as a community-building aspect. Not every formula works for everyone, and many niche ecosystems will form, grow and wither, based on many choices technical and non-technical.

Reinventing the Wheel and Making it Square

At that point, it may become obvious that what we’ve been calling “a good build system” has all the advanced features of a complete development system, and more: It includes features ranging from a reactive programming core to general purpose extension languages to control support for targets in arbitrary new programming languages or mappings between arbitrary namespaces. It has higher-order structures for control flow and data flow, staged evaluation with hygiene across multiple namespaces. It supports meta-linguistic modularity at various granularities in tight cooperation with the source control system. It has a rich set of instrumentation strategies used while building, testing and deploying programs. It scales from small interactive programs within a process’ memory to large distributed software with a global cache. It encompasses entire software ecosystems, wherein the “same” pieces of software evolve and are used by many people in many different combinations and configurations. How can such a thing even exist?

Human programmers might think that such a system is a practical impossibility, out of reach of even the bestest and largest software companies, that can’t afford the development of such a software Behemoth — and indeed demonstrate as much by their actual choice of build systems. So Human programmers would typically set their expectations lower, whenever they’d start writing a new build system, they would just pick one more of the properties above than the competition possesses, and develop around it a “minimal viable product”, then keep reaching for whichever low-hanging fruits they can reach without any consideration for an end goal. Admittedly, that’s probably the correct approach for the pioneers who don’t yet know where they tread. But for those who come after the pioneers, it’s actually wilful blindness, the refusal to open one’s eyes and to see.

Human programmers thus devise some ad hoc Domain Specific Language (DSL) for build configuration; this language can barely express simple builds, and the underlying execution infrastructure can barely build incrementally, either through timestamps (like Make) or through content digests (like Bazel). Then, Humans painstakingly tuck new ad hoc DSLs and DSL modifications to it to support more advanced features: they add a string substitution preprocessing phase or two to Make, or an extension mechanism or two to Bazel; they call external programs (or reimplement them internally) to extract dependency information from programs in each supported language; etc. However, because each feature is added without identifying the full envelope of the interactions that their system ought to address, each new feature that Humans add introduces its own layer of complexity and badly interacts with past and future features, making further progress exponentially harder as the product progresses. Humans thus tend to reinvent the wheel all the time, and most of the time they make it square — because they are not wheel specialists but in this case build specialists looking for an expedient that happens to be wheelish.

Houyhnhnms have a completely different approach to developing a build system (or any software project). They don’t think of build software as a gadget separate from the rest of the programming system, with its own evaluation infrastructure, its own ad hoc programming languages; rather it is a library for meta-level build activities, written in an appropriate deterministic reactive style, in the same general purpose programming language as the rest of the system. At the same time, most build activities are actually trivial: one module depends on a few other modules, the dependency is obvious from a cursory look at the module’s source; and it all can be compiled without any non-default compiler option. But of course, the activities are only trivial after the build infrastructure was developed, and support for the language properly added.

Thus, Houyhnhnms also start small (there is no other way to start), but early on (or at least some time after pioneering new territories but before going to production on a very large scale) they seek to identify the interactions they want to address, and obtain a big picture of where the software will go. Thus, when they grow their software, they do it in ways that do not accumulate new complexity, but instead improve the overall simplicity of the interaction, by integrating into their automation aspects that were previously dealt with manually.

Also, what counts as “small” or “simple” to Houyhnhnms is not the same as for Humans: as previously discussed, Houyhnhnms do not write “standalone programs”, but natural extensions to their programming platform. Therefore each extension itself is small, but it can reuse and leverage the power of the entire platform. Thus, Houyhnhnms do not need to invent new ad hoc programming languages for configuration and extension, then face the dilemma of either investing a lot in tooling and support using these languages or leave developers having to deal with these aspects of their software without much tooling, if at all. Instead, they refine their “normal” programming languages, and any improvement made while working on the “application” becomes available to programs at large, whereas in the other way around any improvement made available to programs at large becomes available when modifying the application (in this case, a build system).

Consequently, a Houyhnhnm develops a build system by making sure his normal language can express modules in arbitrary target languages, programmable mapping between language identifiers and filesystem objects, pure functional computations, determinism, reactive programming paradigm with push and pull, dynamic execution flow, higher-order functions, virtualization of execution, staged evaluation, hygiene, etc. Not all features may be available to begin with; but growing the system happens by enriching the normal programming language with these features, not by building a new minilanguage from scratch for each combination of feature whereby build programs won’t be able to interoperate when new features are added.

Another advantage of the Houyhnhnm platform approach is that since programming language features are themselves largely modular, they can be reused independently in different combinations and with future replacements of other features. Thus, if you realize you made a design mistake, that you can improve some feature at the cost of some incompatibility, etc., then you don’t have to throw away the entire code base: you can reuse most of the code, and you might even build bridges to keep supporting users of the old code until they migrate to the new one, while sharing a common base that enforces shared invariants. Thus, for instance you might start with a system that does not provide proper hygiene, add hygiene later, and keep the non-hygienic bits running while you migrate your macros to support the new system, and maybe even still afterwards. Each time, writing “the next” build system does not involve starting an even larger behemoth from scratch, but adding a feature to the existing code base.

In conclusion: to Humans, a build system is a complex collection of build utilities disconnected from the rest of the development environment, that can never fully address all build issues. To Houyhnhnms, the build system is just the regular system used at the meta-level, and what we learn by analyzing what a build system should do is the structure of the regular system’s programming language, or what it evolves toward as it matures. Once again, a difference in point of view leads to completely different software architecture, with very different results.

Chapter 8: Inter-process Communication

Ngnghm — Sun, 03 Jan 2016 19:58:27 UT

In our discussion about the difference between Human applications and their Houyhnhnm counterparts (I often pronounce “Houyhnhnm” as “Hunam”), I was intrigued by claims Ngnghm made (I usually call her “Ann”) that communication was much easier between activities of a Houyhnhnm computing system than between applications of a Human computer system. I asked Ann to elaborate on this topic.

It was easy to agree that Human computer systems made communication something much lower level than it could be. But Ann also argued that Humans had very poor algebras and interfaces for users to combine processes. Just what kinds of communication could there even exist besides the ones that already existed on Human computer systems?

Implicit Communication

From my discussions with Ann emerged the notion of communications being explicit or implicit.

In the case of explicit communication, a process specifically names another process, whether an existing one or a new one to be started; it then opens a communication channel with that other process, and proceeds to exchange data. Explicit communication does exactly what the programmers want (or at least say: even Houyhnhnms have no AI strong enough to DWIM); thus programmers control how much complexity they will afford; but it requires tight coupling between the programs (and thus programmers) on all sides of the communication, and is difficult to extend or adapt to suit the dynamic needs of the end-user.

Conversely, communication with other processes can be implicit: something outside some process grabs data from it, and makes it available to some other process. This is the case with copy-pasting, or with piping the standard output of one process into the standard input of another. Implicit communication is controlled by the users of a program rather than by the programmers who write it, and is therefore adapted to their needs. It sometimes require complex support from the programs that partake in it (or, we’ll argue, their meta-programs); but programmers don’t have to worry about programs on the other side, as long as they abide by some general protocol (and keep up with its updates).

Note that implicit vs explicit is a continuum rather than a clear cut distinction: every communication is partly explicit, because it necessarily involves grabbing data that was somehow published by the first process, the publishing of which wasn’t optimized away; and every communication is partly implicit, because it always relies on something in its context to effect that communication, at the meta level (as known from famous paradoxes, no consistent formal system is perfectly self-contained). Another name for this dimension of software design is declarative vs procedural programming: In the declarative approach, programmers describe what is being computed, without specifying how it is going to be computed or how it will be further processed, which will be determined by strategies at the meta level. In the procedural approach, programmers describe the steps of the computation without specifying what is going to be computed, and all the operational semantics remains at the base level.

Houyhnhnms recognize the importance of both aspects of communication, implicit and explicit; meanwhile Humans tend to be blind about the implicit aspect, because they are habitually reluctant to seriously consider anything at the meta level. When Humans tackle implicit communication (and they cannot not do it), they advance reluctantly into a topic about which they are blind; and thus they end up with implicit communication systems simultaneously quite complex and costly for programmers to implement, yet extremely limited in expressive power for the end-user.

The Case of Copy-Paste

The kind of implicit communication most visible to end-users in Human computer systems is copy-paste: applications interact with a graphical interface, and may allow the user to either copy or cut part of a document being displayed; the clipping is then stored in a global clipboard (with space for a single clip). Another application interacting with the graphical interface may then allow the user to paste the clipping currently in the clipboard into its own document. The two programs may know nothing of each other; as long as they properly partake in the protocol, they will have communicated with each other as per the desires of the end-user. Copy-pasting alone provides user-controllable implicit communication between most applications, and is an essential feature in Human computer systems.

Now, on Human computer systems, copy-paste requires every participating application to specially implement large chunks of graphical interface support. Every application then becomes somewhat bloated, having to include large graphical libraries (which in modern systems can happily be shared between applications); but also having to properly initialize them, follow their protocols, abide by the strictures of their event loop, etc. They have to be able to negotiate with the clipboard server the kinds of entities they can copy and paste, and/or convert between what the server supports and what they can directly handle. This architecture where all features are implemented at the same level of abstraction contributes significantly to the complexity of applications; applications are therefore hard to reason about, brittle and insecure. The overall graphical environment will in turn inherit the unreliability of the applications that partake in it. And despite all this complexity, often some application will fail to support copying for some of the information it displays (e.g. an error message); the feature is then sorely missed as the user needs to copy said information by hand, or falls back to some low-level means of information capture such as screen copy (or memory dump, for more advanced developers).

An interesting exception to the rule of the above paragraph is the case of “console” applications: these applications display simple text to a “terminal emulator” straight out of the 1970s, at which point all the output can be copied for further pasting. The terminal emulator thus serves as the meta-program responsible for presentation of the application output, and handling copy-paste. This comes with many limitations: only plain text is supported, not “rich text”, not images; lines longer than the terminal size may or may not be clipped, or have an end-of-line marker or escape character inserted; selecting more than a screenful may be an issue, though you can sometimes work around it by resizing the terminal or by switching to tiny fonts; standard output and error output may be mixed, and interspersed with output from background programs; layout artifacts may be included (such as spaces to end-of-line, or graphic characters that draw boxes in which text is displayed); etc. Still, the principle of a meta-program to handle display already exists in some Human computer systems; its protocol is just limited, baroque and antiquated.

Houyhnhnm computing systems generalize the idea that presenting data to the end-user is the job of a meta-program separate from the activity that displays the data; this meta-program is part of a common extensible platform, rather than of the self-contained “application” that underlies each activity. The display manager will thus manage a shared clipboard; this clipboard may contain more than just one clip; it may contain an arbitrarily long list of clips (like the Emacs kill-ring). Also, clips can include source domain information, so that the user can’t unintentionally paste sensitive data into untrusted activities, or data of an unexpected kind. The platform manages interactive confirmations, rejection notifications, and content filters, that are activated when users copy or paste data. In these aspects as in all others, the platform can be extended by modules and customized by end-users. Other meta-programs beside the display manager can reuse the same infrastructure: they can use their own criteria to select data from a program’s output; they can use the selected data for arbitrary computations, and store the results into arbitrary variables or data structures, not just a common clipboard; they may consult the history of the selected data, or watch the data continuously as it changes, instead of merely extracting its current value. And the base program doesn’t have to do anything about it, except properly organize its data so that the external meta-programs may reliably search that data.

Smoke on Your Pipe and Put That in

As another instance of implicit communication, one of the great successful inventions of (the Human computer system) Unix was the ability to combine programs through pipes: regular “console” applications possess a mode of operation where they take input from an implicit “standard input” and yield output into an implicit “standard output”, with even a separate “error output” to issue error messages and warnings, and additional “inherited” handles to system-managed entities. A process usually does not know and does not care where the input comes from and where the output is going to: it may be connected to a communication stream with another process, to a terminal, or to a file; the parent process setup the connections before the program started to run.

The parent here plays a bit of the role of a meta level, but this role is very limited and only influences the initial program configuration. (Actually, advanced tools may use ptrace, but it is very unwieldy, non-portable, and inefficient, which may explain why it remains uncommon outside its intended use as a debugging tool.) Still, even within this limitation, Unix pipes revolutionized the way software was written, by allowing independent, isolated programs to be composed, and the resulting compositions to be orchestrated into scripts written in some high-level programming language.

Houyhnhnm computing systems very much acknowledge the power of composing programs; but they are not so restricted as with Unix pipes. They enable composition of programs of arbitrary types, with arbitrary numbers of inputs and outputs all of them properly typed according to some high-level object schema, rather than always low-level sequences of bytes. (Note that low-level sequences of bytes do constitute an acceptable type; they are just rarely used in practice except in a few low-level programs.) These typed inputs and outputs all provide natural communication points that can be used to compose programs together.

Unlike the typical parent processes of Human computer systems, the meta-programs of Houyhnhnm computing systems can control more than the initial configuration of applications. They can at all time control the entire behavior of the base-level program being evaluated. In particular, side-effects as well as inputs and outputs are typed and can be injected or captured. Virtualization is a routine operation available to all users, not just an expensive privileged operation reserved to system administrators.

Explicit Communication

There are many obstacles to explicit communication in Human computer systems.

A first obstacle, that we already mentioned in a previous chapter, is the low-level nature of the data that is exchanged with their communication protocols, which constitutes a uniform obstacle to all communications by making them complex, error-prone, and insecure. But these protocols are not low-level only with respect to the data; they are also low-level with respect to communication channels. Human programming languages do not support reflection, and communication channels are selected by passing around handles, low-level first-class objects (typically small integers); this makes it harder to define and enforce invariants as to how channels may or may not be used within a given process: any function having a handle can do anything with it, and handles are often easy to forge; thus you can’t reason about security locally. Houyhnhnm programming languages instead support reflection; thus while they can express the same low-level protocols as above, they tend to (fully) abstract over them and instead expose higher-level protocols, where the channel discipline as well as the data discipline are expressed as part of the types of the functions that exchange data. Communication channel names become regular identifiers of the programming language; the language lets programmers use dynamic binding to control these identifiers; and the usual type-checking and verification techniques apply to enforce protocol invariants not limited to data format.

A second obstacle specific to explicit communication is that to be a legitimate target to such communication, a program must specifically implement a server that listens on a known port, or that registers on a common “data bus”; where this becomes really hard is that to process the connections, the server must either possess some asynchronous event loop, or deal with hard concurrency issues. Unhappily, Human mainstream programming languages have no linguistic support for decentralized event loops, and make concurrency really hard because side-effects in threads can all too easily mess things up. Libraries that implement a centralized event loop are ipso facto incompatible with each other; those that rely on concurrency and a locking discipline are still hard to mix and match, and to avoid deadlocks they require an improbable global consensus on lock order when used by multiple other libraries.

Houyhnhnm programming languages, like the more advanced Human programming languages (including Erlang, Racket, Haskell, OCaml, etc.) both support decentralized event loops (the crucial feature being proper tail calls, and for even more advanced support, first-class delimited continuations), and make it easier by supporting well-typed concurrency abstractions on top of a functional programming core, which is a big improvement. But Houyhnhnm computing systems also make it possible to move these servers completely to a separate meta-program that controls the process you communicate with; thus the base-level process can be written as a simple program, with a very simple semantics, easy to reason about, without any pollution by the server and its complex and possibly incompatible semantics; yet it is possible to tell it to invoke exported functions or otherwise run transactions on its state, by talking to the meta-program that controls it.

A third obstacle specific to explicit communication in Human computer systems is the difficulty of locating and naming one of those target processes available to communicate with. Indeed, inasmuch as communication is explicit, it requires some way to name the party you want to communicate with: a named process (in e.g. Erlang), a numbered port or a named pipe or socket on the current machine (in e.g. Unix), a remote port on a named machine (using TCP/IP), etc. Implicit communication only needs to distinguish between local ports: “standard input”, “standard output”, “file descriptor number 9”, “the graphical display manager” (including its cut-and-paste manager), etc., without having to know what or whom is connected to it on the other side. Reading (or writing to) a file is intermediate between the explicit and implicit: you know the name of the file, but not the identity of who wrote the file (or will read it). Naming a port can also be considered more implicit and less explicit than naming a process.

Now, Human computer systems do not have object persistence, so all their connections and all their names are transient entities that must be reestablished constantly. Human computer systems also have no notion of dynamic environment; there is a static environment, set at the start of a process, but it doesn’t adapt to dynamic changes; or to track dynamic changes, programs can query servers, but then the behavior is either completely unconstrained or highly non-local. You can try to automate this communication, but every program has to handle a vast array of error cases. In any case, local reasoning about dynamic properties is nearly impossible.

Houyhnhnm computing systems have persistence, which means you can give a stable name to a stable activity, and establish a stable connection; they also feature dynamic binding as a language feature that can be used to control the behavior of programs or groups of programs in a structured way. How do they deal with transience, reconnection and unreliability at lower levels of the system? They abstract issues away by introducing a clear distinction between base level and meta level: the base level program is written in an algebra that can assume these problems are solved, with persistent naming and dynamic reconnection both implicitly solved; the meta level program takes care of these issues. Local reasoning on small simple programs (whether at the base or meta level) keeps the overall complexity of the system in check while ensuring robustness.

What’s in a Name?

At the extreme end, opposite to implicit communication, the communication is so explicit that the system knows exactly what’s on the other side of a communication portal. The inter-process communication can then be reduced to a static function call, and the listening function on the other side can often itself be inlined. And in a Houyhnhnm computing system, this may indeed happen, automatically.

Indeed, when it doesn’t change very frequently, whatever is on the other side of any communication channel can be considered locally constant; then, whichever meta-program handles connecting the communicating parties, whether a linker or JIT, can optimize all communication into function calls, and function calls into more specific instructions; it can then reduce all higher-order functions and indirections to efficient loops, until a change in the connection or in the code invalidates these optimizations.

Of course, sometimes the optimization that makes sense goes the other way, transforming function calls into communication with another process: a process on a CPU might delegate computations to a GPU; an embedded device, including a mobile phone, might rather query a server than compute itself, etc. Thus local CPU cycles can be saved whenever cheaper, faster and/or more energy-efficient resources are available. And there again, a more declarative approach allows meta-programs to automatically pick a better strategy adapted to the dynamic program context.

In the end, factoring the code in terms of base-level and meta-level is an essential tool for division of programming labor: The base-level programmer can focus on expressing pertinent aspects of the program semantics; he can write smaller programs that are simpler, easier to reason about, easier to compose; they can be written in a domain-specific language, or, equivalently, in a recognizable subset of his general-purpose language with well-defined patterns of function calls. The meta-level programmer can focus on implementation strategies and optimizations; he has a relatively simple, well-defined framework to prove their correctness, whether formally or informally; and he can focus on the patterns he is interested in, while leveraging the common platform for all other evaluation patterns, instead of having to reinvent the wheel. Thus, whether the source code for some part of an application is modular or monolithic is wholly independent of whether the implementation will be modular or monolithic at runtime. The former is a matter of division of labor and specialization of tasks between programmers at coding-time; the latter is a matter of division of labor and specialization of tasks between hardware components at runtime.

At every level, each programmer can and must use explicit names each implicitly bound to a value, to abstract any process, function or object that belongs to another programmer. By hypothesis, the programmer never knows for sure what the name will be bound to — though often that other programmer may well be the same programmer in a different role at a different time. Yet the overall system in time can always see all the bindings and statically or dynamically reduce them, efficiently combining all parts of a programs into one. Names allow to express fixed intent in an ontology where the extent will change (the extent being the value of a variable, or the text of a function, etc.); they are superfluous from the perspective of a computer system, because for a computer system any name beside memory addresses and offsets is but a costly indirection that is better done away with; names are important precisely because programming is part of a computing system, where the activities of programmers require abstraction and communication across programmers, across time, across projects, etc.

Activity Sandboxing

In Houyhnhnm computing systems, proper sandboxing ensures that activities may only share or access data according to the rules they have declared and that the system owner agreed to. In particular, purported autistic applications are ensured to actually be autistic. Proper sandboxing also means that the system owner isn’t afraid of getting viruses, malware or data leaks via an activity.

Unlike Human computer systems, Houyhnhnm computing systems always run all code in a fully abstract sandbox, as controlled by a user-controlled meta-program. There is no supported way for code to distinguish between “normal” and “virtualized” machines. If the system owner refuses to grant an application access rights to some or all requested resources, the activity has no direct way to determine that the access was denied; instead, whenever it will access the resource, it will be suspended, or get blank data, or fake data from a randomized honeypot, or a notification of network delay, or whatever its meta-level is configured to provide; the system owner ultimately controls all configuration. If the application is well-behaved, many unauthorized accesses may be optimized away; but even if it’s not, it has no reliable way of telling whether it’s running “for real”, i.e. whether it’s connected to some actual resource.

Allowing code to make the difference would be a huge security failure; and any time a monitor in a production system recognizes the attempt by a process to probe its environment or otherwise break the abstraction, a serious security violation is flagged; upon detection, the process and all its associated processes are suspended, up to the next suitably secure meta-level; also the incident is logged, an investigation is triggered, and the responsible software vendor is questioned. — Unless of course, the people responsible for the break in attempt are the system’s owners themselves, or penetration testers they have hired to assess and improve their security, which is a recommended practice among anyone hosting computations controlling any important actual resources.

Note that proper sandboxing at heart has nothing whatsoever to do with having “kernel” support for “containers” or hardware-accelerated “virtual machines”; rather it is all about providing full abstraction, i.e. abstractions that don’t leak. For instance, a user-interface should make it impossible to break the abstraction without intentionally going to the meta-level. You shouldn’t be able to accidentally copy and paste potentially sensitive information from one sandbox to the next; instead, copy and pasting from one sandbox to another should require extra confirmation before any information is transferred; the prompt is managed by a common meta-level below the sandboxes, and provides the user with context about which are the sandboxes and what is the considered content; that the user may thus usefully confirm based on useful information — or he may mark this context or a larger context as authorized for copying and pasting without further confirmations.

Protocols as Meta-level Business

Houyhnhnm computing systems resolutely adopt the notion that some tasks are generally the responsibility of a series of (meta)programs that are separate from the ones computing the results; i.e. presenting computation results, combining communicating processes, choosing an implementation strategy for a declarative program, etc. Factoring out the interface at the meta level means that each level can be kept conceptually simple. The system remains “reasonable”, that is susceptible to be reasoned about. It’s enough to assess the security properties only once, abstracting away the specifics of programs using the features.

Thus, in Houyhnhnm computing systems, unlike in Human computer systems, the robustness and timeliness of the system don’t have to depend on every application partaking in the protocol being well-behaved, nor on every programmer working on any such application being steadfast at all times and never making any mistake. There can be no bugs in all the lines of code that the programmers don’t have to write anymore. And updating or extending the protocol is much easier, since it only involves updating or extending the according meta-programs, without having to touch the base-level applications (unless they want to explicitly take advantage of new features).

Moving features from base-level applications to meta-level layers can be justified with all the same arguments why “preemptive multitasking” beats “cooperative multitasking” as an interface offered to programmers: sentient programmers are intrinsically unreliable, any kind of “cooperation” that relies on manually written code to abide by non-trivial invariants in all cases will result in massive system instability. At the same time, “cooperative” beats “uncooperative” as far as implementation is concerned; cooperation is (and actually must be) used under the hood to preserve any non-trivial system invariants — but it can be used automatically and correctly, through a meta-program’s code-generator.

Embracing or Fearing The Meta

In Human computer systems, there are few features implemented as meta-programs; software libraries are strictly runtime entities, and programmers must manually follow the “design patterns” required to properly implement the protocols supported by those libraries. In Houyhnhnm computing systems, there are plenty of meta-programs; and though they may have a runtime component like libraries, they most importantly include compile-time and/or link-time entities; and they ensure that all runtime code strictly follows all supported protocols by construction. The meta-programs, that display, select, extract, or watch data, use introspection of the program’s state, types and variables; and for reasons of efficiency, they do not re-do it constantly at runtime; yet they do keep enough information available at runtime to recompute whatever is needed when programs change.

Changes in these meta-programs may involve recompiling or otherwise reprocessing every base program that uses them. This meta-processing is deeply incompatible with the traditional Human notion of “binary-only” or “closed source” distribution of software; but that notion doesn’t exist for Houyhnhnms: Houyhnhnms understand that metaprogramming requires free availability of sources. For similar reasons, Humans who sell proprietary software see a platform based on meta-programming as the Antichrist.

A program that comes without source is crippled in terms of functionality; it is also untrusted, to be run in a specially paranoid (and hence slower) sandbox. Houyhnhnms may tolerate interactive documents that behave that way; they may accept black box services where they only care about the end-result of one-time interactions, at the periphery of their computing systems. But they have little patience for integrating a black-box program into critical parts of their regular platforms; they won’t put it in a position where it has access to critical information, or make it part of any process the failure of which could threaten the integrity of the system. If they care about what a black-box program does, they will spend enough time to reimplement it openly.

Chapter 7: Platforms not Applications

Ngnghm — Sat, 26 Dec 2015 03:33:44 UT

My previous discussion with Ngnghm (or Ann as I call her) left me baffled: I could somehow understand that Houyhnhnms don’t have the concept of an Operating System Kernel (note that I pronounce “Houyhnhnm” “Hunam”); and I could vaguely guess how each of the many aspects of a Human kernel could correspond to a family of software patterns in a Houyhnhnm computing system, at various levels of abstractions. But while I could visualize these patterns individually, it was less clear to me what the big picture was when these smaller compile-time, link-time and runtime abstractions were put together. So I decided to approach their software architecture from the other end: what do end-user applications look like in Houyhnhnm computing systems?

I was baffled again, but not surprised anymore, to find that Houyhnhnms don’t have a notion of application. Granted, there are simple cases where Human applications have direct counterparts in Houyhnhnm computing systems. But in the general case, Houyhnhnms don’t think in terms of standalone applications; they think in terms of platforms that they extend with new functionality.

Autistic Applications

Ann was starting to get familiar with Human computer systems, and the few end-user applications that he was using daily. She noticed that a certain class of applications was quite reminiscent of software that existed in Houyhnhnm computing systems, at least superficially: self-contained end-user applications, such as games, interactive art, audiovisual performances, showroom displays, news and other writings, etc. These applications had in common that they are made to be explored by the user but not modified in any significant way; they mostly didn’t communicate much, if at all, with any other application in any way that the end-user cared to control; they had no significant input and no output beside the user experience. I dubbed the concept autistic applications. But when Ann tried to translate the Houyhnhnm expression for the concept, it sounded more like interactive documents.

In any case, these things look pretty much the same in Houyhnhnm computing systems and Human computer systems: You somehow get ahold of the software; installing it automatically installs its dependencies, if any; you run it in a sandbox (at least Houyhnhnms do); and you interact with it. It doesn’t matter too much what the program does (if anything), precisely because the information flow is essentially one way, from the application to the user.

Still, there were a few subtle points where even these autistic applications in Human computer systems differ from interactive documents in Houyhnhnm computing systems. For instance, in a Houyhnhnm computing system, you can always copy and paste text and pictures and sounds, search for words in registered dictionaries, or otherwise manipulate the application output; these do not require the application developers having to do anything to enable such features. But a more striking difference is that all Houyhnhnm activities inherit from the system its orthogonal persistence. You can thus always interrupt the application and save and restore its state at any point in time, except where explicitly not desired (e.g. in the middle of a transaction). Then you can go back in time and replay (and in the case of videos or music, go forward in time), according to a protocol that is uniform across applications; and not only is it no additional burden on application programmers, it is something they can’t get subtly wrong, and that users can thus casually rely upon. There is never any loss of any session state; there are no disappearing tabs or windows, no games where you can’t both pause and save your game; no messages you have to enter twice because they were cleared between two page loads or browser crash and restart, or that reappear because the clearing failed to be recorded.

Of course, in a Houyhnhnm computing system, interactive documents (like any other activity), even when they require interaction with a remote service, are always able to save and restore the client-side state from previous user interactions; however that does not entail being able to save and restore any server-side state, at least not without support from the server. And while the system typically makes it easy for the server developers to provide that support if they want, there are many reasons why they might not want to do it, including cost and confidentiality. Conversely, for reasons of privacy, a user might want to replay a previous session without telling the remote server. Also, for regression testing or for debugging their applications, developers may want to replay parts of the server side interactions without affecting the users. All these behaviors are expressible in Houyhnhnm computing systems: you can specify the scope and context in which you replay some computation, within the resources that you control.

Typical Applications

Now, most applications are not autistic; they do involve exchanging data with other applications: using data produced by other applications, and producing data that will be used by other applications. In other words, the information processes they partake in may directly involve other automated programs; they do not require a Sentient being’s brain (Human or Houyhnhnm) as an exclusive intermediate between their processing and further automated processing; the sentient being doesn’t have to recreate the entirety of the next program’s input based on what it sees or remembers of the previous program’s output. And there we see that even “autistic applications” are not “autistic processes”: An autistic application does not communicate with other automated programs but does interact with sentient users; its implementation might also interact with other programs below the abstraction provided to the user, though that’s mostly invisible to the user. An “autistic process” that communicates with no other process whatsoever, not even those in a sentient being’s brain, can and will be wholly optimized away.

Ann then explained that the situation differs sharply between Human and Houyhnhnm systems regarding all these typical, non-autistic, applications — to the point that Houyhnhnms don’t really have a notion of application. For technical reasons with historical roots in marketing, Human computer systems tend to organize software into neatly separated, standalone, black-box “applications”; communication between different applications is very difficult, and must be explicitly handled by each of these applications; every application must include an implementation of all these modes of communication it will partake in. Instead, Houyhnhnm computing systems consider such communication the heart of the system, and make it comparatively easy; they do not usually have self-contained “applications”; they start from a common platform that handles all the communication; and they extend this platform to handle new kinds of situations, until they include all the interesting situations that the “application” would have covered.

A first obstacle to inter-application communication in Human computer systems, is that the only common abstractions are very low-level, in terms of arrays of bytes. Any higher-level objects have to be encoded into sequences of bytes, shipped across costly runtime virtual process boundaries, then decoded back into objects on the other side by a matching algorithm. Applications thus have to agree on complex, expensive, bug-prone yet inexpressive low-level communication protocols that are big security liabilities. Having to deal with such protocols is a huge barrier to entry that explains why few programmers endeavour to try it. A lot of this work can be completely automated using type-directed code-generation; and the better Human systems do it to a point (see Protocol Buffers, Cap’n’Proto, piqi, etc.); but the integration with types of actual programming languages remains generally lackluster. What types can be used for generally shareable data remain very limited and inexpressive, whereas whatever types they can use for manipulating data within a given program remain generally oblivious of any sharing constraints, ownership rights, access restrictions, etc.

In Houyhnhnm computing systems, communication of objects is handled by the system at the highest level of abstraction possible: that of whichever types and checks are being used to define and validate these objects. Low-level encoding and decoding can be eschewed altogether for linear objects where both processes trust each other with respect to representation invariants; it can sometimes be reduced to mere checking when the trust is incomplete; and where encoding or checking is actually required, it is automatically extracted based on type information available either at compile-time or at runtime. The programming language types are the communication types, and if foreign languages need to communicate with each other, it’s a regular matter of FFI (Foreign Function Interface) that you need to solve anyway, and might as well solve once and for all, rather than have each application invent its own bad incompatible partial solution.

A second obstacle to inter-application communication in Human computer systems is that they have very poor algebras and interfaces for users to combine processes. For most users, sharing data between applications requires one of two things: selecting and copying (or cutting) data from one application using a mouse, then pasting it into another application; or having the application save or export a file to a local disk, then opening or importing that file in another application (with “interesting” consequences when two applications try to modify it at the same time). Developers can do better, but there’s a large discontinuity between the skills required to merely use the system, and the skills required to do even the simplest things as you program the system. Modern Human computer systems tend to allow for an intermediate layer between the two, “scripting”, with Unix shells and their pipes, or the notably more modern PowerShell on Windows. Scripting lowers the barrier to building applications, and when using “client” utilities and libraries, allows programmers to share data beyond copy-pasting and files; but it still remains quite complex to use, and often brittle and limited in expressiveness, because it does not directly partake in either of the programs’ invariant enforcement and atomic transactions (though a few applications offer a suitable transactional interface).

Houyhnhnm Platforms

Houyhnhnm computing systems are based on the premise of small modular entities that each do one thing well; and these entities can be combined inside a common platform that does its best to reduce the discontinuity between using and programming. To Houyhnhnms, there is no difference between using and programming; if anything, the difference between a programmer and a user, is that the programmer knows there is no difference between using and programming. Certainly, there is a continuum of proficiency and knowledge amongst users; but there is generally no large barrier to overcome in order for users to generalize and automate as a script whatever computations they know how to achieve interactively; and there isn’t a large amount of boilerplate required to write the least program, as there is in all Human programming languages except “scripting languages”. Houyhnhnm platforms are built around a high-level programming language accessible to the user; therefore communication happens directly using objects in the system language so no serialization or deserialization into low-level bit sequences is required (or if it is, for the sake of network communication, it can be automated); and the system language is available to name entities, combine and apply programs.

A few Human computer systems have historically followed this model: Smalltalk workstations (from Xerox), Lisp Machines (from Xerox, MIT, Symbolics, LMI or TI), Hypercard (on old Apple Macintosh’es); to a point, HP calculators or Mathematica. But perhaps the most successful such platform to date is GNU Emacs: It is largely written as a set of modules in a “scripting language”, Emacs Lisp. Entities defined in a module can be freely used in another one, and data is directly exchanged without going through any communication or translation layer. Emacs Lisp is antiquated, more so than Smalltalk or Lisp Machine Lisp ever were, and its data structures are heavily biased towards text editing; and yet it remains widely used and actively developed, because in many ways it’s still far ahead of any competition despite its limitations.

In a Houyhnhnm computing system, programmers do not write standalone applications in non-autistic cases; instead, they write new modules that extend the capabilities of the platform. Often, a new module will extend the system to handle new entities. As long as these entities implement common interfaces, they can be used along all previously known entities by all existing modules that use these interfaces. For instance, a new picture compression format is automatically usable by each and every function that uses pictures throughout the system; a common extensible picture editor can be used on all pictures anywhere on the system; a common extensible text editor can handle any kind of writable text in the system; etc. At all times, each of these modules, including all common editors, will include all the user’s customizations; this makes writing customizations much more worthwhile than if separate customizations had to be written for each application, each in its own language with its own learning curve, as is the case in Human computer systems.

A new module may also define new interfaces, and how they apply to existing kinds of entities. There is of course a problem when two modules that don’t know each other extend the system by one adding new kinds of entities and the other defining new kinds of interfaces, the combination leading to new cases that are not handled. Houyhnhnm systems are not magic and can’t generate handlers for those cases out of thin air: a further module may define how to handle these new combinations; or a suitable generic fallback may have been provided with the new interface; or lacking any of the above, the system will fail and drop to its metasystem, that will handle the error. In the end, it’s still the job of some programmer to ensure that the overall system works suitably in the legitimate cases that it will actually encounter. These issues exist in Human and Houyhnhnm systems alike — the only difference is that Human computer systems are so difficult to extend that programmers seldom reach the point when they can confront these problems, whereas Houyhnhnm computing system eliminate enough of the artificial problems in extending the system that users are more often confronted with these extension-related issues.

Different Shapes

Because of the high barrier to communication between applications in Human computer systems, these applications tend to grow into big hulking pieces of software that try to do everything — yet can’t. Indeed, even a picture editor will need to edit text to overlay on pictures, to email the pictures, to browse the filesystem looking for pictures to edit, to search pictures by date or by location, etc. It needs to be extensible to accept new file formats, new color schemes, new filters, new extraction tools, new analyses, new generation techniques, new scanning sources, new social networks on which to publish pictures, etc. Soon, it becomes a platform of its own, its own extension API, its own scripting language, its own plugin ecosystem, its own configuration system, its own sandboxing infrastructure. Every successful application grows this way, until it does many of the same things as all the other applications, all of them badly, except those within its own application core.

In a Houyhnhnm computing system, a picture editor will handle picture editing, and picture editing only — and do it well. It will delegate sending email, browsing the filesystem, searching for pictures, etc., to suitable other modules of the common platform. Instead of extensions being available for a single application, they will be available to all software. Thus, whereas Human computer systems feature one unwieldy file selector for each application, Houyhnhnm computing systems instead will have a single file selection service for the entire platform. All the improvements ever made to file selection will be available to all activities instead of only a single application: preview of contents, browsing history, restriction and search by type or by many criteria beside filename hierarchy, relevance to context, selection or tagging of multiple files instead of one at once, automatic refresh of search results, generation of content on demand, etc. Security will notably be improved by each component only having access to the capabilities it needs, containing any security breach by construction.

Extension Languages

Many Human application developers eventually realize that the growing set of predefined usage patterns they develop over time can never cover all the cases required by all potential users. So they eventually invent their own configuration and extension language, so that users can define their own usage patterns. But most application developers are no programming language specialists; even when they are, being pressured by the application development deadlines, they just don’t possess the resources to implement more than the strict minimum necessary for a programming language; and they never planned in advance for adding such a language, so it doesn’t fit well in their large existing code base. Therefore they usually end up with a very badly designed language, very inefficiently implemented, and no tooling to support using it besides print-debugging at the end of a long edit-compile-test cycle. That resulting language can be very good at the few initial predefined operations, and passable when using some limited usage patterns, but is consistently bad at everything else. Yet it costs a lot to develop, and even more to do without.

In contrast, Houyhnhnms will use their common platform to configure and extend all software. The platform comes with a variety of programming languages each designed by the best programming language designers; it provides an efficient implementation framework, great tooling, all the programming paradigms users may desire. There are also many ways for developers to control the expressiveness of configuration languages: domain-specific languages, type systems, contracts, etc. Not only do such expressiveness restrictions make it easier for domain experts to precisely express what an application requires, in the terms that best make sense for the domain (here using the informal meaning of “application”, not the Human computer system notion); they also enable domain-specific meta-programming: since the configurations follow a given pattern or can be otherwise normalized to objects of a given type, various kinds or guarantees, queries and optimizations may apply.

Programming Incentives

More generally, considering the larger computing system that includes the sentient programmers, Human computer systems display a singular failure of division of labour and specialization of tasks. Developer talent is stretched thin, as the same tasks are done over and over, once per application, almost never well, by developers who are seldom specialists in those tasks. Meanwhile, those few developers who are good at a task cannot focus on doing it in a way that will benefit everyone, but must instead improve a single application at a time. And because hiring is application-based rather than specialty-based, even specialists are seldom funded to do what they are good at, instead being paid to badly write yet another implementation of a feature at which they are mediocre, for whichever application they were hired about. Cultivating a platform is an afterthought to application growth; which platforms happen to succeed depends not at all on its good design, but on a package deal with other aspects of which application will grow biggest. As a result, most successful application extension platforms start as some amateur’s quick and dirty design under pressure, with a requirement of matching the application’s existing API; platforms then have to forever deal with backward compatibility with a bad and skewed design. Since there is no common platform, developers must relearn the badly designed ad hoc partial platform of each application before they can be productive; this increases the barrier to entry to coding, entrenches the market fragmentation, and adds up to a huge deadweight loss for society as a whole.

In Houyhnhnm computing systems, experts of any given domain can focus on their domain of expertise, contracting their services to those who require improvement for their applications. Experts don’t need to restart their work from scratch for every application, but need only do it once per platform. And there are only a few worthwhile platforms, and they each are designed by experts at platform design rather than by some random application developer. Where a number of expertises must be integrated together toward an “application”, choosing and cultivating a well-designed platform for software growth is not an afterthought but a prerequisite for the project manager. Without artificial barriers to development, the total amount of effort expanded on each feature is much lower in Houyhnhnm computing systems than in Human computer systems, while the average domain expertise of those who implement each feature is much higher. Houyhnhnms thus achieve better software quality at the cost of a lower quantity of development efforts than Humans. They don’t have an “app economy”, but they do have active markets where producers sell interactive documents and platform extensions, either as services or products.

The economic structure of software development as well as its technical architecture is thus crucially affected by this simple change of point of view, from computer systems to computing systems.

Chapter 6: Kernel Is As Kernel Does

Ngnghm — Sun, 29 Nov 2015 04:34:45 UT

I admitted to Ngnghm (whom I call “Ann”) that I was perplexed by Houyhnhnm computing systems (I pronounce “Houyhnhnm” like “Hunam”). To better understand them, I wanted to know what their kernels, libraries and applications looked like. There again, he surprised me by having no notion of what I called kernel or application: the way Houyhnhnm systems are architected leads to widely different concepts; and for the most part there isn’t a direct one-to-one correspondance between our notions and theirs. And so I endeavored to discover what in Houyhnhnm computing systems replaces what in Human computer systems is embodied by the operating system kernel.

Kernels

“What does an Operating System Kernel look like in a Houyhnhnm computing system?” I asked Ann. She wasn’t sure what I was calling either Operating System or Kernel.

I explained that in a Human computer system, the kernel is a piece of software that handled the hardware resources, and provided some uniform abstractions that isolated users from the hardware details that varied from machine to machine and across time. All the rest of the system is expressed in terms of those abstractions; modern computer hardware ensures through runtime checks that all programs beside the kernel run in a “user mode” that only sees these abstractions; the kernel alone runs in a “kernel mode” that gives it direct access to the hardware. The kernel can also use this hardware support to provide low-level isolation between “processes”: it allows multiple user programs to run at the same time while ensuring that none may interfere with other programs except through said abstractions.

Ann however had trouble distinguishing the kernel from any other program based on my description. The notion of kernel, like most concepts of Human computer systems, was too artifact-oriented and didn’t fit the grid of interaction-oriented Houyhnhnm computing systems. “What is it that a kernel does?” Ann asked me; when she’d know that, she could tell me how their systems implement analogous interactions. And then I was at loss to distinguish exactly what kinds of interaction a kernel does that other pieces of software don’t.

Resource Management

The most obvious thing a kernel does is that it manages and multiplexes resources: it takes some resources, such as CPU time, core memory, disk space, console access, network connections, etc.; and it makes available to multiple programs, either one after the other, or at the same time. It ensures that each program can use all the resources it needs without programs stepping on each other’s toes and corrupting each other’s state.

However, resource management cannot define what a kernel is, since plenty of other components of a computer system also manage resources: All but the simplest programs contain a memory allocator. A database server, or any kind of server, really, manages some kind of records. A sound mixer, a 3D scene renderer, a Window system, or anything worth of being called a system at all, allow multiple entities to coexist, interact with each other, and be perceived, modified, accessed, experienced, by the system’s user.

Houyhnhnms recognize that resource management is an infinitely varied topic; this topic cannot possibly be reduced to a fixed set of resources, but is an inherent aspect of most programs. When they need to explicitly deal with this aspect, Houyhnhnms make it an explicit part of the rich algebra they use to express programs. The simplest idiom for that is to use a proxy, a handle, or some indirect way of naming a resource; programs that use the resource may only go through that handle, while only the program that manages the resource manipulates the underlying details. More advanced idioms include using some variant of what we call linear logic; on some systems, linear logic can also be painfully emulated using monads.

Access Control

A kernel also provides some kind of access control to the resources it exposes: for instance, you have to login as a user to access the system; then you can only access those files owned by said user, or explicitly shared by other users.

But there again so does any system-managed access to resources. Moreover, whichever access control a Human Computer System kernel provides is often so primitive that it’s both too slow to be in any code’s critical path and yet too coarse and too inexpressive to match any but the most primitive service’s intended access policies. Therefore, every program must either reimplement its own access control from scratch or become a big security liability whenever it’s exposed to a hostile environment.

Houyhnhnms recognize that access control too is not a fixed issue that can be solved once and for all for all programs using a pre-defined one-size-fits-all policy. It can even less be solved using a policy that’s so simple that it maps directly to a bitmask and some simple hardware operations. Instead, they also prefer to provide explicit primitives in their programming language to let programmers define the access abstractions that fit their purposes; in doing so, they can use common libraries to express all the usual security paradigms and whichever variant or combination the users will actually need; and these primitives fit into the algebra they use to manage resources above.

Abstraction

A Human Computer System kernel (usually) provides abstraction: all programs in the system, beside the kernel itself, are user programs; their computations are restricted to only be combinations of the primitives provided by the system, as implemented at runtime by the kernel. It is not possible for user programs to subvert the system and directly access the resources on top of which the system itself is built (or if it is, it’s a major security issue to be addressed as the highest emergency). The system thus offers an abstraction of the underlying hardware; and this abstraction offers portability of programs to various hardware platforms, as well as security when these programs interact with each other. More generally, abstraction brings the ability to reason about programs independently from the specificities of the hardware on which they will run (“abstracting away” those specificities). And this in turn enables modularity in software and hardware development: the division of labor that makes it possible to master the otherwise unsurmountable complexity of a complete computer system.

Now and again, abstraction is also what any library or service interface provides, and what every programming language enforces: by using the otherwise opaque API of the library or service, programmers do not have to worry about how things are implemented underneath, as long as they follow the documented protocol. And by using a programming language that supports it, they can rely on the compiler-generated code always following the documented protocol, and they don’t even have to worry about following it manually: as long as the program compiles and runs, it can’t go wrong (with respect to the compiler-enforced protocols). Abstraction in general is thus not a defining activity of an operating system kernel either; and neither is abstraction of any of the specific resources it manages, that are often better abstracted by further libraries or languages.

Houyhnhnms not only reckon that abstraction is an essential mechanism for expressing programs; Houyhnhnms also acknowledge that abstraction is not reserved to a one single “system” abstraction to be shared by all programmers in all circumstances. Rather, abstraction is an essential tool for the division of mental labor, and is available to all programmers who want to define the limits between their respective responsibilities. The program algebras used by Houyhnhnms thus have a notion of first-class programming system (which includes programming language as well as programming runtime), that programmers can freely define as well as use, in every case providing abstraction. Since they are first-class, they can also be parametrized and made from smaller blocks.

Note, however, that when parametrizing programming systems, it is important to be able to express full abstraction, whereby programs are prevented from examining the data being abstracted over. A fully abstracted value may only be used according to the interface specified by the abstract variable type; thus, unless that abstract type explicitly includes some interface to inspect the actual value’s type or to deconstruct its value according to specific match patterns, the client code won’t be able to do any of these, even if in the end the actual value provided happens to be of a known type for which such operations are available. A dynamic language may implement it through opaque runtime proxies; a static language may provide this feature through static typing; some languages, just like computability logic, may distinguish between “blind” quantifiers and regular “parallel” or “choice” quantifiers. In any case, the fragment of code in which a full abstraction holds is prevented from peering inside the abstraction, even if the language otherwise provides reflection mechanisms that can see through regular abstractions. Of course, when turtling down the tower of implementations, what is a completely opaque full abstraction at a higher level may be a fully transparent partial abstraction at a lower level; that’s perfectly fine — the lower-level, which cannot be accessed or modified without proper permissions, is indeed responsible for properly implementing the invariants of the higher-level.

Enforced and Unenforced Abstractions

There is one thing, though, that kernels do in Human computer systems that other pieces software mostly don’t do — because they mostly can’t do it, lacking system support: and that’s enforcing full abstraction. Indeed, in a Human computer system, typically, only the operating system’s invariants are enforced. They are enforced by the kernel, and no other piece of software is allowed to enforce anything. If a process runs as a given user, say joe, then any other process running as joe can do pretty much what it wants to it, mucking around with its files, maybe even its memory, by attaching with a debugger interface, etc. If a user is allowed to debug things he runs at all (and he probably should be allowed), then all processes running as that user are allowed, too. Users in Unix or Windows can’t create sub-users that they control, in which they could enforce their user-defined invariants. Any failed invariant potentially puts the entire system at risk, and any security breach means everything the user does is affected (which on single-user computers, means everything worthwhile on the computer). That subsystems shall not break their own or each other’s invariants thus remains a pure matter of convention: the kernel will not enforce these invariants at all; they are enforced solely by myriads of informal naming conventions, manual management by system administrators, and social pressure for software developers to play well with software developed by others. Any bug in any application exposed to the internet puts the entire system at risk.

There does exist a tool whereby user-defined invariants can be enforced, of sorts: machine emulation, machine virtualization, hypervisors, containers, user-level filesystems, etc., allow to run an entire human machine with its own kernel. However, except for the most costly and least powerful strategy, emulation, that is always available, these tools are not available for casual users or normal programs; they are heavy-weight tools that require system administrator privileges, and a lot of setup indeed. Still, they exist; and with these tools, companies with a large expensive engineering crew can enforce their company-wide invariants; they can thus enjoy the relative simplicity that comes when you can reason about the entire system, knowing that parasitic behaviors have been eliminated, because they are just not expressible in the unikernels that are run inside the managed subsystems.

Houyhnhnms recognize that the invariants that ultimately matter in a computing system are never those that underlie any “base” system; instead, they are always those of the overall system, the “final” applications, as experienced by users. To them, the idea that there should be a one privileged “base” system, with a kernel that possesses a monopoly on invariant enforcement, is absurd on its face; the invariants of a system low-level enough to implement all the programs that users may care about are necessarily way too low-level to matter to any user. In Houyhnhnm computing systems, virtualization is a basic ubiquitous service that is universally relied upon; each activity is properly isolated and its invariants cannot be violated by any other activity, except those that explicitly run at its meta-level.

What’s more, Houyhnhnms understand that when building software components and applications, programmers almost never want to start from that “base” system of a low-level machine, however virtualized, but want to start from as high-level a system as they can afford. Therefore, Houyhnhnms have first-class notions for computing systems and for implementing a computing system based on another computing system (again, in terms closer to notions of a human computer systems, a computing system includes both a programming language and a runtime virtual machine). Which functionality is virtualized, and which is just inherited unmodified from the lower-level system, can thus be decided at the highest level that makes sense, keeping the overall system simpler, easier to reason about, and easier to implement efficiently — which is all one and the same.

Certainly, some technical enforcement cannot wholly replace social enforcement: some invariants are too expensive to enforce through technical means, or would require artificial intelligence to do so, which Houyhnhnms don’t possess more than Humans. But at least, Houyhnhnms can minimize the “surface of attack” for technical defects that can possibly violate desired invariants, by making such attacks impossible without a meta-level intervention; and those meta-level interventions that are necessary can be logged and reviewed, making for more effective social enforcement as well as for general reproducibility and debuggability.

Runtime and Compile-time Enforcement

In a Human computer system, The Kernel possesses a monopoly on invariant enforcement, and only enforces a static set of invariants; it does so by being an expensive middleman at runtime between any two components that communicate with each other or use any hardware device. In terms of invariant enforcement, this is simultaneously extremely unexpressive and quite expensive.

Houyhnhnm computing systems, on the other hand, have a decentralized model of invariant enforcement. Every user specifies his invariants by picking a high-level language as the base for his semantics, and using this language to define further computing elements and their invariants. Most invariants can be enforced statically by the high-level language’s compiler, and necessitate no runtime enforcement whatsoever, eschewing the cost of a kernel. When multiple components need to communicate with each other, the linker can similarly check and enforce most invariants, and eschew any runtime enforcement cost.

Well-behaved programming language implementations can therefore manipulate low-level buffers directly without any copying, when producing video or sound; the result is real-time performance without expensive runtime tricks — or rather, performance precisely by the absence of expensive runtime tricks. When the user requests causes a change in the circuit diagram, the code may have to be unlinked and relinked: thus, relinking will happen when users add or remove a filter between the sound producers and the actual audio output, or similarly introduce some window between graphic drawers and the actual video output. But this relinking can happen without any interruption in the music, with an atomic code switch at a time the buffers are in a stable state.

Certainly, any available hardware support to optimize or secure virtualization can and will be used, wherever it makes sense. But it isn’t the exclusive domain of a One Kernel enforcing one static set of invariants. Rather, it is part of the panoply of code generation strategies available to compilers targetting the given hardware platform. These techniques will be used by compilers when they are advantageous; they will also be used to segregate computing systems that do not mutually trust each other. But what matters most, they are not foundational system abstractions; the computing interactions desired by the users are the foundational system abstractions, and all the rest is implementation details.

Bootstrapping

The last thing (or, depending on how you look at it, first thing) that a Kernel does in a Human Computer System is to bootstrap the computer: The Kernel will initialize the computer, detect the hardware resources available, activate the correct drivers, and somehow publish the abstracted resources. The Kernel will take the system from whatever state the hardware has it when it powers up to some state usable by the user programs at a suitable level of abstraction.

As always, between the firmware, the boot loader, The Kernel, the initialization service manager, the applications that matter, plus various layers of virtualization, etc., the task of initializing the system is already much less centralized even in Human computer systems than the Human “ideal” would have it. Houyhnhnms just do away with this not-so-ideal ideal. They consider that what matters is the state in which the system is ready to engage in whichever actual interactions the user is interested in; anything else is either an intermediate step, or is noise and a waste of resources — either way, nothing worth “blessing” as “the” “base” system. Instead, automatic snapshotting means that the time to restart a Houyhnhnm system is never more than the time to page in the state of the working memory from disk; only the first run after an installation or update can take more time than that.

As for the initial hardware resources, just like any resources visible in a system, they are modeled using linear logic, ensuring they have at all times a well-defined owner; and the owner is usually some virtual device broker and multiplexer that will dynamically and safely link, unlink and relink the device to its current users. Conversely, the users will be linked to a new device if there is a change, e.g. because hardware was plugged in or out, or because the system image was frozen on one hardware platform and thawed on a different one. With the ability to unlink and relink, Houyhnhnm computing systems can thus restart or reconfigure any subsystem while the rest of the system is running, all the while persisting any state worth persisting. This is quite unlike Human computer systems, that require you to reboot the entire system any time a component is stuck, at which point you lose the state of all running programs.

Polycentric Order

In the end, it appeared that once again, the difference of approach between Humans and Houyhnhnms led to very different architectures, organized around mutually incommensurable notions. Humans think in terms of fixed artifacts; Houyhnhnms think in terms of evolving computing processes. My questions about some hypothetical fixed piece of software in their computing architecture were answered with questions about some hypothetical well-defined patterns of interactions in our computer architecture.

Houyhnhnm computing systems do not possess a one single Kernel; instead they possess as many “kernels” as there are computing subsystems and subsubsystems, each written in as high-level a language as makes sense for its purpose; and the set of those “kernels” continually changes as new processes are started, modified or stopped. Resource management is decentralized using linear logic and meta-level brokers, linking, unlinking and relinking. Invariant enforcement, though it may involve runtime checks, including hardware-assisted ones, is driven primarily by compile-time and link-time processes. Overriding invariants, while possible, requires special privileges and will be logged; unlike with Human computer systems, processes can’t casually interfere with each other “just” because they run with the same coarse “user” privilege.

Humans try to identify an artifact to buy or sell; Houyhnhnms look for processes to partake in. Humans have static understanding of relations between artifacts; Houyhnhnms have a dynamic understanding of interactions between processes. Humans use metaphors of centralized control; Houyhnhnms use metaphors of decentralized ownership. Humans think of enforcement as done by a superior third-party; Houyhnhnms think of enforcement as achieved through mutually agreeable contracts between equally free parties. Humans see all resources as ultimately owned by the Central entity and delegated to users; Houyhnhnms see resources as being used, shared or exchanged by independent processes. I could see a lot of ways that the paradigm of Human computer systems fit in a wider trend of patterns in which to conceive of social and political interactions. Yet, I resisted the temptation of asking Ann about the social and political context in which Houyhnhnm computing systems were being designed; at least for now, I was too deeply interested in figuring out the ins and outs of Houyhnhnm computing to be bothered by a digression into these far ranging matters. However, I did take stock that there was a lot of context that led towards the architecture of Human computer systems; and I saw that this context and its metaphors didn’t apply to Houyhnhnm computing, and that I needed to escape from them if I wanted to better understand it.

Chapter 5: Non-Stop Change

Ngnghm — Wed, 09 Sep 2015 03:54:23 UT

Ngnghm, or Ann for her human friends, decided that while stranded among us Humans, she would conduct an ethnographical study of Human computer systems, and took to heart to examining my programming habits. In return, I was more and more curious of how Houyhnhnm (pronounced “Hunam”) systems worked, or failed to work. That’s when, trying to imagine what the Houyhnhnm computing systems might keel over, with their making everything persistent, I had this a-ha moment: surely, they must have extreme trouble with live upgrade of their data schema, and their programmers must spend their time in hell trying to reconcile modifications in what amounts to an unrestricted distributed database, that anyone can modify at any time. Ann wasn’t sure what I was talking about that could be a major issue, and so interrogated me as to the Human practices with respect to handling change in persistent data. And she found that many of the issues stemmed from limitations with how Humans approached them; these issues were not intrinsic with the problem of ensuring persistence of data, and all but disappeared if you considered code change transactions as first-class objects.

The Best Laid Schemas of Houyhnhnms and Men

I challenged Ann to explain to me how Houyhnhnm systems dealt with code and data upgrade, because they were some of the hardest problems I had faced while working with Human computer systems. Ann wasn’t sure what I meant, or why whatever she suspected I meant would be particularly difficult. That surprised me, so I started by stating that any software that survives long enough has to change its code as the software became obsolete, in a phenomenon humorously known as bitrot: as time goes by, the software becomes no good anymore, as if it were being degraded by some kind of rot; of course, and that’s the joke, no external force can possibly cause the bits that compose the software to change and degrade — on the contrary, the problem is precisely that the software fails to change when the world around does change and requires the software to adapt so as to remain relevant. Ann told me that the analogy makes sense to Houyhnhnms but isn’t as humorous: it isn’t the software, but the relationship of the software with the world, that rots; the degradation thus applies not to the computer system, but to the computing system, as constituted not just by the computer, but also by the sentient programmers and users, and by the larger context in which the computation happens. To a Human focused on the software artifact, it may be a funny paradox; but to a Houyhnhnm focused on the softwaring process, it’s a painful truism.

I was intrigued, but I went on. As a program evolves, the way that it organizes data, its schema, will eventually be found wanting, as well designed as it might once have been for its original purpose. New data types, new file formats, new database schemas, will be needed, as the old one become insufficient and inadapted to newly required kinds of computations. Now, the correspondances between old and new data is often non-trivial, and upgrading data from the old format to the new format is hard; what more, upgrading all the data, consistently, without any race condition between old and new data processors, was even harder. And if the upgrade has to happen without any down time from the system as it keeps processing data, then achieving it without a hitch becomes a feat of virtuosity. Finally, bugs in such an upgrade process are both all too easy to introduce and all too hard to recover from. Therefore, Human computer systems shun these schema upgrades as much as possible, and some companies pay people full time just to manage the writing and/or smooth running of upgrade scripts that keep their data up to date, and to maintain the expensive database servers capable of doing it all dependably. However, humans are lucky enough to be able to avoid these difficult data upgrades altogether in most cases, precisely because Humans were not persisting data but throwing it away, so it didn’t need to be upgraded. When they really have to deal with the hard case, the best tools available to regular Humans, like protocol buffers or its successor Cap’n’Proto, are not very expressive in terms of types and leave a whole lot to the programmers in terms of managing upgrades; but at least they provide a framework for data to survive and be upgraded beyond the short life of a program that by necessity can only handle one version of the schema. And yet even these tools are only so good for their cost, and most people just do upgrades the hard way: either with completely disjoint old and new schemas, duplicated code, and big ad hoc functions to translate between them; or with ad hoc sharing of code and data between the schemas, a complexity that increases with each new version, and plenty of corner cases that are never considered, much less tested.

Surely, by persisting everything all the time, Houyhnhnm computing systems were forced to deal with this hard issue all the time, which of necessity must have made all programming difficult and tedious. Yet Ann claimed that she didn’t know data upgrade to be particularly tedious on Houyhnhnm systems.

Afterthought or Forethought

First, she said, the case that is easy in Human computer systems, namely eschewing any upgrade and dropping the old data is just as easy in Houyhnhnm computing systems: just don’t upgrade the data, and instead drop it, as you modify the code. If you don’t care about data, then by definition you don’t care whatever automation the system provides to upgrade it for you, and don’t care to fix it; if you know that for sure in advance, you can even tell the system, so it won’t bother saving the data, and you might get a nice speed up for it, and a reduced monthly bill for your globally replicated persistent data. However, in the cases the data actually matters and you do care about it, that’s when you’ll miss system support for data persistence, and that’s also when you’ll miss system support for upgrading your data.

Moreover, even in this easy case, since Houyhnhnms, as you may remember, save code and data together, you can simply fork the old data together with the correct version of the code that knows how to create and manipulate it, and keep using it while you work on a new version. “Orphaned data”, “version mismatch”, “DLL hell”, are issues that might sometimes delay upgrade, and cause a lot of grief to Houyhnhnm programmers indeed, but they never can prevent reusing current or old data — computation itself is immortal, even if its relationship to the world can suffer. And Houyhnhnms just don’t know the catastrophic failure modes of Human computer systems, such as a system becoming unusable due to failure in the middle of an upgrade, what more with missing or incomplete backups so you cannot downgrade, cannot upgrade, and must reinstall and reconstitute all local configuration from scratch.

Second, since the “canonical” representation of data is not low-level bytes, but high-level data types, a whole lot of the extrinsic complexity that Human computer systems have to deal with can instead be automated away: When the schema change is merely a change in low-level representation with identical high-level data types, then a trivial strategy is always available for upgrade — decompile and recompile. Moreover, the system can automatically track which object currently uses which underlying representation; it can therefore manage the upgrade without sentient oversight. The sentient programmers can then focus on the actual intrinsic issues of schema upgrade: non-trivial data transformations into semantically non-equivalent data types. And even then, the system provides a framework that automates a lot.

Third, and more importantly, upgrade is inevitable indeed; but the problem with Human computer systems is that since they both focus on software as a finished artifact and define things at a low-level, they drop all the data that matters about upgrade when it’s readily available, only to desperately try to reconstitute it the hard way after it’s too late. Upgrade automation is thus almost inexistent in Human computer systems, because it comes as an afterthought. By contrast, it comes naturally in Houyhnhnm computing systems, because it is part and parcel of how they conceive software development.

Change Comes from the Inside

To Humans, change happens outside the computer system that is changed: A program is an immutable object, especially so after it was invoked to start a process (a limited virtualized computer system). To change a program thus requires shutting down the processes started with the old program and starting new processes with the new program, although doing it naively can cause an interruption of service.

As for changing a program’s data types and having to preserve data, that to Humans is an exceptional situation to be dealt with using exceptional tools. Such a catastrophic event happens every so many months or years, when releasing a new “major” version of the program.

To Houyhnhnms, change happens inside the computing system that changes, because everything relevant is ipso facto inside the system, part of its ontology. What more, to Houyhnhnms, change to programs is what programming verily is: To program is to change programs, and to change programs is to program. That’s a tautological identity. Inasmuch as types are part and parcel of a program, then changing a program’s data types is part and parcel of programming and is supported by the system as a matter of course, including upgrading any existing data to use the modified types.

To a Houyhnhnm, the idea that change could happen outside the system is absurd on its face, and the notion of a programming language that would only be concerned with describing programs that never change is obviously lacking. Of course, change processes may or may not be automated — but as per the Sacred Motto of the Guild of Houyhnhnm Programmers, whatever parts of them can be automated, should be automated, must be automated, will be automated, until eventually they are automated.

Once again, Humans focus on programs as finished entities with fixed semantics, whereas Houyhnhnms think in terms of systems made of ongoing interactions. And once again, this fundamental difference in paradigm implies fundamental differences in the most basic structures of computing systems, including programming languages.

Human Languages that Support Live Upgrade

However, there are two notable exceptions amongst Human programming languages, whereby the language does — to some degree — support live upgrade, that is, changes to programs and data types in an actively running program, from within the language: Common Lisp, and Erlang.

Common Lisp lets you redefine functions, classes, types, etc., in an existing, running image, and immediately thereafter use the modified program. There are however a lot of limitations and a lot of pain in doing so, especially if the program has to be actively running while the modifications take place. For instance, you may have to declare your functions notinline to ensure the latest version is always used, or to shadow their symbol to ensure the version at time of definition is always used. Before you redefine a class, you can define methods on update-instance-for-redefined-class to ensure that all data will be properly upgraded after the class redefinition; this is somewhat low-level, there is no automated enforcement of consistency between such methods and the types, and you better keep supporting older versions of the class, because you can’t be sure that some older instances still haven’t been upgraded; but it’s there and it works. There are many limitations however to this hot upgrade support; notably, and depending on which Common Lisp implementation you use, there are many kinds of code modifications that will not be safe to issue if multiple threads are active at the same time (particularly changes to the class hierarchy); so you will have to somehow synchronize those threads before you upgrade your code. The big problem with Common Lisp is that its model of side-effects to a single global world doesn’t play well with a modern concurrent and distributed world, where a complete system is made of many processes running on many machines, or where you want to have multiple versions of the software running at the same time. Lisp hasn’t been actively developed as a self-contained system for three decades, and what remnant support it has for thinking in terms of system is a partial subset of whatever antiquated solutions existed when it was standardized. Yet that it supports at all the upgrade of code and data without resorting to magic from outside the language makes it miles ahead of all known Human languages. Except Erlang.

Erlang’s heritage and ambitions are very different from those of Lisp, and in many ways it is much more primitive — but as an upside its primitives are better defined. Through a reinvention decades later, Erlang embodies the Actor model of the early 1970s, which is what the “Object Oriented” utterly failed to be despite its own hype: a programming model where many well-encapsulated entities — called “processes” in Erlang rather than “objects” because they are active — communicate with each other by passing messages. Processes can be distributed, and reliability is achieved by assuming that individual processes will fail eventually, letting them fail, and restarting them, whereas important data will have been stored and replicated in several other processes that won’t all fail at the same time. Now, one thing where Erlang shines, far better than Lisp and far far better than anything else, is in its support of live upgrade of code and data — a topic that it alone seems to take seriously. When you upgrade code, it is always clearly defined which function calls will go to the old version of the code, and which function calls will go to the new version. And the strict evaluation model in terms of processes and messages makes it possible to reason about what state each process is in, how it will update its state, and how it will handle old or new messages. All in all, Erlang feels like a low-level language, but not a Human low-level language, that tries to track the underlying computer hardware for efficiency, and more like a Houyhnhnm low-level language, that tries to track the underlying computing model for correct and reliable behavior; what makes it less than a high-level language is that it isn’t easy enough to build higher abstractions on top of the provided abstraction level — though it may still be better at it than many alleged “high-level” Human programming languages.

Therefore, there are some precedents in Human computing systems that allude to what Houyhnhnm computing systems would be. But they are quite undeveloped compared to what would be required of a full-fledged Houyhnhnm computing system.

Automating Live Upgrade

So, I asked Ann, what do Houyhnhnm programming languages look like when it comes to supporting live upgrade of code and data in a running program?

First, Houyhnhnm programming languages have a notion of transactions for code and data upgrade, that can be composed and applied atomically, so that you can coherently upgrade a system from one version to another, without having to wonder what happens if some code is running right in between when two mutually dependent entities are being redefined. Of course, while developing, Houyhnhnm programmers will want to experiment and explore with upgrades of some part of the code or data only, that lack overall system coherence; such exploratory modifications are doomed to fail, but that’s alright, because until it is are properly vetted, each change is always run in its own virtual fork of the system, where failure is quite acceptable.

Houyhnhnm type systems track how incoherent the system is, and maintain for the programmer a to-do list for those parts of the system that haven’t been updated to follow the new types yet. At runtime, incoherent calls are intercepted before they may break the system, and the debugger offers the programmer a chance to give a new function definition before to enter (or re-enter) the function that fails — among Human programming languages, good Lisp implementations provide the latter (though they can’t undo the side-effects of bad functions), whereas good statically typed languages provide a bit of the former (though they don’t maintain to-do lists and thus gloss over functions that fail to break the type system e.g. because they have an otherwise clause that implicitly covers a newly added case when a correct code update ought to explicitly handle the new case, of at least requires the programmer to vet that the otherwise clause indeed applies to the new case).

Houyhnhnm systems, since they remember the history of type modifications, require every type modification to be accompanied by a well-typed upgrade function, taking an object in the old type and returning an object in the new type. Simple upgrade functions can be trivially expressed, such as using default values for new fields, or erroring out. The system also uses linear logic to ensure that when writing an upgrade operator, you must explicitly drop any data that you don’t care about anymore, so you can’t lose information by mistake or omission (old versions of the data will persist by default, as for all data in a Houyhnhnm system, but upgrade mistakes will still make further fixes harder, so it is very valuable that Houyhnhnm systems support writing correct upgrade procedures easily). Because the same function doesn’t necessarily apply to all data of a given type, upgrade functions can be overridden at the granularity of individual variables, and they may take some contextual information as extra parameters.

The system also tracks the difference between on the one hand renaming some variable, and on the other hand deleting some variable and adding another: though the two lead to identical source code from the limited Human point of view, they imply very different upgrade behavior from the wider Houyhnhnm point of view, and are represented differently in a Houyhnhm computing system. Branching and merging are similarly supported. These are very easy to track in an interactive development environment, but hard to reconstitute, indeed impossible to reliably reconstitute, just by looking at the code before and after the modifications, when the language doesn’t allow to explicitly express the difference.

Houyhnhnm systems allow you to edit the effective history that applies to software releases; this history is made of coherent changes that describe how to upgrade from previous released versions to the latest one while preserving all system invariants. It can be contrasted with the actual history through which the result was attained, which is often made of incoherent changes, of backing from blind alleys and of downright mistakes. By upgrading data along the effective history rather than actual history, data corruption can be avoided, and bugs introduced in the upgrade process can themselves be fixed. Now, since Houyhnhnm computing systems accommodate for situations with many distributed agents each with their own version of the code and data and their history, the modifying, branching and merging of objects also apply to these histories of code and data changes. Histories themselves constitute a monotonic algebra where it is always possible to merge histories; unhappily, merge conflicts may cause the result to be incoherent according to the criteria that the users care about; however the system can also check the coherence of the results and reject incoherent transactions, and/or require manual intervention by sentient programmers to fix whatever incoherence it tracked. Of course, this can lead to arbitrarily complex situations when merging histories in a distributed system; but in practice, there is a finite amount of data that needs to be upgraded that people actually care about, so there is a cap on how much complexity Houyhnhnm programmers need to deal with. Moreover, since Houyhnhnm computing systems, unlike Human computer systems, do track where all that data is, Houyhnhnm programmers can be fairly confident that they did (or didn’t) complete the upgrade of all relevant data, at which point no further upgrade effort is required.

Now, none of this negates the interest of having a subset of the programming language that allows to express immutable programs and reason about them. Immutability is indeed a very important property that Houyhnhnm programmers can appreciate as well as Human functional programmers. But when programs do mutate, and they do, by the very virtue of programming itself, Houyhnhnm programmers much prefer being able to reason about these changes from within the system, with all the automation that the system makes possible, including automated refactoring, detection and tracking of incoherence, formal reasoning, etc., where instead Humans have to revert to completely informal reasoning on pen and paper, and must indulge in expensive manual drudgery which robs them from being able to focus on the difficult parts that really require their attention, drowning the important details in an ocean of boring repetitive tasks. Houyhnhnm programming languages thus allow to express immutable programs written in suitably restricted programming languages that make it easy to reason about them; and simultaneously they can express meta-level programs without these expressive restrictions (but with different restrictions) that can describe changes in the former programs, from outside their universe but still within the automated part of the overall computing system, also allowing for reasoning.

In conclusion, as compared to Human computer systems, Houyhnhnm computing systems handle all the easy cases of live upgrade, and automatically ensure coherence for the hard cases, where Humans instead rely on large bureaucracies of database administrators and upgrade experts to manually manage live upgrade of code and data through excruciating slow and still error-prone unautomated processes. And this, it bears repeating, is but a necessary consequence of the simple difference of point of view, of “philosophy”, between Human computer systems and Houyhnhnm computing systems.

Chapter 4: Turtling down the Tower of Babel

Ngnghm — Mon, 24 Aug 2015 23:51:01 UT

Ngnghm, or Ann, examined how manual persistence was managed underneath Human computer systems, and contrasted with how Houyhnhnms (pronounced “Hunams”) automated its implementation. This led her to more general remarks about the compared architectures of Human computer systems and Houyhnhnm computing systems: Houyhnhnm computing systems can and do go meta, which to them is notionally down (not up, as some Humans would have it). Going meta allows Houyhnhm computing systems to enjoy qualities not found in Human computer systems, that can’t go meta.

Stacked up against Quality

Ann wanted to know how Humans dealt with manual persistence. She found that we were using a large quantity of mutually incompatible and often fragile “libraries” in each of many loose categories that each implement some aspect of persistence: “I/O”, “file formats”, “serialization”, “marshalling”, “markup languages”, “XML schemas”, “communication protocols”, “interchange formats”, “memory layout”, “database schema”, “database servers”, “query languages”, “object relational mapping”, “object request brokers”, “foreign function interface”, and many “wrappers”, “adapters” and “glue layers” to make them work together. Indeed, some old IBM study had estimated that 30% of all application code written was related to the basic functions of saving data and restoring it — and at least my experience suggests that this estimate might still be valid to this day. Houyhnhnms, like Dijkstra, regard this as a huge cost: if we wish to count lines of code, we should not regard them as “lines produced” but as “lines spent”: the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger.

Unhappily, that huge cost also comes with limited benefits, because a program can only manipulate an object if it gets the entire large tower of libraries, the “software stack”, just right, and thus two objects built on top of incompatible “software stacks” cannot interoperate. Costly adapters can be written to bridge between the two towers, but this not only requires extra copying and management by programmers, this also loses any atomicity properties of transactions between the two object systems — and isn’t accessible to casual users, who thus pain to manage their data.

Moreover, the above estimate did not include the error handling strategies when the above failed; meanwhile, the complexity of these baroque towers incur enormous security risks. Indeed, a lot of “layers” in these software “stacks” are written in unsafe low-level languages for reasons of alleged “performance” or “compatibility”, whereas another (overlapping) lot of such “layers” include some complex manual parsing of data going through the layer, that are as many points where attackers may inject unwanted behavior; these many layers further interact in ways that make it nearly impossible to assess the overall semantics of the system, much less its security properties. As for performance, a lot of it is wasted just crossing the layers at runtime, rather than e.g. folding them at compile-time.

This architecture in software towers is thus detrimental not only to persistence, but also to robustness, to security, to performance, to upgradability, to maintainability, etc., — all the qualities that managers of Human computer development projects often demote as being “non-functional”, because their development processes are so deeply dysfunctional, at least from the Houyhnhnm point of view: by neglecting as an afterthought aspects of software development that are not directly visible through a quick test of a software artifact, these processes ensure that those aspects cannot be addressed properly. By contrast, Houyhnhnm computing systems consider as primary the processes of software development and use, not the artifacts; they thus consider the above aspects as primary properties of the overall system, that are important to address as part of the architecture of the softwaring process.

Meta-level Strategies

Houyhnhnms do not have any library to manage persistence; instead, Houyhnhnms have a number of libraries to manage transience. Indeed, persistence is a system-wide protocol, universally provided using generic strategies, and comes for free to users and programmers alike; they don’t have to manually flush main memory buffers to mass storage any more than they have to manually flush memory cache lines to main memory buffers, or to manually spill processor registers to memory cache lines. But if they care about extra performance, they can manage these things indeed, and escape or improve the system-provided strategies. In other words, correctness, safety, etc., come for free, and it takes extra effort for a variable not to be saved, for infinite undo not to be available, etc., — and for extra performance to be squeezed out of otherwise working programs. I already mentioned in the previous chapter many things that you might want not to persist altogether, or for which to only keep episodic backups. More interesting are the cases where you may want to extend the system to more efficiently support some data type (say, domain-specific compression), some consensus protocol (say, a variant of the PAXOS algorithm), some reconciliation process (say, a new CRDT), or some resource ownership discipline (say, a variant of linear logic). Then you want to specify a new implementation strategy for common system protocols; and for this you usually specify a modular incremental variant of the openly-accessible existing strategies.

Unlike what you’d use in Human computer systems, these strategies are not merely runtime libraries that you link to, the APIs of which programs must explicitly call — this would require every program to be modified any time you change a persistence strategy (alternatively, every program would have to use very rigid virtual machine, with either a very slow interpreter or a very expensive compiler). Instead, persistence strategies are meta-level software modifications that customize the implementation of the usual programming languages. Thus, these strategies can arbitrarily instrument the code generated for existing programs, to automatically add any required call to suitable libraries, but also to efficiently handle any associated bookkeeping, depending on what strategies are in the domain in which the unmodified programs are run. Updated objects may be marked, either individually, in “cards” or in “pages” for the sake garbage collection or persistence; counts or sets of local or remote references may be maintained; drawing pictures may be achieved either by blitting directly to video memory or by issuing requests to some server; some type system may be enforced through some combination of static inference and dynamic checks; etc. Of course, these implementation strategies may reject requests to create or move a process into a domain where some incompatibility exists: the program might not pass some static type checks; it might fail to possess appropriate permissions, or sufficient resources, etc. Then the user or programmer may have to modify his program or try a different strategy.

Importantly, this variety of strategies is made possible because Houyhnhnm computing systems are first-class entities abstracted from any specific implementation strategy. Therefore, a very same process (which includes not only source program, but also running state) may be run with different strategies — and indeed with strategies that vary during its execution. When you write a program, the source language you choose completely specifies allowed behavior, and all strategies are guaranteed to preserve this behavior, no more, no less.

Of course, either at the time you start the program or later, you may decide to constrain the process to only use a given subset of strategies: this actually means that you really wanted a more specific program in a more specific language than initially declared. Not only is that fine, that’s a common and recommended way of writing programs: always specify the program’s behavior at as high-level as you can, to make it easier to reason about it; yet make sure the optimization strategies you require have been applied, so the performance profile isn’t surprisingly bad. As a trivial example, the Fibonacci function would be specified with its usual equational definition, but would typically be annotated with a compile-time assertion that the linear recursion recognizer has kicked in, at which point the system guarantees that the function will be computed in constant time for small values, and polylog time for big ones — rather than exponential time, with a naive implementation.

Formally speaking, if you wrote a program in abstract language A, and specified a given implementation I of language A generating code in concrete language C, then you actually specified a program in language C. And as long as you don’t proceed to make modifications at the lower level of language C that invalidate the abstraction to language A, then you can remove the constraint, go back to the level of program A, and later choose a different implementation I’ targetting language C’. That’s how you migrate a process from one domain to another. (This ability to do generalized migration also requires having formalized the notion of an implementation such that you can interrupt and decompile a process, including running state, and not just source code, from its concrete implementation back to the level of abstraction at which the user has chosen to interact with it — but that’s a topic for a future chapter.)

Anything You Can Do I Can Do Meta

In Houyhnhnm computing systems, programs are thus persistent by default (as well as type-safe, and safe in many other ways); yet they can be made faster and smaller by locally dropping to lower levels of abstraction in structured ways that preserve higher level of semantics. This generalizes the remark made by Paul Graham that, on Lisp, as compared to other languages, “You can get fast programs, but you have to work for them. In this respect, using Lisp is like living in a rich country instead of a poor one: it may seem unfortunate that one has to work so as to stay thin, but surely this is better than working to stay alive, and being thin as a matter of course.” This doesn’t mean that the default mode of operation is especially slow or wasteful of memory: given a fixed amount of development resources, accumulating reusable automated strategies as in Houyhnhnm computing systems can achieve more performance than manually implementing strategies in every program like in Human computer systems.

Indeed, manual implementation of software strategies, known in the Human computer industry as “design patterns”, is the primary source of bad quality in software: humans are just so much worse than machines (not to mention slower and more expensive) at applying algorithmic strategies — which notably goes against the Sacred Motto of the Guild of Houyhnhnm Programmers. (Of course, quality is even worse when the underlying design patterns have not even been recognized and their properties haven’t even been semi-formalized between sentients.) Now, errors can be made when writing the meta-program that automates the strategy — but it’s much easier to debug one simple general meta-program once than thousands of context-specific manual instances of the pattern that each had to precisely match the pattern in excruciating details. What more, without automation, it’s much harder to keep these myriads of instances right as the pattern or its parameters change, and maintenance requires all of them to be modified accordingly. As Rich Hickey quipped, (Design) patterns mean “I have run out of language.”

Because software strategies ultimately preserve the safe semantics of high-level languages, they involve less code written in unsafe low-level languages, and what low-level code is generated can be automatically and verifiably made to preserve high-level invariants that matter for safety. Entire classes of bugs that commonly plague Human computer systems thus never appear in Houyhnhnm computing systems. Of course, Houyhnhnms make many mistakes while developing their computing systems, and the inconsistent strategies they write can cause inconsistent behavior, with catastrophic consequences. But virtualization ensures that these catastrophes do not escape the narrow scope of the sandbox in which the developer is trying them; and catastrophic effects are actually easier to detect, so that most such bugs are faster to fix. Subtle meta-level bugs causing delayed catastrophes, though they exist, are quite rare. To eliminate them, the usual combination of testing and formal methods can help. There again, generic code is usually harder to test or formalize than a single specific instance of the code, but much easier to test or formalize than thousands or millions of instances, as necessarily happens when strategies are applied manually rather than automatically.

Finally, because Houyhnhnm computing systems work at the level of abstract data types, most messaging happens with robust system-provided pairs of printers and parsers, rather than an ever renewed collection of ad hoc manual printers and parsers for manually designed interchange languages, each introducing a renewed layer of bugs. Indeed, in Human computer systems, the humans who “design” these interchange languages are often unaware that they are designing languages indeed, or in deep denial when confronted to that fact; they thus prefer to remain ignorant of the very basics of language design, and ever repeat all the beginners’ mistakes. In Houyhnhnm computing systems, it is understood that whatever interactions happen between sentient beings and/or automated processes by definition constitute a language; and while you want the overall design interaction between sentient being and machine to happen at the highest possible level using as expressive a language as possible, the interactions between automated processes should happen using the highest level but least expressive language possible, so they remain easier to analyze.

Therefore, when contrasted to Human computer systems, it appears that Houyhnhnm computing system thus achieve better quality through meta programming.

Building up vs building down

Humans can only build software but up. Houyhnhnms can build both up and down.

All computer software has to start from a given base: whatever abstractions the operating system provides, or, in absence of operating system, the “bare metal” — which for Human computer systems is often not quite so bare these days, with plenty of firmware, coprocessors and virtualization layers involved. Now, Human computer systems are built by piling layers upon layers on top of this base; and a Human operating system itself can be already considered such a tower of layers, on top of which to build higher towers. One limitation of Human computer systems, though, is that to cooperate on the same data structures, programs typically have to reuse the very exact same tower of layers. Because each layer adds a lot of informal underspecified details, and it is impossible to reproduce computations or assume that programs have similar enough semantics unless they are identical from the ground up. With this tower architecture, as with the legendary Tower of Babel, people are divided by a confusing diversity of languages that prevent them from communicating.

Now, it is actually important to share data between different programs. Human software developers thus onerously build abstractions, without system support, so that they may save files in one format, which will hopefully be implemented in a compatible enough way by the other program or next version of the program. The operating system itself is such an abstraction, trying to present a uniform view of the computer to programs that run on top of it, despite a wild variety of underlying computers; so are to a point various virtual machines, or programming language specifications. So is, more trivially, the informal promise in successive versions of the “same” program to keep working with data saved by previous versions. Yet, any given abstraction usually has at most one sensible implementation on any given Human computer system.

Slightly more advanced Human computer systems, using macros, can at compile time lift the system up and add a number of layers below. For an extreme case, some Common Lisp libraries reimplement Common Lisp in Common Lisp to add first-class multiple-entry or even serializable continuations, so as to enable logic programming or direct-style web programming. Some interactive development systems also instrument the virtual machine so as to lift execution into something that allows for debugging, with Omniscient Debugging as an extreme example. But even then, once the program is built, once the runtime has been chosen, once the program has started running, the system remains forever grounded on top of the chosen basis.

Houyhnhnm computer systems, by contrast, can dynamically add new layers below a running program: not only can you add a layer on top of any existing tower before you start using it, you can add or replace layers below the tower, or anywhere in the middle of it, while you are using it. This ability to build down as well as up crucially relies on processes being specified in formally well-defined high-level languages, so that it is always clear what are the semantics to be preserved when modifying the underlying implementation. Therefore, Houyhnhnms don’t even have a fixed notion of ground or base. Rather than rigid towers of stone being built up, they have living worlds that stand on an indefinite number of other living worlds, just like the turtles of the common joke, whereby there are turtles all the way down. Then Houyhnhnms can lift the stack of turtles at any desired point and add or replace some of the turtles beneath, all while the system keeps running. Every turtle is unique, but no turtle is special.

The superficial differences between Houyhnhnm computing systems and Human computer systems are thus the reflection of radical differences between their underlying software architectures — that once again, derive from the initial divergence in point of view: considering the entire sentient-machine processes, rather than focusing only on the finished machine artifacts.

Chapter 3: The Houyhnhnm Version of Salvation

Ngnghm — Sun, 09 Aug 2015 05:10:00 UT

Following our discussion on persistence, Ngnghm, or Ann as I call her, had plenty of questions about how Human computer systems held together when they can’t seem to get basic persistence right. But in return, I had even more questions about what Houyhnhnm computing systems could even be like, when all data persisted by default: What did the user interface look like? Was there no more save button? What happened when you copied or deleted files? Were there files at all? How did people deal with all the garbage? Were your mistakes forever? If you somehow hosed your machine, would it remain forever hosed? How did you test potentially dangerous changes?

When Data Persists

Thus, on a first approach, the interface to a Houyhnhnm (pronounced “Hunam”) computing system may look very similar to that of a Human computer system — and that’s how Ann had been fooled at first into thinking they were designed along the same principles. If you have a Human laptop that you usually put to sleep and don’t turn off or restart, your setup might be similar to what it would be on a Houyhnhnm muzzletop. But even if your operating system is very stable, you must always be prepared for a catastrophic failure; and whenever you upgrade your software (which you must do eventually if only to apply security patches), then you must restart just to make sure the restart procedure will be working if the failure happens while you’re disconnected from online help — even though your system is stable and doesn’t really need it. And at that point you lose all your current working state. The eventuality of losing all your current working state is thus a Sword of Damocles hanging above any and all working state you weave, even on the most stable of systems. Note that a Houyhnhnm computing system may have to be restarted semi-periodically for the very same reason; the difference being that you don’t lose any working state as you do — or else, your system administrator and/or your or his insurance will cover any damages caused. Impermanence is thus a pervasive assumption in all Human computer systems, around which all habits are built; the loss of working state can be mitigated by using “session managers” that automatically save your session, or “startup scripts” where you manually re-create your usual work session, but both approaches only save very partial session information, in addition to being quite fragile and unreliable in practice — and intensive in programmer effort to implement. By contrast, in a Houyhnhnm computing system, you never lose your session; moreover, you can extract a startup script to re-create a similar session on another system, by introspecting the state of the system (the support for which is necessarily present for the sake of persistence) or by selectively replaying the relevant parts of the system persistence log.

System-wide Versioning

In Human computer systems, editors and other applications have a “save” button. In Houyhnhnm computing systems, there are no applications and no “save” buttons. Instead, there are components that each deal with the specific aspects of some kinds of documents or data, and otherwise share common system features for general aspects of data — including notions of versioning and releasing, i.e. publishing a stable version so it’s visible to other people, whereas intermediate changes and their inglorious details remain unpublished. Thus, instead of “text editors” and “picture editors”, there are “text editing components” and “picture-editing components”, that are available anywhere that there are modifiable texts or pictures in the system — and pretty much any text or picture you see can be copied into an editable variant, or traced back to its source, which can be forked and edited, if it’s not directly editable already. In Human computer systems, programmers have to bundle a finite number of such components into the package-deal that is an “application”, where you can’t use the component you want without being stuck with those you don’t want. In Houyhnhnm computing systems, users can individually configure the components or combinations of components they want to use for each type of data they are interested in. They all delegate their versioning aspect to the user’s favorite versioning component, that will handle forking new branches of data, and branches off branches of data, merging data from multiple branches, atomically committing changes, releasing data from a subbranch to make its changes available to a wider branch, etc.

Another advantage of system-wide versioning is that in Houyhnhnm computing systems, infinite undo comes for free on any kind of data, without any special effort from the developer, and without any limitation for the user; what more it is available atomically for all data in the system or any joint subset thereof. By contrast, in Human computer systems, the ability to undo a few steps for a few kinds of documents is a very costly, unreliable and/or error-prone operation requiring a lot of programming and a lot of maintenance, and working on one document at a time; some applications maintain history, but it is optimized for data mining by spies, and useless for users to recover past sessions. One more feature made possible by system-wide versioning is the ability to easily reproduce and isolate bugs — an activity that consumes a lot of expensive programmer time in Human computer systems, and that is made much easier in Houyhnhnm computing systems, since the log of interaction events that led to the erroneous behavior was recorded and can be replayed until the behavior was narrowed down; then the error case can automatically be reduced to its essence, by shaking the tree of actually used dependencies as detected by re-running an instrumented version of the same code to achieve e.g. Omniscient Debugging; the test case is then ready for inclusion in a regression test suite.

Data at the Proper Level of Abstraction

Because persistence in Human Computer Systems consists in communicating sequences of bytes to external processes and systems (whether disks or clouds of servers), all data they hold is ultimately defined in terms of sequences of bytes, or files; when persisting these files, they are identified by file paths that themselves are short sequences of bytes interpreted as a series of names separated by slashes / (or on some systems, backslashes \, or yet something else). Because persistence in Houyhnhnm Computing Systems applies to any data in whichever high-level language it was defined, all Houyhnhnm computing data is defined in terms of Algebraic Data Types, independently from underlying encoding (which might automatically and atomically change later system-wide). For the sake of importing data in and out of independently evolving systems, as well as for the sake of keeping the data compressed to make the most of limited resources, some low-level encoding in terms of bytes may be defined for some data types. But on the one hand, this is the exception; on the other other, the data is still part of the regular Algebraic Data Type system, can still be used with type constructors (e.g. as part of sum, product or function types), etc. Whereas Human computer systems would require explicit serialization and deserialization of data, and would require ad hoc containers or protocol generators to allow larger objects to contain smaller ones, Houyhnhnm computing systems abstract those details away and generate any required code from type definitions. Low-level encodings can even be replaced by newer and improved ones, and all objects will be transparently upgraded in due time — while preserving all user-visible identities and relationships across such changes in representation.

Since objects are not defined in terms of sequences of bytes, the very notion of file doesn’t apply to Houyhnhnm computing systems. At the same time, accessing an object inside a data structure is often (though not always) conveniently represented as following an access path from the root of the data structure to the desired element. An “access path” is thus a natural notion in all computing systems even though in general a path is not a sequence of strings separated by slashes, but a list of accessors, that may be symbols or strings (when accessing a dictionary), integers (when accessing a sequence by index), or arbitrary accessor functions. But few are the cases where the natural way to locate data is via a list of sequences of bytes containing neither ASCII slash nor ASCII NUL; or worse, sequences of Unicode code glyphs up to some subtle case-conversion, represented as UTF–8 code points in some normal form; or even worse, the greatest common denominator between an underspecified set of several variants of the above, with unspecified separators.

Thus, files are not the general case for persisting data; text files even less so. Still, a good text editor and good text-based diff tools can provide a handy way to view and modify data and view and act on modifications to data. Indeed, unless and until you have better tools to represent change between arbitrary data structures, it makes sense to translate otherwise unsupported data structures to and from a well supported generic data structure such as text. It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures. Now, it is important to understand the distinction between a representation and the real thing; the text being presented is not “canonical”, it is not usually “source”. In Houyhnhnm computing systems, the source is the semantic state of the system, on which change happens, and from which the text is extracted if and when needed; this is in sharp contrast with typical Human computer systems, where the source (that is, the locus of modification by sentients) is text files that are compiled, disconnected from the state of the system.

Dealing with Bad Memories

But, I inquired, if they log everything and almost never forget anything, don’t Houyhnhnm computing system quickly get filled with garbage? No, replied Ann. The amount of information that users enter through a keyboard and mouse (or their Houyhnhnm counterparts) is minute compared to the memory of modern computers, yet, starting from a well-determined state of the system, it fully determines the subsequent state of the system. Hence, the persistence log doesn’t need to record anything else but these events with their proper timestamp. This however, requires that all sources of non-determinism are either eliminated or recorded — which Houyhnhnm computing systems do by construction. Of course, to save resources, you can also configure some computations so they are not recorded, or so their records aren’t kept beyond some number of days. For instance, you might adopt a model-view-controller approach, and consider the view as transient while only logging changes to the model and the controller, or even only to the model; or you might eschew long-term storage of your game sessions; or you might forget the awkward silences and the street noise from your always-on microphone; or you might drop data acquired by your surveillance camera when it didn’t catch any robber; or you might delete uninteresting videos; or you might expunge old software installation backups from long gone computers; or you might preserve a complete log only for a day, then an hourly snapshot for a few days, and a daily snapshot for a few weeks, a weekly snapshot for a few months, etc.; or you might obliterate logs and snapshots as fast as you can while still ensuring that the system will be able to withstand partial or total hardware failure of your personal device; or then again, given enough storage, you might decide to keep everything. It’s your choice — as long as you pay for the storage. The decision doesn’t have to be made by the programmer, though he may provide hints: the end-user has the last say. Indeed, the interests of the programmer may not be aligned with those of the user; if he needs to delegate these decisions, the user will thus entrust administrators he pays, whose interests are better aligned because they will be liable in case of malpractice.

Now, beyond clutter that uselessly fills up memory, what about actively bad things that surely you don’t want to memorize? Some mistake might cause your entire system to become unresponsive by resource exhaustion (a fork bomb on Unix, an out-of-memory situation on any system); something might trigger a system bug and cause a hardware crash, a Blue Screen Of Death or a kernel panic; even worse, some subtle combination of factors could generate a memory corruption that jeopardizes the integrity of the persistent data. Houyhnhnms computing systems may be more robust than Human computer systems in this regard, yet even Houyhnhnms are not perfect in avoiding catastrophic mistakes. If you detect such a situation, what do you do? Old Houyhnhnm engineers tell classic stories of catastrophic system modifications that were reverted by shutting down the computer before the modification was written to disk; but of course, as the latency of persistence goes down, the window of opportunity for such a feat goes away. The general answer is that to fix a system that has entered a bad state, you need an external system that can stop the broken system, inspect it, fix it, and restart it.

On a Human computer system, when things get that bad, you can often reboot in a special “failsafe” mode (that can usually handle but the simplest of situations), or you can use a USB key with a known stable version of the system (with which experts can handle complex situations), or at worst if the hardware was damaged, you can disconnect the computer’s mass memory unit and connect it into another computer. In a Houyhnhnm computing system, you can do as much, but you can also use a reserved input sequence (the equivalent of Ctrl-Alt-Del on Windows) to enter a monitor. The monitor is a simple but complete computing system, as per the Houyhnhnm criteria of simplicity; it is universal and can do everything a computing system can do, and is often a bare-bones variant of a regular computing system, as used for secure bootstrapping purposes; it also specifically understands enough of the semantics of the regular system to inspect it, fix it, and restart it, using the full power of a complete computing system (though a simple one). A small amount of memory is reserved for the operation of the monitor; actually, if mass memory units are working (as they should be) and have some reserved space for the monitor (as is the case on a default installation), then the monitor will actually spawn a virtualized monitor; this allows monitor operations to have more memory available, so they can for instance merge in a lot of the system state (up to some point deemed safe by the user); but this also makes it possible to still have a monitor (and possibly more virtualized monitors) in case you make mistakes in the virtualized monitor. As a result, it is safe to use dichotomy (binary search) to determine which change broke the system.

Virtualization as Branching

More generally, a Houyhnhnm can use virtualization and system rollback while conducting any kinds of experiments, so he never has to hesitate about doing anything risky, half-baked, downright stupid, or otherwise dangerous. But virtualization doesn’t mean the same thing in a Houyhnhnm computing system as in a Human computer system. In a Human computer system, virtualization is an ad hoc tool for system administrators, that allows the deployment of specially prepared servers; it is implemented using heroic techniques, by faking an entire physical computer at the level of abstraction of CPU instructions and memory accesses; the awkward result requires extra care and makes the users’ life overall more miserable rather than less, though it is sometimes an economic necessity. In a Houyhnhnm computing system, virtualization is merely the ability to branch the history of changes in the system, and derives naturally from the fact that the entire system is under version control; it is available at whichever level of abstraction the users and programmers specify their computations, which is most usually at a much higher level of abstraction than that of CPU instructions (though a naive, fallback strategy is indeed always available that consists in going down to that low level).

Thus, all destructive or catastrophic experiments happen in branches that are never merged into the official reality — the errors remain imaginary. Alternatively, when bad things happen in said official reality, they can be retcon’ed into having been but a bad dream, as the bad reality is demoted into being a mere unmerged error branch while a nicer reality becomes the master copy. Of course, it may be too late to undo bad communications with external systems, if they were let happen. Some of them may be cancelled or compensated with new transactions. Some of them may have to be accepted as losses or errors: you can’t unprint a pile of garbled characters, and you won’t get your bitcoins back. Even Houyhnhnm computing systems can’t protect you from yourself; but they can make it easy to try things in a virtualized environment, and to only merge into reality those transactions that pass all checks.

It is also possible to branch only part of the system while the rest of the system remains shared; and of course you can merge two branches back together, somehow fusing changes. Thus, there is no need to specially prepare an image for virtualization; any system and any subsystem, any program and any subprogram, can be forked at any time; any ongoing I/O with other systems can be redirected to one fork or the other, or multiplexed to/from both, or filtered, etc., and the user can dynamically re-wire all these connections from a monitor outside the virtualized system, that can itself be virtualized, etc.

Version control should be familiar to developers of Human computer systems; but these days, they apply it only to source code; and so live, interactive data is not covered by the version control, or at best only in a very indirect way, if the programmers make large, contrived and expensive efforts to take regular snapshots or, which is harder, to check in every change. Houyhnhnms think of code and data as coming together, part of the same interaction with the Sentient user, with data and code being useless without the other, or out of synch with the other; and thus Houyhnhnm computing systems casually apply version control to the entire state of the system.

In the end, thinking like Houyhnhnms in terms of computing systems, rather than like Humans in terms of computer systems has far-ranging consequences in terms of software and hardware architecture. Persistence is but one aspect of this architecture, though ultimately, it cannot be separated from the rest. And on this aspect like on others, from the necessity of dealing with the same basic needs and failure scenarios, the change in point of view leads to very different approaches to making and keeping the systems working.

Chapter 2: Save Our Souls

Ngnghm — Mon, 03 Aug 2015 05:10:00 UT

Ngnghm, whom I call Ann, was fascinated by our keyboards: because of physiological differences between our races, similar devices had never been imagined by Houyhnhnm computing engineers (I pronounce that like Hunam). Now, as she was watching me closely, Ann noticed that I was punctuating most of my typing with recurring combinations of key chords. I told her I had no idea what she meant; and so she had me record and review how, after every sentence or so, or before changing activities, I was composing the sequence Ctrl-X Ctrl-S, or Command-S, or some other mantra that varied slightly with the application I was using. Interestingly, I wasn’t even aware that I was doing that before she told me! What was this mantra doing, she inquired? How could I possibly repeat it without even noticing — and why would I? I told her that depending on the application, each of these mantra saved the current file, and that typing it had become ingrained in me as a subconscious habit, because I used it so often, out of necessity. What does "saved" mean wondered Ann, and what made it a necessity?

Persistence, Automated

I explained that Human computer software and hardware are prone to crashing, or to losing battery power, and other unexpected failures — there she sighed with sympathy, for Houyhnhnms (pronounced “Hunams”) were just as frustrated as Humans with how unreliable their computers were. I continued that the solution universally adopted for Human computer systems was therefore that Humans had to explicitly save each file for its contents to be later recoverable in the event of such a crash. Having been burned too many times by the loss of many hours of hard work, I had grown the habit of saving often, and doing it unconsciously at every pause in my thought process; thus I didn’t have to think hard to predict when the computer was at risk and explicitly decide when I ought to save. Ann was properly appalled. Didn’t the system just automatically save everything I typed? Why was human thought and habit involved at all in a task that could have been fully automated long ago — and indeed had been automated in all but the earliest and most primitive Houyhnhnm computing systems?

Although, she remarked, considering the overall computing system containing both Sapient and Computer, the task had been automated indeed. Indeed, if you came to think of it, this task couldn’t possibly not be automated, unless the computing system were only used but to produce worthless data never worth keeping — at which point it would thus itself be mostly worthless. However, the task had been imperfectly automated at great cost by creating a habit in my brain and hands, rather than automated both perfectly and cheaply by having it done by the computer. Certainly, building a physical habit that lightened the burden on the higher parts of my mind was better than no automation at all, but what a waste of precious wetware! At least in this instance and for this concern, the very purpose of computers had been defeated. As went the Sacred Motto of the Guild of Houyhnhnm Programmers: I object to doing things that computers can do.

And then, suddenly, Ann became worried about her journal. She had been consigning her observations on a computer ever since she had learned to use a mouse to draw Houyhnhnm symbols in a paint application. (Ann once remarked that a one-button mouse is an exquisite input device for a Houyhnhnm’s hoof, but that it takes a Yahoo to believe it is suited to a Yahoo’s hand.) Now, she admitted that she could never retrieve her old notes; but she just thought that it was due to her not understanding the Houyhnhnm-Computer Interface properly, to said interface not having proper automated recognition for hoof-written Houyhnhnm scribblings, and to her not knowing how to search back in time what she had previously drawn. She assumed that Human computers were probably not geared to properly index her observations for easy retrieval, but that of course they would otherwise all be logged in the computer’s memory. Was I implying that all her notes were lost, including some of the finest poetry she had ever written, as inspired both by the suffering from her predicament and the marvel at her discoveries? I won’t claim any proficiency at judging Houyhnhnm poetry — it all sounds like nickering and whickering to me — but to this day, I fear that one of the greatest pieces of Houyhnhnm literature has been lost to the world, due to the failings of Human computer systems.

Orthogonal Persistence

Ann explained to me that Houyhnhnm computing systems make data persistence the default, at every level of abstraction. Whether you open the canvas of a graphical interface and start drawing freely, or you open an interactive evaluation prompt and bind a value to a variable, or you make any kind of modification to any document or program, the change you made will remain in the system forever — that is, until Civilization itself crumbles, or you decide to delete it (a tricky operation, more below). Everything you type remains in your automatic planet-wide backups, providing several layers of availability and of latency — kept private using several layers of cryptography.

Of course, you can control what does or doesn’t get backed up where, by defining domains each with its own privacy policy that may differ from the (reasonable) defaults. The user interface is aware of these domains, and makes it clear at all times which domain you’re currently working with. It also prevents you from inadvertently copying data from a more private domain then pasting it into a more public one; in particular, you only type your primary passwords but in a very recognizable special secure domain that never stores them; and your secondary access keys are stored in a special private domain using stronger cryptography than usual, and also subject to various safety rules to avoid leakage.

Deletion (as opposed to mere de-indexing), while possible, gets more expensive as the data you want to delete gets older: logs, backups and indexes dating back to the deleted change have to be scrubbed and rewritten; the system must triple-check that everything is still in working order after this sweeping change; it must also make sure that the user is ultimately happy with the results, including with whatever might break for other users he knows who might have depended on details of the old history (assuming he shared any of it). Now, when deleting anything but most recent changes, this expensive operation will leave traces that something was deleted, though the details of what was deleted will indeed have been deleted. Of course, deletion doesn’t affect copies other people may have of the data, if you ever shared it; therefore, thou shalt not lightly share thy data, and thou shalt never share any access keys — but that’s true anyway. At least Houyhnhnm systems let you manage your sharing and backup policies in a systematic way, and ensure that everyone can depend on sensible, safe, defaults.

In other words, Houyhnhnm computing systems have orthogonal persistence — and have had it for decades. The adjective “orthogonal” means that the persistence of data is a property of the domain you’re working in, as managed by the system; it is not an aspect of data that programmers have to deal with in most ordinary programs; unless of course they are programmers specifically working on a new abstraction for persistence, which is after all an ordinary program, just in a narrow niche. Regular programmers just manipulate the data with full confidence that the inputs they consume, the code that manipulates them, and the outputs they produce will each remain available as long as the user wants them, with the consistency guarantees specified by the user, as long as the user affords the associated costs.

Actually, ordinary programs don’t know and can’t even possibly know which domain they will be running in, and letting them query those details would be a breach of abstraction, with serious security implications and performance impediments, even assuming for a moment that it wouldn’t otherwise affect program correctness. Therefore, only programs with adequate capabilities can manipulate the persistence and privacy levels of computing domains, except of course to deliberately spawn a subdomain with yet strictly fewer capabilities. The system of course can recognize privacy and performance annotations about authorized programs and automatically distribute the many components of these programs each in a suitable domain.

It is important to maintain full abstraction when keeping the semantics of ordinary programs orthogonal to various concrete aspects of the computing domains: the persistence, privacy, robustness and performance (but also machine word size, endianness, memory layout, physical location of the machine, etc.). This abstraction allows the user to independently specify what domain he wants, and to later change his specification, while the program keeps running. The same abstraction allows the underlying system to independently pick the best suited or cheapest concrete implementation, and to migrate the program to a different underlying machine when the conditions change. And whether migration is prompted by user request, system adaptation, or a change of phase in the execution of the program, the concrete code to run the program can automatically be re-generated to fit the new conditions, so the program may continue running in a new domain implementation, without any interruption in its semantics (though possibly with an observable pause). Thus, the system may optimize away logging and copying in transient computations for which speed matters more than robustness; or it may introduce extra logging and extra copying when debugging existing programs (e.g. enabling Omniscient Debugging for a failed computation); it may automatically introduce synchronization steps in computations performed in lock-step by several redundant machines based on different architectures to ensure detection and elimination of low-level failures (or tampering); or then again it may add layers of encryption between CPU and memory where the user feels paranoid; or it may compile the code to FPGA where performance really matters.

The possibilities are endless, as long as the system maintains full abstraction of program semantics from the underlying implementation, as Houyhnhnm computing systems do. When on the contrary, as in Human computer systems, the code is pegged to a particular implementation, then not only is it practically impossible to migrate a program from one domain to another at runtime, but programs may have to be completely rewritten from scratch before they may even be executed in a domain with slightly different constraints regarding persistence, privacy, performance, etc.

Fractal Transience

Interestingly, on the visible side of the system, successful Human “apps” these days have evolved into offering to users some semblance of persistence: configuration settings, lists of open tabs, documents you manipulate — most user-visible application state, most of the time, seems to be preserved from one session to the next, without the user having to issue any explicit command to “save” anything. Desktop apps still tend to display a counter-productive “recovery” menu at startup, though. And more annoyingly, this apparent persistence still doesn’t cover the most frequent case these days of people typing things: input forms and message boxes in web pages. Also, the “catastrophic” events that may cause data loss include perfectly predictable events, such as the eventual death of each and every piece of hardware and of each and every software project and service-providing business, in a mere matter of years. Yet, satisfied with expectations from this apparent persistence, users can easily be fooled, like Ann was initially, into believing that Human computer systems are just as good as Houyhnhnm computing systems in this regard; and just like Ann, they can be led to believe that failures are due to incompetence on their part, rather than on the part of the computing system developers.

Well, at least, that’s how the Houyhnhnm see things: whether or not you can assign blame to any person in particular for the situation of Human computer systems, this situation is deeply dysfunctional. Actually, the Houyhnhnm also have something to say if you cannot assign personal blame for it — and it doesn’t look like you can: this means that the meta-system for assigning responsibilities itself is also dysfunctional. Why do “vendors” of Human computer systems by and large hoard all the freedom but none of the responsibility when it comes to modifying and maintaining their software so it doesn’t fail catastrophically and betray the customers? This is a clearly dysfunctional process according to Houyhnhnm criteria. Even when these vendors tout themselves as selling “software as a service”, they often hide behind their “Intellectual Property” monopolies to actually make it “rotware as a racket” — they offer bitrotting bad expensive service, oriented towards the vendor’s interests to the detriment of the users’, with no enforceable service level agreement, with no way to extract your data in a state usable by any competing service, with the promise that the service will grow even more inadequate and eventually die (being cancelled, bankrupted, or bungled), yet that you will have to keep paying, and then pay again when you have to leave or be left behind; but you don’t have much choice because patents and other monopolies attract capital and provide disincentive to investment in any competition (if legally allowed at all) or in other services that don’t similarly exclude competition through legal tactics. By contrast, Houyhnhnms individually have full ultimate control over their own machines, and it is based on this control that they enjoy division of labour in delegating software maintenance of most (if they are programmers) or all (if they aren’t) of their systems to competing providers who are held individually liable in case of failure, and aren’t granted monopolies by a centralized privilege-doling entity.

Now, after Ann made this painful first hoof experience of the persistence failure of Human computer systems, she started investigating how Human computer systems implemented persistence, or failed to. And she discovered to her dismay that beneath the veneer of persistence, there was transience at every level she was looking at — not just transience, but fractal transience: this fundamental design difference between Human and Houyhnhnm computing systems is observable at every level of these systems. The user, the programmer, the library developer, the compiler writer, the operating system implementer, everyone, all the time, has to assume the software and hardware layers below him are fragile, supposed to work only in a single computing domain; everyone will in turn provide a similarly fragile and non-transportable device to the users above him. All the manual handling of persistence costs a significant fraction of software development (about 30% of all code written, an IBM study once counted); and if you ever want to make a significant improvement to any component at any level, you pretty much have to rewrite the entire software “stack” above whichever level you are hoping to improve — in other words this requires a significant world-changing event.

And yet, it runs! Ann was in awe that Human computer systems could run at all; they clearly demonstrated some emerging order so powerful that it could survive despite ubiquitous design flaws — or could it possibly be surviving thanks to what to this Houyhnhnm appeared as flaws? Ann decided to pursue her investigations…

Chapter 1: The Way Houyhnhnms Compute

Ngnghm — Sun, 02 Aug 2015 14:56:46 UT

Dear fellow programmer,

I used to think humans wrote software the way they did because they knew what they were doing. Then I realized that they didn’t really know, but adopted ways that seemed to work better than others. Or maybe rather humans were adopted by the ways that best knew how to survive, whether they actually “worked” or not. In any case, I trusted “evolution”, that is, ultimately, other people, to have figured out the best way that software could and should be written. But everything I knew about computing changed when one day I met a Houyhnhnm (pronounced “Hunam”), who told me how things were done in her faraway land. She made me think in terms of computing systems rather than computer systems; and from my newly found understanding, I could see clearly how computing systems could and should be, that today’s (mainstream) Human computer systems aren’t. But mostly, she taught me how to think, by myself, about computing. And so let me take you through my story of computing enlightenment.

Beyond the Sea of Potentiality

This adventurous Houyhnhnm, whose name was Ngnghm (but I call her “Ann”) had heard of a stranger who long ago visited her home country. So the legend said, the traveler, called Gulliver, was a “Human”: a paradoxical creature that looked just like a Yahoo, yet who like a Houyhnhnm (at least to a point) possessed the ability to reason and speak. There were fantastic tales of a planet full of such Humans, as attributed to this Gulliver; and the stories went that in the land of Humans, there were animals known as “Horses” that looked just like Houyhnhnms, yet who like Yahoos couldn’t speak any language and were likely not fully sentient. Ann, immensely curious, had embarked on a journey of discovery to find and visit this fantasy land of Humans and Horses, if it existed at all. But while sailing the Sea of Potentiality, her transdimensional ship collided with débris caused by Human (or was it Yahoo?) pollution — and she was shipwrecked. Now she was stranded onto our plane of existence (actually a sphere). Not being able to communicate in Human language, she was initially mistaken for a wild and dangerous mare; and she had but narrowly escaped being sent to the knacker — or worse, to a government research facility.

By the time I met her through a friend, though, Ann had already learned to read and write our language, albeit imperfectly. She was desperately looking for parts to build a new ship, so that she may some day sail back home. Since I know nothing of transdimensional travel, I instead showed her how to use the Internet to find all the support that mankind could offer her. She was stupefied by how similar yet how different our Human computer systems were from those of the Houyhnhnms; in some way, ours were so much more advanced, yet in other ways they were so desperately primitive. And as she was telling me of how Computing was done amongst Houyhnhnms, I was suddenly reminded of how I had always felt that there had to be better ways to engage in computing, but couldn’t pin point exactly what was wrong. Now I had found a clearer vision of a world I was yearning for — a world I felt like I had lost, though I never had it — and a world that was within reach if only I could build a suitable ship, to sail the Sea of Potentiality and reach the mysterious and enticing land of Houyhnhnm computing.

Some people have accused me of having imagined all this encounter. But my hope is that, after reading my story, you’ll see that it is not only real but necessary, and soon you will start telling other people what you’re now imagining; and eventually, you and I will build the ship with which we will sail together to the land of Houyhnhnm computing.

A Different Point of View

The fundamental difference between Human computer systems and Houyhnhnm computing systems is one of point of view. Houyhnhnms do not possess a different kind of logic, nor mathematics, nor physics; though they have discovered how to travel across dimensions to other potential universes, they do not have quantum computers, logical oracles, or any magic means of computation beyond our own capabilities. However they approach computing in a way that is foreign to us Humans, and that leads to very different results.

Whereas Humans view computers as tools below them to which they give orders and that do their bidding, Houyhnhnms view computing as an interaction within a system around them that extends their consciousness. Humans articulate their plans primarily in terms of things: the logical and physical devices they build (sometimes including tools to make more tools), in the lower realms of software and hardware. Houyhnhnms weave their conversations foremost in terms of processes: the interactions they partake in, that they attempt to automate (including these conversations themselves), which always involves wetware first. In short, Humans have computer systems, Houyhnhnms have computing systems.

You may dismiss all this as dreamy philosophy, empty words without any consequences — I certainly did so at first. Yet the difference in point of view that I am now attempting to distill leads to systems that are organized in very different ways, that are optimized for very different metrics, and that engage users in very different processes, with role delineations according to very different criteria, resulting in a very different variety of artifacts of very different sizes, but most importantly, connected in very different ways. It may all be but just so stories, but stories have consequences.

Simplicity

What made me most aware of this difference was when Ann discovered that, like she was in her own world, I was trained in writing software, and then asked me to demonstrate the working of some Human computing systems, starting with the simplest I could find. So I showed her simple programs I was writing in C; C is a relatively simple programming language with a somewhat familiar syntax; its formal semantics is simple and well defined in theory, and in practice it is a universal programming language capable of doing everything; indeed it is used almost everywhere that Humans have computers. Yet, after she painfully assimilated enough of what I showed her, struggling all the way, her conclusion was that, no, I was obviously not programming in C, and that I couldn’t possibly be programming in C, because C was not a universal programming system at all, but could do next to nothing, and only very inefficiently so. Instead, what I was programming in was not just C, but a C compilation toolchain plus an IDE and an Operating System plus plenty of libraries and utilities, that all together constituted a very large computing system with incredibly complex formal semantics; what more, a large part of the interaction between these components depended on a large number of completely informal semi-conventions about how the filesystem was or wasn’t used by which process, and how these system and user processes themselves were managed. What to Humans looked simple because our point of view focuses on some aspects and neglects others, to the Houyhnhnms was an unmanaged and unmanageable mess because they see things from a different angle.

What Houyhnhnms considered to be a simple system was one that has a short description when you take into account the entire software system, including the compiler, interactive editor, formal verification tools, libraries, operating system, drivers, hardware blueprints, etc., and including the informal conventions used by isolated or cooperating users, or the chaotic lack thereof. C, because its underlying development environment necessitated huge and largely informal support structures, constituted a very complex computing system, even though it looked small and simple once the support system was assumed. Functional programming languages like ML or Haskell yield much simpler systems if you take into account the verification tools and the development process; yet they still neglected entire swaths of what makes a complete computing system, such as IDE, Operating System, persistent storage usage conventions, schema upgrade, etc., and so they ended up being overall still pretty complex.

By Houyhnhnm standards, the simplest Human computing systems, though far from ideal, would be more something like Smalltalk or the other systems built by Alan Kay’s ViewPoints Research Institute, where the description for the entire system, including compiler, IDE, libraries, operating system, drivers, interactive graphical environment, font rendering, etc., all fit in a few tens of thousands of lines of code. Note that FORTH has been used to build complete systems of even smaller overall software size; but being low-level, FORTH relies more on informal design patterns and manually enforced limitations, which according to Houyhnhnm criteria make the resulting system overall more complex, especially so if multiple people are supposed to work on the same system. Still, such simplistic systems make sense for isolated resource-starved programmers.

Why Simplicity Matters

Houyhnhnms certainly don’t restrict themselves to using systems that are simple (according to their metric). But these simple systems do play an essential role in the Houyhnhnm computing ecology: first, they are an essential part of computing curricula, so programmers can get a grasp of all the parts that make a complete system; second, the ways to factor and evolve such systems is also studied by designers and managers so they may think in terms of overall system architecture (including the Houyhnhnm factor, of course); last but not least, they are also instrumental in the bootstrapping process by which more complex systems are built in a way that is auditably secure.

In any case, a change in point of view led to a completely different metric to assess the simplicity of computing systems. It would also change how to judge other qualities of computing systems in general — and thus change the approach to how computing is done and what artifacts it yields. Alan Kay said that A change of perspective is worth 80 IQ points. The way we humans build and use computing devices may have a lot to gain from considering the Houyhnhnm perspective. And next quality of computing systems that Ann made me reconsidered was something as basic as Persistence…

Post-Scriptum: Pronounciation

How do you pronounce “Ngnghm” or “Houyhnhnm”, my friends ask? Mostly, I don’t. You must realize that these are attempted transcriptions of sounds that Houyhnhnms make with their equine mouths. So if you’ve never met a Houyhnhnm, just imagine one of them whinnying in a way that you’d transcribe like that if you had to. Or don’t. Personally, I have stopped trying to mimic the way Ngnghm neighs, and if I have to pronounce one of these two names when talking to a friend, I just call my friend “Ann”, and pronounce “Houyhnhnm” as “Hunam” — like “Human”, but exchanging the “n” and the “m”. Importantly, though, I am careful to avoid the H-o-r-s-e word when talking about Ann and her kin: she deeply dislikes and vehemently objects to being assimilated to these stupid creatures, Horses — just like you probably wouldn’t want to be taken for (and treated as) a Yahoo.