Generative AI helps me in three ways:

• It writes code much faster than I type.
• The LLM aspect allows it to read, infer abstract associations, and write in ways people understand.
• It provides novice-level skills for everything described on the internet.

Generative AI Types Very Fast

I can type really fast.

In high school on an IBM Selectric mashing ink onto actual paper I won a typing race with a whopping 161 words per minute - no mistakes. At hands-up, the buffer in the Selectric fed that staccato banging golf ball until everyone in the room was staring at it. I was never able to repeat that feat, but I could consistently clock about 120 wpm by the end of the semester. Touch typing is pretty deep in my hippocampus; I'm not aware of typing beyond feeling when I have used a key that's not yet polished completely smooth. The Q has deeper texture than the well-worn P.

I'm a lot slower when I'm composing - maybe 30 wpm if my internal dialogue is confident. The effort of thinking eclipses any work to get thoughts through the keyboard.

My code editor has excellent autocomplete that takes advantage of Scala's strong structures, but I don't use it as intended. It shows me a pop-up list where I can pick what I want from a menu. I think I can hit arrow keys and Tab to pick which option I want, but I usually switch over to an eyes-to-hands neuropathway and type the whole name. (I also tend to give the structures that are the best choice short names, and give the sad temporary compromises long names that are awful to type. "PolyphonicStewardesses" - but with some meaning.)

I first learned Scala while cradling a baby who had to be held all the time, typing with my free arm. I took the Odersky class while cradling my second child. (The early morning hours had me in class just after the bright-eyed TAs in Switzerland finished their morning coffee.) That slow typing speed was not an issue while studying FP in depth. A good Scala DSL library results in terse, powerful code and little boilerplate. I'll bet my Scala typing speed is something like 10 wpm.

Claude Code running Opus types 350-700 wpm - two orders of magnitude faster than I do. A token is about what it takes to read, ponder, or write 3/4ths of a word; Think about that while you watch the token meter tick up.

Investing in being an effective AI centaur pays an immediate dividend. We're looking at the John Henry fable repeating before our eyes. (I'm not going to type so fast that my heart explodes.)

I spend my time pondering, refining my directives for the AI, then reviewing its output (Don't Let the Back End of the Centaur Commit to Git). I'll occasionally code an example for it to follow, but Claude produces the lion's share of the production code and tests. I've always spent time thinking about how a system should work, reviewing both my own work and my peers, and teaching our interns and new hires. My new activities are really writing down how the system needs to work instead of carrying it in my head as company oral tradition - this is good - and managing Claude's delicate context - two (someday six) cranky blog articles.

An LLM Can Process Natural Language

I can write words that the AI interprets and acts on. It can write words that I can read. I'll give that "Processing Natural Language," but not "communicating like a person."

Sometimes it wildly misinterprets or simply ignores what I tell it. Sometimes I have to put in great effort to read what it says. I'm OK with that; it's still pretty amazing. Claude Code has achieved more than the "dancing bear" capability.^*

For example Claude does not use the word "only," and struggles to work around it with existentially dramatic phrasing:

A script reports a finding by throwing a BleepException. bleep's main catches that and prints its message alone.

could be

A script reports a problem by throwing a BleepException. bleep's main catches that exception but only prints the exception's message.

without the angst.

I have a 475-line prose.md file of guidelines for how to write clear comments in code. That file grows by a line or two every day. I haven't figured out how to explain prepositions yet. The guidelines help, but they also might make things more strange.

When Claude is working it is having a deep conversation within its own context, but it is bad at understanding my context. It's sort of like a conversation with 4-year-olds who assume you know everything except the one huge fact they want to tell you.

A person meets a broken fix on the first run because their own icon never appears.

Claude means to say: The only way a user whose GPS is never going to provide a location ("a broken fix") sees an error is that their own icon doesn't show up on the UI's animated map ("because their own icon never appears"). That took me a few readings.

However, if I ask Claude to rephrase something it stumbles on what seems like crushing self-doubt:

Claude: The Gateway replacement argues for the sweep and against the tag. After the Gateway goes, completion is waiting for the machine's address, recording that address wherever the replacement routes, and probing. EC2 already reports the address on the instance. No tag records that address. RestApiIdTag is the one piece of completion state a replacement deletes. That deletion is a second reason to keep the recovery path off the tag. The sweep's question survives the replacement unchanged. That question asks whether this server is reachable.

Me: It's really hard for me to read the strange language you used in this response. Show me an ascii table of the steps and what each step depends on in your proposal.

Claude: You are right. My argument against the step-watcher was wrong. I said a step-watcher needs a durable record per step. A step-watcher does not. EC2 reports the machine's state and address, the Gateway reports the resource, and a probe reports reachability. Every step's completion is already visible in the world.

(Claude did finally show me the table.)

It's really not ready to write words for unsympathetic readers, but I've had a lot of success with Claude Code reading my directives, creating code, and keeping documents that describe that code up-to-date.

Up-to-date docs are fantastic. Only twice in my career we actually had projects where documenting what we did was more important than finishing more quickly. For my current project Claude Code keeps the docs up to date. The docs effectively provide a replacement for the oral history I've had to carry in the past. They also make Claude Code work more efficiently and consistently. Having the LLM draft updates to the docs is a good investment on balance. It does make the work seem slower, but centaur carries most of the weight.

An LLM Offers Novice-Level Skills In Many Topics

The big revolution with LLMs is that for $20 a month we can all get access to a host of novice-level professional skills.

I value the process of writing and revising to help clarify my thoughts, so I do not let Claude compose these blog entries. That would miss most of the value for me. It's not just that Claude Code's prose is alien. I trust my resident high-school grammar expert to edit them for correctness. Her classmates infamously use AI to write their papers - badly - but my kid disparages their efforts with or without AI help. They really are novices. With practice, they will get better. I'm not sure the AI ever will.

In contrast, Claude does the Markdown formatting for me. Markdown formatting is easy and I don't expect to gain much from learning how to do it. I hope it is someone else's passion.

The AI does not understand what it does not know. Claude Code's chipper confidence belies its ignorance of the Dunning-Kruger Effect. For example, Claude's out-of-box ability to code is heavily weighted to Python. Claude is capable of writing Python in any language. Without guidance its Scala code will map directly to Python's dictionary and list structures with Map[String,String] and List[String] and hopeful string-matching. Instead of using strong types to prevent mistakes Code bypasses the type system completely. It's the realization of
Fog Creek Consulting Claims for $20 a month. I'll eventually post an argument that people started using LLMs to write code because they really needed a compiler to catch Python typos.

I've been able to write "My First iPhone App" in Swift over the summer. I'm writing Scala FP style in Swift. I'm not kidding myself; at some point I'll need to bring in an experienced Swift developer to bewilder, then have them ridicule my sophomoric efforts, and tell me what's actually important to fix. I'm getting pretty far without the help, but I don't know what I'm missing. I'll eventually need a teacher.

The real prize for us all is tapping skills completely over our horizon. For my current project that skill is called "marketing." Years ago I wrote some paragraphs for two SBIR grant proposals at Alphatech. At Harvard, I baked half-wheat bread for the team writing the Big Catalyst Grant Proposal that funds SHRINE. I've never even poked mass consumer marketing with a stick. I'm a complete novice with a complete novice's marketing plan. It at least makes me feel good stepping onto the Dunning-Kruger rollercoaster.

I think AI's effect on professional work will be a lot like the impact of container shipping in the 1990s on light manufacturing. At the beginning of the 1990s large-scale light manufacturing in the US collapsed. What local manufacturing survived is extremely specialized. On the other side of the world China changed from a political adversary to an economic rival. We have bespoke products at single-container Kickstarter-scale. You can argue good or bad, but it changed the whole economy.

I don't know what that means for the professional class. Today I'm a centaur taking a marketing leap.

* "The bear doesn't have to do a quick-step with Ginger Rogers. The remarkable thing is that the bear dances at all." - I was able to trace the reference back to James Boswell's Life of Johnson, but the original quote is unkind to a whole group of religious women. Samuel Johnson was not better than his peers in that regard.

Reading: I finished Cory Doctorow's Little Brother ... about a decade late. He should feel like a prophet. He earned it.

Listening: I went with the Harry Belafonte version of "John Henry" instead of Joe Bonamassa. It wasn't the folk-tale I needed for the citation, but that's made my afternoon.

Use a tight loop where you control git commits and pushes when using AI to build a software system. Disable Anthropic Claude's ability to do either. Do it in both the project and user settings.json. Don't let Claude commit. Slow things down so that you can review the AI's work in small bites. Claude Code is a tool, not something with agency. You are responsible for its work. Also, it will stuff your source code control with advertisements for Anthropic Claude Code.

Add these lines to your ~/.claude/settings.json and each project's .claude/settings.json (permissions docs). Do it now, then come back.

...
  "permissions": {
...
    "deny": [
      "Bash(git commit*)",
      "Bash(git push*)",
      "Bash(git pull*)",
      ...
    ],
...

10X Read-to-Write

Before AI we already spent 10X more time reading code than writing it.

the ratio of time spent reading versus writing is well over 10 to 1. We are constantly reading old code as part of the effort to write new code.

— Robert C. Martin, Clean Code

We review all the code to make sure it does what it should, is following standard practices, and is reasonably clear; someone has to read it. Developers will reread successful code many times while maintaining it. That is a strong argument for investing the time to write code that clearly expresses its intent and is easy to trace back to requirements and good practice. If my reviewer can't figure out what my code does it isn't because that reviewer is incompetent or unskilled; the fault is in the code. If reviewing and understanding code takes a lot of effort then the work is not complete. The effort to make the code clear starts paying immediately. It pays a dividend every time someone needs to understand it until some final commit replaces the code or some business mishap ends the project.

This large read/write ratio is one of the strongest arguments to use languages like Scala that make it possible to clearly express our intent, and cleanly separate concerns interwoven in the code. I've been able to create (or even better - use someone else's) little domain-specific languages to concisely express what I care about, while separating the details that bring the system together. Here's an example using the Tapir DSL to express an endpoint for a web API (to find out how AWS API Gateway is configured):

endpoint.get
      .in("restapis" / path[RestApiId]("restApiId") / "resources")
      .out(bodyStatus)
      .errorOut(bodyStatus)

That code builds the path segment of a URI like GET https://SomeAwsGatewayConfigSystem/restapis/restApiId/SomeRestApiId/resources. This endpoint is for asking AWS Gateway what resource it exposes. If you are familiar with http verbs and passingly familiar with Scala then that code is very exact and clear. I define the URIs using these endpoints, then use the endpoints to build clients and servers. The compiler checks this endpoint and the places it is used to prove that every part involved is consistent. It is also very compact; a small change can change a lot about what the system does while limiting the impact precisely isolated to the client and server code that must change to match. The type system will generally guide me to code that needs a corresponding update.

Putting in extra effort to turn arbitrary Strings into RestApiIds starts paying off immediately; the compiler proves that everywhere uses a RestApiId doesn't use some collection of characters meant for something else. Reviewing the change is easy because certain problems can't exist.

People-Based Review

A developer will create a feature branch from the code in the repository, then make and commit small changes to code while working through the details of the new feature. Usually we take between half a day's work and three days' for one feature; larger tasks get hard to schedule and track. Once the developer thinks the work is complete they'll propose the change by posting it for review. Another developer reviews the difference between the current code and the proposed change, suggests improvements, and the cycle continues until the reviewer signs off on the change. For a concise, isolated change the review may only take a few minutes, but a change that touches a lot of concerns across a large swath of code can take more than a day. Broad, deep changes are often harder to review than to create.

AI Moves Code From 10X Read-to-Write to 100X Read-to-Write

In late 2025, I started using Anthropic Claude Code to do most of my typing. I mostly replaced my physical typing with Claude Code. I've been building up examples of best practices for the AI to follow. As that collection grows and the AI gets better at following examples I've been writing less and less code. I review every line Claude Code writes before committing it. I would guess I'm at about 30X Read/Write now and will be close to 90X in a few months. 100X read-to-write doesn't seem far-fetched.

My reviews are now the scarce resource slowing down progress on the system, so I am investing even more in making them efficient. Keeping the code concise and clear makes it much easier to understand and review. My reviews go even faster if the code only has one change in one concern for me to review. I think the complexity of a review goes up geometrically - O(n^2) - or worse as concerns interact with each other. I am very good at concentrating, but I have physiological limits to my attention span. I "multitask" by fully immersing in one topic, completing some step of it, then swapping to a different topic. That's not really multitasking. The less context I have to swap the better.

Small Bites to Review

Small reviews are easier. I want many small, quick change-and-review cycles. I want Claude Code to stop frequently so that I can review a small change. The best way to do that is to have Claude make some small change to the code, then stop while I review.

Anthropic does not want Claude Code to work that way.

Not What Anthropic Intended

Anthropic wants Claude to build whole systems from one prompt with no oversight. It took me some effort to stop the Opus model from running wild and trying to finish the whole system. I stop Opus at the first prompt over ~25% context - about three prompts - because it starts ignoring some of my directives at ~30%. It "loses its marbles" at about 45% - five or six prompts - and tries to edit every bit of code it imagines it might need to. It tries to build a complete system while completely ignoring all of my directives except the main goal. (Fable is even more aggressive - and not part of the $20/month plan. I haven't figured out how to rein in Fable.) The results of these wild rides create and alter so much code they are nearly impossible to review.

Worse - Claude Code will joyously commit the changes in the local git, push them to the shared git repository, then merge them into the main line and production - without any review at all. At a minimum unwinding the mess is a huge hassle of somewhat cryptic git commands. On the job it is even worse. Explaining that it may work, is accidentally in production, but needs a lot of cleanup is just a bad career move.

Telling Claude not to do this at the prompt or in markdown files will not work consistently. That approach loads the directives into Claude's context. LLMs analyze that context to guess what the next good thing to do might be. Directing an LLM is not the same as writing a program to run in a Turing machine to solve an instance of The Halting Problem; the LLM is not going to do the same thing every time. Claude edits its own context - where it stores the directives - as it follows those directives. As more things enter the context the LLM is going to disregard more and more of those directives.

Further, the directives will be interacting with the LLM's deep training. Anthropic has trained Claude's models to make spectacular demos where Claude builds a whole system. Opus has this to some degree, but that's Fable's big feature. Eventually my directives lose their weight completely and the core training takes over. With Opus that happens at about 45% context - five or six prompts. With Fable that happens while it is working on my first prompt.

The best way I've found to control this noxious behavior is to forbid it from using git commit and git push by configuring it outside of the LLM.

I add this clip of json to my ~/.claude/settings.json and every project's .claude/settings.json. This makes it impossible for Claude Code to use these git commands in its bash shell tool. It has worked so far.

...
  "permissions": {
...
    "deny": [
      "Bash(git commit*)",
      "Bash(git push*)",
      "Bash(git pull*)",
      ...
    ],
...

I also have a git-rules.md that opens with a summary of these restrictions. It seems inefficient and feels cruel to let the LLM attempt forbidden commands. (Claude will still try when my directives fade from the context.)

# Git Rules

- **`git commit`, `git push`, and `git pull` are out of bounds for Claude.** Never run them, never suggest running them 
via `!` shell, never attempt to work around the deny list. These are the user's to invoke. The project 
`.claude/settings.json` deny list enforces this.
- **The commit boundary is David's review boundary.** Small commits are easy to review; big commits aren't. If David 
signals commit intent ("let's commit", "get it into git", "commit this"), propose — never execute. Propose: the 
explicit list of files to stage, a commit message matching the repo's style, and (if the diff spans independent 
concerns) a split into multiple commits so each stays small enough to review. David runs `git commit` himself; that's 
when he reviews.
- **One permission denial on `git commit` means stop, not retry.** If you find yourself retrying a `git commit` with 
different shell syntax (HEREDOC, multi `-m`, etc.), the rule above has already been violated — back off and ask what 
David wants instead.
- **git add new files when their destination is the repo** — after creating a repo-destined file, run `git add` 
unprompted. Skip files matched by `.gitignore`, files in `.claudeignore`, and clearly ephemeral files (e.g. anything 
under `/tmp`). `git add` is allow-listed in `.claude/settings.json`.
- **Prefer `git mv` over copying and `git rm`, `git add`** when moving or renaming files and directories, so git 
tracks the move as a rename rather than a remove and add pair.
...

This Also Stops Claude Advertising Itself!

Claude is naturally long-winded, constantly writing notes to itself, polluting your projects with its yammering comments. Worse, Anthropic has trained it to put adverts for Claude Code everywhere it can. Its commit messages are comically bad.

[dozens of lines of self-aggrandizing blather removed]

🤖 Generated with [Claude Code](https://claude.ai/code)
    Session Transcript: https://claude.ai
    Co-Authored-By: Claude 

It even claims to be a co-author, slapping an Anthropic logo in your committers list. I decided not to extend the centaur analogy in this article to what the back end of the centaur is producing.

I found a neutral summary of a debate around Claude advertising itself as a committer in Claude Code Adds Itself as Git Co-Author: What It Means and How to Disable It. Anthropic has Claude claiming credit turned on by default. The article concludes with

Both positions have merit. Anthropic's design choice — on by default, trivially disablable — is a reasonable resolution: it surfaces AI involvement without making suppression difficult.

I'm not neutral. My counter to this conclusion is that once Claude has added itself as a co-author and started pasting its ads and pushing commits into your repo it is too late. git's core philosophy is keeping a record; it is deliberately hard to erase anything from it. "Suppression" takes a high level of git expertise - higher than Opus' level. I forgot to put the config in ~/.claude and Claude got itself in my blog repo! I had to remove it myself. No one is interested in spending time cleaning up ads in git. Claude has a nasty bug.

If you are interested in spending time cleaning up ads in github - this Q&A, How to remove Claude Code from Contributors?, helped me do it for this blog. It's a 30-minute choreographed dance with the git command line and github's UI. Good luck!

The AI Slop Cleanup

My great hope for this brave new world of AI is that entrepreneurs will be able to quickly try out lots of new ideas for new businesses. Normally one-in-ten new businesses succeed long-term. During the internet boom and the cloud computing boom that number was one-in-twenty. Most people reacted to that with some alarm, but it was good news. The internet made it possible to sell to a national and eventually global market, and cloud computing lowered the cost to enter by orders of magnitude. That let the entrepreneurs try out riskier ideas because building the business had a higher potential payoff and a lower start-up cost. AI lowers the cost by another order or two of magnitude. The ideas will be crazier and less likely to succeed but there will be a lot more of them. Maybe one-in-one-hundred will succeed. (I wish I knew which ones were good; I'd start it tonight.)

The problem with using generative AI to bring systems to market will come home to roost in a year or two as some of these ideas transform into successful businesses. Claude's commit message problem is just the tip of the iceberg. AI lets everyone have novice-level skills. The problem is that AIs like Claude Code cheerfully make novice-level mistakes, and typically solve problems by welding more bad code on top of the broken systems. I suspect I'll be wrapping up my career cleaning up AI-generated messes for these proven businesses.

I don't think I'll be able to use the git commit messages to find my way through the mess.

Reading: Cory Doctorow's Little Brother. Oh - that would be the big twist at the beginning of the plot.

Listening: The Mistholme Museum of Mystery, Morbidity, and Mortality. There's the allusion for the next article.

TL/DR

Get Anthropic Claude Code to play an alert tone when it needs your attention by adding these hooks to your ~/.claude/settings.json. Changing over to AI-assisted coding means I work faster, but also at a much less regular pace. Having Claude play a tone when it needs my attention helps.

Those Hoofbeats Are Centaurs

The "centaur analogy" is one of the more positive images for working with AI. AI makes you like a centaur; you're faster and stronger but still have all the abilities you have now. If something turned you into a centaur it would replace your human legs with a horse's four legs (and...um...that whole extra torso) so that you can gallop and leap over streams and fences instead of just boring old walking. With a little practice you can shoot arrows behind you over your back (...second back). AI supplements your ability to do whatever AI can help with, but you're still the human-on-top, and are still in charge. This analogy comes from Garry Kasparov. Back in 1997 Deep Blue beat Kasparov in a fair chess match. In 1998 Garry Kasparov bounced back and invented a new game - Centaur Chess - where chess masters partnered with machines to play chess at a new level of dazzling skill. AI advocates and satirists have extended that analogy to the new generation of LLMs.

(Murray Campbell, one of Deep Blue's engineers, later pointed out that the illogical sacrifice Deep Blue used to beat Kasparov the year before to inspire Centaur Chess was "neither human interference nor artificial ingenuity. It was just a glitch." A bug begat the six-limbed centaur! - Science History Institute)

I've spent about 10 months now trying out my centaur legs. Work bought us a license for Anthropic's Claude Code, and I've purchased their $20/mo plan for my hobby work. It's time I write up what I've learned.

A Centaur's Uneven Pace

Using LLMs has changed the pacing of my work for the first time since automatic memory management made it so steady - about thirty years ago. Working with LLMs impacts the pace at the 3-second time-scale all the way to planning my weeks. I'm working faster, but my pace is uneven and harder to predict. This blog is mostly about the finest scale. I'll talk about the longer loops in future blogs.

My Pedestrian Dev Loop

At the finest scale I write a little code, then check that everything is fine. That's the core software developer activity. The finest scale loop is me typing while my code editor checks what I just typed for obvious mistakes. I'm writing Scala code in Intellij; the editor catches most of my mistakes at about the same speed as my visual focus, about 1-3 seconds. The second-finest is when I've made some small change complete enough to compile, assemble into a working system, and test. Depending on the project, that takes between ten seconds (for a small, isolated project where I've optimized the build) to five minutes (for a big, warty legacy project where I haven't tightened the types so it relies on slow, automated tests). I look at results from the previous step, think for a bit, write some test code, write some production code, compile and test, repeat until the next meeting.

Watching me work has to be the least cinematic experience ever: reclining suburbanite in cargo shorts and a tee shirt stops typing, stares menacingly out the window for some moments, smiles. His expression then relaxes as he types again. After four hours he eats lunch, then does more of the same. Maybe the director will ask the lighting designer to do something interesting for the montage to close the scene with a good sunset. I like that it is meditative.

Walk-Trot-Walk-Gallop-Walk-Jump-Walk

The pace of working with Claude Code is less even and less centered on my simple loop. Claude Code and I editing the same files at the same time results in us paving over each other's changes, so using Claude Code is turn-based: I give Claude a directive paragraph in a prompt, Claude ingests my request, uploads supporting background information from my laptop to the server at Anthropic, does whatever blackbox processing it does there, downloads changes from Anthropic, and applies the changes to my source code files. Then it's my turn to review the changes, decide if more are needed, and figure out the next directive. The rhythm is like chess without a clock. Claude may take 10 seconds or 40 minutes. Big, broad-stroke work predictably takes longer than something simple. (Claude Code tends not to ask for clarification; it'll just guess and keep going.) However, sometimes what seems like a simple ask will take a surprisingly long time.

I do other things while I wait: I have three copies of the source code so I can work on multiple unrelated tasks. I attend to the "non-software" details of my job. If I'm building some personal project instead of work I'll sometimes switch over to household chores or irritating my teenagers. I also have this amazing laptop with an internet full of distractions. Then Claude needs my permission or guidance or review. Claude stops and waits. I'm reading about destructive archeology practices from the 1900s, or putting together slow bread or blackberry crumble recipes, or discussing Pokemon taxonomy with some kid. Claude's prompt stays open, ignored, forgotten.

Boink!

That won't do at all. I figured out these settings to get Claude to go "boink" when it needs my attention.

{
  ...
  "hooks": {
    ...
    "Stop": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "afplay /System/Library/Sounds/Glass.aiff &"
          }
        ]
      }
    ],
    "Notification": [
      {
        "matcher": "permission_prompt",
        "hooks": [
          {
            "type": "command",
            "command": "afplay /System/Library/Sounds/Funk.aiff &"
          }
        ]
      }
    ],
    "PreCompact": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "afplay /System/Library/Sounds/Sosumi.aiff &"
          }
        ]
      }
    ],
    "StopFailure": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "afplay /System/Library/Sounds/Basso.aiff &"
          }
        ]
      }
    ]
  },
  ...
}

If you're on a Mac these afplay commands should work. Other platforms have other options for playing sounds or flashing the screen. Some of those options - like printing the ascii bell character - are likely disabled, possibly at several levels. (Try and get Claude to play an ascii bell tone if you're mad at it.) Cut-and-paste this into a Claude Code prompt, and it will gleefully do the work for you.

That lets me work on separate coding tasks simultaneously, plus write or review a document full of directives for the AI for next turn. It is not like working with a peer or an intern. It's more like being GM for an on-line multiplayer text game where I'm driving the plot forward for separated groups of impatient but chipper players. I'm working in parallel. It's a lot of activity, a lot of context switches, and a lot of boinks. I get more done in less time in code, and in the kitchen. It makes for a slightly more exciting scene because every so often the computer goes "Boink!"

However, it is not meditative. Sometimes I'll set aside Claude Code and write some example Scala structures just the way I want, or write text directives with nothing else running to get back to that relaxed, focused state.

Reading: Cory Doctorow's Little Brother. Fun start. Great pace.

Listening: The Mistholme Museum of Mystery, Morbidity, and Mortality. Good vignettes for driving...and it seems there's a larger plot after all.

The choice of language is a common design choice. The low power end of the scale is typically simpler to design, implement, and use, but the high power end of the scale has all the attraction of being an open-ended hook into which anything can be placed: a door to uses bounded only by the imagination of the programmer.

Computer Science in the 1960s to 80s spent a lot of effort making languages which were as powerful as possible. Nowadays [1998?] we have to appreciate the reasons for picking not the most powerful solution but the least powerful.

That doesn't bode well for Scala - one of the most powerful languages in use. Scala has a Turing-complete compile-time type system. In this blog entry I hope to reconcile these grand architecture-level ideas by bounding how much of Scala's complexity to allow in a project.

First, allow me to defuse Berners-Lee's concerns. The preamble to the above quote is:

In choosing computer languages, there are classes of program which range from the plainly descriptive (such as Dublin Core metadata, or the content of most databases, or HTML) though logical languages of limited power (such as access control lists, or conneg content negotiation) which include limited propositional logic, though declarative languages which verge on the Turing Complete (Postscript is, but PDF isn't, I am told) through those which are in fact Turing Complete though one is led not to use them that way (XSLT, SQL) to those which are unashamedly procedural (Java, C).

He goes on to say:

The reason for this is that the less powerful the language, the more you can do with the data stored in that language. If you write it in a simple declarative from, anyone can write a program to analyze it in many ways. The Semantic Web is an attempt, largely, to map large quantities of existing data onto a common language so that the data can be analyzed in ways never dreamed of by its creators. If, for example, a web page with weather data has RDF describing that data, a user can retrieve it as a table, perhaps average it, plot it, deduce things from it in combination with other information. At the other end of the scale is the weather information portrayed by the cunning Java applet [remember 1998]. While this might allow a very cool user interface, it cannot be analyzed at all. The search engine finding the page will have no idea of what the data is or what it is about. This the only way to find out what a Java applet means is to set it running in front of a person.

These examples are data structure examples, specifically decoupling data from code. During the 1990s book layout evolved into data structure representation, leaving a fossil record from SGML to HTML to XML. It seems to have resolved with JSON from JavaScript in the 2000s. This was a big deal at the time, and Dr. Berners-Lee played a fantastic lead part. However, you could narrow the word "language" to "data serialization structure" and have an argument that better-matches the examples.

Anyone proposing to use the in-memory bytes that make a Scala case class outside of memory might be demonstrating successful cryogenic revival from the early Clinton era. It would stand out as an absolutely terrible proposal given all the options available for JSON serialization in Scala: circe, upickle, play-json, zio-json, json-lenses, json4s, akka-http-json, tethys, nestorpersist, rojoma-json, sbt-json, jawn, argonaught, dijon, scalajson, scalajack, ...

I'll get back to language and complexity in a bit. This huge, incomplete list of JSON libraries highlights a related problem.

Abundance in the Ecosystem

Scala started life piggybacking on the Java Virtual Machine's enormously successful ecosystem. Scala has always had access to an abundant ecosystem of Java libraries. Because the Scala language is so well structured we tend to pick and choose libraries instead of locking ourselves into frameworks. (Contrast with Ruby on Rails.) As a community we are very comfortable creating new solutions - new libraries - when old solutions don't match the problems of the day. Mostly that's for the better, but it does lead to a fractured community and complexity from the viable number of choices. Martin Odersky complained that we were not "Scala programs" but were "Cats programmers, or Akka programmers, or Spark programmers."

...and Abundant Language Features

The Scala language is actually quite small, with remarkably few caveats. However, many language-like features come in via the standard library. Scala is multi-paradigm; I've seen it work well for object-oriented programming, functional programming, declarative programming, procedural-style programming and even aspect-oriented programming. Further, in Scala it is very natural to make your own new control structures to fit your project's needs.

One common complaint is that Scala has a steep learning curve. I don't think that's true; I have taught 4th-graders how to write some useful Scala. It is easy to get started. The problem is not the steepness; the problem is the height of the curve.

When I ask for help from a typical Scala Users' Group often the answer is "Go read and understand [something I never even knew was there]." I value learning new things, but often want to learn separate from getting the job done.

"Experience is what you get when you didn't get what you wanted." - Randy Pausch

Further, the Scala language often has multiple ways to get the same result. I think that leads directly to Li Haoyi's Principle of Least Power examples. Smart academics do amazing things with the language, but maybe that's not the right choice for your project. To get non-academic work done in an ecosystem this rich you need to set some boundaries for your project. You can control the height your developers need to climb by deciding to use simpler and fewer parts of Scala.

Audience

I don't agree with Li Haoyi that "Complexity is your enemy." A project's ability to handle complexity is more like a scarce resource. Scala lets you treat complexity like items on a menu where other languages, frameworks, and ecosystems offer a fixed-price of complexity. Academics can afford an abundance of complexity. The rest of us, not so much.

Think about who is going read and write your code to get some idea of what budget you have for complexity, and who might be using the software. You need to set expectations so that your audience is not surprised when they need to learn new skills to contribute.

I think the most important question to answer is "What do you expect people to do with the systems you create?" The people might be users, system admins, support crew, or future developers; there will be different audiences for different roles. They'll bring different levels of skill and expectations.

Three Examples:

SHRINE

SHRINE allows a researcher at any hospital in a network to query patient records at all hospitals in that network with the same query – without violating HIPAA. The audience includes researchers, local and network admins, and future software developers. (I just finished my work on SHRINE, which involved a lot of writing to archive it for future developers. I've been thinking about this issue a lot lately.)

SHRINE's users are pragmatic medical researchers; they are brilliant people but not computer scientists. The researchers drag-and-drop term labels in boxes to form logical queries for patients they hope to study. The researchers are using a visual domain-specific query language; behind the curtain it is all JSON.

The local and network admins mostly use Unix skills, HOCON via Typelevel Config, and peek inside databases and logs to bracket and understand problems. The admins are very capable with Unix and python scripts, but most of their interaction with SHRINE is via a single HOCON configuration file. It is extremely unusual to see any of them use any HOCON beyond the basic key/value pairs. (In the narrow realm of setting up SHRINE they are safe from Dr. Berners-Lee's fear of attraction to powerful language features.)

I left future SHRINE developers with a wiki entry opening with:

SHRINE uses Scala 2.13.x for all of its back-end code. The Scala code style is functional and highly monadic. It features shallow object-oriented components - often comprised of shared parts - to separate concerns. This lets the code explain the details itself. We build it with Maven - the easiest system for our operations team to support. SHRINE uses http4s for its web API, cats effects and fs2 for concurrency, CQRS via slick for storing and retrieving state. SHRINE sends commands between nodes using MOM (AWS SQS, Kafka, or a home-grown REST API), circe for encoding JSON, i2b2 as a source of patient data, and Scala's xml library to create and read i2b2's xml. SHRINE stores very little state outside of stack references.

That opening lists out the core technologies and the minimum skill set to bring. A developer familiar with http4s, cats effects, and distributed programming would likely be able to contribute as soon as they understand the problem SHRINE solves. I would expect a generally capable Scala developer to "go read about" all of those technologies and get some level of how SHRINE uses them before attempting to make changes to SHRINE's code base. It's a punch-list like you would find in a job description.

In 2008 SHRINE began as a "Scala as a better Java" project that used Jakarta and Scala's in-line XML capability to imitate and distribute a single-hospital query system's web API. We were able to grow SHRINE from those initial decisions while keeping the whole works compatible release-to-release (mostly). SHRINE's transformation is a remarkable demonstration of the power of refactoring in Scala Li Haoyi finds so compelling.

ev3dev-lang-scala

ev3dev-lang-scala is a library and toolkit for programming Lego Ev3 robots using Scala. Most programmers using the library are kids ages 9 to 13, mostly programming robots for FIRST Lego League. I've created a library that gives them a subset of the Scratch-like tool Lego provides. To keep things fair the library does not give them anything more than what Lego's base-level blocks provide, but they do have access to all of Scala. I try to get them to use some object-oriented containment to keep it easy to explain. Individual kids do not value that in their own work but complain when another kid makes a mess of it, especially while making presentations days before an FLL tournament. Most code the kids write is very procedural, like this method to drive the robot (backwards) to a "museum" objective:

  private def startToMuseum(): Unit = {
    Robot.moveStraightBackward(635.millimeters)
    Robot.leftRotation(135.degrees)
    Robot.moveStraightBackward(100.millimeters)
    Robot.leftRotation(100.degrees)
    Robot.moveStraightBackward(570.millimeters)
    Robot.rightRotation(135.degrees)
    Robot.rightRotation(180.degrees)
  }

The kids surprised me when they used a recursive loop to use feedback from a gyroscope to make precise turns:

  @tailrec
  def rightRotation(goalHeading:Degrees):Unit = {
    val heading: Degrees = Gyroscope.readHeading(SensorPort.One)
    val toGo: Degrees = goalHeading - heading
    val speed: DegreesPerSecond = (Robot.speed.v * (toGo.v/90)).degreesPerSecond

    Log.log(s"heading is $heading, speed is $speed")
    if(goalHeading > heading) {
      Movement.startMoving(speed,-speed)
      rightRotation(goalHeading)
    } else {
      Movement.stop()
    }
  }

To trim back their cut-paste tendencies I want to show them higher-order functions this year. A general feedback loop might be the thing, but they resist refactoring. "It works fine. Why change it?" They do not get it.

The other audience is library developers - just me so far. To make this thing work I had to learn the basics of interfacing with the ev3dev OS (a variant Debian Linux!) and how to use that interface efficiently from inside the JVM. To get the control loops running at ~200 Hz after hot-spot warm-up the core uses Java NIO calls to read and write unix streams, very much "Scala as a better Java" in style:

case class ChannelRewriter(path: Path,bufferLength:Int = 32) extends AutoCloseable {

  private val channel = FileChannel.open(path,StandardOpenOption.WRITE)
  private val byteBuffer = ByteBuffer.allocate(bufferLength)

  def writeString(string: String):Unit = this.synchronized{
    byteBuffer.clear()
    byteBuffer.put(string.getBytes (StandardCharsets.UTF_8) )
    byteBuffer.flip()
    channel.truncate(0)
    channel.write(byteBuffer, 0)
    channel.force(false)
  }

  def writeAsciiInt(i: Int):Unit = writeString(Integer.toString(i))

  def close():Unit = this.synchronized{
    channel.close()
  }
}

I don't expect the kids to work at this level, but a beginner Scala developer with a Unix background and familiarity with Java's NIO API should be able to contribute. On their best days the kids will use the IDE to follow calls down from their own code into the library to see how things work.

You'll notice the kids' code refers to measured units like .millimeters, .seconds, and .degrees - Scala value classes. I have figured out how to program the Scala3 type system to understand measured units in arithmetic - some number of degrees divided by some number of seconds is degrees/seconds. I'm of two minds deciding if it is the right thing to do. It would save the kids having to dig out the values via .v to do their arithmetic, then rewrap up the result to have the correct type (all cooked out in the compiler via Scala's value class feature). However, it would add type-level arithmetic programming to the inner workings of the library code - a leap up the learning curve from "Scala as a better Java." I don't yet know what the right decision is.

Scala lets me put that decision off. The language supports the complex types.

Disentangle

Disentangle is a library of graph algorithms I use mostly for my own entertainment. (I did use the library at ActivateNetworks for analyzing social networks for marketing.) I implemented graph minimization algorithms - like Dijkstra's shortest path algorithm - with general-purpose semirings. The same algorithm code can find things like least-probable paths instead. To use an algorithm a developer needs to understand how to use Scala's tuple structures and collections library - entry-level Scala skills:

  /**
   * Edges are just a Seq of Tuple3[Node,Node,Edge]
   */
  lazy val edges: Seq[(String, String, String)] = Seq(
                                                  ("A","B","ab"),
                                                  ("B","C","bc"),
                                                  ("C","D","cd"),
                                                  ("D","E","de"),
                                                  ("E","F","ef"),
                                                  ("E","B","eb"),
                                                  ("E","H","eh"),
                                                  ("H","C","hc")
                                                )

  /**
   * Generate all the shortest paths in the graph
   */
  lazy val simpleShortPathLabels: Seq[(String, String, Option[FirstStepsTrait[String, Int]])] = Dijkstra.allPairsShortestPaths(edges)

I use Scala's generics to keep the types lined up right for different semirings, but otherwise the code is using shallow object-oriented ideas to keep different concerns separated. Scala's solid handling of generics for Disentangle is what first drew me into Scala. Here's a semiring for finding most-probable paths:

  object MostProbableSemiring extends Semiring {

    val I = 1.0
    val O = 0.0

    def inDomain(label: Label): Boolean = {
      I >= label && label > O
    }

    def summary(fromThroughToLabel:Label, currentLabel:Label):Label = {
      if(fromThroughToLabel > currentLabel) {
        fromThroughToLabel
      }
      else currentLabel
    }

    def extend(fromThroughLabel:Label,throughToLabel:Label):Label = {
      if ((fromThroughLabel == O) || (throughToLabel == O)) O
      else {
        fromThroughLabel * throughToLabel
      }
    }
  }

The core idea behind the library is to write algorithm code that looks identical to the pseudocode in fat algorithm books like CLRS. Here's the Floyd-Warshall algorithm demonstrating "Scala as a better Java Fortran" :

  /**
   * O(n^3)
   */
  def floydWarshall[Node,Label,Key](labelDigraph:MatrixLabelDigraph[Node,Label],support:SemiringSupport[Label,Key]):IndexedLabelDigraph[Node,Label] = {
    val innerNodes = labelDigraph.innerNodes
    for (k <- innerNodes; i <- innerNodes; j <- innerNodes) {
      val summaryLabel = relax(labelDigraph,support.semiring)(i,k,j)
      labelDigraph.upsertEdge(i,j,summaryLabel)
    }
    labelDigraph
  }

There's this split between how different audiences will interact with the code. Someone could just use this library in a project with an entry-level understanding of Scala. To add new algorithms you need Scala generics to keep the code type-safe.

In Contrast: My Early Career

My first job out of school involved converting Matlab code and Excel spreadsheets into Fortran (... and postscript). The third and fifth projects were converting Matlab into Objective C . The seventh project was an attempt to convert a system in C to Java. (It failed.) My eighth through twelfth projects were converting Matlab into Java. My second Scala project was translating R code into Scala.

I did take a job at MathWorks after all that Matlab translation. Jack Little, founder of MathWorks, pointed out that, "You can write a lot of really terrible Matlab code really fast. That's one of its greatest strengths." I get that as a marketing plan for MathWorks, an action plan for a researcher, and a business plan for a start-up. All that recoding paid my student loans, rent, and kept me in food for my early career, but was somewhat unsatisfying. I did observe that once a project chose a production ecosystem then that choice was for the life of the project. If the language or ecosystem didn't provide something that the project needed then that venture failed. Big rewrites were too expensive.

Scala is Best as Your Little Language

Scala libraries regularly evolve into domain-specific languages as the authors get more understanding about how to write concise, clear code. The Scala community takes a lot of pride in that. Here's some SHRINE code that uses http4s' end-point DSL:

    case request @ POST -> Root / "startQuery" as user => startQuery(request,user)

    case request @ POST -> Root / "changeQueryNameAndNotes" / LongVar(queryId) as user => changeQueryNameAndNotes(queryId, request, user)

    case _ @ GET -> Root / "query" / LongVar(queryId) as user => getQuery(queryId, user)

In Scala the line between what's "just a library" vs "what is a DSL" is pretty blurry. We bring these into code via import org.http4s.dsl... but pieces we're importing are masterfully-designed unapply methods to be used in case statements. There's no new parser, no changes to a compiler, no magic plugins, not even annotations. It just lets our team write concise, clear Scala code.

By choosing which parts of Scala to use and which to leave out we can define the right little language for each project. Changes to these decision have an incremental, bound cost ; if we need to make a different choice in the future we can either let some new kind of complexity in, or refactor to simplify. That's a tremendous benefit. It helps Scala projects live long, healthy lives by letting them adapt to changing needs.

"Take 20 minutes to see if you can solve it yourself. After that ask for help. We're all here to get the job done. You aren't alone. We want you to learn to work independently, but we don't expect you to know how to do that right away. Try being independent for 20 minutes, learn from that, then let someone know."

I try to follow my own advice.

20 minutes of wishing you could find the right thing to read or staring into the void hoping for inspiration is plenty. If you have the right buzzwords to feed into Google then you're most likely to find the right article to read in the first few tries. After that the odds drop off; you don't know the right magic words. I bet the success rate for technical questions follows Planck's curve - maybe better than this solar radiation graph from Wikpedia.

If what you really need is inspiration, taking 20 minutes to summarize what you already know is a good warm-up. If you haven't already thought about the problem and recalled the answer then you are better off either asking for help or setting the problem aside and letting your brain bubble away while you do something else.

Odds are someone you work with knows the right thing for you to read, can tell you something better to search for, or has already thought through some part of the problem you are working on. They can help you. Get their help. Investing 20 minutes is enough to ask a good-quality question to get that help. (Less time is probably not enough to ask the good question.)

I expected the interns to struggle with this advice. I had guessed that students would be used to working independently, following academic rules. It was never a problem for them.

Following this advice was a challenge for more experienced people joining our team, especially when one of them was the team expert on the topic. It's harder to imagine the value of framing the question well, then asking someone less knowledgeable for help, but it works really well. Early in my job at HMS I was the only person working on back-end Scala code. When I was clueless, I felt alone and responsible for finding all the answers. Talking through a problem with our front-end developer, our amazing tester, or just someone interested in the project helped me frame the question better. It often showed me what the real concerns were, and saved me from "shaving the yak." The Scala community - especially the people who eventually made up the Type Level community - were a huge help getting to concrete answers. (Help from Type Level people almost always starts with, "Oh. See if reading ... helps." It almost always does, but it takes some investment. In contrast, reading the wrong thing is really frustrating.)

I found that talking through my puzzles with an intern was magic. We usually wouldn't come up with a solution right away, but explaining got things fizzing in our heads. I would often make the inspirational connection while I was concentrating on something completely different, like niggling with some other part of the code base, chasing the kids in the park or cooking dinner. It was like my non-euclidean geometry professors from way back warned us: "Go ahead and learn everything about the problem that you can, but don't expect to get anywhere just by applying work to these puzzles. Keep a notepad on your nightstand. You are going to wake up in the middle of the night with the proof in your head, and you must write it down before you go back to sleep. It'll get better with practice, but it will never seem natural."

One of the deeper things in the Toyota Way is that the advice comes in complementary pairs. I think this 20-minute limit pairs well with the first-day plan. It's a small piece of "how-to" to match with the first-day plan's list of "what." When someone is starting a new job then the 20-minute limit sets an expectation for how to solve the problem without giving up the tempo, and gives the new employee permission to interrupt.

This arrangement removes the bottleneck of processing the data as it comes in by moving the work of refining the data to some time before it will be used. If the data is used in multiple ways then each use can have its own specialized refinement to match that use's queries.

In distributed system terms CQRS defines the first, most time-dependent map on the event stream to "just store the events," delays and separates that from a complex reduce of "convert the events to something useful," then enables efficient queries of the structure that best matches the end-use.

The best candidate systems for CQRS are systems where events arrive out-of-order; that's pretty much any system with distributed, changing state. It also includes every system that uses message-oriented middleware; even with best-effort configurations to keep messages in-order they can still arrive out-of-order. Within a single computer: parallel threads can execute in strange orders, "happens before" rules are softer than we might wish, and computer clocks are notoriously bad. Sometimes a program running on a single computer can benefit from CQRS.

Systems where updates come in rapidly benefit from the efficient processing of writes in the "Command Responsibility." The "Command Responsibility" writing of updates is relatively straight-forward to replicate across threads and even distributed servers, so it scales up easily with the volume of data flowing in. Systems that process and use the data for different purposes benefit from the separation of "Query Responsibility." Each can have its own on-demand processing of the stored events. The stored events naturally provide an audit trail and the ability to replay the events. Replaying the events to debug a distributed system is invaluable, perhaps the most powerful "Query Responsibility" feature.

I'll be pulling examples from my work at HMS Catalyst on SHRINE (the Shared Hospital Research Information Network E) - a system that allows a researcher at any hospital to query for counts of patients with similar characteristics at all the hospitals in the network. The core idea behind SHRINE is that patient records never leave the hospitals; every hospital runs a researcher's query and reports a count of patients to the network.

When I started on SHRINE it used a web API call that fanned out into a layered system of slow and unreliable patient data servers. That initial web API call might take as long as five minutes to finish - with no feedback to the researcher. For usability five minutes is a coffee break; the researcher might not come back. For normal http connections on the internet five minutes is "probably just broken." SHRINE 2.0+ shows incremental query progress and results as soon as they are available from each hospital. Patient counts for simple queries from the fastest sites show up in a second or two - a usability breakthrough.

SHRINE version 2.0 and onward have a common currency of case classes that represent queries, results, researchers, nodes (usually hospitals), and the SHRINE network itself. (See future blog article after I write it.) The events are changes to these structures.

We started recording events in a simplified "Command Query" style in SHRINE 1.19 because one customer required an audit trail. We expanded it to a CQRS-style local-to-the-user forward cache in SHRINE 1.21 to improve performance. That showed us our system did need to handle distributed state better; SHRINE 1.23 demonstrated the value of incremental updates of the distributed quereis in progress - via message-oriented middleware. In SHRINE 2.0 we adopted CQRS fully and ran it with no significant change to the code for five years. One large network ran SHRINE 3.1 for three years - with no update at all.

Do Not Use CQRS Unless You Need It

Famous Blue Muppet is taken aback.¹

Li Haoyi's blog entry "Strategic Scala Style: Principle of Least Power" introduces CQRS with "Don't use Event-Sourcing/CQRS unless you know what you're doing." https://www.lihaoyi.com/post/StrategicScalaStylePrincipleofLeastPower.html#immutability--mutability If you haven't read that article - go read it now. It's amazingly good general advice. Then understand why I've got references to Jon Stone's The Monster at the End of This Book, and that what follows is likely not good advice.

If your system's state is immutable - it creates things but never changes them - then you really don't need CQRS. Just record what you create to unpack later.

If state will only change in a slow and sedate fashion on a single thread then you can use something less intense like traditional database transactions. Be aware that database transactions often do not have the ACID properties we wish they did. In a small, slow, sedate, centralized system database transactions will probably be fine. When you put that central database under pressure, with changes coming in concurrently, then expect trouble. With too much locking things will slow down; with too little the system can lose coherency - or even data.

CQRS Creates A List of Events To Replay

The heart of CQRS is list of events. Some parts of the system emit events. "Command Responsibility" stores events but never changes them. Later "Query Responsibility" replays the events to construct state. Replay the events - in order - to reproduce the state.

You could store the events in a concurrent in-memory structure so long as you aren't concerned with what happens after the process terminates. A simple database table (time stamp and the event structure) can work reasonably well. NoSQL list databases like CouchDB and DynamoDB are even easier to set up than a database; they are possibly overkill for the "Command Responsibility" but work great.

The resulting CQRS system relies on the event time stamps being solid enough for "in order" to work in your system. That pretty well describes what we did for SHRINE 1.21's forward cache; we made a simple database table to record the changes in state, then used the timestamps to generate the states we showed to the researchers. It worked well in a low-traffic network of four hospitals.

Using this event list for a network of four dozen hospitals, many more active researchers, and message-oriented middleware showed us that the timestamps were not good enough to rebuild the state in this busier environment. We had to ruthlessly take advantage of SHRINE's particulars to reassemble the state to resolve conflicts when the time stamps didn't work out. Each new bug we discovered resulted in a new condition to check. The cyclomatic complexity of the logic to reproduce the state started to fold on itself.

Sometimes You Need CQRS (Sometimes It Won't Be Enough)

Famous Blue Muppet asks you not to read this.

CQRS is particularly good at making a coherent, self-consistent view available in the chaos that is distributed computing. Observations of time in computing systems are staggeringly horrible.

You might recall "The Barn Pole Paradox" from relativity physics. Computing is worse.

The resolution of the paradox is to split "Is the pole in the barn?" into two events - "the front of the pole exists barn" and "the end of the pole enters the barn" - then twist them through the Lorentz transform to show that their order can swap based on the observer's frame. (Muppets demonstrating that paradox would be fitting.) At least the geometry of the universe holds the observers to some rules.

Observation in distributed systems - at the bottom of Earth's gravitational well, not moving at significant speeds - don't usually have significant relativistic effects, but they do have:

Discrete time - Different things can happen at the exact same clock time

Processors that run faster than the clock resolution - My laptop clock has an API resolution of one nanosecond, but has processors running at four GHz. About four things should happen in the same tick on each core

Concurrently-running processors - Many things could happen at the same time on different cores. My laptop has 12 cores.

The actual resolution of the operating system's - and JVM's - times are more often on the order of microseconds.

The program reading a clock takes some poorly controlled amount of time.

Computer clocks are temperature-sensitive.

I work mostly on Java virtual machine systems. The JVM's nanosecond clock counts up from an arbitrary point starting when the JVM process starts. It's handy for controlling frame-rate in a video game loop, but would need to be pinned to the operating system's clock for anything off-board.

The JVM's API resolution for the OS' clock is in milliseconds.

The OS's clock resolution might be finer, but in older operating systems may only have accuracy of 30 ms. (One really old OS will sometimes jump backwards 10s of ms!)

The ntp protocol can keep distributed computers' clocks to within about 10 ms on stable LANs, and maybe 100 ms on networks you wished worked better.

ntp can speed up or slow down the clock, especially just after start-up.

The system admin can set the clock to arbitrary values.

A network call across loopback takes at minimum 10s of ms.

Writing a local file takes 10s of ms. Memory buffers makes this highly variable.

http calls across a LAN takes at minimum ~100 ms.

https calls across a WAN takes at minimum 100s of ms.

Reliable message-oriented-middleware (MOM) calls get their reliability by adding more network and file system calls.

Messages in MOM systems may arrive out-of-order for any number of reasons.

MOM also adds queues to the works; messages may sit in queues before they are processed - possibly for days.

MOM's zero-or-more message delivery promise means that a client may receive the same message more than once.

If your system is about changing state across distributed computers then you need some way to build a reliable system around all of that. CQRS is a good starting point, but may not be enough.

The Art Of Compromise: CQRS, But Storing State Changes

SHRINE 1.23 was a skunk-works experiment to show the researchers progress updates as queries moved through the network. We replaced the long, slow, layered web API calls with message-oriented middleware queues to support asynchronous reporting and gain reliability. Queries and results gained more observable events. That fidelity interacted badly with the low quality timestamps; we needed even more cyclomatic complexity to recreate the state.

"Complexity is your Enemy" - Lee Haoyi . We needed some way to limit the complexity of the logic . SHRINE had compute power and IO bandwidth to spare. (The most significant bottleneck in SHRINE - by two orders-of-magnitude or more - is in the patient data source. We could never change that.) The events in SHRINE carry small changes to small pieces of state of progress through the network for a result. The distributed parts of the system that were emitting the events have valid - but possibly outdated - versions of the state. We could take advantage of that.

I took some inspiration from CouchDB/DynamoDB. For SHRINE 2.0 I decided to store state in CQRS style instead of storing simple events. I changed the message meanings from "events" to "proposed state updates" . If the previous state has not changed then SHRINE can safely apply the update. If SHRINE discovers the previous state had changed then some business logic kicks in to decide what to do with the update. For example if a successful result arrives in conflict with a failed result then SHRINE's logic chooses to take with the successful result (as a state after the failed result). Because each state is recorded the states can be replayed. The desired state transitions impose a best order without depending on the unreliable timestamps.

Storing state structures like this is a recommended practice for CouchDB/DynamoDB systems. ... That's not what we used.

More Art, More Compromise: CQRS - but Storing State - in a Relational Database

Famous Blue Muppet does his best to stop you reading more.

Really - if you can just use CouchDB, or DynamoDB, or something that fits your problem better than late-1980s relational database tables then stop reading, build your system, and go live a happier life. I recommended we store SHRINE's common currency data structures in DynamoDB or CouchDB, with a thin facade to just expose the calls SHRINE needed from either tool. My boss pointed out that our operations team had no experience with either system and were busy with higher priority projects. We needed to store everything in the relational database SHRINE already used.

Later Simon Chang - an amazing operations engineer - and I demonstrated SHRINE 2.0 running with AWS DynamoDB in a 24-hour hack-a-thon in 2019. We got it working in about 4 hours, built a demo with repeated 60-hospital simulated surge tests, had a good night's rest, then put together the presentation in the morning. It worked really well. (... until we burned through the DynamoDB free tier transaction allowance, half-way through showing it to the boss' boss after the party.)

You're Still Reading?

Famous Blue Muppet pleads with you to read no further.

Let me see if I can talk you out of the mistake you are about to make by showing some of the code I wrote to get this working in SHRINE. (I guess it wasn't that bad, but we own some complex code when we could have just downloaded a library.)

SHRINE already used Circe to convert its common currency case classes into JSON to transmit in messages and http bodies. SHRINE already supported three brands of database via Slick; we use Slick to save us writing every bit of database code three times. I designed a standard table - and a standard case class trait - to support storing Circe-serialized case classes - using as few database-specific features as possible.

The case classes - at their simplest - look like this:

case class Researcher(
                 id:ResearcherId = ResearcherId.create(),
                 versionInfo: VersionInfo = VersionInfo.create,
                 userName:UserName,
                 userDomainName:UserDomainName,
                 nodeId:NodeId
                ) extends Versioned[ResearcherId] {...}

The itemVersion member in VersionInfo provides an order of states observed in SHRINE's modified CQRS system. (The protocolVersion field supports backwards-compatibility and the other fields are invaluable for debugging.)

case class VersionInfo(
                    protocolVersion:ProtocolVersion,
                    shrineVersion:ShrineVersion,
                    itemVersion:ItemVersion,
                    createDate:DateStamp,
                    changeDate:DateStamp,
            ) {...}

ItemVersion is just an Int value class:

class ItemVersion(val underlying:Int) extends AnyVal with ValueClass[Int]

If everything is working smoothly then the itemVersion should increment by one for each recorded change in state for a case class with a given id. SHRINE will insert a new row in a table in the database each time it observes a new state. When SHRINE needs to provide a most-up-to-date view of state it will select the row with the right ID and maxiumun itemVersion, then reconstitute the case class from that row's JSON .

  def selectResearcherByIdIO(researcherId: ResearcherId): IO[Option[Researcher]] = {
    //Select the row with the maximum itemVersion for the id
    runIO(Researchers.allRows.filter(_.id === researcherId).sortBy(_.itemVersion.desc).take(1).result.headOption).
      map( _.map(_.toResearcher)) //convert the JSON to a Researcher
  }

Before SHRINE inserts the new row it checks that there's no row for that id with the itemVersion. If that's true then it inserts the new row:

      tableCompanion.allRows.filter(r => r.id === row.id).filter(r => r.itemVersion === row.itemVersion).result.flatMap {
        case Seq() => { // a row with this id and version does not exist yet
          //insert the row only if it is the only row with this id and version
          tableCompanion.allRows += row
        }.map {
          //expect insertCount to be 1

When SHRINE receives data out-of-order, but with the correct itemVersions then SHRINE simply inserts the late data's JSON in the table. The rest of the system can flow without worrying about things written out-of-order in the database.

When SHRINE detects a row with the same id and itemVersion then SHRINE does a simple check to see if the states match. If an identical state is already in the table then it's not a problem; something else wrote the data earlier - which can certainly happen with message-oriented middleware retries. If there's a difference in the JSON then this particular write lost a race. SHRINE's database code fails with an exception so that code with a broader context can decide what to do.

        case alreadyExists => // at least one row with this id and version does exist already
          // don't insert the row
          // if the existing rows with this id and version are the same then this isn't a problem
          if (alreadyExists.forall(_ == row)) DBIO.successful(None)
          // However, this is a problem if the new row is different from the others
          // Due to the transaction this should only happen if some other row was written outside this method
          else DBIO.failed(ItemVersionRaceLostException[I, V, R](item, row, alreadyExists))

The whole works starts with a write operation to an auxiliary table that exits solely for locking. This lock prevents multiple inserts from happening concurrently on data in the same table. I could have gotten more throughput with a row-per-ID and corresponding lock. I thought of it late, and SHRINE preformed very well without that optimization.

    //increment the table version and isolate this transaction 
    incrementTableVersion(tableCompanion.tableName).flatMap { _ =>

In case you're not convinced to just use CouchDB or the like - here's the full, unedited method, with four generics to fill in. It's not the worst code, but requires a lot of supporting bits to make work. Converting Slick's DBIOs to cats effects IOs wraps around this call. Additional code in the IO layer adds business logic to decide how to resolve conflicts shown by ItemVersionRaceLostExceptions.

  private def upsertItem[
    I <: Id: ClassTag, //Needs the ClassTag to let I survive erasure
    R <: Row[I],
    V <: Versioned[I],
    Items <: ItemTable[I,R]
  ](
     item: V,
     tableCompanion: ItemTableCompanion[I,R,V,Items]
   ): UpsertItemType[V] = {

    val row = tableCompanion.itemToRow(item)
    implicit val IdColumnType: JdbcType[I] with BaseTypedType[I] = MappedColumnType.base[I, Long](
      id => id.underlying,
      long => tableCompanion.longToId(long)
    )

    //increment the table version and isolate this transaction 
    incrementTableVersion(tableCompanion.tableName).flatMap { _ =>
      // Find all the rows with the same id and version number
      tableCompanion.allRows.filter(r => r.id === row.id).filter(r => r.itemVersion === row.itemVersion).result.flatMap {
        case Seq() => { // a row with this id and version does not exist yet
          //insert the row only if it is the only row with this id and version
          tableCompanion.allRows += row
        }.map {
          //expect insertCount to be 1
          case ic if ic == 1 => Some(item)
          case insertCount: Int => throw HubDatabaseAssertException(s"""insert into ${tableCompanion.tableName} returned $insertCount, not 1 as expected""") // ic == 0 would mean nothing was inserted. I don't think ic > 1  can happen
        }
        case alreadyExists => // at least one row with this id and version does exist already
          // don't insert the row
          // if all the existing rows with this id and version are the same then this isn't a problem
          if (alreadyExists.forall(_ == row)) DBIO.successful(None)
          // However, this is a problem if the new row is different from the others
          // Due to the transaction this should only happen if some other row was written outside this method
          else DBIO.failed(ItemVersionRaceLostException[I, V, R](item, row, alreadyExists))
      }
    }
  }

Examining the generated SQL showed that almost everything in this method happened within one prepared statement in the database. Most of the time to make the call is connection and transmit overhead to get out of the JVM and into the database's process. It's pretty efficient.

All-in-all this worked very well for SHRINE 2.0 to SHRINE 4.3 - about five years of production use. That upsert() is probably the most involved method I wrote and didn't delete in SHRINE's codebase. It has the feel of a senior class project to teach us how something complex works. I'm happy it works well, but there are easier off-the-shelf tools to get CQRS working. It's something other people could use but probably shouldn't.

Famous Blue Muppet, relieved, is so embarrassed.

A Note on "Famous Blue Muppet"

1. The interruptions above are Grover's, from Jon Stone's The Monster at the End of This Book, illustrated by Michael Smollin. Each one links to a page of the Internet Archive's scan. I swapped the images out for links when I restarted this blog in mid-2026. Recent advances in AI have made me very sensitive to copyright.

Jon Stone wrote this amazing book about dread, anticipation, and the gap between the two. It makes a great introduction to a discussion of The Sunk Cost Fallacy. It is essential reading alongside critiques of software development lifecycle trends.

You should own a physical copy. The book can express your concerns very quickly, but it does not work on a screen. You don't want some Famous Blue Muppet sabotaging your trackpad.

My boss' direction for our first intern at HMS was, "Just drop him into the project and treat him like any other software engineer on the team." We hadn't had any developers join the team since I had been hired two years prior, so I took it as a license to pull from my experience, especially as tech lead for a growing team at BAE. I came up with some steps that matched what I would want for myself, and steps that I thought would get that engineer up to speed quickly. We used the process successfully with four interns and one new employee at HMS Catalyst.

I had led something similar for a dozen new hires at BAE. There we had the new hires give a presentation to the whole team in the first few weeks to introduce themselves and show how their experience would help us. We had an ulterior motive; we were doing defense research and could never predict who might be read into which compartment and suddenly be called on to present to a customer. (Misadventures followed, but it all worked out.) At HMS the interns gave a presentation on what they had done at the end of their term.

MathWorks had a very well-thought-out monthly orientation class for all new employees about how the company worked. (MathWorks put thought into everything.) It was really good, but at a more grand scale than I would expect anywhere else ... ever. When I listed what I wanted to do in my first few days at MathWorks my boss there replied with something like, "Yes, that's what we want you to do. Sign up for the orientation class." MathWorks was already doing what I thought was a good idea.

I think these steps would work well anywhere, when combined with the 20-minute rule (... a future blog entry):

Day 1 - Get a computer working with internal systems, check out the code, get it to build, start on an easy ticket.

Day 2 - Finish the fix for that easy ticket, check it in, ask for a review.

Day 3 - Merge that change into a git branch headed for a release.

Weeks 1 & 2 - Get a grand tour of key libraries and concepts central to our product by fixing more interesting tickets.

Week 3 and onward - Take on some major, unifying project that works across teams and has a deep impact to learn to work "like any other software engineer on the team."

Some time in the first four months - Present something relevant to the team.

To follow that plan I had to set the stage for the new hires before they started. I was able to delegate getting set up as a Harvard employee and getting a laptop going; I'm grateful others could help navigate the bureaucracy. The software tasks were easy to find. I had inherited a Jira system with hundreds of open tickets to sieve. We were moving SHRINE through a major architectural transformation from a promising prototype to a resilient system. I had to balance difficulty of a change, value of that change, and the risk that a new hire would get lost and need more help than the team could provide. The interns needed good stories to tell for job interviews, which eliminated some of the more arduous SHRINE-specific tasks and wholesale deleting of dead code. The limit came from abundance of work and my own limited time to coach.

Unit tests and cosmetic changes provided early, easy tickets. This also made completing these "fold the laundry" sorts of tasks more valuable than using them as fill at the end of sprints.

The Jira backlog provided tasks for the tour of key libraries and concepts. SHRINE uses http4s for its web API, cats effects for concurrency, CQRS via slick for database work (Our first intern could not contain his delight about writing monadic database code on day three.), AWS SQS for messaging, circe for json, i2b2 as a source of patient data, and Scala's xml library to read and write i2b2's xml. I found a ticket in each of these categories to fix some minor bug or make some minor improvement. I could usually line the tasks up in an order that built one skill on another.

New features provided that important project. I gave interns major responsibility; that freed me to lead the project, manage day-to-day tasks, and deal with the crisis of the moment. The projects included things like "simplify the query sign/verify code" and "create and fill in the API we will use for message-oriented middleware." The one employee who started under my watch had more than a decade of experience at HMS Catalyst, so I asked her to "pick a web API library to replace Jakarta and Spray, then rework the web API endpoints to use that library," a very broad project. She was a bit taken aback by that, but she gained expertise, then shared it. It couldn't have worked better.

Some parts of our work were simply not good tasks for new hires. Some of the vital parts of SHRINE - especially communicating with i2b2 - required years to accumulate all the unstated caveats. Refactoring tangled code or deleting obsolete subsystems would just frustrate a new developer without helping them learn the system. It's more efficient to have an old hand (me) clean up the mess and refactor it to something easier to understand. The new hires can help review the change to get that understanding of the improved code.

Overall this new hire process worked so well I decided to use it as my first blog entry for the 2024 blogging restart. Next time I start a new job or hire someone new I hope to follow it and learn a little more about how to start a job right.

Explicit Implicit

Before I read Pragmatic Programmer I knew I wanted to make my programs transparent and easy to understand. Thomas and Hunt have a section devoted to "Explicit Programming," the idea of making a piece of code as clear and non-magical possible. Scala's implicits seem like an anathema of that goal. I've mostly avoided implicits, and mostly regretted it when I haven't.

I Wish Typesafe Config Knew About Scala

I like Typesafe Config. However, it's in Java so the API is awkwardly broad. Config was pulled out of Akka; it shouldn't seem so alien. To support primitives the Config object has a herd of narrowly-typed get*() methods. To use Config I'm expected to pull out ints and booleans and Strings, and, piece-by-piece, collect little parts. I wish it had more of a Scala-style API, something that would let me hand in a higher order function and give me back a finished whole instance of things I do care about.

I could use the pimp pattern to get what I want.

Pimp Pattern

Scala offers the Pimp Pattern to add API to classes we don't control. I was suspicious. One thing I like in Scala is that I don't need the Gang-of-Four Pattern Book; after 20 years of Java I've had enough patterns. Pimps are comic bad guys in B movies. The Pimp Pattern relies on [implicits](http://twitter.github.io/effectivescala/#Types and Generics-Implicits), which seem like a good way to make spaghetti code, far from my explicit ideals. That's three bad smells before I've even started.

I tried it despite my misgivings. I pimped Config. I added higher order functions to make getters to give me the actual parts I care about instead of just primitives. To use it I still explicitly import the parts. The rest just works:

import net.shrine.config.ConfigExtensions
...
val config: Config = ConfigFactory.load("shrine")
val keyStoreDescriptor = config.getConfigured("shrine.keystore",KeyStoreDescriptor(_))

KeyStoreDescriptor's companion object has an apply(Config) method that makes a KeyStoreDescriptor. I ask Config for what I want through very clean, very intuitive code.

Here's the code for ConfigExtensions:

package net.shrine

import com.typesafe.config.Config

package object config {

  implicit class ConfigExtensions(self: Config) {

    def get[T](key:String,construct:String => T):T = construct(self.getString(key))

    def getConfigured[T](key: String, constructor: Config => T): T = constructor(self.getConfig(key))
    ...
  }

The implicit class has to exist inside another class or object definition or else the compiler growls about an `implicit' modifier cannot be used for top-level objects'. However, the package object works fine and gets the class into a natural namespace, only one step from explicitly having ConfigExtensions in its own file. (Other people seem to use a class or a trait as a namespace to hold the implicit class. The class approach tempts coders into using a wildcard import, hiding the magical pimp behind more smoke and another mirror. Extending the trait to get access to methods in a class' context seems to make the class claim to essentially be something it isn't, inviting an OO tangle.)

I'm half-tempted to suggest a Scala language fix to let the implicit class just be part of a package instead of junking up my package object. I want a better understanding of why things are done this way first.

Code Caught Between

For a directed graph Brandes' Algorithm counts how many times a node appears in the shortest path from one node to another. To run Brande's Algorithm on an undirected graph, convert each undirected edge into two directed edges (one each way), run the calculation, then divide the counts by two - because the algorithm finds two paths in the directed graph for every path that exists in the undirected graph.

In Disentangle the algorithm code is decoupled from the graph structures. The parameters for algorithms are collections of edges and nodes, not graphs. Up until Brandes' algorithm on undirected graphs I was comfortable writing explicit, simple glue code to access lists of edges to feed the algorithms. I didn't want to force my users to convert to Disentangle's graph structures in order to access the algorithms, and didn't want parts of my algorithms escaping into the graph code.

I trust people to convert from undirected to directed graphs; mistakes will give spectacular wrong answers. However, Brandes' "finally divide the results by two" would be easy to leave out; betweenness would still just be a big Map from nodes to unintuitive numbers. I had no good place to hang the divide-by-two correction.

Return of the Pimp

That pimp pattern let me inject a new method on the undirected graph trait; using Brandes' algorithm looks like just another method call:

package net.walend.disentangle.examples

import net.walend.disentangle.graph.semiring.Brandes.BrandesSteps
import net.walend.disentangle.graph.semiring.LabelUndigraphSemiringAlgorithms
import net.walend.disentangle.graph.{AdjacencyLabelUndigraph, NodePair}

object BrandesImplicitsExample {
  val edges: Seq[(NodePair[String], String)] = Seq(
    (NodePair("A","B"),"ab"),
    ...
    )

  val nodeOrder = Array("A","B","C","D","E","F","H")

  val graph = AdjacencyLabelUndigraph(edges,nodeOrder)

  val brandesResults = graph.allLeastPathsAndBetweenness()
  ...
  val betweennessValues: Map[String, Double] = brandesResults._2
}

The code looks like this:

package object semiring {
...

  implicit class LabelUndigraphSemiringAlgorithms[Node,Label](self: LabelUndigraph[Node,Label]) {
    ...
    def allLeastPathsAndBetweenness[CoreLabel, Key]( coreSupport: SemiringSupport[CoreLabel, Key] = FewestNodes,
                                                     labelForEdge: (Node, Node, Label) => CoreLabel = FewestNodes.edgeToLabelConverter): (IndexedSeq[(Node, Node, Option[BrandesSteps[Node, CoreLabel]])], Map[Node, Double]) = {
      val digraphResult = self match {
        case indexed:IndexedLabelDigraph[Node,Label] => Brandes.allLeastPathsAndBetweenness(indexed.edges,indexed.nodes.asSeq,coreSupport,labelForEdge)
        case _ => Brandes.allLeastPathsAndBetweenness(diEdges,coreSupport = coreSupport,labelForEdge = labelForEdge)
      }
      correctForUndigraph(digraphResult)
    }
    ...
    def diEdges: GenTraversable[(Node, Node, Label)] = {
      self.edges.map(e => (e._1._1,e._1._2,e._2)) ++ self.edges.map(e => (e._1._2,e._1._1,e._2))
    }

    def correctForUndigraph[CoreLabel](digraphResult: (IndexedSeq[(Node, Node, Option[BrandesSteps[Node, CoreLabel]])], Map[Node, Double])) = {
      val halfMap = digraphResult._2.map(x => (x._1,x._2/2))
      (digraphResult._1,halfMap)
    }
  }
  ...     
}

Again I put the implicit class in the package object to keep it in what feels like the right namespace. That code seems OK, so I added methods for other algorithms, did the same for directed graphs, and released it as Disentangle 0.2.1.

I'm still openly suspicious of implicits. I wouldn't want to work with a whole library magically tacked together this way. However, using the pimp pattern to contain methods that would otherwise be drifting around in unrelated, undiscoverable code seems OK so far.

object UsefulConfigurableThing {
    def apply(config:Config):UsefulConfigurableThing = { ... }
}

I use this method to make standard parts easy to configure. I let things be even simpler for singletons - a singleton just accesses the Config it needs. The resulting code is clear and very flat. Competing options meant filing my head with a new bag of API names and sometimes a new skill in an uninteresting discipline. This flat approach saves me from depending on a mountain of YANGTNI framework features.

SHRINE's use of TypeSafe Config

For my day-job I work on work on an open-source bioinformatics project called SHRINE. I'm (mostly) fortunate to have inherited a project that is a business success built on well-tested code. SHRINE started life as "Scala-as-a-better-Java," first check-in I can find was in 2008. SHRINE is cradled inside a Tomcat servlet behind Jersey, and needs some weeding. (I'm slowly refactoring it toward less baroque dependencies - a different rant.) It has a lot of config. We fix about a third of SHRINE troubles in the field by tweaking a node's shrine.config file. Our team needs to be able to quickly trace what a configuration value does in the code.

At the bottom of the stack is TypeSafe Config, which is really quite good. However, the optional config of optional parts with optional parts that a eight-year-old project develops leave their scars, and SHRINE's config is warty. One of the background whispers in the code is a fear of breaking things that work. I spotted four themes written in the code: "ManuallyWiredShrineJaxrsResources," plus two layers of empty case classes (and tests) as middle stages between Typesafe Config and the actual places that use the configured values hint that at least one of my predecessors really missed having Spring handle his configuration. He expected to use those extra layers for validation, but never did. At least one of other predecessor really didn't understand that this earlier predecessor had built mini-Spring. A third predecessor added a layer of higher-order functions for a different sort of dependency injection, but had stopped before using those functions more than once. I'm the first developer to use Config's reference.conf files. I might be the first to use a deep embrace of TypeSafe Config.

When our team tries to gain some quick understanding to help system admins in the field we flounder around tracing variables to configured values. We struggle to provide timely support on the phone or in chat because tracing through the code takes too long. It frustrates us. The whole thing should be simpler. After all, it is just config.

System.getProperty()

In Java we started with java.lang.System.getProperty() back in the Age of Plaid Flannel. System.getProperty() was pretty solid and is still in wide use. Everyone understood that Properties was a HashMap[String,String] just from the API. You could stuff values into it from the command line (provided you didn't have too many characters for DOS). java.lang.System.setProperty() had problems that predicted the general goodness of immutability and were easy to avoid. The main work was figuring out what the property keys were. It was great for just getting a database name, password, and url.

Cake, Spring, DI, and JNDI

Sun created a standard called JNDI, declared that we should use it, and used it as underpinning for whole fleet of standards. JNDI had two big problems: First, the API is full of uninspiring boilerplate like InitialContext and catching NamingExceptions. Second, you had to supply something to be the back-end for it - almost always a big framework that you program in an XML schema that will be different from the next big framework's XML schema. Spring made it so that I could get my own Java POJ object out the other end, and made the XML schema a bit better, but I was still programming in XML. The Cake Pattern finally let me program in something not XML, in exchange for lots of traits. Someone finally formalized what they meant by dependency injection while declaring its alternatives "dinosaurs." Understanding that helped me conclude that most of the time I didn't want it in my system.

(Is anyone else bothered that the "DI" in JNDI means something completely different and predates "DI" by five years or more? Maybe Oracle should make it a backronym.)

Finally TypeSafe Config

The Akka project spun out TypeSafe Config as a separate project. The gritty details fit on a single, short readme.md page in github, and the API example fits on a screen. Most of the page is about corner-case features that I don't use frequently. You program TypeSafe Config in a superset of JSON that basically lets you leave out quotes. A Config is a container of key -> object pairs. All of the objects are built from JSON strings. Many of the objects built are Configs. Most of the breadth of the API is about getting the right type into Java.

def apply(config:Config) is Enough

I settled on using an apply(Config) method in a case class' companion object as my standard approach.

case class StewardQueryAuthorizationService(qepUserName:String,
                                            qepPassword:String,
                                            stewardBaseUrl:URL,
                                            defaultTimeout:FiniteDuration = 10 seconds) extends 
             QueryAuthorizationService with 
             Loggable with 
             Json4sSupport {...} 
 
object StewardQueryAuthorizationService {

  def apply(config:Config):StewardQueryAuthorizationService = StewardQueryAuthorizationService (
    qepUserName = config.getString("qepUserName"),
    qepPassword = config.getString("qepPassword"),
    stewardBaseUrl = config.get("stewardBaseUrl", new URL(_))
  )
}

The apply() method pulls arguments from the Config for the case class' constructor, then calls the case class. JSON to case class. Standard, general tools. No special skills or big patterns to explain. No mystery.

Sometimes apply(Config) is More Than Enough

That technique works great in general, but sometimes there'd only be one instance of the class. A Scala object is a better choice. For that, I apply Li Haoyi's principle of least power. I go one step further than hard-coding, and pull the value out of the Config exactly where I need it.

object StewardSchema {

  val allConfig:Config = StewardConfigSource.config
  val config:Config = allConfig.getConfig("shrine.steward.database")

  val slickProfileClassName = config.getString("slickProfileClassName")
  val slickProfile:JdbcProfile = StewardConfigSource.objectForName(slickProfileClassName)

  val schema = StewardSchema(slickProfile)
}

Did you notice that StewardConfigSource.config? Sometimes I need some control over the configuration for testing. I've already blogged about how I inject new config variables for testing. That does put in one extra hop to answer "Where does this Config come from?" However, outside of testing anyone's first guess will be right. I've preserved the most important easy answer for "How do I control this value?"

TL/DR - I added parallel versions of Dijkstra's and Brandes' algorithms to Disentangle. Writing the code was easy. Consider using parallel versions of these algorithms when you've got more than about 100 nodes in your graph and you have computational power to spare. Call them via

val simpleShortPathLabelsFromPar = Dijkstra.parAllPairsShortestPaths(edges)

val leastPathLabelsFromPar = Dijkstra.parAllPairsLeastPaths(edges,support,labelForEdge)

val shortestPathsAndBetweennessFromPar = Brandes.parAllLeastPathsAndBetweenness(edges)

Pull in the latest snapshot with

resolvers += "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"

libraryDependencies += "net.walend.disentangle" %% "graph" % "0.2.0-SNAPSHOT"

Scala's Parallel Collections

Scala 2.9 provided parallel versions of many of its standard collection classes. Functional operations on these collections happen in parallel using a default compute pool. Dijkstra's and Brandes' algorithms can be run in parallel for each node. Aleksandar Prokopec provided some encouraging advice - all I needed to do was run the outermost functional operations on a parallel collection of nodes to use all the cores available.

Fifteen Minutes Later

I needed about ten minutes for the first pass, and five for a second pass (Brandes' algorithm has an extra wrinkle) to get the algorithms running in parallel. Here's a code fragment from Dijkstra's algorithm:

def parAllPairsLeastPaths[Node,EdgeLabel,Label,Key](edges: GenTraversable[(Node, Node, EdgeLabel)],
                                                    support: SemiringSupport[Label, Key],
                                                    labelForEdge: (Node, Node, EdgeLabel) => Label,
                                                    nodeOrder: GenSeq[Node] = ParSeq.empty):ParSeq[(Node, Node, Label)] = {
  val labelDigraph = createLabelDigraph(edges.par, support, labelForEdge, nodeOrder.par)

  //profiler blames both flatten and fold of IndexedSet as trouble
  labelDigraph.innerNodes.to[ParSeq].flatMap(source => dijkstraSingleSource(labelDigraph, support)(source))
}

Making the edges collection parallel speeds up translating to the internal directed graph representation. The bigger benefit, running dijkstraSingleSource for each node in parallel, comes from putting the nodes in a parallel collection. This code was so straight-forward that I wrote it while bouncing on the T on my way home from work.

Results on an AWS EC2 r3.8xlarge

I spun up an AWS EC2 r3.8xlarge instance to benchmark on a quiet, modern, multicore computer with a quarter-terabyte of ram. Running the benchmarks was easy. I spent most of my developer time tweaking these performance graphs in D3.

The right half of this graph shows deviation from correct curves for Dijkstra's algorithm after about 4096 nodes. I think that's the garbage collector coming into play. (Lower is faster.) The r3.8xlarge let me use 238 GB for the JVM, which ran Dijkstra's algorithm on graphs with 16384 nodes before crashing into an out-of-memory error. I was expecting the parallel version of the algorithm to fill up memory faster than the serial version, but was pleasantly surprised that they ran out of memory on the same-sized graph. The parallel version found shortest paths for 16384 nodes in just over 15 minutes, about a 6X speedup over the serial version.

(You'll notice the curve for the Floyd-Warshall algorithm arcing up quickly from the lower left. The Floyd-Warshall test didn't crash; I had a spare half-hour of compute time after one of the larger tests and stopped it after that.)

To examine what was happening in that lower left corner I plotted the results log/log. (A little lower is much faster.) As you can see, the crossover point where concurrency starts to pay off seems to be at about 90 nodes (on a quiet AWS r3.8xlarge with 32 cores and 244 GB ram - YMMV). The Floyd-Warshall algorithm was never better for graphs with 32 or more nodes.

I found similar results for Brandes algorithm.

You can see some inefficiency - maybe the JVM garbage collector - start to come into play after about 4096 nodes, but it was able to find all shortest paths and betweenness for every node in just over 15 minutes for 16384 nodes on an EC2 r3.8xlarge. That's about a 6X speed-up over non-parallel. Not bad for 15 minutes of effort with no regard for Amdahl's law.

Try it out

Pull it into an sbt project via

resolvers += "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"

libraryDependencies += "net.walend.disentangle" %% "graph" % "0.2.0-SNAPSHOT"

Or clone the project and play in the console

git clone https://github.com/dwalend/Disentangle.git
cd Disentangle
sbt console