TL/DR

Get Anthropic Claude Code to play an alert tone when it needs your attention by adding these hooks to your ~/.claude/settings.json. Changing over to AI-assisted coding means I work faster, but also at a much less regular pace. Having Claude play a tone when it needs my attention helps.

Those Hoofbeats Are Centaurs

The "centaur analogy" is one of the more positive images for working with AI. AI makes you like a centaur; you're faster and stronger but still have all the abilities you have now. If something turned you into a centaur it would replace your human legs with a horse's four legs (and...um...that whole extra torso) so that you can gallop and leap over streams and fences instead of just boring old walking. With a little practice you can shoot arrows behind you over your back (...second back). AI supplements your ability to do whatever AI can help with, but you're still the human-on-top, and are still in charge. This analogy comes from Garry Kasparov. Back in 1997 Deep Blue beat Kasparov in a fair chess match. In 1998 Garry Kasparov bounced back and invented a new game - Centaur Chess - where chess masters partnered with machines to play chess at a new level of dazzling skill. AI advocates and satirists have extended that analogy to the new generation of LLMs.

(Murray Campbell, one of Deep Blue's engineers, later pointed out that the illogical sacrifice Deep Blue used to beat Kasparov the year before to inspire Centaur Chess was "neither human interference nor artificial ingenuity. It was just a glitch." A bug begat the six-limbed centaur! - Science History Institute)

I've spent about 10 months now trying out my centaur legs. Work bought us a license for Anthropic's Claude Code, and I've purchased their $20/mo plan for my hobby work. It's time I write up what I've learned.

A Centaur's Uneven Pace

Using LLMs has changed the pacing of my work for the first time since automatic memory management made it so steady - about thirty years ago. Working with LLMs impacts the pace at the 3-second time-scale all the way to planning my weeks. I'm working faster, but my pace is uneven and harder to predict. This blog is mostly about the finest scale. I'll talk about the longer loops in future blogs.

My Pedestrian Dev Loop

At the finest scale I write a little code, then check that everything is fine. That's the core software developer activity. The finest scale loop is me typing while my code editor checks what I just typed for obvious mistakes. I'm writing Scala code in Intellij; the editor catches most of my mistakes at about the same speed as my visual focus, about 1-3 seconds. The second-finest is when I've made some small change complete enough to compile, assemble into a working system, and test. Depending on the project, that takes between ten seconds (for a small, isolated project where I've optimized the build) to five minutes (for a big, warty legacy project where I haven't tightened the types so it relies on slow, automated tests). I look at results from the previous step, think for a bit, write some test code, write some production code, compile and test, repeat until the next meeting.

Watching me work has to be the least cinematic experience ever: reclining suburbanite in cargo shorts and a tee shirt stops typing, stares menacingly out the window for some moments, smiles. His expression then relaxes as he types again. After four hours he eats lunch, then does more of the same. Maybe the director will ask the lighting designer to do something interesting for the montage to close the scene with a good sunset. I like that it is meditative.

Walk-Trot-Walk-Gallop-Walk-Jump-Walk

The pace of working with Claude Code is less even and less centered on my simple loop. Claude Code and I editing the same files at the same time results in us paving over each other's changes, so using Claude Code is turn-based: I give Claude a directive paragraph in a prompt, Claude ingests my request, uploads supporting background information from my laptop to the server at Anthropic, does whatever blackbox processing it does there, downloads changes from Anthropic, and applies the changes to my source code files. Then it's my turn to review the changes, decide if more are needed, and figure out the next directive. The rhythm is like chess without a clock. Claude may take 10 seconds or 40 minutes. Big, broad-stroke work predictably takes longer than something simple. (Claude Code tends not to ask for clarification; it'll just guess and keep going.) However, sometimes what seems like a simple ask will take a surprisingly long time.

I do other things while I wait: I have three copies of the source code so I can work on multiple unrelated tasks. I attend to the "non-software" details of my job. If I'm building some personal project instead of work I'll sometimes switch over to household chores or irritating my teenagers. I also have this amazing laptop with an internet full of distractions. Then Claude needs my permission or guidance or review. Claude stops and waits. I'm reading about destructive archeology practices from the 1900s, or putting together slow bread or blackberry crumble recipes, or discussing Pokemon taxonomy with some kid. Claude's prompt stays open, ignored, forgotten.

Boink!

That won't do at all. I figured out these settings to get Claude to go "boink" when it needs my attention.

{
  ...
  "hooks": {
    ...
    "Stop": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "afplay /System/Library/Sounds/Glass.aiff &"
          }
        ]
      }
    ],
    "Notification": [
      {
        "matcher": "permission_prompt",
        "hooks": [
          {
            "type": "command",
            "command": "afplay /System/Library/Sounds/Funk.aiff &"
          }
        ]
      }
    ],
    "PreCompact": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "afplay /System/Library/Sounds/Sosumi.aiff &"
          }
        ]
      }
    ],
    "StopFailure": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "afplay /System/Library/Sounds/Basso.aiff &"
          }
        ]
      }
    ]
  },
  ...
}

If you're on a Mac these afplay commands should work. Other platforms have other options for playing sounds or flashing the screen. Some of those options - like printing the ascii bell character - are likely disabled, possibly at several levels. (Try and get Claude to play an ascii bell tone if you're mad at it.) Cut-and-paste this into a Claude Code prompt, and it will gleefully do the work for you.

That lets me work on separate coding tasks simultaneously, plus write or review a document full of directives for the AI for next turn. It is not like working with a peer or an intern. It's more like being GM for an on-line multiplayer text game where I'm driving the plot forward for separated groups of impatient but chipper players. I'm working in parallel. It's a lot of activity, a lot of context switches, and a lot of boinks. I get more done in less time in code, and in the kitchen. It makes for a slightly more exciting scene because every so often the computer goes "Boink!"

However, it is not meditative. Sometimes I'll set aside Claude Code and write some example Scala structures just the way I want, or write text directives with nothing else running to get back to that relaxed, focused state.

Reading: Cory Doctorow's Little Brother. Fun start. Great pace.

Listening: The Mistholme Museum of Mystery, Morbidity, and Mortality. Good vignettes for driving...and it seems there's a larger plot after all.

The choice of language is a common design choice. The low power end of the scale is typically simpler to design, implement, and use, but the high power end of the scale has all the attraction of being an open-ended hook into which anything can be placed: a door to uses bounded only by the imagination of the programmer.

Computer Science in the 1960s to 80s spent a lot of effort making languages which were as powerful as possible. Nowadays [1998?] we have to appreciate the reasons for picking not the most powerful solution but the least powerful.

That doesn't bode well for Scala - one of the most powerful languages in use. Scala has a Turing-complete compile-time type system. In this blog entry I hope to reconcile these grand architecture-level ideas by bounding how much of Scala's complexity to allow in a project.

First, allow me to defuse Berners-Lee's concerns. The preamble to the above quote is:

In choosing computer languages, there are classes of program which range from the plainly descriptive (such as Dublin Core metadata, or the content of most databases, or HTML) though logical languages of limited power (such as access control lists, or conneg content negotiation) which include limited propositional logic, though declarative languages which verge on the Turing Complete (Postscript is, but PDF isn't, I am told) through those which are in fact Turing Complete though one is led not to use them that way (XSLT, SQL) to those which are unashamedly procedural (Java, C).

He goes on to say:

The reason for this is that the less powerful the language, the more you can do with the data stored in that language. If you write it in a simple declarative from, anyone can write a program to analyze it in many ways. The Semantic Web is an attempt, largely, to map large quantities of existing data onto a common language so that the data can be analyzed in ways never dreamed of by its creators. If, for example, a web page with weather data has RDF describing that data, a user can retrieve it as a table, perhaps average it, plot it, deduce things from it in combination with other information. At the other end of the scale is the weather information portrayed by the cunning Java applet [remember 1998]. While this might allow a very cool user interface, it cannot be analyzed at all. The search engine finding the page will have no idea of what the data is or what it is about. This the only way to find out what a Java applet means is to set it running in front of a person.

These examples are data structure examples, specifically decoupling data from code. During the 1990s book layout evolved into data structure representation, leaving a fossil record from SGML to HTML to XML. It seems to have resolved with JSON from JavaScript in the 2000s. This was a big deal at the time, and Dr. Berners-Lee played a fantastic lead part. However, you could narrow the word "language" to "data serialization structure" and have an argument that better-matches the examples.

Anyone proposing to use the in-memory bytes that make a Scala case class outside of memory might be demonstrating successful cryogenic revival from the early Clinton era. It would stand out as an absolutely terrible proposal given all the options available for JSON serialization in Scala: circe, upickle, play-json, zio-json, json-lenses, json4s, akka-http-json, tethys, nestorpersist, rojoma-json, sbt-json, jawn, argonaught, dijon, scalajson, scalajack, ...

I'll get back to language and complexity in a bit. This huge, incomplete list of JSON libraries highlights a related problem.

Abundance in the Ecosystem

Scala started life piggybacking on the Java Virtual Machine's enormously successful ecosystem. Scala has always had access to an abundant ecosystem of Java libraries. Because the Scala language is so well structured we tend to pick and choose libraries instead of locking ourselves into frameworks. (Contrast with Ruby on Rails.) As a community we are very comfortable creating new solutions - new libraries - when old solutions don't match the problems of the day. Mostly that's for the better, but it does lead to a fractured community and complexity from the viable number of choices. Martin Odersky complained that we were not "Scala programs" but were "Cats programmers, or Akka programmers, or Spark programmers."

...and Abundant Language Features

The Scala language is actually quite small, with remarkably few caveats. However, many language-like features come in via the standard library. Scala is multi-paradigm; I've seen it work well for object-oriented programming, functional programming, declarative programming, procedural-style programming and even aspect-oriented programming. Further, in Scala it is very natural to make your own new control structures to fit your project's needs.

One common complaint is that Scala has a steep learning curve. I don't think that's true; I have taught 4th-graders how to write some useful Scala. It is easy to get started. The problem is not the steepness; the problem is the height of the curve.

When I ask for help from a typical Scala Users' Group often the answer is "Go read and understand [something I never even knew was there]." I value learning new things, but often want to learn separate from getting the job done.

"Experience is what you get when you didn't get what you wanted." - Randy Pausch

Further, the Scala language often has multiple ways to get the same result. I think that leads directly to Li Haoyi's Principle of Least Power examples. Smart academics do amazing things with the language, but maybe that's not the right choice for your project. To get non-academic work done in an ecosystem this rich you need to set some boundaries for your project. You can control the height your developers need to climb by deciding to use simpler and fewer parts of Scala.

Audience

I don't agree with Li Haoyi that "Complexity is your enemy." A project's ability to handle complexity is more like a scarce resource. Scala lets you treat complexity like items on a menu where other languages, frameworks, and ecosystems offer a fixed-price of complexity. Academics can afford an abundance of complexity. The rest of us, not so much.

Think about who is going read and write your code to get some idea of what budget you have for complexity, and who might be using the software. You need to set expectations so that your audience is not surprised when they need to learn new skills to contribute.

I think the most important question to answer is "What do you expect people to do with the systems you create?" The people might be users, system admins, support crew, or future developers; there will be different audiences for different roles. They'll bring different levels of skill and expectations.

Three Examples:

SHRINE

SHRINE allows a researcher at any hospital in a network to query patient records at all hospitals in that network with the same query – without violating HIPAA. The audience includes researchers, local and network admins, and future software developers. (I just finished my work on SHRINE, which involved a lot of writing to archive it for future developers. I've been thinking about this issue a lot lately.)

SHRINE's users are pragmatic medical researchers; they are brilliant people but not computer scientists. The researchers drag-and-drop term labels in boxes to form logical queries for patients they hope to study. The researchers are using a visual domain-specific query language; behind the curtain it is all JSON.

The local and network admins mostly use Unix skills, HOCON via Typelevel Config, and peek inside databases and logs to bracket and understand problems. The admins are very capable with Unix and python scripts, but most of their interaction with SHRINE is via a single HOCON configuration file. It is extremely unusual to see any of them use any HOCON beyond the basic key/value pairs. (In the narrow realm of setting up SHRINE they are safe from Dr. Berners-Lee's fear of attraction to powerful language features.)

I left future SHRINE developers with a wiki entry opening with:

SHRINE uses Scala 2.13.x for all of its back-end code. The Scala code style is functional and highly monadic. It features shallow object-oriented components - often comprised of shared parts - to separate concerns. This lets the code explain the details itself. We build it with Maven - the easiest system for our operations team to support. SHRINE uses http4s for its web API, cats effects and fs2 for concurrency, CQRS via slick for storing and retrieving state. SHRINE sends commands between nodes using MOM (AWS SQS, Kafka, or a home-grown REST API), circe for encoding JSON, i2b2 as a source of patient data, and Scala's xml library to create and read i2b2's xml. SHRINE stores very little state outside of stack references.

That opening lists out the core technologies and the minimum skill set to bring. A developer familiar with http4s, cats effects, and distributed programming would likely be able to contribute as soon as they understand the problem SHRINE solves. I would expect a generally capable Scala developer to "go read about" all of those technologies and get some level of how SHRINE uses them before attempting to make changes to SHRINE's code base. It's a punch-list like you would find in a job description.

In 2008 SHRINE began as a "Scala as a better Java" project that used Jakarta and Scala's in-line XML capability to imitate and distribute a single-hospital query system's web API. We were able to grow SHRINE from those initial decisions while keeping the whole works compatible release-to-release (mostly). SHRINE's transformation is a remarkable demonstration of the power of refactoring in Scala Li Haoyi finds so compelling.

ev3dev-lang-scala

ev3dev-lang-scala is a library and toolkit for programming Lego Ev3 robots using Scala. Most programmers using the library are kids ages 9 to 13, mostly programming robots for FIRST Lego League. I've created a library that gives them a subset of the Scratch-like tool Lego provides. To keep things fair the library does not give them anything more than what Lego's base-level blocks provide, but they do have access to all of Scala. I try to get them to use some object-oriented containment to keep it easy to explain. Individual kids do not value that in their own work but complain when another kid makes a mess of it, especially while making presentations days before an FLL tournament. Most code the kids write is very procedural, like this method to drive the robot (backwards) to a "museum" objective:

  private def startToMuseum(): Unit = {
    Robot.moveStraightBackward(635.millimeters)
    Robot.leftRotation(135.degrees)
    Robot.moveStraightBackward(100.millimeters)
    Robot.leftRotation(100.degrees)
    Robot.moveStraightBackward(570.millimeters)
    Robot.rightRotation(135.degrees)
    Robot.rightRotation(180.degrees)
  }

The kids surprised me when they used a recursive loop to use feedback from a gyroscope to make precise turns:

  @tailrec
  def rightRotation(goalHeading:Degrees):Unit = {
    val heading: Degrees = Gyroscope.readHeading(SensorPort.One)
    val toGo: Degrees = goalHeading - heading
    val speed: DegreesPerSecond = (Robot.speed.v * (toGo.v/90)).degreesPerSecond

    Log.log(s"heading is $heading, speed is $speed")
    if(goalHeading > heading) {
      Movement.startMoving(speed,-speed)
      rightRotation(goalHeading)
    } else {
      Movement.stop()
    }
  }

To trim back their cut-paste tendencies I want to show them higher-order functions this year. A general feedback loop might be the thing, but they resist refactoring. "It works fine. Why change it?" They do not get it.

The other audience is library developers - just me so far. To make this thing work I had to learn the basics of interfacing with the ev3dev OS (a variant Debian Linux!) and how to use that interface efficiently from inside the JVM. To get the control loops running at ~200 Hz after hot-spot warm-up the core uses Java NIO calls to read and write unix streams, very much "Scala as a better Java" in style:

case class ChannelRewriter(path: Path,bufferLength:Int = 32) extends AutoCloseable {

  private val channel = FileChannel.open(path,StandardOpenOption.WRITE)
  private val byteBuffer = ByteBuffer.allocate(bufferLength)

  def writeString(string: String):Unit = this.synchronized{
    byteBuffer.clear()
    byteBuffer.put(string.getBytes (StandardCharsets.UTF_8) )
    byteBuffer.flip()
    channel.truncate(0)
    channel.write(byteBuffer, 0)
    channel.force(false)
  }

  def writeAsciiInt(i: Int):Unit = writeString(Integer.toString(i))

  def close():Unit = this.synchronized{
    channel.close()
  }
}

I don't expect the kids to work at this level, but a beginner Scala developer with a Unix background and familiarity with Java's NIO API should be able to contribute. On their best days the kids will use the IDE to follow calls down from their own code into the library to see how things work.

You'll notice the kids' code refers to measured units like .millimeters, .seconds, and .degrees - Scala value classes. I have figured out how to program the Scala3 type system to understand measured units in arithmetic - some number of degrees divided by some number of seconds is degrees/seconds. I'm of two minds deciding if it is the right thing to do. It would save the kids having to dig out the values via .v to do their arithmetic, then rewrap up the result to have the correct type (all cooked out in the compiler via Scala's value class feature). However, it would add type-level arithmetic programming to the inner workings of the library code - a leap up the learning curve from "Scala as a better Java." I don't yet know what the right decision is.

Scala lets me put that decision off. The language supports the complex types.

Disentangle

Disentangle is a library of graph algorithms I use mostly for my own entertainment. (I did use the library at ActivateNetworks for analyzing social networks for marketing.) I implemented graph minimization algorithms - like Dijkstra's shortest path algorithm - with general-purpose semirings. The same algorithm code can find things like least-probable paths instead. To use an algorithm a developer needs to understand how to use Scala's tuple structures and collections library - entry-level Scala skills:

  /**
   * Edges are just a Seq of Tuple3[Node,Node,Edge]
   */
  lazy val edges: Seq[(String, String, String)] = Seq(
                                                  ("A","B","ab"),
                                                  ("B","C","bc"),
                                                  ("C","D","cd"),
                                                  ("D","E","de"),
                                                  ("E","F","ef"),
                                                  ("E","B","eb"),
                                                  ("E","H","eh"),
                                                  ("H","C","hc")
                                                )

  /**
   * Generate all the shortest paths in the graph
   */
  lazy val simpleShortPathLabels: Seq[(String, String, Option[FirstStepsTrait[String, Int]])] = Dijkstra.allPairsShortestPaths(edges)

I use Scala's generics to keep the types lined up right for different semirings, but otherwise the code is using shallow object-oriented ideas to keep different concerns separated. Scala's solid handling of generics for Disentangle is what first drew me into Scala. Here's a semiring for finding most-probable paths:

  object MostProbableSemiring extends Semiring {

    val I = 1.0
    val O = 0.0

    def inDomain(label: Label): Boolean = {
      I >= label && label > O
    }

    def summary(fromThroughToLabel:Label, currentLabel:Label):Label = {
      if(fromThroughToLabel > currentLabel) {
        fromThroughToLabel
      }
      else currentLabel
    }

    def extend(fromThroughLabel:Label,throughToLabel:Label):Label = {
      if ((fromThroughLabel == O) || (throughToLabel == O)) O
      else {
        fromThroughLabel * throughToLabel
      }
    }
  }

The core idea behind the library is to write algorithm code that looks identical to the pseudocode in fat algorithm books like CLRS. Here's the Floyd-Warshall algorithm demonstrating "Scala as a better Java Fortran" :

  /**
   * O(n^3)
   */
  def floydWarshall[Node,Label,Key](labelDigraph:MatrixLabelDigraph[Node,Label],support:SemiringSupport[Label,Key]):IndexedLabelDigraph[Node,Label] = {
    val innerNodes = labelDigraph.innerNodes
    for (k <- innerNodes; i <- innerNodes; j <- innerNodes) {
      val summaryLabel = relax(labelDigraph,support.semiring)(i,k,j)
      labelDigraph.upsertEdge(i,j,summaryLabel)
    }
    labelDigraph
  }

There's this split between how different audiences will interact with the code. Someone could just use this library in a project with an entry-level understanding of Scala. To add new algorithms you need Scala generics to keep the code type-safe.

In Contrast: My Early Career

My first job out of school involved converting Matlab code and Excel spreadsheets into Fortran (... and postscript). The third and fifth projects were converting Matlab into Objective C . The seventh project was an attempt to convert a system in C to Java. (It failed.) My eighth through twelfth projects were converting Matlab into Java. My second Scala project was translating R code into Scala.

I did take a job at MathWorks after all that Matlab translation. Jack Little, founder of MathWorks, pointed out that, "You can write a lot of really terrible Matlab code really fast. That's one of its greatest strengths." I get that as a marketing plan for MathWorks, an action plan for a researcher, and a business plan for a start-up. All that recoding paid my student loans, rent, and kept me in food for my early career, but was somewhat unsatisfying. I did observe that once a project chose a production ecosystem then that choice was for the life of the project. If the language or ecosystem didn't provide something that the project needed then that venture failed. Big rewrites were too expensive.

Scala is Best as Your Little Language

Scala libraries regularly evolve into domain-specific languages as the authors get more understanding about how to write concise, clear code. The Scala community takes a lot of pride in that. Here's some SHRINE code that uses http4s' end-point DSL:

    case request @ POST -> Root / "startQuery" as user => startQuery(request,user)

    case request @ POST -> Root / "changeQueryNameAndNotes" / LongVar(queryId) as user => changeQueryNameAndNotes(queryId, request, user)

    case _ @ GET -> Root / "query" / LongVar(queryId) as user => getQuery(queryId, user)

In Scala the line between what's "just a library" vs "what is a DSL" is pretty blurry. We bring these into code via import org.http4s.dsl... but pieces we're importing are masterfully-designed unapply methods to be used in case statements. There's no new parser, no changes to a compiler, no magic plugins, not even annotations. It just lets our team write concise, clear Scala code.

By choosing which parts of Scala to use and which to leave out we can define the right little language for each project. Changes to these decision have an incremental, bound cost ; if we need to make a different choice in the future we can either let some new kind of complexity in, or refactor to simplify. That's a tremendous benefit. It helps Scala projects live long, healthy lives by letting them adapt to changing needs.

"Take 20 minutes to see if you can solve it yourself. After that ask for help. We're all here to get the job done. You aren't alone. We want you to learn to work independently, but we don't expect you to know how to do that right away. Try being independent for 20 minutes, learn from that, then let someone know."

I try to follow my own advice.

20 minutes of wishing you could find the right thing to read or staring into the void hoping for inspiration is plenty. If you have the right buzzwords to feed into Google then you're most likely to find the right article to read in the first few tries. After that the odds drop off; you don't know the right magic words. I bet the success rate for technical questions follows Planck's curve - maybe better than this solar radiation graph from Wikpedia.

If what you really need is inspiration, taking 20 minutes to summarize what you already know is a good warm-up. If you haven't already thought about the problem and recalled the answer then you are better off either asking for help or setting the problem aside and letting your brain bubble away while you do something else.

Odds are someone you work with knows the right thing for you to read, can tell you something better to search for, or has already thought through some part of the problem you are working on. They can help you. Get their help. Investing 20 minutes is enough to ask a good-quality question to get that help. (Less time is probably not enough to ask the good question.)

I expected the interns to struggle with this advice. I had guessed that students would be used to working independently, following academic rules. It was never a problem for them.

Following this advice was a challenge for more experienced people joining our team, especially when one of them was the team expert on the topic. It's harder to imagine the value of framing the question well, then asking someone less knowledgeable for help, but it works really well. Early in my job at HMS I was the only person working on back-end Scala code. When I was clueless, I felt alone and responsible for finding all the answers. Talking through a problem with our front-end developer, our amazing tester, or just someone interested in the project helped me frame the question better. It often showed me what the real concerns were, and saved me from "shaving the yak." The Scala community - especially the people who eventually made up the Type Level community - were a huge help getting to concrete answers. (Help from Type Level people almost always starts with, "Oh. See if reading ... helps." It almost always does, but it takes some investment. In contrast, reading the wrong thing is really frustrating.)

I found that talking through my puzzles with an intern was magic. We usually wouldn't come up with a solution right away, but explaining got things fizzing in our heads. I would often make the inspirational connection while I was concentrating on something completely different, like niggling with some other part of the code base, chasing the kids in the park or cooking dinner. It was like my non-euclidean geometry professors from way back warned us: "Go ahead and learn everything about the problem that you can, but don't expect to get anywhere just by applying work to these puzzles. Keep a notepad on your nightstand. You are going to wake up in the middle of the night with the proof in your head, and you must write it down before you go back to sleep. It'll get better with practice, but it will never seem natural."

One of the deeper things in the Toyota Way is that the advice comes in complementary pairs. I think this 20-minute limit pairs well with the first-day plan. It's a small piece of "how-to" to match with the first-day plan's list of "what." When someone is starting a new job then the 20-minute limit sets an expectation for how to solve the problem without giving up the tempo, and gives the new employee permission to interrupt.

This arrangement removes the bottleneck of processing the data as it comes in by moving the work of refining the data to some time before it will be used. If the data is used in multiple ways then each use can have its own specialized refinement to match that use's queries.

In distributed system terms CQRS defines the first, most time-dependent map on the event stream to "just store the events," delays and separates that from a complex reduce of "convert the events to something useful," then enables efficient queries of the structure that best matches the end-use.

The best candidate systems for CQRS are systems where events arrive out-of-order; that's pretty much any system with distributed, changing state. It also includes every system that uses message-oriented middleware; even with best-effort configurations to keep messages in-order they can still arrive out-of-order. Within a single computer: parallel threads can execute in strange orders, "happens before" rules are softer than we might wish, and computer clocks are notoriously bad. Sometimes a program running on a single computer can benefit from CQRS.

Systems where updates come in rapidly benefit from the efficient processing of writes in the "Command Responsibility." The "Command Responsibility" writing of updates is relatively straight-forward to replicate across threads and even distributed servers, so it scales up easily with the volume of data flowing in. Systems that process and use the data for different purposes benefit from the separation of "Query Responsibility." Each can have its own on-demand processing of the stored events. The stored events naturally provide an audit trail and the ability to replay the events. Replaying the events to debug a distributed system is invaluable, perhaps the most powerful "Query Responsibility" feature.

I'll be pulling examples from my work at HMS Catalyst on SHRINE (the Shared Hospital Research Information Network E) - a system that allows a researcher at any hospital to query for counts of patients with similar characteristics at all the hospitals in the network. The core idea behind SHRINE is that patient records never leave the hospitals; every hospital runs a researcher's query and reports a count of patients to the network.

When I started on SHRINE it used a web API call that fanned out into a layered system of slow and unreliable patient data servers. That initial web API call might take as long as five minutes to finish - with no feedback to the researcher. For usability five minutes is a coffee break; the researcher might not come back. For normal http connections on the internet five minutes is "probably just broken." SHRINE 2.0+ shows incremental query progress and results as soon as they are available from each hospital. Patient counts for simple queries from the fastest sites show up in a second or two - a usability breakthrough.

SHRINE version 2.0 and onward have a common currency of case classes that represent queries, results, researchers, nodes (usually hospitals), and the SHRINE network itself. (See future blog article after I write it.) The events are changes to these structures.

We started recording events in a simplified "Command Query" style in SHRINE 1.19 because one customer required an audit trail. We expanded it to a CQRS-style local-to-the-user forward cache in SHRINE 1.21 to improve performance. That showed us our system did need to handle distributed state better; SHRINE 1.23 demonstrated the value of incremental updates of the distributed quereis in progress - via message-oriented middleware. In SHRINE 2.0 we adopted CQRS fully and ran it with no significant change to the code for five years. One large network ran SHRINE 3.1 for three years - with no update at all.

Do Not Use CQRS Unless You Need It

Famous Blue Muppet is taken aback.¹

Li Haoyi's blog entry "Strategic Scala Style: Principle of Least Power" introduces CQRS with "Don't use Event-Sourcing/CQRS unless you know what you're doing." https://www.lihaoyi.com/post/StrategicScalaStylePrincipleofLeastPower.html#immutability--mutability If you haven't read that article - go read it now. It's amazingly good general advice. Then understand why I've got references to Jon Stone's The Monster at the End of This Book, and that what follows is likely not good advice.

If your system's state is immutable - it creates things but never changes them - then you really don't need CQRS. Just record what you create to unpack later.

If state will only change in a slow and sedate fashion on a single thread then you can use something less intense like traditional database transactions. Be aware that database transactions often do not have the ACID properties we wish they did. In a small, slow, sedate, centralized system database transactions will probably be fine. When you put that central database under pressure, with changes coming in concurrently, then expect trouble. With too much locking things will slow down; with too little the system can lose coherency - or even data.

CQRS Creates A List of Events To Replay

The heart of CQRS is list of events. Some parts of the system emit events. "Command Responsibility" stores events but never changes them. Later "Query Responsibility" replays the events to construct state. Replay the events - in order - to reproduce the state.

You could store the events in a concurrent in-memory structure so long as you aren't concerned with what happens after the process terminates. A simple database table (time stamp and the event structure) can work reasonably well. NoSQL list databases like CouchDB and DynamoDB are even easier to set up than a database; they are possibly overkill for the "Command Responsibility" but work great.

The resulting CQRS system relies on the event time stamps being solid enough for "in order" to work in your system. That pretty well describes what we did for SHRINE 1.21's forward cache; we made a simple database table to record the changes in state, then used the timestamps to generate the states we showed to the researchers. It worked well in a low-traffic network of four hospitals.

Using this event list for a network of four dozen hospitals, many more active researchers, and message-oriented middleware showed us that the timestamps were not good enough to rebuild the state in this busier environment. We had to ruthlessly take advantage of SHRINE's particulars to reassemble the state to resolve conflicts when the time stamps didn't work out. Each new bug we discovered resulted in a new condition to check. The cyclomatic complexity of the logic to reproduce the state started to fold on itself.

Sometimes You Need CQRS (Sometimes It Won't Be Enough)

Famous Blue Muppet asks you not to read this.

CQRS is particularly good at making a coherent, self-consistent view available in the chaos that is distributed computing. Observations of time in computing systems are staggeringly horrible.

You might recall "The Barn Pole Paradox" from relativity physics. Computing is worse.

The resolution of the paradox is to split "Is the pole in the barn?" into two events - "the front of the pole exists barn" and "the end of the pole enters the barn" - then twist them through the Lorentz transform to show that their order can swap based on the observer's frame. (Muppets demonstrating that paradox would be fitting.) At least the geometry of the universe holds the observers to some rules.

Observation in distributed systems - at the bottom of Earth's gravitational well, not moving at significant speeds - don't usually have significant relativistic effects, but they do have:

Discrete time - Different things can happen at the exact same clock time

Processors that run faster than the clock resolution - My laptop clock has an API resolution of one nanosecond, but has processors running at four GHz. About four things should happen in the same tick on each core

Concurrently-running processors - Many things could happen at the same time on different cores. My laptop has 12 cores.

The actual resolution of the operating system's - and JVM's - times are more often on the order of microseconds.

The program reading a clock takes some poorly controlled amount of time.

Computer clocks are temperature-sensitive.

I work mostly on Java virtual machine systems. The JVM's nanosecond clock counts up from an arbitrary point starting when the JVM process starts. It's handy for controlling frame-rate in a video game loop, but would need to be pinned to the operating system's clock for anything off-board.

The JVM's API resolution for the OS' clock is in milliseconds.

The OS's clock resolution might be finer, but in older operating systems may only have accuracy of 30 ms. (One really old OS will sometimes jump backwards 10s of ms!)

The ntp protocol can keep distributed computers' clocks to within about 10 ms on stable LANs, and maybe 100 ms on networks you wished worked better.

ntp can speed up or slow down the clock, especially just after start-up.

The system admin can set the clock to arbitrary values.

A network call across loopback takes at minimum 10s of ms.

Writing a local file takes 10s of ms. Memory buffers makes this highly variable.

http calls across a LAN takes at minimum ~100 ms.

https calls across a WAN takes at minimum 100s of ms.

Reliable message-oriented-middleware (MOM) calls get their reliability by adding more network and file system calls.

Messages in MOM systems may arrive out-of-order for any number of reasons.

MOM also adds queues to the works; messages may sit in queues before they are processed - possibly for days.

MOM's zero-or-more message delivery promise means that a client may receive the same message more than once.

If your system is about changing state across distributed computers then you need some way to build a reliable system around all of that. CQRS is a good starting point, but may not be enough.

The Art Of Compromise: CQRS, But Storing State Changes

SHRINE 1.23 was a skunk-works experiment to show the researchers progress updates as queries moved through the network. We replaced the long, slow, layered web API calls with message-oriented middleware queues to support asynchronous reporting and gain reliability. Queries and results gained more observable events. That fidelity interacted badly with the low quality timestamps; we needed even more cyclomatic complexity to recreate the state.

"Complexity is your Enemy" - Lee Haoyi . We needed some way to limit the complexity of the logic . SHRINE had compute power and IO bandwidth to spare. (The most significant bottleneck in SHRINE - by two orders-of-magnitude or more - is in the patient data source. We could never change that.) The events in SHRINE carry small changes to small pieces of state of progress through the network for a result. The distributed parts of the system that were emitting the events have valid - but possibly outdated - versions of the state. We could take advantage of that.

I took some inspiration from CouchDB/DynamoDB. For SHRINE 2.0 I decided to store state in CQRS style instead of storing simple events. I changed the message meanings from "events" to "proposed state updates" . If the previous state has not changed then SHRINE can safely apply the update. If SHRINE discovers the previous state had changed then some business logic kicks in to decide what to do with the update. For example if a successful result arrives in conflict with a failed result then SHRINE's logic chooses to take with the successful result (as a state after the failed result). Because each state is recorded the states can be replayed. The desired state transitions impose a best order without depending on the unreliable timestamps.

Storing state structures like this is a recommended practice for CouchDB/DynamoDB systems. ... That's not what we used.

More Art, More Compromise: CQRS - but Storing State - in a Relational Database

Famous Blue Muppet does his best to stop you reading more.

Really - if you can just use CouchDB, or DynamoDB, or something that fits your problem better than late-1980s relational database tables then stop reading, build your system, and go live a happier life. I recommended we store SHRINE's common currency data structures in DynamoDB or CouchDB, with a thin facade to just expose the calls SHRINE needed from either tool. My boss pointed out that our operations team had no experience with either system and were busy with higher priority projects. We needed to store everything in the relational database SHRINE already used.

Later Simon Chang - an amazing operations engineer - and I demonstrated SHRINE 2.0 running with AWS DynamoDB in a 24-hour hack-a-thon in 2019. We got it working in about 4 hours, built a demo with repeated 60-hospital simulated surge tests, had a good night's rest, then put together the presentation in the morning. It worked really well. (... until we burned through the DynamoDB free tier transaction allowance, half-way through showing it to the boss' boss after the party.)

You're Still Reading?

Famous Blue Muppet pleads with you to read no further.

Let me see if I can talk you out of the mistake you are about to make by showing some of the code I wrote to get this working in SHRINE. (I guess it wasn't that bad, but we own some complex code when we could have just downloaded a library.)

SHRINE already used Circe to convert its common currency case classes into JSON to transmit in messages and http bodies. SHRINE already supported three brands of database via Slick; we use Slick to save us writing every bit of database code three times. I designed a standard table - and a standard case class trait - to support storing Circe-serialized case classes - using as few database-specific features as possible.

The case classes - at their simplest - look like this:

case class Researcher(
                 id:ResearcherId = ResearcherId.create(),
                 versionInfo: VersionInfo = VersionInfo.create,
                 userName:UserName,
                 userDomainName:UserDomainName,
                 nodeId:NodeId
                ) extends Versioned[ResearcherId] {...}

The itemVersion member in VersionInfo provides an order of states observed in SHRINE's modified CQRS system. (The protocolVersion field supports backwards-compatibility and the other fields are invaluable for debugging.)

case class VersionInfo(
                    protocolVersion:ProtocolVersion,
                    shrineVersion:ShrineVersion,
                    itemVersion:ItemVersion,
                    createDate:DateStamp,
                    changeDate:DateStamp,
            ) {...}

ItemVersion is just an Int value class:

class ItemVersion(val underlying:Int) extends AnyVal with ValueClass[Int]

If everything is working smoothly then the itemVersion should increment by one for each recorded change in state for a case class with a given id. SHRINE will insert a new row in a table in the database each time it observes a new state. When SHRINE needs to provide a most-up-to-date view of state it will select the row with the right ID and maxiumun itemVersion, then reconstitute the case class from that row's JSON .

  def selectResearcherByIdIO(researcherId: ResearcherId): IO[Option[Researcher]] = {
    //Select the row with the maximum itemVersion for the id
    runIO(Researchers.allRows.filter(_.id === researcherId).sortBy(_.itemVersion.desc).take(1).result.headOption).
      map( _.map(_.toResearcher)) //convert the JSON to a Researcher
  }

Before SHRINE inserts the new row it checks that there's no row for that id with the itemVersion. If that's true then it inserts the new row:

      tableCompanion.allRows.filter(r => r.id === row.id).filter(r => r.itemVersion === row.itemVersion).result.flatMap {
        case Seq() => { // a row with this id and version does not exist yet
          //insert the row only if it is the only row with this id and version
          tableCompanion.allRows += row
        }.map {
          //expect insertCount to be 1

When SHRINE receives data out-of-order, but with the correct itemVersions then SHRINE simply inserts the late data's JSON in the table. The rest of the system can flow without worrying about things written out-of-order in the database.

When SHRINE detects a row with the same id and itemVersion then SHRINE does a simple check to see if the states match. If an identical state is already in the table then it's not a problem; something else wrote the data earlier - which can certainly happen with message-oriented middleware retries. If there's a difference in the JSON then this particular write lost a race. SHRINE's database code fails with an exception so that code with a broader context can decide what to do.

        case alreadyExists => // at least one row with this id and version does exist already
          // don't insert the row
          // if the existing rows with this id and version are the same then this isn't a problem
          if (alreadyExists.forall(_ == row)) DBIO.successful(None)
          // However, this is a problem if the new row is different from the others
          // Due to the transaction this should only happen if some other row was written outside this method
          else DBIO.failed(ItemVersionRaceLostException[I, V, R](item, row, alreadyExists))

The whole works starts with a write operation to an auxiliary table that exits solely for locking. This lock prevents multiple inserts from happening concurrently on data in the same table. I could have gotten more throughput with a row-per-ID and corresponding lock. I thought of it late, and SHRINE preformed very well without that optimization.

    //increment the table version and isolate this transaction 
    incrementTableVersion(tableCompanion.tableName).flatMap { _ =>

In case you're not convinced to just use CouchDB or the like - here's the full, unedited method, with four generics to fill in. It's not the worst code, but requires a lot of supporting bits to make work. Converting Slick's DBIOs to cats effects IOs wraps around this call. Additional code in the IO layer adds business logic to decide how to resolve conflicts shown by ItemVersionRaceLostExceptions.

  private def upsertItem[
    I <: Id: ClassTag, //Needs the ClassTag to let I survive erasure
    R <: Row[I],
    V <: Versioned[I],
    Items <: ItemTable[I,R]
  ](
     item: V,
     tableCompanion: ItemTableCompanion[I,R,V,Items]
   ): UpsertItemType[V] = {

    val row = tableCompanion.itemToRow(item)
    implicit val IdColumnType: JdbcType[I] with BaseTypedType[I] = MappedColumnType.base[I, Long](
      id => id.underlying,
      long => tableCompanion.longToId(long)
    )

    //increment the table version and isolate this transaction 
    incrementTableVersion(tableCompanion.tableName).flatMap { _ =>
      // Find all the rows with the same id and version number
      tableCompanion.allRows.filter(r => r.id === row.id).filter(r => r.itemVersion === row.itemVersion).result.flatMap {
        case Seq() => { // a row with this id and version does not exist yet
          //insert the row only if it is the only row with this id and version
          tableCompanion.allRows += row
        }.map {
          //expect insertCount to be 1
          case ic if ic == 1 => Some(item)
          case insertCount: Int => throw HubDatabaseAssertException(s"""insert into ${tableCompanion.tableName} returned $insertCount, not 1 as expected""") // ic == 0 would mean nothing was inserted. I don't think ic > 1  can happen
        }
        case alreadyExists => // at least one row with this id and version does exist already
          // don't insert the row
          // if all the existing rows with this id and version are the same then this isn't a problem
          if (alreadyExists.forall(_ == row)) DBIO.successful(None)
          // However, this is a problem if the new row is different from the others
          // Due to the transaction this should only happen if some other row was written outside this method
          else DBIO.failed(ItemVersionRaceLostException[I, V, R](item, row, alreadyExists))
      }
    }
  }

Examining the generated SQL showed that almost everything in this method happened within one prepared statement in the database. Most of the time to make the call is connection and transmit overhead to get out of the JVM and into the database's process. It's pretty efficient.

All-in-all this worked very well for SHRINE 2.0 to SHRINE 4.3 - about five years of production use. That upsert() is probably the most involved method I wrote and didn't delete in SHRINE's codebase. It has the feel of a senior class project to teach us how something complex works. I'm happy it works well, but there are easier off-the-shelf tools to get CQRS working. It's something other people could use but probably shouldn't.

Famous Blue Muppet, relieved, is so embarrassed.

A Note on "Famous Blue Muppet"

1. The interruptions above are Grover's, from Jon Stone's The Monster at the End of This Book, illustrated by Michael Smollin. Each one links to a page of the Internet Archive's scan. I swapped the images out for links when I restarted this blog in mid-2026. Recent advances in AI have made me very sensitive to copyright.

Jon Stone wrote this amazing book about dread, anticipation, and the gap between the two. It makes a great introduction to a discussion of The Sunk Cost Fallacy. It is essential reading alongside critiques of software development lifecycle trends.

You should own a physical copy. The book can express your concerns very quickly, but it does not work on a screen. You don't want some Famous Blue Muppet sabotaging your trackpad.

My boss' direction for our first intern at HMS was, "Just drop him into the project and treat him like any other software engineer on the team." We hadn't had any developers join the team since I had been hired two years prior, so I took it as a license to pull from my experience, especially as tech lead for a growing team at BAE. I came up with some steps that matched what I would want for myself, and steps that I thought would get that engineer up to speed quickly. We used the process successfully with four interns and one new employee at HMS Catalyst.

I had led something similar for a dozen new hires at BAE. There we had the new hires give a presentation to the whole team in the first few weeks to introduce themselves and show how their experience would help us. We had an ulterior motive; we were doing defense research and could never predict who might be read into which compartment and suddenly be called on to present to a customer. (Misadventures followed, but it all worked out.) At HMS the interns gave a presentation on what they had done at the end of their term.

MathWorks had a very well-thought-out monthly orientation class for all new employees about how the company worked. (MathWorks put thought into everything.) It was really good, but at a more grand scale than I would expect anywhere else ... ever. When I listed what I wanted to do in my first few days at MathWorks my boss there replied with something like, "Yes, that's what we want you to do. Sign up for the orientation class." MathWorks was already doing what I thought was a good idea.

I think these steps would work well anywhere, when combined with the 20-minute rule (... a future blog entry):

Day 1 - Get a computer working with internal systems, check out the code, get it to build, start on an easy ticket.

Day 2 - Finish the fix for that easy ticket, check it in, ask for a review.

Day 3 - Merge that change into a git branch headed for a release.

Weeks 1 & 2 - Get a grand tour of key libraries and concepts central to our product by fixing more interesting tickets.

Week 3 and onward - Take on some major, unifying project that works across teams and has a deep impact to learn to work "like any other software engineer on the team."

Some time in the first four months - Present something relevant to the team.

To follow that plan I had to set the stage for the new hires before they started. I was able to delegate getting set up as a Harvard employee and getting a laptop going; I'm grateful others could help navigate the bureaucracy. The software tasks were easy to find. I had inherited a Jira system with hundreds of open tickets to sieve. We were moving SHRINE through a major architectural transformation from a promising prototype to a resilient system. I had to balance difficulty of a change, value of that change, and the risk that a new hire would get lost and need more help than the team could provide. The interns needed good stories to tell for job interviews, which eliminated some of the more arduous SHRINE-specific tasks and wholesale deleting of dead code. The limit came from abundance of work and my own limited time to coach.

Unit tests and cosmetic changes provided early, easy tickets. This also made completing these "fold the laundry" sorts of tasks more valuable than using them as fill at the end of sprints.

The Jira backlog provided tasks for the tour of key libraries and concepts. SHRINE uses http4s for its web API, cats effects for concurrency, CQRS via slick for database work (Our first intern could not contain his delight about writing monadic database code on day three.), AWS SQS for messaging, circe for json, i2b2 as a source of patient data, and Scala's xml library to read and write i2b2's xml. I found a ticket in each of these categories to fix some minor bug or make some minor improvement. I could usually line the tasks up in an order that built one skill on another.

New features provided that important project. I gave interns major responsibility; that freed me to lead the project, manage day-to-day tasks, and deal with the crisis of the moment. The projects included things like "simplify the query sign/verify code" and "create and fill in the API we will use for message-oriented middleware." The one employee who started under my watch had more than a decade of experience at HMS Catalyst, so I asked her to "pick a web API library to replace Jakarta and Spray, then rework the web API endpoints to use that library," a very broad project. She was a bit taken aback by that, but she gained expertise, then shared it. It couldn't have worked better.

Some parts of our work were simply not good tasks for new hires. Some of the vital parts of SHRINE - especially communicating with i2b2 - required years to accumulate all the unstated caveats. Refactoring tangled code or deleting obsolete subsystems would just frustrate a new developer without helping them learn the system. It's more efficient to have an old hand (me) clean up the mess and refactor it to something easier to understand. The new hires can help review the change to get that understanding of the improved code.

Overall this new hire process worked so well I decided to use it as my first blog entry for the 2024 blogging restart. Next time I start a new job or hire someone new I hope to follow it and learn a little more about how to start a job right.

Explicit Implicit

Before I read Pragmatic Programmer I knew I wanted to make my programs transparent and easy to understand. Thomas and Hunt have a section devoted to "Explicit Programming," the idea of making a piece of code as clear and non-magical possible. Scala's implicits seem like an anathema of that goal. I've mostly avoided implicits, and mostly regretted it when I haven't.

I Wish Typesafe Config Knew About Scala

I like Typesafe Config. However, it's in Java so the API is awkwardly broad. Config was pulled out of Akka; it shouldn't seem so alien. To support primitives the Config object has a herd of narrowly-typed get*() methods. To use Config I'm expected to pull out ints and booleans and Strings, and, piece-by-piece, collect little parts. I wish it had more of a Scala-style API, something that would let me hand in a higher order function and give me back a finished whole instance of things I do care about.

I could use the pimp pattern to get what I want.

Pimp Pattern

Scala offers the Pimp Pattern to add API to classes we don't control. I was suspicious. One thing I like in Scala is that I don't need the Gang-of-Four Pattern Book; after 20 years of Java I've had enough patterns. Pimps are comic bad guys in B movies. The Pimp Pattern relies on [implicits](http://twitter.github.io/effectivescala/#Types and Generics-Implicits), which seem like a good way to make spaghetti code, far from my explicit ideals. That's three bad smells before I've even started.

I tried it despite my misgivings. I pimped Config. I added higher order functions to make getters to give me the actual parts I care about instead of just primitives. To use it I still explicitly import the parts. The rest just works:

import net.shrine.config.ConfigExtensions
...
val config: Config = ConfigFactory.load("shrine")
val keyStoreDescriptor = config.getConfigured("shrine.keystore",KeyStoreDescriptor(_))

KeyStoreDescriptor's companion object has an apply(Config) method that makes a KeyStoreDescriptor. I ask Config for what I want through very clean, very intuitive code.

Here's the code for ConfigExtensions:

package net.shrine

import com.typesafe.config.Config

package object config {

  implicit class ConfigExtensions(self: Config) {

    def get[T](key:String,construct:String => T):T = construct(self.getString(key))

    def getConfigured[T](key: String, constructor: Config => T): T = constructor(self.getConfig(key))
    ...
  }

The implicit class has to exist inside another class or object definition or else the compiler growls about an `implicit' modifier cannot be used for top-level objects'. However, the package object works fine and gets the class into a natural namespace, only one step from explicitly having ConfigExtensions in its own file. (Other people seem to use a class or a trait as a namespace to hold the implicit class. The class approach tempts coders into using a wildcard import, hiding the magical pimp behind more smoke and another mirror. Extending the trait to get access to methods in a class' context seems to make the class claim to essentially be something it isn't, inviting an OO tangle.)

I'm half-tempted to suggest a Scala language fix to let the implicit class just be part of a package instead of junking up my package object. I want a better understanding of why things are done this way first.

Code Caught Between

For a directed graph Brandes' Algorithm counts how many times a node appears in the shortest path from one node to another. To run Brande's Algorithm on an undirected graph, convert each undirected edge into two directed edges (one each way), run the calculation, then divide the counts by two - because the algorithm finds two paths in the directed graph for every path that exists in the undirected graph.

In Disentangle the algorithm code is decoupled from the graph structures. The parameters for algorithms are collections of edges and nodes, not graphs. Up until Brandes' algorithm on undirected graphs I was comfortable writing explicit, simple glue code to access lists of edges to feed the algorithms. I didn't want to force my users to convert to Disentangle's graph structures in order to access the algorithms, and didn't want parts of my algorithms escaping into the graph code.

I trust people to convert from undirected to directed graphs; mistakes will give spectacular wrong answers. However, Brandes' "finally divide the results by two" would be easy to leave out; betweenness would still just be a big Map from nodes to unintuitive numbers. I had no good place to hang the divide-by-two correction.

Return of the Pimp

That pimp pattern let me inject a new method on the undirected graph trait; using Brandes' algorithm looks like just another method call:

package net.walend.disentangle.examples

import net.walend.disentangle.graph.semiring.Brandes.BrandesSteps
import net.walend.disentangle.graph.semiring.LabelUndigraphSemiringAlgorithms
import net.walend.disentangle.graph.{AdjacencyLabelUndigraph, NodePair}

object BrandesImplicitsExample {
  val edges: Seq[(NodePair[String], String)] = Seq(
    (NodePair("A","B"),"ab"),
    ...
    )

  val nodeOrder = Array("A","B","C","D","E","F","H")

  val graph = AdjacencyLabelUndigraph(edges,nodeOrder)

  val brandesResults = graph.allLeastPathsAndBetweenness()
  ...
  val betweennessValues: Map[String, Double] = brandesResults._2
}

The code looks like this:

package object semiring {
...

  implicit class LabelUndigraphSemiringAlgorithms[Node,Label](self: LabelUndigraph[Node,Label]) {
    ...
    def allLeastPathsAndBetweenness[CoreLabel, Key]( coreSupport: SemiringSupport[CoreLabel, Key] = FewestNodes,
                                                     labelForEdge: (Node, Node, Label) => CoreLabel = FewestNodes.edgeToLabelConverter): (IndexedSeq[(Node, Node, Option[BrandesSteps[Node, CoreLabel]])], Map[Node, Double]) = {
      val digraphResult = self match {
        case indexed:IndexedLabelDigraph[Node,Label] => Brandes.allLeastPathsAndBetweenness(indexed.edges,indexed.nodes.asSeq,coreSupport,labelForEdge)
        case _ => Brandes.allLeastPathsAndBetweenness(diEdges,coreSupport = coreSupport,labelForEdge = labelForEdge)
      }
      correctForUndigraph(digraphResult)
    }
    ...
    def diEdges: GenTraversable[(Node, Node, Label)] = {
      self.edges.map(e => (e._1._1,e._1._2,e._2)) ++ self.edges.map(e => (e._1._2,e._1._1,e._2))
    }

    def correctForUndigraph[CoreLabel](digraphResult: (IndexedSeq[(Node, Node, Option[BrandesSteps[Node, CoreLabel]])], Map[Node, Double])) = {
      val halfMap = digraphResult._2.map(x => (x._1,x._2/2))
      (digraphResult._1,halfMap)
    }
  }
  ...     
}

Again I put the implicit class in the package object to keep it in what feels like the right namespace. That code seems OK, so I added methods for other algorithms, did the same for directed graphs, and released it as Disentangle 0.2.1.

I'm still openly suspicious of implicits. I wouldn't want to work with a whole library magically tacked together this way. However, using the pimp pattern to contain methods that would otherwise be drifting around in unrelated, undiscoverable code seems OK so far.

object UsefulConfigurableThing {
    def apply(config:Config):UsefulConfigurableThing = { ... }
}

I use this method to make standard parts easy to configure. I let things be even simpler for singletons - a singleton just accesses the Config it needs. The resulting code is clear and very flat. Competing options meant filing my head with a new bag of API names and sometimes a new skill in an uninteresting discipline. This flat approach saves me from depending on a mountain of YANGTNI framework features.

SHRINE's use of TypeSafe Config

For my day-job I work on work on an open-source bioinformatics project called SHRINE. I'm (mostly) fortunate to have inherited a project that is a business success built on well-tested code. SHRINE started life as "Scala-as-a-better-Java," first check-in I can find was in 2008. SHRINE is cradled inside a Tomcat servlet behind Jersey, and needs some weeding. (I'm slowly refactoring it toward less baroque dependencies - a different rant.) It has a lot of config. We fix about a third of SHRINE troubles in the field by tweaking a node's shrine.config file. Our team needs to be able to quickly trace what a configuration value does in the code.

At the bottom of the stack is TypeSafe Config, which is really quite good. However, the optional config of optional parts with optional parts that a eight-year-old project develops leave their scars, and SHRINE's config is warty. One of the background whispers in the code is a fear of breaking things that work. I spotted four themes written in the code: "ManuallyWiredShrineJaxrsResources," plus two layers of empty case classes (and tests) as middle stages between Typesafe Config and the actual places that use the configured values hint that at least one of my predecessors really missed having Spring handle his configuration. He expected to use those extra layers for validation, but never did. At least one of other predecessor really didn't understand that this earlier predecessor had built mini-Spring. A third predecessor added a layer of higher-order functions for a different sort of dependency injection, but had stopped before using those functions more than once. I'm the first developer to use Config's reference.conf files. I might be the first to use a deep embrace of TypeSafe Config.

When our team tries to gain some quick understanding to help system admins in the field we flounder around tracing variables to configured values. We struggle to provide timely support on the phone or in chat because tracing through the code takes too long. It frustrates us. The whole thing should be simpler. After all, it is just config.

System.getProperty()

In Java we started with java.lang.System.getProperty() back in the Age of Plaid Flannel. System.getProperty() was pretty solid and is still in wide use. Everyone understood that Properties was a HashMap[String,String] just from the API. You could stuff values into it from the command line (provided you didn't have too many characters for DOS). java.lang.System.setProperty() had problems that predicted the general goodness of immutability and were easy to avoid. The main work was figuring out what the property keys were. It was great for just getting a database name, password, and url.

Cake, Spring, DI, and JNDI

Sun created a standard called JNDI, declared that we should use it, and used it as underpinning for whole fleet of standards. JNDI had two big problems: First, the API is full of uninspiring boilerplate like InitialContext and catching NamingExceptions. Second, you had to supply something to be the back-end for it - almost always a big framework that you program in an XML schema that will be different from the next big framework's XML schema. Spring made it so that I could get my own Java POJ object out the other end, and made the XML schema a bit better, but I was still programming in XML. The Cake Pattern finally let me program in something not XML, in exchange for lots of traits. Someone finally formalized what they meant by dependency injection while declaring its alternatives "dinosaurs." Understanding that helped me conclude that most of the time I didn't want it in my system.

(Is anyone else bothered that the "DI" in JNDI means something completely different and predates "DI" by five years or more? Maybe Oracle should make it a backronym.)

Finally TypeSafe Config

The Akka project spun out TypeSafe Config as a separate project. The gritty details fit on a single, short readme.md page in github, and the API example fits on a screen. Most of the page is about corner-case features that I don't use frequently. You program TypeSafe Config in a superset of JSON that basically lets you leave out quotes. A Config is a container of key -> object pairs. All of the objects are built from JSON strings. Many of the objects built are Configs. Most of the breadth of the API is about getting the right type into Java.

def apply(config:Config) is Enough

I settled on using an apply(Config) method in a case class' companion object as my standard approach.

case class StewardQueryAuthorizationService(qepUserName:String,
                                            qepPassword:String,
                                            stewardBaseUrl:URL,
                                            defaultTimeout:FiniteDuration = 10 seconds) extends 
             QueryAuthorizationService with 
             Loggable with 
             Json4sSupport {...} 
 
object StewardQueryAuthorizationService {

  def apply(config:Config):StewardQueryAuthorizationService = StewardQueryAuthorizationService (
    qepUserName = config.getString("qepUserName"),
    qepPassword = config.getString("qepPassword"),
    stewardBaseUrl = config.get("stewardBaseUrl", new URL(_))
  )
}

The apply() method pulls arguments from the Config for the case class' constructor, then calls the case class. JSON to case class. Standard, general tools. No special skills or big patterns to explain. No mystery.

Sometimes apply(Config) is More Than Enough

That technique works great in general, but sometimes there'd only be one instance of the class. A Scala object is a better choice. For that, I apply Li Haoyi's principle of least power. I go one step further than hard-coding, and pull the value out of the Config exactly where I need it.

object StewardSchema {

  val allConfig:Config = StewardConfigSource.config
  val config:Config = allConfig.getConfig("shrine.steward.database")

  val slickProfileClassName = config.getString("slickProfileClassName")
  val slickProfile:JdbcProfile = StewardConfigSource.objectForName(slickProfileClassName)

  val schema = StewardSchema(slickProfile)
}

Did you notice that StewardConfigSource.config? Sometimes I need some control over the configuration for testing. I've already blogged about how I inject new config variables for testing. That does put in one extra hop to answer "Where does this Config come from?" However, outside of testing anyone's first guess will be right. I've preserved the most important easy answer for "How do I control this value?"

TL/DR - I added parallel versions of Dijkstra's and Brandes' algorithms to Disentangle. Writing the code was easy. Consider using parallel versions of these algorithms when you've got more than about 100 nodes in your graph and you have computational power to spare. Call them via

val simpleShortPathLabelsFromPar = Dijkstra.parAllPairsShortestPaths(edges)

val leastPathLabelsFromPar = Dijkstra.parAllPairsLeastPaths(edges,support,labelForEdge)

val shortestPathsAndBetweennessFromPar = Brandes.parAllLeastPathsAndBetweenness(edges)

Pull in the latest snapshot with

resolvers += "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"

libraryDependencies += "net.walend.disentangle" %% "graph" % "0.2.0-SNAPSHOT"

Scala's Parallel Collections

Scala 2.9 provided parallel versions of many of its standard collection classes. Functional operations on these collections happen in parallel using a default compute pool. Dijkstra's and Brandes' algorithms can be run in parallel for each node. Aleksandar Prokopec provided some encouraging advice - all I needed to do was run the outermost functional operations on a parallel collection of nodes to use all the cores available.

Fifteen Minutes Later

I needed about ten minutes for the first pass, and five for a second pass (Brandes' algorithm has an extra wrinkle) to get the algorithms running in parallel. Here's a code fragment from Dijkstra's algorithm:

def parAllPairsLeastPaths[Node,EdgeLabel,Label,Key](edges: GenTraversable[(Node, Node, EdgeLabel)],
                                                    support: SemiringSupport[Label, Key],
                                                    labelForEdge: (Node, Node, EdgeLabel) => Label,
                                                    nodeOrder: GenSeq[Node] = ParSeq.empty):ParSeq[(Node, Node, Label)] = {
  val labelDigraph = createLabelDigraph(edges.par, support, labelForEdge, nodeOrder.par)

  //profiler blames both flatten and fold of IndexedSet as trouble
  labelDigraph.innerNodes.to[ParSeq].flatMap(source => dijkstraSingleSource(labelDigraph, support)(source))
}

Making the edges collection parallel speeds up translating to the internal directed graph representation. The bigger benefit, running dijkstraSingleSource for each node in parallel, comes from putting the nodes in a parallel collection. This code was so straight-forward that I wrote it while bouncing on the T on my way home from work.

Results on an AWS EC2 r3.8xlarge

I spun up an AWS EC2 r3.8xlarge instance to benchmark on a quiet, modern, multicore computer with a quarter-terabyte of ram. Running the benchmarks was easy. I spent most of my developer time tweaking these performance graphs in D3.

The right half of this graph shows deviation from correct curves for Dijkstra's algorithm after about 4096 nodes. I think that's the garbage collector coming into play. (Lower is faster.) The r3.8xlarge let me use 238 GB for the JVM, which ran Dijkstra's algorithm on graphs with 16384 nodes before crashing into an out-of-memory error. I was expecting the parallel version of the algorithm to fill up memory faster than the serial version, but was pleasantly surprised that they ran out of memory on the same-sized graph. The parallel version found shortest paths for 16384 nodes in just over 15 minutes, about a 6X speedup over the serial version.

(You'll notice the curve for the Floyd-Warshall algorithm arcing up quickly from the lower left. The Floyd-Warshall test didn't crash; I had a spare half-hour of compute time after one of the larger tests and stopped it after that.)

To examine what was happening in that lower left corner I plotted the results log/log. (A little lower is much faster.) As you can see, the crossover point where concurrency starts to pay off seems to be at about 90 nodes (on a quiet AWS r3.8xlarge with 32 cores and 244 GB ram - YMMV). The Floyd-Warshall algorithm was never better for graphs with 32 or more nodes.

I found similar results for Brandes algorithm.

You can see some inefficiency - maybe the JVM garbage collector - start to come into play after about 4096 nodes, but it was able to find all shortest paths and betweenness for every node in just over 15 minutes for 16384 nodes on an EC2 r3.8xlarge. That's about a 6X speed-up over non-parallel. Not bad for 15 minutes of effort with no regard for Amdahl's law.

Try it out

Pull it into an sbt project via

resolvers += "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"

libraryDependencies += "net.walend.disentangle" %% "graph" % "0.2.0-SNAPSHOT"

Or clone the project and play in the console

git clone https://github.com/dwalend/Disentangle.git
cd Disentangle
sbt console

resolvers += "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"

libraryDependencies += "net.walend.disentangle" %% "graph" % "0.2.0-SNAPSHOT"

Disentangle

I renamed ScalaGraphMinimizer to Disentangle. I wanted to change the name for a while. ScalaGraphMinimizer is internet-unique, but was nearly impossible to say using a human mouth. My work with semiring-based minimization algorithms is mature at this point and I'd like project to grow in new directions. While working on parallel versions of algorithms this fall I realized ScalaGraphMinimizer was about to outgrow its name.

I picked the name Disentangle to provide a double-meaning of optimism of purpose and guidance for the code.

On its surface, Disentangle is a library of algorithms to help provide human insight for complex graph structures. In my day job too often I've projected a picture of a hairball of a graph on a screen in a dark room, narrating it with words like, "This graph is really complicated, but - next slide please - I found these interesting things that you may care about." Disentangle has already helped tell some stories and I plan to keep grown the library in that theme.

The second meaning is guidance for me creating the library and a promise to people using it. I want Disentangle to be a non-invasive library used via a few clean method calls, not a project-defining framework. It uses some of Scala's common-currency parts - collections and tuples - as the core of Disentangle's API. Specifically it does not provide a domain-specific language or require developers to extend its internal structures.

The above differentiates Disentangle from its predecessors. Some structures, like lists, sets, and maps, are ubiquitous, almost universal, and having them in a common core library makes sense. Other structures like a master-and-servant set of queues of AtomicMarkedReferences are purpose-built for their specialized tasks. Graphs fall in a gap between these extremes because graphs have a lot of lot of commonality and a lot of variation. Other graph frameworks require developers to begin by supplying graphs defined in that framework's structures. Developers who attempt to use those frameworks frequently give up when the framework doesn't meet their needs. They rip out the first attempt (or - far worse - leave it in to gum up the works) and replace it with custom one-use code. I hope that happens less with Disentangle.

Graphs, Collections, and Tuples

Computer languages are usually pretty good at defining graphs, at least directed graphs. A general-purpose programming language ought to provide a good starting point for general-purpose graph algorithms without demanding too much from developers. Scala - as a better Java - provides a lot of easily-accessible features through its type system. I chose those features for Disentangle's surface API. Disentangle's starting point is a collection of Tuples and a simple, single API call on an object. Code that uses it looks like this:

/**
 * Edges are just a Seq of Tuple3[Node,Node,Edge]
 */
val edges: Seq[(String, String, String)] = Seq(
                                              ("A","B","ab"),
                                              ("B","C","bc"),
                                              ("C","D","cd"),
                                              ("D","E","de"),
                                              ("E","F","ef"),
                                              ("E","B","eb"),
                                              ("E","H","eh"),
                                              ("H","C","hc")
                                            )

import net.walend.disentangle.graph.semiring.{Dijkstra,FirstStepsTrait}

/**
 * Generate all the shortest paths in the graph
 */
val simpleShortPathLabels = Dijkstra.allPairsShortestPaths(edges)

or with types defined:

val simpleShortPathLabels: Seq[(String, String, Option[FirstStepsTrait[String, Int]])] = 
    Dijkstra.allPairsShortestPaths(edges)

FirstStepsTrait has a Set of possible first steps to take on shortest paths, and the length of those paths.

Semirings Inside

Other frameworks get into trouble when you want something just a little beyond exactly what their algorithms can deliver. To customize the code you have to fork and bend the whole algorithm. However, Disentangle's implementations of the Floyd-Warshall algorithm, Dijkstra's algorithm, and Brandes' algorithm take a semiring to define what "shortest path" means. Dijkstra.allPairsShortestPaths() is a wrapper method around allPairsLeastPaths(), which takes a SemiringSupport object. The default version finds paths with the fewest nodes. Here's a code snippet from inside of object Dijkstra:

def defaultSupport[Node] = AllPathsFirstSteps[Node,Int,Int](FewestNodes)

def allPairsShortestPaths[Node,EdgeLabel](edges:GenTraversable[(Node,Node,EdgeLabel)],
                                      nodeOrder:GenSeq[Node] = Seq.empty
                                    ):Seq[(Node,Node,Option[FirstStepsTrait[Node, Int]])] = {
  val support = defaultSupport[Node]
  allPairsLeastPaths(edges, support, support.convertEdgeToLabel(FewestNodes.convertEdgeToLabel), nodeOrder)
}

Pick Weights via Semirings

My most specific complaint with other frameworks is their baked-in selection of what to use for weights. Disentangle is semiring-based, so you can supply the semiring -- you can use a weight that matches your needs. I created a straight-forward double-based semiring - LeastWeights - to provide traditional Double weights. It's easy to copy and bend to your needs.

import net.walend.disentangle.graph.semiring.{AllPathsFirstSteps,LeastWeights}
val support: AllPathsFirstSteps[String, Double, Double] = 
  new AllPathsFirstSteps(LeastWeights)

(If the edges aren't Doubles already you'll need to supply some function to convert your edges to Doubles. I wrote this cheezy hack as an example.)

def stringToDouble(fromNode:String,toNode:String,edge:String):Double = 
  edge.map(_.hashCode().toDouble).product

Instead of calling allPairsShortestPaths() call allPairsLeastPaths() to generate all the shortest paths.

val leastPathLabels: Seq[(String, String, support.Label)] = 
  Dijkstra.allPairsLeastPaths(edges,support,support.convertEdgeToLabel[String](stringToDouble))

Other Semirings

Because Disentangle is semiring-based, it's possible to create your own semirings have standard algorithms solve for something other than a "least path." For example, here's one for most probable paths.

Try it out

Pull it into an sbt project via

resolvers += "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"

libraryDependencies += "net.walend.disentangle" %% "graph" % "0.2.0-SNAPSHOT"

Or clone the project and play in the console

git clone https://github.com/dwalend/Disentangle.git
cd Disentangle
sbt console

TL/DR - I found a workable compromise for testing with different configurations using typesafe config without starting a new JVM or turning the code inside out. See this gist.

Simple Structures

TypeSafe Config code loads the config from a hierarchy of sources - primarily files on the classpath and java.lang.System properties. I can access my system's configuration from anywhere by calling

  val config = ConfigFactory.load();
  val dbUrl = config.getString("database.url")

I want to keep that simplicity. However, I also needed a good way to test with different configurations. My day-job project, Shrine's Data Steward Web App, has three different major configurations, maybe a dozen variations.

Specifically, I want the code to use simple structures that pull their configurations from a typesafe Config object. I want to be able to replicate the technique shamelessly without sharing code between subprojects. Further, I do not want that Config exposed in my system's Scala API; I should be able to use Scala's singleton [objects] (https://raseshmori.wordpress.com/2013/06/20/scala-part-4-classes-objects/) where I need singletons. At the day job, the configuration is unchanging once set - except for during tests. Having it front-and-center distracts from more important details. I should not have to couple my code via constructor parameters that make me wish for an aspect-oriented style. After all it's just config.

Three Compromises

I thought of three options. None are perfect, but one is good enough.

Start a New Process

The most brute force approach is to use a different JVM process for testing each configuration. That's how I'd have handled this back in the age of ant. However, Shrine uses maven. Creating a new JVM process for tests is out-of-model in Shrine and possibly in any maven project. I skipped from ant to sbt. During the age of maven I was at MathWorks where we used make. Maven feels as alien as a three-fingered glove to me. I don't know a tasteful way to spin up several new processes for running tests in the same maven subproject.

Config Parameter Objects

The older code in Shrine uses config parameter objects, skipping typesafe config, to test parts of the system in isolation. A maven subproject uses typesafe config to drive various abstract-factory-pattern-inspired parts to construct parts for Shrine. Testing different configurations means constructing specialized configuration helpers instead of using a simple Config. The approach definitely solves the problem, but it adds a long, twisty maze between the config files where a value is defined and the code that actually uses it. Following that path adds about five minutes to each task involving a config key-value pair. The approach seems particularly invasive because singletons that could have been Scala objects have to be constructed class instances. The existing pattern tightly couples the Shrine system around a single instance of a ShrineConfig class to keep the whole works from becoming a furball. Testing more than a single subsystem is very difficult. I found myself repeating my criticisms of Spring mixed with profane mutterings about GLOOP. Someday I want to clean up that part of the code, not add to the confusion.

Higher-Order Config Hack

I decided to wrap typesafe config with just a little mutability and use defs where I need configurable values. Using defs for configurable values forces them to be reevaluated each time the owning code accesses them; there will always be a little in-memory overhead. However, any part of the system can access the Config when needed, with a key's name right next to the def that supplies the value.

  def dbUrl = ExampleConfigSource.config.getString("database.url")

My first hack at the solution was to set and clean up system properties in a try/finally block, and use ConfigFactory's resetCache method. The project is a web app showing a database; I can afford a little compute overhead, but it just seemed sloppy. I won't share some of the uglier code, but the progression to something clean went fine. The second hack was to put the try/finally into a higher-order function. The third step started to look less hacky. I replaced the cache flush and system properties with API to use Config's withFallback() to get the default (cached and unchanging) Config. I put the changeable Config inside an AtomicReference for minimal concurrent safety. Finally I dressed it up in a Scala-style Try/Success/Failure . It's not fool-proof, but should be fine for running one test at a time.

Here's what the code looks like:

import java.util.concurrent.atomic.AtomicReference
import scala.util.{Failure, Success, Try}
import com.typesafe.config.{Config, ConfigFactory}

/**
 * Use to tweak a Config without clearing and reloading a new config (for testing).
 *
 * @author dwalend
 */
class AtomicConfigSource(baseConfig:Config) {
  val atomicConfigRef = new AtomicReference[Config](ConfigFactory.empty())
 
  /**
   * Get the atomic Config. Be sure to use defs for all 
   * config values that might be changed.
   */
  def config:Config = atomicConfigRef.get().withFallback(baseConfig)
 
  /**
   * Use the config in a block of code with just one key/value replaced.
   */
  def configForBlock[T](key:String,value:AnyRef,origin:String)(block: => T):T = {
    val configPairs = Map(key -> value)
    configForBlock(configPairs,origin)(block)
  }
 
  /**
   * Use the config in a block of code.
   */
  def configForBlock[T](configPairs:Map[String, _ <: AnyRef],origin:String)(block: => T):T = {
    import scala.collection.JavaConverters.mapAsJavaMapConverter
 
    val configPairsJava:java.util.Map[String, _ <: AnyRef] = configPairs.asJava
    val blockConfig:Config = ConfigFactory.parseMap(configPairsJava,origin)
    val originalConfig:Config = atomicConfigRef.getAndSet(blockConfig)
    val tryT:Try[T] = Try(block)
 
    val ok = atomicConfigRef.compareAndSet(blockConfig,originalConfig)
 
    tryT match {
      case Success(t) => {
        if(ok) t
        else throw new IllegalStateException(
          s"Expected config from ${blockConfig.origin()} to be from ${atomicConfigRef.get().origin()} instead.")
      }
      case Failure(x) => {
        if(ok) throw x
        else throw new IllegalStateException(
          s"Throwable in block and expected config from ${blockConfig.origin()} to be from ${atomicConfigRef.get().origin()} instead.",x)
      }
    }
  }
}

To use it, I create a Scala object to hold the config:

/**
 * A little object to let you reach your config from anywhere.
 * 
 * @author dwalend
 */
object ExampleConfigSource {
  //load from application.conf and the usual typesafe config sources
  val atomicConfig = new AtomicConfigSource(ConfigFactory.load()) 
 
  def config:Config = atomicConfig.config
 
  def configForBlock[T](key:String,value:AnyRef,origin:String)(block: => T):T = 
    atomicConfig.configForBlock(key,value,origin)(block)
}

To change config in a test, wrap the test code in a configForBlock:

  "Steward" should " accept query requests with no topic in 'just log and approve everything' mode " in {

    ExampleConfigSource.configForBlock("shrine.steward.createTopicsMode", 
                                        CreateTopicsMode.TopicsIgnoredJustLog.name){

      Post(s"/qep/requestQueryAccess/user/${researcherUserName}",InboundShrineQuery(5,"test query","Not even using a topic")) ~>
        addCredentials(qepCredentials) ~>
        route ~> check {
        assertResult(OK)(status)
      }
    }
  }