DEV Community: Andrew (he/him)

There is No Such Thing as a Unit Test

Andrew (he/him) — Wed, 30 Oct 2024 16:17:20 +0000

"The terms 'unit test' and 'integration test' have always been rather murky, even by the slippery standards of most software terminology."

-- Martin Fowler

"...there is no such thing as a Unit Test"

-- Michael Belivanakis

What is a Unit Test?

I typed the above question into a popular search engine, and the first three results I got back were as follows (emphasis mine)

A unit test is a block of code that verifies the accuracy of a smaller, isolated block of application code, typically a function or method. -- aws.amazon.com

A unit test is a way of testing a unit - the smallest piece of code that can be logically isolated in a system. In most programming languages, that is a function, a subroutine, a method or property. -- smartbear.com

Unit is defined as a single behaviour exhibited by the system under test (SUT), usually corresponding to a requirement. While it may imply that it is a function or a module (in procedural programming) or a method or a class (in object-oriented programming) it does not mean functions/methods, modules or classes always correspond to units. From the system-requirements perspective only the perimeter of the system is relevant, thus only entry points to externally-visible system behaviours define units. -- Kent Beck via Wikipedia

A problem I often see with modern software testing is that we lean toward the second definition too often: a unit is "the smallest piece of code that can be logically isolated in a system". The word "can" is carrying a lot of weight in this sentence. We can logically isolate just about anything.

"Is a unit a file?"

-- "Definitely not", I can hear you say.

"Assuming an object-oriented program, how about a class?"

-- "Probably not", you say with slightly less conviction.

"How about a method?"

-- "Probably", with a bit more confidence, agreeing with the first two results above.

"What if that method is 300 lines of code?"

-- "Ooh, yeah, you should probably break that out into smaller methods."

Suppose we do this. Let's break our 300-line method into, say, 10 methods of 30 lines each, as some CS professors seem to teach their students that this is a good rule of thumb for function length.

// before
def original(x: String, y: Int): Boolean = {

  // ...
  // hundreds of lines of code
  // ...

}

// after
def improved(x: String, y: Int): Boolean = {

  val intermediateValueA = a(x)
  val intermediateValueB = b(intermediateValueA, y)
  val intermediateValueC = c(intermediateValueB)
  // ...
  val intermediateValueH = h(intermediateValueG)
  val intermediateValueI = i(intermediateValueH)

  j(intermediateValueI)

}

private def a(x: String): Long = {
  // ...
}

private def b(z: Long, y: Int): Double = {
  // ...
}

// eight more private functions...

All of these methods could be private (or whatever your language's equivalent of that is). In that case, they can only be accessed by the class which contains them. They used to be all in one method anyway, so we can be sure nobody else is using this logic anywhere else.

But now we face another decision: should we write unit tests for all of these individual methods? For many of us, our gut reaction will be "yes". This could make refactoring more difficult in the future, though, because "the closer your tests are to the implementation the more sensitive they are to changes".

Anecdotally, I've worked on codebases with hundreds of tests like this, all tightly coupled to the production implementation. Adding one field to a class meant updating a hundred or more tests which didn't care about this field at all, but needed the new field in order to compile. The test changes regularly took longer to implement than the production changes.

Writing unit tests for these smaller methods might also require us to make them more public than they need to be; the world outside this class doesn't care about these individual methods, all it cares about is the one improved method which now ties them all together.

The real question is, are these functions "units" of code?

The answer is no.

As Kent Beck would say, these are not "entry points to externally-visible system behaviours".

The only externally-visible entry point in the refactored example above is the improved function, just as the original function was initially. But these pervasive ideas that...

large functions should be broken up into smaller ones, and
a "unit test" is a "test of a single function"

...combine to produce a result that is much worse than the sum of its parts: huge suites of tests tightly-coupled to unnecessarily-public production code that take too long to write and are difficult to maintain.

Outcomes like this lead many developers to believe things like...

"Most Unit Testing is Waste"

-- James O. Coplien

Kinds of Tests

Rather than thinking of tests along the traditional unit / integration / end-to-end spectrum, I think it's helpful to think along a few other dimensions

is this test fast or slow?
is this a black-box test or a white-box test?
is this test informed by development or does it inform development?

Fast and Slow Tests

Let me start by asserting that "fast" is not synonymous with "good", and "slow" is not synonymous with "bad" in this context.

Fast tests are tests that run in a few seconds, milliseconds, microseconds, or less. Fast tests, therefore, must be entirely in-memory. They do no disk IO and they make no network calls. They can be run every single time a code change is made without being a roadblock to development speed, and should therefore be run as part of the developer's inner loop. Every time you compile, you can run these tests.

Slow tests take several seconds, minutes, or hours to run. The dividing line between fast and slow tests is somewhere around 2-5 seconds. Slow tests may require reading large input files from disk, doing lots of computation, or communicating across the network. That is: they are IO, CPU, or network bound. Contract tests (which often spin up Docker containers) and performance tests (which may run gigabytes of data or thousands of requests through the system) are examples of slow tests. These tests should be run less regularly, as they can impact development speed: before each commit to main / master is probably fine for tests shorter than a few minutes, daily or weekly might be a good cadence for tests much longer than that.

Black-Box and White-Box Tests

Black-box tests make no assumptions about the internals of the thing they are testing. They provide inputs and assert on observable outputs, and that's it. The observable output is usually a return value from a method, but a black-box test might instead assert that a side effect has occurred, like that a log line has been written, or that a metric has been recorded, or that some state has been mutated.

White-box tests specifically test the internals of the thing they are testing. They are introspective. Tests which have assertions like "when 'x' happens, function a() should call function b()" are white-box tests. They are explicitly testing how something should happen (how some code is implemented), rather than testing only that it has happened. Tests which rely heavily on mocking frameworks are often white-box tests, asserting that such-and-such a method has (or hasn't) been called in response to some inputs.

If you don't care about how something is implemented -- just that that it does what it's supposed to do -- you should write a black-box test. This is usually the case, so opt for black-box tests as a default.

Development-Informed Tests and Development-Informing Tests

Development-informed tests are written reactively, in that the production code is written first, and the tests are written afterward. Development-informed tests codify the behaviour of the system as-is. Traditional "unit tests" are almost exclusively development-informed tests.

Development-informing tests are written proactively, in that a test is written first, and the production code is written after. Test-Driven Development (TDD) is a software development methodology which encourages writing only development-informing tests, ensuring that 100% of the system's behaviour is always codified in tests.

Development-informing tests can also provide confidence that some tricky piece of logic has been implemented correctly. For example, you might write a regex to parse U.S. phone numbers, and -- at the same time -- add a handful of tests to ensure that you catch things like

area codes surrounded by parentheses
spaces vs. no spaces vs. hyphens
the presence or absence of a +1 country code

It can be hard to be sure -- just by staring at the regular expression -- that it catches all of these cases. Usually it's more convincing to just write a handful of simple tests to convince yourself that the most common edge cases are being handled correctly.

I always write bug fix tests in a development-informing way, as well. First, I write a test which should pass, but which I expect to fail due to the presence of a bug. Then, I fix the bug in the production code, ensuring that the test now passes. This process shows that -- had the test existed originally -- it would have caught the bug. This gives confidence that the bug should not reappear in the future.

"Most Unit Testing is Waste"

The three ways of looking at tests outlined above can provide insight into why developers like James O. Coplien believe that most unit testing is a waste of time.

Most Unit Tests are Development-Informed

In my experience, TDD is not practiced by most developers.

Most tests, therefore, are development-informed. A developer writes some production code and then writes a test, usually to ensure that some code coverage minimum is reached.

These tests are not written to catch bugs, and they are not written to help a developer think through some difficult implementation, and so their value is not immediately apparent.

Most Unit Tests do not test "Externally-Visible System Behaviours"

As mentioned earlier, the twin practices of (1) breaking large functions up into smaller ones and (2) writing tests for each function rather than for each externally-visible system behaviour leads to a proliferation of tests tightly-coupled to the production implementation. These tests are, by their nature, fragile. They must be updated whenever the smallest implementation detail is changed, even if the externally-visible system behaviour is identical.

This often happens when using mocking frameworks, since every method called on a mocked object must be declared, with its return value specified.

In the worst-case scenario, developers will sometimes copy-and-paste the production implementation directly into the test, asserting that the "expected" result from the test implementation equals the "actual" result from the production code. This kind of white-box test unquestionably adds no value, even if it does increase "code coverage".

A New Test Pyramid

The traditional test pyramid aims to emphasise to developers that they should be mostly writing "unit tests", with fewer "integration tests" and only a handful of "end-to-end tests". Although different formulations of the pyramid may use different terms for the latter two levels, almost all agree that the base of the pyramid should be composed of "unit tests". Google recommends a 70% / 20% / 10% split of unit / integration / end-to-end tests.

The idea is that you should cover most of the code's logic in small, fast tests which can be run over and over during the inner loop of development. Your integration tests should cover interactions between units; and your end-to-end tests should validate that an end user's actions result in some expected overall outcome.

That advice is fine, provided all developers agree what constitutes a "unit test" or an "integration test". Clearly this is not the case. (See the search engine results at the top of this blog post.) However, we can use the objective criteria above (fast vs. slow, black-box vs. white-box, development-informed vs. development-informing) to construct a New Test Pyramid.

The Base

Opt for black-box tests wherever possible. Where external dependencies are required, prefer fake implementations rather than mocks (and add corresponding contract tests to ensure that external dependency behaves as you think it does). This keeps the entire test in-memory, making it fast enough to run before each commit to main / master. You will find that most of your tests are these fast, black-box tests.

Note that this is not synonymous with "unit test". As discussed above, traditional "unit tests" are usually fast, but are sometimes white-box tests, and are often development-informed.

The Middle

Prefer development-informing tests over development-informed tests (prefer a TDD style of development). Development-informed tests are often written by rote and offer little value.

Prefer slow black-box tests over fast white-box tests. The former are easier to maintain as they are less tightly-coupled to the production implementation.

Traditional "integration tests" and "end-to-end" tests both fall into the "slow, black-box tests" category.

The Top

Write as few white-box tests as possible. That is, do as little code introspection as possible. Only test observable outputs.

Write development-informed tests only when necessary. If the production implementation works, it works. If it doesn't, you will find a bug, write a development-informing test, and fix the bug. This process is described above.

Conclusion

The traditional unit / integration / end-to-end categorization of tests is fuzzy at best. Differing interpretations of what constitutes a "unit test", combined with well-intentioned but misapplied advice on keeping code readable by reducing the number of lines per function, class, etc. has led to a proliferation of hard-to-maintain, low-value test suites that negatively impact developer productivity.

Categorizing tests objectively, using the three criteria described above, can lead to more maintainable tests which provide more value.

Code as Art

Andrew (he/him) — Sat, 17 Aug 2024 17:37:27 +0000

Banner Image: Quine Relay -- Copyright (c) 2013, 2014 Yusuke Endoh (@mametter), @hirekoke

Digital Art

In almost all circumstances, code is a means to an end.

The phrase "computer programming" itself describes the activity of programming a computer to accomplish a particular task. Often, that task is logical. Most computer programs do something "useful", whether that's calculating the best route from your home to work, balancing a budget, or running your smart fridge's Twitter client.

But many programs are written solely to produce something of aesthetic value.

Recently, Generative AI applications like Midjourney, which can produce visual art from text prompts, have seen a lot of press. But consider also dwitter.net, where JavaScript programs of 140 characters or fewer create moving 3D landscapes, gyrating fractals, and even self-playing games of Pong. And also programmers like Grzegorz Witczak, Ben Evans, and Diana A. Smith who create photorealistic works of art using only CSS, a language otherwise used mainly to style fonts, borders, and backgrounds on web pages.

Digital art is not a new phenomenon. Nearly as long as computers have existed, programmers have been using them to express themselves aesthetically.

Traditional artists have been inspired by digital art, as well. Consider Dalí's Painting of Gala looking at the Mediterranean sea which from a distance of 20 meters is transformed into a portrait of Abraham Lincoln (Homage to Rothko), produced in 1976, which appears to show a pixelated image of Abraham Lincoln when viewed from a distance.

For most digital artists, code is a medium, like marble for a sculptor, or a canvas for a painter. Code is the material through which the art is expressed. Though it takes time and skill to produce, a canvas itself is not usually seen as art itself. But could it be?

Readability

Developers often talk about writing "clean" or "elegant" code. Most programming languages are "general-purpose", meaning they can be used to accomplish a variety of tasks. Accordingly, there are often many different ways to accomplish the same task, using the same programming language.

Robert C. Martin's Clean Code: A Handbook of Agile Software Craftsmanship gives many examples of "unclean" code, with tips and rules of thumb for rewriting the code in a "clean" way.

In the overwhelming majority of cases, the most important aspect of code is its functionality -- usually to perform some computational task. A secondary aspect of the code is its readability, closely related to the "cleanliness", "elegance", or "aesthetics" of the code.

As a quick example, consider this 3x3 matrix

val identity = Array(1, 0, 0, 0, 1, 0, 0, 0, 1)

Which elements are on the diagonal of this matrix?

It's much more readable if we manually format it as follows

val identity = Array(1, 0, 0,
                     0, 1, 0,
                     0, 0, 1)

These two arrays are functionally identical. But aesthetically, the source code used to construct the latter array is more "readable" than the former. Functionality is paramount, but aesthetics are a close second.

But what if a program (or an entire programming language) were designed primarily for its aesthetic qualities?

Unreadability

Formatting code for readability -- or to follow established patterns and idioms -- is the norm. But some programmers do the opposite (intentionally or otherwise).

Every year, the International Obfuscated C Code Contest (IOCCC) accepts submissions of C code which accomplish some task, but do it in an unusual, hard-to-decipher way. These programs are functional, but are purposely formatted so as to be as unreadable as possible (they are obfuscated).

Some programming languages have been designed specifically to accomplish as much as possible in as few characters as possible. APL is a "serious" (non-"esoteric") and very terse programming language which programmers may be familiar with, but many code golf languages have been created specifically to minimize source code size in a similar way.

And some programmers choose to format their code so that their entire ray-tracing C++ program can fit on the back of a business card, or so the source code of their ouroborous quine written in Ruby looks like a dragon (see the header image of this blog post), or so that they look like and can be read as poems. All of these examples use "serious" programming languages in an "unserious" way.

Esoteric Languages

There are also many programming languages where the aesthetics of the source code is more important than what is actually written in that code. As most programming languages are written using ASCII or Unicode character sets, some of these look like ASCII art...

　　　　_, ._
　　（　・ω・）　んも〜
　　○=｛=｝〇,
　 　|:::::::::＼, ', ´
､､､､し ､､､((（.＠）ｗｖｗｗＷＷｗｖｗｗＷｗｗｖｗｗｗｗＷＷＷｗｗＷｗ
ｗＷＷＷＷＷＷｗｗｗｗＷｗｗｖｗＷＷｗＷｗｗｖｗＷＷＷ

...though most just look like jumbled messes of characters

 72_1_108::_3+:}
    0          _ """
     11_{78_23_" " "
     4         4 ".{@
     _         " {
     1           }"
     08_100_33"""""

Some esoteric programming languages are non-textual. The source code of these programs can be visual, as with Piet

or Nice (which does not yet have a working interpreter)

Other languages are auditory, meaning programs are encoded as sound. Usually, these programs can also be written as individual notes, or as sheet music

Listen to the above program

In many cases, the simplest implementation of a program in these languages is not the most aesthetically pleasing. Programmers will opt for beautiful programs over efficient ones.

For example, the following Piet program gets a number as input from the user, squares it, and prints the result

...but so does this one

Code as Art

"There is no generally agreed definition of what constitutes art" [Wikipedia]

Even in languages which aren't specifically designed to have "beautiful" source code, some programmers opt to format their code in ways which trade efficiency, practicality, or readability for aesthetic appeal.

There is no single, correct solution to a problem which can be solved by a program. Different programs can accomplish the same task in different ways. Weighing tradeoffs and choosing one solution over another is part of the art of computer programming.

The human desire to be creative and express one's individuality does not end where a keyboard begins. Hundreds of esoteric programming languages can attest to that.

Whether textual, visual, or auditory, code can be used as a medium to create art, but it can also be art itself.

Drop Everything and Review

Andrew (he/him) — Wed, 24 Apr 2024 23:07:27 +0000

The Inner and Outer Development Loops

Feedback is critical when developing software; the earlier it can be given, the better. Static code analysis tools will immediately write red squigglies underneath code, as it's being written, to let a programmer know that there's a syntax error. (In fact, as I'm writing this blog post, I'm getting blue underlines from my spellchecker.) Unit tests, similarly, give very quick feedback, validating the logic of any newly-written code, and ensuring that there are no regressions (that the new code doesn't cause previously-written tests to break).

This very fast code-build-test feedback loop is often called the inner loop of software development. It requires no input from other people, and the time for a complete loop can be as short as the time it takes static analysis to run, which might be only a few hundred milliseconds. The only blockers in the inner loop are the speed of the compiler or interpreter and the speed at which the developer can program.

The outer loop, by contrast, usually refers to the process of deploying code to a live environment and iterating based on integration with the larger system, acceptance tests, performance tests, and so on (though different sources disagree about which steps should be included in the outer loop). This loop is much more involved and, accordingly, takes much more time to provide feedback to the developer. Depending on how fast your CI / CD pipeline runs, and whether your QA tests are automated or manual, it could take anywhere from minutes to days.

But I propose that there is a middle loop, as well, which often gets incorrectly lumped into the outer loop: the code review loop.

The Middle Loop

The Middle Development Loop involves getting feedback from developers and other stakeholders, adjusting your code according to that feedback, and then requesting additional feedback. It is distinct from the inner loop, which is traced by a single developer in a single codebase in a single environment, as well as from the outer loop, which is traced by many developers across many environments. The key differentiator here is that this is the first time that multiple stakeholders will be looking at the same new bit of code.

This often happens during code review, but feedback can come in many forms: pair programming, mob programming, synchronous code review, as well as traditional asynchronous code review. The latter is, in many workplaces, the standard: you create a PR, add some reviewers, and go off and do something else while you wait for feedback.

This is usually fine, provided you have lots of tasks to work on and can context switch easily from one to another while waiting for reviews. And while this is often the case, in my experience, the opposite happens just as often: you're building some feature across multiple code bases, or in multiple steps, and you need reviews and approvals at each stage. In this case, each time a review is required, it becomes a blocker.

Developers can get frustrated waiting for reviews and glob multiple fixes into a single PR to reduce the number of reviews required, or they might ping other developers directly to ask for reviews. Those pinged might be taken out of their flow state, or even feel harassed, if contacted over and over.

For all parties involved, it can be a very frustrating experience.

If you have the power to do so in your organization, push for synchronous code reviews, which greatly reduce the size of the middle loop, or for synchronous coding, like pair or mob programming, which eliminates the middle loop entirely. For many reading this, though, I know this is not an option. You've got async code reviews and you're stuck with them. In that case, let's try to make the best of a less-than-ideal situation: what is it that makes async review frustrating?

The problem with async code review is that it often unnecessarily inflates the duration of the middle loop.

Code review can be a thankless job. When's the last time you heard of someone getting promoted for being the most thorough reviewer on a team? Plus, it's easy to hide in a crowd, and if 10 developers are added as reviewers to a PR which only needs 2 approvals, they may ignore the request, assuming that their teammates will pick up the slack. Even when a developer does review some code, they may delay doing so. Waiting until you're finished your current task, or until after lunch, or even until after your next meeting, to review a PR may not seem like a delay for you. In fact, in may seem downright prompt. But to the developer waiting for a review to continue their work, they are blocked.

The next time you put up a PR, you will also need reviews. And just like your coworkers, you will want them as quickly as possible. So remember that when someone requests a code review, the courteous thing to do is to Drop Everything and Review (DEAR).

Minimize the Middle Loop, Dear

After handling any active production incidents, the first thing on a developer's // TODO list should always be reviewing code.

If you've just sat down at your desk and are looking for something to work on, checking which pull requests you've been assigned to should become a reflex.

Don't start that little bug fix, don't read that next blog post (unless it's this one), don't check your inbox.

Remember DEAR and do the courteous thing: Drop Everything and Review.

What Are Const Generics and How Are They Used in Rust?

Andrew (he/him) — Mon, 25 Mar 2024 15:02:39 +0000

I was working through an example in the repo for the Bevy game engine recently and came across this code

/// Update the speed of `Time<Virtual>.` by `DELTA`
fn change_time_speed<const DELTA: i8>(mut time: ResMut<Time<Virtual>>) {
    let time_speed = (time.relative_speed() + DELTA as f32)
        .round()
        .clamp(0.25, 5.);

    // set the speed of the virtual time to speed it up or slow it down
    time.set_relative_speed(time_speed);
}

This is a function (fn) which takes a mutable argument called time. The type of time, ResMut<Time<Virtual>>, comes after the colon, :.

The thing that caught my eye here was the generic parameter: <const DELTA: i8>. What is that?

Here's another example from Bevy

pub unsafe fn read<T>(self) -> T {
    let ptr = self.as_ptr().cast::<T>().debug_ensure_aligned();
    // -- snip --
}

The read function takes a generic type parameter T and uses it in two places: in the body of the function, and as a return type. Programmers who are familiar with generics know that an unconstrained T is a placeholder that means "any type"; it could be String or bool or anything else.

In languages with a global type hierarchy, like Java, a value t: T has some operations which can be performed on it, like .toString(), because every type T in Java extends the base Object type. Rust has no such global type hierarchy, and no root Object type, so there's not much at all you can do with an instance of an unconstrained type.

Going back to the first example, const DELTA: i8 clearly already has a type, appearing after the colon, :. (It is i8, an 8-bit signed integer.) So what is it doing sitting between those angle brackets (<>) where generic parameters usually sit?

In this position, const DELTA: i8 is acting as a const generic.

What Are Const Generics?

Const generic parameters are a new (ish) kind of generic parameter in Rust, similar to type parameters (e.g. T) and lifetime parameters (e.g. 'a). In the same way that a function (or method, struct, enum, impl, trait, or type alias) can use a generic type parameter, it can also use const generic parameters.

Const generic parameters are what power [T; N] type annotation of arrays in Rust. They are why [T; 3] (an array of three T values) and [T; 4] (an array of four T values) are different types, but different types which can be handled generically as specific implementations of [T; N].

Const generic parameters allow items to be generic over constant values, rather than over types.

The difference can be subtle. Here's a simple example

fn add<const b: i8>(a: i8) -> i8 {
  a + b
}

Here, b is not a "type parameter"; it is a value, and so it can be treated exactly as a value, used in expressions, and so on. But since it is const, the value of b must be known at compile time. For example, the following will not compile

fn example(a: i8, b: i8) -> i8 {
    add::<b>(a) // error: attempt to use a non-constant value in a constant
}

The logic in this function, of course, could also be expressed like

fn add(a: i8, b: i8) -> i8 {
  a + b
}

...so what's the benefit of const generics? Let's look at some other examples

Using Const Generics to Enforce Correctness

There are a few examples from linear algebra where const generics are very helpful. For example, the dot product of two vectors a and b, is defined for any two vectors of any dimensionality (length), provided they have the same dimensionality

struct Vector<const N: usize>([i32; N]);

impl<const N: usize> Vector<N> {
    fn dot(&self, other: &Vector<N>) -> i32 {
        let mut result = 0;
        for index in 0..N {
            result += self.0[index] * other.0[index]
        }
        result
    }
}

We get a compile-time error if we try to find the dot product of two vectors with different numbers of elements

fn main() {
    let a = Vector([1, 2, 3]);
    let b = Vector([4, 5, 6]);

    assert_eq!(a.dot(&b), 32); // ok: a and b have the same length

    let c = Vector([7, 8, 9, 10]);

    a.dot(&c); // error: expected `&Vector<3>`, but found `&Vector<4>`
}

Const generics can be applied to matrix multiplication, as well. Two matrices can be multiplied only if the first one has M rows and N columns and the second has N rows and P columns. The resulting matrix will have M rows and P columns.

struct Matrix<const nRows: usize, const nCols: usize>([[i32; nCols]; nRows]);

impl<const M: usize, const N: usize> Matrix<M, N> {
    fn multiply<const P: usize>(&self, other: &Matrix<N, P>) -> Matrix<M, P> {
        todo!()
    }
}

Here, we again get a compile-time error if we ignore this constraint

fn main() {
    let a = Matrix([[1, 2, 3], [4, 5, 6]]); // 2 x 3 matrix
    let b = Matrix([[1, 2, 3, 4], [2, 3, 4, 5], [3, 4, 5, 6]]); // 3 x 4 matrix

    a.multiply(&b); // ok: 2 x 4 matrix

    let c = Matrix([[1, 2, 3], [2, 3, 4]]); // 2 x 3 matrix

    a.multiply(&c); // error: expected `&Matrix<3, <unknown>>`, but found `&Matrix<2, 3>`
}

These constraints can be enforced at runtime without const generics, but const generics can help shift these issues left, catching them earlier in the development process, tightening the inner dev loop.

Using Const Generics to Conditionally Implement `trait`s

(Adapted from Nora's example here.)

Const generics also enable really powerful patterns, like compile-type checks on values in signatures. For example...

struct Assert<const COND: bool> {}

...this struct takes a constant generic bool parameter, COND. If we define a trait IsTrue...

trait IsTrue {}

impl IsTrue for Assert<true> {}

...we can conditionally implement traits by requiring some Assert to impl IsTrue, like so

trait IsOdd<const N: i32> {}

impl<const N: i32> IsOdd<N> for i32 where Assert<{N % 2 == 1}>: IsTrue {}

The above Assert<{N % 2 == 1}> requires #![feature(generic_const_exprs)] and the nightly toolchain. See https://github.com/rust-lang/rust/issues/76560 for more info.

Above, trait IsOdd is implemented for the i32 type, but only on values N which satisfy N % 2 == 1. We can use this trait to get compile-time checks that constant (hard-coded) i32 values are odd

fn do_something_odd<const N: i32>() where i32: IsOdd<N> {
    println!("oogabooga!")
}

fn do_something() {
    do_something_odd::<19>();
    do_something_odd::<42>(); // does not compile
    do_something_odd::<7>();
    do_something_odd::<64>(); // does not compile
    do_something_odd::<8>(); // does not compile
}

The above will generate a compiler error like

error[E0308]: mismatched types
  --> src/main.rs:70:5
   |
70 |     do_something_odd::<42>();
   |     ^^^^^^^^^^^^^^^^^^^^^^^^ expected `false`, found `true`
   |
   = note: expected constant `false`
              found constant `true`

Using Const Generics to Avoid Complex Return Types

Finally, const generics can be used to make code more readable, and more performant. The example from the beginning of this post comes from Bevy, and the reason const generics are used there is because Bevy is expecting a function pointer as an argument to a method

fn main() {
    App::new()
        // -- snip --
        .add_systems(
            Update,
            (
                // -- snip --
                change_time_speed::<1>.run_if(input_just_pressed(KeyCode::ArrowUp)),
                // -- snip --
            ),
        )
        .run();
}

change_time_speed::<1>, above, is a function pointer. We can rearrange this method to take an argument, rather than using a const generic parameter...

change_time_speed_2(1).run_if(input_just_pressed(KeyCode::ArrowUp)),

...but then we would have to change the return type as well

/// Update the speed of `Time<Virtual>.` by `DELTA`
fn change_time_speed_2(delta: i8) -> impl FnMut(ResMut<Time<Virtual>>) {
    move |mut time| {
        let time_speed = (time.relative_speed() + delta as f32)
            .round()
            .clamp(0.25, 5.);

        // set the speed of the virtual time to speed it up or slow it down
        time.set_relative_speed(time_speed);
    }
}

To many, the original function may be more readable

/// Update the speed of `Time<Virtual>.` by `DELTA`
fn change_time_speed<const DELTA: i8>(mut time: ResMut<Time<Virtual>>) {
    let time_speed = (time.relative_speed() + DELTA as f32)
        .round()
        .clamp(0.25, 5.);

    // set the speed of the virtual time to speed it up or slow it down
    time.set_relative_speed(time_speed);
}

Remember, as well, that Rust uses monomorphization of generics to improve runtime performance. So not only is the const generic version of this function more readable, but it's possible (though I haven't benchmarked) that it's more performant as well. Either way, it's good to know that there are multiple ways to attack a problem, and to be able to weigh the pros and cons of each approach.

Hopefully this discussion has helped you to understand what const generics are, and how they can be used in Rust.

Make Invalid States Unrepresentable

Andrew (he/him) — Fri, 02 Feb 2024 17:07:17 +0000

Suppose you have a Person class in your program, and that a Person has an age. What type should the age be?

`age` as a `String`

case class Person(age: String)

"Of course it shouldn't be a String" you might think. But why? The reason is that we can then end up with code like

val person = Person("Jeff")

If we ever wanted to do anything with an age: String, we would need to validate it everywhere

def isOldEnoughToSmoke(person: Person): Boolean = {
  Try(person.age.toInt) match {
    case Failure(_) => throw new Exception(s"cannot parse age '${person.age}' as numeric")
    case Success(value) => value >= 18
  }
}

def isOldEnoughToDrink(person: Person): Boolean = {
  Try(person.age.toInt) match {
    case Failure(_) => throw new Exception(s"cannot parse age '${person.age}' as numeric")
    case Success(value) => value >= 21
  }
}

// etc.

This is cumbersome for the programmer writing the code, and makes it difficult for any programmer reading the code, as well.

We could move this validation to a separate method

def parseAge(age: String): Int = {
  Try(age.toInt) match {
    case Failure(_) => throw new Exception(s"cannot parse age '$age' as numeric")
    case Success(value) => value
  }
}

def isOldEnoughToSmoke(person: Person): Boolean =
  parseAge(person.age) >= 18

def isOldEnoughToDrink(person: Person): Boolean =
  parseAge(person.age) >= 21

...but this is still not ideal. The code is a bit cleaner, but we still need to parse a String into an Int every time we want to do anything numeric (comparison, arithmetic, etc.) with the age. This is often called "stringly-typed" data.

This can also move the program into an illegal state by throwing an Exception. If we're going to fail anyway, we should fail fast. We can do better.

`age` as an `Int`

Your first instinct might have been to make age an Int, rather than a String

case class Person(age: Int)

If so, you have good instincts. An age: Int is much nicer to work with

def isOldEnoughToSmoke(person: Person): Boolean =
  person.age >= 18

def isOldEnoughToDrink(person: Person): Boolean =
  person.age >= 21

This

is easier to write
is easier to read
fails fast

You cannot construct an instance of the Person class with a String age now. That is an invalid state. We have made it unrepresentable, using the type system. The compiler will not allow this program to compile.

Problem solved, right?

val person = Person(-1)

This is clearly an invalid state as well. A person cannot have a negative age.

val person = Person(90210)

This is also invalid -- it looks like someone accidentally entered their ZIP code instead of their age.

So how can we constrain this type even further? How can we make even more invalid states unrepresentable?

`age` as an `Int` with constraints

at runtime

We can enforce runtime constraints in any statically-typed language

case class Person(age: Int) {
  assert(age >= 0 && age < 150)
}

In Scala, assert will throw a java.lang.AssertionError if the assertion fails.

Now we can be sure that the age for any Person will always be within the range [0, 150). Both

val person = Person(-1)

and

val person = Person(90210)

will now fail. But they will fail at runtime, halting the execution of our program.

This is similar to what we saw in "age as a String", above. This is still not ideal. Is there a better way?

at compile time

Many languages allow compile-time constraints, as well. Usually this is accomplished through macros, which inspect the source code during a compilation phase. These are often referred to as refined types.

Scala has quite good support for refined types across multiple libraries. A solution using the refined library might look something like

case class Person(age: Int Refined GreaterEqual[0] And Less[150])

A limitation of this approach is that the field(s) to be constrained at compile-time must be literal, hard-coded values. Compile-time constraints cannot be enforced on, for example, values provided by a user. By that point, the program has already been compiled. In this case, we can always fall back to runtime constraints, which is often what these libraries do.

For now, we'll continue with runtime constraints only, since often that's the best we can do.

`age` as an `Age` with constraints

From simplest to most complex implementation, we moved left to right in the diagram below

String => Int => Int with constraints

This increase in complexity directly correlates with the accuracy with which we're modelling this data

"The problems tackled have inherent complexity, and it takes some effort to model them appropriately." [source]

The move left-to-right above should be driven by the requirements of your system. You should not implement compile-time refinements, for example, unless you have lots of hard-coded values to validate at compile time: otherwise you aren't gonna need it. Every line of code has a cost to implement and maintain. Avoiding premature specification is just as important as avoiding premature generalization, though it's always easier to move from more specific types to less specific types, so prefer specificity over generalization.

Every bit of data has a context, as well. There is no such thing as a "pure" Int value floating around in the universe. An age can be modelled as an Int, but it's different from a weight, which could also be modelled as an Int. The labels we attach to these raw values are the context

case class Person(age: Int, weight: Int) {
  assert(age >= 0 && age < 150)
  assert(weight >= 0 && weight < 500)
}

There is one more problem for us to solve here. Suppose I'm 81kg and 33 years old

val me = Person(81, 33)

That compiles, but... it shouldn't. I swapped my weight and age!

An easy way to avoid this confusion is to define some more types. In this case, newtypes

case class Age(years: Int) {
  assert(years >= 0 && years < 150)
}

case class Weight(kgs: Int) {
  assert(kgs >= 0 && kgs < 500)
}

case class Person(age: Age, weight: Weight)

The name newtype for this pattern comes from Haskell. This is a simple way to ensure that we don't accidentally swap values with the same underlying type. The following, for example, will not compile

val age = Age(33)
val weight = Weight(81)

val me = Person(weight, age) // does not compile!

We could also use tagged types. In Scala, the simplest possible example of this looks something like

trait AgeTag
type Age = Int with AgeTag

object Age {
  def apply(years: Int): Age = {
    assert(years >= 0 && years < 150)
    years.asInstanceOf[Age]
  }
}

trait WeightTag
type Weight = Int with WeightTag

object Weight {
  def apply(kgs: Int): Weight = {
    assert(kgs >= 0 && kgs < 500)
    kgs.asInstanceOf[Weight]
  }
}

case class Person(age: Age, weight: Weight)

val p0 = Person(42, 42) // does not compile -- an Int is not an Age
val p1 = Person(Age(42), 42) // does not compile -- an Int is not a Weight  
val p2 = Person(Age(42), Weight(42)) // compiles!
val p3 = Person(Weight(42), Weight(42)) // does not compile -- a Weight is not an Age

This makes use of the fact that function application f() is syntactic sugar in Scala for an apply() method. So f() is equivalent to f.apply().

This approach allows us to model the idea that an Age / a Weight is an Int, but an Int is not an Age / a Weight. This means we can treat an Age / a Weight as an Int and add, subtract, or do whatever other Int-like things we want to do.

Mixing these two approaches in one example, you can see the difference between newtypes and tagged types. You must extract the "raw value" from a newtype. You do not need to do this with a tagged type

// `Age` is a tagged type
trait AgeTag
type Age = Int with AgeTag

object Age {
  def apply(years: Int): Age = {
    assert(years >= 0 && years < 150)
    years.asInstanceOf[Age]
  }
}

// `Weight` is a newtype
case class Weight(kgs: Int) {
  assert(kgs >= 0 && kgs < 500)
}

// `Age`s can be treated as `Int`s, because they _are_ `Int`s
assert(40 == Age(10) + Age(30))

// `Weight`s are not `Int`s, they _contain_ `Int`s
Weight(10) + Weight(30) // does not compile

// To add `Weight`s, we must "unwrap" them
Weight(10).kgs + Weight(30).kgs

In some languages, the "unwrapping" of newtypes can be done automatically. This can make newtypes as ergonomic as tagged types. For example, in Scala, this could be done with an implicit conversion

implicit def weightAsInt(weight: Weight): Int = weight.kgs

// `Weight`s are not `Int`s, but they can be _converted_ to `Int`s
Weight(10) + Weight(30) // this now compiles

Further refinements

The important point of the above discussion is that, as much as possible, we want to make invalid states unrepresentable.

"Jeff" is an invalid age. Age isn't a string, it is a number.
-1 is an invalid age. Age cannot be negative, it should be 0 or positive, and probably less than about 150.
My age is not 88. An age should be easily distinguishable from other integral values, like weight.

Everything discussed above implemented these refinements on the concept of "age", one at a time.

We can make further refinements if there is a need for those refinements.

For example, suppose we want to send a "Happy Birthday!" email to a Person on their birthday. Rather than an Age, we now need a date of birth.

case class Date(year: Year, month: Month, day: Day)

case class Year(value: Int, currentYear: Int) {
  assert(value >= 1900 && value <= currentYear)
}

case class Month(value: Int) {
  assert(value >= 1 && value <= 12)
}

case class Day(value: Int) {
  assert(value >= 1 && value <= 31)
}

case class Person(dateOfBirth: Date, weight: Weight) {
  def age(currentDate: Date): Age = {
    ??? // TODO calculate Age from dateOfBirth
  }
}

The amount of information provided by dateOfBirth is strictly greater than the amount of information provided by Age. We can calculate someone's age from their date of birth, but we cannot do the opposite.

The above implementation leaves much to be desired, though -- there are lots of invalid states. A better way to implement this would be for Month to be an enum, and for Day validity to depend on the Month (February never has 30 days, for example)

case class Year(value: Int, currentYear: Int) {
  assert(value >= 1900 && value <= currentYear)
}

sealed trait Month

case object January extends Month
case object February extends Month
case object March extends Month
case object April extends Month
case object May extends Month
case object June extends Month
case object July extends Month
case object August extends Month
case object September extends Month
case object October extends Month
case object November extends Month
case object December extends Month

case class Day(value: Int, month: Month) {
  month match {
    case February => assert(value >= 1 && value <= 28)
    case April | June | September | November => assert(value >= 1 && value <= 30)
    case _ => assert(value >= 1 && value <= 31)
  }
}

case class Date(year: Year, month: Month, day: Day)

Always prefer low-cardinality types to high-cardinality types, when possible. It limits the number of possible invalid states. In most languages, enums are the way to go here (in Scala 2, an enum can be modelled using a sealed trait, as shown above). But there are still invalid states hiding above. Can you find them?

In some cases, stringly-typed data validated using regular expressions can be replaced entirely by enums. Could you model Canadian postal codes such that it's impossible to construct an invalid one?

Use the above knowledge to go forth and make invalid states unrepresentable.

Start building browser games with Rust!

Andrew (he/him) — Mon, 15 Jan 2024 14:59:15 +0000

Interested in gamedev in Rust?

Me too!

So I put together this bare-minimum tutorial for using SDL2 and compiling a pure Rust app to WASM! Making browser games in Rust with WebAssembly just got a bit easier.

https://github.com/awwsmm/hello-rust-sdl2-wasm

What This Senior Developer Learned From His First Big Rust Project

Andrew (he/him) — Tue, 09 Jan 2024 15:15:48 +0000

cover photo by Pixabay

Here is a bit of background on me

according to my company's org chart on Workday, my current title is "Senior Consultant"
I've been writing code full-time in various capacities for over a decade and I've been professionally developing software for about five years
in graduate school, I did data analysis and visualization almost entirely in C++
for the last four years, my primary development language has been Scala

This blend of "close-to-the-metal" development in C++ and FP-style development in Scala has led me to Rust, which I see as a pretty usable middle ground between the two. So over the past year I've been learning Rust by building small projects and leading weekly book clubs.

Over the holidays, I decided to take this a step further and build my first "big" Rust project. Here's how that went down...

TL;DR: if you're only interested in the technical discussion, and not so much the project background, skip to the section on implementation. If you just want my conclusions, skip to the end.

The Project

https://github.com/awwsmm/rust-mvp

The idea I had was to build a small Internet of Things (IoT) system. The explicit goals of the project were

to build some services which used very few resources, capable of running in environments where the size of the Java Runtime Environment (JRE) would make Scala or Java development impossible
the services should run on separate nodes and somehow discover each other on the network, without the need for hard-coded IPs
the services should be able to send messages to (and receive messages from) one another
there should be some simulated data in the system, which can be visualized (or, at least, exported to a spreadsheet for visualization)

In addition, I work for a consulting company, and the client we are engaged with was OOO over Christmas. So another goal of this project was to have all of this completed, from scratch, in just five working days.

I managed to recruit two other developers* who helped build some of the foundations of the project in those first five days; in the two weeks since, I've built out the rest of the project by myself. In general, I consider the effort a success, but am hoping that whoever reads this might be able to leave some valuable feedback which could improve future efforts of this kind.

* Huge shout-out to boniface and davidleacock!

Planning

The other two developers and I spent the week before Christmas planning and discussing the project, but not coding. We were hoping that "team of three developers builds a Rust IoT MVP in just five days" would be an effective sales tool for ourselves and our company. It was very ambitious, and the work soon spilled over into about four person-weeks total (which is still not bad, if you ask me).

I prepped by writing some sketches, as I called them. These were little projects that (I'd hoped) would become the building blocks of our MVP. These sketches included

I also created a custom Rust-based container image for the CD project, which includes the necessary libraries for the CI project, like rustfmt for formatting, clippy for linting, and grcov for code coverage reporting.

While I originally thought of containerizing these applications, running them in Kubernetes (K8s), and letting K8s do the service discovery, I realized that that approach wouldn't square with "real life", where the services would somehow have to discover each other on a LAN. mDNS seemed the best choice to emulate real-life service discovery on a network.

Finally, we had to plan the domain itself. We came up with something quite similar to this example from Bridgera

a Sensor collects data from the Environment, and somehow communicates that data to...
a Controller, which assesses that data and (optionally) sends a Command to...
an Actuator, which has some effect on...
the Environment, which, in our example, generates fake data and has some internal state which can be modified via Actuator Commands

These four kinds of Devices -- Sensors, Actuators, and the Controller and Environment, are the services in this system. They connect to each other via mDNS.

As we were short on time and resources, all of this was to be done in software, with no actual interaction with any hardware sensors or actuators. Because of this, we needed a simulated Environment, which could generate fake data for our Sensors.

From the outset, we realized it was important to have Ubiquitous Language around these concepts. We worked to refine and document our understanding of the domain, and keep our model as clear and as small as possible. Nothing unnecessary or confusing should sneak through.

Implementing

Anyway, down to the nitty-gritty.

Cargo Workspace

This project is structured as a Cargo workspace, where there are multiple crates in a single repo. The idea behind this was that, in a real-life scenario, we would want to publish separate library crates for Actuators, Sensors, and so on. If you are a third-party developer creating software for (for example) a smart lightbulb, you might not care about the Sensor library. Your device only has an effect on the environment, it doesn't probe it in any way.

Setting up a project as a Cargo workspace is straightforward, and allows you to pull out "common" code into one or more separate crates, which adheres to the DRY principle and just generally makes the whole project easier to develop and maintain.

Dependencies

In the interest of keeping the resulting binaries and containers as small as possible, I steered this project away from the big frameworks (tokio, actix, etc.), opting to "roll our own" solutions wherever we could. Currently, the project has only eight dependencies

mdns-sd for mDNS networking
chrono for UTC timestamps and timestamp de/serialization
rand for random number generation
local-ip-address for local IP address discovery
phf for compile-time static Maps
log the rust-lang official logging framework
env_logger a minimal logging implementation
plotly for graphing data in the Web UI

Even some of these are not strictly necessary. We could

do away with chrono by rolling our own timestamp de/serialization
remove phf by just creating this single static Map at runtime
do away with log and env_logger by reverting to using println!() everywhere

mdns-sd and local-ip-address are critical; they ensure the Devices on the network can connect to one another. rand is critical for the Environment, and appears only in that crate's dependencies. plotly is critical to the Web UI, hosted by the Controller, which (as of this writing) shows just a live plot and nothing else.

Finally, for containerization of services, we used rust:alpine base image in a multi-stage build. Only a single dependency needed to be installed in the initial stage, musl-dev, which is required by the local-ip-address crate.

The final sizes of the four binaries produced (for the Controller, Environment, and one implementation each of the Sensor and Actuator interfaces) ranged from 3.6MB to 4.8MB, an order of magnitude smaller than the JRE, which clocks in around 50-100MB, depending on configuration.

The containers were a bit larger, coming in around 13.5MB to 13.7MB. This is still peanuts compared to container image sizes I'm used to for Scala-based projects -- I find that Scala container images are typically in the 100s of MBs range, so < 15MB is a breath of fresh air.

Service Discovery and Messaging

As this sketch shows, it's actually really straightforward to get two services to discover each other via mDNS with the mdns-sd crate. Once services knew about each other, they could communicate.

The easiest way that I know of for two services on a network to communicate with each other is over HTTP. So in this project

Service A discovers Service B via mDNS, retrieving its ServiceInfo
Service A opens a TcpStream by connecting to Service B using the address extracted from its ServiceInfo
every service (including Service B) opens a TcpListener to its own address, listening for incoming TCP connections
Service A sends a Message to Service B via its TcpStream, Service B receives it on its TcpListener, handles it, and sends a response to Service A, closing the socket

These Messages don't necessarily need to be HTTP-formatted messages, but it makes it easier to interact with them "from the outside" (via curl) if they are.

Similarly, the data points (called Datums in this project) sent via HTTP don't need to be serialized to JSON, but they are, because it makes it easier to interact with that data in a browser, or on the command-line.

Construction of HTTP-formatted messages and de/serialization of JSON was all done by hand in this repo, to avoid bringing in unnecessary dependencies.

TIP: one "gotcha" I encountered in writing the service discovery code was that each service needs its own mDNS ServiceDaemon. In the original demo, a single daemon was instantiated and clone()d, with the clones passed into each service. But then only the Actuator (or only the Sensor) would see, for example, the Environment come online. It would consume the ServiceEvent announcing that device's discovery on the network, and the next service wouldn't be able to see it come online. So, heads-up: create a separate daemon for each service which needs to listen to events.

Common Patterns and Observations

With the basic project structure in place, and with the services able to communicate, I noticed a few patterns reoccurring as the project came to life.

`Arc<Mutex<Everything>>`

In this project, the Devices have some state which is often updated across threads. For instance, the Controller uses one thread to constantly look for new Sensors and Actuators on the network, and adds any it finds to its memory.

To safely update data shared across multiple threads, I found myself wrapping lots of things in Arc<Mutex<...>> boxes, following this example from The Book.

I'd be interested in knowing if there's a better / more ergonomic / more idiomatic way of doing this.

Cloning before `move`-ing into a new thread

Another pattern that appears a few times in this repo is something like

fn my_method(&self) {
    let my_field = self.field.clone();
    std::thread::spawn(move || {
        // do something with my_field
    })
}

We cannot rearrange this to

fn my_method(&self) {
    std::thread::spawn(move || {
        let my_field = self.field; // will not compile
        // do something with my_field
    })
}

because "Self cannot be shared between threads safely" (E0277). Similarly, anything wrapped in an Arc<...> needs to be cloned as well

fn my_other_method(&self) {
    let my_arc = Arc::clone(&self.arc);
    std::thread::spawn(move || {
        // do something with my_arc
    })
}

I've ended up with a few thread::spawn sites with big blocks of cloned data just above them.

There's an RFC for this issue, which has been open since 2018. It looks like it's making some progress lately, but it could be a while before we no longer need to manually clone everything that gets moved into a thread.

It's too easy to `.unwrap()`

This project is not very large -- it's about 5000 lines of Rust code, by my estimate. But in those 5000 lines, .unwrap() appears over 100 times.

When developing something new, it's easier (and more fun) to focus on the "happy path" and leave exception handling for later. Rust makes this pretty easy: assume success, call .unwrap() on your Option or Result, and move on; it's very easy to bypass proper error handling. But it's a pain in the neck to add it in later (imagine adding error handling for all 100+ of those .unwrap() sites).

It would be better, in my opinion, to keep on top of these .unwrap()s as they appear.

Near the end of this MVP, as I counted all of these sites with missing error handling, I found myself longing for a clippy rule which would disallow any .unwrap()...

As it turns out, there already are unwrap_used and expect_used lints which can be used to error out if either of these methods are called. I will definitely be enabling these lints on my personal projects in the future, and I hope that they will eventually become the default.

Parsing

I wrote a lot of custom parsing code.

A common pattern I followed was to impl Display for some type, then add a pub fn parse() method to turn the serialized version back into the appropriate type.

This is probably not the best way to do this -- user-friendly strings for display are different things from compact serialized representations for message passing and persistence. If I were to do this again, I would probably use a crate like serde for de/serialization, and save impl Display for a user-friendly string representation.

In addition, I "rolled my own" routing. When an HTTP request was found on a TcpStream, I would manually check the start_line (something like POST /command HTTP/1.1) to route to the appropriate endpoint. In the future, I might leave this to an external crate... maybe something like hyper.

`pub struct`s should implement `PartialEq` when possible

I think this is probably a good rule of thumb for any pub data type: implement PartialEq when appropriate, so consumers of your crate can test for equality. The ServiceInfo type in mdns-sd does not derive PartialEq. This means I couldn't easily test for equality of two ServiceInfos in tests.

In lieu of this, I checked that every pub method on two instances returned the same values. This was kind of a pain, resulting in big blocks of

assert_eq!(actual.foo(), expected.foo());
assert_eq!(actual.bar(), expected.bar());
assert_eq!(actual.baz(), expected.baz());
// ...

It would have been nice to just write

assert_eq!(actual, expected)

instead.

`trait`s implementing other `trait`s can get messy, fast

In this project, there's a trait Device with an abstract method called get_handler()

// examples in this section are abridged for clarity
pub trait Device {
    fn get_handler(&self) -> Handler;
}

The Sensor and Actuator traits both implement Device, and provide default implementations of get_handler()

pub trait Sensor: Device {
    fn get_handler(&self) -> Handler {
        // some default implementation here for all `Sensor`s
    }
}

pub trait Actuator: Device {
    fn get_handler(&self) -> Handler {
        // some default implementation here for all `Actuator`s
    }
}

But then there are the concrete implementations of Sensor and Actuator

pub struct TemperatureSensor {
  // ...
}

impl Sensor for TemperatureSensor {}

impl Device for TemperatureSensor {
    fn get_handler(&self) -> Handler {
        Sensor::get_handler(self)
    }
}

pub struct TemperatureActuator {
  // ...
}

impl Actuator for TemperatureActuator {}

impl Device for TemperatureActuator {
    fn get_handler(&self) -> Handler {
        Actuator::get_handler(self)
    }
}

There already is a concrete implementation of get_handler() in Sensor / Actuator, so we don't actually need anything in the impl Sensor / impl Actuator blocks (unless there are other abstract methods), but we do need this awkward impl Device in each case.

As far as Device "knows", TemperatureActuator hasn't implemented its abstract method. But we know that Actuator has, and that TemperatureActuator implements Actuator. There seems to be some information missing here that the compiler could fill in, theoretically, but currently isn't.

Rust could use a more robust `.join()` method on slices

Other languages let you specify a start and end parameter when joining an array of strings, so you could easily do something like

["apple", "banana", "cherry"].join("My favourite fruits are: ", ", ", ". How about yours?")
//                                 |--------- start ---------| |sep|  |------- end -------|

which would result in a string like "My favourite fruits are: apple, banana, cherry. How about yours?", but Rust doesn't yet have this functionality. This would be a great little quality-of-life addition to the slice primitive type.

All of my `Result` error types are `String`s

This is certainly the easiest way to quickly build something while mostly ignoring failures, but at some point, I should go back and replace these with proper error types. Clients should be able to match on the type of the error, rather than having to parse the message, to figure out what failed.

Any Result types which leak to the external world (to clients) should probably have proper Err variants, and not just String messages. This is another thing I wish clippy had a lint for: no &str or String Err types.

`S: Into<String>` instead of `&str`

Rust will automatically coerce &Strings to &strs, and so the traditional wisdom is that function arguments should be of type &str, so the user doesn't need to construct a new String to pass to a function which takes a string argument. If you already have a String, you can just call as_ref() on it to get a &str.

But Rust will only do a single implicit coercion at a time. So we can't convert some type T: Into<String> into a String and then into a &str. This is why I opted for S: Into<String> instead of &str arguments in a few places. &str implements Into<String> and so does any type which implements Into<String> (or Display).

It is definitely less performant, since we're copying data on the heap, but also a bit more ergonomic, since we don't need to pass t.to_string().as_ref() (when t: T and T: Into<String>) to the function, but just t itself.

Apparently I'm not the first person to discover this pattern, either: Into<String> returns 176,000 hits on GitHub.

Conclusion

I learned a lot in building this project: about mDNS networking, the nitty-gritty of HTTP message formats, and writing bigger projects in Rust. To summarize the points I raised above...

Things I know I need to do better

I shouldn't be using Display for serialization. In the future, I will look into using a crate like serde instead.
I shouldn't be using String for all of my Err variants. Clients of the library crates I'm producing should be able to handle an error without having to parse a string message. In the future, I will build error enums as soon as I start producing errors.

Things I'm looking forward to from the Rust community

Explicit clone-ing prior to a move closure is a pain. I'm following this GitHub issue in hopes that this becomes more ergonomic in the future.
A clippy error for String / &str Err variants would be nice, as well.
Rust could use a more robust .join() method on string slices, with start and end parameters. As far as I can tell, this issue is not yet being tracked. After this article is published, I hope to open an RFC for this small feature.
I'm hoping that eventually the compiler will be smart enough to know that when B: A and C: B, where A defines some abstract method and B implements that abstract method, that c: C already has that method implemented, without having to explicitly tell the compiler about that implementation. But that might be a ways off.

Things I still have questions about

Is Arc<Mutex<Everything>> really the best way to mutate data across multiple threads? Or is there a more idiomatic (or safer) way of doing this?

Things I would recommend to other Rust developers

Please impl PartialEq on any pub data type published by your crate, whenever possible. Your clients will thank you (hopefully).
Don't be afraid to use S: Into<String> instead of &str. It might be less performant, but it's also more ergonomic, and you're definitely not the first person to do it.
Enable clippy's unwrap_used and expect_used lints, to force yourself to tackle error scenarios head-on, instead of pushing them aside to deal with them later.

If you've made it this far... thanks for reading!

Please direct any feedback you may have about the above article to the email address on my CV. This was a fantastic learning experience and I'm excited to do some more serious Rust development in the near future.

Software Development is About Compromise

Andrew (he/him) — Mon, 10 Apr 2023 13:39:01 +0000

Trade-Offs in Software Development

"Where's all my CPU and memory gone?"

-- thegeomaster

Software development is -- and has always been -- about trade-offs.

The CAP theorem tells us that we need to choose between consistency, availability, and partition tolerance when designing distributed data stores.

Caches can make information available more quickly, but it might be out of date, and it will definitely use more storage space than making a fresh request each time.

And, as always, money can be a factor. SSDs are faster than HDDs, but they are more expensive per GB. And microservices in a distributed system should each have their own database, but cloud computing costs can mean that shared DBs are more cost-effective.

Weighing the pros and cons of varied solutions to a problem was something I needed to tackle recently when redesigning part of my personal website, awwsmm.com. Here's how that went down...

A Case Study: My Website

The Setup

My website is a pretty minimal Next.js site, written in TypeScript, hosted on GitHub, built and deployed by Vercel.

In general, Next.js allows you to have two kinds of pages: static pre-rendered pages, and dynamic pages, rendered "just in time", when a visitor to your website tries to visit that page.

My website has static blog posts, as well as "project" pages, which give a quick overview of personal projects that I've been working on lately.

My initial design was for a given project page (for example, this one for my website itself) to have an up-to-date commit history, interleaved with occasional "log entries", summarizing big sweeping changes made to these projects. (Kind of like release notes, but with more detail behind why certain changes are being made.)

So what was the problem?

The Problem

The problem was that these project pages were statically rendered; they were built in advance.

I would request the commit history of a project from GitHub and write it to a local cache, saved in the repository.

When a new PR is opened against the repo containing my website, Vercel runs a test deployment. When the test deployment looks good, I hit "merge" and the new changes are added to the repo in a merge commit.

But, because the cache is created only when I'm developing locally, this merge commit is not a part of the commit history in the cache. It couldn't possibly be. It would require updating the cache file, which would introduce changes not included in that commit, which would require another commit, ad infinitum.

This means that the commit history for the awwsmm.com project page is always at least one commit behind master.

This bothered me, and I wanted to see if I could fix it.

First Attempt: Environment-Aware Caching

My first attempt at a solution was (what I'm going to call) "environment-aware caching".

What if, after the deployment passed and the new PR was merged into master, Vercel ignored the cache, only using it as a backup? During deployment, we could hit the GitHub API again, which should then have the new commit, right?

This required knowing which environment the build was running on. This is straightforward, as Vercel populates a VERCEL_ENV environment variable to "production", "preview", or "deployment", depending on where the build is running.

But this also required maintaining a cache which we hopefully would never fall back to, which seemed kind of silly.

It also meant that each release would have to be deployed at least twice: once to satisfy the checks before merging the PR, and once after the PR was merged to pick up the new commit on master. I'd have to remember to do this "double deployment" to keep things up-to-date.

Finally, there was some human error here in that I tried to use the same caching mechanism for "last updated" dates on my blog posts, confusing the issue.

All in all, this solution was pretty complex to maintain (all I want is a static blog), and it put me off of keeping my website up-to-date for a while. When I finally came back here after a few months, I decided that a simpler solution was in order.

Second Attempt: Server-Side Rendering

So how about trying to render the project pages "just in time"?

Vercel's server-side rendering (SSR) also generates static pages, but it renders them only when the user navigates to the page, not during deployment.

"This is great!" I thought. I could just request the commit history when the page is requested, and it would always be up-to-date.

Unfortunately, requesting and processing 100 commits from GitHub seemed to be too much to ask. I was consistently waiting about 3 seconds for the page to load, which is really bad. Project pages with shorter commit histories loaded a bit faster, but there was still a noticeable delay. Sending the request to GitHub, awaiting a response, processing the response, generating the resulting page, and displaying it just took too much time.

This approach would also send a request to GitHub every time a user loaded that page.

This second issue could be solved by fine-tuning the Cache-Control header sent along with the request, such that I could guarantee that the 5000-requests-per-hour (authenticated) limit would never be exceeded.

But the first issue remained a problem.

Third Attempt: Non-Blocking Server-Side Rendering

"Maybe I'm the problem" I thought.

"Maybe it's the way I (think I) am blocking inside of getServerSideProps."

I rearranged this method to not await anywhere, but return a Promise which is a result of other Promises chained together with then, and nothing else.

I was hoping that, with everything done in a non-blocking way in the return value, Vercel could work some magic to speed up the call. (Maybe it could run the request as soon as the user hovered over the link to the page?)

But it didn't help. I was still stuck at ~3 seconds of loading time. Worth a try, at least.

Fourth Attempt: GraphQL

"Maybe the response from GitHub is taking so long because it's returning too much data?"

I wasn't using most of the response anyway; all I cared about was the commit hash, the message, and the date. I'd never used GraphQL before, but I knew that it could be used in situations like this, where you wanted to request only particular data from an endpoint.

So I learned enough about GraphQL and GitHub's API to request only the commit data I cared about.

This sped up the page a bit locally but not in production on Vercel. It was still taking about 3 seconds.

Fifth Attempt: Timeout / Fallback to Cache

"Well, if it's loading fast enough locally, but not remotely, maybe I can set a timeout threshold?"

My thought was that I could cache a "stale" version of the page to display if the "live" version of the page took too long to load. Maybe longer than 500ms or so.

So I would now generate a cache when building locally, and save the cache to the repo. In production, I would attempt to request fresh data from GitHub, but if it took longer than 500ms, I would fall back to the cache.

...but it still took 3 seconds to render the page remotely.

"Why?" I thought, pulling my hair out in frustration.

Other project pages loaded quickly... maybe Vercel was just (somwehow?) slow to process this 600-line JSON file, generate the hundreds of components for it, style all of them, and display them.

I found issues on the vercel repo of people complaining about similar problems. Maybe it just took a few seconds for Vercel to spin up a runner to render the page? (That didn't seem to square with the load times of the other project pages.)

I took solace in the fact that other people on the Internet also thought that getServerSideProps is weird and unintuitive.

I assessed other possible avenues for investigation: timing the response?, actually reading the docs? (no thanks), incremental static generation?, edge functions?.

I was beginning to lose the will to live... all I wanted was a blog.

Regrouping

I took a breather and came back to it all after a little while.

"What is it I actually want?"

up-to-date commit histories
fast page loads

The problem was that those two things were -- if not mutually exclusive, at least -- in competition with each other. Up-to-the-second commit histories would require a request to GitHub as soon as the user requested the project page. Which meant that all of the data fetching, and processing, and rendering would have to be done quickly.

This didn't seem like too big of an ask to me, but apparently it was. Maybe it was my code, maybe it was the Lambda cold start times, maybe it was something on Vercel's end... whatever it was, it was pinning me to 3-second page load times.

"Maybe I can request and process most of the history in advance, then only request recent history when a user clicks on a page?"

This sounded overcomplicated, though. (Unlikely, but) what if a project had had 100 new commits since I last deployed my website (and generated the cache)? Then I would have the same problem: trying to request and render 100 commits onto this history page.

So I compromised, and came up with a simpler solution:

generate project/ pages statically

This solves the load time issue -- by the time we get to production, the pages will have already been generated, so nothing needs to be done except displaying them.

But this means that we will not have an up-to-date commit history.

accept that the commit history will not be up-to-date

I'd already sunk so much time into this, I was ready to compromise.

Nobody coming to my website will care if my commit histories are slightly out of date. So I decided to just put a disclaimer at the top and bottom of the histories, saying something like "for the most up-to-date commit history, see GitHub".

Now, we can generate pages in advance, keep load times to basically zero, and we don't have to worry about complex caching solutions, or what environment the build is running in, or GitHub rate limits, or anything.

My compromise was that my time and energy were worth more to me than having this one feature on my website be exactly the way I had envisioned it initially.

Software Development is About Compromise

In my case, I tried and tried to get my website to do what I wanted it to, but my two requirements: up-to-date commit histories and fast loading times, were in direct competition with each other.

Having an up-to-date history will require a request when the user clicks the button, and processing the result of that request, which will take time.

Surely there are more complex solutions which balance these two better, but in the end, having these pages be up-to-date is not critical for my blog. Choosing where to use my time is another trade-off. I learned a lot about Vercel, GraphQL, and Promises in JavaScript / TypeScript during this process, but ultimately, "see GitHub" is good enough for me. And that's a compromise I'm willing to accept.

The AI Assistance Paradox: How ChatGPT Helps, But Can Never Replace Human Ingenuity in Programming

Andrew (he/him) — Sat, 08 Apr 2023 02:03:14 +0000

Programming is a unique and rewarding field that demands a combination of technical skill and human creativity. Over the years, I have encountered numerous challenges that have required me to rely on the ingenuity of the human spirit to find solutions.

At the same time, I've also seen the emergence of AI tools that can assist programmers in their work. For example, chatbots like ChatGPT can help programmers automate repetitive tasks, generate code snippets, and even offer suggestions for optimizing performance. These tools are incredibly powerful and can save programmers a significant amount of time and effort.

However, despite the impressive capabilities of AI tools like ChatGPT, they will never replace programmers. Programming is not simply about generating code; it's about solving complex problems and finding creative solutions. It requires a deep understanding of the needs and desires of the users, as well as the ability to think outside the box and innovate.

AI tools like ChatGPT lack the human ingenuity and creativity that are essential to programming. They may be able to generate code, but they cannot understand the nuances of human behavior or anticipate the needs and desires of users. They cannot think creatively or come up with innovative solutions to complex problems. In short, they lack the human spirit that is so integral to the programming process.

To illustrate this point, let me share another anecdote from my own experience. I once worked on a project for a healthcare company that required me to develop a custom software solution for their patient management system. The system needed to be fast, responsive, and secure, with robust reporting capabilities that would allow doctors and administrators to easily access and analyze patient data.

To accomplish this, I turned to my trusted tools of the trade: Python, Django, and MySQL. I also utilized a number of AI tools, including natural language processing algorithms, machine learning models, and predictive analytics software.

These tools were incredibly useful in helping me automate certain tasks and generate insights from the patient data. However, they were not enough on their own. I had to rely on my own ingenuity and creativity to design a system that met the specific needs of the healthcare company. I had to understand the needs of the doctors and administrators, anticipate their wants and needs, and create a system that was not only functional but also intuitive and easy to use.

In the end, it was the combination of technical skill and human ingenuity that allowed me to create a successful solution for the healthcare company. And while AI tools like ChatGPT may be able to assist programmers in their work, they will never be able to fully replace the human spirit that is so essential to the programming process.

This article was generated by ChatGPT.

Elegant Multi-Line Shell Strings

Andrew (he/him) — Fri, 18 Mar 2022 01:08:56 +0000

Photo by Wendy van Zyl from Pexels

The State of Multi-Line Strings

Multi-line strings in shells are a pain.

Suppose you want to create a file, using a shell script, which contains the following content

export default class Greeter {
  greet(name: string) { return 'Hello, ' + name + '!'; }
}

How can this be achieved?

Some Methods

Method 1: a multi-line variable

This simple solution works when the variable definition is not indented at all

var="export default class Greeter {
  greet(name: string) { return 'Hello, ' + name + '!'; }
}"

$ echo $var
export default class Greeter {
  greet(name: string) { return 'Hello, ' + name + '!'; }
}

But what if we're defining $var within a function, and we want it indented along with the rest of the function body?

function my_function() {
  var="export default class Greeter {
    greet(name: string) { return 'Hello, ' + name + '!'; }
  }"
  echo $var
}

$ my_function
export default class Greeter {
    greet(name: string) { return 'Hello, ' + name + '!'; }
  }

Oh, well, that's obviously not what we want.

So, simple variable assignment: it works in a very limited subset of cases, when the variable definition is not indented at all. Let's try another method.

Method 2: a single-line variable with '`\n`'s for line breaks

In this method, we replace all of the line breaks in the multiline string with \n line break characters:

function my_function() {
  var="export default class Greeter {\n  greet(name: string) { return 'Hello, ' + name + '!'; }\n}"
  echo $var
}

$ my_function
export default class Greeter {
  greet(name: string) { return 'Hello, ' + name + '!'; }
}

The result looks good, but the method is messy. What if we want to reformat this like

export default class Greeter {
  greet(name: string) {
    return 'Hello, ' + name + '!';
  }
}

That would involve adding more \n characters, and spaces to match the indentation. It's not extremely straightforward:

function my_function() {
  var="export default class Greeter {\n  greet(name: string) {\n    return 'Hello, ' + name + '!';\n  }\n}"
  echo $var
}

So, explicit line break characters: this works if the text you want formatted won't change often, and if the readability of the implementation doesn't matter. If you want the text-generating code itself to be readable or maintainable, this is not a great solution.

So what else can we do?

Method 3: Heredocs

Multiline strings are what Heredocs were made for:

In computing, a here document (here-document, here-text, heredoc, hereis, here-string or here-script) is a file literal or input stream literal: it is a section of a source code file that is treated as if it were a separate file. The term is also used for a form of multiline string literals that use similar syntax, preserving line breaks and other whitespace (including indentation) in the text.

So let's see how well they work for our problem

function my_function() {
  var=$(cat <<EOF
  export default class Greeter {
    greet(name: string) { return 'Hello, ' + name + '!'; }
  }
  EOF)
  echo $var
}

$ my_function
/Users/andrew/test.sh:8: parse error near `var=$(cat <<EOF'

Oh, uh, yeah, obviously the delimiter sequence (EOF in this case), cannot be indented, and must appear on a line by itself, so we have to write

function my_function() {
  var=$(cat <<EOF
  export default class Greeter {
    greet(name: string) { return 'Hello, ' + name + '!'; }
  }
EOF
  )
  echo $var
}

...which is fine, but sort of breaks indentation of the rest of the function body. It also doesn't work:

$ my_function
  export default class Greeter {
    greet(name: string) { return 'Hello, ' + name + '!'; }
  }

Just like Method #1, this method adds to the output the whitespace we used to indent the function body, which we don't want.

So how can we preserve only the indentation we want (and maybe get rid of that ugly heredoc delimiter)?

My Method

Here's how I do it

function my_function() {
  var="$(sed -e 's/^[ ]*\| //g' -e '1d;$d' <<'--------------------'
    | 
    | export default class Greeter {
    |   greet(name: string) { return `Hello, ${name}!`; }
    | }
    | 
--------------------
    )"
  echo $var
}

I use pipe characters | to define a "margin", which I then strip out using sed. sed -e 's/^[ ]*\| //g' will remove any number of space characters ([ ]*) at the beginning of the line (^), followed by a pipe (|), followed by one space character ([ ]).

This "margin" method was inspired by Scala's String#stripMargin functionality, which behaves in a very similar way.

The second sed expression, -e '1d;$d', removes the first and last line. I add blank lines to provide a bit of visual whitespace around the content I want to write to the variable. If you don't want one or both of these blank lines, remove them with this slight variation on my method

function my_function() {
  var="$(sed -e 's/^[ ]*\| //g' <<'--------------------'
    | export default class Greeter {
    |   greet(name: string) { return `Hello, ${name}!`; }
    | }
--------------------
    )"
  echo $var
}

I don't mind the line of hyphens, either, as the EOF replacement, because it sort of acts like the top and bottom margin of the content. But, if you put the heredoc delimiter in quotes, as I have above, you can also include whitespace in it. So you could do something like

function my_function() {
  var="$(sed -e 's/^[ ]*\| //g' <<'    +'
    | export default class Greeter {
    |   greet(name: string) { return `Hello, ${name}!`; }
    | }
    +
    )"
  echo $var
}

Though I personally think this leaves a bit too much whitespace under the content. Also, many syntax highlighting algorithms have trouble with this.

With any of these variations, you can indent the content to whatever level you like

function my_function_1() {
  var="$(sed -e 's/^[ ]*\| //g' <<'--------------------'
    | export default class Greeter {
    |   greet(name: string) { return `Hello, ${name}!`; }
    | }
--------------------
    )"
  echo $var
}

function my_function_2() {
  var="$(sed -e 's/^[ ]*\| //g' <<'--------------------'
| export default class Greeter {
|   greet(name: string) { return `Hello, ${name}!`; }
| }
--------------------
    )"
  echo $var
}

function my_function_3() {
  var="$(sed -e 's/^[ ]*\| //g' <<'--------------------'
              | export default class Greeter {
              |   greet(name: string) { return `Hello, ${name}!`; }
              | }
--------------------
    )"
  echo $var
}

$ my_function_1; my_function_2; my_function_3
export default class Greeter {
  greet(name: string) { return `Hello, ${name}!`; }
}
export default class Greeter {
  greet(name: string) { return `Hello, ${name}!`; }
}
export default class Greeter {
  greet(name: string) { return `Hello, ${name}!`; }
}

The flexibility -- combined with the visual aesthetics -- of this method is why it's recently become my go-to for multi-line strings in the shell.

Check out more of my writing at awwsmm.com

What's Wrong This Time? Part III: The Deep End

Andrew (he/him) — Sun, 06 Mar 2022 21:48:25 +0000

Photo by Cristian Palmer on Unsplash

Part III: The Deep End

At this point, I wasn't sure where to go next.

Why should these blog posts now have a seemingly random publication date, which was not the current date, and was not the date they were first added to the repo? What was going on?

So I did what most programmers do in a situation like this: I added some console.log()s.

Digging Deeper

I wanted to know what commits were available, and what information they had. In other words, I wanted better observability into what was going on in this bit of the codebase.

My first idea was to just print out the dates of every line returned from git log. The output of that (for a given blog post) looked something like

lines returned from git log
2022-01-09T20:48:23+00:00
2022-01-06T09:18:41+00:00

...okay, not extremely useful. Why are there only two commits here? This is just the same information we had on the website. Maybe printing out the hash of each commit would be a bit more helpful? Then I could check if there was anything unusual about these commits

lines returned from git log
69e038a919e448251fa2211a9fcf3fda914812fe @ 2022-01-09T20:48:23+00:00
d5cf8fbc05891ac9d8d7067b5cb1fb195dc2cf99 @ 2022-01-06T09:18:41+00:00

Now we can search GitHub for that commit dc2cf99.

But this commit doesn't add or update any blog posts... so why is it being returned from git log <path/to/blogpost>?

What if I git logged a file that has definitely been around since the first commit, like index.tsx. I tried printing out every log line for this file and saw the following on Vercel

lines returned from git log index.tsx
69e038a919e448251fa2211a9fcf3fda914812fe @ 2022-01-09T20:48:23+00:00
88c420835d35a008de808b7cef04980a15b029bc @ 2022-01-09T12:55:49+00:00
0a882cf5062e4c0ac4505ed609ca77f14b35a76a @ 2022-01-08T20:15:44+00:00
d4a9a360c38398cdd41825aa0fe193e8176cb4fd @ 2022-01-07T22:41:52+00:00
3acb76c1f6c6d1b4cdb76939496e251220aa29ea @ 2022-01-06T20:09:17+00:00

It only goes back five commits! The commit history looked the same for other long-lived files as well. Only ever going back to that last commit on January 6.

Running the same code on my local machine gives many more commits, going back all the way to the first commit on January 2.

What gives?

The Shallow End

At this point, I wasn't sure how much more debugging I could do. So I started doing a bit of research.

And I found this issue ("How to unshallow repo?") on the Vercel GitHub repo

"Hello, I need to define a variable at build time that depends on git describe which depends on git history, but it seems the repo in vercel build enviroment is a shallow clone with only few last commits."

That sounds like my problem! And it sounds like it's caused by Vercel making a shallow clone of my git repo before building. I'd never encountered shallow cloning in the wild before, but I knew of it as a concept, which is how I found that GitHub issue.

So how can we work around this? We simply won't have the information available at build time to determine the correct "published" and "last updated" dates for a given blog post.

But there's always a way to work around these kinds of limitations. In this case, that involves a cache.

Cache Rules Everything Around Me

There are a few ways we could solve this problem. We could, for instance, use the GitHub API to pull commit information from the repo hosted on GitHub.com. I chose not to do this as I preferred to keep the solution self-contained: we have all the information available at build time when running locally, so how could we make that information available when building for production (on Vercel), as well? (Where we'll have a shallow clone of the repo.)

Rather than make API calls over the internet for information which is available locally, I thought we could simply save this information in a cache, and then use that cache when building on Vercel.

The workflow I came up with for writing blog posts (and caching the important git info) looked something like this

draft a wip- post (these are ignored for version control by my .gitignore)
when the draft is ready, git commit it to the development branch and push to Vercel
for...
- new blog posts (where the only commit in git log is the current commit), Vercel assumes the post is brand-new and uses the date of the current commit for the "published" and "last updated" times
- old blog posts (where more than one commit references this blog post), Vercel looks for cached "published" and "last updated" times, and throws an error if it doesn't find any

There are a few small problems with this.

First, when do we update the cache? You'll notice that there is no step in the workflow above for ensuring that the cache is up-to-date. Since we only have access to the required information when building locally, we have to update the cache when building locally. But when does this information get pushed to the remote repo? We have to enforce that, as well.

Second, the above workflow has a problem when we merge the development branch into the master branch when promoting a new release to production -- the merge commit itself means that the "new" blog post is now in two commits. As outlined above, this will cause Vercel to throw an error if the post isn't in the cache (it won't be).

So... What Now?

I've got some hacky fixes for the above problems implemented.

For instance, I've got a pre-push git hook which runs a build before each git push. This means that -- in theory -- the cache is always up to date. But of course, I need to make sure to git add it in the next commit.

As for the "merging creates a new commit" issue, I've tried two solutions so far.

The first was to distinguish between commits on the development branch and commits on the master branch. Only blog posts with commits on master should be considered as "old". This works great when running locally, but the clone that Vercel creates seems to rename this development branch to master when building a preview deployment. So that's a no-go.

The second solution (which I'm currently using) is to simply ignore merge commits.

So far, the above appears to be working. But it feels like an overly complex and fragile solution, and I hope to improve upon it in the future. Maybe just querying GitHub for the commit history is easier than going through all of this cache trouble.

Conclusion

So that's it! The goal was simple: get rid of arbitrary "published" times on blog posts and pull that data directly from the project's git history. But the solution ended up being much more complex and nuanced than I had initially planned.

But along the way, I learned some new tools and tricks, I learned a bit more about how my repo is built and deployed on Vercel, and I have some ideas for how I can make things more streamlined in the future. And that's what this is all meant to be, really, a learning experience.

In the future, maybe I'll do away with this overly-complex caching mechanism, but I do want to get the "published" and "last updated" dates from the repo's git history. This initial solution, while messy, does the job for now.

What's Wrong This Time? Part II: Electric Bugaloo

Andrew (he/him) — Sun, 23 Jan 2022 19:26:25 +0000

Part II: The Bugs

There's an old programming joke that goes something like

There are only two hard problems in computer science: cache invalidation, naming things, and off-by-one errors.

I think we should add a third (fourth?) problem to that list: sorting things.

Sorting Things

There are lots of different ways to sort things in computer science. C.S. students learn about time and space complexity of these sorting algorithms, YouTubers make cool visualisations of them, and occasionally, a guy named Tim will invent a new one.

But there's one aspect of sorting algorithms that -- for me, at least -- seems completely impossible: remembering in which direction things are sorted.

If you say to a group of people: "okay, everyone, stand in a single-file line, ordered by height", the next question you might ask is "okay, but in which direction?" Who should stand at the front of the line? The shortest person or the tallest person?

In programming, we define comparison functions, which describe how to order whatever objects we're interested in.

Some comparison functions seem obvious. For example, in TypeScript, using the default string comparison...

const array: string[] = ["cherry", "apple", "banana"]
array.sort()
//...

...we would expect array to be sorted alphabetically, with apple as the first (0th) element of the sorted array

//...
console.log(array)    // [ 'apple', 'banana', 'cherry' ]
console.log(array[0]) // apple

Note that <array>.sort() in JavaScript sorts the array "in place", so that the original, unsorted array no longer exists afterward. In some languages, and for some sorting algorithms, arrays are not sorted in place, and a new array will be returned. This new array should be assigned to a new variable.

But often we will be working with objects more complex than strings, and we will need to define custom comparison functions. These are functions which take two elements of type T and return a number, and are used to sort arrays of type T:

type T = string

const newArray: T[] = ["cherry", "apple", "banana"]

function comparison(t1: T, t2: T): number {
  return t1.charCodeAt(0) - t2.charCodeAt(0)
}

newArray.sort(comparison)

console.log(newArray)    // ?
console.log(newArray[0]) // ?

Without reading the docs, will the console.log()s above give the same result as the earlier ones? How about something a bit simpler -- sorting an array of numbers:

type T = number

const newArray: T[] = [42, 2112, 19]

function comparison(t1: T, t2: T): number {
  return t2 - t1
}

newArray.sort(comparison)

console.log(newArray)    // ?
console.log(newArray[0]) // ?

Will the first element above be 19? Or 2112? Are you sure?

I understand the utility of sorting algorithms, and I understand the need for a ternary (greater than, less than, or equal) return value, and hence number as the return type instead of boolean, but comparison functions are just one of those things that I've always had to test every time. Sometimes in development, and sometimes in production.

So What Happened?

With what we learned above, you should now be able to see what went wrong with my initial code. The problem was here

    // get the blog post date from its git commit date
    const gitLog = SlugFactory.git.log({ file: `blog/${slug.params.slug}.md` });

    return gitLog.then(lines => {
      const dates = lines.all.map(each => each.date);

      // if blog post hasn't been committed yet, use current date
      const date = dates[0] ?? new Date().toISOString();

      return new FrontMatter(slug.params.slug, title, description, date, rawContent);
    });

git log returns commits sorted by date, such that newer commits come first and later commits come afterward. So dates[0], above, is the newest commit returned from git log, and each blog post was being given a "publication" date of the most recent commit in which that post was modified.

When were these blog posts most recently modified? Well, all of them were modified in that same commit, because the point of the commit was to remove the date parameter from the front matter. Essentially, I was mixing up the lastUpdated date and the published date. One of these is the first element in the list (dates[0]) and one of them is the last element in the list (dates[dates.length-1]).

So like I said, there are four hard problems in computer science.

On To The Next One

With that fixed, we're off to the races, right?

Oh... well, that's not right.

Those two posts were both committed on January 2 (Hello, World! and Git Hooks), not on January 6. So why did they both have the wrong date?

That's right, it's another bug... Or is it?

Find out in the thrilling final installation of this debugging mystery!

DEV Community: Andrew (he/him)

There is No Such Thing as a Unit Test

What is a Unit Test?

Kinds of Tests

Fast and Slow Tests

Black-Box and White-Box Tests

Development-Informed Tests and Development-Informing Tests

"Most Unit Testing is Waste"

Most Unit Tests are Development-Informed

Most Unit Tests do not test "Externally-Visible System Behaviours"

A New Test Pyramid

The Base

The Middle

The Top

Conclusion

Code as Art

Digital Art

Readability

Unreadability

Esoteric Languages

Code as Art

Drop Everything and Review

The Inner and Outer Development Loops

The Middle Loop

Minimize the Middle Loop, Dear

What Are Const Generics and How Are They Used in Rust?

What Are Const Generics?

Using Const Generics to Enforce Correctness

Using Const Generics to Conditionally Implement traits

Using Const Generics to Avoid Complex Return Types

Make Invalid States Unrepresentable

age as a String

age as an Int

age as an Int with constraints

at runtime

at compile time

age as an Age with constraints

Further refinements

Start building browser games with Rust!

What This Senior Developer Learned From His First Big Rust Project

The Project

Planning

Implementing

Cargo Workspace

Dependencies

Service Discovery and Messaging

Common Patterns and Observations

Arc<Mutex<Everything>>

Cloning before move-ing into a new thread

It's too easy to .unwrap()

Parsing

pub structs should implement PartialEq when possible

traits implementing other traits can get messy, fast

Rust could use a more robust .join() method on slices

All of my Result error types are Strings

S: Into<String> instead of &str

Conclusion

Software Development is About Compromise

Trade-Offs in Software Development

Software development is -- and has always been -- about trade-offs.

A Case Study: My Website

The Setup

The Problem

First Attempt: Environment-Aware Caching

Second Attempt: Server-Side Rendering

Third Attempt: Non-Blocking Server-Side Rendering

Fourth Attempt: GraphQL

Fifth Attempt: Timeout / Fallback to Cache

Regrouping

Software Development is About Compromise

The AI Assistance Paradox: How ChatGPT Helps, But Can Never Replace Human Ingenuity in Programming

Elegant Multi-Line Shell Strings

The State of Multi-Line Strings

Some Methods

Method 1: a multi-line variable

Method 2: a single-line variable with '\n's for line breaks

Method 3: Heredocs

My Method

What's Wrong This Time? Part III: The Deep End

Digging Deeper

Using Const Generics to Conditionally Implement `trait`s

`age` as a `String`

`age` as an `Int`

`age` as an `Int` with constraints

`age` as an `Age` with constraints

`Arc<Mutex<Everything>>`

Cloning before `move`-ing into a new thread

It's too easy to `.unwrap()`

`pub struct`s should implement `PartialEq` when possible

`trait`s implementing other `trait`s can get messy, fast

Rust could use a more robust `.join()` method on slices

All of my `Result` error types are `String`s

`S: Into<String>` instead of `&str`

Method 2: a single-line variable with '`\n`'s for line breaks