Automated Acceptance Criteria

A well written user story helps put me into a thoughtful frame of mind.

I have a dream (Story Card)

What is my dream story card? I don't mean: What's the story I'd most like to work on! I mean: What should a virtual story card look like (as opposed to card stock on a wall)? This may be a trivial question. But for me the user experience working with stories is very important: I work with them daily, write them, discuss them, work on them, accept them, etc. I want the feel of the card to be thoughtful, like that fellow to the right.

And more than that. I am lazy, impatient, hubristic. The acceptance criteria, I want them testable, literally testable in that each has a matching test I can execute. Given my laziness, I don't want to switch systems and run tests; I'd like to execute acceptance criteria directly from the story card.

So is there a system like that today? No. There are bits and pieces though.

Difficult to use systems, waiting on tests, ugly interfaces, counter-intuitive controls: these are a few of my least favorite things.

Not all story card systems are equal

Some story card systems are particularly awkward to read, understand or use. Special demerits for:

• A hard-coded workflow that the team cannot be change to fit how they work: the team is expected to fit the tool
• Workflow state transition buttons are nice, but not so nice is unconfigurable labels, especially when the button labels are misleading
• A hard-coded or limited hierarchy of stories, so if a team uses epics or features or themes or whatever to organize stories, and there are more than one level to this, the team is out of luck
• Lack of quality RESTful support, in particular, no simple identifier for story cards, so linking directly to cards is opaque, useless or completely absent

A scenario

Post-development testing on this team is a fairly ordinary role. The developers say: a new web page is ready. Testers then validate the same page features each time for each new page, simple things:

• Can an account with security role X log in?
• Can X submit the form? (Or not submit if forbidden?)
• What form defaults appear for role X? Do they reflect role X?

(Yes, I know — what about developer testing? Bear with me.)

On it goes, the same work each time. Redundant, repetitive, error-prone, fiddly. And worst of all—boring, BORING! This is traditional, manual "user testing" at its worst.

The QA is the Advocate for Value. Strive to make them awesome!

What's to be done? Can we fix this?

After all, the testing is valuable: nobody wants broken web pages; everybody wants to log in. But the tester is valuable, too, more valuable even: is this the most valuable testing a human could do? Surely people are more clever, more insightful than this. And what about all the other page features not tested because of time spent on the basics?

Well, people are more clever than this.

A great topic for Retrospective, as the team finds means to overcoming whatever slows them down.

Clearing the path

What guides testing? If you're using nearly any form of modern user story writing, this includes something like "Acceptance Criteria". These are the gates that permit story development to be called successful: the testers are gatekeepers in the manual testing world. In the manual world these criteria might be congregated into a single "Requirements Document" or similar (think: big, upfront design).

We can do better! Gatekeeper-style testing assumes a linear path from requirements to implementation to testing, just as waterfall considers these activities as distinct phases. But we know agile approaches do better than waterfall in most cases. Why should we build our teams to mirror waterfall? Of course the answer is to structure teams to look agile, just as the team itself practices agile values.

So how do we make Acceptance Criteria more agile?

Enter the Three Amigos

In current agile practice, a story card is not a ready to play until approved by the The Three Amigos: BA, Dev, QA. Each plays their part, brings their perspective, contributes to meeting team-agreed "Definition of Ready".

BA, Dev, QA: Who is which? Let agilists try out any role, or all the roles.

A key component of playable cards are the Acceptance Criteria — answering the question, "What does success look like?" when a story is finished.

The perspectives include:

• BA: Is the story told right? — What is the way to describe the work?
• Dev: Is the story the right size? — What is the complexity of the work?
• QA: Is it the right story to tell? — What is the value of the work?

What are Acceptance Criteria?

But where does this simple testing come from? Any software delivery process beyond "winging it" has some requirements

Well-written agile stories have Acceptance Criteria. What are these? An Acceptance Criteria (AC) is a statement in a story that a tester (QA) can use to validate that all or a portion of the story is complete. The formulaic phrasing for ACs I like best is:

GIVEN some state of the world
WHEN some change happens
THEN some observable outcome results

Generally for web applications this means when I do something with a web page in the application, then the page changes in some particular way or submits a request and the response page has some particular quality or property (or the negative case that is does not have that quality or property).

In some cases it is even simpler: just check that a particular web address fully loads, for example, when testing login access to pages.

So what's the question?

Wherever possible we want to automate. If something can be done for us by a computing machine, we don't want to spend human time on it. Humans can move on to more interesting, valuable work when existing tasks can be automated.

Consider the Test Pyramid (image right): automating lower-value tests focuses people on higher-value ones. You get more human attention and insight on the kinds of tests which best improve the value of software. You win more.

The Story

This is a sample story card with a simplistic implementation of AACs. Live buttons call back into the story system, to be mapped to calls in the testing system. (Another implementation might have the buttons directly call to the testing system, avoiding an extra call into the story system but showing details in the page source about the testing system.)

Title

Narrative

AS A AAC author
I WANT a mock executable story
SO THAT others can see the value

Details

No actual criteria were validated in the execution of these tests. This is only a mock.

Acceptance criteria

Summary: 1 missing, 1 untested, 1 running, 1 passed, 1 failed, 1 errored, 1 disabled

GIVEN magical thinking
WHEN in Missingland
THEN there's no test

No test (yet) - create one!

GIVEN magical thinking
WHEN in Newland
THEN nothing has happened yet

Test never run - be the first!

GIVEN magical thinking
WHEN in Fastland
THEN tests run quickly

15% done (3s)

GIVEN magical thinking
WHEN in Happyland
THEN Unicorns

GIVEN magical thinking
WHEN in Sadland
THEN there be Dragons

Expected: Dragons, got: Puppies

GIVEN magical thinking
WHEN in Crazyland
THEN nothing works right

Test timed out after 90 seconds

GIVEN magical thinking
WHEN in Slowland
THEN tests are disabled

@dev1 @qa2: BLOCKED on widget spanner

Acceptance Criteria states

Every AC potentially has a message from the testing system giving more detail on state. These are noted below.

Missing

This AC has no matching test in the testing system. Use the Create button to create a new test. This does not run the test.

The message is boilerplate to remind users to create tests.

Untested

The AC has a matching test in the testing system, but the test has never been run. Use the Test button to run the test.

Typically there is no message for this state.

Running

The matching test for the AC is running in the testing system. Use the Cancel button to stop the test, or wait for it to complete.

The message, is supported by the testing system, should give a notion of progress. See REST and long-running jobs for how to do this.

Passed

The matching test for the AC passed last time it ran. Use the Test button to run the test again.

Typically there is no message for this state.

Failed

The matching test for the AC failed last time it ran. Use the Test button to run the test again.

The message must give a notion of why the test failed.

Errored

The matching test for the AC errored last time it ran. Use the Test button to run the test again.

The message must give a notion of why the test errored.

Disabled

The matching test for the AC is disabled in the testing system. Update the testing system to reenable.

The message, if supported, should give a reason the test is disabled when available.

Potential problems

Nothing is free. Potential problems with AACs include:

Integrations

There are no existing integrations along these lines. You need to build your own against JIRA, Mingle, FitNesse, Cucumber, etc. Whether the story system drives the interaction, or the test system does, may depend on the exact combination. Best would be if both systems can call the other.

Scaling

As more AACs run, the complete suite takes longer. For example, adding 1 minute of AAC/story, and 5 stories/iteration, in a 12 iteration project takes 60 minutes to run. This is not specific to AACs but a general problem with acceptance tests. It's still much cheaper than the manual steps for each test, but prohibitive for a developer to run the whole suite locally.

Best practice is for the 3 amigos to run only the tests specific to a story before calling that story Ready to Accept.

Update

Until I published this post on Blogger, I really wasn't certain how it would look on that platform. I manually tested the visuals from Chrome with the local post, and it looked good. After seeing it in Blogger, however, the "aside" sections are laid out poorly, overlapping the text. I won't relay the page: mistakes are the best way to improve, and a subtext of this post is Experiment & Learn Rapidly. Public experiements are the most faithful kind: no opportunity to fudge and pretend all was well on the first try.

Sproingk lives!

Sproingk lives!

After months of instability, I again have a fully working pipeline and a live web page for Sproingk, my demo project combining the latest public betas in:

• Kotlin (1.2.10, soon 1.2.20)
• Spring Boot (2.0.0 M7, soon RC1)
• Springfox (2.7.0)
• Hygienic build tooling (ktlint 0.14.0, jacoco 0.7.10 SNAPSHOT)
• Circle CI (2.0)
• Boxfuse (1.30.*.*, updates frequently)

If you're interested in any combination of these, please take a gander.

About some of the items

Kotlin (It's not just for Android!) is really where JVM programming is headed. And it's fun.

Springfox gives you a lovely UI for your REST API. (In the demo, try the "greeting controller".)

Boxfuse make minimal shrink-wrapped Linux images of your software, and handles blue/green AWS deployments.

A language stack for 2018

(I talk about myself in the post more than usual. I'm expressing opinions more than usual, rather than observations and advice. Caveat lector. Also, this post is link-rich.)

The stack

After reading Eric S. Raymond (ESR)'s posts on the post-"C" world, I realized that I, too, live in that world. What would my ideal language stack look like for 2018?

For context, here are ESR's posts I have in mind:

1. The long goodbye to C
2. The big break in computer languages
3. Language engineering for great justice
4. C, Python, Go, and the Generalized Greenspun Law

Read them? Good. So this is the language stack I have in mind:

• Python — By default
• Kotlin (JVM) — When you need it
• Go — When you must

Each of these languages hits a sweet spot, and displaces an earlier language which was itself a sweet spot of its time:

• Bash and Perl → Python
• Java → Python and Kotlin
• "C" and C++ → Kotlin and Go

An interesting general trend here: Not just replace a language with a more modern equivalent, but also move programming further away from the hardware. As ESR points out, Moore's law and improving language engineering have raised the bar.

(ESR's thinking has evolved over time, a sign of someone who has given deep and sustained thought to the subject.)

About my experience with these languages

Python

I have moderate experience in Python spread out since the mid-90s. At that time, I was undecided between Python, Ruby and Perl. Over my career I worked heavily in Perl (it paid well, then), some in Ruby (mostly at ThoughtWorks), and gravitated strongly to Python, especially after using it at Macquarie commodities trading where it was central to their business.

Kotlin

I've been a Kotlin fan since it was announced. It scratches itches that Java persistently gives, and JetBrains is far more pleasant a "benevolent dictator" than Oracle: JetBrains continually sought input and feedback from the community in designing the language, for example. Java has been my primary language since the late 90s, bread and butter in most projects. If JetBrains keeps Kotlin in its current directions, I expect it to displace Java, and deservedly so.

Go

This is where I am relying on the advice of others more than personal experience. I started programming with "C" and LISP (Emacs), and quickly became an expert (things being relative) in C++. Among other C++ projects, I implemented for INSO (Microsoft Word multilingual spell checker) a then new specification for UNICODE/CJKV support in C++ (wstring and friends). I still love "C" but could do without C++. Go seems to be the right way to head, especially with garbage collection.

With Ken Thompson and Rob Pike behind it, intelligent luminaries like ESR pitching for it, and colleagues at ThoughtWorks excited to start new Go projects, it's high time I make up this gap.

What makes a "modern" language?

I'm looking for specific things in a "modern" language, chiefly:

Good community

Varying and strong:

• Python
• Kotlin
• Go

You can explore the figures at TIOBE and StackOverflow.

Kotlin is the interesting case. Though low in the rankings, because of 100% interoperability running on the JVM, it's easy to call Java from Kotlin and call Kotlin from Java, so the whole Java ecosystem is available to Kotlin natively.

Further, Google fully supports Kotlin on Android as a first-class language, so there is a wealth of interoperability there. (The story for iOS is more nuanced, with Kotlin/Native including that platform as a target but in progress.)

Lastly, Kotlin/Native is bringing similar interoperability between Kotlin and Go.

(This is related to having a rich ecosystem.)

Garbage collection

(A quick primer (2011) on garbage collection.)

Among Go's advantages over "C" and C++ is solid garbage collection out of the box, though Boehm a valiant effort. It's only been since 1959. No modern programmer—short of special cases—should manually manage memory.

Kotlin gets a head start here. It's built on the JVM (when targeting that environment), which has arguably the world's greatest GC (or at least most tested). It definitely gives you a choice of garbage collectors. There's too much to say about GC on the JVM for this post except that it is first-rate.

Python has garbage collection. Though not as strong as the JVM, it continues to improve. It is unusual for GC to become a limiting factor in a Python program; you will know if it does.

(If you are in "C" or C++, and want some of the benefits of GC, do consider the Boehm-Demers -Weiser garbage collector. No, you won't get the fullest benefits of a language built for GC, but you'll get enough to help a lot, even if just for leak detection.)

Static type inference

Kotlin really demonstrates where Java could improve by leaps rather than baby steps: automatic type inference. At least Java is heading in the right direction. Don't tell the computer how to run your program when it can figure this our for itself! Keep your focus on what the program is for.

Interestingly, Kotlin and Go have strong typing out of the box, but not Python. Python has built-in support for typing, and an excellent optional implementation, mypy, however this is type declaration not type inference, so it loses a bit there. And none of the common alternatives (Ruby, Perl, PHP, etc.) have type inference either. I'll need to check again in a few years, and possibly update this choice.

(Paul Chiusana writes on the value of static type checking.)

Rich ecosystem

Of the languages, Kotlin is a standout here. Because it is a JVM language, it is 100% compatible with Java, and can fully use the rich world of Java libraries and frameworks. The main weakness for Kotlin is lack of tooling: because more advanced tools may make assumptions about bytecode, Kotlin's particular choices of emitted bytecode sometimes confuse them.

(JetBrains has surveyed on the state of ecosystems for programming languages, related to having a good community.)

Close behind is Python, "batteries included" and all, and it has a better organized and documented standard library than Java. For some problem domains, Python has a richer ecosystem, for example, SciPy and NumPy is the best math environment available in any language. (Some specialty languages like MATLAB deserve mention—an early employer of mine.) I may need to reconsider my ranking Kotlin over Python here.

Go is, frankly, too new to have developed an equivalent ecosystem, and full-blown package management is still a work in progress.

Concision and convenience

A common thread in recent language development is lower ceremony: fewer punctuation marks and boilerplate; make the machine do more work, you do less. Kotlin provides the most obvious example compared to Java. Go is know for cleanness and brevity. And Python ranks high here as well.

(Donnie Berkholz writes an interesting post on ranking language expressiveness.)

Code samples

The classic "Hello, World!" showing three things:

• Writing a "main" callable from the command line
• Using standard output to print to console
• String formatting to build the output message

This doesn't, of course, give a sense of how these languages in their full spectrum, but does give a first taste.

Java

Java in a file named MyStuff.java:

package my.stuff;

public final class MyStuff {
    public static final String LANGUAGE = "Java";

    public static void main(final String... args) {
        System.out.println(String.format("Hello, World, from %s", language));
    }
}

Kotlin

Kotlin in a file named my-program.kt:

package my.stuff

const val LANGUAGE = "Kotlin"

fun main(args: Array<String>) = println("Hello, World, from $LANGUAGE")

Go

But also compare Go to C++:

package main

import "fmt"

const Language = "Go"

func main() {
    fmt.Println("Hello, World, from", Language)
}

C++

And C++:

#include <iostream>

int
main()
{
  std::cout << "Hello, World!" << std::endl;

  return 0;
}

Python

And for completeness, Python compared to Perl and BASH:

#!/usr/bin/python

language = 'Python'

print('Hello, World, from {}'.format(language))

Perl

Any Perl:

#!/usr/bin/perl

use strict;
use warnings;

my $language = "Perl";

printf("Hello, World, from %s\n", $language);

BASH

Unfairly simple:

#!/bin/bash

language=BASH

echo "Hello World, from $language"

See also

• Mixing Kotlin and Go

Update

As usual, I never catch as many problems in my writing as I do after reading it posted publically. Many small edits, and an added, explicit mention of wstring.

Footnotes

1. A surprising take on Python versus JavaScript from Michael Bolin. And I do not feel Kotlin (JS) is ready yet for front-end work.
2. In contrast to ESR's thoughtful posts, a nice Steve Yegge rant in favor of Kotlin. (I enjoy both their writing styles.)
3. YMMV — I use Perl as a example, but it could be Ruby or PHP or similar. And some might prefer Node.js to Python (but don't: see footnote 1. The exact choice is a matter of preference: I prefer Python, and some would keep Ruby (for example).
4. Mike Vanier wrote a similar list for "scalable computer programming languages" in 2001, at least for the technical elements.
5. Mike Hearn points out potential pitfalls with Go GC.