Color your world

My coworkers use many ad hoc or single-purpose scripts, things like: checking system status, wrappers for build systems, launching services locally, etc. My UNIX background tells me, "keep it simple, avoid output; Silence is Golden."

Somehow my younger colleagues aren't impressed.

So to avoid acting my age I started sprinkling color into my scripts, and it worked. Feedback was uniformly positive. And true to my UNIX roots, I provided command line flags to disable color.

Some lessons for budding BASHers:

1. Yes, experiment and learn, but be sure to do your research. The Internet has outstanding help for BASH.
2. Learn standard idioms (see below).
3. Don't overdo it. Color for summary lines and warnings have more impact when the rest of the text is plain.
4. Keep functions small, just as you would in other languages. BASH is a programming language with a command line, so keep your good habits when writing shell.
5. Collaborate, pair! This comes naturally to my fellows. Coding is more enjoyable, goes faster and has fewer bugs.

Idioms

Most of these idioms appear in testing with bash, a simple BASH BDD test framework I wrote for demonstration.

Process command-line flags

this=false
that=true
while getopts :htT-: opt
do
    [[ - == $opt ]] && opt="${OPTARG%%=*}" OPTARG="${OPTARG#*=}"
    case $opt in
    h | help ) print_help ; exit 0 ;;
    t | this ) this=true ;;
    T | no-that ) that=false ;;
    esac
done
shift $((OPTIND - 1))

Keep boolean toggles simple

run_faster=true

if $run_faster
then
    use_faster_algorithm "$@"
else
    use_more_correct_algorithm "$@"
fi

Simple coloring

pgreen="\e[32m"
pred="\e[31m"
preset="\e0m"

if $success
then
    echo -e "${pgreen}PASS${preset} $test_name"
else
    echo -e "${pred}FAIL${preset} $test_name - $failure_reason"
EOM
fi

Consistent exit codes

function check_it {
    local -r failed=$1
    local -r syntax_error=$2
    if $syntax_error
    then
        return 2
    elif $failed
    then
        return 1
    else
        return 0
    fi
}

Coda

There are many more idioms to learn, hopefully this taste catches your interest. I was careful to include others mixed in with these (what does local -r do?) to whet the appetite for research. Go try the BASH debugger sometime.

UPDATE: Fixed thinko. ANSI escape codes need to be handled by echo -e or printf, not sent directly via cat!

Do not test Java getters and setters

An excellent project from Osman Shoukry to automate testing of Java getters and setters—that is when you have getters and setters to test. There's the rub: do not write getters or setters.

For starters they violate encapsulation, exposing your objects innards to others. Ok, but there are frameworks which require them, even in 2016. What to do?

Generate them:

@Getter
@Setter
@RequiredArgsConstructor
public final class SampleBean {
    private final String left;
    private String right;
}

With what result?

public final class SampleBean {
    private final String left;
    private String right;

    @java.lang.SuppressWarnings("all")
    @javax.annotation.Generated("lombok")
    public String getLeft() {
        return this.left;
    }

    @java.lang.SuppressWarnings("all")
    @javax.annotation.Generated("lombok")
    public String getRight() {
        return this.right;
    }

    @java.lang.SuppressWarnings("all")
    @javax.annotation.Generated("lombok")
    public void setRight(final String right) {
        this.right = right;
    }

    @java.beans.ConstructorProperties({"left"})
    @java.lang.SuppressWarnings("all")
    @javax.annotation.Generated("lombok")
    public SampleBean(final String left) {
        this.left = left;
    }
}

This code need never be tested. Test the generator, not the generated code. If the generator is correct, so is the generated code. In this case, Lombok is heavily tested.

As an alternative to the constructor taking field values, Jesse Wilson has interesting advice.

BDD-style fluent testing in BASH

I wanted to impress on my colleagues that BASH was still hip, still relevant. And that I wasn't an old hacker. So I wrote a small BDD test framework for BASH.

Techniques

Fluent coding relies on several BASH features:

• Variable expansion happens before executing commands
• A shell function is indistinguishable from a program: they are called the same way
• Local function variables are dynamically scoped but only within a function, so are visible to other functions called within that scope, directly or indirectly through further function calls

Together with Builder pattern, it's easy to write given/when/then tests. (Builder pattern here solves the problem not of telescoping constructors, but massive, arbitrary argument lists.)

So when you run:

function c {
    echo "$message"
}

function b {
    "$@"
}

function a {
    local message="$1"
    shift
    "$@"
}

a "Bob's your uncle" b c

You see the output:

Bob's your uncle

How does this work?

First BASH expands variables. In function a this means that after the first argument is remembered and removed from the argument list, "$@" expands to b c. Then b c is executed.

Then BASH calls the function b with argument "c". Similarly b expands "$@" to c and calls it.

Finally as $message is visible in functions called by a, c prints the first argument passed to a (as it was remebered in the variable $message), or "Bob's your uncle" in this example.

Running the snippet with xtrace makes this clear (assuming the snippet is saved in a file named example):

bash -x example
+ a 'Bob'\''s your uncle' b c
+ local 'message=Bob'\''s your uncle'
+ shift
+ b c
+ c
+ echo 'Bob'\''s your uncle'
Bob's your uncle

So the test functions for given_jar, when_run and then_expect (along with other, similar functions) work the same way. Keep this in mind.

Application

So how does this buy me fluent BDD?

Given these functions:

function then_expect {
    local expectation="$1"
    shift

    if [[ some_test "$pre_condition" "$condition" "$expectation" ]]
    then
        echo "PASS: $scenario"
        return 0
    else
        echo "FAIL: $scenario"
        return 1
    fi
}

function when {
    local condition="$1"
    shift
    "$@"
}

function given {
    local pre_condition="$1"
    shift
    "$@"
}

function scenario {
    local scenario="$1"
    shift
    "$@"
}

When you write:

scenario "Some test case" \
    given "Something always true" \
    when "Something you want to test" \
    then_expect "Some outcome"

Then it executes:

some_test "Something always true" "Something you want to test" "Some outcome"

And prints one of:

PASS Some test case
FAIL Some test case

A real example at https://github.com/binkley/shell/tree/master/testing-bash.

Bash long options

UPDATED: Long options with arguments in the "name=value" style. The original post neglected this important case.

For years I've never know quite the right way to handle long options in Bash without significant, ugly coding. The usual sources (Advanced Bash-Scripting Guide, The Bash Hackers Wiki, others) are not much help. An occasional glimpse appears on StackOverflow, but not well explained or voted.

Solution

Working with a colleague yesterday, we found this:

name=Bob
while getopts :hn:-: opt
do
    [[ - == $opt ]] && opt=${OPTARG%%=*} OPTARG=${OPTARG#*=}
    case $opt in
    h | help ) print_help ; exit 0 ;;
    n | name ) name=$OPTARG ;;
    * ) print_usage >&2 ; exit 2 ;;
    esac
done
shift $((OPTIND - 1))
echo "$0: $name"

$ ./try-me -h
Usage: ./try-me [-h|--help][-n|--name <name>]
$ ./try-me --help
Usage: ./try-me [-h|--help][-n|--name <name>]
$ ./try-me -n Fred
./try-me: Fred
$ ./try-me --name=Fred
./try-me: Fred

Magic!

I checked with bash 3.2 and 4.3. At least for these, the '-' option argument has a bit of magic when it takes an argument. When the argument to '-' starts with a dash, as in --help (here "-help" is the argument to the '-' option), getopts drops the argument's leading '-', and OPTARG is just the text ("help" in this example). Only '-' has this magic.

Add a quick check for '-' at the top of the while-loop, and the case-block is simple and clear.

Bob's your uncle.

UPDATE: Followup on Bash long options.

Hand-rolling builders in Java

I showed off a hand-rolled example Java builder pattern today. It has some benefits over existing builder solutions, but is more work than I like:

1. All parts of the builder are immutable; you can pass a partially built object for another object to complete (I'm looking at you, Lombok)
2. It's syntactically complete; that is, code won't compile without providing all arguments for the thing to build
3. It's easy to generalize; in fact, I'm thinking about an annotation processor to generate it for you (but not there quite yet)

@EqualsAndHashCode
@ToString
public final class CartesianPoint {
    public final int x;
    public final int y;

    public static Builder builder() {
        return new Builder();
    }

    private CartesianPoint(final int x, final int y) {
        this.x = x;
        this.y = y;
    }

    public static final class Builder {
        public WithX x(final int x) {
            return new WithX(x);
        }

        @RequiredArgsConstructor
        public static final class WithX {
            private final int x;

            public WithY y(final int y) {
                return new WithY(y);
            }

            @RequiredArgsConstructor
            public final class WithY {
                private final int y;

                public CartesianPoint build() {
                    return new CartesianPoint(x, y);
                }
            }
        }
    }
}

That was a lot to say! Which is why most times you don't hand-roll builders. Usage is obvious:

@Test
public void shouldBuild() {
    final CartesianPoint point = CartesianPoint.builder().
            x(1).
            y(2).
            build();
    assertThat(point.x).isEqualTo(1)
    assertThat(point.y).isEqualTo(2);
}

Adding caching for equivalent values is not hard:

public static final class Builder {
    private static final ConcurrentMap
            cache = new ConcurrentHashMap<>();

    public WithX x(final int x) {
        return new WithX(x);
    }

    @RequiredArgsConstructor
    public static final class WithX {
        private final int x;

        public WithY y(final int y) {
            return new WithY(y);
        }

        @RequiredArgsConstructor
        public final class WithY {
            private final int y;

            public CartesianPoint build() {
                final CartesianPoint point = new CartesianPoint(x, y);
                final CartesianPoint cached = cache.
                        putIfAbsent(point, point);
                return null == cached ? point : cached;
            }
        }
    }
}

Maven testing module

My usual practice is to put test dependencies in my Maven parent POM when working on a multi-module project. And I usually have a "testing" module as well for shared test resources such as a logback-test.xml to quiet down test output.

The test dependencies look like clutter in my parent POM, and they are, or so I recently realized.

As all my non-test modules use the "testing" module as a test dependency, I clean this up by moving my test dependencies out of the parent POM and into the "testing" module alongside the common resources. So my layout looks like:

Parent POM

Common properties such as dependency versions, dependencies management marks the "testing" module as "test" scope.

Testing POM

Test dependencies not marked as "test" scope—consumers of this module will mark it as "test", and its transitive dependencies will automatically be "test" as well.

Non-test POMs

Use the "testing" module as a dependency—specified in the parent POM dependencies management—, no test dependencies or resources inherited from parent POM.

Examples:

• Parent POM
• Testing POM
• Non-test POM

Java 8 shim method references

So I'm working on Spring Boot autoconfiguration for Axon Framework. I run into a nice interface in Axon framework that is unfortunately too specific. So I generalize. The original, pared down:

public interface AuditDataProvider {
    Map<String, Object> provideAuditDataFor(CommandMessage<?> command);
}

Aha! A SAM interface, interesting. So I craft my look-a-like:

public interface AuditDataProvider {
    Map<String, Object> provideAuditDataFor(Message<?> command);
}

Not much difference. Note the method parameter is Message rather than CommandMessage. This works fine as the implementation I have in mind uses getMetaData(), defined in Message and inherited by CommandMessage—so the original Axon interface is overspecified, using a more specific parameter type than needed.

(Keep this in mind: most times use the most general type you can.)

Ah, but other parts of the Axon framework ask for an AuditDataProvider (the original code, above) and I'm defining a new, more general interface. I cannot extend the original with mine; Java correctly complains that I am widening the type: all CommandMessages are Messages, but not all Messages are CommandMessages.

Java 8 method references to the rescue!

public interface MessageAuditDataProvider {
    Map<String, Object> provideAuditDataFor(final Message<?> message);

    default AuditDataProvider asAuditDataProvider() {
        return this::provideAuditDataFor;
    }
}

Because I accept a supertype in my new interface relative to the original, my method reference works simply and cleanly.

Followup: Feature Toogles for Spring

The original technique in Spring Techniques: Feature toggles for controller request handler methods works well in the small but failed for our large project. We have too many snowflakes, customized replumbing of Spring and Boot, and destructive interference forced another path. So we went with AOP, the magical fallback in such situations, a pity.

But help is on the way!

The Togglz project is close to an official solution for the 2.3.0 release (no timeline announced). I am pleased with the solution taken and contributed a small bit. Here's the documentation commit. Please try 2.3.0-RC1 when it goes to Maven Central.

Modern maven

I've pushed my first release of Modern-J, a maven archetype (project starter), to github. Mostly this is for myself, to have a decent maven archetype for starting spikes and projects.

One thing I learned about maven is dealing with version mismatch in dependencies. The technique is not to modify <dependency> blocks with exclusions but to add a <dependencyManagement> block:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>${junit.version}</version>
        </dependency>
    </dependencies>
</dependencyManagement>

(My POM sets "junit.version" to 4.12.)

This resolves the dependency mismatch between current JUnit (4.12) and the JUnit for System-Rules (4.11), a wonderful JUnit @Rule I hope to see eventually bundled with JUnit itself.

UPDATE: Hat tip to Qulice who beat me there first, though I'm not as strict.

Java generic exception specifiers

I'm not sure it's widely appreciated that throws clauses can take generic parameters, just as return type or arguments. You can leverage this to improve your error handling. Note the helpful type inference provided by the compiler:

public final class ErrorHandlingMain {
    public static void main(final String... args) {
        final Result<String, RuntimeException> fooResult
                = success("foo");
        final Result<String, Exception> barResult
                = failure(new IOException("bar")); // Note #1

        out.println(fooResult.get());  // Note #2
        out.println(get(fooResult));   // Note #3
        try {
            out.println(barResult.get());  // Note #4
        } catch (final Exception e) {
            out.println(e);
        }
        try {
            out.println(get(barResult));
        } catch (final Exception e) {
            out.println(e);
        }
    }

    public static <T, E extends Exception>
    T get(final Result<T, E> result)
            throws E {
        return result.get();
    }

    @FunctionalInterface
    public interface Result<T, E extends Exception> {
        T get()
                throws E;

        static <T> Result<T, RuntimeException>
        success(final T value) {
            return () -> value;
        }

        static <T, E extends Exception> Result<T, E>
        failure(
                final E exception) {
            return () -> {
                throw exception;
            };
        }
    }
}

(Unusual formatting to help with screen width.)

1. Note type widening from IOException to Exception. Reversing those types won't compile.
2. Compiler sees RuntimeException, does not require try/catch.
3. Likewise for static methods.
4. Compiler sees Exception, requires try/catch.

Spring Techniques: Feature toggles for controller request handler methods

Maria Gomez, a favorite colleague, asked a wonderful question, "How can I have feature toggles on Spring MVC controller request handler methods?" Existing Java feature toggle libraries focus on toggling individual beans, or using if/else logic inside methods, and don't work at the method level.

Given a trivial example toggle:

@Documented
@Retention(RUNTIME)
@Target(METHOD)
public @interface Enabled {
    boolean value();
}

I'd like my controller to work like this:

@RestController
@RequestMapping(PATH)
public class HelloWorldController {
    public static final String PATH = "/hello-world";

    private static final String texanTemplate = "Howdy, %s!";
    private static final String russianTemplate = "Привет, %s!";
    private final AtomicLong counter = new AtomicLong();

    @Enabled(true)
    @RequestMapping(value = "/{name}", method = GET)
    public Greeting sayHowdy(@PathVariable("name") final String name) {
        return new Greeting(counter.incrementAndGet(),
                format(texanTemplate, name));
    }

    @Enabled(false)
    @RequestMapping(value = "/{name}", method = GET)
    public Greeting sayPrivet(@PathVariable("name") final String name) {
        return new Greeting(counter.incrementAndGet(),
                format(russianTemplate, name));
    }
}

(Greeting is a simple struct turned into JSON by Spring.)

To make the example a little more sophisticated, I'd like to use a "3rd-party library" to decide on which features to activate (think "Togglz" or "FF4J", say):

@Component
public class EnabledChecker {
    public boolean isMapped(final Enabled enabled) {
        return null == enabled || enabled.value();
    }
}

Originally I investigated Spring's RequestCondition classes, thinking I could do the same as @RequestMapping(... match conditions ...). However, this is tricky! Spring uses these conditions to decide which method to invoke for each HTTP request, not when deciding which methods should be treated as the handler for a given HTTP path. Taking this route, Spring complains at wiring time of duplicate handlers for the same request path.

The right way is to control the initial wiring of request handler methods, not decide later. First extend RequestMappingHandlerMapping (what a mouthful!):

public class EnabledRequestMappingHandlerMapping
        extends RequestMappingHandlerMapping {
    @Autowired
    private EnabledChecker checker;

    @Override
    protected RequestMappingInfo getMappingForMethod(final Method method,
            final Class<?> handlerType) {
        final Enabled enabled = findAnnotation(method, Enabled.class);
        final boolean mapped = checker.isMapped(enabled);
        return mapped ? super.getMappingForMethod(method, handlerType) : null;
    }
}

Note this is not directly a bean (no @Component). We need one more bit, to override the factory method that creates these handler mappings:

@Configuration
public class EnabledWebMvcConfigurationSupport
        extends WebMvcConfigurationSupport {
    @Override
    protected RequestMappingHandlerMapping createRequestMappingHandlerMapping() {
        return new EnabledRequestMappingHandlerMapping();
    }
}

And Bob's your uncle. EnabledWebMvcConfigurationSupport ensures the returned ReqeustMappingHandlerMapping is injected, and so the "3rd-party library" is available to consult.

Full code in Github.

Looking for problematic logging with JUnit

Stefan Birkner's System Rules is one of my favorite JUnit extension libraries. I commonly use it to verify System.out and System.err, for example validating audit trail logging.

Growing tired of the same boilerplate, I rolled some simple rules into an aggregated JUnit @Rule, called NiceLoggingRule. It enforces:

• No logging to System.err
• No WARN or ERROR logging to System.out

A more sophisticated version would let the user decide on more than "log level" as to what is an acceptable log line, but it gives a good demonstration of writing complex JUnit rules:

public final class NiceLoggingRule
        implements TestRule {
    private static final Pattern NEWLINE = compile("\n");

    private final SystemOutRule sout = new SystemOutRule().
            enableLog().
            muteForSuccessfulTests();
    private final SystemErrRule serr = new SystemErrRule().
            enableLog();

    private final Pattern logLinePattern;
    private final Predicate<String> problematic;
    private final RuleChain delegate;

    public NiceLoggingRule(final String logLinePattern,
            final Predicate<String> problematic) {
        this.logLinePattern = compile(logLinePattern);
        this.problematic = problematic;
        delegate = outerRule(NiceLoggingStatement::new).
                around(sout).
                around(serr);
    }

    @Override
    public Statement apply(final Statement base,
            final Description description) {
        return delegate.apply(base, description);
    }

    private final class NiceLoggingStatement
            extends Statement {
        private final Statement base;
        private final Description description;

        private NiceLoggingStatement(final Statement base,
                final Description description) {
            this.base = base;
            this.description = description;
        }

        @Override
        public void evaluate()
                throws Throwable {
            base.evaluate();
            checkSystemErr(description);
            checkSystemOut(description);
        }

        private void checkSystemErr(final Description description) {
            final String cleanSerr = serr.getLogWithNormalizedLineSeparator();
            final List<String> errors = NEWLINE.splitAsStream(cleanSerr).
                    collect(toList());
            if (!errors.isEmpty())
                fail("Output to System.err from " + description + ":\n"
                        + cleanSerr);
        }

        private void checkSystemOut(final Description description) {
            final String cleanSout = sout.getLogWithNormalizedLineSeparator();
            final List<LogLine> problems = NEWLINE.splitAsStream(cleanSout).
                    map(LogLine::new).
                    filter(LogLine::problematic).
                    collect(toList());
            if (!problems.isEmpty())
                fail(problems.stream().
                        map(Object::toString).
                        collect(joining("",
                                "Problems to System.out from " + description
                                        + ":\n", "")));
        }
    }

    private final class LogLine {
        @Nonnull
        private final String line;
        @Nonnull
        private final String level;

        private LogLine(@Nonnull final String line) {
            final Matcher match = logLinePattern.matcher(line);
            if (!match.find()) // Not match! Ignore trailing CR?NL
                fail(format(
                        "Log line does not match expected pattern (%s): %s",
                        logLinePattern.pattern(), line));
            this.line = line;
            level = match.group("level");
        }

        public boolean problematic() {
            return problematic.test(level);
        }

        @Override
        public String toString() {
            return line;
        }
    }
}

For example, using it with Spring Boot's default log pattern one might write a factory helper:

public final class SpringDefaultNiceLoggingRule {
    private static final String logLevels = Stream.of(LogLevel.values()).
            filter(level -> OFF != level).
            map(Enum::name).
            collect(joining("|"));

    public static NiceLoggingRule springDefaultNiceLoggingRule() {
        return new NiceLoggingRule(
                "^(?<timestamp>\\d{4,4}-\\d{2,2}-\\d{2,2} \\d{2,2}:\\d{2,2}:\\d{2,2}\\.\\d{3,3}) +(?<level>"
                        + logLevels + ") +",
                SpringDefaultNiceLoggingRule::problematic);
    }

    private static boolean problematic(final String level) {
        return 0 > INFO.compareTo(LogLevel.valueOf(level));
    }
}

Then a simple Spring Boot unit test becomes:

@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(classes = MockServletContext.class)
public final class RootControllerTest {
    @Rule
    public final NiceLoggingRule niceLogging = springDefaultNiceLoggingRule();

    private MockMvc mvc;

    @Before
    public void setUp()
            throws Exception {
        mvc = standaloneSetup(new RootController()).build();
    }

    @Test
    public void shouldGetRoot()
            throws Exception {
        mvc.perform(get("/").
                accept(APPLICATION_JSON_UTF8)).
                andExpect(status().isOk()).
                andExpect(jsonPath("$.message", equalTo("Hello, world!")));
    }
}

RESTful helper script

I find this script useful working on modern RESTful services. It shows both the headers and the formatted JSON response body. The idea is that most times you provide a URL and want to see the full response. If you need extra flags for curl just add them (e.g., user/password). If you want to customize jq—say, filter for just a particular piece of the response—use a double-dash ("--") to separate curl and jq arguments:

An example with Spring Boot (plus some custom actuator endpoints). Note "jq" colorizes the output on the command line (below is plain text):

$ ~/bin/jurlq http://localhost:8081/remote-hello/health
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Strict-Transport-Security: max-age=31536000 ; includeSubDomains
X-Application-Context: remote-hello:8081
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Date: Mon, 21 Dec 2015 13:28:07 GMT

{
  "status": "UP",
  "cpu": {
    "status": "UP",
    "processors": 8,
    "system-loadavg": -1,
    "process-cpu-load": 0.22963892677420872,
    "process-cpu-time": "PT42.71875S",
    "system-cpu-load": -1
  },
  "file": {
    "status": "UP",
    "usable-disk": 55486464000,
    "total-disk": 299694551040
  },
  "java": {
    "status": "UP",
    "start-time": "2015-12-21T07:27:06.970-06:00",
    "uptime-beats": 0,
    "vm-name": "Java HotSpot(TM) 64-Bit Server VM",
    "vm-vendor": "Oracle Corporation",
    "vm-version": "25.66-b17"
  },
  "memory": {
    "status": "UP",
    "committed-virtual-memory": 991350784,
    "free-physical-memory": 1952854016,
    "free-swap-space": 1203003392,
    "total-physical-memory": 8540618752,
    "total-swap-space": 10354229248
  },
  "os": {
    "status": "UP",
    "arch": "amd64",
    "name": "Windows 10",
    "version": "10.0"
  },
  "threads": {
    "status": "UP",
    "count": 22,
    "daemon-count": 20,
    "peak-count": 22,
    "started-count": 26
  },
  "diskSpace": {
    "status": "UP",
    "free": 55486464000,
    "threshold": 10485760
  },
  "configServer": {
    "status": "UNKNOWN",
    "error": "no property sources located"
  },
  "hystrix": {
    "status": "UP"
  }
}

UPDATE: Tried Github Gist for the source, but it does not show in my blog feed reader. Here's the script:

#!/bin/bash

curl_args=()
for arg
do
    case "$arg" in
    -- ) shift ; break ;;
    * ) curl_args=("${curl_args[@]}" "$arg") ; shift ;;
    esac
done

jq_args=("${@-.}")

curl -s -D - "${curl_args[@]}" | tr -d '\r' | {
    while read line
    do
        case "$line" in
        '' ) echo ; break ;;
        * ) echo "$line" ;;
        esac
    done

    exec jq "${jq_args[@]}"
}

Spring REST showcase

I've noodled for some time now at mini-projects showcasing Spring REST with as many bells and whistles as I could pack in before it broke. I've never reached a satisfactory conclusion, which is more a testament to my mercurial temperament than to Spring. My ideal project would include:

• No XML, if avoidable (Lombok, I'm looking at you)
• No properties files (thank you, Spring YAML support)
• Annotations and code generation (thank you, Spring Boot and Lombok)
• Good documentation (Swagger and RAML, why do you need to be so tricky?)
• Complete REST adherence (HATEOAS, you are still an ugly child, sorry to say that)
• Modern Java
• No external container (Spring Boot to the rescue!)
• Many example integration points (and this is why I stay with you Spring)
• Maven build (sorry Gradle, I gave up makefiles so I would never again debug my build)
• Production support (Spring actuator: genius; see OSI)
• Cloud support
• Full CD pipeline (Boxfuse, you may save me yet; Github and Travis, you're still the best)

Essentially I want to implement a showcase REST microservice adhering to Larry Wall's three great virtues of a programmer: Laziness, Impatience and Hubris.

Well, there's always yet another Github repo.

Updates I'll keep updating this post as I find new things to desire in a showcase project.

Wrong, but technically right

tldr; — Immutable objects should have all-final fields.

I'm reading one of those "interview questions" posts and this jumped out at me:

Question 3: Does all property of Immutable Object needs to be final? (answer) Not necessary, as stated above you can achieve same functionality by making member as non-final but private and not modifying them except in constructor. Don't provide setter method for them and if it is a mutable object, then don't ever leak any reference for that member. Remember making a reference variable final, only ensures that it will not be reassigned a different value, but you can still change individual properties of object, pointed by that reference variable. This is one of the key point, Interviewer like to hear from candidates.

This is unfortunate. Although technically true, in my experience I've found the non-final case to come up <1% of the time. It's terribly misleading for less experienced developers to see an answer like this without having the context that goes with it.

To understand why I'm trepidatious, you can't go wrong with the C2 wiki on value objects. The right pattern for immutable objects in Java goes something like:

@EqualsAndHashCode
@RequiredArgsConstructor(staticName = "valueOf")
@ToString
public final class HiMomImImmutable {
    public final int i;
    public final String s;
}

Here I use Lombok annotations as shorthand for the equivalent Java code. If you're not familiar with Lombok, please become so—it's both a time-saver and a bug-killer in your code.

Lombok has a @Value annotation for these. However I disagree that with their choice that fields should be private and accessed with getters. To me this makes sense when consuming frameworks requiring bean-like getters, but not otherwise.

Back to the post, the author's answer is unclear: he mentions modifying final fields in the constructor. But of course, any field marked final may be assigned in the constructor—must be so assigned—if not initialized in declaration. Then if you have complex logic to initialize (itself a code smell), try this:

public SomeClass(/* args */) {
    this.someField = aStaticMethod(/* some other args */);
}

This encapsulates the initialization logic in a static method as the final field may be initialized only once, and not referred to before initialization.

Approximating tuples in Java

What is a tuple? It's an ordered collection of individually typed elements. Wikipedia has a complicated explanation that comes down to the same thing in the context of ordinary programming. The Python implementation of tuples is a good example.

There are many related concepts, such as:

Arrays and lists

Ordered but all elements are the same type

Ordered dictionaries (maps)

Ordered but all values are of the same type, and they are named

Aggregates (structs, unions)

Individually typed but named and unordered

The closest matches to tuples in current Java are "structs" (more below) and arrays or lists, each with drawbacks. For most use of tuples, arrays and lists are non-starters: elements are all of the same nominal type. There is a lot of thought around how to do this best in Java.

What do I mean by saying Java has "structs" ala "C"? The vast bulk of Java code hides fields away with private, but exposes them with getter methods (and setters when non-final). This is the "Java Bean" anti-pattern. An alternative is to simply expose fields directly:

public final class CartesianPoint {
    public final int x;
    public final int y;

    public CartesianPoint(int x, int y) {
        this.x = x;
        this.y = y;
    }
}

This is almost a tuple. However:

• CartesianPoint extends java.lang.Object but should not be an object in the OOP sense
• Elements are accessed by name, e.g., point.x, rather than position
• Elements are unordered—you cannot write a general function to process the first and second elements of the tuple without knowing 1) the struct type and 2) the names of the fields

But for many use cases this can be close enough, for example as a return type to simulate multiple value return. Hopefully Java 10 brings value types which resolves the java.lang.Object issue. This may even bring true tuples!

This example can be improved with Lombok:

@EqualsAndHashCode
@RequiredArgsConstructor(staticName = "valueOf")
@ToString(includeFieldNames = false)
public final class CartesianPoint {
    public final int x;
    public final int y;
}

Calling code would look like:

public CartesianPoint locate() {
    int x = computeX();
    int y = computeY();

    return CartesianPoint.valueOf(x, y);
}

Notice the visual similarity of the local variables x and y to the corresponding fields in CartesianPoint. Users of CartesianPoint would see:

public static void main(final String... args) {
    Boat boat = Boat.rowBoat();
    CartesianPoint point = boat.locate();

    out.printf("%s -> x is %d, y is %d%n", point, point.x, point.y);
}

$ ./float-boat 1 2
CartesianPoint(1, 2) -> x is 1, y is 2

If you want to go hog wild, the ordered and unnamed qualities of tuples can be simulated though not without significant noise:

Function<CartesianPoint, Integer> first = p -> p.x;
Function<CartesianPoint, Integer> second = p -> p.y;

out.printf("first is %d, second is %d%n",
        first.apply(point),
        second.apply(point));

Update

Streaming functionally:

Function<CartesianPoint, Integer> first = p -> p.x;
Function<CartesianPoint, Integer> second = p -> p.y;
Stream.of(first, second).
    map(f -> f.apply(point)).
    forEach(out::println);

What's wrong with Java 8 series by Pierre-Yves Saumont

Pierre-Yves Saumont wrote a series of articles for DZone. I feel remiss for having missed them. For example, What's Wrong in Java 8, Part IV: Monads explores java.util.Optional. Do not be misled by the post titles—yes, he criticizes Java 8 for what it could have been—, but he covers functional thinking in Java with depth, skill and panache.

The full list of "What's Wrong with Java 8" articles:

1. Currying vs Closures
2. Functions & Primitives
3. Streams and Parallel Streams
4. Monads
5. Tuples
6. Strictness
7. Streams again

And all his DZone articles.

Tracking Java 8 stream count

I had a coding problem to fail when a Java 8 stream was empty. One way is illustrated below in example1. Another approach was interesting, shown in example2

public final class Streamy {
    private static <T, E extends RuntimeException> void example1(
            final Stream<T> items,
            final Consumer<? super T> process,
            final Supplier<E> thrown)
            throws E {
        if (0 == items.
                peek(process).
                count())
            throw thrown.get();
    }

    private static <T, E extends RuntimeException> void example2(
            final Stream<T> items,
            final Consumer<? super T> process,
            final Supplier<E> thrown)
            throws E {
        final AtomicBoolean foundSome = new AtomicBoolean();
        try (final Stream<T> stream = items) {
            stream.
                    onClose(() -> {
                        if (!foundSome.get())
                            throw thrown.get();
                    }).
                    peek(__ -> foundSome.set(true)).
                    forEach(process);
        }
    }

    public static void main(final String... args) {
        example1(Stream.of(), out::println, RuntimeException::new);
        example2(Stream.of(), out::println, RuntimeException::new);
    }
}

Comparing them I find:

• Using count() is shorter and more clear
• Using onClose() is more expressive

I found it odd to use peek() in example1 to execute the real purpose of the code, and was happy to discover onClose() though disappointed to need try-with-resources for it to run.

It was unfortunate that the more expressive approach (peek() for side effect, forEach() for processing, onClose for post-processing check) was also harder to understand.

Downloading sources and javadocs automatically

Thanks to Ted Wise, I taught my "modern java" build to automatically download sources and javadocs. Using maven:

<plugin>
    <artifactId>maven-dependency-plugin</artifactId>
    <version>${maven-dependency-plugin.version}</version>
    <executions>
        <execution>
            <id>download-sources</id>
            <phase>generate-sources</phase>
            <goals>
                <goal>sources</goal>
            </goals>
        </execution>
        <execution>
            <id>download-javadocs</id>
            <phase>generate-sources</phase>
            <configuration>
                <classifier>javadoc</classifier>
            </configuration>
            <goals>
                <goal>resolve</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Struggling with Travis CI and Maven 3.3

I struggled with fixing Travis CI for my "labs" code. The main problem is lack of support for Maven 3.3; my POM enforcer requires it. Eventually I got something working with help from the Internet, though I'm not happy with it. It manually downloads and uses maven 3.3:

sudo: false
language: java
jdk:
  - oraclejdk8
# TODO: Gross until Travis support setting maven version or upgrades to 3.3
before_install:
  - wget http://apache.claz.org/maven/maven-3/3.3.3/binaries/apache-maven-3.3.3-bin.tar.gz
  - tar zxvf apache-maven-3.3.3-bin.tar.gz
  - chmod +x apache-maven-3.3.3/bin/mvn
  - export M2_HOME=$PWD/apache-maven-3.3.3
  - export PATH=$PWD/apache-maven-3.3.3/bin:${PATH}
  - hash -r
before_script:
  - export M2_HOME=$PWD/apache-maven-3.3.3
  - export PATH=$PWD/apache-maven-3.3.3/bin:${PATH}
  - hash -r
script: mvn verify -Dgpg.skip=true

Bonus

On another project using Travis CI I ran afoul of the 10,000 line limit for displaying build output in the web UI. Workaround, add this to the build_install section:

  - echo 'MAVEN_OPTS="-Dorg.slf4j.simpleLogger.defaultLogLevel=warn"' >~/.mavenrc

If you're using maven 3.3 or better, Karl Heinz Marbaise has a better approach with .mvn/jvm.config.

Spring advice

Today I had an exchange with an erstwhile colleague, one of those talented few equally comfortable in C++ and Java, great with demanding clients and fellow developers. He asked me for thoughts on Spring dependency injection. Branching out a bit, I replied:

Matter of taste/opinion.

Some things I prefer:

• Avoid setter injection if at all possible. I want my beans to be finished after DI, not changeable at runtime on accident (very nasty bug to track down) Constructor injection most preferred, followed by field
• Use standard annotations rather than Spring ones (@Inject), though @Value is needed for injected configuration if you're not using a dedicated configuration object
• Spring Java configuration beats XML almost all the time
• If the program doesn't need Spring DI, I prefer using Guice or Dagger. They're simpler, everything is at compile time, and error messages are better. The Spring non-DI libraries play fine with others (e.g., spring-jdbc)
• Avoid mixing business objects with wiring unless it makes sense. For example, JdbcTemplate should be new'ed as needed, injecting only the DataSource. Injecting the template is an anti-pattern
• Do use Spring Boot or Dropwizard, et al, if it makes sense. Big time savers, lots of good integration prepackaged
• For webby programs (apps, services) remember to test unit, controller and integration separately. Spring boot intro page has excellent code examples

Things get more interesting with multiple configurations and with cloud. For example:

• Do you make separate builds for each env, or one build with multiple configurations? Latter is traditional in EE world, former much better for cloud (devops, immutable, unikernel, etc)
• Do you pull in configuration externally, e.g., Spring Cloud Config, Netflix Archaius or Apache Zookeeper. I like this approach, but more complex and overkill for simple programs. Nearly mandatory for microservices, and strong choice when in cloud

Did I answer fairly?

Blog code 6

(Updated) I've published my blog code version 6 to Maven Central with javadocs. The previous version was 0.5; having a leading "0." was silly (there won't be a 1.0).

Interesting changes:

• Added the YAML modules. This is an annotation processor which takes YAML descriptions of Java data structures, and generates immutable classes for them.
• Added the Matching class for a pattern-matching DSL in Java. I'll improve on over time.

Automating dependency versions in Maven

Reading the most excellent Modern Java series of posts, I realized I could automate a command line step I always did manually—updating dependency versions:

$ mvn versions:update-properties

I can automate that! The simplest possible POM demonstrating:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>example-group</groupId>
    <artifactId>example-artifact</artifactId>
    <version>0-SNAPSHOT</version>

    <properties>
        <example-dependency.version>1</example-dependency.version>
        <generateBackupPoms>false</generateBackupPoms>
        <versions-maven-plugin.version>2.2</versions-maven-plugin.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>example-group</groupId>
            <artifactId>example-dependency</artifactId>
            <version>${example-dependency.version}</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>versions-maven-plugin</artifactId>
                <version>${versions-maven-plugin.version}</version>
                <executions>
                    <execution>
                        <id>update-dependencies</id>
                        <phase>validate</phase>
                        <goals>
                            <goal>update-properties</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

My hypothetical maven build says:

$ mvn clean
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building example-artifact 0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ example-artifact ---
[INFO]
[INFO] --- versions-maven-plugin:2.2:update-properties (update-dependencies) @ example-artifact ---
[INFO] Property ${example-dependency.version}: Leaving unchanged as 1.2.3
[INFO] Property ${versions-maven-plugin.version}: Leaving unchanged as 2.2

The key is using <phase>validate</phase> in the plugin execution. The first lifecycle phase for maven is "validate". Setting the property "generateBackupPoms" to "false" prevents pom.xml.versionsBackup files from littering the project. You are using source control, aren't you?

Update

This approach is best during development, especially for greenfield projects. I would not recommend it for stable or legacy projects where dependency versions need to be kept constant.

JUnit @Rule for a SQL transaction

I needed to run some Java test code in a SQL transaction. It modifies the database, changes that should be 1) invisible to other tests, and 2) thrown away after tests finish.

I am using JUnit (of course) so I wrote a @Rule to provide a transaction to my test methods (note I'm using Lombok for convenience):

@RequiredArgsConstructor
public final class SQLTransactionRule
        extends ExternalResource {
    private final DataSource original;
    private final String user;
    private final String password;
    private final Consumer testDataSource;

    private Connection conn;

    public SQLTransactionRule(final DataSource original,
            final Consumer testDataSource) {
        this(original, null, null, testDataSource);
    }

    @Override
    protected void before()
            throws Throwable {
        super.before();
        beginTransaction();
        testDataSource.accept(testDataSource(conn));
    }

    @Override
    protected void after() {
        testDataSource.accept(null);
        rollbackTransaction();
        super.after();
    }

    private void beginTransaction() {
        try {
            final Connection conn = original.getConnection(user, password);
            conn.setAutoCommit(false);
            this.conn = conn;
        } catch (final SQLException e) {
            throw new RuntimeException(
                    format("Cannot create transaction from %s: %s", original,
                            getRootCause(e)), e);
        }
    }

    private void rollbackTransaction() {
        try {
            conn.rollback();
        } catch (final SQLException e) {
            throw new RuntimeException(
                    format("Cannot rollback transaction to %s: %s", original,
                            getRootCause(e)), e);
        }
    }

    private DataSource testDataSource(final Connection conn) {
        return (DataSource) newProxyInstance(getClass().getClassLoader(),
                new Class[]{DataSource.class}, (proxy, method, args) -> {
                    System.out.println("SQLTransactionRule.testDataSource");
                    if (Object.class.equals(method.getDeclaringClass()))
                        return method.invoke(proxy, args);
                    else if ("getConnection".equals(method.getName()))
                        return conn;
                    else
                        return method.invoke(original, args);
                });
    }
}

The motivation for the Consumer of "testDataSource" and for the proxied data source was avoidance of dependencies. Original code has this convenience method, but I decided that was a concern of the test code not the @Rule:

public JdbcTemplate newJdbcTemplate() {
    return new JdbcTemplate(testDataSource);
}

Of course there are tests for the @Rule as well! And this post on Mockito was handy.

@RunWith(MockitoJUnitRunner.class)
public final class SQLTransactionRuleTest {
    @Mock
    private DataSource original;
    @Mock
    private Connection conn;

    @Rule
    public final ExpectedException thrown = ExpectedException.none();

    private SQLTransactionRule rule;
    private DataSource testDataSource;

    @Before
    public void setUp()
            throws SQLException {
        when(original.getConnection(anyString(), anyString())).
                thenReturn(conn);
        rule = new SQLTransactionRule(original,
                testDataSource -> this.testDataSource = testDataSource);
    }

    @Test
    public void shouldExecuteStatement()
            throws Throwable {
        final boolean[] called = {false};
        executeStatement(() -> called[0] = true);
        assertThat(called[0], is(true));
    }

    @Test
    public void shouldPublishTestDataSource()
            throws Throwable {
        executeStatement(
                () -> assertThat(testDataSource, is(notNullValue())));
    }

    @Test
    public void shouldReuseOriginalConnection()
            throws Throwable {
        executeStatement(() -> assertThat(testDataSource.getConnection(),
                is(sameInstance(conn))));
    }

    @Test
    public void shouldTransact()
            throws Throwable {
        executeStatement(() -> verify(conn).setAutoCommit(eq(false)));
        verify(conn).rollback();
    }

    @Test
    public void shouldDescribe()
            throws Throwable {
        thrown.expect(AssertionError.class);
        thrown.expectMessage(containsString("alpha"));
        thrown.expectMessage(containsString("beta"));

        executeStatement(() -> assertThat("alpha", is(equalTo("beta"))));
    }

    @FunctionalInterface
    private interface RunMe {
        void execute()
                throws Throwable;
    }

    private void executeStatement(final RunMe statement)
            throws Throwable {
        rule.apply(new Statement() {
            @Override
            public void evaluate()
                    throws Throwable {
                statement.execute();
            }
        }, EMPTY).evaluate();
    }
}