Monday, May 23, 2016

Automating github

I got tired of downloading my own scripts from Github when working among multiple projects. So I automated it, of course. The bitsh project reuses a test script from the shell project, and now the Makefile for bitsh is simply:

SHELL=/bin/bash

test:
	@[ -t 1 ] && flags=-c ; \
	./test-bitsh.sh -i -- $$flags t

When run-tests is updated in Github, bitsh automatically picks up the changes. And I learned the value of ETag.

By the way, why "bitsh"? I hunted around for project names combining "git" and "bash" and found most of them already taken. Beggars can't be choosers.

UPDATE: I found the <TAB> character got munged by Blogger to a single space. This is unfortunate as you cannot copy out a valid Makefile. One solution is to put in an HTML tab entity.

Sunday, May 22, 2016

Metaprogramming with Bash

Most programmers do not take full advantage of the languages they work in, though some languages make this a real challenge. Take metaprogramming, or programs that have some self-knowledge. LISP-family languages make this easy and natural; those with macros even more so. Bytecode languages (think Java), and even more so object code languages (think "C"), fall back on extra-linguistic magic such as AOP rewriting.

Text-based languages lay in a middle ground. Best known is Bash. Rarely do programmers take full advantage of Bash features, and few would think of metaprogramming. Not as clean as LISP macros, it is still straight forward.

As an example of function rewriting note line 21. The surrounding function body redefines an existing function, incorporating the original's body into itself. This is very similar to aspect-oriented programming with a "around" advice. Much care is taken to preserve the original function context.

This is a fully working script—give it a try!

#!/bin/bash

export PS4='+${BASH_SOURCE}:${LINENO}: ${FUNCNAME[0]:+${FUNCNAME[0]}(): } '

pgreen=$(printf "\e[32m")
pred=$(printf "\e[31m")
pboldred=$(printf "\e[31;1m")
preset=$(printf "\e[0m")
pcheckmark=$(printf "\xE2\x9C\x93")
pballotx=$(printf "\xE2\x9C\x97")
pinterrobang=$(printf "\xE2\x80\xBD")

function _register {
    case $# in
    1 ) local -r name=$1 arity=0 ;;
    2 ) local -r name=$1 arity=$2 ;;
    esac
    read -d '' -r wrapper <<EOF
function $name {
    # Original function
    $(declare -f $name | sed '1,2d;$d')

    __e=\$?
    shift $arity
    if (( 0 < __e || 0 == \$# ))
    then
        __tally \$__e
    else
        "\$@"
        __e=\$?
        __tally \$__e
    fi
}
EOF
    eval "$wrapper"
}

let __passed=0 __failed=0 __errored=0
function __tally {
    local -r __e=$1
    $__tallied && return $__e
    __tallied=true
    case $__e in
    0 ) let ++__passed ;;
    1 ) let ++__failed ;;
    * ) let ++__errored ;;
    esac
    _print_result $__e
    return $__e
}

function _print_result {
    local -r __e=$1
    case $__e in
    0 ) echo -e $pgreen$pcheckmark$preset $_scenario_name ;;
    1 ) echo -e $pred$pballotx$preset $_scenario_name ;;
    * ) echo -e $pboldred$pinterrobang$preset $_scenario_name ;;
    esac
}

function check_exit {
    (( $? == $1 ))
}

function make_exit {
    local -r e=$1
    shift
    (exit $e)
    "$@"
}

function check_d {
    [[ $PWD == $1 ]]
}

function change_d {
    cd $1
}

function variadic {
    :
}

function early_return {
    return $1
}

function eq {
    [[ "$bob" == nancy ]]
}

function normal_return {
    (exit $1)
}

function f {
    local -r bob=nancy
}

function AND {
    "$@"
}

function SCENARIO {
    local -r _scenario_name="$1"
    shift
    local __tallied=false
    local __e=0
    pushd $PWD >/dev/null
    "$@"
    __tally $?
    popd >/dev/null
}

_register f
_register normal_return 1
_register variadic 1
_register eq 
_register early_return 1
_register change_d 1
_register check_exit 1

echo "Expect 10 passes, 4 failures and 4 errors:"
SCENARIO "Normal return pass direct" normal_return 0
SCENARIO "Normal return fail direct" normal_return 1
SCENARIO "Normal return error direct" normal_return 2
SCENARIO "Normal return pass indirect" f AND normal_return 0
SCENARIO "Normal return fail indirect" f AND normal_return 1
SCENARIO "Normal return error indirect" f AND normal_return 2
SCENARIO "Early return pass indirect" f AND early_return 0
SCENARIO "Early return fail indirect" f AND early_return 1
SCENARIO "Early return error indirect" f AND early_return 2
SCENARIO "Early return pass direct" early_return 0
SCENARIO "Early return fail direct" early_return 1
SCENARIO "Early return error direct" early_return 2
SCENARIO "Variadic with none" f AND variadic
SCENARIO "Variadic with one" f AND variadic apple
SCENARIO "Local vars" f AND eq
here=$PWD
SCENARIO "Change directory" change_d /tmp
SCENARIO "Check directory" check_d $here
SCENARIO "Check exit" make_exit 1 AND check_exit 1

(( 0 == __passed )) || ppassed=$pgreen
(( 0 == __failed )) || pfailed=$pred
(( 0 == __errored )) || perrored=$pboldred
cat <<EOS
$ppassed$__passed PASSED$preset, $pfailed$__failed FAILED$preset, $perrored$__errored ERRORED$preset
EOS

Practical use of this script as a test framework would pull out SCENARIO and AND to a separate source script, included with "." or source, put the registered functions and their helpers in another source script, and provide command-line parsing to pick out which tests to execute. test-bitsh.sh is an example in progress.

Friday, May 20, 2016

Color your world

My coworkers use many ad hoc or single-purpose scripts, things like: checking system status, wrappers for build systems, launching services locally, etc. My UNIX background tells me, "keep it simple, avoid output; Silence is Golden."

Somehow my younger colleagues aren't impressed.

So to avoid acting my age I started sprinkling color into my scripts, and it worked. Feedback was uniformly positive. And true to my UNIX roots, I provided command line flags to disable color.

Some lessons for budding BASHers:

  1. Yes, experiment and learn, but be sure to do your research. The Internet has outstanding help for BASH.
  2. Learn standard idioms (see below).
  3. Don't overdo it. Color for summary lines and warnings have more impact when the rest of the text is plain.
  4. Keep functions small, just as you would in other languages. BASH is a programming language with a command line, so keep your good habits when writing shell.
  5. Collaborate, pair! This comes naturally to my fellows. Coding is more enjoyable, goes faster and has fewer bugs.

Idioms

Most of these idioms appear in testing with bash, a simple BASH BDD test framework I wrote for demonstration.

Process command-line flags

this=false
that=true
while getopts :htT-: opt
do
    [[ - == $opt ]] && opt="${OPTARG%%=*}" OPTARG="${OPTARG#*=}"
    case $opt in
    h | help ) print_help ; exit 0 ;;
    t | this ) this=true ;;
    T | no-that ) that=false ;;
    esac
done
shift $((OPTIND - 1))

Keep boolean toggles simple

run_faster=true

if $run_faster
then
    use_faster_algorithm "$@"
else
    use_more_correct_algorithm "$@"
fi

Simple coloring

pgreen="\e[32m"
pred="\e[31m"
preset="\e0m"

if $success
then
    echo -e "${pgreen}PASS${preset} $test_name"
else
    echo -e "${pred}FAIL${preset} $test_name - $failure_reason"
EOM
fi

Consistent exit codes

function check_it {
    local -r failed=$1
    local -r syntax_error=$2
    if $syntax_error
    then
        return 2
    elif $failed
    then
        return 1
    else
        return 0
    fi
}

Coda

There are many more idioms to learn, hopefully this taste catches your interest. I was careful to include others mixed in with these (what does local -r do?) to whet the appetite for research. Go try the BASH debugger sometime.

UPDATE: Fixed thinko. ANSI escape codes need to be handled by echo -e or printf, not sent directly via cat!

Tuesday, May 03, 2016

Do not test Java getters and setters

An excellent project from Osman Shoukry to automate testing of Java getters and setters—that is when you have getters and setters to test. There's the rub: do not write getters or setters.

For starters they violate encapsulation, exposing your objects innards to others. Ok, but there are frameworks which require them, even in 2016. What to do?

Generate them:

@Getter
@Setter
@RequiredArgsConstructor
public final class SampleBean {
    private final String left;
    private String right;
}

With what result?

public final class SampleBean {
    private final String left;
    private String right;

    @java.lang.SuppressWarnings("all")
    @javax.annotation.Generated("lombok")
    public String getLeft() {
        return this.left;
    }

    @java.lang.SuppressWarnings("all")
    @javax.annotation.Generated("lombok")
    public String getRight() {
        return this.right;
    }

    @java.lang.SuppressWarnings("all")
    @javax.annotation.Generated("lombok")
    public void setRight(final String right) {
        this.right = right;
    }

    @java.beans.ConstructorProperties({"left"})
    @java.lang.SuppressWarnings("all")
    @javax.annotation.Generated("lombok")
    public SampleBean(final String left) {
        this.left = left;
    }
}

This code need never be tested. Test the generator, not the generated code. If the generator is correct, so is the generated code. In this case, Lombok is heavily tested.

As an alternative to the constructor taking field values, Jesse Wilson has interesting advice.

Friday, April 08, 2016

BDD-style fluent testing in BASH

I wanted to impress on my colleagues that BASH was still hip, still relevant. And that I wasn't an old hacker. So I wrote a small BDD test framework for BASH.

Techniques

Fluent coding relies on several BASH features:

  • Variable expansion happens before executing commands
  • A shell function is indistinguishable from a program: they are called the same way
  • Local function variables are dynamically scoped but only within a function, so are visible to other functions called within that scope, directly or indirectly through further function calls

Together with Builder pattern, it's easy to write given/when/then tests. (Builder pattern here solves the problem not of telescoping constructors, but massive, arbitrary argument lists.)

So when you run:

function c {
    echo "$message"
}

function b {
    "$@"
}

function a {
    local message="$1"
    shift
    "$@"
}

a "Bob's your uncle" b c

You see the output:

Bob's your uncle

How does this work?

First BASH expands variables. In function a this means that after the first argument is remembered and removed from the argument list, "$@" expands to b c. Then b c is executed.

Then BASH calls the function b with argument "c". Similarly b expands "$@" to c and calls it.

Finally as $message is visible in functions called by a, c prints the first argument passed to a (as it was remebered in the variable $message), or "Bob's your uncle" in this example.

Running the snippet with xtrace makes this clear (assuming the snippet is saved in a file named example):

bash -x example
+ a 'Bob'\''s your uncle' b c
+ local 'message=Bob'\''s your uncle'
+ shift
+ b c
+ c
+ echo 'Bob'\''s your uncle'
Bob's your uncle

So the test functions for given_jar, when_run and then_expect (along with other, similar functions) work the same way. Keep this in mind.

Application

So how does this buy me fluent BDD?

Given these functions:

function then_expect {
    local expectation="$1"
    shift

    if [[ some_test "$pre_condition" "$condition" "$expectation" ]]
    then
        echo "PASS: $scenario"
        return 0
    else
        echo "FAIL: $scenario"
        return 1
    fi
}

function when {
    local condition="$1"
    shift
    "$@"
}

function given {
    local pre_condition="$1"
    shift
    "$@"
}

function scenario {
    local scenario="$1"
    shift
    "$@"
}

When you write:

scenario "Some test case" \
    given "Something always true" \
    when "Something you want to test" \
    then_expect "Some outcome"

Then it executes:

some_test "Something always true" "Something you want to test" "Some outcome"

And prints one of:

PASS Some test case
FAIL Some test case

A real example at https://github.com/binkley/shell/tree/master/testing-bash.

Friday, March 25, 2016

Bash long options

UPDATED: Long options with arguments in the "name=value" style. The original post neglected this important case.

For years I've never know quite the right way to handle long options in Bash without significant, ugly coding. The usual sources (Advanced Bash-Scripting Guide, The Bash Hackers Wiki, others) are not much help. An occasional glimpse appears on StackOverflow, but not well explained or voted.

Solution

Working with a colleague yesterday, we found this:

name=Bob
while getopts :hn:-: opt
do
    [[ - == $opt ]] && opt=${OPTARG%%=*} OPTARG=${OPTARG#*=}
    case $opt in
    h | help ) print_help ; exit 0 ;;
    n | name ) name=$OPTARG ;;
    * ) print_usage >&2 ; exit 2 ;;
    esac
done
shift $((OPTIND - 1))
echo "$0: $name"
$ ./try-me -h
Usage: ./try-me [-h|--help][-n|--name <name>]
$ ./try-me --help
Usage: ./try-me [-h|--help][-n|--name <name>]
$ ./try-me -n Fred
./try-me: Fred
$ ./try-me --name=Fred
./try-me: Fred

Magic!

I checked with bash 3.2 and 4.3. At least for these, the '-' option argument has a bit of magic when it takes an argument. When the argument to '-' starts with a dash, as in --help (here "-help" is the argument to the '-' option), getopts drops the argument's leading '-', and OPTARG is just the text ("help" in this example). Only '-' has this magic.

Add a quick check for '-' at the top of the while-loop, and the case-block is simple and clear.

Bob's your uncle.

Wednesday, March 02, 2016

Hand-rolling builders in Java

I showed off a hand-rolled example Java builder pattern today. It has some benefits over existing builder solutions, but is more work than I like:

  1. All parts of the builder are immutable; you can pass a partially built object for another object to complete (I'm looking at you, Lombok)
  2. It's syntactically complete; that is, code won't compile without providing all arguments for the thing to build
  3. It's easy to generalize; in fact, I'm thinking about an annotation processor to generate it for you (but not there quite yet)
@EqualsAndHashCode
@ToString
public final class CartesianPoint {
    public final int x;
    public final int y;

    public static Builder builder() {
        return new Builder();
    }

    private CartesianPoint(final int x, final int y) {
        this.x = x;
        this.y = y;
    }

    public static final class Builder {
        public WithX x(final int x) {
            return new WithX(x);
        }

        @RequiredArgsConstructor
        public static final class WithX {
            private final int x;

            public WithY y(final int y) {
                return new WithY(y);
            }

            @RequiredArgsConstructor
            public final class WithY {
                private final int y;

                public CartesianPoint build() {
                    return new CartesianPoint(x, y);
                }
            }
        }
    }
}

That was a lot to say! Which is why most times you don't hand-roll builders. Usage is obvious:

@Test
public void shouldBuild() {
    final CartesianPoint point = CartesianPoint.builder().
            x(1).
            y(2).
            build();
    assertThat(point.x).isEqualTo(1)
    assertThat(point.y).isEqualTo(2);
}

Adding caching for equivalent values is not hard:

public static final class Builder {
    private static final ConcurrentMap
            cache = new ConcurrentHashMap<>();

    public WithX x(final int x) {
        return new WithX(x);
    }

    @RequiredArgsConstructor
    public static final class WithX {
        private final int x;

        public WithY y(final int y) {
            return new WithY(y);
        }

        @RequiredArgsConstructor
        public final class WithY {
            private final int y;

            public CartesianPoint build() {
                final CartesianPoint point = new CartesianPoint(x, y);
                final CartesianPoint cached = cache.
                        putIfAbsent(point, point);
                return null == cached ? point : cached;
            }
        }
    }
}

Monday, February 29, 2016

Maven testing module

My usual practice is to put test dependencies in my Maven parent POM when working on a multi-module project. And I usually have a "testing" module as well for shared test resources such as a logback-test.xml to quiet down test output.

The test dependencies look like clutter in my parent POM, and they are, or so I recently realized.

As all my non-test modules use the "testing" module as a test dependency, I clean this up by moving my test dependencies out of the parent POM and into the "testing" module alongside the common resources. So my layout looks like:

Parent POM
Common properties such as dependency versions, dependencies management marks the "testing" module as "test" scope.
Testing POM
Test dependencies not marked as "test" scope—consumers of this module will mark it as "test", and its transitive dependencies will automatically be "test" as well.
Non-test POMs
Use the "testing" module as a dependency—specified in the parent POM dependencies management—, no test dependencies or resources inherited from parent POM.

Examples:

Friday, February 26, 2016

Java 8 shim method references

So I'm working on Spring Boot autoconfiguration for Axon Framework. I run into a nice interface in Axon framework that is unfortunately too specific. So I generalize. The original, pared down:

public interface AuditDataProvider {
    Map<String, Object> provideAuditDataFor(CommandMessage<?> command);
}

Aha! A SAM interface, interesting. So I craft my look-a-like:

public interface AuditDataProvider {
    Map<String, Object> provideAuditDataFor(Message<?> command);
}

Not much difference. Note the method parameter is Message rather than CommandMessage. This works fine as the implementation I have in mind uses getMetaData(), defined in Message and inherited by CommandMessage—so the original Axon interface is overspecified, using a more specific parameter type than needed.

(Keep this in mind: most times use the most general type you can.)

Ah, but other parts of the Axon framework ask for an AuditDataProvider (the original code, above) and I'm defining a new, more general interface. I cannot extend the original with mine; Java correctly complains that I am widening the type: all CommandMessages are Messages, but not all Messages are CommandMessages.

Java 8 method references to the rescue!

public interface MessageAuditDataProvider {
    Map<String, Object> provideAuditDataFor(final Message<?> message);

    default AuditDataProvider asAuditDataProvider() {
        return this::provideAuditDataFor;
    }
}

Because I accept a supertype in my new interface relative to the original, my method reference works simply and cleanly.

Monday, February 22, 2016

Followup: Feature Toogles for Spring

The original technique in Spring Techniques: Feature toggles for controller request handler methods works well in the small but failed for our large project. We have too many snowflakes, customized replumbing of Spring and Boot, and destructive interference forced another path. So we went with AOP, the magical fallback in such situations, a pity.

But help is on the way!

The Togglz project is close to an official solution for the 2.3.0 release (no timeline announced). I am pleased with the solution taken and contributed a small bit. Here's the documentation commit. Please try 2.3.0-RC1 when it goes to Maven Central.

Sunday, February 21, 2016

Modern maven

I've pushed my first release of Modern-J, a maven archetype (project starter), to github. Mostly this is for myself, to have a decent maven archetype for starting spikes and projects.

One thing I learned about maven is dealing with version mismatch in dependencies. The technique is not to modify <dependency> blocks with exclusions but to add a <dependencyManagement> block:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>${junit.version}</version>
        </dependency>
    </dependencies>
</dependencyManagement>

(My POM sets "junit.version" to 4.12.)

This resolves the dependency mismatch between current JUnit (4.12) and the JUnit for System-Rules (4.11), a wonderful JUnit @Rule I hope to see eventually bundled with JUnit itself.

UPDATE: Hat tip to Qulice who beat me there first, though I'm not as strict.

Monday, February 15, 2016

Java generic exception specifiers

I'm not sure it's widely appreciated that throws clauses can take generic parameters, just as return type or arguments. You can leverage this to improve your error handling. Note the helpful type inference provided by the compiler:

public final class ErrorHandlingMain {
    public static void main(final String... args) {
        final Result<String, RuntimeException> fooResult
                = success("foo");
        final Result<String, Exception> barResult
                = failure(new IOException("bar")); // Note #1

        out.println(fooResult.get());  // Note #2
        out.println(get(fooResult));   // Note #3
        try {
            out.println(barResult.get());  // Note #4
        } catch (final Exception e) {
            out.println(e);
        }
        try {
            out.println(get(barResult));
        } catch (final Exception e) {
            out.println(e);
        }
    }

    public static <T, E extends Exception>
    T get(final Result<T, E> result)
            throws E {
        return result.get();
    }

    @FunctionalInterface
    public interface Result<T, E extends Exception> {
        T get()
                throws E;

        static <T> Result<T, RuntimeException>
        success(final T value) {
            return () -> value;
        }

        static <T, E extends Exception> Result<T, E>
        failure(
                final E exception) {
            return () -> {
                throw exception;
            };
        }
    }
}

(Unusual formatting to help with screen width.)

  1. Note type widening from IOException to Exception. Reversing those types won't compile.
  2. Compiler sees RuntimeException, does not require try/catch.
  3. Likewise for static methods.
  4. Compiler sees Exception, requires try/catch.