Old scripting

The landscape

I'm helping someone with a scripting problem on an old system and an old shell. How old? Try IBM AIX 6.1, first released in 2007, and ksh93 "e" released in ... 1993. At least the AIX is a version of 6.1 from 2014! (Kudos to IBM for treating long-term support seriously.)

A second point to ponder. The goal is to improve remote scripting—running scripts on a remote machine. In this environment, ssh exists but is not used. The remote execution tool chosen is rexec, considered one of the most dangerous tools possible. But my remit is not to address the insecurity, just to improve the scripting. (They know this is a bad, and are actively working to eventually resolve.)

So, given these constraints, what problem am I solving?

Example problem

This environment makes extensive use of remotely executed scripts to wire together a distributed, locally-hosted system. Current scripts duplicate the same approach, each implemented as a one-off: Copy a script to a remote machine with rcp; use rexec to invoke the script, capturing the output to a file on the remote host; copy the captured file back to the local host; process the output file; sometimes clean up the remote host afterwards.

Some gotchas to watch out for with ksh93e or rexec:

• Function tracing - Using the standard xtrace setting to trace script execution in ksh93 has problems with tracing functions, and requires using old-style function syntax
• Variable scope - To keep variables local to a function in ksh93, you must use the new-style function syntax (note the conflict with tracing)
• Exit broken with trap - When calling exit to quit a remote script, trap does not get a correct $? variable (it is always 0, as exit succeeded in returning a non-0 exit status). Instead one must "set" $? with the code of a failing command, and then leave with a plain call to exit
• No pipefail - Release "e" of ksh93 just does not know anything about set -o pipefail, and there is no uninstrusive workaround. This now common feature showed up in release "g"
• No exit code - Would you believe rexec does not itself exit with the exit code of the remote command, never has, and never will? It always exits 0 if the remote command could be started.
• Buffered stderr - Empirically, rexec (at least the version with this AIX) buffers the stderr stream of remote commands, and only flushes when rexec exits, so the sense of ordering between stdout, stderr and the command-line prompt is even worse than usual (the actual handling is unspecified)

This problem and environment triggers a memory: The last time I worked on AIX was in 1994, and it was almost the same problem! I really thought I had escaped those days.

A solution

So I refactored. I couldn't change the use of rexec—this environment is not ready for SSH key management—, I couldn't replace KSH93 with BASH or replace AIX with Linux, but I could do something about the imperfect duplication and random detritus files.

The solution

Note the need to call a fail function instead of exit directly because of poor interaction with trap.

Assuming some help, such as a global progname variable (which could simply be $0), and avoiding remote temporary files:

_transfer_exit_code() {
    while read line
    do
        case $line in
            ^[0-9] | ^[1-9][0-9] | ^11[0-9] | ^12[0-7] ) return ${line#^} ;;
            * ) printf '%s\n' "$line" ;;
        esac
    done
    return 1  # ksh93e lacks pipefail; we get here when 'rscript' failed
}

rscript() {
    case $# in
        0 | 1 )
            echo "$progname: BUG: Usage: rexec SCRIPT-NAME HOSTNAME [ARGS]..." >&2 ;;
        * ) script_name=$1 ; shift
            hostname=$1 ; shift ;;
    esac
    # Trace callers script if we ourselves are being traced
    case $- in
        *x* ) _set_x='set -x' ;;
    esac

    rexec $hostname /usr/bin/ksh93 -s "$@" <<EOS | _transfer_exit_code
set - "$@"  # Only reasonable way to pass through function arguments

# Work around AIX ksh93 return code of exit ignored by trap
fail() {
    return \$1
}

# Our hook to capture the exit code for rexec who dumbly swallows it
trap 'rc=\$?; echo ^\$rc; exit \$rc' EXIT

PS4='+$script_name:\$(( LINENO - 14 )) (\$SECONDS) '
$_set_x

# The callers script
$(cat)
EOS
}

Example use

#!/usr/bin/ksh93

progname=${0##*/}

PS4='+$progname:$LINENO ($SECONDS) '

usage() {
    echo "Usage: $0 [-d] HOSTNAME"
}

. rexec.ksh

debug=false
while getopts :d opt
do
    case $opt in
        d ) debug=true ;;
        * ) usage >&2 ; exit 2 ;;
    esac
done
shift $(( OPTIND - 1 ))

case $# in
    1 ) hostname=$1 ;;
    * ) usage >&2 ; exit 2 ;;
esac

$debug && set -x

script_name=My-Remote-Script

tmp=${TMPDIR-/tmp}/$progname.$RANDOM
trap 'rm -f $tmp' EXIT

rscript $script_name $hostname Katy <<'EOS' >$tmp
echo $#: $1
fail 3
EOS

case $? in
    3 ) ;;
    * ) echo "$0: Did not pass through exit code" >&2 ; exit 1 ;;
esac

case "$(<$tmp)" in
    '1: Katy' ) ;;
    * ) echo "$0: Did not pass through arguments" >&2 ; exit 1 ;;
esac

Source

The code is in GitHub.

Quick diff tip, make, et al

I'm using make for a simple shell project, to run tests before committing. The check was trivial:

SHELL = bash

test:
	@./run-tests t | grep 'Summary: 16 PASSED, 0 FAILED, 0 ERRORED' >/dev/null

This has the nice quality of Silence is Golden: say nothing when all is good. However, it loses the quality of Complain on Failure: it simply fails without saying why.

A better solution, preserving both qualities:

SHELL = bash

test:
	@diff --color=auto \
	    <(./run-tests t | grep 'Summary: .* PASSED, .* FAILED, .* ERRORED') \
	    <(echo 'Summary: 16 PASSED, 0 FAILED, 0 ERRORED')

It still says nothing when all is good, but now shows on failure how many tests went awry. Bonus: color for programmers who like that sort of thing.

Why set SHELL to bash? I'm taking advantage of Process Substitution. Essentially the command outputs inside the subshells are turned into special kinds of files, and diff likes to compare files. Ksh and Zsh also support process substitution, so I'm going with the most widely available option.

UPDATE:

Why are my arguments to diff ordered like that? In usual testing language, I'm comparing "actual" vs "expected", and more commonly you'll see programmers list "expected" first.

diff by default colors the left-hand input in RED, and the right-hand input in GREEN. On failure, it makes more sense to color "actual" in red and "expected" in green. Example output on failure:

$ make
1c1
< Summary: 17 PASSED, 1 FAILED, 0 ERRORED
---
> Summary: 19 PASSED, 0 FAILED, 0 ERRORED
make: *** [Makefile:4: test] Error 1

Followup on Bash long options

A followup on Bash long options.

The top-level option parsing while-loop I discussed works fine for regular options. Sometimes you need special parsing for subcommand options. A hypothetical example might be:

$ my-script --toplevel-thing my-subcommand --something-wonderful option-arg

Here the --toplevel-thing option is for my-script, and --something-wonderful option and its option-arg is for my-subcommand. Regular getopts parsing will try to handle all options for the top level, failing to distinguish subcommand options as separate. Further, getopts in a function does not behave quite as expected.

One solution is simple and hearkens back to the pre-getopts days. For the top level:

while (( 0 < $# ))
do
    case $1 in
        --toplevel-thing ) _toplevel_thing=true ; shift ;;
        -* ) usage >&2 ; exit 2 ;;
        * ) break ;;
    esac
done

Using a while-loop with explicit breaks avoids looking too far along the command line, and wrongly consuming options meant for subcommands. Rechecking $# each time through the loop breaks gracefully. Similarly, for subcommands expressed in a function:

function my-subcommand {
    while (( 0 < $# ))
    do
        case $1 in
            --something-special ) local option_arg="$2" ; shift 2 ;;
            * ) usage >&2 ; exit 2 ;;
        esac
    done
    # Rest of my-subcommand, using `option_arg` if provided

This uses the same pattern as the top level so you avoid needing to remember to handle top level one way, and subcommand another.

An example script using this pattern.

Gee wilickers

I have a common command-line pattern I grew tired of typing. An example

$ mvn verify | tee verify.out

I use this pattern so often as I want to both watch the build on screen, and have a save file to grep when something goes wrong. Sometimes I also find myself telling the computer:

$ mvn verify | tee verify.out
$ mv verify.out verify-old.out
$ $EDITOR pom.xml
$ mvn verify | tee verify.out
$ diff verify-old.out verify.out

I want to see what changed in my build. But ... too much typing! So I automated with gee, a mashup of git and tee. You can think of it as source control for <STDOUT>.

Now I can type:

$ gee -o verify.out mvn verify
$ gee -g git diff HEAD^  # Did build output change?

How does it work? gee keeps a git repository in a hidden directory (.gee), committing program output there using tee. It follows simple conventions for file name and commit message (changeable with flags), and accepts <STDIN>:

$ mvn verify | gee -o verify.out -m 'After foobar edit'

Automating github

I got tired of downloading my own scripts from Github when working among multiple projects. So I automated it, of course. The bitsh project reuses a test script from the shell project, and now the Makefile for bitsh is simply:

SHELL=/bin/bash

test:
	@[ -t 1 ] && flags=-c ; \
	./test-bitsh.sh -i -- $$flags t

When run-tests is updated in Github, bitsh automatically picks up the changes. And I learned the value of ETag.

By the way, why "bitsh"? I hunted around for project names combining "git" and "bash" and found most of them already taken. Beggars can't be choosers.

UPDATE: I found the <TAB> character got munged by Blogger to a single space. This is unfortunate as you cannot copy out a valid Makefile. One solution is to put in an HTML tab entity.

Metaprogramming with Bash

Most programmers do not take full advantage of the languages they work in, though some languages make this a real challenge. Take metaprogramming, or programs that have some self-knowledge. LISP-family languages make this easy and natural; those with macros even more so. Bytecode languages (think Java), and even more so object code languages (think "C"), fall back on extra-linguistic magic such as AOP rewriting.

Text-based languages lay in a middle ground. Best known is Bash. Rarely do programmers take full advantage of Bash features, and few would think of metaprogramming. Not as clean as LISP macros, it is still straight forward.

As an example of function rewriting note line 21. The surrounding function body redefines an existing function, incorporating the original's body into itself. This is very similar to aspect-oriented programming with a "around" advice. Much care is taken to preserve the original function context.

This is a fully working script—give it a try!

#!/bin/bash

export PS4='+${BASH_SOURCE}:${LINENO}: ${FUNCNAME[0]:+${FUNCNAME[0]}(): } '

pgreen=$(printf "\e[32m")
pred=$(printf "\e[31m")
pboldred=$(printf "\e[31;1m")
preset=$(printf "\e[0m")
pcheckmark=$(printf "\xE2\x9C\x93")
pballotx=$(printf "\xE2\x9C\x97")
pinterrobang=$(printf "\xE2\x80\xBD")

function _register {
    case $# in
    1 ) local -r name=$1 arity=0 ;;
    2 ) local -r name=$1 arity=$2 ;;
    esac
    read -d '' -r wrapper <<EOF
function $name {
    # Original function
    $(declare -f $name | sed '1,2d;$d')

    __e=\$?
    shift $arity
    if (( 0 < __e || 0 == \$# ))
    then
        __tally \$__e
    else
        "\$@"
        __e=\$?
        __tally \$__e
    fi
}
EOF
    eval "$wrapper"
}

let __passed=0 __failed=0 __errored=0
function __tally {
    local -r __e=$1
    $__tallied && return $__e
    __tallied=true
    case $__e in
    0 ) let ++__passed ;;
    1 ) let ++__failed ;;
    * ) let ++__errored ;;
    esac
    _print_result $__e
    return $__e
}

function _print_result {
    local -r __e=$1
    case $__e in
    0 ) echo -e $pgreen$pcheckmark$preset $_scenario_name ;;
    1 ) echo -e $pred$pballotx$preset $_scenario_name ;;
    * ) echo -e $pboldred$pinterrobang$preset $_scenario_name ;;
    esac
}

function check_exit {
    (( $? == $1 ))
}

function make_exit {
    local -r e=$1
    shift
    (exit $e)
    "$@"
}

function check_d {
    [[ $PWD == $1 ]]
}

function change_d {
    cd $1
}

function variadic {
    :
}

function early_return {
    return $1
}

function eq {
    [[ "$bob" == nancy ]]
}

function normal_return {
    (exit $1)
}

function f {
    local -r bob=nancy
}

function AND {
    "$@"
}

function SCENARIO {
    local -r _scenario_name="$1"
    shift
    local __tallied=false
    local __e=0
    pushd $PWD >/dev/null
    "$@"
    __tally $?
    popd >/dev/null
}

_register f
_register normal_return 1
_register variadic 1
_register eq 
_register early_return 1
_register change_d 1
_register check_exit 1

echo "Expect 10 passes, 4 failures and 4 errors:"
SCENARIO "Normal return pass direct" normal_return 0
SCENARIO "Normal return fail direct" normal_return 1
SCENARIO "Normal return error direct" normal_return 2
SCENARIO "Normal return pass indirect" f AND normal_return 0
SCENARIO "Normal return fail indirect" f AND normal_return 1
SCENARIO "Normal return error indirect" f AND normal_return 2
SCENARIO "Early return pass indirect" f AND early_return 0
SCENARIO "Early return fail indirect" f AND early_return 1
SCENARIO "Early return error indirect" f AND early_return 2
SCENARIO "Early return pass direct" early_return 0
SCENARIO "Early return fail direct" early_return 1
SCENARIO "Early return error direct" early_return 2
SCENARIO "Variadic with none" f AND variadic
SCENARIO "Variadic with one" f AND variadic apple
SCENARIO "Local vars" f AND eq
here=$PWD
SCENARIO "Change directory" change_d /tmp
SCENARIO "Check directory" check_d $here
SCENARIO "Check exit" make_exit 1 AND check_exit 1

(( 0 == __passed )) || ppassed=$pgreen
(( 0 == __failed )) || pfailed=$pred
(( 0 == __errored )) || perrored=$pboldred
cat <<EOS
$ppassed$__passed PASSED$preset, $pfailed$__failed FAILED$preset, $perrored$__errored ERRORED$preset
EOS

Practical use of this script as a test framework would pull out SCENARIO and AND to a separate source script, included with "." or source, put the registered functions and their helpers in another source script, and provide command-line parsing to pick out which tests to execute. test-bitsh.sh is an example in progress.

Color your world

My coworkers use many ad hoc or single-purpose scripts, things like: checking system status, wrappers for build systems, launching services locally, etc. My UNIX background tells me, "keep it simple, avoid output; Silence is Golden."

Somehow my younger colleagues aren't impressed.

So to avoid acting my age I started sprinkling color into my scripts, and it worked. Feedback was uniformly positive. And true to my UNIX roots, I provided command line flags to disable color.

Some lessons for budding BASHers:

1. Yes, experiment and learn, but be sure to do your research. The Internet has outstanding help for BASH.
2. Learn standard idioms (see below).
3. Don't overdo it. Color for summary lines and warnings have more impact when the rest of the text is plain.
4. Keep functions small, just as you would in other languages. BASH is a programming language with a command line, so keep your good habits when writing shell.
5. Collaborate, pair! This comes naturally to my fellows. Coding is more enjoyable, goes faster and has fewer bugs.

Idioms

Most of these idioms appear in testing with bash, a simple BASH BDD test framework I wrote for demonstration.

Process command-line flags

this=false
that=true
while getopts :htT-: opt
do
    [[ - == $opt ]] && opt="${OPTARG%%=*}" OPTARG="${OPTARG#*=}"
    case $opt in
    h | help ) print_help ; exit 0 ;;
    t | this ) this=true ;;
    T | no-that ) that=false ;;
    esac
done
shift $((OPTIND - 1))

Keep boolean toggles simple

run_faster=true

if $run_faster
then
    use_faster_algorithm "$@"
else
    use_more_correct_algorithm "$@"
fi

Simple coloring

pgreen="\e[32m"
pred="\e[31m"
preset="\e0m"

if $success
then
    echo -e "${pgreen}PASS${preset} $test_name"
else
    echo -e "${pred}FAIL${preset} $test_name - $failure_reason"
EOM
fi

Consistent exit codes

function check_it {
    local -r failed=$1
    local -r syntax_error=$2
    if $syntax_error
    then
        return 2
    elif $failed
    then
        return 1
    else
        return 0
    fi
}

Coda

There are many more idioms to learn, hopefully this taste catches your interest. I was careful to include others mixed in with these (what does local -r do?) to whet the appetite for research. Go try the BASH debugger sometime.

UPDATE: Fixed thinko. ANSI escape codes need to be handled by echo -e or printf, not sent directly via cat!

BDD-style fluent testing in BASH

I wanted to impress on my colleagues that BASH was still hip, still relevant. And that I wasn't an old hacker. So I wrote a small BDD test framework for BASH.

Techniques

Fluent coding relies on several BASH features:

• Variable expansion happens before executing commands
• A shell function is indistinguishable from a program: they are called the same way
• Local function variables are dynamically scoped but only within a function, so are visible to other functions called within that scope, directly or indirectly through further function calls

Together with Builder pattern, it's easy to write given/when/then tests. (Builder pattern here solves the problem not of telescoping constructors, but massive, arbitrary argument lists.)

So when you run:

function c {
    echo "$message"
}

function b {
    "$@"
}

function a {
    local message="$1"
    shift
    "$@"
}

a "Bob's your uncle" b c

You see the output:

Bob's your uncle

How does this work?

First BASH expands variables. In function a this means that after the first argument is remembered and removed from the argument list, "$@" expands to b c. Then b c is executed.

Then BASH calls the function b with argument "c". Similarly b expands "$@" to c and calls it.

Finally as $message is visible in functions called by a, c prints the first argument passed to a (as it was remebered in the variable $message), or "Bob's your uncle" in this example.

Running the snippet with xtrace makes this clear (assuming the snippet is saved in a file named example):

bash -x example
+ a 'Bob'\''s your uncle' b c
+ local 'message=Bob'\''s your uncle'
+ shift
+ b c
+ c
+ echo 'Bob'\''s your uncle'
Bob's your uncle

So the test functions for given_jar, when_run and then_expect (along with other, similar functions) work the same way. Keep this in mind.

Application

So how does this buy me fluent BDD?

Given these functions:

function then_expect {
    local expectation="$1"
    shift

    if [[ some_test "$pre_condition" "$condition" "$expectation" ]]
    then
        echo "PASS: $scenario"
        return 0
    else
        echo "FAIL: $scenario"
        return 1
    fi
}

function when {
    local condition="$1"
    shift
    "$@"
}

function given {
    local pre_condition="$1"
    shift
    "$@"
}

function scenario {
    local scenario="$1"
    shift
    "$@"
}

When you write:

scenario "Some test case" \
    given "Something always true" \
    when "Something you want to test" \
    then_expect "Some outcome"

Then it executes:

some_test "Something always true" "Something you want to test" "Some outcome"

And prints one of:

PASS Some test case
FAIL Some test case

A real example at https://github.com/binkley/shell/tree/master/testing-bash.

Bash long options

UPDATED: Long options with arguments in the "name=value" style. The original post neglected this important case.

For years I've never know quite the right way to handle long options in Bash without significant, ugly coding. The usual sources (Advanced Bash-Scripting Guide, The Bash Hackers Wiki, others) are not much help. An occasional glimpse appears on StackOverflow, but not well explained or voted.

Solution

Working with a colleague yesterday, we found this:

name=Bob
while getopts :hn:-: opt
do
    [[ - == $opt ]] && opt=${OPTARG%%=*} OPTARG=${OPTARG#*=}
    case $opt in
    h | help ) print_help ; exit 0 ;;
    n | name ) name=$OPTARG ;;
    * ) print_usage >&2 ; exit 2 ;;
    esac
done
shift $((OPTIND - 1))
echo "$0: $name"

$ ./try-me -h
Usage: ./try-me [-h|--help][-n|--name <name>]
$ ./try-me --help
Usage: ./try-me [-h|--help][-n|--name <name>]
$ ./try-me -n Fred
./try-me: Fred
$ ./try-me --name=Fred
./try-me: Fred

Magic!

I checked with bash 3.2 and 4.3. At least for these, the '-' option argument has a bit of magic when it takes an argument. When the argument to '-' starts with a dash, as in --help (here "-help" is the argument to the '-' option), getopts drops the argument's leading '-', and OPTARG is just the text ("help" in this example). Only '-' has this magic.

Add a quick check for '-' at the top of the while-loop, and the case-block is simple and clear.

Bob's your uncle.

UPDATE: Followup on Bash long options.

RESTful helper script

I find this script useful working on modern RESTful services. It shows both the headers and the formatted JSON response body. The idea is that most times you provide a URL and want to see the full response. If you need extra flags for curl just add them (e.g., user/password). If you want to customize jq—say, filter for just a particular piece of the response—use a double-dash ("--") to separate curl and jq arguments:

An example with Spring Boot (plus some custom actuator endpoints). Note "jq" colorizes the output on the command line (below is plain text):

$ ~/bin/jurlq http://localhost:8081/remote-hello/health
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Strict-Transport-Security: max-age=31536000 ; includeSubDomains
X-Application-Context: remote-hello:8081
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Date: Mon, 21 Dec 2015 13:28:07 GMT

{
  "status": "UP",
  "cpu": {
    "status": "UP",
    "processors": 8,
    "system-loadavg": -1,
    "process-cpu-load": 0.22963892677420872,
    "process-cpu-time": "PT42.71875S",
    "system-cpu-load": -1
  },
  "file": {
    "status": "UP",
    "usable-disk": 55486464000,
    "total-disk": 299694551040
  },
  "java": {
    "status": "UP",
    "start-time": "2015-12-21T07:27:06.970-06:00",
    "uptime-beats": 0,
    "vm-name": "Java HotSpot(TM) 64-Bit Server VM",
    "vm-vendor": "Oracle Corporation",
    "vm-version": "25.66-b17"
  },
  "memory": {
    "status": "UP",
    "committed-virtual-memory": 991350784,
    "free-physical-memory": 1952854016,
    "free-swap-space": 1203003392,
    "total-physical-memory": 8540618752,
    "total-swap-space": 10354229248
  },
  "os": {
    "status": "UP",
    "arch": "amd64",
    "name": "Windows 10",
    "version": "10.0"
  },
  "threads": {
    "status": "UP",
    "count": 22,
    "daemon-count": 20,
    "peak-count": 22,
    "started-count": 26
  },
  "diskSpace": {
    "status": "UP",
    "free": 55486464000,
    "threshold": 10485760
  },
  "configServer": {
    "status": "UNKNOWN",
    "error": "no property sources located"
  },
  "hystrix": {
    "status": "UP"
  }
}

UPDATE: Tried Github Gist for the source, but it does not show in my blog feed reader. Here's the script:

#!/bin/bash

curl_args=()
for arg
do
    case "$arg" in
    -- ) shift ; break ;;
    * ) curl_args=("${curl_args[@]}" "$arg") ; shift ;;
    esac
done

jq_args=("${@-.}")

curl -s -D - "${curl_args[@]}" | tr -d '\r' | {
    while read line
    do
        case "$line" in
        '' ) echo ; break ;;
        * ) echo "$line" ;;
        esac
    done

    exec jq "${jq_args[@]}"
}

Census data with Bash

For fun I played with the 2012 US census data on metropolitan areas. I wanted to answer a simple questions, which "big" cities are growing the fastest?

First a note on file format. If you are splitting this line on comma, you want 3 fields:

apple,"banana,split",coconut

But it is easy to get 4 fields instead. My quick fix was to change comma to colon for field separators with this bit of python in a file named "csv-comma-to-colon.py":

import csv
import fileinput

for row in csv.reader(fileinput.input()):
    print(':'.join(row))

Simple, effective. An alternative I did not explore enough was csvprintf.

Once I have colon field separators, the rest writes itself. Seeing the CSV file from the census has non-data header and footer lines, I filter and sort:

sed '/^[^1-9]/d' CBSA-EST2012-05.csv \
    | python ./csv-comma-to-colon.py \
    | sort -t: -k4 -nr \
    | head -50 \
    | sort -t: -k6 -nr \
    | head -10 \
    | cut -d: -f2,4,6

Breaking it down:

1. Skip non-data lines from the CSV file.
2. Convert comma field separators to colon.
3. Reorder the data by city size, descending.
4. Keep only the top 50 biggest cities.
5. Reorder the remaining data by growth rate.
6. Keep only the 10 fastest growing cities.
7. Display only these columns: city name, current population, growth rate.

There may be better ways, this was enough for some quick fun. I would be remiss not to show the results nicely formatted for the web:

Metropolitan Area	Population	Growth
Austin-Round Rock, TX	1,834,303	3.0
Raleigh, NC	1,188,564	2.2
Orlando-Kissimmee-Sanford, FL	2,223,674	2.2
Houston-The Woodlands-Sugar Land, TX	6,177,035	2.1
Dallas-Fort Worth-Arlington, TX	6,700,991	2.0
San Antonio-New Braunfels, TX	2,234,003	1.9
Phoenix-Mesa-Scottsdale, AZ	4,329,534	1.8
Denver-Aurora-Lakewood, CO	2,645,209	1.8
Nashville-Davidson--Murfreesboro--Franklin, TN	1,726,693	1.7
Las Vegas-Henderson-Paradise, NV	2,000,759	1.7

Krumins does Bash one-liners

Beginning of a series on Bash one-liners from Peter Krumins. If it's from Peter, you know it'll be good.

Find broken symlinks

Helpful sysadmin tip from Jason White (jasonjgw), philosopher cum Linux expert, on finding broken symlinks.

readlink for better scripting

Fritz Thomas solves a common shell scripting problem for me: finding where your script lives. This script saved as ~/bin/x:

#!/bin/bash
echo "$0"
readlink -f "$0"

With ~/bin in PATH produces:

$ cd ~/bin; ./x
./x
/home/where/the/heart/is/bin/x
$ cd; x
/home/where/the/heart/is/bin/x
/home/where/the/heart/is/bin/x

Just what I need! (Yes, quite an odd home directory.)

Do note the caveats for readlink(1).

Shell gotcha: when comments execute

The problem: a colleague has a script which dies early, something like this (with the -x flag):

+ [ 2 -eq 2 ]
+ break
+ exit

The corresponding shell (KSH, but same results in BASH):

if [ $cond1 -eq $cond2 ]
then
   break
fi

# Lots of commented out code

more commands ...

The curiosity: why the early exit? The logic does not have an exit anywhere nearby; the commands following do not call exit.

Follow Sherlock Holmes' dictum, How often have I said to you that when you have eliminated the impossible, whatever remains, however improbable, must be the truth? (The Sign of the Four) The only thing left is the commented out code.

Sure enough:

# Blah, blah
# Something with $(hidden command) buried inside
# More blahbage

I had completely forgotten that the shell expands variables even inside comments! In this case, the expanded $(hidden command) blew up and caused the shell to exit prematurely. Caveat plicator.

Speeding up Cygwin login

On Windows I use Cygwin extensively. So I open a lot of login shells. However these shells are sometimes slow to load, especially when my host is busy with other jobs.

While editing my ~/.bash_profile I noticed code like this:

if [ -d "some path element" ]; then
    PATH="some path element:$PATH"
fi

This is a common idiom. However, it is making a separate fork to /bin/test every time.

A simple improvement:

if [[ -d "some path element" ]]; then
    PATH="some path element:$PATH"
fi

A little trying it out and the verdict: Wow, big improvement.

Cygwin BASH wrapper for Maven deploy:deploy-file

I frequently use Maven's deploy:deploy-file mojo for building project-local repositories.

Many interesting 3^rd-party Java libraries are not (yet) available from public Maven repositories (the otherwise excellent dev.java.net projects are particularly weak in this respect), but I want to use them in my project POM. What to do?

I store them with my project. This is not an ideal solution, but it is practical.

To help me deploying jars, sources and javadocs I wrote a simple BASH wrapper script for Cygwin (to run under Linux/UNIX requires a trivial change). Here is typical use:

$ mvn-file-deploy ~/IdeaProjects/Scratch/lib cool-library-1.0.4.jar org.cool.library:cool-library:1.0.4
$ mvn-file-deploy -S ~/IdeaProjects/Scratch/lib cool-library-1.0.4-sources.jar org.cool.library:cool-library:1.0.4
$ mvn-file-deploy -J ~/IdeaProjects/Scratch/lib cool-library-1.0.4-javadocs.jar org.cool.library:cool-library:1.0.4

And the script itself:

#!/bin/bash

function help()
{
    cat <<'EOH' | fmt
Usage: mvn-file-deploy [options] REPO_PATH FILE_PATH DESCRIPTOR [CLASSIFIER]

Use mvn-file-deploy as a helper for 'mvn deploy:deploy-file' when using Cygwin.

EOH
    cat <<'EOH'
Options:
  -h, --help          Print this help message
      --version       Print program version
  -S, --sources       Deploy sources
  -J, --javadocs      Deploy javadocs
  -q, --quiet         Do not print maven command
EOH
    cat <<'EOH' | fmt

Other options are passed to Maven.  (BROKEN)

Examples:

The most common use is to run it like this:

    $ mvn-file-deploy C:/my/project/lib C:/some/interesting.jar groupId:artifactId:version

Or if you use a classifier (e.g., jdk5):

But sometimes like this.

    $ mvn-file-deploy C:/my/project/lib C:/some/interesting.jar groupId:artifactId:version classifier

Report bugs to <binkley@alumni.rice.edu>.
EOH
}

function version()
{
    cat <<'EOV' | fmt
mvn-file-deploy 1.0
Copyright (C) 2007 JPMorganChase, Inc.  All rights reserved.
EOV
}

function fatal()
{
    echo "$0: $*" >&2
    exit 2
}

function usage() 
{
    cat <<'EOU' | fmt >&2
usage: mvn-file-deploy [options] REPO_PATH FILE_PATH DESCRIPTOR [CLASSIFIER]
EOU
}

eval set -- "$(getopt -o hJSq --long help,javadocs,sources,quiet,version -n "$0" -- "$@")"

declare -a maven_opts
declare -a args

for opt
do
    shift
    case "$opt" in
        -h|--help ) help ; exit 0 ;;
        -J|--javadocs ) classifier=-Dclassifier=javadocs ;;
        -S|--sources ) classifier=-Dclassifier=sources ;;
        --version ) version ; exit 0 ;;
        -q|--quiet ) quiet=t ; maven_opts=("${maven_opts[@]}" "$opt") ;;
        -- ) break ;;
        -* ) maven_opts=("${maven_opts[@]}" "$opt") ;;
    esac
done

case $# in
    3 ) repo_path="$1"
        file_path="$2"
        descriptor="$3" ;;
    4 ) repo_path="$1"
        file_path="$2"
        descriptor="$3"
        classifier=-Dclassifier="$4" ;;
    * ) usage ; exit 2 ;;
esac

(cd "$repo_path" 2>/dev/null) || \
    fatal "No such repository path: $repo_path"

repo_url="file://$(cygpath -am "$repo_path")"
file_path="$(cygpath -am "$file_path")"

[[ -r "$file_path" ]] || \
    fatal "No such artifact: $file_path"

triplet=($(echo $descriptor | tr : ' '))

(( 3 == ${#triplet[*]} )) || \
    fatal "Descriptor not groupId:artifactId:version format: $descriptor"

packaging=-Dpackaging="${file_path##*.}"

[[ -n "$quiet" ]] ||
    echo mvn "${maven_opts[@]}" deploy:deploy-file -Durl="$repo_url" \
        -DrepositoryId=project -Dfile="$file_path" -DgroupId=${triplet[0]} \
        -DartifactId=${triplet[1]} -Dversion=${triplet[2]} \
        $packaging $classifier
exec mvn "${maven_opts[@]}" deploy:deploy-file -Durl="$repo_url" \
    -DrepositoryId=project -Dfile="$file_path" -DgroupId=${triplet[0]} \
    -DartifactId=${triplet[1]} -Dversion=${triplet[2]} \
    $packaging $classifier

Download web pages in BASH

BASH is quite amazing. From Bhaskar V. Karambelkar comes Bash shell tricks and this gem (with some small corrections):

function headers()
{
    server=$1
    port=${2:-80}
    exec 5<>/dev/tcp/$server/$port
    echo -ne "HEAD / HTTP/1.0\r\nHost: $server:$port\r\n\r\n" >&5
    cat <&5
    exec 5<&-;
}

I work behind a corporate firewall, so I need web proxy settings. No problem, BASH is still amazing:

function webget()
{
    declare -a parts
    parts=($(echo $1 | tr / ' '))
    protocol=${parts[0]}
    server=${parts[1]}
    path=$(echo $1 | sed "s,$protocol//$server,,")

    exec 5<>/dev/tcp/$http_proxy_server/$http_proxy_port
    echo -ne "GET $path HTTP/1.0\r\nHost: $server\r\n\r\n" >&5
    cat <&5
    exec 5<&-;
}

Usage is obvious:

$ webget http://www.ccil.org/jargon/
# Out pops the top page for Jargon File Resources

UPDATE: Also useful for talking to SMTP (email) servers:

$ exec 5<>/dev/tcp/localhost/smtp
$ read -u 5 line
$ echo $line
220 my.full.host.name ESMTP ...
$ echo QUIT >&5
$ read -u 5 line
$ echo $line
221 2.0.0 my.full.host.name closing connection

This also shows that BASH knows to map the SMTP service to port 25.

Getting more from streams in Bash

A handy trick in Bash is to split a stream to process it in more than one way without saving a temporary copy:

# Make 3 a copy of 1 (stdout)
exec 3>&1
result="$(command to generate stream \
    | tee /dev/fd/3 \
    | command to process stream)"
# Now work with $result
# The original stream also went to the console

The actual case I worked with this morning was a very simple demonstration:

#!/bin/bash

exec 3>&1

echo -e "Bob is my friend.\nFred is my friend, too." \
    | tee /dev/fd/3 \
    | cut -f1 -d' ' >&2

Which simply prints:

Bob is my friend.
Fred is my friend.

to stdout and to stderr it prints:

Bob
Fred