Tuesday, August 30, 2005

Deleting things with Ant

One thing about Ant I have noticed is that file operations are slow and recursive operations are especially so. Consider this toy Ant script to delete a directory in the background while carrying on in the foreground:

<project name="parallel-delete" default="all">
    <property name="old.dir" value="build"/>
    <available file="${old.dir}" property="old.dir.exists"/>

    <target name="all" depends="rename-old-dir">
        <parallel>
            <apply executable="rm">
                <arg value="-rf"/>
                <dirset dir="." includes="${old.dir}.*"/>
            </apply>

            <sequential>
                <!-- Real work goes here -->
                <sleep seconds="1"/>
                <echo message="Snoozing..."/>
            </sequential>
        </parallel>
    </target>

    <target name="rename-old-dir" if="old.dir.exists">
        <tempfile property="tmp.dir" prefix="${old.dir}."/>
        <!-- Slow as all get out.
        <move todir="${tmp.dir}">
            <fileset dir="${old.dir}"/>
        </move> -->
        <!-- Deprecated, but works better than move.
        <rename src="${old.dir}" dest="${tmp.dir}"/> -->
        <!-- Fastest of all but only works on UNIX/Cygwin.
        <exec executable="mv">
            <arg value="${old.dir}"/>
            <arg value="${tmp.dir}"/>
        </exec>
    </target>
</project>

Notice the commented sections in the rename-old-dir target? Each of them has drawbacks. For my testing I setup with:

$ cp -a /usr/include build

This gave me a large directory tree to test with. You can decide from the comments which approach is least worst for you.

The actual technique—deleting in the background—is an interesting one and shaves considerable time off a large project wanting a clean rebuild. There is also the funny business with <apply/> also owing to Ant problems with file operations on a large, recursive tree. I cannot nest a <dirset/> element inside a <delete/> task (the <dirset/> type is, unfortunately, rather less useful than <fileset/>), but I need to delete directories, not files, and the name of the directory is a pattern. (I may have left over temporary directories to delete from previous, aborted runs.)

This solution is workable and fast, but with the drawback of targeting UNIX and Cygwin. Windows-only users get the short end of the development stick again.

My project is sitting on Ant 1.6.2; perhaps 1.6.3 addresses some of these. The syntax for file operations on directories is improving (e.g., <move file="src/dir" tofile="new/dir/to/move/to"/>) so some of my concerns may be already addressed.

More on the <parallel/> trick

My sample Ant script runs the cleanup in the background as foreground work continues. If the foreground work finishes, the script continues and the background completes. An alternative is to surround the background work with <daemons/>; then if the foreground finishes the script exits leaving the background work incomplete. For a cleanup task this isn't a terrible choice and one could have a vacuum process or task for removing detritus from previous incomplete background work with the benefit of having the Ant script finish faster.

UPDATE: Food for thought: here is a similar build script with make instead of Ant:

build.dir = build

all:
        : Do your work here
        sleep 1
        echo Snoozing...

rebuild: clean all

clean:
        test -d $(build.dir) && mv $(build.dir) $(build.dir).$$$$ || true
        ($(RM) -rf $(build.dir).* &)

Monday, August 29, 2005

Nice Ant 1.6.3 improvement for default values

Apache Ant 1.6.3 added a nice feature to the condition task. There is now an else attribute. The niceness is best illustrated by example:

<property environment="env"/>

<!-- The old, pre-1.6.3 way:-->
<condition property="foo.bar" value="${env.FOO_BAR}">
    <isset property="env.FOO_BAR"/>
</condition>
<condition property="foo.bar" value="Lives somewhere else">
    <not>
        <isset property="env.FOO_BAR"/>
    </not>
</condition>

And with Ant 1.6.3:

<property environment="env"/>

<!-- The new, 1.6.3 way:-->
<condition property="foo.bar" value="${env.FOO_BAR}"
        else="Lives somewhere else">
    <isset property="env.FOO_BAR"/>
</condition>

It could be more concise still, but this is an improvement.

Saturday, August 20, 2005

A home page

I enjoy posting code snippets but sometimes a full project is more helpful. On my home page are several links to downloads for several of my more interesting (to me) posts. Cheers!

Friday, August 19, 2005

GNU autotools lessons learned

Along the way to autoconfiscating portions of our Windows to Linux port for my company's desktop softare, I discovered the autoreconf gem. It handles most of what I had stuffed into a autogen.sh script.

One nit. I have some custom macros in an m4/ subdirectory which add support for --enable-debug, --enable-profile and --enable-release flags to ./configure. (Why aren't these standard, or at least the macros standard?) autoreconf supports an -I m4 option to pass these to autoconf and autoheader, but not to aclocal.

Drat!

However, thanks to GNU Automake by Example, I found that I can put a ACLOCAL_AMFLAGS = -I m4 line in the top-level Makefile.am to pass -I m4 to aclocal. This is an unfortunate code duplication, but better than simply having the feature broken.

I also discovered autoupdate which brought my configure.ac file up to current standards. Nifty.

Lastly, I saw that autoreconf is actually quite clever. If I have never run ./configure, the --make option does nothing as it does not know how I wish to configure the project (the install directory, for example). However once I have run ./configure, autoreconf reuses the settings from that first run for subsequent runs of ./configure and then dashes off with make afterwards.

Thursday, August 11, 2005

Teaching configure about build flags

Cobbled together from several sources around the Internet, I put together a solution this morning to a question posed by one of our developers: how do you make debug v release builds with GNU autotools?

A first pass at answering produced the ac_build_types.m4 macro for autoconf. First, some usage. Here is configure.ac:

AC_PREREQ(2.59)
AC_INIT([my_project], [0.0.0], [binkley@alumni.rice.edu])
AC_CONFIG_SRCDIR([config.h.in])
AC_CONFIG_HEADER([config.h])
AM_INIT_AUTOMAKE

AC_BUILD_TYPES

dnl Rest of file...

The only thing different from a standard configure.ac is the addition of AC_BUILD_TYPES. The effect of that shows in ./configure:

$ ./configure --help
# ...
Optional Features:
  --disable-FEATURE       do not include FEATURE (same as --enable-FEATURE=no)
  --enable-FEATURE[=ARG]  include FEATURE [ARG=yes]
  --enable-debug          build debug version
  --enable-profile        build profiling version
  --enable-release        build release version
# ...

Now there are flags to build debug, profiling and release builds.

Last is the macro itself:

C_DEFUN([AC_BUILD_TYPES],
[
AC_ARG_ENABLE(debug,
      [  --enable-debug          build debug version],
      CFLAGS="$CFLAGS -DDEBUG -O0 -g3 -Wall"
      CXXFLAGS="$CXXFLAGS -DDEBUG -O0 -g3 -Wall")
AC_ARG_ENABLE(gprof,
      [  --enable-profile        build profiling version],
      CFLAGS="$CFLAGS -pg"
      CXXFLAGS="$CXXFLAGS -pg"
      LDFLAGS="$LDFLAGS -pg")
AC_ARG_ENABLE(release,
      [  --enable-release        build release version],
      CFLAGS="$CFLAGS -DNDEBUG -g0 -O3"
      CXXFLAGS="$CXXFLAGS -DNDEBUG -g0 -O3")
])

That's all! Save the definition into something like ac_build_types.m4 and run aclocal -Idirectory-containing-macros as part of creating your ./configure for other developers.

Wednesday, August 10, 2005

Almost there

As part of exploring a Windows to Linux port, I looked into generating Windows DLLs with automake and libtool. After some experimenting, I have almost everything I want in an example library, Foo:

  1. Builds static and shared libraries for Linux
  2. Builds static and shared libraries for Windows
  3. Windows libraries can depend on Cygwin
  4. Windows libraries can be entirely independent of Cygwin

Here's how I build for that last case:

$ ./autogen.sh
$ CC="cc -mno-cywin" CXX="cc -mno-cygwin" LTCC="cc" ./configure --prefix=/tmp/foo
$ make install

This installs everything under /tmp/foo (where I can easily clean up between test runs).

Why LTCC="cc"? It turns out that libtool writes a temporary wrapper for the main and just this temporary wrapper needs the full Cygwin environment. Libtool provides LTCC for building the wrapper independently of CC used for everything else. Without the extra setting, everything actually works but produces spurious errors during building. (If you want an explanation of -mno-cygwin, see the manpage for GCC under Cygwin.)

I posted this under the title Almost there. There is one more detail to work out. Even though the DLL has no Cygwin depedencies, it is still named cygfoo-0-0-0.dll rather than foo.dll as expected. Everythings works correctly, but this is annoying. You cannot rename the DLL either as this breaks loading for programs linked against it. When I figure this last bit out, I'll post an update to this entry.

Runtime traces for C

While porting some software from Windows to Linux, I needed to see backtraces. If there is no global exception handler and an exception in C++ is not caught, it aborts the program. The reports I got went along the lines of some program output:

$ run_foo
Aborted.

Not very helpful!

So I wrote a small pair of trace macros based on the GNU C Library's backtrace facility. Although this facility is only available for UNIX and Linux platforms, my tracing macros still are helpful under Windows sans backtracing.

#ifndef TRACE_H_
#define TRACE_H_

#include <stdio.h>
#include <stdlib.h>

#define TRACE_BACKTRACE_FRAMES 10

#ifdef __GNUC__
# define __FUNCTION__ __PRETTY_FUNCTION__
#endif

/* Emacs-style output */
#ifdef EMACS
# define TRACE_PREFIX   fprintf (stderr, "%s:%d:%s", __FILE__, __LINE__, __FUNCTION__)
#else
# define TRACE_PREFIX   fprintf (stderr, "%s(%d):%s", __FILE__, __LINE__, __FUNCTION__)
#endif /* EMACS */

#ifdef linux
# include <execinfo.h>

# define TRACE_DUMP()   do {     void *array[TRACE_BACKTRACE_FRAMES];     int n = backtrace (array, sizeof (array) / sizeof (void *));     char **symbols = backtrace_symbols (array, n);     int i;      for (i = 0; i < n; ++i)       fprintf (stderr, " -[%d]%s\n", i, symbols[i]);      free (symbols);   } while (0)
#else
# define TRACE_DUMP()
#endif /* linux */

#define TRACE()   TRACE_PREFIX; fprintf (stderr, "\n");   TRACE_DUMP ()

#define TRACE_MSG(MSG)   TRACE_PREFIX; fprintf (stderr, ": %s\n", MSG);   TRACE_DUMP ()

#endif /* TRACE_H_ */

Use the EMACS define to switch between Emacs-style and regular line tracing.

UPDATE: I meant to provide sample output:

trace.c(9):bob: Some interesting debug message.
 -[0]./trace(bob+0x51) [0x804878d]
 -[1]./trace(main+0x21) [0x8048817]
 -[2]/lib/tls/libc.so.6(__libc_start_main+0xd0) [0x4021de80]
 -[3]./trace [0x80486a1]

One thing jumps out immediately: this is not Java. But one can tease out that main called bob and bob wrote a trace message. Notice this:

trace.c(9):bob: Some interesting debug message.
 -[0]./trace(main+0x51) [0x8048841]
 -[1]/lib/tls/libc.so.6(__libc_start_main+0xd0) [0x4021de80]
 -[2]./trace [0x80486a1]

That is the same output with -O3 passed to GCC. The compiler inlines away the trivial bob function. -O2 did not inline the function.