Friday, August 30, 2013

Quick tour of Java 8

Bienvenido David III provides a quick tour of Java 8 in Everything About Java 8.

Rebooting Jigsaw redux

Alex Blewitt has interesting observations on the next reboot of Jigsaw, the project to modularize the JDK, in Jigsaw, Second Cut. Amusing bit:

There has been a module system available for Java from the early days, known as JSR 8 and more recently as OSGi and JSR 291, that is used by all major Java EE application servers, and many other systems and applications in wide use today (JIRA, Eclipse) as well as specific vertical markets (embedded systems, home automation). Jigsaw follows in the footsteps of design successes such as java.util.Date, Java IO NIO NIO2 NIO2.2 and the java.logging package, and aims to ignore best practices to developer an inferior system for use by the JDK itself.

Ultimately he hopes for OSGi-lite.

Thursday, August 29, 2013

Netflix contributes RxJava

Somehow I missed this earlier: Functional Reactive in the Netflix API with RxJava by Ben Christensen. It begins:

Our recent post on optimizing the Netflix API introduced how our web service endpoints are implemented using a "functional reactive programming" (FRP) model for composition of asynchronous callbacks from our service layer.

This post takes a closer look at how and why we use the FRP model and introduces our open source project RxJava – a Java implementation of Rx (Reactive Extensions).

Open recursion

Excellent explanation of open recursion from Bob Nystrom.

Sunday, August 25, 2013

Providing failure policy with generics and annotations

Java generics with annotations provides a simple technique for specifying failure policies in an API. The exception in a throws clause may be a generic parameter. The @Nullable and @Nonnull annotations express intent.

In the example, Base takes an Exception as a generic parameter. Three implementations demonstrate the choices:

  1. ReturnsNull is for "finder" type APIs, where a null return is in good taste.
  2. ThrowsUnchecked is when failure is fatal, and should be handled higher in the stack.
  3. ThrowsChecked is when failure is transient, and can be handled by the caller (e.g., network failures).

Because the exception is defined at a class level, this technique is best for SAM-style APIs or when a common exception is shared by a few calls.

static void main(final String... args) {
    new ReturnsNull().returnSomething();
    System.out.println("Returned null");
    try {
        new ThrowsUnchecked().returnSomething();
    } catch (final RuntimeException ignored) {
        System.out.println("Threw unchecked");
    }
    try {
        new ThrowsChecked().returnSomething();
    } catch (final Exception ignored) {
        System.out.println("Threw checked");
    }
}

interface Base<E extends Exception> {
    Object returnSomething() throws E;
}

final class ReturnsNull implements Base<RuntimeException> {
    @Nullable @Override
    public String returnSomething() {
        return null;
    }
}

final class ThrowsUnchecked implements Base<RuntimeException> {
    @Nonnull @Override
    public String returnSomething() {
        throw new RuntimeException();
    }
}

final class ThrowsChecked implements Base<Exception> {
    @Nonnull @Override
    public String returnSomething() throws Exception {
        throw new Exception();
    }
}

Tuesday, July 30, 2013

Extending String in Java with enums

Without extension methods in Java 8 (or without changing JVM language) one cannot extend String. Or can one?

public final class ExtendStringWithEnum {
    public static void main(final String... args) {
        out.println(format("%s: %s", APPLE, toCrAzYcAsE(APPLE)));
        out.println(format("%s: %s", BANANA, BANANA.toCrAzYcAsE()));
    }

    public enum EnumConstants {
        APPLE("apple"), BANANA("banana");
        private final String value;

        EnumConstants(final String value) {
            this.value = value;
        }

        public String toCrAzYcAsE() {
            return ExtendStringWithEnum.toCrAzYcAsE(value);
        }

        @Override
        public String toString() {
            return value;
        }
    }

    public static final class StringConstants {
        public static final String APPLE = "apple";
        public static final String BANANA = "banana";
    }

    public static String toCrAzYcAsE(final String value) {
        final StringBuilder builder = new StringBuilder(value.length());
        for (int i = 0, x = value.length(); i != x; ++i)
            builder.append(
                    0 == i % 2 ? toUpperCase(value.charAt(i))
                            : toLowerCase(value.charAt(i)));
        return builder.toString();
    }
}

Given suitable static imports, the code in "main" for APPLE is how I write it using string constants; the code for BANANA is how I write it for enum constants wrapping strings, and how I would like to write it for strings as well using the static "toCrAzYcAsE" methods. Maybe Java 9.

Thursday, July 25, 2013

OOPs

John D. Cook notes in Too many objects the condition of OOP gone mad:

For example, I had a discussion with a colleague once on how to represent depth in an oil well for software we were writing. I said “Let’s just use a number.”

double depth;

My suggestion was laughed off as hopelessly crude. We need to create depth objects! And not just C++ objects, COM objects! We couldn’t send a double out into the world without first wrapping it in the overhead of an object.

Wednesday, July 17, 2013

Martin Thompson on GC

Another great post from Martin Thompson, Java Garbage Collection Distilled. In spite of the unassuming title, this is a must-read for advanced Java developers. Introduction:

Serial, Parallel, Concurrent, CMS, G1, Young Gen, New Gen, Old Gen, Perm Gen, Eden, Tenured, Survivor Spaces, Safepoints, and the hundreds of JVM startup flags. Does this all baffle you when trying to tune the garbage collector while trying to get the required throughput and latency from your Java application? If it does then do not worry, you are not alone. Documentation describing garbage collection feels like man pages for an aircraft. Every knob and dial is detailed and explained but nowhere can you find a guide on how to fly. This article will attempt to explain the tradeoffs when choosing and tuning garbage collection algorithms for a particular workload.

The focus will be on Oracle Hotspot JVM and OpenJDK collectors as those are the ones in most common usage. Towards the end other commercial JVMs will be discussed to illustrate alternatives.

I especially like his diagram for G1:

Enjoy!

Tuesday, July 09, 2013

Example RMI code

Syed Muhammad Humayun posts Java Remote Method Invocation (RMI) with Secure Socket Layer (SSL), a succinct, clear example of using SSL with vanilla RMI in Java. RMI never had good examples that I saw, Sayed's contribution goes into toolbox of tricks.

Monday, June 24, 2013

Thursday, May 30, 2013

Bloaty XMLBeans

Nikita Salnikov-Tarnovski writes Reducing memory consumption by 20x. One change, dropping XML beans, reduced footprint from 1.5G to 214M. It's not quite that simple, and there's more tuning for the headline 20x improvement, but this is the gist of it. Amazing.

Tuesday, May 28, 2013

For old time's sake, Uncle Bob

I forgot how good Uncle Bob can be. Thank you, The Old Reader: Transformation Priority and Sorting. Quoting:

In this post we explore the Transformation Priority Premise in the context of building a sort algorithm.

We also explore comic books as a pedagogical tool.

Want to read the rest? (In the interest of fair use you should follow the link.)

Sunday, May 26, 2013

Is it live, or is it structured deferral?

A delightful ACM paper by Paul E. McKenney, Structured Deferral: Synchronization via Procrastination. The "motivating example":

In this example, Schrödinger would like to construct an in-memory database to keep track of the animals in his zoo. Births would of course result in insertions into this database, while deaths would result in deletions. The database is also queried by those interested in the health and welfare of Schrödinger's animals.

Schrödinger has numerous short-lived animals such as mice, resulting in high update rates. In addition, there is a surprising level of interest in the health of Schrödinger's cat,19 so much so that Schrödinger sometimes wonders whether his mice are responsible for most of these queries. Regardless of their source, the database must handle the large volume of cat-related queries without suffering from excessive levels of contention. Both accesses and updates are typically quite short, involving accessing or mutating an in-memory data structure, and therefore synchronization overhead cannot be ignored.

Schrödinger also understands, however, that it is impossible to determine exactly when a given animal is born or dies. For example, suppose that his cat's passing is to be detected by heartbeat. Seconds or even minutes will be required to determine that the poor cat's heart has in fact stopped. The shorter the measurement interval, the less certain the measurement, so that a pair of veterinarians examining the cat might disagree on exactly when death occurred. For example, one might declare death 30 seconds after the last heartbeat, while another might insist on waiting a full minute, in which case the veterinarians disagree on the state of the cat during the second half of the minute after the last heartbeat.

Fortunately, Heisenberg8 has taught Schrödinger how to cope with such uncertainty. Furthermore, the delay in detecting the cat's passing permits use of synchronization via procrastination. After all, given that the two veterinarians' pronouncements of death were separated by a full 30 seconds, a few additional milliseconds of software procrastination is perfectly acceptable.

Among the pleasures in this paper: Linux's RCU, hazard pointers, and "read-only cat-heavy performance of Schrödinger's hash table".

Enjoy!

Friday, May 24, 2013

If it's broke, don't fix it

Saul Caganoff writes Netflix: Dystopia as a Service on Adrian Cockroft keynote describing Netflix's seemingly impossible infrastructure. The opening:

"As engineers we strive for perfection; to obtain perfect code running on perfect hardware, all perfectly operated." This is the opening from Adrian Cockroft, Cloud Architect at Netflix in his recent Keynote Address at Collaboration Summit 2013. Cockroft continues that the Utopia of perfection takes too long, that time to market trumps quality and so we compromise. But rather than lamenting the loss of "static, better, cheaper" perfection, Cockroft describes how the Netflix "Cloud Native" architecture embraces the Dystopia of "broken and inefficient" to deliver "sooner" and "dynamic". Cockroft says that "the new engineering challenge is not to construct perfection but to construct highly agile and highly available services from ephemeral and often broken components."

Were I looking for a new challenge, as interesting as Google is, I'd go to Netflix—unless I got to work on asteroid mining!


Do you read the Netflix Tech Blog — why not?

Thursday, May 23, 2013

Census data with Bash

For fun I played with the 2012 US census data on metropolitan areas. I wanted to answer a simple questions, which "big" cities are growing the fastest?

First a note on file format. If you are splitting this line on comma, you want 3 fields:

apple,"banana,split",coconut

But it is easy to get 4 fields instead. My quick fix was to change comma to colon for field separators with this bit of python in a file named "csv-comma-to-colon.py":

import csv
import fileinput

for row in csv.reader(fileinput.input()):
    print(':'.join(row))

Simple, effective. An alternative I did not explore enough was csvprintf.

Once I have colon field separators, the rest writes itself. Seeing the CSV file from the census has non-data header and footer lines, I filter and sort:

sed '/^[^1-9]/d' CBSA-EST2012-05.csv \
    | python ./csv-comma-to-colon.py \
    | sort -t: -k4 -nr \
    | head -50 \
    | sort -t: -k6 -nr \
    | head -10 \
    | cut -d: -f2,4,6

Breaking it down:

  1. Skip non-data lines from the CSV file.
  2. Convert comma field separators to colon.
  3. Reorder the data by city size, descending.
  4. Keep only the top 50 biggest cities.
  5. Reorder the remaining data by growth rate.
  6. Keep only the 10 fastest growing cities.
  7. Display only these columns: city name, current population, growth rate.

There may be better ways, this was enough for some quick fun. I would be remiss not to show the results nicely formatted for the web:

Metropolitan Area Population Growth
Austin-Round Rock, TX 1,834,303 3.0
Raleigh, NC 1,188,564 2.2
Orlando-Kissimmee-Sanford, FL 2,223,674 2.2
Houston-The Woodlands-Sugar Land, TX 6,177,035 2.1
Dallas-Fort Worth-Arlington, TX 6,700,991 2.0
San Antonio-New Braunfels, TX 2,234,003 1.9
Phoenix-Mesa-Scottsdale, AZ 4,329,534 1.8
Denver-Aurora-Lakewood, CO 2,645,209 1.8
Nashville-Davidson--Murfreesboro--Franklin, TN 1,726,693 1.7
Las Vegas-Henderson-Paradise, NV 2,000,759 1.7

Wednesday, May 22, 2013

Netflix delivers Java tools

Netflix continues as a Java technology powerhouse, delivering one open source tool or framework after another. The latest posting from their excellent blog is Brian Moore on Garbage Collection Visualization, a tool for turning gc.log into usable graphs.

The JVM heap teaser shot:

Sunday, May 19, 2013

Java for Fun - Shazam with Roy van Rijn

Roy van Rijn writes Creating Shazam in Java. This is way more fun than most Java posts:

If I record a song and look at it visually it looks like this:


(all the red dots are ‘important points’)

(I would hang that on my wall, wouldn't you?)

He has more fun code on GitHub.

Thursday, May 16, 2013

Escaping TFS

From 2011 Derek Hammer writes TFS is destroying your development capacity with advice on escaping TFS. How little has changed.

UPDATED: A delightful take from Prasanna Pendse, Top 10 Version Control Features of TFS.

Wednesday, May 15, 2013

Code imitates math

Jeff Preshing posted in 2010 High-Resolution Mandelbrot in Obfuscated Python, Python code to draw the Mandelbrot set (h/t Marcus Holtermann). Can you spot the relationship?

_                                      =   (
                                        255,
                                      lambda
                               V       ,B,c
                             :c   and Y(V*V+B,B,  c
                               -1)if(abs(V)<6)else
               (              2+c-4*abs(V)**-0.4)/i
                 )  ;v,      x=1500,1000;C=range(v*x
                  );import  struct;P=struct.pack;M,\
            j  ='<QIIHHHH',open('M.bmp','wb').write
for X in j('BM'+P(M,v*x*3+26,26,12,v,x,1,24))or C:
            i  ,Y=_;j(P('BBB',*(lambda T:(T*80+T**9
                  *i-950*T  **99,T*70-880*T**18+701*
                 T  **9     ,T*i**(1-T**45*2)))(sum(
               [              Y(0,(A%3/3.+X%v+(X/v+
                               A/3/3.-x/2)/1j)*2.5
                             /x   -2.7,i)**2 for  \
                               A       in C
                                      [:9]])
                                        /9)
                                       )   )

Tuesday, May 14, 2013

Praising the opponents with PostgreSQL

I just read an unusual article, PostgreSQL at a glance, a lengthy, favorable review of a competing product by Kim Sung Kyu, Senior Software Engineer at CUBRID. The opening:

PostgreSQL shows excellent functionalities and performance. Considering its high quality, it may seem strange that PostgreSQL is not more popular. However, PostgreSQL continues to make progress. This article will discuss this database.

This is a good review of PostgreSQL, on CUBRID's blog no less. To openly write about alternative products speaks well of your own.

Linux C10M

Great list, the key is separation of concerns: The Secret To 10 Million Concurrent Connections -The Kernel Is The Problem, Not The Solution.

Wednesday, May 08, 2013

More Zing for Java

From LMAX Exchange Getting Up To 50% Improvement in Latency From Azul's Zing JVM:

The results LMAX Exchange are seeing are remarkable: a 10-20% improvement in the mean latency, increasing to around a 50% improvement at the 99th percentile. Moreover

At the max/99.99th percentile with HotSpot the number would jump all over the place so it is hard to produce a relative comparison, except to say that the Zing values are much more stable. A bad run with HotSpot could easily be an order of magnitude worse.

In terms of throughput

Zing gives us the ability to lift what we call our "red-line" - the throughput value at which the latency starts to drop off a cliff. This effect often manifests as a second order effect of GC pauses. If we get a stall that is sufficiently long, we will start to drop packets. The process of packet redelivery can create back-pressure throughout the rest of the system sending client latencies through the roof. Having a more efficient collector with very short predictable pauses should allow us to increase our "red-line".

Most eye popping for me:

Whilst these figures are impressive, the variations, caused primarily by stop-the-world pauses in the CMS collector that is part of HotSpot, are becoming a significant problem. LMAX Exchange tried upgrading to the CMS version in JDK 7, but encountered around a 20% increase in the length of GC pauses for the same work load. The reasons for this weren't entirely clear, but Barker suggested it was probably down to a need to re-tune the collector. That Zing's collector (C4) typically requires little or no tuning was a major selling point for LMAX Exchange.

I think that we really needed to do retuning of our GC setting and investigating whether JDK 7 specific options like -XX:+UseCondCardMark and -XX:+UseNUMA should be applied. One of the other big reasons to go with Azul is the reduced need to tune the collector. The general recommendation is that you should re-tune from scratch on each new version of the JDK, which sounds fine in theory, but can be impractical. Collector tuning in Oracle JDK is essentially walking through a large search space for a result that meets your needs. Experience, knowledge and guess-work can crop significant chunks off that search space, but even then an extensive tuning exercise can take weeks. For example, our full end-to-end performance test takes 1 hour (10 minutes build & deploy, 10 minutes warm-up, 40 minutes testing), so I could reasonably run 8 runs a day. If you consider the number of different collectors (CMS, Parallel, Serial,...) and all of their associated options (new and old sizes, survivor spaces, survivor ratios,...) how many runs do I need to do to get effective coverage of that search space: 20, 30, more? With Zing the defaults work significantly better than a finely tuned Oracle JDK. We still have some investigation over whether we can get a bit more out of the Zing VM through tuning (e.g. fewer collector threads as our allocation rate is relatively low). However, tuning Zing is just that, i.e. looking to eke out the very best from the system; compared to the Oracle JDK where tuning from the defaults can be the difference between usable and unusable. The effort involving in tuning does come with an opportunity cost. I would much rather have the developers that would typically be involved with GC tuning (they are probably the ones that have the best working knowledge of software performance) be focusing on improving the performance of other areas of the system.

Concluding:

Part of the reason Zing is so attractive to these companies is that it remains the only collector that eliminates stop-the-world pauses from the young generation as well as the old generation. Whilst young generation pauses are shorter, where an application is particularly performance sensitive they still matter. As a result, Tene told us, "All we have to do is point to newgen pauses in other JVMs and say: 'those too will be gone'."

...Furthermore, the fact that it can handle multi-GB-per-sec allocations without worsening latencies or pauses, makes it very appealing for developers who have been trying hard not to allocate things because "it hurts". With Zing, you can use plenty of memory to go fast, instead of trying to avoid using it so that things won't hurt and jitter.

For production use Zing is priced on an annual subscription/server. Unsurprisingly the vendor is reluctant to reveal pricing information, though it is in line with a supported Oracle or IBM JVM.

The Future of Java

Kindly transcribed in Cliff Click, Charlie Hunt and Doug Lea on 'The Future of the JVM'. Juicy bit:

My View: worst feature was a synchronized StringBuffer. I have blogged about this a couple of times. I would add mutable Date objects.

Preaching to the choir, brother.