Message Digest

MD5 throughput was ~15 GBytes/minute on a quad core Intel i5 @ 2.67GHz running Ubuntu Linux 10.04 and Java(TM) SE Runtime Environment (build 1.6.0_22-b04) Java HotSpot(TM) Server VM (build 17.1-b03, mixed mode)

The throughput was the same whether the digest was computed from bytes on disk or memory. org.apache.commons.codec.digest.DigestUtils org.apache.commons.codec.binary.Hex

From file

      String filename = "src/test/resources/somefile";
      is = new FileInputStream(filename);
      assertEquals("bd6b1dc1ac767b018003572a767d2d0d", DigestUtils.md5Hex(is));

Pipe filter stream

      is = new DigestInputStream( new BufferedInputStream(new FileInputStream(filename)), MessageDigest.getInstance("MD5"));

  /*
   * Implementations using DigestInputStream would not need the byte handling as it would be just one filter in a pipe
   * that is being pumped for some other purpose.
   */
      final byte[] buf = new byte[2048];
      while (dis.read(buf) == -1)
      {
      }

      final MessageDigest d = dis.getMessageDigest();
      assertEquals("bd6b1dc1ac767b018003572a767d2d0d", Hex.encodeHexString(d.digest()));