core/src/main/java/com/twitter/elephantbird/util/HdfsUtils.java
public static long getDirectorySize(Path path, FileSystem fs, PathFilter filter)
      throws IOException {
    PathSizeVisitor visitor = new PathSizeVisitor();
    PathFilter composite = new PathFilters.CompositePathFilter(
      PathFilters.newExcludeDirectoriesFilter(fs.getConf()),
      filter);
    walkPath(path, fs, composite, visitor);
    return visitor.getSize();
  }
public static long getDirectorySize(Path path, FileSystem fs, PathFilter filter)
      throws IOException {
    PathSizeVisitor visitor = new PathSizeVisitor();
    PathFilter composite = new PathFilters.CompositePathFilter(
      PathFilters.newExcludeDirectoriesFilter(fs.getConf()),
      filter);
    walkPath(path, fs, composite, visitor);
    return visitor.getSize();
  }
Calculates the total size of all the contents of a directory that are accepted
by filter. All subdirectories will be searched recursively and paths in subdirectories
that are accepted by filter will also be counted.

Does not include the size of directories themselves
(which are 0 in HDFS but may not be 0 on local file systems)

To get the size of a directory without filtering, use
{@link #getDirectorySize(Path, FileSystem)} which is much more efficient.

@param path path to recursively walk
@param fs FileSystem for this path
@param filter path filter for which paths size's to include in the total
NOTE: you do *not* need to filter out directories, this will be done for you
@return size of the directory in bytes
@throws IOException