<commit_before>HDFS for Go
===========

[![build](https://travis-ci.org/colinmarc/hdfs.svg?branch=master)](https://travis-ci.org/colinmarc/hdfs)

`hdfs` is a native go client for hdfs, using the protocol buffers interface the
namenode provides. It implements protocol version 9, which means it supports
Hadoop 2.0.0 and up (including CDH5).

It tries to be idiomatic by aping the stdlib `os` package where possible. This
includes implementing `os.FileInfo` for file status, and returning
`os.ErrNotExist` for missing files, for example.

The best place to get started is the
[Godoc](https://godoc.org/github.com/colinmarc/hdfs).

```go

client, _ := hdfs.New("namenode:8020")

file, _ := client.Open("/mobydick.txt")

buf := make([]byte, 59)
file.ReadAt(buf, 48847)

fmt.Println(string(buf))
// => Abominable are the tumblers into which he pours his poison.
```

The `hdfs` Binary
-----------------

The library also ships with a command line client for hdfs. Like the library,
its primary aim is to be idiomatic, by enabling your favorite unix verbs:

```
$ hdfs --help
Usage: ./hdfs COMMAND
The flags available are a subset of the POSIX ones, but should behave similarly.

Valid commands:
  ls [-la] [FILE]...
  rm [-r] FILE...
  mv [-fT] SOURCE... DEST
  mkdir [-p] FILE...
  touch [-amc] FILE...
  chmod [-R] OCTAL-MODE FILE...
  chown [-R] OWNER[:GROUP] FILE...
  cat SOURCE...
  head [-n LINES | -c BYTES] SOURCE...
  tail [-n LINES | -c BYTES] SOURCE...
  get SOURCE [DEST]
  getmerge SOURCE DEST

```

It's also pretty fast compared to `hadoop -fs`, and, best of all, comes with
bash tab completion!

You can install it with `go get`, or by cloning and running `make install`
(alternatively, running `make` will create a binary in the source directory, so
you can install it where you want). To enable tab completion, source
`cmd/hdfs/bash_completion`, or drop it into the bash completion directory for
platform (on linux, this is usually `/etc/bash_completion.d`).

Acknowledgements
----------------

This library is heavily indebted to [snakebite](https://github.com/spotify/snakebite).<commit_msg>readme cleanup<commit_after>HDFS for Go
===========

[![build](https://travis-ci.org/colinmarc/hdfs.svg?branch=master)](https://travis-ci.org/colinmarc/hdfs)

`hdfs` is a native go client for hdfs, using the protocol buffers interface the
namenode provides. It implements protocol version 9, which means it supports
Hadoop 2.0.0 and up (including CDH5).

It tries to be idiomatic by aping the stdlib `os` package where possible. This
includes implementing `os.FileInfo` for file status, and returning
`os.ErrNotExist` for missing files, for example.

The best place to get started is the
[Godoc][1].

```go

client, _ := hdfs.New("namenode:8020")

file, _ := client.Open("/mobydick.txt")

buf := make([]byte, 59)
file.ReadAt(buf, 48847)

fmt.Println(string(buf))
// => Abominable are the tumblers into which he pours his poison.
```

The `hdfs` Binary
-----------------

The library also ships with a command line client for hdfs. Like the library,
its primary aim is to be idiomatic, by enabling your favorite unix verbs:

```
$ hdfs --help
Usage: ./hdfs COMMAND
The flags available are a subset of the POSIX ones, but should behave similarly.

Valid commands:
  ls [-la] [FILE]...
  rm [-r] FILE...
  mv [-fT] SOURCE... DEST
  mkdir [-p] FILE...
  touch [-amc] FILE...
  chmod [-R] OCTAL-MODE FILE...
  chown [-R] OWNER[:GROUP] FILE...
  cat SOURCE...
  head [-n LINES | -c BYTES] SOURCE...
  tail [-n LINES | -c BYTES] SOURCE...
  get SOURCE [DEST]
  getmerge SOURCE DEST

```

It's also pretty fast compared to `hadoop -fs`, and, best of all, comes with
bash tab completion!

You can install it with `go get github.com/colinmarc/hdfs`, or by cloning and
running `make install`. Alternatively, running `make` will create a binary in
the source directory, so you can install it where you want.

To enable tab completion, source `cmd/hdfs/bash_completion`, or drop it into the
bash completion directory for your platform (on linux, this is usually
`/etc/bash_completion.d`).

Acknowledgements
----------------

This library is heavily indebted to [snakebite][2].

[1]: https://godoc.org/github.com/colinmarc/hdfs
[2]: https://github.com/spotify/snakebite
{'fraction_non_alphanumeric': 0.13793103448275862, 'fraction_numerical': 0.00844475721323012, 'mean_word_length': 3.58494623655914, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5148885', 'n_tokens_mistral': 1456, 'n_tokens_neox': 1363, 'n_words': 542}