<commit_before># go-cdbmap

go-cdbmap is a pure [Go](http://golang.org/) package to read and write cdb ("constant database") files, forked from [jbarham's go-cdb library](https://github.com/jbarham/go-cdb/).

The cdb file format is a machine-independent format with the following features:

 - *Fast lookups:* A successful lookup in a large database normally takes just two disk accesses. An unsuccessful lookup takes only one.
 - *Low overhead:* A database uses 2048 bytes, plus 24 bytes per record, plus the space for keys and data.
 - *No random limits:* cdb can handle any database up to 4 gigabytes. There are no other restrictions; records don't even have to fit into memory.

See the original cdb specification and C implementation by <NAME>
at http://cr.yp.to/cdb.html.

## Installation

Assuming you have a working Go environment, installation is simply:

	go get github.com/clee/go-cdbmap

The package documentation can be viewed online at
http://gopkgdoc.appspot.com/pkg/github.com/clee/go-cdbmap
or on the command line by running `go doc github.com/clee/go-cdbmap`.

The usage is extremely simple; here is an example program that shows usage of the entire API.

```go
package main

import (
	"github.com/clee/go-cdbmap"
)

func main() {
	// Read a cdb-formatted file into a map[string][]string
	c, err := cdbmap.Open("example.cdb")
	m, err := c.Map()
	if err != nil {
		panic(err)
	}

	// Take a map[string][]string and turn it into a cdb file
	nc := cdbmap.NewFromMap(m)
	nc.Write("/tmp/test.cdb")
}

```

## Utilities

The go-cdbmap package includes ports of the programs `cdbdump` and `cdbmake` from
the [original implementation](http://cr.yp.to/cdb/cdbmake.html).<commit_msg>Add more detailed example code<commit_after># go-cdbmap

go-cdbmap is a pure [Go](http://golang.org/) package to read and write cdb ("constant database") files, forked from [jbarham's go-cdb library](https://github.com/jbarham/go-cdb/).

The cdb file format is a machine-independent format with the following features:

 - *Fast lookups:* A successful lookup in a large database normally takes just two disk accesses. An unsuccessful lookup takes only one.
 - *Low overhead:* A database uses 2048 bytes, plus 24 bytes per record, plus the space for keys and data.
 - *No random limits:* cdb can handle any database up to 4 gigabytes. There are no other restrictions; records don't even have to fit into memory.

See the original cdb specification and C implementation by <NAME>
at http://cr.yp.to/cdb.html.

## Installation

Assuming you have a working Go environment, installation is simply:

	go get github.com/clee/go-cdbmap

The package documentation can be viewed online at
http://gopkgdoc.appspot.com/pkg/github.com/clee/go-cdbmap
or on the command line by running `go doc github.com/clee/go-cdbmap`.

The usage is extremely simple; here is an example program that shows usage of the entire API.

```go
package main

import (
	"github.com/clee/go-cdbmap"
	"fmt"
)

func main() {
	// Read a cdb-formatted file into a map[string][]string
	c, err := cdbmap.Open("example.cdb")
	m, err := c.Map()
	if err != nil {
		panic(err)
	}
	// Now that we have a map, we can loop over the keys...
	for key, values := range m {
		fmt.Printf("key: %s [\n", key)

		// And all of the values, too
		for _, value := range values {
			fmt.Printf("\t%s\n", value)
		}
		fmt.Printf("]\n")
	}

	// Take a map[string][]string and turn it into a cdb file
	nc := cdbmap.NewFromMap(m)
	nc.Write("/tmp/test.cdb")
}
```

## Utilities

The go-cdbmap package includes ports of the programs `cdbdump` and `cdbmake` from
the [original implementation](http://cr.yp.to/cdb/cdbmake.html).
{'fraction_non_alphanumeric': 0.11208550287750069, 'fraction_numerical': 0.0038366675801589476, 'mean_word_length': 3.828042328042328, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3883304', 'n_tokens_mistral': 1221, 'n_tokens_neox': 1171, 'n_words': 464}