<commit_before>msgp
=======

This is a code generation tool for serializing Go `struct`s using the [MesssagePack](http://msgpack.org) standard. It is targeted 
at the `go generate` [tool](http://tip.golang.org/cmd/go/#hdr-Generate_Go_files_by_processing_source).

### Use

*** Note: `go generate` is a go 1.4+ feature. ***

In a source file, include the following directive:

```go
//go:generate msgp
```

The `msgp` command will generate `Unmarshal`, `Marshal`, `WriteTo`, and `ReadFrom` methods for all exported struct
definitions in the file. You will need to include that directive in every file that contains structs that 
need code generation. The generated files will be named {filename}_gen.go by default (but can 
be overridden with th `-o` flag.)

Field names can be overridden in much the same way as the `encoding/json` package. For example:

```go
type Person struct {
	Name       string `msg:"name"`
	Address    string `msg:"address"`
	Age        int    `msg:"age"`
	Hidden     string `msg:"-"` // this field is ignored
	unexported bool             // this field is also ignored
}
```

Running `msgp` on a file containing the above declaration would generate the following methods:

```go
func (z *Person) Marshal() ([]byte, error)

func (z *Person) Unmarshal(b []byte) error

func (z *Person) WriteTo(w io.Writer) (n int, err error)

func (z *Person) ReadFrom(r io.Reader) (n int, err error)
```

The `msgp/enc` package has a function called `CopyToJSON` which can take MessagePack-encoded binary
and translate it directly into JSON. It has reasonably high performance, and works much the same way that `io.Copy` does.


### Features

 - Proper handling of pointers, nullables, and recursive types
 - Backwards-compatible decoding logic
 - Efficient and readable generated code
 - JSON interoperability
 - Support for embedded fields (see Benchmark 1) and inline comma-separated declarations (see Benchmark 2)
 - Support for Go's `complex64` and `complex128` types
 - Generation of both `[]byte`-oriented and `io.Reader/io.Writer`-oriented methods.


### Status

Very alpha. Here are the known limitations:

 - The only currently supported map types are `map[string]string` and `map[string]interface{}`.
 - All fields of a struct that are not Go built-ins are assumed to satisfy the `io.WriterTo` and `io.ReaderFrom`
   interface. This will only *actually* be the case if the declaration for that type is in a file touched by the code generator.
 - Like most serializers, `chan` and `func` fields are ignored, as well as non-exported fields.

We will (soon) offer support for `map[string]T`, where `T` is any other supported type. We do *not* plan on 
supporting maps keyed on non-string fields, as it would severly limit JSON interoperability.

### Performance

As you might imagine, the generated code is quite a lot more performant than reflection-based serialization. Here 
are the two built-in benchmarking cases provided. Each benchmark writes the struct to a buffer and then extracts 
it again.


##### Benchmark 1

Type specification:
```go
type TestType struct {
	F   *float64          `msg:"float"`
	Els map[string]string `msg:"elements"`
	Obj struct {
		ValueA string `msg:"value_a"`
		ValueB []byte `msg:"value_b"`
	} `msg:"object"`
	Child *TestType `msg:"child"`
}
```

|  Method | Time | Heap Use | Heap Allocs |
|:-------:|:----:|:--------:|:-----------:|
| msgp codegen | 2553ns | 129 B | 5 allocs |
| [ugorji/go](http://github.com/ugorji/go) | 8467ns | 2015 B | 43 allocs |
| encoding/json | 11976ns | 2004 B | 33 allocs |



##### Benchmark 2

Type specification:
```go
type TestFast struct {
	Lat, Long, Alt float64
	Data []byte
}
```
|  Method | Time | Heap Use | Heap Allocs |
|:-------:|:----:|:--------:|:-----------:|
| msgp codegen | 1151ns | 57 B | 1 allocs |
| [ugorji/go](http://github.com/ugorji/go) | 5199ns | 1227 B | 27 allocs |
| encoding/json | 8120ns | 1457 B | 11 allocs |
<commit_msg>todo list<commit_after>msgp
=======

This is a code generation tool for serializing Go `struct`s using the [MesssagePack](http://msgpack.org) standard. It is targeted 
at the `go generate` [tool](http://tip.golang.org/cmd/go/#hdr-Generate_Go_files_by_processing_source).

### Use

*** Note: `go generate` is a go 1.4+ feature. ***

In a source file, include the following directive:

```go
//go:generate msgp
```

The `msgp` command will generate `Unmarshal`, `Marshal`, `WriteTo`, and `ReadFrom` methods for all exported struct
definitions in the file. You will need to include that directive in every file that contains structs that 
need code generation. The generated files will be named {filename}_gen.go by default (but can 
be overridden with th `-o` flag.)

Field names can be overridden in much the same way as the `encoding/json` package. For example:

```go
type Person struct {
	Name       string `msg:"name"`
	Address    string `msg:"address"`
	Age        int    `msg:"age"`
	Hidden     string `msg:"-"` // this field is ignored
	unexported bool             // this field is also ignored
}
```

Running `msgp` on a file containing the above declaration would generate the following methods:

```go
func (z *Person) Marshal() ([]byte, error)

func (z *Person) Unmarshal(b []byte) error

func (z *Person) WriteTo(w io.Writer) (n int, err error)

func (z *Person) ReadFrom(r io.Reader) (n int, err error)
```

The `msgp/enc` package has a function called `CopyToJSON` which can take MessagePack-encoded binary
and translate it directly into JSON. It has reasonably high performance, and works much the same way that `io.Copy` does.


### Features

 - Proper handling of pointers, nullables, and recursive types
 - Backwards-compatible decoding logic
 - Efficient and readable generated code
 - JSON interoperability
 - Support for embedded fields (see Benchmark 1) and inline comma-separated declarations (see Benchmark 2)
 - Support for Go's `complex64` and `complex128` types
 - Generation of both `[]byte`-oriented and `io.Reader/io.Writer`-oriented methods.


### Status

Very alpha. Here are the known limitations:

 - The only currently supported map types are `map[string]string` and `map[string]interface{}`.
 - All fields of a struct that are not Go built-ins are assumed to satisfy the `io.WriterTo` and `io.ReaderFrom`
   interface. This will only *actually* be the case if the declaration for that type is in a file touched by the code generator.
 - Like most serializers, `chan` and `func` fields are ignored, as well as non-exported fields.

We will (soon) offer support for `map[string]T`, where `T` is any other supported type. We do *not* plan on 
supporting maps keyed on non-string fields, as it would severly limit JSON interoperability.

### Performance

As you might imagine, the generated code is quite a lot more performant than reflection-based serialization. Here 
are the two built-in benchmarking cases provided. Each benchmark writes the struct to a buffer and then extracts 
it again.


##### Benchmark 1

Type specification:
```go
type TestType struct {
	F   *float64          `msg:"float"`
	Els map[string]string `msg:"elements"`
	Obj struct {
		ValueA string `msg:"value_a"`
		ValueB []byte `msg:"value_b"`
	} `msg:"object"`
	Child *TestType `msg:"child"`
}
```

|  Method | Time | Heap Use | Heap Allocs |
|:-------:|:----:|:--------:|:-----------:|
| msgp codegen | 2553ns | 129 B | 5 allocs |
| [ugorji/go](http://github.com/ugorji/go) | 8467ns | 2015 B | 43 allocs |
| encoding/json | 11976ns | 2004 B | 33 allocs |



##### Benchmark 2

Type specification:
```go
type TestFast struct {
	Lat, Long, Alt float64
	Data []byte
}
```
|  Method | Time | Heap Use | Heap Allocs |
|:-------:|:----:|:--------:|:-----------:|
| msgp codegen | 1151ns | 57 B | 1 allocs |
| [ugorji/go](http://github.com/ugorji/go) | 5199ns | 1227 B | 27 allocs |
| encoding/json | 8120ns | 1457 B | 11 allocs |


### TODO

 - Support JSON method generation. The front-end for the existing generator will work fine; we just need new templates.
 - Support non-struct types.
 - Support finer-grained directive control (e.g. read string as []byte and vice-versa), probably via struct tags
 - Support `Seek`ing internal fields in the encoded binary

If you've found a bug, please open an issue! Thanks.

{'fraction_non_alphanumeric': 0.12428831011508176, 'fraction_numerical': 0.01720169594185342, 'mean_word_length': 3.7177142857142855, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2417118', 'n_tokens_mistral': 2666, 'n_tokens_neox': 2436, 'n_words': 1050}