<commit_before>+++
author = ["<NAME>"]
date = "2015-12-12T00:05:27-05:00"
linktitle = "SHIELD Backup System"
series = ["Advent 2015"]
title = "shield backup system"

+++

# Go in SHIELD

## Quick background: What is SHIELD?

Like many software projects, SHIELD was conceived out of need. In particular, we had a
client who deployed their own Cloud Foundry with BOSH and they needed regular backups of
Cloud Foundry itself and its services, such as Redis, PostgreSQL, Docker ... you get the idea.
For the interested, [here](http://gnuconsulting.com/blog/2014/09/07/intro-to-cloud-foundry-and-bosh/)
is a quick summary of the basics of BOSH and Cloud Foundry.

//FIXME - trim the next paragraph down more.

Since our client was using vSphere we initially considered/tried using vSphere's
snapshot abilities, but ultimately switched to a custom solution four a couple reasons.
One was that BOSH doesn't work well with traditional snapshots because of the way it
resurrects failed VMs. Since the IDs for VMs, persistent disk, etc. will change restoring
a snapshot with now-outdated IDs led to issues like the persistent disk going missing
because the restored snapshot didn't have the correct ID. The other reason was a change
in the client requirements. In addition to wanting to be able to do broader restores in the
event of meteor strikes, they also wanted to be able to do more granular restores for
day to day activities. For example, if operations wanted to restore specific
containers or credentials or if developers wanted to restore specific databases after
tinkering. Since this new requirement lent itself to a custom solution, we rolled
both requirements into SHIELD's design.

## Functions in Structs
Something we employed both in the CLI and in the server-side code was the ability
to validate the received data to help keep out both non-sense and malicious values. A
streamlined example of the validation functions from our CLI form looks like this:

```go
package main

import "fmt"

type FieldValidator func(name string, value int) error

type Form struct {
	Fields []*Field
}

type Field struct {
	Name      string
	Value     int
	Validator FieldValidator
}

func NewForm() *Form {
	f := Form{}
	return &f
}

func (f *Form) NewField(name string, value int, fcn FieldValidator) error {
	f.Fields = append(f.Fields, &Field{Name: name, Value: value, Validator: fcn})
	return nil
}

func InputIsNotBigEnough(name string, value int) error {
	if value < 3600 {
		return fmt.Errorf("input value must be greater than 3600")
	}
	return nil
}

func InputIsNotSmallEnough(name string, value int) error {
	if value > 3600 {
		return fmt.Errorf("input value must be less than 3600")
	}
	return nil
}

func (f *Form) Show() error {
	for _, field := range f.Fields {
		err := field.Validator(field.Name, field.Value)
		if err == nil {
			err = fmt.Errorf("value is valid")
		}
		fmt.Printf("name: '%s', value: '%d', validate: '%v'\n", field.Name, field.Value, err)
	}
	return nil
}

func main() {
	in := NewForm()
	in.NewField("number", 3599, InputIsNotBigEnough)
	in.NewField("another number", 3601, InputIsNotSmallEnough)
	in.NewField("last number", 2500, func(n string, v int) error {
		if v != 2500 {
			return fmt.Errorf("value is not 2500")
		}
		return nil
	})
	in.Show()
}
```

<small>[Click here](http://play.golang.org/p/GhW7dL-pCv) to test on the Go Playground.</small>

The `FieldValidator` type allows for any function as long as it takes in a `string` and `int` and outputs
an `error`. This allows us to pass different named functions, or even anonymous functions, as an argument.
Since the function is stored in the struct, all three functions can just be called using dot notation just
like the other elements of the struct: `field.Validator` (ln 43).

### Interfaces

Take a quick look at the following code:

```go
package main

import (
	"encoding/json"
	"fmt"
)

type JSONMessage struct {
	Message []string `json:"message"`
}

func (m JSONMessage) JSON() string {
	b, err := json.Marshal(m)
	if err != nil {
		return fmt.Sprintf(`{"error":"failed to unmarshal JSON: %s"}`, err)
	}
	return string(b)
}

type JSONDate struct {
	Holiday string `json:"holiday"`
	Date    int    `json:"date"`
}

func (d JSONDate) JSON() string {
	b, err := json.Marshal(d)
	if err != nil {
		return fmt.Sprintf(`{"error":"failed to unmarshal JSON: %s"}`, err)
	}
	return string(b)
}

type ToJSON interface {
	JSON() string
}

func JSONify(j ToJSON) string {
	return fmt.Sprintf("%s", j.JSON())
}

func main() {
	message := JSONMessage{[]string{"Merry Christmas", "Happy New Year"}}
	hdate := JSONDate{"First Day of Hanukkah", 20151206}
	fmt.Printf("%s\n", JSONify(message))
	fmt.Printf("%s\n", JSONify(hdate))
}
```

<small>[Click here](http://play.golang.org/p/SaZV522EBc) to test on the Go Playground.</small>

In SHIELD we used an interface for JSON payloads similar to the above, but for error handling.
Specifically we used a single function for writing the JSON error the client calling the API
endpoint. Tying this into what we did above with validation functions in structs, this
provided a consistent way to pass JSON error payloads since Go does not support function
overloading.
<commit_msg>Testing with embedded Go Playground<commit_after>+++
author = ["<NAME>"]
date = "2015-12-12T00:05:27-05:00"
linktitle = "SHIELD Backup System"
series = ["Advent 2015"]
title = "shield backup system"

+++

# Go in SHIELD

## Quick background: What is SHIELD?

Like many software projects, SHIELD was conceived out of need. In particular, we had a
client who deployed their own Cloud Foundry with BOSH and they needed regular backups of
Cloud Foundry itself and its services, such as Redis, PostgreSQL, Docker ... you get the idea.
For the interested, [here](http://gnuconsulting.com/blog/2014/09/07/intro-to-cloud-foundry-and-bosh/)
is a quick summary of the basics of BOSH and Cloud Foundry.

//FIXME - trim the next paragraph down more.

Since our client was using vSphere we initially considered/tried using vSphere's
snapshot abilities, but ultimately switched to a custom solution four a couple reasons.
One was that BOSH doesn't work well with traditional snapshots because of the way it
resurrects failed VMs. Since the IDs for VMs, persistent disk, etc. will change restoring
a snapshot with now-outdated IDs led to issues like the persistent disk going missing
because the restored snapshot didn't have the correct ID. The other reason was a change
in the client requirements. In addition to wanting to be able to do broader restores in the
event of meteor strikes, they also wanted to be able to do more granular restores for
day to day activities. For example, if operations wanted to restore specific
containers or credentials or if developers wanted to restore specific databases after
tinkering. Since this new requirement lent itself to a custom solution, we rolled
both requirements into SHIELD's design.

## Functions in Structs
Something we employed both in the CLI and in the server-side code was the ability
to validate the received data to help keep out both non-sense and malicious values. A
streamlined example of the validation functions from our CLI form looks like this:

```go
package main

import "fmt"

type FieldValidator func(name string, value int) error

type Form struct {
	Fields []*Field
}

type Field struct {
	Name      string
	Value     int
	Validator FieldValidator
}

func NewForm() *Form {
	f := Form{}
	return &f
}

func (f *Form) NewField(name string, value int, fcn FieldValidator) error {
	f.Fields = append(f.Fields, &Field{Name: name, Value: value, Validator: fcn})
	return nil
}

func InputIsNotBigEnough(name string, value int) error {
	if value < 3600 {
		return fmt.Errorf("input value must be greater than 3600")
	}
	return nil
}

func InputIsNotSmallEnough(name string, value int) error {
	if value > 3600 {
		return fmt.Errorf("input value must be less than 3600")
	}
	return nil
}

func (f *Form) Show() error {
	for _, field := range f.Fields {
		err := field.Validator(field.Name, field.Value)
		if err == nil {
			err = fmt.Errorf("value is valid")
		}
		fmt.Printf("name: '%s', value: '%d', validate: '%v'\n", field.Name, field.Value, err)
	}
	return nil
}

func main() {
	in := NewForm()
	in.NewField("number", 3599, InputIsNotBigEnough)
	in.NewField("another number", 3601, InputIsNotSmallEnough)
	in.NewField("last number", 2500, func(n string, v int) error {
		if v != 2500 {
			return fmt.Errorf("value is not 2500")
		}
		return nil
	})
	in.Show()
}
```

<small>[Click here](http://play.golang.org/p/GhW7dL-pCv) to test on the Go Playground.</small>

<iframe src="http://play.golang.org/p/GhW7dL-pCv" frameborder="0" style="width: 100%; height: 100%"><a href="http://play.golang.org/p/GhW7dL-pCv">see this code in play.golang.org</a></iframe>

The `FieldValidator` type allows for any function as long as it takes in a `string` and `int` and outputs
an `error`. This allows us to pass different named functions, or even anonymous functions, as an argument.
Since the function is stored in the struct, all three functions can just be called using dot notation just
like the other elements of the struct: `field.Validator` (ln 43).

### Interfaces

Take a quick look at the following code:

```go
package main

import (
	"encoding/json"
	"fmt"
)

type JSONMessage struct {
	Message []string `json:"message"`
}

func (m JSONMessage) JSON() string {
	b, err := json.Marshal(m)
	if err != nil {
		return fmt.Sprintf(`{"error":"failed to unmarshal JSON: %s"}`, err)
	}
	return string(b)
}

type JSONDate struct {
	Holiday string `json:"holiday"`
	Date    int    `json:"date"`
}

func (d JSONDate) JSON() string {
	b, err := json.Marshal(d)
	if err != nil {
		return fmt.Sprintf(`{"error":"failed to unmarshal JSON: %s"}`, err)
	}
	return string(b)
}

type ToJSON interface {
	JSON() string
}

func JSONify(j ToJSON) string {
	return fmt.Sprintf("%s", j.JSON())
}

func main() {
	message := JSONMessage{[]string{"Merry Christmas", "Happy New Year"}}
	hdate := JSONDate{"First Day of Hanukkah", 20151206}
	fmt.Printf("%s\n", JSONify(message))
	fmt.Printf("%s\n", JSONify(hdate))
}
```

<small>[Click here](http://play.golang.org/p/SaZV522EBc) to test on the Go Playground.</small>

<iframe src="http://play.golang.org/p/SaZV522EBc" frameborder="0" style="width: 100%; height: 100%"><a href="http://play.golang.org/p/SaZV522EBc">see this code in play.golang.org</a></iframe>

In SHIELD we used an interface for JSON payloads similar to the above, but for error handling.
Specifically we used a single function for writing the JSON error the client calling the API
endpoint. Tying this into what we did above with validation functions in structs, this
provided a consistent way to pass JSON error payloads since Go does not support function
overloading.

{'fraction_non_alphanumeric': 0.10194264569842738, 'fraction_numerical': 0.016836262719703978, 'mean_word_length': 4.338765432098765, 'pattern_counts': {'":': 4, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4812858', 'n_tokens_mistral': 3520, 'n_tokens_neox': 3263, 'n_words': 1408}