<commit_before>
# nubis-docker-builder

Docker image for building Nubis AMIs

[Install docker](https://docs.docker.com/engine/installation/linux/ubuntu/)


### Getting image and using image

```bash
IMAGE_VERSION=$(curl -k -s -S "https://registry.hub.docker.com/v1/repositories/nubisproject/nubis-builder/tags" | jq --raw-output '.[]["name"]' | sort --field-separator=. --numeric-sort --reverse | grep -m 1 "^v")

docker pull nubisproject/nubis-builder:${IMAGE_VERSION}

ACCOUNT='<account-to-build-in>'

aws-vault exec ${ACCOUNT} -- docker run -u $UID:$(id -g) -it --env-file ~/.docker_env -e GIT_COMMIT_SHA=$(git rev-parse HEAD) -v $PWD:/nubis/data nubisproject/nubis-builder:${IMAGE_VERSION}

aws-vault exec ${ACCOUNT} -- docker run -u $UID:$(id -g) -it --env-file ~/.docker_env -e GIT_COMMIT_SHA=$(git rev-parse HEAD) -v $PWD:/nubis/data nubisproject/nubis-builder:${IMAGE_VERSION} --build-region us-east-1 --copy-regions 'ap-northeast-1,ap-northeast-2,ap-southeast-1,ap-southeast-2,eu-central-1,eu-west-1,sa-east-1,us-east-1,us-west-1,us-west-2' build
```

### Building image locally
You should never need to do this since we have automated builds, but if you want to build locally this is how you do it

```bash

cd <Docker file folder location>
docker build -t nubis-builder .
<commit_msg>Fix mdl checks for nubis/docker/README.md<commit_after># nubis-docker-builder

Docker image for building Nubis AMIs

[Install docker](https://docs.docker.com/engine/installation/linux/ubuntu/)

## Getting image and using image

```bash
DOCKER_URL='https://registry.hub.docker.com/v1/repositories/nubisproject/nubis-builder/tags'
IMAGE_VERSION=$(curl -k -s -S "${DOCKER_URL}" \
                | jq --raw-output '.[]["name"]' \
                | sort --field-separator=. --numeric-sort --reverse \
                | grep -m 1 "^v")

docker pull nubisproject/nubis-builder:${IMAGE_VERSION}

ACCOUNT='<account-to-build-in>'

aws-vault exec ${ACCOUNT} -- docker run \
                            -u $UID:$(id -g) -it \
                            --env-file ~/.docker_env \
                            -e GIT_COMMIT_SHA=$(git rev-parse HEAD) \
                            -v $PWD:/nubis/data \
                            nubisproject/nubis-builder:${IMAGE_VERSION}

COPY_REGIONS='ap-northeast-1,ap-northeast-2,ap-southeast-1,ap-southeast-2,eu-central-1,eu-west-1,sa-east-1,us-east-1,us-west-1,us-west-2'
aws-vault exec ${ACCOUNT} -- docker run \
                            -u $UID:$(id -g) -it \
                            --env-file ~/.docker_env \
                            -e GIT_COMMIT_SHA=$(git rev-parse HEAD) \
                            -v $PWD:/nubis/data \
                            nubisproject/nubis-builder:${IMAGE_VERSION} \
                            --build-region us-east-1 \
                            --copy-regions ${COPY_REGIONS} \
                            build
```

## Building image locally

You should never need to do this since we have automated builds, but if you want
to build locally this is how you do it

```bash

cd <Docker file folder location>
docker build -t nubis-builder .

{'fraction_non_alphanumeric': 0.14863563402889246, 'fraction_numerical': 0.008346709470304976, 'mean_word_length': 2.2834562697576395, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4855425', 'n_tokens_mistral': 1116, 'n_tokens_neox': 1046, 'n_words': 236}