.drone
    README.md
    drone.yml
.github
    ISSUE_TEMPLATE
        BOUNTY.yml
    depcheck.yml
    dependabot.yml
    pull_request_template.md
    workflows
        bump-formula-pr.yml
        check_docs.yml
        depcheck.yml
        publish.yml
        scripts.yml
        stale.yml
        test.yml
        test_packages.yml
.golangci.yml
ADOPTERS.md
CHANGELOG.md
CODE_OF_CONDUCT.md
GOVERNANCE.md
MAINTAINERS.md
README.md
SECURITY.md
build-image
    README.md
cmd
    agent-operator
        DEVELOPERS.md
        README.md
        main.go
    agent
        entrypoint.go
        flow.go
        main.go
        service.go
        service_windows.go
        test-local-file.txt
    agentctl
        main.go
component
    all
        all.go
    common
        appendable
            appendable.go
        config
            types.go
        relabel
            relabel.go
    component.go
    component_health.go
    discovery
        discovery.go
        kubernetes
            k8s.go
    local
        file
            detector.go
            file.go
            file_test.go
    metrics
        globalrefmap.go
        globalrefmap_test.go
        mapping.go
        mutate
            mutate.go
        receiver.go
        receiver_test.go
        remotewrite
            remote_write.go
            types.go
        scrape
            scrape.go
            scrape_test.go
            types.go
    registry.go
    registry_test.go
    remote
        s3
            s3.go
            s3_test.go
            types.go
            watcher.go
    targets
        mutate
            mutate.go
            mutate_test.go
docs
    README.md
    developer
        contributing.md
        downstream-prometheus.md
        releasing.md
    flow
        _index.md
        concepts
            _index.md
            component_controller.md
            components.md
            configuration_language.md
        config-language
            _index.md
            components.md
            expressions
                _index.md
                function_calls.md
                operators.md
                referencing_outputs.md
                types_and_values.md
            files.md
            syntax.md
        getting_started.md
        monitoring
            _index.md
            component_metrics.md
            controller_metrics.md
            debugging.md
        reference
            _index.md
            cli
                _index.md
                run.md
            components
                _index.md
                local.file.md
                metrics.scrape.md
                remote.s3.md
                targets.mutate.md
            config-blocks
                _index.md
                logging.md
            flags.md
            stdlib
                _index.md
                concat.md
                env.md
                json_decode.md
        tutorials
            _index.md
            prometheus-metrics.md
            your-first-component.md
    rfcs
        0000-template.md
        0001-designing-in-the-open.md
        0002-integrations-in-operator.md
        0003-new-metrics-subsystem.md
        0004-agent-flow.md
    sources
        README.md
        _index.md
        api
            _index.md
        configuration
            _index.md
            create-config-file.md
            dynamic-config.md
            flags.md
            integrations
                _index.md
                apache-exporter-config.md
                cadvisor-config.md
                consul-exporter-config.md
                dnsmasq-exporter-config.md
                ebpf-config.md
                elasticsearch-exporter-config.md
                github-exporter-config.md
                integrations-next
                    _index.md
                    app-agent-receiver-config.md
                    eventhandler-config.md
                    snmp-config.md
                    vsphere-config.md
                kafka-exporter-config.md
                memcached-exporter-config.md
                mongodb_exporter-config.md
                mysqld-exporter-config.md
                node-exporter-config.md
                postgres-exporter-config.md
                process-exporter-config.md
                redis-exporter-config.md
                snmp-config.md
                statsd-exporter-config.md
                windows-exporter-config.md
            logs-config.md
            metrics-config.md
            scraping-service.md
            server-config.md
            traces-config.md
        cookbook
            _index.md
            dynamic-configuration
                01_Basics
                    01_Structure.md
                    01_assets
                        agent-1.yml
                        final.yml
                        server-1.yml
                    01_config.yml
                    02_Instances.md
                    02_assets
                        agent-1.yml
                        final.yml
                        metrics-1.yml
                        metrics_instances-1.yml
                        metrics_instances-2.yml
                        server-1.yml
                    02_config.yml
                    03_Integrations.md
                    03_assets
                        agent-1.yml
                        final.yml
                        integrations-node.yml
                        integrations-redis.yml
                        metrics-1.yml
                        metrics_instances-1.yml
                        metrics_instances-2.yml
                        server-1.yml
                    03_config.yml
                    04_Logs_and_Traces.md
                    04_assets
                        agent-1.yml
                        final.yml
                        integrations-node.yml
                        integrations-redis.yml
                        logs-1.yml
                        metrics-1.yml
                        metrics_instances-1.yml
                        metrics_instances-2.yml
                        server-1.yml
                        traces-1.yml
                    04_config.yml
                02_Templates
                    01_Looping.md
                    01_assets
                        agent-1.yml
                        final.yml
                    01_config.yml
                    02_Datasources.md
                    02_assets
                        agent-1.yml
                        final.yml
                        fruit.json
                    02_config.yml
                    03_Datasource_and_Objects.md
                    03_assets
                        agent-1.yml
                        computers.json
                        final.yml
                    03_config.yml
                03_Advanced_Datasources
                    01_AWS.md
                    01_assets
                        agent-1.yml
                        final.yml
                    01_config.yml
                README.md
                _index.md
        operation-guide
            _index.md
        operator
            _index.md
            add-custom-scrape-jobs.md
            architecture.md
            custom-resource-quickstart.md
            getting-started.md
            helm-getting-started.md
        set-up
            _index.md
            install-agent-binary.md
            install-agent-docker.md
            install-agent-macos.md
            install-agent-on-windows.md
            quick-starts.md
        upgrade-guide
            _index.md
|
| -- |
example
    docker-compose
        README.md
        grafana
            dashboards
                agent-operational.json
                agent-remote-write.json
                agent-tracing-pipeline.json
                agent.json
            datasources
                datasource.yml
        jsonnetfile.json
        jsonnetfile.lock.json
        load-generator
            load-generator.json
    k3d
        README.md
        environment
            spec.json
        jsonnetfile.json
        jsonnetfile.lock.json
        lib
            load-generator
                load-generator-config.json
        smoke
            spec.json
operations
    agent-flow-mixin
        jsonnetfile.json
    river-jsonnet
        README.md
        jsonnetfile.json
packaging
    linux_packages_test.go
    rpm
        gpg-sign.sh
pkg
    agentctl
        cardinality.go
        cardinality_test.go
        samples.go
        sync.go
        sync_test.go
        wal_iterator.go
        walstats.go
        walstats_test.go
    agentproto
        agent.pb.go
        func.go
    build
        build.go
    client
        client.go
        grafanacloud
            client.go
            client_test.go
    cluster
        cluster.go
        cluster_test.go
        gossip.go
        gossip_test.go
    config
        config.go
        config_test.go
        dynamicconfig.go
        dynamicloader.go
        dynamicloader_configs_test.go
        dynamicloader_template_test.go
        features
            features.go
            features_test.go
        integrations.go
        integrations_test.go
        remote_config.go
        remote_config_test.go
    crow
        crow.go
        metrics.go
        samples.go
        samples_test.go
    flow
        componenttest
            componenttest.go
            context.go
        config.go
        config_test.go
        flow.go
        flow_http.go
        flow_http_test.go
        flow_test.go
        internal
            controller
                component.go
                component_references.go
                component_write.go
                component_write_test.go
                loader.go
                loader_test.go
                metrics.go
                queue.go
                queue_test.go
                scheduler.go
                scheduler_test.go
                value_cache.go
                value_cache_test.go
                wrap_registerer.go
            dag
                dag.go
                marshal.go
                marshal_test.go
                ops.go
                ops_test.go
                tarjan.go
                tarjan_test.go
                walk.go
            graphviz
                graphviz.go
                grpahviz_test.go
            testcomponents
                doc.go
                passthrough.go
                tick.go
        logging
            config.go
            logger.go
        rivertypes
            optional_secret.go
            optional_secret_test.go
            secret.go
            secret_test.go
    integrations
        agent
            agent.go
        apache_http
            apache_http.go
        cadvisor
            cadvisor.go
            cadvisor_stub.go
            cadvisor_test.go
            common.go
        collector_integration.go
        config
            config.go
        consul_exporter
            consul_exporter.go
        dnsmasq_exporter
            dnsmasq_exporter.go
        ebpf_exporter
            common.go
            ebpf.go
            ebpf_stub.go
            ebpf_test.go
        elasticsearch_exporter
            elasticsearch_exporter.go
            tls.go
        github_exporter
            github_exporter.go
            github_test.go
        handler_integration.go
        install
            install.go
            install_test.go
        integration.go
        kafka_exporter
            kafka_exporter.go
            kafka_test.go
        manager.go
        manager_test.go
        memcached_exporter
            memcached_exporter.go
        mongodb_exporter
            logruskit.go
            mongodb_exporter.go
            mongodb_test.go
        mysqld_exporter
            mysqld-exporter.go
            mysqld_test.go
        node_exporter
            collectors.go
            config.go
            config_test.go
            node_exporter.go
            node_exporter_linux.go
            node_exporter_test.go
        postgres_exporter
            postgres_exporter.go
            postgres_exporter_test.go
            postgres_test.go
        process_exporter
            config.go
            process-exporter.go
            process-exporter_linux.go
        redis_exporter
            redis_exporter.go
            redis_exporter_test.go
        register.go
        register_test.go
        snmp_exporter
            common
                common.go
            snmp.go
            snmp_exporter.go
        statsd_exporter
            metrics.go
            statsd_exporter.go
        stub_integration.go
        v2
            agent
                agent.go
            apache_http
                apache_http.go
            app_agent_receiver
                app_agent_receiver.go
                config.go
                config_test.go
                handler.go
                handler_test.go
                logs_exporter.go
                logs_exporter_test.go
                payload.go
                payload_test.go
                receiver_metrics_exporter.go
                receiver_metrics_test.go
                sourcemaps.go
                sourcemaps_test.go
                testdata
                    foo.js
                    payload.json
                    payload_2.json
                traces_exporter.go
                traces_test.go
                utils.go
                utils_test.go
            autoscrape
                appender.go
                autoscrape.go
                autoscrape_test.go
            common
                metrics.go
            controller.go
            controller_httpintegration_test.go
            controller_metricsintegration_test.go
            controller_test.go
            controller_updateintegration_test.go
            eventhandler
                eventhandler.go
                eventhandler_test.go
                integration.go
            integrations.go
            metricsutils
                metricshandler_integration.go
                metricshandler_integration_test.go
                versionshim.go
            register.go
            register_test.go
            snmp_exporter
                snmp.go
                snmp_exporter.go
            subsystem.go
            subsystem_test.go
            targetgroup.go
            utils.go
            vmware_exporter
                vmware_exporter.go
            workers.go
        windows_exporter
            config.go
            config_windows.go
            windows_exporter.go
            windows_exporter_windows.go
    logs
        config.go
        config_test.go
        http.go
        http_test.go
        logs.go
        logs_test.go
    metrics
        agent.go
        agent_test.go
        cleaner.go
        cleaner_test.go
        cluster
            client
                client.go
            cluster.go
            config.go
            config_watcher.go
            config_watcher_test.go
            configapi
                types.go
            node.go
            node_test.go
            validation.go
            validation_test.go
        http.go
        http_test.go
        instance
            configstore
                api.go
                api_test.go
                codec.go
                codec_test.go
                errors.go
                mock.go
                remote.go
                remote_test.go
                store.go
                unique.go
            errors.go
            global.go
            group_manager.go
            group_manager_test.go
            host_filter.go
            host_filter_test.go
            instance.go
            instance_integration_test.go
            instance_test.go
            manager.go
            manager_test.go
            marshal.go
            marshal_test.go
            modal_manager.go
            noop.go
        wal
            series.go
            util.go
            wal.go
            wal_test.go
    operator
        apis
            monitoring
                v1alpha1
                    deployment.go
                    group.go
                    types.go
                    types_integrations.go
                    types_logs.go
                    types_metrics.go
                    zz_generated.deepcopy.go
        assets
            assets.go
        build_hierarchy.go
        build_hierarchy_test.go
        clientutil
            clientutil.go
            merge.go
        config
            config.go
            config_references.go
            config_references_test.go
            config_test.go
            fs_importer.go
            integration_templates_test.go
            logs_templates_test.go
            metrics_templates_test.go
            utils.go
            utils_test.go
        defaults.go
        hierarchy
            hierarchy.go
            hierarchy_test.go
            list.go
            selector.go
        kubelet.go
        kubelet_test.go
        logutil
            log.go
        operator.go
        operator_test.go
        reconciler.go
        reconciler_integrations.go
        reconciler_integrations_test.go
        reconciler_logs.go
        reconciler_metrics.go
        resources_integrations.go
        resources_logs.go
        resources_metrics.go
        resources_pod_template.go
        resources_pod_template_test.go
    river
        ast
            ast.go
            walk.go
        diag
            diag.go
            printer.go
            printer_test.go
        internal
            rivertags
                rivertags.go
                rivertags_test.go
            stdlib
                stdlib.go
            value
                capsule.go
                decode.go
                decode_benchmarks_test.go
                decode_test.go
                errors.go
                number_value.go
                raw_function.go
                tag_cache.go
                type.go
                type_test.go
                value.go
                value_object.go
                value_object_test.go
                value_test.go
        parser
            error_test.go
            internal.go
            internal_test.go
            parser.go
            parser_test.go
        printer
            printer.go
            printer_test.go
            trimmer.go
            walker.go
        scanner
            scanner.go
            scanner_test.go
        token
            builder
                builder.go
                builder_test.go
                token.go
                value_tokens.go
            file.go
            token.go
        types.go
        vm
            constant.go
            error.go
            op_binary.go
            op_unary.go
            vm.go
            vm_benchmarks_test.go
            vm_block_test.go
            vm_errors_test.go
            vm_stdlib_test.go
            vm_test.go
    server
        config.go
        flags.go
        logger.go
        logger_test.go
        logger_windows.go
        server.go
        server_test.go
        signal_context.go
        testdata
            windows
                README.md
                agent-config.yml
        tls.go
        tls_certstore_stub.go
        tls_certstore_windows.go
        tls_certstore_windows_test.go
        tls_test.go
    traces
        automaticloggingprocessor
            automaticloggingprocessor.go
            automaticloggingprocessor_test.go
            factory.go
        config.go
        config_test.go
        contextkeys
            keys.go
        instance.go
        internal
            traceutils
                server.go
        noopreceiver
            factory.go
            receiver.go
        promsdprocessor
            factory.go
            prom_sd_processor.go
            prom_sd_processor_test.go
        pushreceiver
            factory.go
            receiver.go
        remotewriteexporter
            exporter.go
            exporter_test.go
            factory.go
        servicegraphprocessor
            factory.go
            processor.go
            processor_test.go
            store.go
            store_test.go
            testdata
                trace-sample.json
                unpaired-trace-sample.json
        traces.go
        traces_test.go
    usagestats
        reporter.go
        reporter_test.go
        stats.go
    util
        compare_yaml.go
        defaults.go
        k8s
            k8s.go
            k8s_test.go
            objects.go
            resources.go
        strings.go
        structwalk
            structwalk.go
            structwalk_test.go
        subset
            subset.go
            subset_test.go
        test_logger.go
        trigger.go
        trigger_test.go
        unregisterer.go
        untab.go
        yaml.go
        yaml_test.go
production
    grafana-agent-mixin
        jsonnetfile.json
    grafanacloud-install.ps1
    grafanacloud-install.sh
    kubernetes
        README.md
        build
            build.sh
            jsonnetfile.json
            jsonnetfile.lock.json
            templates
                bare
                    spec.json
                base-sigv4
                    spec.json
                base
                    spec.json
                loki
                    spec.json
                operator
                    spec.json
                traces
                    spec.json
        install-bare.sh
    tanka
        grafana-agent-operator
            jsonnetfile.json
        grafana-agent
            jsonnetfile.json
            v1
                README.md
            v2
                README.md
 |
tools
    crow
        README.md
        main.go
    release-note.md
    rivereval
        main.go
    riverfmt
        main.go
    smoke
        internal
            smoke.go
            tasks.go
            tasks_test.go
        main.go

# Grafana Agent Documentation

This directory contains documentation for Grafana Agent. It is split into two
parts:

* `user/`: Documentation for users. This directory is hosted on
  [grafana.com/docs/agent](https://grafana.com/docs/agent/latest/) and we
  recommend interacting with it there instead of viewing the markdown on
  Github.
* `developer/`: Documentation for contributors and maintainers.
* `rfcs/`: RFCs for proposals relating to Grafana Agent.

## Preview the website

Run `make docs`. This launches a preview of the website with the current grafana docs at `http://localhost:3002/docs/agent/latest/` which will refresh automatically when changes are made to content in the `sources` directory.
Make sure Docker is running.

### Community Projects

Below is a list of community-led projects for working with Grafana Agent. These projects are not maintained or supported by Grafana Labs.

#### Helm (Kubernetes Deployment)

A publically available release of a Grafana Agent Helm chart is maintained [here](https://github.com/DandyDeveloper/charts/tree/master/charts/grafana-agent). Contributions and improvements are welcomed. Full details on rolling out and supported options can be found in the [readme](https://github.com/DandyDeveloper/charts/blob/master/charts/grafana-agent/README.md).

This *does not* require the Grafana Agent Operator to rollout / deploy.

#### Juju (Charmed Operator)

The [grafana-agent-k8s](https://github.com/canonical/grafana-agent-operator) charmed operator runs with [Juju](https://juju.is) the Grafana Agent on Kubernetes.
The Grafana Agent charmed operator is designed to work with the [Logs, Metrics and Alerts](https://juju.is/docs/lma2) observability stack.
# Kubernetes Config

This directory contains Kubernetes manifest templates for rolling out the Agent.

Manifests:

- Metric collection (StatefulSet): [`agent-bare.yaml`](./agent-bare.yaml)
- Log collection (DaemonSet): [`agent-loki.yaml`](./agent-loki.yaml)
- Trace collection (Deployment): [`agent-traces.yaml`](./agent-traces.yaml)

⚠️  **These manifests do not include the Agent's configuration (ConfigMaps)**,
which are necessary to run the Agent.

For sample configurations and detailed installation instructions, please head to:

- [Grafana Agent Metrics Kubernetes Quickstart](https://grafana.com/docs/grafana-cloud/quickstart/agent-k8s/k8s_agent_metrics/)
- [Grafana Agent Logs Kubernetes Quickstart](https://grafana.com/docs/grafana-cloud/quickstart/agent-k8s/k8s_agent_logs/)
- [Grafana Agent Traces Kubernetes Quickstart](https://grafana.com/docs/grafana-cloud/quickstart/agent-k8s/k8s_agent_traces/)

## Manually Applying

Since the manifest files are just templates, note that they are *not* ready for
applying out of the box and you will have to manually perform the following steps:

1. Download the manifest as `manifest.yaml`

2. Modify your copy of the manifest, replacing relevant variables with the appropriate values

3. Apply the modified manifest file: `kubectl -ndefault apply -f manifest.yaml`.

This directory also contains an `install-bare.sh` script that is used inside of
Grafana Cloud instructions. If using the Grafana Agent outside of Grafana Cloud,
it is recommended to follow the steps above instead of calling this script
directly.

## Rebuilding the manifests

The manifests provided are created using Grafana Labs' production
[Tanka configs](../tanka/grafana-agent) with some default values. If you want to
build the YAML file with some custom values, you will need the following pieces
of software installed:

1. [Tanka](https://github.com/grafana/tanka) >= v0.8
2. [`jsonnet-bundler`](https://github.com/jsonnet-bundler/jsonnet-bundler) >= v0.2.1

See the [`template` Tanka environment](./build/templates) for the current
settings that initialize the Grafana Agent Tanka configs.

To build the YAML files, execute the `./build/build.sh` script or run `make example-kubernetes`
from the project's root directory.
# `river-jsonnet` library

The `river-jsonnet` library makes it possible to return River-formatted config
files using Jsonnet.

To manifest a River configuration file, call `river.manifestRiver(value)`.

Field names from objects are expected to follow one of the three forms:

* `<name>` for River attributes (e.g., `foobar`).
* `block <name>` for unlabeled River blocks (e.g., `block exporter.node`)
* `block <name> <label>` for labeled River blocks (.e.g, `block metrics.remote_write default`).

Instead of following these naming conventions, helper functions are provided to
make it easier:

* `river.attr(name)` returns a field name that can be used as an attribute.
* `river.block(name, label="")` returns a field name that represents a block.

In addition to the helper functions, `river.expr(literal)` is used to inject a
literal River expression, so that `river.expr('env("HOME")')` is manifested as
the literal River expression `env("HOME")`.

## Limitations

* Manifested River files always have attributes and object keys in
  lexicographic sort order, regardless of how they were defined in Jsonnet.
* The resulting River files are not pretty-printed to how the formatter would
  print files.

## Example

```jsonnet
local river = import 'github.com/grafana/agent/operations/river-jsonnet/main.libsonnet';

river.manifestRiver({
  attr_1: "Hello, world!",

  [river.block("some_block", "foobar")]: {
    expr: river.expr('env("HOME")'),
    inner_attr_1: [0, 1, 2, 3],
    inner_attr_2: {
      first_name: "John",
      last_name: "Smith",
    },
  },
})
```

results in

```river
attr_1 = "Hello, world"
some_block "foobar" {
  expr = env("HOME")
  inner_attr_1 = [0, 1, 2, 3]
  inner_attr_2 = {
    "first_name" = "John",
    "last_name" = "Smith",
  }
}
```
# `k3d` Examples

## Agent Environment

The `k3d` example uses `k3d` and `tanka` to produce a Kubernetes environment
that implements a full Grafana Agent environment for testing.

### Requirements

- A Unix-y command line (macOS or Linux will do).
- Kubectl
- Docker
- [Tanka >= v0.9.2](https://github.com/grafana/tanka)
- [k3d >= v4.0.0,<= v5.2.2](https://github.com/rancher/k3d)
- [jsonnet-bundler >= v0.4.0](https://github.com/jsonnet-bundler/jsonnet-bundler)

### Getting Started

Build latest agent images with `make agent-image agentctl-image` in the project root directory if there are local changes to test.

Run the following to create your cluster:

```bash
# Create a new k3d cluster
./scripts/create.bash

# Import images into k3d if they are not available on docker hub
k3d image import -c agent-k3d grafana/agent:main
k3d image import -c agent-k3d grafana/agentctl:main

# Ensure jsonnet is up to date before applying environment
jb install
tk apply ./environment

# Navigate to grafana.k3d.localhost:30080 in your browser to view dashboards

# Delete the k3d cluster when you're done with it
k3d cluster delete agent-k3d
```

## Smoke Test Environment

The smoke test environment is used to validate samples end to end.

### Running

Smoke Test environment is invoked via `/scripts/smoke-test.bash`

This tool will spin up cluster of Grafana Agent, Cortex, Avalanche, Smoke and [Crow](../../tools/crow/README.md) instances. The Smoke deployment will then periodically kill instances and check for any failed alerts. At the end of the duration (default 3h) it will end the testing.

For users who do not have access to the `us.gcr.io/kubernetes-dev` container registry, do the following to run the smoke test:

* Build the Smoke and Crow images locally (from the root project directory):
```
make grafana-agent-crow-image agent-smoke-image
```
* Run the smoke test using `/scripts/smoke-test.bash` script.
* `Smoke` and `Crow` pods will fail because the images are not imported into the cluster. Import them running:
```
k3d image import -c agent-smoke-test us.gcr.io/kubernetes-dev/grafana/agent-smoke:main
k3d image import -c agent-smoke-test us.gcr.io/kubernetes-dev/grafana/agent-crow:main
```

### What to look for?

These alerts are viewable [here](http://prometheus.k3d.localhost:50080/alerts).

Prometheus alerts are triggered:
- If any Crow instances are not running or Crow samples are not being propagated correctly.
- If any Vulture instances are not running or Vulture samples are not being propagated correctly.
- If any Grafana Agents are not running or Grafana Agent limits are outside their norm.

NOTE: The alerts might be in pending until the system settles down.

![](./assets/pending_alert.png)

An alert firing will look similar to the below.

![](./assets/alert_firing.png)

If at the end of the test any issues are found they will look similar to the below.

![](./assets/console_failure.png)

### How to trigger an alert?

Changing the avalanche setting for label_count to 1000, located [here](../../production/tanka/grafana-agent/smoke/avalanche/main.libsonnet). This will ensure the [GrafanaAgentMemHigh](http://prometheus.k3d.localhost:50080/graph?g0.expr=ALERTS%7Balertname%3D%22GrafanaAgentMemHigh%22%7D&g0.tab=1&g0.stacked=0&g0.show_exemplars=0.g0.range_input=1h.) alert exceeds the limit.

![](./assets/trigger_change.png)

### Architecture

By default, a k3d cluster will be created running the following instances

- agent-single - single instance
- agent-cluster - 3 Grafana Agents in clustered configuration
- crow-cluster - serves the agent cluster
- crow-single - serves the single agent
- cortex
- avalanche - selection of avalanche instances serving traffic
- smoke - scales avalanche replicas and introduces chaos by deleting agent pods during testing
- vulture - emits traces and checks if are stored properly
- tempo

Crow and Vulture instances will check to see if the metrics and traces that were scraped shows up in the prometheus endpoint and then will emit metrics on the success of those metrics. This success/failure result will trigger an alert if it is incorrect.

### Metrics Flow

![](./assets/metrics_flow.png)

### Traces Flow

![](./assets/traces_flow.png)

### Avalanche

Avalanche is used to add some additional load on the system and general testing.
---
aliases:
- /docs/agent/latest/README/
draft: "True"
---

<p align="center"><img src="assets/logo_and_name.png" alt="Grafana Agent logo"></p>

Grafana Agent is an telemetry collector for sending metrics, logs,
and trace data to the opinionated Grafana observability stack. It works best
with:

* [Grafana Cloud](https://grafana.com/products/cloud/)
* [Grafana Enterprise Stack](https://grafana.com/products/enterprise/)
* OSS deployments of [Grafana Loki](https://grafana.com/oss/loki/), [Prometheus](https://prometheus.io/), [Cortex](https://cortexmetrics.io/), and [Grafana Tempo](https://grafana.com/oss/tempo/)

- Grafana Agent uses less memory on average than Prometheus – by doing less
  (only focusing on `remote_write`-related functionality).
- Grafana Agent allows for deploying multiple instances of the Agent in a
  cluster and only scraping metrics from targets that are running on the same host.
  This allows for distributing memory requirements across the cluster
  rather than pressurizing a single node.
<p align="center"><img src="docs/sources/assets/logo_and_name.png" alt="Grafana Agent logo"></p>

Grafana Agent is a telemetry collector for sending metrics, logs,
and trace data to the opinionated Grafana observability stack. It works best
with:

* [Grafana Cloud](https://grafana.com/products/cloud/)
* [Grafana Enterprise Stack](https://grafana.com/products/enterprise/)
* OSS deployments of [Grafana Loki](https://grafana.com/oss/loki/), [Prometheus](https://prometheus.io/), [Grafana Mimir](https://grafana.com/oss/mimir/), and [Grafana Tempo](https://grafana.com/oss/tempo/)

Users of Prometheus operating at a massive scale (i.e., millions of active
series) can struggle to run an unsharded singleton Prometheus instance: it becomes a
single point of failure and requires a giant machine with a lot of resources
allocated to it. Even with proper sharding across multiple Prometheus instances,
using Prometheus to send data to a cloud vendor can seem redundant: why pay for
cloud storage if data is already stored locally?

The Grafana Agent uses the same code as Prometheus, but tackles these issues
by only using the most relevant parts of Prometheus for interaction with hosted
metrics:

1. Service Discovery
2. Scraping
3. Write Ahead Log (WAL)
4. Remote Write

On top of these, the Grafana Agent enables easier sharding mechanisms that
enable users to shard Agents across their cluster and lower the memory requirements
per machine.

A typical deployment of the Grafana Agent for Prometheus metrics can see
up to a 40% reduction in memory usage with equal scrape loads.

The Grafana Agent it can be used to send Prometheus metrics to any system that
supports the Prometheus `remote_write` API.

## Trade-offs

By heavily optimizing Prometheus for remote write and resource reduction, some
trade-offs have been made:

- You can't query the Agent; you can only query metrics from the remote write
  storage.
- Recording rules aren't supported.
- Alerts aren't supported.
- When sharding the Agent, if your node has problems that interrupt metric
  availability, metrics tracking that node won't be sent for alerting on.

While the Agent can't use recording rules and alerts, `remote_write` systems such
as Mimir currently support server-side rules and alerts. Note that this trade-off
means that reliability of alerts are tied to the reliability of the remote system
and alerts will be delayed at least by the time it takes for samples to reach
the remote system.

## Getting Started

When using Kubernetes this [link](https://grafana.com/docs/grafana-cloud/quickstart/agent-k8s) offers the best guide.

Other installation methods can be found in our
[Grafana Agent](https://grafana.com/docs/agent/latest/set-up/) documentation.

More detailed [documentation](./docs/README.md) is provided as part of the
repository.

## Example

The [`example/`](./example) folder contains docker-compose configs and a local
k3d/Tanka environment. Both examples deploy the Agent, Cortex and Grafana for
testing the agent. See the [docker-compose README](./example/docker-compose/README.md)
and the [k3d example README](./example/k3d/README.md) for more information.

## Prometheus Vendoring

The Grafana Agent vendors a downstream Prometheus repository maintained by
[Grafana Labs](https://github.com/grafana/prometheus). This is done so
experimental features Grafana Labs wants to contribute upstream can first be
tested and iterated on quickly within the Agent. We aim to keep the
experimental changes to a minimum and upstream changes as soon as possible.

For more context on our vendoring strategy, read our
[downstream repo maintenance guide](./docs/developer/downstream-prometheus.md).

## Getting Help

If you have any questions or feedback regarding the Grafana Agent:

* Ask a question on the Agent Slack channel. To invite yourself to the Grafana
  Slack, visit https://slack.grafana.com/ and join the #agent channel.
* Alternatively ask questions on the
  [Discussions page](https://github.com/grafana/agent/discussions).
* [File an issue](https://github.com/grafana/agent/issues/new) for bugs, issues
  and feature suggestions.
* Attend the [Grafana Agent Community Call](https://docs.google.com/document/d/1TqaZD1JPfNadZ4V81OCBPCG_TksDYGlNlGdMnTWUSpo).

## Contributing

Any contributions are welcome and details can be found
[in our contributors guide](./docs/developer/contributing.md).
The files here are for usage with windows TLS. They need to be installed to the Users certificate store in the `My` location if using the included agent configuration. 

Running ` .\curl.exe -v -GET --key .\client_key_unencrypted.key --cert .\client_cert.crt --insecure https://localhost:12345/metrics` will let you see the correct metrics. NOTE: You will need to likely download [curl](https://curl.se/windows/dl-7.82.0_2/curl-7.82.0_2-win64-mingw.zip) compiled explicitly with windows crypto support.
When updating the drone.yml file the file must be re-signed using `make drone`. This is limited to Grafana employees for security reasons.

Drone environment variables will need to be setup beforehand.

```
export DRONE_SERVER=<url>
export DRONE_TOKEN=<token>
```
# Example

This directory contains a Docker Compose 3 environment that can be used to test
Grafana Agent.

By default, the following services are exposed:

1. Cortex for storing metrics (localhost:9009)
2. Grafana for visualizing telemetry (localhost:3000)
3. Loki for storing logs (localhost:3100)
4. Tempo for storing traces (localhost:3200)
5. Avalanche for a sample /metrics endpoint to scrape (localhost:9001).

Run the following to bring up the environment:

```
docker-compose up -d
```

By default, the Docker Compose environment doesn't include a Grafana Agent
container. This lets you test the agent externally, especially useful when
validating code changes. You can enable the included Grafana Agent by passing
`agent` to the profiles list: `docker compose --profile=agent up -d`. When
running, the Agent exposes its HTTP endpoint at localhost:12345. This address
can be changed with the `--server.http.address` flag (e.g.,
`--server.http.address=127.0.0.1:8000`).

The Docker Compose environment heavily relies on profiles to enable optional
features. You can pass multiple profiles by passing the flag multiple times:
`docker compose --profile agent --profile integrations up -d`.

## Running Integrations

The Docker Compose environment includes example services to point integrations
at (e.g., mysql, redis, consul, etc.). You can run all integrations using the
`integrations` profile, or run individual services with a profile name matching
the integration (i.e., enabling the `dnsmasq_exporter` profile to specifically
enable the dnsmasq service).

Enabling specific integration profiles is useful when you only want to test a
single integration, as the `integrations` profile can be resource intensive.

## Visualizing

Grafana is exposed at `http://localhost:3000`, and includes some useful
dashboards:

* The `Agent` dashboard gives a very high-level overview of running agents.

* The `Agent Prometheus Remote Write` dashboard visualizes the current state of
  writing metrics to Cortex.

* The `Agent Tracing Pipeline` dashbaord visualizes the current state of the
  tracing pipeline (if spans are being processed).

* The `Agent Operational` dashboard shows resource consumption of Grafana
  Agent. Not all panels will have data here, as they rely on metrics from other
  sources (i.e., cAdvisor).
# Tanka Configs

**STATUS**: Abandoned. Use v0 (parent directory) or v2 instead.

This directory contains the Tanka configs that we use to deploy the Grafana
Agent. It is marked as `v1` and is incompatible with the `v0` configs
found in the [parent directory](../).

This library is currently a work in progress and backwards-incompatible changes
may occur. Once the library is considered complete, no further backwards
incompatible changes will be made.

## Capabilities

This library is significantly more flexible than its `v0` counterpart. It tries
to allow to deploy and configure the Agent in a feature matrix:

| Mechanism        | Metrics | Logs      | Traces | Integrations |
| ---------------- | ------- | --------- | ------ | ------------ |
| DaemonSet        | Yes     | Yes       | Yes    | Yes          |
| Deployment       | Yes     | No        | No     | No           |
| Scraping Service | Yes     | No        | No     | No           |

The library can be invoked multiple times to get full coverage. For example, you
may wish to deploy a scraping service for scalable metrics collection, and a
DaemonSet with just Loki Logs for log collection.

Trying to use the library in incompatible ways will generate errors. For
example, you may not deploy a scraping service with Loki logs collection.

## API

## Generate Agent Deployment

- `new(name, namespace)`: Create a new DaemonSet. This is the default mode to
  deploy the Agent.  Enables host filtering.
- `newDeployment(name, namespace)`: Create a new single-replica Deployment.
  Disables host filtering.
- `newScrapingService(name, namespace, replicas)`: (Not yet available). Create a
  scalable deployment of clustered Agents. Requires being given a KV store such as Redis or ETCD.

## Configure Metrics

- `withMetricsConfig(config)`: Creates a metrics config block.
- `defaultMetricsConfig`: Default metrics config block.
- `withMetricsInstances(instances)`: Creates a metrics instance config to
  tell the Agent what to scrape.
- `withRemoteWrite(remote_writes)`: Configures locations to remote write metrics
   to. Controls remote writes globally.
- `scrapeInstanceKubernetes`: Default metrics instance config to scrape from
  Kubernetes.

## Configure Logs

- `withLogsConfig(config)`: Creates a Logs config block to pass to the Agent.
- `newLogsClient(client_config)`: Creates a new client configuration to pass
  to `withLogsClients`.
- `withLogsClients(clients)`: Add a set of clients to a Logs config block.
- `scrapeKubernetesLogs`: Default Logs config that collects logs from Kubernetes
  pods.

## Configure Traces

- `withTracesConfig(config)`: Creates a Traces config block to pass to the Agent.
- `withTracesRemoteWrite(remote_write)`: Configures one or multiple locations to push spans to.
- `withTracesSamplingStrategies(strategies)`: Configures strategies for trace collection.
- `withTracesScrapeConfigs(scrape_configs)`: Configures scrape configs to attach
   labels to incoming spans.
- `tracesScrapeKubernetes`: Default scrape configs to collect meta information
   from pods. Aligns with the labels from `scrapeInstanceKubernetes` and
   `scrapeKubernetesLogs` so logs, metrics, and traces all use the same set of
   labels.

## General

- `withImages(images)`: Use custom images.
- `withConfigHash(include=true)`: Whether to include a config hash annotation.
- `withPortsMixin(ports)`: Mixin ports from `k.core.v1.containerPort` against
   the container and service.
# Grafana Agent build images

The Grafana Agent build images are used for CI workflows to manage builds of
Grafana Agent.

There are two images:

* `grafana/agent-build-image:vX.Y.Z` (for building Linux containers)
* `grafana/agent-build-image:vX.Y.Z-windows` (for building Windows containers)

(Where `vX.Y.Z` is replaced with some semantic version, like v0.14.0).

## Pushing new images

Once a commit is merged to main which updates the build-image Dockerfiles, a
maintainer must push a tag matching the pattern `build-image/vX.Y.Z` to the
grafana/agent repo. For example, to create version v0.15.0 of the build images,
a maintainer would push the tag `build-image/v0.15.0`.

Automation will trigger off of this tag being pushed, building and pushing the
new build images to Docker Hub.

A follow-up commit to use the newly pushed build images must be made.
# Tanka Configs

**STATUS**: Work in progress, use of these configs is not recommended for production.

This directory contains the Tanka configs that we use to deploy the Grafana
Agent. It is marked as `v2` and is incompatible previous versions of the library
located in other directories.

This library is currently a work in progress and backwards-incompatible changes
may occur. Once the library is considered complete, no further backwards
incompatible changes will be made.

## Capabilities

This library is significantly simplified over the `v0` and `v1` counterparts.
Since there are many ways to combine the various functionalities of the Grafana
Agent, the `v2` library aims to stay out of your way and provide optional composible
helpers that may be useful for some people.

Users of the library will pick a controller for their deployment. They are
expected to know what feature are compatible with which controller:

| Controller       | Metrics              | Logs      | Traces | Integrations |
| ---------------- | -------------------  | --------- | ------ | ------------ |
| DaemonSet        | If host filtering    | Yes       | Yes    | No           |
| Deployment       | Yes                  | No        | No     | Yes          |
| StatefulSet      | Yes                  | No        | No     | Yes          |

Creating an incompatible deployment will cause runtime issues when running the
Agent (for example, if configuring Logs with a StatefulSet, you will only get
logs from the node the pods are running on).

To get full coverage of features, you must create multiple deployments of the
library. You may wish to combine a StatefulSet for metrics and integrations, a
Deployment for Traces, and a DaemonSet for logs.

## API

## Generate Agent Deployment

- `new(name='grafana-agent', namespace='')`: Create a new Agent without a
   controller.
- `withDeploymentController(replicas=1)`: Attach a Deployment as the Agent
  controller. Number of replicas may optionally be given.
- `withDaemonSetController()`: Attach a DaemonSet as the Agent controller.
- `withStatefulSetController(replicas=1, volumeClaims=[])`: Attach a StatefulSet
  as the Agent controller. Number of replicas and a set of volume claim
  templates may be given.

## Generate Scraping Service Syncer

The Scraping Service Syncer is used to sync metrics instance configs against the
scraping service config management API.

- `newSyncer(name='grafana-agent-sycner', namespace='', config={})`

## General

- `withAgentConfig(config)`: Provide a custom Agent config.
- `withArgsMixin(config)`: Pass a map of additional flags to set.
- `withMetricsPort(port)`: Value for the `http-metrics` port (default 80)
- `withImagesMixin(images)`: Use custom images instead of the defaults.
- `withConfigHash(include=true)`: Whether to include a config hash annotation.
- `withPortsMixin(ports=[])`: Mixin ports from `k.core.v1.containerPort` against
   the container and service.
- `withVolumesMixin(volumes=[])`: Volume to attach to the pod.
- `withVolumeMountsMixin(mounts=[])`: Volume mounts to attach to the container.

## Helpers

- `newKubernetesMetrics(config={})`: Creates a set of metrics scrape_configs for
  collecting metrics from Kubernetes pods.
- `newKubernetesLogs(config={})`: Creates a set of logs scrape_configs for
  collecting logs from Kubernetes pods.
- `newKubernetesTraces(config={})`: Creates a set of traces scrape_configs for
  associating spans with metadata from discovered Kubernetes pods.
- `withLogVolumeMounts(config={})`: Adds volume mounts to the controller for collecting
  logs.
- `withLogPermissions(config={})`: Runs the container as privileged and as the root user
  so logs can be collected properly.
- `withService(config)`: Add a service for the deployment, statefulset, or daemonset.
  Note that this must be called after any ports are added via `withPortsMixin`.
# Crow

Crow is a tool similar to Tempo Vulture and Loki Canary that is used to smoke test Grafana Agent. Crow works by generating metrics, then validating them against Prometheus. Crow uses two endpoints; the traditional `/metrics` and then `/validate` that generates the results of Crow checking for successful samples.

Note: The `/validate` endpoint should only be checked by the Grafana Agent instance that is configured to remote_write.
---
draft: true
---

# Dynamic Configuration Cookbook

The purpose of the cookbook is to guide you through common scenarios of using dynamic configuration. Each folder contains increasingly more complex use cases, but feel free to jump in wherever you feel appropriate.

## Basics

[Basics](01_Basics) covers
- [Structure](01_Basics/01_Structure.md) of how agent and server templates are loaded
- [Instances](01_Basics/02_Instances.md) of metrics and metrics instances are loaded
- [Integrations](01_Basics/03_Integrations.md) of how integrations are loaded
- [Logs and Traces](01_Basics/04_Logs_and_Traces.md) of how traces and logs are loaded

[Templates](02_Templates) covers
- [Looping](02_Templates/01_Looping.md) covers basic command usage and simple loops
- [Datasource](02_Templates/02_Datasources.md) covers usage of datasource which are external datastores you can use to pull in data
- [Datasources and Objects](02_Templates/03_Datasource_and_Objects.md) covers the usage of complex json objects

[Advanced Datasources](03_Advanced_Datasources) covers non file based datasources
- [AWS](03_Advanced_Datasources/01_AWS.md) covers querying EC2 for instances
# Grafana Agent Operator

The Grafana Agent Operator is a Kubernetes operator that makes it easier to
deploy the Grafana Agent and easier to collect telemetry data from your pods.
It is currently in **beta**, and is subject to change at any time.

It works by watching for [Kubernetes custom resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
that specify how you would like to collect telemetry data from your Kubernetes
cluster and where you would like to send it. They abstract Kubernetes-specific
configuration that is more tedious to perform manually. The Grafana Agent
Operator manages corresponding Grafana Agent deployments in your cluster by
watching for changes against the custom resources.

Metric collection is based on the [Prometheus
Operator](https://github.com/prometheus-operator/prometheus-operator) and
supports the official v1 ServiceMonitor, PodMonitor, and Probe CRDs from the
project. These custom resources represent abstractions for monitoring services,
pods, and ingresses. They are especially useful for Helm users, where manually
writing a generic SD to match all your charts can be difficult (or impossible!)
or where manually writing a specific SD for each chart can be tedious.

## Roadmap

- [ ] Helm chart
- [ ] Logs support
- [ ] Traces support
- [ ] Integrations support

## Documentation

Refer to the project's [documentation](../../docs/operator) for how to install
and get started with the Grafana Agent Operator.

## Developer Reference

The [Maintainer's Guide](./DEVELOPERS.md) includes
basic information to help you understand how the code works. This can be very
useful if you are planning on working on the operator.
