<commit_before>[[howto]]
= '`How-to`' guides

[partintro]
--
This section provides answers to some common '`how do I do that...`' type of questions
that often arise when using Spring Cloud Data Flow. 

If you are having a specific problem that we don't cover here, you might want to check out
http://stackoverflow.com/tags/spring-cloud-dataflow[stackoverflow.com] to see if someone has
already provided an answer; this is also a great place to ask new questions (please use
the `spring-cloud-dataflow` tag).

We're also more than happy to extend this section; If you want to add a '`how-to`' you
can send us a {github-code}[pull request].
--

== Configure Maven Properties

You can set the maven properties such as local maven repository location, remote maven repositories and their authentication credentials including
the proxy server properties via commandline properties when starting the Dataflow server or using the `SPRING_APPLICATION_JSON` environment property
for the Dataflow server.

The remote maven repositories need to be configured explicitly if the apps are resolved using maven repository as except `local` Data Flow server, other
 Data Flow server implementations (that use maven resources for app artifacts resolution) have no default value for remote repositories.
 The `local` server has `https://repo.spring.io/libs-snapshot` as the default remote repository.
 
To pass the properties as commandline options:

[source,bash]
----
$ java -jar <dataflow-server>.jar --maven.localRepository=mylocal
--maven.remote-repositories.repo1.url=https://repo1
--maven.remote-repositories.repo1.auth.username=repo1user
--maven.remote-repositories.repo1.auth.password=<PASSWORD>
--maven.remote-repositories.repo2.url=https://repo2 --maven.proxy.host=proxyhost
--maven.proxy.port=9018 --maven.proxy.auth.username=proxyuser
--maven.proxy.auth.password=<PASSWORD>
----

or, using the `SPRING_APPLICATION_JSON` environment property:

[source,json]
----
export SPRING_APPLICATION_JSON='{ "maven": { "local-repository": "local","remote-repositories": { "repo1": { "url": "https://repo1", "auth": { "username": "repo1user", "password": "<PASSWORD>" } },
"repo2": { "url": "https://repo2" } }, "proxy": { "host": "proxyhost", "port": 9018, "auth": { "username": "proxyuser", "password": "<PASSWORD>" } } } }'
----

Formatted JSON:

[source,json]
----
SPRING_APPLICATION_JSON='{
  "maven": {
    "local-repository": "local",
    "remote-repositories": {
      "repo1": {
        "url": "https://repo1",
        "auth": {
          "username": "repo1user",
          "password": "<PASSWORD>"
        }
      },
      "repo2": {
        "url": "https://repo2"
      }
    },
    "proxy": {
      "host": "proxyhost",
      "port": 9018,
      "auth": {
        "username": "proxyuser",
        "password": "<PASSWORD>"
      }
    }
  }
}'
----

NOTE: Depending on Spring Cloud Data Flow server implementation, you may have to pass the
environment properties using the platform specific environment-setting capabilities. For instance,
in Cloud Foundry, you'd be passing them as `cf set-env SPRING_APPLICATION_JSON`.


== Logging

Spring Cloud Data Flow is built upon several Spring projects, but ultimately the dataflow-server is a 
Spring Boot app, so the logging techniques that applies to any link:http://docs.spring.io/spring-boot/docs/current/reference/html/howto-logging.html#howto-logging[Spring Boot] 
application is applicable here as well. 


While troubleshooting, following are the two primary areas where enabling the DEBUG logs could be 
useful.

=== Deployment Logs
Spring Cloud Data Flow builds upon link:https://github.com/spring-cloud/spring-cloud-deployer[Spring Cloud Deployer] SPI 
and the platform specific dataflow-server uses the respective link:https://github.com/spring-cloud?utf8=%E2%9C%93&query=deployer[SPI implementations]. 
Specifically, if we were to troubleshoot deployment specific issues; such as the network errors, it'd 
be useful to enable the DEBUG logs at the deployers and the libraries used by it. 

. For instance, if you'd like to enable DEBUG logs for the link:https://github.com/spring-cloud/spring-cloud-deployer/tree/master/spring-cloud-deployer-local[local-deployer], 
you'd be starting the server with following.

+
[source,bash]
----
$ java -jar <dataflow-server>.jar --logging.level.org.springframework.cloud.deployer.spi.local=DEBUG
----
+

(_where, `org.springframework.cloud.deployer.spi.local` is the global package for everything local-deployer
related_)

. For instance, if you'd like to enable DEBUG logs for the link:https://github.com/spring-cloud/spring-cloud-deployer-cloudfoundry[cloudfoundry-deployer], 
you'd be setting the following environment variable and upon restaging the dataflow-server, we will 
see more logs around request, response and the elaborate stack traces (_upon failures_). The cloudfoundry-deployer 
uses link:https://github.com/cloudfoundry/cf-java-client[cf-java-client], so we will have to enable DEBUG 
logs for this library. 


+
[source,bash]
----
$ cf set-env dataflow-server JAVA_OPTS '-Dlogging.level.cloudfoundry-client=DEBUG'
$ cf restage dataflow-server
----
+

(_where, `cloudfoundry-client` is the global package for everything `cf-java-client` related_)

. If there's a need to review Reactor logs, which is used by the `cf-java-client`, then the following 
would be helpful.

+
[source,bash]
----
$ cf set-env dataflow-server JAVA_OPTS '-Dlogging.level.cloudfoundry-client=DEBUG -Dlogging.level.reactor.ipc.netty=DEBUG'
$ cf restage dataflow-server
----
+

(_where, `reactor.ipc.netty` is the global package for everything `reactor-netty` related_)

NOTE: Similar to the `local-deployer` and `cloudfoundry-deployer` options as discussed above, there
are equivalent settings available for Apache YARN, Apache Mesos and Kubernetes varaints, too. Check out the 
respective link:https://github.com/spring-cloud?utf8=%E2%9C%93&query=deployer[SPI implementations] to 
find out more details about the packages to configure for logging.

=== Application Logs

The streaming applications in Spring Cloud Data Flow are Spring Boot applications and they can be 
independently setup with logging configurations. 

For instance, if you'd have to troubleshoot the `header` and `payload` specifics that's being passed 
around between source, processor and sink channels, you'd be deploying the stream with the following
options.


[source,bash]
----
dataflow:>stream create foo --definition "http --logging.level.org.springframework.integration=DEBUG | transform --logging.level.org.springframework.integration=DEBUG | log --logging.level.org.springframework.integration=DEBUG" --deploy
----

(_where, `org.springframework.integration` is the global package for everything Spring Integration related, 
which is responsible for messaging channels_)<commit_msg>Polishing how-to guide for logging<commit_after>[[howto]]
= '`How-to`' guides

[partintro]
--
This section provides answers to some common '`how do I do that...`' type of questions
that often arise when using Spring Cloud Data Flow. 

If you are having a specific problem that we don't cover here, you might want to check out
http://stackoverflow.com/tags/spring-cloud-dataflow[stackoverflow.com] to see if someone has
already provided an answer; this is also a great place to ask new questions (please use
the `spring-cloud-dataflow` tag).

We're also more than happy to extend this section; If you want to add a '`how-to`' you
can send us a {github-code}[pull request].
--

== Configure Maven Properties

You can set the maven properties such as local maven repository location, remote maven repositories and their authentication credentials including
the proxy server properties via commandline properties when starting the Dataflow server or using the `SPRING_APPLICATION_JSON` environment property
for the Dataflow server.

The remote maven repositories need to be configured explicitly if the apps are resolved using maven repository as except `local` Data Flow server, other
 Data Flow server implementations (that use maven resources for app artifacts resolution) have no default value for remote repositories.
 The `local` server has `https://repo.spring.io/libs-snapshot` as the default remote repository.
 
To pass the properties as commandline options:

[source,bash]
----
$ java -jar <dataflow-server>.jar --maven.localRepository=mylocal
--maven.remote-repositories.repo1.url=https://repo1
--maven.remote-repositories.repo1.auth.username=repo1user
--maven.remote-repositories.repo1.auth.password=<PASSWORD>
--maven.remote-repositories.repo2.url=https://repo2 --maven.proxy.host=proxyhost
--maven.proxy.port=9018 --maven.proxy.auth.username=proxyuser
--maven.proxy.auth.password=<PASSWORD>
----

or, using the `SPRING_APPLICATION_JSON` environment property:

[source,json]
----
export SPRING_APPLICATION_JSON='{ "maven": { "local-repository": "local","remote-repositories": { "repo1": { "url": "https://repo1", "auth": { "username": "repo1user", "password": "<PASSWORD>" } },
"repo2": { "url": "https://repo2" } }, "proxy": { "host": "proxyhost", "port": 9018, "auth": { "username": "proxyuser", "password": "<PASSWORD>" } } } }'
----

Formatted JSON:

[source,json]
----
SPRING_APPLICATION_JSON='{
  "maven": {
    "local-repository": "local",
    "remote-repositories": {
      "repo1": {
        "url": "https://repo1",
        "auth": {
          "username": "repo1user",
          "password": "<PASSWORD>"
        }
      },
      "repo2": {
        "url": "https://repo2"
      }
    },
    "proxy": {
      "host": "proxyhost",
      "port": 9018,
      "auth": {
        "username": "proxyuser",
        "password": "<PASSWORD>"
      }
    }
  }
}'
----

NOTE: Depending on Spring Cloud Data Flow server implementation, you may have to pass the
environment properties using the platform specific environment-setting capabilities. For instance,
in Cloud Foundry, you'd be passing them as `cf set-env SPRING_APPLICATION_JSON`.


== Logging

Spring Cloud Data Flow is built upon several Spring projects, but ultimately the dataflow-server is a 
Spring Boot app, so the logging techniques that apply to any link:http://docs.spring.io/spring-boot/docs/current/reference/html/howto-logging.html#howto-logging[Spring Boot]
application are applicable here as well.


While troubleshooting, following are the two primary areas where enabling the DEBUG logs could be 
useful.

=== Deployment Logs
Spring Cloud Data Flow builds upon link:https://github.com/spring-cloud/spring-cloud-deployer[Spring Cloud Deployer] SPI 
and the platform specific dataflow-server uses the respective link:https://github.com/spring-cloud?utf8=%E2%9C%93&query=deployer[SPI implementations]. 
Specifically, if we were to troubleshoot deployment specific issues; such as the network errors, it'd 
be useful to enable the DEBUG logs at the underlying deployer and the libraries used by it.

. For instance, if you'd like to enable DEBUG logs for the link:https://github.com/spring-cloud/spring-cloud-deployer/tree/master/spring-cloud-deployer-local[local-deployer], 
you'd be starting the server with following.

+
[source,bash]
----
$ java -jar <dataflow-server>.jar --logging.level.org.springframework.cloud.deployer.spi.local=DEBUG
----
+

(_where, `org.springframework.cloud.deployer.spi.local` is the global package for everything local-deployer
related_)

. For instance, if you'd like to enable DEBUG logs for the link:https://github.com/spring-cloud/spring-cloud-deployer-cloudfoundry[cloudfoundry-deployer], 
you'd be setting the following environment variable and upon restaging the dataflow-server, we will 
see more logs around request, response and the elaborate stack traces (_upon failures_). The cloudfoundry-deployer 
uses link:https://github.com/cloudfoundry/cf-java-client[cf-java-client], so we will have to enable DEBUG 
logs for this library. 


+
[source,bash]
----
$ cf set-env dataflow-server JAVA_OPTS '-Dlogging.level.cloudfoundry-client=DEBUG'
$ cf restage dataflow-server
----
+

(_where, `cloudfoundry-client` is the global package for everything `cf-java-client` related_)

. If there's a need to review Reactor logs, which is used by the `cf-java-client`, then the following 
would be helpful.

+
[source,bash]
----
$ cf set-env dataflow-server JAVA_OPTS '-Dlogging.level.cloudfoundry-client=DEBUG -Dlogging.level.reactor.ipc.netty=DEBUG'
$ cf restage dataflow-server
----
+

(_where, `reactor.ipc.netty` is the global package for everything `reactor-netty` related_)

NOTE: Similar to the `local-deployer` and `cloudfoundry-deployer` options as discussed above, there
are equivalent settings available for Apache YARN, Apache Mesos and Kubernetes variants, too. Check out the
respective link:https://github.com/spring-cloud?utf8=%E2%9C%93&query=deployer[SPI implementations] to 
find out more details about the packages to configure for logging.

=== Application Logs

The streaming applications in Spring Cloud Data Flow are Spring Boot applications and they can be 
independently setup with logging configurations. 

For instance, if you'd have to troubleshoot the `header` and `payload` specifics that are being passed
around source, processor and sink channels, you'd be deploying the stream with the following
options.


[source,bash]
----
dataflow:>stream create foo --definition "http --logging.level.org.springframework.integration=DEBUG | transform --logging.level.org.springframework.integration=DEBUG | log --logging.level.org.springframework.integration=DEBUG" --deploy
----

(_where, `org.springframework.integration` is the global package for everything Spring Integration related, 
which is responsible for messaging channels_)

These properties can also be specified via `deployment` properties when deploying the stream.

[source,bash]
----
dataflow:>stream deploy foo --properties "app.*.logging.level.org.springframework.integration=DEBUG"
----
{'fraction_non_alphanumeric': 0.11144621571161856, 'fraction_numerical': 0.005601034037052995, 'mean_word_length': 4.393880712625871, 'pattern_counts': {'":': 64, '<': 19, '<?xml version=': 0, '>': 22, 'https://': 26, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6342160', 'n_tokens_mistral': 4083, 'n_tokens_neox': 3870, 'n_words': 1399}