<commit_before>[[getting-started]]
= Getting started

[partintro]
--
If you're just getting started with Spring Cloud Data Flow, this is the section
for you! Here we answer the basic "`what?`", "`how?`" and "`why?`" questions. You'll
find a gentle introduction to Spring Cloud Data Flow along with installation instructions.
We'll then build our first Spring Cloud Data Flow application, discussing some core principles as
we go.
--

[[getting-started-system-requirements]]
== System Requirements

You need Java installed (Java 8 or later), and to build, you need to have Maven installed as well.

You need to have an RDBMS for storing stream, task and app states in the database. The `local` Data Flow server by default uses embedded H2 database for this.

You also need to have link:https://redis.io[Redis] running if you are running any streams that involve analytics applications. Redis may also be required run the unit/integration tests.

For the deployed streams and tasks to communicate, either link:http://www.rabbitmq.com[RabbitMQ] or link:http://kafka.apache.org[Kafka] needs to be installed.

[[getting-started-deploying-spring-cloud-dataflow]]
== Deploying Spring Cloud Data Flow Local Server

. Download the Spring Cloud Data Flow Server and Shell apps:
+
[source,bash,subs=attributes]
----
wget http://repo.spring.io/{version-type-lowercase}/org/springframework/cloud/spring-cloud-dataflow-server-local/{project-version}/spring-cloud-dataflow-server-local-{project-version}.jar

wget http://repo.spring.io/{version-type-lowercase}/org/springframework/cloud/spring-cloud-dataflow-shell/{project-version}/spring-cloud-dataflow-shell-{project-version}.jar
----
+
. Launch the Data Flow Server
+
.. Since the Data Flow Server is a Spring Boot application, you can run it just by using `java -jar`.
+
[source,bash,subs=attributes]
----
$ java -jar spring-cloud-dataflow-server-local-{project-version}.jar
----
+
. Launch the shell:
+
[source,bash,subs=attributes]
----
$ java -jar spring-cloud-dataflow-shell-{project-version}.jar
----
+
If the Data Flow Server and shell are not running on the same host, point the shell to the Data Flow server URL:
+
[source,bash]
----
server-unknown:>dataflow config server http://198.51.100.0
Successfully targeted http://198.51.100.0
dataflow:>
----
+
By default, the application registry will be empty. If you would like to register all out-of-the-box stream applications built with the Kafka binder in bulk, you can with the following command. For more details, review how to <<streams.adoc#spring-cloud-dataflow-register-stream-apps, register applications>>.
+
[source,bash,subs=attributes]
----
$ dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-kafka-10-maven
----
+
NOTE: Depending on your environment, you may need to configure the Data Flow Server to point to a custom
Maven repository location or configure proxy settings.  See <<getting-started-maven-configuration>> for more information.
+
. You can now use the shell commands to list available applications (source/processors/sink) and create streams. For example:
+
[source,bash]
----
dataflow:> stream create --name httptest --definition "http --server.port=9000 | log" --deploy
----
+
NOTE: You will need to wait a little while until the apps are actually deployed successfully
before posting data.  Look in the log file of the Data Flow server for the location of the log
files for the `http` and `log` applications.  Tail the log file for each application to verify
the application has started.
+
Now post some data
[source,bash]
----
dataflow:> http post --target http://localhost:9000 --data "hello world"
----
Look to see if `hello world` ended up in log files for the `log` application.

[NOTE]
====
When deploying locally, each app (and each app instance, in case of `count>1`) gets a dynamically assigned `server.port`
unless you explicitly assign one with `--server.port=x`. In both cases, this setting is propagated as a configuration
property that will override any lower-level setting that you may have used (_e.g._ in `application.yml` files).
====


[TIP]
====
In case you encounter unexpected errors when executing shell commands, you can
retrieve more detailed error information by setting the exception logging level
to `WARNING` in `logback.xml`:

[source,xml]
----
<logger name="org.springframework.shell.core.JLineShellComponent.exceptions" level="WARNING"/>
----

====

[[getting-started-maven-configuration]]
=== Maven Configuration
If you want to override specific maven configuration properties (remote repositories, proxies, etc.) and/or run the Data Flow Server behind a proxy,
you need to specify those properties as command line arguments when starting the Data Flow Server. For example:

[source,bash,subs=attributes]
----
$ java -jar spring-cloud-dataflow-server-local-{project-version}.jar --maven.localRepository=mylocal
--maven.remote-repositories.repo1.url=https://repo1
--maven.remote-repositories.repo1.auth.username=user1
--maven.remote-repositories.repo1.auth.password=<PASSWORD>
--maven.remote-repositories.repo1.snapshot-policy.update-policy=daily
--maven.remote-repositories.repo1.snapshot-policy.checksum-policy=warn
--maven.remote-repositories.repo1.release-policy.update-policy=never
--maven.remote-repositories.repo1.release-policy.checksum-policy=fail
--maven.remote-repositories.repo2.url=https://repo2
--maven.remote-repositories.repo2.policy.update-policy=always
--maven.remote-repositories.repo2.policy.checksum-policy=fail
--maven.proxy.host=proxy1
--maven.proxy.port=9010 --maven.proxy.auth.username=proxyuser1
--maven.proxy.auth.password=<PASSWORD>
----

By default, the protocol is set to `http`. You can omit the auth properties if the proxy doesn't need a username and password. Also, the maven `localRepository` is set to `${user.home}/.m2/repository/` by default.
Like in the above example, the remote repositories can be specified along with their authentication (if needed). If the remote repositories are behind a proxy, then the proxy properties can be specified as above.

The repository policies can be specified per remote repository configuration as above.
The key `policy` is applicable to both `snapshot` and the `release` repository policies.

As these are Spring Boot `@ConfigurationProperties` you can also specify them as environment variables, e.g. MAVEN_REMOTE_REPOSITORIES_REPO1_URL.
Another common option, is to set the properties using the `SPRING_APPLICATION_JSON` environment variable.
An example of how the JSON is structured is shown below:

[source,bash]
----
$ SPRING_APPLICATION_JSON='{ "maven": { "local-repository": null,
"remote-repositories": { "repo1": { "url": "https://repo1", "auth": { "username": "repo1user", "password": "<PASSWORD>" } }, "repo2": { "url": "https://repo2" } },
"proxy": { "host": "proxyhost", "port": 9018, "auth": { "username": "proxyuser", "password": "<PASSWORD>" } } } }' java -jar spring-cloud-dataflow-server-local-{project-version}.jar
----

[[getting-started-application-configuration]]
== Application Configuration
You can use the following configuration properties of the Data Flow server to customize how applications are deployed.

[source,properties,indent=0,subs="verbatim,attributes,macros"]
----
spring.cloud.deployer.local.workingDirectoriesRoot=java.io.tmpdir # Directory in which all created processes will run and create log files.

spring.cloud.deployer.local.deleteFilesOnExit=true # Whether to delete created files and directories on JVM exit.

spring.cloud.deployer.local.envVarsToInherit=TMP,LANG,LANGUAGE,"LC_.*. # Array of regular expression patterns for environment variables that will be passed to launched applications.

spring.cloud.deployer.local.javaCmd=java # Command to run java.

spring.cloud.deployer.local.shutdownTimeout=30 # Max number of seconds to wait for app shutdown.

spring.cloud.deployer.local.javaOpts= # The Java options to pass to the JVM
----

When deploying the application you can also set deployer properties prefixed with `deployer.<name of application>`, So for example to set Java options for the time application in the `ticktock` stream, use the following stream deployment properties.<commit_msg>Add RepositoryPolicies doc<commit_after>[[getting-started]]
= Getting started

[partintro]
--
If you're just getting started with Spring Cloud Data Flow, this is the section
for you! Here we answer the basic "`what?`", "`how?`" and "`why?`" questions. You'll
find a gentle introduction to Spring Cloud Data Flow along with installation instructions.
We'll then build our first Spring Cloud Data Flow application, discussing some core principles as
we go.
--

[[getting-started-system-requirements]]
== System Requirements

You need Java installed (Java 8 or later), and to build, you need to have Maven installed as well.

You need to have an RDBMS for storing stream, task and app states in the database. The `local` Data Flow server by default uses embedded H2 database for this.

You also need to have link:https://redis.io[Redis] running if you are running any streams that involve analytics applications. Redis may also be required run the unit/integration tests.

For the deployed streams and tasks to communicate, either link:http://www.rabbitmq.com[RabbitMQ] or link:http://kafka.apache.org[Kafka] needs to be installed.

[[getting-started-deploying-spring-cloud-dataflow]]
== Deploying Spring Cloud Data Flow Local Server

. Download the Spring Cloud Data Flow Server and Shell apps:
+
[source,bash,subs=attributes]
----
wget http://repo.spring.io/{version-type-lowercase}/org/springframework/cloud/spring-cloud-dataflow-server-local/{project-version}/spring-cloud-dataflow-server-local-{project-version}.jar

wget http://repo.spring.io/{version-type-lowercase}/org/springframework/cloud/spring-cloud-dataflow-shell/{project-version}/spring-cloud-dataflow-shell-{project-version}.jar
----
+
. Launch the Data Flow Server
+
.. Since the Data Flow Server is a Spring Boot application, you can run it just by using `java -jar`.
+
[source,bash,subs=attributes]
----
$ java -jar spring-cloud-dataflow-server-local-{project-version}.jar
----
+
. Launch the shell:
+
[source,bash,subs=attributes]
----
$ java -jar spring-cloud-dataflow-shell-{project-version}.jar
----
+
If the Data Flow Server and shell are not running on the same host, point the shell to the Data Flow server URL:
+
[source,bash]
----
server-unknown:>dataflow config server http://198.51.100.0
Successfully targeted http://198.51.100.0
dataflow:>
----
+
By default, the application registry will be empty. If you would like to register all out-of-the-box stream applications built with the Kafka binder in bulk, you can with the following command. For more details, review how to <<streams.adoc#spring-cloud-dataflow-register-stream-apps, register applications>>.
+
[source,bash,subs=attributes]
----
$ dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-kafka-10-maven
----
+
NOTE: Depending on your environment, you may need to configure the Data Flow Server to point to a custom
Maven repository location or configure proxy settings.  See <<getting-started-maven-configuration>> for more information.
+
. You can now use the shell commands to list available applications (source/processors/sink) and create streams. For example:
+
[source,bash]
----
dataflow:> stream create --name httptest --definition "http --server.port=9000 | log" --deploy
----
+
NOTE: You will need to wait a little while until the apps are actually deployed successfully
before posting data.  Look in the log file of the Data Flow server for the location of the log
files for the `http` and `log` applications.  Tail the log file for each application to verify
the application has started.
+
Now post some data
[source,bash]
----
dataflow:> http post --target http://localhost:9000 --data "hello world"
----
Look to see if `hello world` ended up in log files for the `log` application.

[NOTE]
====
When deploying locally, each app (and each app instance, in case of `count>1`) gets a dynamically assigned `server.port`
unless you explicitly assign one with `--server.port=x`. In both cases, this setting is propagated as a configuration
property that will override any lower-level setting that you may have used (_e.g._ in `application.yml` files).
====


[TIP]
====
In case you encounter unexpected errors when executing shell commands, you can
retrieve more detailed error information by setting the exception logging level
to `WARNING` in `logback.xml`:

[source,xml]
----
<logger name="org.springframework.shell.core.JLineShellComponent.exceptions" level="WARNING"/>
----

====

[[getting-started-maven-configuration]]
=== Maven Configuration
If you want to override specific maven configuration properties (remote repositories, proxies, etc.) and/or run the Data Flow Server behind a proxy,
you need to specify those properties as command line arguments when starting the Data Flow Server. For example:

[source,bash,subs=attributes]
----
$ java -jar spring-cloud-dataflow-server-local-{project-version}.jar --maven.localRepository=mylocal
--maven.remote-repositories.repo1.url=https://repo1
--maven.remote-repositories.repo1.auth.username=user1
--maven.remote-repositories.repo1.auth.password=<PASSWORD>
--maven.remote-repositories.repo1.snapshot-policy.update-policy=daily
--maven.remote-repositories.repo1.snapshot-policy.checksum-policy=warn
--maven.remote-repositories.repo1.release-policy.update-policy=never
--maven.remote-repositories.repo1.release-policy.checksum-policy=fail
--maven.remote-repositories.repo2.url=https://repo2
--maven.remote-repositories.repo2.policy.update-policy=always
--maven.remote-repositories.repo2.policy.checksum-policy=fail
--maven.proxy.host=proxy1
--maven.proxy.port=9010 --maven.proxy.auth.username=proxyuser1
--maven.proxy.auth.password=<PASSWORD>
----

By default, the protocol is set to `http`. You can omit the auth properties if the proxy doesn't need a username and password. Also, the maven `localRepository` is set to `${user.home}/.m2/repository/` by default.
Like in the above example, the remote repositories can be specified along with their authentication (if needed). If the remote repositories are behind a proxy, then the proxy properties can be specified as above.

The repository policies can be specified per remote repository configuration as above.
The key `policy` is applicable to both `snapshot` and the `release` repository policies.

You can refer https://github.com/eclipse/aether-core/blob/4cf5f7a406b516a45d8bf15e7dfe3fb3849cb87b/aether-api/src/main/java/org/eclipse/aether/repository/RepositoryPolicy.java#L16[Repository Policies] for the list of
supported repository policies.

As these are Spring Boot `@ConfigurationProperties` you can also specify them as environment variables, e.g. MAVEN_REMOTE_REPOSITORIES_REPO1_URL.
Another common option, is to set the properties using the `SPRING_APPLICATION_JSON` environment variable.
An example of how the JSON is structured is shown below:

[source,bash]
----
$ SPRING_APPLICATION_JSON='{ "maven": { "local-repository": null,
"remote-repositories": { "repo1": { "url": "https://repo1", "auth": { "username": "repo1user", "password": "<PASSWORD>" } }, "repo2": { "url": "https://repo2" } },
"proxy": { "host": "proxyhost", "port": 9018, "auth": { "username": "proxyuser", "password": "<PASSWORD>" } } } }' java -jar spring-cloud-dataflow-server-local-{project-version}.jar
----

[[getting-started-application-configuration]]
== Application Configuration
You can use the following configuration properties of the Data Flow server to customize how applications are deployed.

[source,properties,indent=0,subs="verbatim,attributes,macros"]
----
spring.cloud.deployer.local.workingDirectoriesRoot=java.io.tmpdir # Directory in which all created processes will run and create log files.

spring.cloud.deployer.local.deleteFilesOnExit=true # Whether to delete created files and directories on JVM exit.

spring.cloud.deployer.local.envVarsToInherit=TMP,LANG,LANGUAGE,"LC_.*. # Array of regular expression patterns for environment variables that will be passed to launched applications.

spring.cloud.deployer.local.javaCmd=java # Command to run java.

spring.cloud.deployer.local.shutdownTimeout=30 # Max number of seconds to wait for app shutdown.

spring.cloud.deployer.local.javaOpts= # The Java options to pass to the JVM
----

When deploying the application you can also set deployer properties prefixed with `deployer.<name of application>`, So for example to set Java options for the time application in the `ticktock` stream, use the following stream deployment properties.
{'fraction_non_alphanumeric': 0.11068473438062118, 'fraction_numerical': 0.009113802614222328, 'mean_word_length': 4.763303386316517, 'pattern_counts': {'":': 32, '<': 23, '<?xml version=': 0, '>': 35, 'https://': 11, 'lorem ipsum': 0, 'www.': 2, 'xml': 4}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5831525', 'n_tokens_mistral': 4823, 'n_tokens_neox': 4549, 'n_words': 1735}