<commit_before>---
layout: post
title: "sparklyr finally connected to Spark"
excerpt: "I followed the offical tutorial at first, but it does not work will either with a unstable network environment or a fresh system."
tags:
  - coding
  - bugs
  - R
  - EN
---


I was trying to use R package `sparklyr` to install Spark and
connect to it. Since it is my second day with my new
laptop@Thinkpad_T460P, just installed Ubuntu 16.04 alongside
Windows 10. Apparently, I had not installed Java JDK yet.


## Introduction
- This passage may help you with the following 2 problems:
  1. Easily Install Spark with `sparklyr` under a unstable
  network environment.
  2. Solve the problem "JAVA_HOME is not set" when connecting to Spark.
- This passage works under the following environment:
  - system: x86_64, linux-gnu, Ubuntu 16.04
  - R version 3.3.3 (2017-03-06)
  - RStudio version 1.0.136
  - sparklyr version 0.5.3

## Easily Install
At first, I followed the official tutorial as every one else
would prefer. It seems pretty simple, the only thing I need to
do is running the following code.

```
install.packages("sparklyr")
library(sparklyr)
spark_install(version = "1.6.2")
```
Then here comes out the first problem: I can't successfully download the Spark
*.tar when ever the network breaks for more than a threshold time, the downloading process
would break off.

To solve the problem I found another command when typing `?spark_install`,
install spark from a tar file.

```
spark_install_tar(tarfile)
```
So I used a download manager(there are lots of good Linux download
managers like uGet, SteadyFlow, kGet, XDM, etc.). After installing, we can
check for the installing path and version.

```
spark_installed_versions()
spark_install_dir()
```

## Easily Connect
Speak at the first: I had set JAVA_HOME variables both in `/etc/profile`
and `/etc/environment`, along with sourcing the configuration.

```
> sc <- spark_connect(master = "local")
```
I met the second problem when connecting to Spark.

```
Error in force(code) :
  Failed while connecting to sparklyr to port (8880) for sessionid (6026): Gateway in port (8880) did not respond.
Path: /home/floatsd/.cache/spark/spark-1.6.2-bin-hadoop2.6/bin/spark-submit<commit_msg>test code block syntax<commit_after>---
layout: post
title: "sparklyr finally connected to Spark"
excerpt: "I followed the offical tutorial at first, but it does not work will either with a unstable network environment or a fresh system."
tags:
  - coding
  - bugs
  - R
  - EN
---


I was trying to use R package `sparklyr` to install Spark and
connect to it. Since it is my second day with my new
laptop@Thinkpad_T460P, just installed Ubuntu 16.04 alongside
Windows 10. Apparently, I had not installed Java JDK yet.


## Introduction
- This passage may help you with the following 2 problems:
  1. Easily Install Spark with `sparklyr` under a unstable
  network environment.
  2. Solve the problem "JAVA_HOME is not set" when connecting to Spark.
- This passage works under the following environment:
  - system: x86_64, linux-gnu, Ubuntu 16.04
  - R version 3.3.3 (2017-03-06)
  - RStudio version 1.0.136
  - sparklyr version 0.5.3

## Easily Install
At first, I followed the official tutorial as every one else
would prefer. It seems pretty simple, the only thing I need to
do is running the following code.

```css
install.packages("sparklyr")
library(sparklyr)
spark_install(version = "1.6.2")
```
Then here comes out the first problem: I can't successfully download the Spark
*.tar when ever the network breaks for more than a threshold time, the downloading process
would break off.

To solve the problem I found another command when typing `?spark_install`,
install spark from a tar file.

```ruby
spark_install_tar(tarfile)
```
So I used a download manager(there are lots of good Linux download
managers like uGet, SteadyFlow, kGet, XDM, etc.). After installing, we can
check for the installing path and version.

```html
spark_installed_versions()
spark_install_dir()
```

## Easily Connect
Speak at the first: I had set JAVA_HOME variables both in `/etc/profile`
and `/etc/environment`, along with sourcing the configuration.

```
> sc <- spark_connect(master = "local")
```
I met the second problem when connecting to Spark.

```
Error in force(code) :
  Failed while connecting to sparklyr to port (8880) for sessionid (6026): Gateway in port (8880) did not respond.
Path: /home/floatsd/.cache/spark/spark-1.6.2-bin-hadoop2.6/bin/spark-submit
{'fraction_non_alphanumeric': 0.08370436331255565, 'fraction_numerical': 0.02626892252894034, 'mean_word_length': 4.105681818181818, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2974750', 'n_tokens_mistral': 1478, 'n_tokens_neox': 1302, 'n_words': 617}