<commit_before>---
layout: post
title:  "Spark环境搭建"
keywords: "spark,hadoop"
description: "在linux环境搭建Spark"
category: spark 
tags: [spark]
---
###服务器环境
```
~$ cat /proc/version
Linux version 4.4.0-24-generic (buildd@lcy01-30) (gcc version 5.3.1 20160413 (Ubuntu 5.3.1-14ubuntu2.1) ) #43-Ubuntu SMP Wed Jun 8 19:25:16 UTC 2016
```
###Spark安装教程
####JDK安装
```
~$ java -version
java version "1.8.0_92"
Java(TM) SE Runtime Environment (build 1.8.0_92-b14)
Java HotSpot(TM) Server VM (build 25.92-b14, mixed mode)
```
####Scala安装
 * 版本:2.10.5
 * [下载地址](http://www.scala-lang.org/)
 * 环境变量配置
```
export SCALA_HOME=/home/coldface/software/scala-2.10.5
export PATH=${SCALA_HOME}/bin:$PATH
```
 * 使修改生效执行source /etc/profile
 * 在命令行输入scala,出现下列信息说明安装成功
```
~$ scala
Welcome to Scala 2.10.5 (Java HotSpot(TM) Server VM, Java 1.8.0_92).
Type in expressions for evaluation. Or try :help.

scala>
```
####Spark安装
 * 版本:spark-1.6.1-bin-hadoop2.6
 * [下载地址](http://spark.apache.org/downloads.html)
 * 环境变了配置
```
export SPARK_HOME=/home/coldface/software/spark-1.6.1-bin-hadoop2.6
export PATH=${SPARK_HOME}/bin:$PATH
```
 * 使修改生效执行source /etc/profile
 * 在spark安装bin目录执行./spark-shell
```
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.1
      /_/

Using Scala version 2.10.5 (Java HotSpot(TM) Server VM, Java 1.8.0_92)
```
####运行Spark WordCount示例
 * 在命令行输入:./spark-shell开启spark
 * 把输入文件加载进RDD
 ```
 scala> var textFile = sc.textFile("/home/coldface/data.txt")
 ```
 * 执行MapReduce操作
 ```
 scala> val workCounts=textFile.flatMap(line => line.split(" ")).map(word=>(word,1)).reduceByKey((a, b) => a + b)
 ```
 * 查看结果(查询每个单词出现的次数)
 ```
 scala> workCounts.collect()
[Stage 0:>                                                          (0 + 0[Stage 0:>                                                          (0 + 2[Stage 0:=============================>                             (1 + 1                                                                          res2: Array[(String, Int)] = Array((Hamstra,1), (Zhu	Databricks,1), (Liang	Hortonworks,1), (Das,1), (Sharma,1), (Haberman,1), (Webtrends,1), (Dai,1), (Andy,1), (Graves,1), (Bizo,1), (Dave,1), (IBM,2), (Databricks,,1), (Alluxio,,1), (Dudziak,1), (DB,1), (Or	Databricks,1), (B.V.,1), (Intel,2), (Tsai	Netflix,1), (ClearStory,1), (Vanzin	Cloudera,1), (Data,2), (MIT,1), (Arbor,1), (University,1), (Zaharia,1), (Ram,1), (Wenchen,1), (Alibaba,1), (Robert,1), (Fan	Databricks,1), (Thomas,2), (Yin,1), (UC,4), ("",52), (Meng	Databricks,1), (McNamara,1), (Health,1), (Databricks,6), (Patrick,1), (Josh,1), (Huai	Databricks,1), (Ryza	Clover,1), (Charles,1), (Reiss,1), (Matei,1), (Xia,1), (Ryan,1), (Bradley	Databricks,1), (Davies,1), (Facebook,1), (Michigan,,1), (Mridul,1), (Michael,1), (Rashid,1), (Marc...
 ```
 
<commit_msg>spark<commit_after>---
layout: post
title:  "Spark环境搭建"
keywords: "spark,hadoop"
description: "在linux环境搭建Spark"
category: spark 
tags: [spark]
---
###服务器环境
```
~$ cat /proc/version
Linux version 4.4.0-24-generic (buildd@lcy01-30) (gcc version 5.3.1 20160413 (Ubuntu 5.3.1-14ubuntu2.1) ) #43-Ubuntu SMP Wed Jun 8 19:25:16 UTC 2016
```
###Spark安装教程
####JDK安装
```
~$ java -version
java version "1.8.0_92"
Java(TM) SE Runtime Environment (build 1.8.0_92-b14)
Java HotSpot(TM) Server VM (build 25.92-b14, mixed mode)
```
####Scala安装
 * 版本:2.10.5
 * [下载地址](http://www.scala-lang.org/)
 * 环境变量配置
 
   ```
   export SCALA_HOME=/home/coldface/software/scala-2.10.5
   export PATH=${SCALA_HOME}/bin:$PATH
   ```
 * 使修改生效执行source /etc/profile
 * 在命令行输入scala,出现下列信息说明安装成功
 
 ```
 ~$ scala
 Welcome to Scala 2.10.5 (Java HotSpot(TM) Server VM, Java 1.8.0_92).
 Type in expressions for evaluation. Or try :help.

 scala>
```
####Spark安装
 * 版本:spark-1.6.1-bin-hadoop2.6
 * [下载地址](http://spark.apache.org/downloads.html)
 * 环境变量配置
 
 ```
 export SPARK_HOME=/home/coldface/software/spark-1.6.1-bin-hadoop2.6
 export PATH=${SPARK_HOME}/bin:$PATH
 ```
 * 使修改生效执行source /etc/profile
 * 在spark安装bin目录执行./spark-shell
 
 ```
 Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.1
      /_/

 Using Scala version 2.10.5 (Java HotSpot(TM) Server VM, Java 1.8.0_92)
 ```
####运行Spark WordCount示例
 * 在命令行输入:./spark-shell开启spark
 * 把输入文件加载进RDD
 
 ```
 scala> var textFile = sc.textFile("/home/coldface/data.txt")
 ```
 * 执行MapReduce操作
 
 ```
 scala> val workCounts=textFile.flatMap(line => line.split(" ")).map(word=>(word,1)).reduceByKey((a, b) => a + b)
 ```
 * 查看结果(查询每个单词出现的次数)
 
 ```
 scala> workCounts.collect()
[Stage 0:>                                                          (0 + 0[Stage 0:>                                                          (0 + 2[Stage 0:=============================>                             (1 + 1                                                                          res2: Array[(String, Int)] = Array((Hamstra,1), (Zhu	Databricks,1), (Liang	Hortonworks,1), (Das,1), (Sharma,1), (Haberman,1), (Webtrends,1), (Dai,1), (Andy,1), (Graves,1), (Bizo,1), (Dave,1), (IBM,2), (Databricks,,1), (Alluxio,,1), (Dudziak,1), (DB,1), (Or	Databricks,1), (B.V.,1), (Intel,2), (Tsai	Netflix,1), (ClearStory,1), (Vanzin	Cloudera,1), (Data,2), (MIT,1), (Arbor,1), (University,1), (Zaharia,1), (Ram,1), (Wenchen,1), (Alibaba,1), (Robert,1), (Fan	Databricks,1), (Thomas,2), (Yin,1), (UC,4), ("",52), (Meng	Databricks,1), (McNamara,1), (Health,1), (Databricks,6), (Patrick,1), (Josh,1), (Huai	Databricks,1), (Ryza	Clover,1), (Charles,1), (Reiss,1), (Matei,1), (Xia,1), (Ryan,1), (Bradley	Databricks,1), (Davies,1), (Facebook,1), (Michigan,,1), (Mridul,1), (Michael,1), (Rashid,1), (Marc...
 ```
 

{'fraction_non_alphanumeric': 0.22001047668936616, 'fraction_numerical': 0.057971014492753624, 'mean_word_length': 3.0281293952180026, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3296701', 'n_tokens_mistral': 2714, 'n_tokens_neox': 2531, 'n_words': 424}