安裝 Spark in Ubuntu 12.04 小記

最近由於研究需要

要開始研究 Apache Spark

先從大家常安裝的 Ubuntu 12.04 開始

之後來研究 CentOS 與 openSUSE

:-)

安裝 Spark in Ubuntu 12.04

OS: Ubuntu 12.04 LTS

java:  openjdk 7

scala: 2.9.1

spark: 1.4.0 with pre-build for Hadoop 2.6 and later

因為用的是 EasyCloud服務, 所以已經有先 sudo 為 root

先更新 apt-get

#apt-get   update

看看有沒有 jps  指令

#jps

如果沒有就安裝 openjdk 套件

#apt-get  install  openjdk-7-jdk

再次 測試 jps

#jps

查詢 scala 套件

#apt-cache   search   ^scala

安裝 scala

#apt-get   install  scala

查詢 scala 版本

# apt-cache   showpkg   scala

進入  scala shell 界面

# scala

Welcome to Scala version 2.9.1 (OpenJDK 64-Bit Server VM, Java 1.6.0_24).

Type in expressions to have them evaluated.

Type :help for more information.

scala>

按 Ctrl + d 離開

下載 Spark

https://spark.apache.org/downloads.html

選取 Spark 版本

選取 套件類型

點選 Download Spark後面的連結  spark-1.4.0-bin-hadoop2.6.tgz

選擇最接近的下載點

由於我是在 VM 裡面所以使用 wget 方式

# wget    http://www.gtlib.gatech.edu/pub/apache/spark/spark-1.4.0/spark-1.4.0-bin-hadoop2.6.tgz

# ls

spark-1.4.0-bin-hadoop2.6.tgz

解壓縮 spark

# tar   zxvf   spark-1.4.0-bin-hadoop2.6.tgz

觀察相關資訊

# ls

spark-1.4.0-bin-hadoop2.6  spark-1.4.0-bin-hadoop2.6.tgz

# ls spark-1.4.0-bin-hadoop2.6/bin/

beeline             pyspark2.cmd      spark-class       sparkR.cmd        spark-submit

beeline.cmd         pyspark.cmd       spark-class2.cmd  spark-shell       spark-submit2.cmd

load-spark-env.cmd  run-example       spark-class.cmd   spark-shell2.cmd  spark-submit.cmd

load-spark-env.sh   run-example2.cmd  sparkR            spark-shell.cmd

pyspark             run-example.cmd   sparkR2.cmd       spark-sql

# spark-1.4.0-bin-hadoop2.6/bin/spark-shell

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

15/07/05 01:54:41 INFO SecurityManager: Changing view acls to: root

15/07/05 01:54:41 INFO SecurityManager: Changing modify acls to: root

15/07/05 01:54:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)

15/07/05 01:54:41 INFO HttpServer: Starting HTTP Server

15/07/05 01:54:42 INFO Utils: Successfully started service 'HTTP class server' on port 49845.

Welcome to

     ____              __

    / __/__  ___ _____/ /__

   _\ \/ _ \/ _ `/ __/  '_/

  /___/ .__/\_,_/_/ /_/\_\   version 1.4.0

     /_/

Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.6.0_24)

Type in expressions to have them evaluated.

Type :help for more information.

15/07/05 01:54:48 WARN Utils: Your hostname, nchc-vm resolves to a loopback address: 127.0.1.1; using 10.2.0.8 instead (on interface eth0)

15/07/05 01:55:18 INFO SparkILoop: Created sql context (with Hive support)..

SQL context available as sqlContext.

scala>

先記起來

~ enjoy it