Spark SQL 到底怎么搭建起来

Question

Spark SQL 到底怎么搭建起来

Answer 1

Spark on yarn已搭建好，开始使用SparkSql，做如下工作1、将Hive-site.xml copy至$SPARK_HOME/conf目录，注意配置hive.metastore.uris、hive.metastore.client.socket.timeout2、复制MySQL-connector-Java.jar 到$SPARK_HOME/lib目录3、配置spark-env.shexport SPARK_CLASSPATH=$SPARK_CLASSPATH:/usr/lib/spark/lib/mysql-connector-java.jar:/usr/lib/hive/lib/*export SPARK_CLASSPATH=$SPARK_CLASSPATH:/usr/lib/spark/lib/mysql-connector-java.jar:/usr/lib/hive/lib/*4、开始使用./bin/spark-sql --master yarn --num-executors 30 --executor-cores 4 --executor-memory 8g./bin/spark-sql --master yarn --num-executors 30 --executor-cores 4 --executor-memory 8g