异常说明
SparkSession初始化一个lazy的HiveClient用于Hive MetaStore连接，一旦使用其内部就会实例化一个org.apache.spark.sql.hive.HiveSessionState对象，执行SessionState.start方法时会根据HiveConf里的配置执行一系列操作，包括创建一些临时的工作目录等。这些目录如果没有正确的权限会导致这个过程失败，致使连接不上Server.
客户端异常
hzyaoqin@hadoop980:~/spark-packages/spark-2.1.0-bin-ne-1.3.0-SNAPSHOT$ bin/beeline -u 'jdbc:hive2://hadoop980.hz.163.org:10001/default;principal=hive/hadoop980.hz.163.org@TEST.MAMMUT.NETEASE.COM;hive.server2.proxy.user=hzyaoqin;spark.sql.warehouse.dir=/user/hzyaoqin/warehouse'Connecting to jdbc:hive2://hadoop980.hz.163.org:10001/default;principal=hive/hadoop980.hz.163.org@TEST.MAMMUT.NETEASE.COM;hive.server2.proxy.user=hzyaoqin;spark.sql.warehouse.dir=/user/hzyaoqin/warehouse17/09/06 17:17:14 INFO jdbc.Utils: Supplied authorities: hadoop980.hz.163.org:1000117/09/06 17:17:14 INFO jdbc.Utils: Resolved authority: hadoop980.hz.163.org:1000117/09/06 17:17:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable17/09/06 17:17:15 INFO jdbc.HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://hadoop980.hz.163.org:10001/default;principal=hive/hadoop980.hz.163.org@TEST.MAMMUT.NETEASE.COM;hive.server2.proxy.user=hzyaoqin;spark.sql.warehouse.dir=/user/hzyaoqin/warehouseCan't overwrite cause with java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':Beeline version 1.2.1.spark2 by Apache Hive0: jdbc:hive2://hadoop980.hz.163.org:10001/de (closed)>
其中异常主体为，
Can't overwrite cause with java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':
， 实际意为初始化HiveSessionState时发生了不可名状的事情；进一步去看服务端日志；
注：HiveSession级别的日志没有做到客户端这边，只有Operation级别的日志可以看哦
服务端异常
Error opening session: org.apache.spark.SparkException: Failed Init SparkSession for user[hzyaoqin]java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState': at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl.createSparkSession(SparkHiveSessionImpl.scala:135) at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl.getOrCreateSparkSession(SparkHiveSessionImpl.scala:109) at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl.open(SparkHiveSessionImpl.scala:186) at org.apache.spark.sql.hive.thriftserver.multitenancy.ThriftServerSessionManager.openSession(ThriftServerSessionManager.scala:225) at org.apache.spark.sql.hive.thriftserver.multitenancy.ThriftServerCLIService.openSessionWithImpersonation(ThriftServerCLIService.scala:91) at org.apache.spark.sql.hive.thriftserver.multitenancy.ThriftClientCLIService.getSessionHandle(ThriftClientCLIService.scala:252) at org.apache.spark.sql.hive.thriftserver.multitenancy.ThriftClientCLIService.OpenSession(ThriftClientCLIService.scala:266) at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253) at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)Caused by: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState': at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:1120) at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:111) at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:110) at org.apache.spark.sql.SparkSession$Builder$$anonfun$11.apply(SparkSession.scala:888) at org.apache.spark.sql.SparkSession$Builder$$anonfun$11.apply(SparkSession.scala:888) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) at scala.collection.mutable.HashMap.foreach(HashMap.scala:99) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:888) at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl$$anon$1.run(SparkHiveSessionImpl.scala:124) at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl$$anon$1.run(SparkHiveSessionImpl.scala:118) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl.createSparkSession(SparkHiveSessionImpl.scala:118) ... 15 moreCaused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:1117) ... 31 moreCaused by: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog': at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:183) at org.apache.spark.sql.internal.SharedState.<init>(SharedState.scala:90) at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:102) at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:102) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession.scala:102) at org.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:101) at org.apache.spark.sql.internal.SessionState.<init>(SessionState.scala:157) at org.apache.spark.sql.hive.HiveSessionState.<init>(HiveSessionState.scala:32) ... 36 moreCaused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:180) ... 44 moreCaused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:264) at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:367) at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:271) at org.apache.spark.sql.hive.HiveExternalCatalog.<init>(HiveExternalCatalog.scala:65) ... 49 moreCaused by: java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=hzyaoqin, access=EXECUTE, inode="/tmp/hive/public/hzyaoqin/9869b00f-0629-4290-ad0f-71f3c9ed0566":hdfs:hadoop:drwxr-x--- at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1722) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:108) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3863) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:843) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415)
非常明显,hzyaoqin用户没有用户hdfs所属目录的执行权限；Permission denied: user=hzyaoqin, access=EXECUTE, inode="/tmp/hive/public/hzyaoqin/9869b00f-0629-4290-ad0f-71f3c9ed0566":hdfs:hadoop:drwxr-x---
这个目录为hive.exec.strachdir（缺省为/tmp/hive/public）下创建username/sessionId的临时目录，我们多租户下执行都是用hzyaoqin这个执行账号去真正执行的，这个目录如果是hdfs用户的，必然不是有本次session创建的，应该是以前的遗留目录
解决方案
1、删除/tmp/hive/public/hzyaoqin（如果这目录没啥用），重新连接
2、重新配置hive.exec.strachdir到另一个有权限的目录，可以在连接串中动态的session级别指定或classpath中的hive-site.xml进行server级别的全局设置，前者重新连接，后者重启server。
作者：风景不美
链接：https://www.jianshu.com/p/5ae027ba7c9c