如何在执行“org.apache.spark.sql.DataSet.collectAsList()

Question

我正在创建一个 Java RESTAPI Spring Boot 应用程序，它使用 spark 从服务器获取一些数据。当我尝试从数据集转换为列表时，它失败了。

我已经尝试使用 jdk8 和 jdk11 来编译和执行代码，但我得到相同的“java.lang.IllegalArgumentException：不支持的类文件主要版本 55”，过去，我通过更新 Java 版本解决了这个问题，但它是不为此工作。

我在用着：

• JDK 11.0.2

• 春季启动 2.1.4

• 火花 2.4.2

这是我正在执行的代码：

Dataset<Row> dataFrame = sparkSession.read().json("/home/data/*.json");

        dataFrame.createOrReplaceTempView("events");

        Dataset<Row> resultDataFrame = sparkSession.sql("SELECT * FROM events WHERE " + predicate); 

        Dataset<Event> eventDataSet = resultDataFrame.as(Encoders.bean(Event.class));

        return eventDataSet.collectAsList();

查询有效，实际上在调试时您可以在 resultDataFrame 和 eventDataSet 中看到信息。

我希望输出是正确的事件列表，但我得到了例外：

[http-nio-8080-exec-2] ERROR org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.IllegalArgumentException: Unsupported class file major version 55] with root cause

java.lang.IllegalArgumentException: Unsupported class file major version 55

    at org.apache.xbean.asm6.ClassReader.<init>(ClassReader.java:166)

    at org.apache.xbean.asm6.ClassReader.<init>(ClassReader.java:148)

    at org.apache.xbean.asm6.ClassReader.<init>(ClassReader.java:136)

    at org.apache.xbean.asm6.ClassReader.<init>(ClassReader.java:237)

    at org.apache.spark.util.ClosureCleaner$.getClassReader(ClosureCleaner.scala:49)

    at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:517)

    at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:500)

    at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)

.....

评论更新：对于 Java 8，我将 pom 更改为针对 java 8：

<java.version>1.8</java.version>

然后更新项目，maven clean，maven install 然后运行。获得相同的版本 55 错误

Answer 1

从 spark-core 依赖项中排除默认的 XBean 工件并添加最新版本的 XBean 工件，它对我有用。<dependencies>&nbsp; &nbsp; <dependency>&nbsp; &nbsp; &nbsp; &nbsp; <groupId>org.apache.xbean</groupId>&nbsp; &nbsp; &nbsp; &nbsp; <artifactId>xbean-asm6-shaded</artifactId>&nbsp; &nbsp; &nbsp; &nbsp; <version>4.10</version>&nbsp; &nbsp; </dependency>&nbsp; &nbsp; <dependency>&nbsp; &nbsp; &nbsp; &nbsp; <groupId>org.apache.spark</groupId>&nbsp; &nbsp; &nbsp; &nbsp; <artifactId>spark-core_2.11</artifactId>&nbsp; &nbsp; &nbsp; &nbsp; <version>2.4.1</version>&nbsp; &nbsp; &nbsp; &nbsp; <exclusions>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <exclusion>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <groupId>org.apache.xbean</groupId>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <artifactId>xbean-asm6-shaded</artifactId>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </exclusion>&nbsp; &nbsp; &nbsp; &nbsp; </exclusions>&nbsp; &nbsp; </dependency></dependencies>

Answer 2

问题的根本原因是我瞄准了错误的 JDK 的符号链接，这就是它不起作用的原因。JAVA_HOME 的目标是 jdk11 并且 eclipse 正在运行它。

Answer 3

由于大多数python开发人员为该项目生成了virutalenv，因此您可以使用以下代码段来检查pyspark工作所需的不同组件的版本。错误的原因是java版本不兼容。pyspark期望java version 1.8+而不是jdk-11。Major version 55对应于jdk-11您在此处看到的仅检查官方 spark 文档以了解版本兼容性。import subprocess# subprocess to find the java , scala and python versioncmd1 = "java -version"cmd2 = "scala -version"cmd3 = "python --version"cmd4 = "whoami"arr = [cmd1, cmd2, cmd3, cmd4]for cmd in arr:&nbsp; &nbsp; process = subprocess.Popen(cmd.split(" "), stdout=subprocess.PIPE,stderr=subprocess.PIPE )&nbsp; &nbsp; stdout,stderr=process.communicate()&nbsp; &nbsp; logging.info(stdout.decode("utf-8") + " | "&nbsp; + stderr.decode("utf-8"))logging.info(os.getenv("JAVA_HOME"))logging.info(os.getenv("HOME"))您将获得以下输出：INFO:root: | openjdk version "1.8.0_252"OpenJDK Runtime Environment (build 1.8.0_252-8u252-b09-1~18.04-b09)OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)INFO:root: | Scala code runner version 2.12.2 -- Copyright 2002-2017, LAMP/EPFL and Lightbend, Inc.INFO:root:Python 3.6.9INFO:root:training