Spring - Spark：jar/依赖项之间的冲突

Question

我有一个 Spring Web 应用程序（内置于 Maven），用它连接到我的 Spark 集群（4 个工作节点和 1 个主节点）和我的 cassandra 集群（4 个节点）。应用程序启动，工作线程与主节点通信，并且 cassandra 集群也在运行。但是，当我通过网络应用程序的界面进行 PCA（spark mllib）或任何其他计算（聚类、pearson、spearman）时，出现以下错误：

java.lang.ClassCastException：无法将 scala.collection.immutable.List$SerializationProxy 的实例分配给 scala.collection.Seq 类型的字段 org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_在 org.apache.spark.rdd.MapPartitionsRDD 的实例中

这个错误肯定是因为 spring-boot-maven-plugin 和 apache Spark 之间的 Jars 冲突。

Answer 1

解决了！我只需要更改插件即可。可能是jar包之间有冲突。这就是我所拥有的：&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <plugin>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <groupId>org.springframework.boot</groupId>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <artifactId>spring-boot-maven-plugin</artifactId>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <version>2.0.9.RELEASE</version>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <executions>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <execution>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <goals>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <goal>repackage</goal>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </goals>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </execution>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </executions>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <configuration>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <requiresUnpack>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <dependency>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <groupId>org.apache.spark</groupId>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <artifactId>spark-core_2.11</artifactId>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dependency>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <dependency>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <groupId>org.apache.spark</groupId>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <artifactId>spark-streaming_2.11</artifactId>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dependency>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </requiresUnpack>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </configuration>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </plugin>我将其替换为：<plugin>&nbsp; &nbsp; &nbsp; &nbsp; <groupId>org.apache.maven.plugins</groupId>&nbsp; &nbsp; &nbsp; &nbsp; <artifactId>maven-compiler-plugin</artifactId>&nbsp; &nbsp; &nbsp; &nbsp; <version>3.8.0</version>&nbsp; &nbsp; &nbsp; &nbsp; <configuration>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <source>1.8</source>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <target>1.8</target>&nbsp; &nbsp; &nbsp; &nbsp; </configuration>&nbsp; &nbsp; </plugin>和<plugin>&nbsp; &nbsp; &nbsp; &nbsp; <groupId>org.apache.maven.plugins</groupId>&nbsp; &nbsp; &nbsp; &nbsp; <artifactId>maven-shade-plugin</artifactId>&nbsp; &nbsp; &nbsp; &nbsp; <version>2.4.1</version>&nbsp; &nbsp; &nbsp; &nbsp; <dependencies>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <dependency>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <groupId>org.springframework.boot</groupId>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <artifactId>spring-boot-maven-plugin</artifactId>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <version>2.0.9.RELEASE</version>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dependency>&nbsp; &nbsp; &nbsp; &nbsp; </dependencies>&nbsp; &nbsp; &nbsp; &nbsp; <configuration>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <keepDependenciesWithProvidedScope>true</keepDependenciesWithProvidedScope>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <createDependencyReducedPom>true</createDependencyReducedPom>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <filters>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <filter>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <artifact>*:*</artifact>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <excludes>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <exclude>META-INF/*.SF</exclude>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <exclude>META-INF/*.DSA</exclude>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <exclude>META-INF/*.RSA</exclude>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </excludes>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </filter>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </filters>&nbsp; &nbsp; &nbsp; &nbsp; </configuration>&nbsp; &nbsp; &nbsp; &nbsp; <executions>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <execution>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <phase>package</phase>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <goals>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <goal>shade</goal>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </goals>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <configuration>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <transformers>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <transformer&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <resource>META-INF/spring.handlers</resource>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </transformer>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <transformer&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <resource>META-INF/spring.factories</resource>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </transformer>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <transformer&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <resource>META-INF/spring.schemas</resource>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </transformer>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <transformer&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <transformer&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <mainClass>com.example.app.Mtbapp1</mainClass>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </transformer>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </transformers>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </configuration>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </execution>&nbsp; &nbsp; &nbsp; &nbsp; </executions>&nbsp; &nbsp; </plugin>