使用 spark-submit 部署应用程序：应用程序已添加到调度程序，但尚未激活

Question

我有带有 Linux Centos 12G 内存的 VirtualBox。我需要将 2 个应用程序部署到以非分布式配置运行的 hadoop。这是我的 YARN 配置：

<configuration>

<property>

    <name>yarn.nodemanager.pmem-check-enabled</name>

    <value>false</value>

</property>

<property>

   <name>yarn.nodemanager.aux-services</name>

   <value>mapreduce_shuffle</value>

</property>

<property>

   <name>yarn.resourcemanager.address</name>

   <value>0.0.0.0:8032</value>

</property>

<property>

  <name>yarn.scheduler.maximum-allocation-vcores</name>

  <value>130</value>

</property>

<property>

   <name>yarn.nodemanager.vmem-check-enabled</name>

   <value>false</value>

   <description>Whether virtual memory limits will be enforced for containers</description>

</property>

<property>

   <name>yarn.scheduler.maximum-allocation-mb</name>

   <value>4048</value>

</property>

<property>

   <name>yarn.nodemanager.vmem-pmem-ratio</name>

   <value>1</value>

   <description>Ratio between virtual memory to physical memory when

setting memory limits for containers</description>

</property>

<property>

   <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>

   <value>1</value>

</property>

</configuration>

我部署了第一个应用程序，它运行正常：

spark-submit --master yarn --deploy-mode 客户端 --name OryxBatchLayer-ALSExample --class com.cloudera.oryx.batch.Main --files oryx.conf --driver-memory 500m --driver-java-options "-Dconfig.file=oryx.conf" --executor-memory 500m --executor-cores 1 --conf spark.executor.extraJavaOptions="-Dconfig.file=oryx.conf" --conf spark.ui.port= 4040 --conf spark.io.compression.codec=lzf --conf spark.logConf=true --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.speculation=true --conf spark. ui.showConsoleProgress=false --conf spark.dynamicAllocation.enabled=false --num-executors=1 oryx-batch-2.8.0-SNAPSHOT.jar

8088 处的 YARN 管理器表示我正在使用 8 个 vcore 中的 2 个和 8g 内存中的 2 个：

Answer 1

修复是编辑~/hadoop-3.1.0/etc/hadoop/capacity-scheduler.xml并将 .1 更新为 1：<property>&nbsp; &nbsp; <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>&nbsp; &nbsp; <value>1</value>&nbsp; &nbsp; <description>&nbsp; &nbsp; &nbsp; Maximum percent of resources in the cluster which can be used to run&nbsp; &nbsp; &nbsp; application masters i.e. controls number of concurrent running&nbsp; &nbsp; &nbsp; applications.&nbsp; &nbsp; </description>&nbsp; </property>