alluxio安装 集群模式

tar -zxf alluxio-1.8.0-hadoop-2.7.tar.gzcd alluxio-1.8.0-hadoop-2.7

2、配置alluxio-site.properties

cp conf/alluxio-site.properties.template conf/alluxio-site.propertiescat > conf/alluxio-site.properties << EOF##关联hdfs配置文件alluxio.underfs.hdfs.configuration=/etc/hadoop/conf/core-site.xml:/etc/hadoop/conf/hdfs-site.xml##下面是高可用的配置alluxio.master.hostname=xg-sensorsdata1.xcar.com.cn
alluxio.zookeeper.enabled=truealluxio.zookeeper.address=xg-sensorsdata1.xcar.com.cn:2181,xg-sensorsdata2.xcar.com.cn:2181,xg-sensorsdata3.xcar.com.cn:2181#配置日志文件目录alluxio.master.journal.folder=hdfs://standyNamenode/alluxio/journal##配置用户模拟，允许yarn用户模拟任意用户alluxio.master.security.impersonation.yarn.users=*

3、配置集群woker节点

cat > conf /workers << EOF
xg-sensorsdata2.xcar.com.cn
xg-sensorsdata3.xcar.com.cn
xg-sensorsdata4.xcar.com.cn
EOF

4、修改日志路径

sed -i "s#\${alluxio.logs.dir}#\/var\/log\/alluxio#g" conf/log4j.properties`

5、分发安装包到woker节点

echo 'xg-sensorsdata2.xcar.com.cn
xg-sensorsdata3.xcar.com.cn
xg-sensorsdata4.xcar.com.cn' | xargs -I {} scp -r alluxio-1.8.0-hadoop-2.7 {}:/root/

集成hdfs

1、修改hdfs组件的core-site.xml，增加如下配置项

alluxio.zookeeper.address=xg-sensorsdata1.xcar.com.cn:2181,xg-sensorsdata2.xcar.com.cn:2181,xg-sensorsdata3.xcar.com.cn:2181
alluxio.zookeeper.enabled=truefs.alluxio.impl=alluxio.hadoop.FileSystem

2、分发客户端依赖包

cd /root/alluxio-1.8.0-hadoop-2.7/client
cp alluxio-1.8.0-client.jar /usr/hdp/current/hadoop-client/libecho 'xg-sensorsdata2.xcar.com.cn
xg-sensorsdata3.xcar.com.cn
xg-sensorsdata4.xcar.com.cn' | xargs -I {} scp alluxio-1.8.0-client.jar {}:/usr/hdp/current/hadoop-client/lib

3、重启影响的组件

集成tez（llap模式下跑不通）

cd /root/alluxio-1.8.0-hadoop-2.7/client
cp alluxio-1.8.0-client.jar /usr/hdp/current/tez-client/libecho 'xg-sensorsdata2.xcar.com.cn
xg-sensorsdata3.xcar.com.cn
xg-sensorsdata4.xcar.com.cn' | xargs -I {} scp alluxio-1.8.0-client.jar {}:/usr/hdp/current/tez-client/lib

修改tez-site.xml

tez.use.cluster.hadoop-libs=true  ##此配置需要集群的运行hive tez任务的节点都安装了tez-client

集成hive

1、分发客户端依赖包

cd /root/alluxio-1.8.0-hadoop-2.7/client
cp alluxio-1.8.0-client.jar /usr/hdp/current/hive-server2-hive2/libecho 'xg-sensorsdata2.xcar.com.cn
xg-sensorsdata3.xcar.com.cn
xg-sensorsdata4.xcar.com.cn' | xargs -I {} scp alluxio-1.8.0-client.jar {}:/usr/hdp/current/hive-server2-hive2/lib

2、重启hive

集成spark

1、分发alluxio-client包

cd /root/alluxio-1.8.0-hadoop-2.7/client
cp alluxio-1.8.0-client.jar /usr/hdp/current/spark2-client/jars/echo 'xg-sensorsdata2.xcar.com.cn
xg-sensorsdata3.xcar.com.cn
xg-sensorsdata4.xcar.com.cn' | xargs -I {} scp alluxio-1.8.0-client.jar {}:/usr/hdp/current/spark2-client/jars/

问题集：

1、ramfs is smaller than the configured size: ramfs size: 1024.00MB, configured size: 4096.00MB

alluxio无法动态修改ramfs的大小

Exception in thread "main" java.lang.RuntimeException: java.lang.IllegalStateException: ramfs is smaller than the configured size: ramfs size: 1024.00MB, configured size: 4096.00MB
at alluxio.worker.AlluxioWorkerProcess.<init>(AlluxioWorkerProcess.java:176)
at alluxio.worker.WorkerProcess$Factory.create(WorkerProcess.java:45)
at alluxio.worker.WorkerProcess$Factory.create(WorkerProcess.java:37)
at alluxio.worker.AlluxioWorker.main(AlluxioWorker.java:55)
Caused by: java.lang.IllegalStateException: ramfs is smaller than the configured size: ramfs size: 1024.00MB, configured size: 4096.00MB
at alluxio.worker.block.meta.StorageTier.checkEnoughMemSpace(StorageTier.java:146)
at alluxio.worker.block.meta.StorageTier.initStorageTier(StorageTier.java:100)
at alluxio.worker.block.meta.StorageTier.newStorageTier(StorageTier.java:169)
at alluxio.worker.block.BlockMetadataManager.<init>(BlockMetadataManager.java:67)
at alluxio.worker.block.BlockMetadataManager.createBlockMetadataManager(BlockMetadataManager.java:82)
at alluxio.worker.block.TieredBlockStore.<init>(TieredBlockStore.java:128)
at alluxio.worker.block.DefaultBlockWorker.<init>(DefaultBlockWorker.java:129)
at alluxio.worker.block.BlockWorkerFactory.create(BlockWorkerFactory.java:43)
at alluxio.worker.block.BlockWorkerFactory.create(BlockWorkerFactory.java:26)
at alluxio.worker.AlluxioWorkerProcess$1.call(AlluxioWorkerProcess.java:129)
at alluxio.worker.AlluxioWorkerProcess$1.call(AlluxioWorkerProcess.java:125)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

作者：ron_yang
链接：https://www.jianshu.com/p/5f5dd1583154