基于Docker的Spark-Hbase-Hadoop的HA高可用环境
基于Docker的Spark-Hbase-Hadoop的HA高可用环境
为了配置一个基于 Hadoop 和 HBase 的分布式系统，以下是准备 Dockerfile 和补充必要环境设置的步骤。
资源配置
- 3 个 Hadoop Master 节点
- 3 个 Hadoop Worker 节点
- 3 个 ZooKeeper 节点
配置ZooKeeper
pull官方的ZooKeeper镜像
docker pull zookeeper
写docker-compose.yml
version: '3.1'
services:
zoo1:
image: zookeeper:3.7.1-temurin
container_name: zoo1
restart: always
hostname: zoo1
ports:
- 2181:2181
environment:
ZOO_MY_ID: 1
ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181
networks:
zookeeper-cluster:
ipv4_address: 10.10.1.10
volumes:
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
zoo2:
image: zookeeper:3.7.1-temurin
container_name: zoo2
restart: always
hostname: zoo2
ports:
- 2182:2181
environment:
ZOO_MY_ID: 2
ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181
networks:
zookeeper-cluster:
ipv4_address: 10.10.1.11
volumes:
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
zoo3:
image: zookeeper:3.7.1-temurin
container_name: zoo3
restart: always
hostname: zoo3
ports:
- 2183:2181
environment:
ZOO_MY_ID: 3
ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181
networks:
zookeeper-cluster:
ipv4_address: 10.10.1.12
volumes:
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
networks:
zookeeper-cluster:
name: zookeeper-cluster
ipam:
config:
- subnet: "10.10.1.0/24"
运行Docker-compose
docker-compose up -d
构建镜像spark-hbase-sshpem:3.4
ssh 协议选择的是PEM 原因是hadoop3.4在故障转移的时候，对ssh加密协议更加严格
准备Dockerfile
Dockerfile 本身只配置了基础系统环境和 JDK。
# 指定路径下创建文件Dockerfile
touch /路径/Dockerfile
Dockerfile文件内容
Dockerfile for HBase and Hadoop Setup (ARM64 架构)
FROM ubuntu:22.04
# 环境变量设置
ENV HADOOP_HOME=/opt/hadoop
ENV HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
ENV HADOOP_COMMON_HOME=$HADOOP_HOME
ENV HADOOP_HDFS_HOME=$HADOOP_HOME
ENV YARN_HOME=$HADOOP_HOME
ENV HADOOP_MAPRED_HOME=$HADOOP_HOME
ENV HBASE_HOME=/opt/hbase
ENV HBASE_CONF_DIR=$HBASE_HOME/conf
ENV SPARK_HOME=/opt/spark
ENV SPARK_CONF_DIR=$SPARK_HOME/conf
ENV JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
ENV PYSPARK_PYTHON=/root/anaconda3/envs/pyspark/bin/python3.8
ENV PYSPARK_DRIVER_PYTHON=$PYSPARK_PYTHON
# 更新 PATH 环境变量
ENV PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH
# 以 root 用户执行
USER root
# 更新并安装依赖包
RUN apt-get update && \
apt-get install -y sudo openjdk-8-jdk openssh-server openssh-client vim psmisc && \
ssh-keygen -t rsa -m PEM -P '' -f ~/.ssh/id_rsa && \
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys && \
chmod 0600 ~/.ssh/authorized_keys && \
mkdir -p /data/hdfs && \
mkdir -p /data/hdfs/journal/node/local/data
# 启动 SSH 服务
RUN service ssh start
# 暴露端口
EXPOSE 9870 9868 9864 9866 8088 8020 16000 16010 16020 22
# 容器启动时启动 SSH
CMD ["/usr/sbin/sshd", "-D"]
Dockerfile for HBase and Hadoop Setup (AMD64 架构)
FROM ubuntu:22.04
# 环境变量设置
ENV HADOOP_HOME=/opt/hadoop
ENV HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
ENV HADOOP_COMMON_HOME=$HADOOP_HOME
ENV HADOOP_HDFS_HOME=$HADOOP_HOME
ENV YARN_HOME=$HADOOP_HOME
ENV HADOOP_MAPRED_HOME=$HADOOP_HOME
ENV HBASE_HOME=/opt/hbase
ENV HBASE_CONF_DIR=$HBASE_HOME/conf
ENV SPARK_HOME=/opt/spark
ENV SPARK_CONF_DIR=$SPARK_HOME/conf
ENV JAVA_HOME=/usr/lib/jvm/java-8-openjdk-arm64
ENV PYSPARK_PYTHON=/root/anaconda3/envs/pyspark/bin/python3.8
ENV PYSPARK_DRIVER_PYTHON=$PYSPARK_PYTHON
# 更新 PATH 环境变量
ENV PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH
# 以 root 用户执行
USER root
# 更新并安装依赖包
RUN apt-get update && \
apt-get install -y sudo openjdk-8-jdk openssh-server openssh-client vim psmisc && \
ssh-keygen -t rsa -m PEM -P '' -f ~/.ssh/id_rsa && \
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys && \
chmod 0600 ~/.ssh/authorized_keys && \
mkdir -p /data/hdfs && \
mkdir -p /data/hdfs/journal/node/local/data
# 启动 SSH 服务
RUN service ssh start
# 暴露端口
EXPOSE 9870 9868 9864 9866 8088 8020 16000 16010 16020 22
# 容器启动时启动 SSH
CMD ["/usr/sbin/sshd", "-D"]
启动Dockerfile
docker build -t mid-spark-hbase-sshpem:3.4
下载Anaconda配置Python环境
访问 Anaconda 下载页面 并选择适合的版本。
利用wget命令下载指定文件
wget https://repo.anaconda.com/archive/Anaconda3-2023.07-2-Linux-x86_64.sh
bash ./Anaconda3-2023.07-2-Linux-x86_64.sh
# 加入bash后会自动配置环境
运行一下中间过度容器
docker run -d --name mid-image mid-spark-hbase-sshpem:3.4
- 进入容器安装Anaconda
# 拷贝Anaconda进入mid-image
docker cp ./Anaconda3-2023.07-2-Linux-x86_64.sh mid-image:/
docker exec -it mid-image bash
chmod a+x /Anaconda3-2023.07-2-Linux-x86_64.sh
bash Anaconda3-2023.07-2-Linux-x86_64.sh
- Anaconda运行依赖部分Linux库，官方给出Debain了需要下载的
apt-get install libgl1-mesa-glx libegl1-mesa libxrandr2 libxrandr2 libxss1 libxcursor1 libxcomposite1 libasound2 libxi6 libxtst6
- 出现安装成功但是没有conda的虚拟环境解决方案
source ~/.bashrc
# 让新的配置生效
5 .创建一个虚拟环境
conda create -n pyspark python=3.8
将中间容器打包为一个image
- 创建最终的镜像
docker commit mid-image spark-hbase-sshpem:3.4
- 删除容器
# 方法一
docker stop mid-image
docker rm mid-image
# 方法二
docker rm -f mid-image
安装 Hadoop 、HBase 和 Spark
Dockerfile 本身只配置了基础系统环境和 JDK。接下来需要手动下载和安装 Hadoop 、 HBase 和 Spark。
下载和安装 Hadoop
- 访问 Hadoop 下载页面 并选择适合你的版本。
- 可以使用2024年9月6号已经验证hadoop可用性的下载：
wget https://dlcdn.apache.org/hadoop/common/hadoop-3.4.0/hadoop-3.4.0.tar.gz
下载和安装 HBase
- 访问 HBase 下载页面 并选择适合的版本。
- 可以使用2024年9月6号已经验证hbase可用性的下载：
wget https://dlcdn.apache.org/hbase/2.5.10/hbase-2.5.10-hadoop3-bin.tar.gz
下载Spark文件
访问 Spark 下载页面 并选择适合的版本。
利用wget命令下载指定文件
wget https://archive.apache.org/dist/spark/spark-3.4.1/spark-3.4.1-bin-hadoop3.tgz
下载玩并且解压到/opt/docker-data下，当然你可以自己选择目录
如何没有权限请修用sudo 创建以及移动文件夹，最终用sudo chmod 777 -R /opt/docker-data 完成权限修改
准备创建spark-hbase容器
创建一个docker-compose.yml
version: '3'
services:
hadoop-master1:
image: spark-hbase-sshpem:3.4
container_name: hadoop-master1
hostname: hadoop-master1
stdin_open: true
tty: true
command:
- sh
- -c
- |
/usr/sbin/sshd -D
volumes:
- type: bind
source: /opt/docker-data/hadoop-3.4.0
target: /opt/hadoop
- type: bind
source: /opt/docker-data/hbase-2.5.10-hadoop3
target: /opt/hbase
- type: bind
source: /opt/docker-data/spark-3.4.1-bin-hadoop3
target: /opt/spark
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
ports:
- "8020:8020"
- "8042:8042"
- "9870:9870"
- "8088:8088"
- "8032:8032"
- "10020:10020"
- "16000:16000"
- "16010:16010"
networks:
zookeeper-cluster:
ipv4_address: 10.10.1.20
hadoop-master2:
image: spark-hbase-sshpem:3.4
container_name: hadoop-master2
hostname: hadoop-master2
stdin_open: true
tty: true
command:
- sh
- -c
- |
/usr/sbin/sshd -D
volumes:
- type: bind
source: /opt/docker-data/hadoop-3.4.0
target: /opt/hadoop
- type: bind
source: /opt/docker-data/hbase-2.5.10-hadoop3
target: /opt/hbase
- type: bind
source: /opt/docker-data/spark-3.4.1-bin-hadoop3
target: /opt/spark
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
ports:
- "28020:8020"
- "18042:8042"
- "29870:9870"
- "28088:8088"
- "28032:8032"
- "20020:10020"
networks:
zookeeper-cluster:
ipv4_address: 10.10.1.21
hadoop-master3:
image: spark-hbase-sshpem:3.4
container_name: hadoop-master3
hostname: hadoop-master3
stdin_open: true
tty: true
command:
- sh
- -c
- |
/usr/sbin/sshd -D
volumes:
- type: bind
source: /opt/docker-data/hadoop-3.4.0
target: /opt/hadoop
- type: bind
source: /opt/docker-data/hbase-2.5.10-hadoop3
target: /opt/hbase
- type: bind
source: /opt/docker-data/spark-3.4.1-bin-hadoop3
target: /opt/spark
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
ports:
- "38020:8020"
- "28042:8042"
- "39870:9870"
- "38088:8088"
- "38032:8032"
- "30020:10020"
networks:
zookeeper-cluster:
ipv4_address: 10.10.1.22
hadoop-worker1:
image: spark-hbase-sshpem:3.4
container_name: hadoop-worker1
hostname: hadoop-worker1
stdin_open: true
tty: true
command:
- sh
- -c
- |
/usr/sbin/sshd -D
volumes:
- type: bind
source: /opt/docker-data/hadoop-3.4.0
target: /opt/hadoop
- type: bind
source: /opt/docker-data/hbase-2.5.10-hadoop3
target: /opt/hbase
- type: bind
source: /opt/docker-data/spark-3.4.1-bin-hadoop3
target: /opt/spark
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
ports:
- "9867:9867"
- "38042:8042"
- "9866:9866"
- "9865:9865"
- "9864:9864"
networks:
zookeeper-cluster:
ipv4_address: 10.10.1.23
hadoop-worker2:
image: spark-hbase-sshpem:3.4
container_name: hadoop-worker2
hostname: hadoop-worker2
stdin_open: true
tty: true
command:
- sh
- -c
- |
/usr/sbin/sshd -D
volumes:
- type: bind
source: /opt/docker-data/hadoop-3.4.0
target: /opt/hadoop
- type: bind
source: /opt/docker-data/hbase-2.5.10-hadoop3
target: /opt/hbase
- type: bind
source: /opt/docker-data/spark-3.4.1-bin-hadoop3
target: /opt/spark
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
ports:
- "29867:9867"
- "48042:8042"
- "29866:9866"
- "29865:9865"
- "29864:9864"
networks:
zookeeper-cluster:
ipv4_address: 10.10.1.24
hadoop-worker3:
image: spark-hbase-sshpem:3.4
container_name: hadoop-worker3
hostname: hadoop-worker3
stdin_open: true
tty: true
command:
- sh
- -c
- |
/usr/sbin/sshd -D
volumes:
- type: bind
source: /opt/docker-data/hadoop-3.4.0
target: /opt/hadoop
- type: bind
source: /opt/docker-data/hbase-2.5.10-hadoop3
target: /opt/hbase
- type: bind
source: /opt/docker-data/spark-3.4.1-bin-hadoop3
target: /opt/spark
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
ports:
- "39867:9867"
- "58042:8042"
- "39866:9866"
- "39865:9865"
- "39864:9864"
networks:
zookeeper-cluster:
ipv4_address: 10.10.1.25
networks:
zookeeper-cluster:
external: true
运行Docker-compose
docker-compose up -d
修改Hadoop 、 Hbase 和 Spark的配置文件
Hadoop先关配置文件
1. hadoop-env.sh
# 末尾添加
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_MAPRED_HOME=/opt/hadoop
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export HDFS_RESOURCEMANAGER_USER=root
export HDFS_NODEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
export YARN_RESOURCEMANAGER_USER=root
export HDFS_ZKFC_USER=root
export HDFS_JOURNALNODE_USER=root
2. core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<!-- HDFS 默认文件系统设置 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://mycluster</value>
</property>
<!-- Zookeeper Quorum 设置 -->
<property>
<name>ha.zookeeper.quorum</name>
<value>zoo1:2181,zoo2:2181,zoo3:2181</value>
</property>
<!-- Hadoop Proxy 用户配置 -->
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
<!-- 启用 WebHDFS -->
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<!-- HTTP/HTTPS 配置 -->
<property>
<name>dfs.http.policy</name>
<value>HTTP_ONLY</value>
</property>
<!-- CORS 支持 -->
<property>
<name>dfs.webhdfs.rest-cors-enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.webhdfs.rest-cors-allowed-origins</name>
<value>*</value>
</property>
<property>
<name>dfs.webhdfs.rest-cors-allowed-methods</name>
<value>GET, POST, PUT, DELETE, OPTIONS</value>
</property>
<property>
<name>dfs.webhdfs.rest-cors-allowed-headers</name>
<value>Content-Type, Authorization, Access-Control-Allow-Origin</value>
</property>
<property>
<name>dfs.webhdfs.rest-cors-allowed-credentials</name>
<value>true</value>
</property>
</configuration>
3. hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<!-- 配置副本数量 -->
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<!-- 禁用HDFS权限检查 -->
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<!-- 设置NameNode存储目录 -->
<property>
<name>dfs.namenode.name.dir</name>
<value>/data/hdfs/namenode</value>
</property>
<!-- 设置DataNode存储目录 -->
<property>
<name>dfs.namenode.data.dir</name>
<value>/data/hdfs/datanode</value>
</property>
<!-- 自动失效切换时的首选活跃节点 -->
<!--
<property>
<name>dfs.ha.automatic-failover.preferred-active</name>
<value>nn1</value>
</property>
-->
<!-- 配置高可用集群 -->
<property>
<name>dfs.nameservices</name>
<value>mycluster</value>
</property>
<property>
<name>dfs.ha.namenodes.mycluster</name>
<value>nn1,nn2,nn3</value>
</property>
<!-- NameNode 节点的 RPC 地址 -->
<property>
<name>dfs.namenode.rpc-address.mycluster.nn1</name>
<value>hadoop-master1:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.mycluster.nn2</name>
<value>hadoop-master2:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.mycluster.nn3</name>
<value>hadoop-master3:8020</value>
</property>
<!-- NameNode 节点的 HTTP 地址 -->
<property>
<name>dfs.namenode.http-address.mycluster.nn1</name>
<value>hadoop-master1:9870</value>
</property>
<property>
<name>dfs.namenode.http-address.mycluster.nn2</name>
<value>hadoop-master2:9870</value>
</property>
<property>
<name>dfs.namenode.http-address.mycluster.nn3</name>
<value>hadoop-master3:9870</value>
</property>
<!-- <property>
<name>dfs.ha.automatic-failover.preferred-active</name>
<value>nn1</value>
</property> -->
<!-- 共享编辑日志的目录 -->
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://hadoop-master1:8485;hadoop-master2:8485;hadoop-master3:8485/mycluster</value>
</property>
<!-- 配置客户端失效切换 -->
<property>
<name>dfs.client.failover.proxy.provider.mycluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!-- 配置在安全模式下不允许成为活跃节点 -->
<!--
<property>
<name>dfs.ha.nn.not-become-active-in-safemode</name>
<value>true</value>
</property>
-->
<!-- 日志节点的目录 -->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/data/hdfs/journal/node/local/data</value>
</property>
<!-- 启用自动失效切换 -->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<!-- SSH fencing 配置 -->
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>30000</value> <!-- 设置为 30 秒 -->
</property>
<property>
<name>dfs.ha.fencing.ssh.timeout</name>
<value>60000</value> <!-- 设置为 60 秒 -->
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<property>
<name>dfs.hosts</name>
<value>/etc/hadoop/conf/dfs.include</value> <!-- 包含的 DataNode 节点 -->
</property>
<property>
<name>dfs.hosts.exclude</name>
<value>/etc/hadoop/conf/dfs.exclude</value> <!-- 排除的 DataNode 节点 -->
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>zoo1:2181,zoo2:2181,zoo3:2181</value>
</property>
</configuration>
4. yarn-site.xml
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>cluster1</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2,rm3</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>hadoop-master1</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>hadoop-master2</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm3</name>
<value>hadoop-master3</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>hadoop-master1:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>hadoop-master2:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm3</name>
<value>hadoop-master3:8088</value>
</property>
<property>
<name>hadoop.zk.address</name>
<value>zoo1:2181,zoo2:2181,zoo3:2181</value>
</property>
</configuration>
5. workers
hadoop-worker1
hadoop-worker2
hadoop-worker3
Hbase相关配置
1. hbase-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- 设置集群为分布式模式 -->
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<!-- 设置 HBase 根目录，确保路径指向 HDFS 的高可用配置 -->
<!-- <property>
<name>hbase.rootdir</name>
<value>hdfs://hadoop-master1:8020/hbase</value>
</property> -->
<property>
<name>hbase.rootdir</name>
<value>hdfs://mycluster/hbase</value>
</property>
<!-- 临时目录设置 -->
<property>
<name>hbase.tmp.dir</name>
<value>./tmp</value>
</property>
<!-- 允许使用本地文件系统 -->
<property>
<name>hbase.unsafe.stream.capability.enforce</name>
<value>false</value>
</property>
<!-- ZooKeeper 配置 -->
<property>
<name>hbase.zookeeper.quorum</name>
<value>zoo1,zoo2,zoo3</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<!-- 启用 ZooKeeper 管理 HBase Master 的故障转移 -->
<property>
<name>hbase.master.wait.on.zk</name>
<value>true</value>
</property>
<!-- 设置 ZooKeeper 中用于监控 HBase Master 状态的路径 -->
<property>
<name>hbase.master.znode</name>
<value>/hbase/master</value>
</property>
</configuration>
2. hbase-env.sh
# 末尾添加
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HBASE_MANAGES_ZK=false
export HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP="true"
export HADOOP_HOME=/opt/hadoop
export HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
3. backup-masters
hadoop-master2
hadoop-master1
hadoop-master3
4. regionservers
hadoop-master2
hadoop-master3
hadoop-master1
添加Hadoop的配置文件
将 core-site.xml 和 hdfs-site.xml 复制到 Hbase 下的 conf 目录中
Spark中的配置文件
- spark-env.sh
# Java和Hadoop相关环境变量
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
YARN_CONF_DIR=/opt/hadoop/etc/hadoop
# Spark Worker节点配置
SPARK_WORKER_CORES=1
SPARK_WORKER_MEMORY=1g
SPARK_WORKER_PORT=7078
SPARK_WORKER_WEBUI_PORT=8081
## 设置历史服务器
# 将Spark程序运行的历史日志存储到HDFS的/sparklog目录中
SPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory=hdfs://mycluster/sparklog/ -Dspark.history.fs.cleaner.enabled=true"
# 启用Zookeeper来管理Master高可用性
SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER \
-Dspark.deploy.zookeeper.url=zoo1:2181,zoo2:2181,zoo3:2181 \
-Dspark.deploy.zookeeper.dir=/spark"
- spark-defaults.conf
spark.eventLog.enabled true
# 设置spark日志记录的路径
spark.eventLog.dir hdfs://mycluster/sparklog/
# 设置spark日志是否启动压缩
spark.eventLog.compress true
# 使用Zookeeper作为恢复模式来启用Master节点的高可用性
spark.deploy.recoveryMode ZOOKEEPER
# 设置Zookeeper集群的URL，用逗号分隔多个Zookeeper节点
spark.deploy.zookeeper.url zoo1:2181,zoo2:2181,zoo3:2181
# 指定Zookeeper存储Spark Master状态的路径
spark.deploy.zookeeper.dir /spark
- log4j2.properties
#修改日志记录登记为WARN
rootLogger.level = WARN
- workers
hadoop-worker1
hadoop-worker2
hadoop-worker3
- fairscheduler.xml.template
<?xml version="1.0"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<allocations>
<pool name="production">
<schedulingMode>FAIR</schedulingMode>
<weight>1</weight>
<minShare>2</minShare>
</pool>
<pool name="test">
<schedulingMode>FIFO</schedulingMode>
<weight>2</weight>
<minShare>3</minShare>
</pool>
</allocations>
初始化Hadoop
- 初始化脚本
# SSH 配置检查
docker exec hadoop-master1 ssh -o StrictHostKeyChecking=no hadoop-master2 exit
docker exec hadoop-master1 ssh -o StrictHostKeyChecking=no hadoop-master3 exit
docker exec hadoop-master2 ssh -o StrictHostKeyChecking=no hadoop-master1 exit
docker exec hadoop-master2 ssh -o StrictHostKeyChecking=no hadoop-master3 exit
docker exec hadoop-master3 ssh -o StrictHostKeyChecking=no hadoop-master1 exit
docker exec hadoop-master3 ssh -o StrictHostKeyChecking=no hadoop-master2 exit
# 启动 journalnode
docker exec hadoop-master1 /opt/hadoop/bin/hdfs --daemon start journalnode
docker exec hadoop-master2 /opt/hadoop/bin/hdfs --daemon start journalnode
docker exec hadoop-master3 /opt/hadoop/bin/hdfs --daemon start journalnode
# 可以不启动 worker 节点上的 journalnode
docker exec hadoop-worker1 /opt/hadoop/bin/hdfs --daemon start journalnode
docker exec hadoop-worker2 /opt/hadoop/bin/hdfs --daemon start journalnode
docker exec hadoop-worker3 /opt/hadoop/bin/hdfs --daemon start journalnode
# 初始化 NameNode
docker exec hadoop-master1 bash /opt/hadoop/bin/hdfs namenode -format
docker exec hadoop-master1 /opt/hadoop/bin/hdfs --daemon start namenode
# Bootstrap Standby
docker exec hadoop-master2 bash /opt/hadoop/bin/hdfs namenode -bootstrapStandby
docker exec hadoop-master2 /opt/hadoop/bin/hdfs --daemon start namenode
docker exec hadoop-master3 bash /opt/hadoop/bin/hdfs namenode -bootstrapStandby
docker exec hadoop-master3 /opt/hadoop/bin/hdfs --daemon start namenode
# 停止 DFS
docker exec hadoop-master1 /opt/hadoop/sbin/stop-dfs.sh
#首次格式化
docker exec hadoop-master1 /opt/hadoop/bin/hdfs zkfc -formatZK
# Zookeeper 数据重新格式化（如果需要）
# docker exec -it hadoop-master1 bash
# /opt/hadoop/bin/hdfs zkfc -formatZK
# exit
# 启动 zkfc 和 DFS/YARN
docker exec hadoop-master1 /opt/hadoop/bin/hdfs --daemon start zkfc
docker exec hadoop-master1 /opt/hadoop/sbin/start-dfs.sh
docker exec hadoop-master1 /opt/hadoop/sbin/start-yarn.sh
- 运行hadoop服务脚本
#! /bin/bash
echo "starting all journalnode"
docker exec hadoop-master1 /opt/hadoop/bin/hdfs --daemon start journalnode
docker exec hadoop-master2 /opt/hadoop/bin/hdfs --daemon start journalnode
docker exec hadoop-master3 /opt/hadoop/bin/hdfs --daemon start journalnode
docker exec hadoop-worker1 /opt/hadoop/bin/hdfs --daemon start journalnode
docker exec hadoop-worker2 /opt/hadoop/bin/hdfs --daemon start journalnode
docker exec hadoop-worker3 /opt/hadoop/bin/hdfs --daemon start journalnode
echo "starting hadoop-master1..."
docker exec hadoop-master1 /opt/hadoop/bin/hdfs --daemon start namenode
sleep 2
echo "starting hadoop-master2..."
docker exec hadoop-master2 /opt/hadoop/bin/hdfs --daemon start namenode
echo "starting hadoop-master3..."
docker exec hadoop-master3 /opt/hadoop/bin/hdfs --daemon start namenode
sleep 2
echo "starting zkfc..."
docker exec hadoop-master1 /opt/hadoop/bin/hdfs --daemon start zkfc
echo "starting dfs..."
docker exec hadoop-worker1 /opt/hadoop/sbin/start-dfs.sh
docker exec hadoop-worker2 /opt/hadoop/sbin/start-dfs.sh
docker exec hadoop-worker3 /opt/hadoop/sbin/start-dfs.sh
sleep 3
echo "starting yarn..."
docker exec hadoop-master1 /opt/hadoop/sbin/start-yarn.sh
docker exec hadoop-master2 /opt/hadoop/sbin/start-yarn.sh
docker exec hadoop-master3 /opt/hadoop/sbin/start-yarn.sh
echo "Done!"
- 停止hadoop脚本
最有效的停止方法是停止你的docker集群
echo "stoping yarn..."
docker exec hadoop-master1 /opt/hadoop/sbin/stop-yarn.sh
docker exec hadoop-master2 /opt/hadoop/sbin/stop-yarn.sh
docker exec hadoop-master3 /opt/hadoop/sbin/stop-yarn.sh
sleep 3
echo "stoping dfs..."
docker exec hadoop-worker1 /opt/hadoop/sbin/stop-dfs.sh
docker exec hadoop-worker2 /opt/hadoop/sbin/stop-dfs.sh
docker exec hadoop-worker3 /opt/hadoop/sbin/stop-dfs.sh
echo "stoping zkfc..."
docker exec hadoop-master1 /opt/hadoop/bin/hdfs --daemon stop zkfc
sleep 2
echo "stoping hadoop-master3..."
docker exec hadoop-master3 /opt/hadoop/bin/hdfs --daemon stop namenode
echo "stoping hadoop-master2..."
docker exec hadoop-master2 /opt/hadoop/bin/hdfs --daemon stop namenode
sleep 2
echo "stoping hadoop-master1..."
docker exec hadoop-master1 /opt/hadoop/bin/hdfs --daemon stop namenode
echo "stoping all journalnode"
docker exec hadoop-worker3 /opt/hadoop/bin/hdfs --daemon stop journalnode
docker exec hadoop-worker2 /opt/hadoop/bin/hdfs --daemon stop journalnode
docker exec hadoop-worker1 /opt/hadoop/bin/hdfs --daemon stop journalnode
docker exec hadoop-master3 /opt/hadoop/bin/hdfs --daemon stop journalnode
docker exec hadoop-master2 /opt/hadoop/bin/hdfs --daemon stop journalnode
docker exec hadoop-master1 /opt/hadoop/bin/hdfs --daemon stop journalnode
测试Hbase功能
登录到HbaseMaster shell中
# 随机登录一个docker容器
docker exec -it hadoop-master2 bash
./opt/hbase/bin/start-hbase.sh
./opt/hbase/bin/hbase shell
测试status命令
status
输出：
1 active master, 2 backup masters, 3 servers, 0 dead, 0.0000 average load
测试Spark功能
测试standalone
# 登录到active的hadoop服务上
./opt/spark/bin/pyspark
# 登录上后输入
spark
Spark运行成功
Spark On HA模式
启动服务
# 顺便就把woker节点启动了
docker exec hadoop-master2 /opt/spark/sbin/start-all.sh
docker exec hadoop-master3 /opt/spark/sbin/start-master.sh
docker exec hadoop-master1 /opt/spark/sbin/start-master.sh
Spark运行成功
Status: ALIVE 查看hadoop-master 1 ~ 3 , 一定会有一台服务是
报错处理
- 未有创建sparklog目录在hdfs中
java.io.FileNotFoundException: File does not exist: hdfs://mycluster/sparklog
在hdfs系统中创建即可
结语
到这里，基于Docker的spark-hbase-hadoop的HA高可用环境就搭建完成了
本文是原创文章，采用 CC BY-NC-ND 4.0 协议，完整转载请注明来自 David
阅读建议
评论
匿名评论
隐私政策
你无需删除空行，直接评论以获取最佳展示效果
最近发布