dolphinscheduler集群部署教程

文章目录 前言一、架构规划二、配置集群免密登录1. 配置root用户集群免密登录1.1 hadoop101节点操作1.2 hadoop102节点操作1.3 hadoop103节点操作 2. 创建用户2.1 hadoop101节点操作2.2 hadoop102节点操作2.3 hadoop103节点操作 三、安装准备1. 安装条件2. 安装jdk3. 安装MySQL4. zookeeper集群搭建4.1 部署zookeeper集群4.1.1 hadoop101节点操作4.1.2 hadoop102节点操作4.1.3 hadoop103节点操作 4.2 启动zookeeper集群4.2.1 hadoop101节点操作4.2.2 hadoop102节点操作4.2.3 hadoop103节点操作 4.3 zookeeper命令4.3.1 启动zookeeper4.3.2 停止zookeeper4.3.3 重启zookeeper4.3.4 查看zookeeper状态4.3.5 进入zookeeper客户端 四、安装dolphinscheduler1. 下载并解压dolphinscheduler2. 修改安装配置文件2.1 修改 install_env.sh 文件2.2 修改 dolphinscheduler_env.sh 文件 3. 配置mysql数据源3.1 修改MySQL安全策略3.2 查看数据库3.3 创建数据库3.4 创建MySQL用户3.5 配置MySQL驱动 4. 初始化数据库5. 修改目录权限6. 安装部署dolphinscheduler7. 登录 DolphinScheduler8. dolphinscheduler命令8.1 一键停止集群所有服务8.2 一键开启集群所有服务8.3 启停 Master8.4 启停 Worker8.5 启停 Api8.6 启停 Alert 总结

---

前言

本文将介绍如何安装和配置 DolphinScheduler，一个分布式任务调度系统。首先，我们将讨论架构规划，包括主机IP、主机名和安装的服务。然后，我们将配置集群之间的免密登录，创建用户并设置密码。接下来，我们将安装所需的软件和库，如JDK、MySQL和ZooKeeper。最后，我们将下载和部署 DolphinScheduler，并演示如何登录和使用该系统。

---

一、架构规划 主机ip主机名安装服务192.168.198.101hadoop101master、worker、apiServers、mysql、zookeeper192.168.198.102hadoop102master、worker、zookeeper192.168.198.103hadoop103worker、alertServer、zookeeper

---

二、配置集群免密登录 1. 配置root用户集群免密登录

分别在hadoop101、hadoop102和hadoop103节点执行下面命令，用于配置集群之间互相免密登录。 执行ssh-copy-id命令过程中需要根据提示输入yes和root用户密码。

1.1 hadoop101节点操作 ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa ssh-copy-id hadoop101 ssh-copy-id hadoop102 ssh-copy-id hadoop103 1.2 hadoop102节点操作 ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa ssh-copy-id hadoop101 ssh-copy-id hadoop102 ssh-copy-id hadoop103 1.3 hadoop103节点操作 ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa ssh-copy-id hadoop101 ssh-copy-id hadoop102 ssh-copy-id hadoop103 2. 创建用户

分别在hadoop101、hadoop102和hadoop103节点创建dolphinscheduler用户并设置密码。

分别在hadoop101、hadoop102和hadoop103节点切换到dolphinscheduler用户，然后配置集群之间互相免密登录。

dolphinscheduler用户的密码：dolphinscheduler

2.1 hadoop101节点操作 useradd dolphinscheduler echo "dolphinscheduler" | passwd --stdin dolphinscheduler sed -i '$adolphinscheduler ALL=(ALL) NOPASSWD: NOPASSWD: ALL' /etc/sudoers sed -i 's/Defaults requirett/#Defaults requirett/g' /etc/sudoers su dolphinscheduler ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa ssh-copy-id hadoop101 ssh-copy-id hadoop102 ssh-copy-id hadoop103

退出dolphinscheduler用户。

exit 2.2 hadoop102节点操作 useradd dolphinscheduler echo "dolphinscheduler" | passwd --stdin dolphinscheduler sed -i '$adolphinscheduler ALL=(ALL) NOPASSWD: NOPASSWD: ALL' /etc/sudoers sed -i 's/Defaults requirett/#Defaults requirett/g' /etc/sudoers su dolphinscheduler ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa ssh-copy-id hadoop101 ssh-copy-id hadoop102 ssh-copy-id hadoop103

退出dolphinscheduler用户。

exit 2.3 hadoop103节点操作 useradd dolphinscheduler echo "dolphinscheduler" | passwd --stdin dolphinscheduler sed -i '$adolphinscheduler ALL=(ALL) NOPASSWD: NOPASSWD: ALL' /etc/sudoers sed -i 's/Defaults requirett/#Defaults requirett/g' /etc/sudoers su dolphinscheduler ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa ssh-copy-id hadoop101 ssh-copy-id hadoop102 ssh-copy-id hadoop103

退出dolphinscheduler用户。

exit

---

三、安装准备 1. 安装条件 jdk版本：1.8+MySQL版本：5.7+MySQL驱动版本：8.0.16+zookeeper版本：3.4.6+ 2. 安装jdk

下载地址：https://www.oracle.com/java/technologies/downloads/#java8

下载后上传到hadoop101、hadoop102和hadoop103节点的/tmp目录下。

然后分别在hadoop101、hadoop102和hadoop103节点执行下面命令，用于创建目录、解压，并设置系统级环境变量。

mkdir -p /opt/module tar -zxvf /tmp/jdk-8u401-linux-x64.tar.gz -C /opt/module/ echo >> /etc/profile echo '#JAVA_HOME' >> /etc/profile echo "export JAVA_HOME=/opt/module/jdk1.8.0_401" >> /etc/profile echo 'export PATH=$PATH:$JAVA_HOME/bin' >> /etc/profile source /etc/profile 3. 安装MySQL

在hadoop101安装MySQL。

mysql5.7.44自动化安装教程mysql5.7.37自动化安装教程 4. zookeeper集群搭建 4.1 部署zookeeper集群 4.1.1 hadoop101节点操作 wget --no-check-certificate https://archive.apache.org/dist/zookeeper/zookeeper-3.8.1/apache-zookeeper-3.8.1-bin.tar.gz -P /tmp tar -zxvf /tmp/apache-zookeeper-3.8.1-bin.tar.gz -C /opt/soft/zookeeper cd /opt/soft/zookeeper/apache-zookeeper-3.8.1-bin cp ./conf/zoo_sample.cfg ./conf/zoo.cfg sed -i "s|^dataDir=.*|dataDir=./tmp/zookeeper|" ./conf/zoo.cfg sed -i "s|^clientPort=.*|clientPort=12181|" ./conf/zoo.cfg echo >> ./conf/zoo.cfg echo "server.1=10.80.69.72:12888:13888" >> ./conf/zoo.cfg echo "server.2=10.80.69.73:12888:13888" >> ./conf/zoo.cfg echo "server.3=10.80.69.74:12888:13888" >> ./conf/zoo.cfg echo >> ./conf/zoo.cfg echo 'admin.serverPort=18080' >> ./conf/zoo.cfg ./bin/zkServer.sh start echo 1 > ./tmp/zookeeper/myid ./bin/zkServer.sh restart 4.1.2 hadoop102节点操作 wget --no-check-certificate https://archive.apache.org/dist/zookeeper/zookeeper-3.8.1/apache-zookeeper-3.8.1-bin.tar.gz -P /tmp tar -zxvf /tmp/apache-zookeeper-3.8.1-bin.tar.gz -C /opt/soft/zookeeper cd /opt/soft/zookeeper/apache-zookeeper-3.8.1-bin cp ./conf/zoo_sample.cfg ./conf/zoo.cfg sed -i "s|^dataDir=.*|dataDir=./tmp/zookeeper|" ./conf/zoo.cfg sed -i "s|^clientPort=.*|clientPort=12181|" ./conf/zoo.cfg echo >> ./conf/zoo.cfg echo "server.1=10.80.69.72:12888:13888" >> ./conf/zoo.cfg echo "server.2=10.80.69.73:12888:13888" >> ./conf/zoo.cfg echo "server.3=10.80.69.74:12888:13888" >> ./conf/zoo.cfg echo >> ./conf/zoo.cfg echo 'admin.serverPort=18080' >> ./conf/zoo.cfg ./bin/zkServer.sh start echo 2 > ./tmp/zookeeper/myid ./bin/zkServer.sh restart 4.1.3 hadoop103节点操作 wget --no-check-certificate https://archive.apache.org/dist/zookeeper/zookeeper-3.8.1/apache-zookeeper-3.8.1-bin.tar.gz -P /tmp tar -zxvf /tmp/apache-zookeeper-3.8.1-bin.tar.gz -C /opt/soft/zookeeper cd /opt/soft/zookeeper/apache-zookeeper-3.8.1-bin cp ./conf/zoo_sample.cfg ./conf/zoo.cfg sed -i "s|^dataDir=.*|dataDir=./tmp/zookeeper|" ./conf/zoo.cfg sed -i "s|^clientPort=.*|clientPort=12181|" ./conf/zoo.cfg echo >> ./conf/zoo.cfg echo "server.1=10.80.69.72:12888:13888" >> ./conf/zoo.cfg echo "server.2=10.80.69.73:12888:13888" >> ./conf/zoo.cfg echo "server.3=10.80.69.74:12888:13888" >> ./conf/zoo.cfg echo >> ./conf/zoo.cfg echo 'admin.serverPort=18080' >> ./conf/zoo.cfg ./bin/zkServer.sh start echo 3 > ./tmp/zookeeper/myid ./bin/zkServer.sh restart 4.2 启动zookeeper集群 4.2.1 hadoop101节点操作 cd /opt/soft/zookeeper/apache-zookeeper-3.8.1-bin ./bin/zkServer.sh restart 4.2.2 hadoop102节点操作 cd /opt/soft/zookeeper/apache-zookeeper-3.8.1-bin ./bin/zkServer.sh start 4.2.3 hadoop103节点操作 cd /opt/soft/zookeeper/apache-zookeeper-3.8.1-bin ./bin/zkServer.sh start 4.3 zookeeper命令 4.3.1 启动zookeeper cd /opt/soft/zookeeper/apache-zookeeper-3.8.1-bin ./bin/zkServer.sh start 4.3.2 停止zookeeper cd /opt/soft/zookeeper/apache-zookeeper-3.8.1-bin ./bin/zkServer.sh stop 4.3.3 重启zookeeper cd /opt/soft/zookeeper/apache-zookeeper-3.8.1-bin ./bin/zkServer.sh restart 4.3.4 查看zookeeper状态 cd /opt/soft/zookeeper/apache-zookeeper-3.8.1-bin ./bin/zkServer.sh status 4.3.5 进入zookeeper客户端 cd /opt/soft/zookeeper/apache-zookeeper-3.8.1-bin ./bin/zkCli.sh -server hadoop101:2181

---

四、安装dolphinscheduler

在hadoop101节点下载并解压dolphinscheduler。

以下操作都在hadoop101节点进行。

1. 下载并解压dolphinscheduler wget --no-check-certificate https://archive.apache.org/dist/dolphinscheduler/3.1.4/apache-dolphinscheduler-3.1.4-bin.tar.gz -P /tmp mkdir -p /opt/soft/dolphinscheduler tar -zxvf /tmp/apache-dolphinscheduler-3.1.4-bin.tar.gz -C /opt/soft/dolphinscheduler 2. 修改安装配置文件 2.1 修改 install_env.sh 文件

根据架构规划修改 /opt/module/dolphinscheduler/apache-dolphinscheduler-3.1.4-bin/bin/env/install_env.sh 文件。内容如下所示。

# 需要安装DolphinScheduler相关组件的主机 ips=${ips:-"hadoop101,hadoop102,hadoop103"} # SSH协议端口号，默认为22。 sshPort=${sshPort:-"22"} # 需要安装master节点的主机 masters=${masters:-"hadoop101,hadoop102"} # 需要安装worker节点的主机 workers=${workers:-"hadoop101:default,hadoop102:default,hadoop103:default"} # 需要安装警报服务的主机 alertServer=${alertServer:-"hadoop103"} # 需要安装api服务的主机 apiServers=${apiServers:-"hadoop101"} # 安装dolphinscheduler的目录。如果不存在，将由install.sh脚本创建。 installPath=${installPath:-"/opt/module/dolphinscheduler/dolphinscheduler"} # 部署dolphinscheduler的用户 deployUser=${deployUser:-"root"} # zookeeper的根目录，目前DolphinScheduler默认的注册服务器是zookeeper。 zkRoot=${zkRoot:-"/dolphinscheduler"} 2.2 修改 dolphinscheduler_env.sh 文件

修改 /opt/module/dolphinscheduler/apache-dolphinscheduler-3.1.4-bin/bin/env/dolphinscheduler_env.sh 文件。内容如下所示。

# 修改为自己的jdk安装目录 export JAVA_HOME=${JAVA_HOME:-/opt/module/jdk1.8.0_391} # 修改MySQL配置 export DATABASE=${DATABASE:-mysql} export SPRING_PROFILES_ACTIVE=${DATABASE} export SPRING_DATASOURCE_URL="jdbc:mysql://hadoop101:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false" export SPRING_DATASOURCE_USERNAME="dolphinscheduler" export SPRING_DATASOURCE_PASSWORD="dolphinscheduler" # DolphinScheduler服务相关配置 export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none} export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC} export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10} # 注册中心配置，修改为自己的zookeeper监听地址 export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper} export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-hadoop101:2181,hadoop102:2181,hadoop103:2181} # 这些环境变量根据自己的需要更改，没有保持默认即口 export HADOOP_HOME=${HADOOP_HOME:-/opt/soft/hadoop} export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/opt/soft/hadoop/etc/hadoop} export SPARK_HOME1=${SPARK_HOME1:-/opt/soft/spark1} export SPARK_HOME2=${SPARK_HOME2:-/opt/soft/spark2} export PYTHON_HOME=${PYTHON_HOME:-/opt/soft/python} export HIVE_HOME=${HIVE_HOME:-/opt/soft/hive} export FLINK_HOME=${FLINK_HOME:-/opt/soft/flink} export DATAX_HOME=${DATAX_HOME:-/opt/soft/datax} export SEATUNNEL_HOME=${SEATUNNEL_HOME:-/opt/soft/seatunnel} export CHUNJUN_HOME=${CHUNJUN_HOME:-/opt/soft/chunjun} export PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_HOME/bin:$SEATUNNEL_HOME/bin:$CHUNJUN_HOME/bin:$PATH 3. 配置mysql数据源 3.1 修改MySQL安全策略

修改安全策略为LOW，使其可以设置简单密码，根据自己的需要更改。

mysql -h hadoop101 -P 3306 -uroot -proot -e "set global validate_password_length=4;" mysql -h hadoop101 -P 3306 -uroot -proot -e "set global validate_password_policy=LOW;" 3.2 查看数据库

查看dolphinscheduler数据库是否存在。

mysql -h hadoop101 -P 3306 -uroot -proot -e "SHOW DATABASES;" | grep dolphinscheduler

若存在则删除。

mysql -h hadoop101 -P 3306 -uroot -proot -e "DROP DATABASE dolphinscheduler;" 3.3 创建数据库

创建dolphinscheduler数据库

mysql -h hadoop101 -P 3306 -uroot -proot -e "CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;" 3.4 创建MySQL用户

创建 dolphinscheduler 用户并授予操作 dolphinscheduler 数据库的权限。

此处创建dolphinscheduler用户并设置密码为dolphinscheduler。

mysql -h hadoop101 -P 3306 -uroot -proot -e "CREATE USER 'dolphinscheduler'@'%' IDENTIFIED BY 'dolphinscheduler';"

授予操作 dolphinscheduler 数据库的权限并刷新权限。

mysql -h hadoop101 -P 3306 -uroot -proot -e "GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'dolphinscheduler'@'%' IDENTIFIED BY 'dolphinscheduler';" mysql -h hadoop101 -P 3306 -uroot -proot -e "flush privileges;" 3.5 配置MySQL驱动

下载并配置MySQL驱动。

wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.16/mysql-connector-java-8.0.16.jar -P /tmp cd /opt/soft/dolphinscheduler/apache-dolphinscheduler-3.1.4-bin cp /tmp/mysql-connector-java-8.0.16.jar ./worker-server/libs cp /tmp/mysql-connector-java-8.0.16.jar ./api-server/libs cp /tmp/mysql-connector-java-8.0.16.jar ./alert-server/libs cp /tmp/mysql-connector-java-8.0.16.jar ./master-server/libs cp /tmp/mysql-connector-java-8.0.16.jar ./tools/libs cp /tmp/mysql-connector-java-8.0.16.jar ./standalone-server/libs/standalone-server cd /opt/soft/dolphinscheduler wget https://repo1.maven.org/maven2/commons-cli/commons-cli/1.5.0/commons-cli-1.5.0.jar -P /tmp cp /tmp/commons-cli-1.5.0.jar apache-dolphinscheduler-3.1.4-bin/worker-server/libs cp /tmp/commons-cli-1.5.0.jar apache-dolphinscheduler-3.1.4-bin/api-server/libs cp /tmp/commons-cli-1.5.0.jar apache-dolphinscheduler-3.1.4-bin/alert-server/libs cp /tmp/commons-cli-1.5.0.jar apache-dolphinscheduler-3.1.4-bin/master-server/libs cp /tmp/commons-cli-1.5.0.jar apache-dolphinscheduler-3.1.4-bin/tools/libs 4. 初始化数据库

初始化dolphinscheduler数据库。

cd /opt/soft/dolphinscheduler/apache-dolphinscheduler-3.1.4-bin bash tools/bin/upgrade-schema.sh

初始化成功如下图所示。

5. 修改目录权限

修改apache-dolphinscheduler-3.1.4-bin目录权限到dolphinscheduler用户。

cd /opt/soft/dolphinscheduler chown -R dolphinscheduler:dolphinscheduler apache-dolphinscheduler-3.1.4-bin 6. 安装部署dolphinscheduler cd /opt/soft/dolphinscheduler/apache-dolphinscheduler-3.1.4-bin bash ./bin/install.sh 7. 登录 DolphinScheduler

浏览器访问地址 http://localhost:12345/dolphinscheduler/ui 即可登录系统UI。默认的用户名和密码是 admin/dolphinscheduler123

如下图所示。

8. dolphinscheduler命令 8.1 一键停止集群所有服务 cd /opt/soft/dolphinscheduler/apache-dolphinscheduler-3.1.4-bin bash ./bin/stop-all.sh 8.2 一键开启集群所有服务 cd /opt/soft/dolphinscheduler/apache-dolphinscheduler-3.1.4-bin bash ./bin/start-all.sh 8.3 启停 Master cd /opt/soft/dolphinscheduler/apache-dolphinscheduler-3.1.4-bin bash ./bin/dolphinscheduler-daemon.sh stop master-server bash ./bin/dolphinscheduler-daemon.sh start master-server 8.4 启停 Worker cd /opt/soft/dolphinscheduler/apache-dolphinscheduler-3.1.4-bin bash ./bin/dolphinscheduler-daemon.sh start worker-server bash ./bin/dolphinscheduler-daemon.sh stop worker-server 8.5 启停 Api cd /opt/soft/dolphinscheduler/apache-dolphinscheduler-3.1.4-bin bash ./bin/dolphinscheduler-daemon.sh start api-server bash ./bin/dolphinscheduler-daemon.sh stop api-server 8.6 启停 Alert cd /opt/soft/dolphinscheduler/apache-dolphinscheduler-3.1.4-bin bash ./bin/dolphinscheduler-daemon.sh start alert-server bash ./bin/dolphinscheduler-daemon.sh stop alert-server

---

总结

本文详细介绍了安装和配置 DolphinScheduler 的步骤。我们首先进行了架构规划，确定了主机IP、主机名和安装的服务。然后，我们配置了集群之间的免密登录，并创建了用户并设置了密码。接下来，我们安装了所需的软件和库，如JDK、MySQL和ZooKeeper。最后，我们下载并部署了 DolphinScheduler，并演示了登录和使用该系统的过程。

希望本教程对您有所帮助！如有任何疑问或问题，请随时在评论区留言。感谢阅读！