基于 aarch64 编译 Hadoop 2.9.2 Native Libraries 完整套件
后知后觉 暂无评论

CentOS Hadoop

某个项目上需要使用旧版本的 Hadoop 2.9.2 版本，但是服务器是 ARMv8 架构，然而官方并未提供编译好的 aarch64 包，因此需要自行构建。

背景

需要部署 Apache Kylin 一个比较旧的版本，相关的组件对新版支持的不是很好，并且项目方使用的是国产飞腾芯片的服务器，因此需要针对 aarch64 完整编译一遍包含（Hadoop 项目只有 3.3.1 等少数几个版本提供了官方编译的的 aarch64 架构包）Native Libraries 在内的完整套件。

过程

基础环境 CentOS 7.9 (aarch64) 最小化安装实例，CPU 是飞腾 FT-S2500（国产 ARMv8，指令集好像略有缺失，不过无伤大雅）。建议编译前完整升级一遍系统后重启继续操作。

$ lsb_release -a
LSB Version:    :core-4.1-aarch64:core-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.9.2009 (AltArch)
Release:        7.9.2009
Codename:       AltArch

环境

Oracle JDK

因为 Hadoop 及相关套件都是基于 Java 编写的，先安装基础环境。理论上 OpenJDK 亦可使用，不过谨慎起见，在 Oracle JDK 官网下载 JDK 安装包。

## 若下载版本为 8u333
sudo yum install jdk-8u333-linux-aarch64.rpm

安装后检查版本

$ java -version
java version "1.8.0_333"
Java(TM) SE Runtime Environment (build 1.8.0_333-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.333-b02, mixed mode)

Maven

然后部署 Maven ，提供 Java 构建环境。

wget https://dlcdn.apache.org/maven/maven-3/3.8.6/binaries/apache-maven-3.8.6-bin.tar.gz
sudo tar xf apache-maven-3.8.6-bin.tar.gz -C /opt/

添加全局环境变量

sudo vim /etc/profile.d/maven.sh
## 添加以下内容
export JAVA_HOME=/usr/java/jdk1.8.0_333-aarch64
export PATH=$JAVA_HOME/bin:$PATH
export M2_HOME=/opt/apache-maven-3.8.6
export MAVEN_HOME=/opt/apache-maven-3.8.6
export PATH=$M2_HOME/bin:$PATH
## 保存后生效环境环境变量或者重新登录终端
source /etc/profile

System Depends

然后准备 Native Libraries 编译环境，官方演示编译环境为 Ubuntu，因此依赖需要更换为 RedHat 系的包名。

sudo yum groups install 'Development Tools'

然后安装 Native Depends

sudo yum install bzip2-devel cyrus-sasl-devel \
                 fuse fuse-devel \
                 openssl-devel \
                 protobuf-compiler protobuf-devel \
                 snappy-devel zlib-devel

还有两个依赖在 EPEL 中，需要先安装 epel-release，然后执行

sudo yum install libzstd libzstd-devel

Build Tools

需要注意的是 CentOS 7.x 仓库的 cmake 版本 2.8 过低，需要使用 3.x 版本 cmake 进行编译，稳妥起见，使用和官方手册相同版本的 cmake 3.13.4。

wget https://cmake.org/files/v3.13/cmake-3.13.4.tar.gz
tar xf cmake-3.13.4.tar.gz
cd cmake-3.13.4/
./configure
sudo make install

安装完成后检查系统组件版本

$ cmake --version
cmake version 3.13.4

CMake suite maintained and supported by Kitware (kitware.com/cmake).
$ protoc --version
libprotoc 2.5.0

构建

下载 Hadoop 2.9.2 版本源码包，解压并开始构建

wget https://github.com/apache/hadoop/archive/refs/tags/rel/release-2.9.2.tar.gz
tar xf release-2.9.2.tar.gz
cd hadoop-rel-release-2.9.2/
mvn clean package -Pdist,native, -DskipTests -Dtar -Dmaven.javadoc-skip=true -X

四核构建大约 20 分钟，看到以下提示即为构建成功。

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 2.9.2:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [01:59 min]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [01:34 min]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 30.543 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 15.560 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.063 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 50.035 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 23.330 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [02:01 min]
[INFO] Apache Hadoop Auth ................................. SUCCESS [01:06 min]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  6.202 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [02:01 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  1.405 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 18.210 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.017 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 42.071 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 29.702 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [  2.715 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 16.634 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 21.782 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  1.012 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [  8.016 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.023 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.014 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [  4.913 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 46.442 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  1.417 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.015 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 17.066 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 10.991 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  0.991 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 16.677 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [  6.296 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [  6.971 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  0.332 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [  1.721 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  0.911 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  0.928 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [  1.307 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [01:01 min]
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [ 15.226 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.012 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  0.898 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  0.628 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.038 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [  2.999 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [  3.329 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.060 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [  5.554 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [  4.658 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  1.054 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [  2.517 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  1.550 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  1.897 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  0.608 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  1.465 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  1.479 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  5.560 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  1.318 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  0.681 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  0.777 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  1.445 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  1.297 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  0.742 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  0.615 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  0.869 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  1.918 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  1.136 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [01:01 min]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [  9.709 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ 11.114 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [  3.316 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.229 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  1.542 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [  4.110 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  7.966 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  8.533 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.028 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 25.411 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  0.377 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [  0.042 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  20:34 min
[INFO] Finished at: 2022-12-19T09:57:17+08:00
[INFO] ------------------------------------------------------------------------

生成的安装包在 hadoop-dist/target/ 目录下 hadoop-2.9.2.tar.gz 为最终编译的产品包。

常见问题

a) 如果提示部分库无法下载怎么处理？

可以尝试换用阿里云的 Maven 镜像仓库或者搭建自建代理，然后使用命令配置使用代理。

export MAVEN_OPTS="-DproxyHost=127.0.0.1 -DproxyPort=8080"

b) 编译报错如下

[INFO] --- hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) @ hadoop-hdfs-native-client ---
...
[WARNING] Located all JNI components successfully.
[WARNING] CUSTOM_OPENSSL_PREFIX = 
[WARNING] -- Performing Test THREAD_LOCAL_SUPPORTED
[WARNING] -- Performing Test THREAD_LOCAL_SUPPORTED - Success
[WARNING] CMake Error at /usr/share/cmake3/Modules/FindProtobuf.cmake:465 (file):
[WARNING]   file STRINGS file "/usr/include/google/protobuf/stubs/common.h" cannot be
[WARNING]   read.
[WARNING] Call Stack (most recent call first):
[WARNING]   main/native/libhdfspp/CMakeLists.txt:45 (find_package)
[WARNING] 
[WARNING] 
[WARNING] CMake Error at /usr/share/cmake3/Modules/FindProtobuf.cmake:471 (math):
[WARNING]   math cannot parse the expression: " / 1000000": syntax error, unexpected
[WARNING]   exp_DIVIDE, expecting exp_PLUS or exp_MINUS or exp_OPENPARENT or exp_NUMBER
[WARNING]   (2).
[WARNING] Call Stack (most recent call first):
[WARNING]   main/native/libhdfspp/CMakeLists.txt:45 (find_package)
[WARNING] 
[WARNING] 
[WARNING] CMake Error at /usr/share/cmake3/Modules/FindProtobuf.cmake:472 (math):
[WARNING]   math cannot parse the expression: " / 1000 % 1000": syntax error,
[WARNING]   unexpected exp_DIVIDE, expecting exp_PLUS or exp_MINUS or exp_OPENPARENT or
[WARNING]   exp_NUMBER (2).
[WARNING] Call Stack (most recent call first):
[WARNING]   main/native/libhdfspp/CMakeLists.txt:45 (find_package)
[WARNING] 
[WARNING] 
[WARNING] CMake Error at /usr/share/cmake3/Modules/FindProtobuf.cmake:473 (math):
[WARNING]   math cannot parse the expression: " % 1000": syntax error, unexpected
[WARNING]   exp_MOD, expecting exp_PLUS or exp_MINUS or exp_OPENPARENT or exp_NUMBER
[WARNING]   (2).
[WARNING] Call Stack (most recent call first):
[WARNING]   main/native/libhdfspp/CMakeLists.txt:45 (find_package)
[WARNING] 
[WARNING] 
[WARNING] CMake Warning at /usr/share/cmake3/Modules/FindProtobuf.cmake:495 (message):
[WARNING]   Protobuf compiler version 2.5.0 doesn't match library version
[WARNING]   ERROR.ERROR.ERROR
[WARNING] Call Stack (most recent call first):
[WARNING]   main/native/libhdfspp/CMakeLists.txt:45 (find_package)
[WARNING] 
[WARNING] 
[WARNING] -- Could NOT find GSASL (missing: GSASL_LIBRARIES GSASL_INCLUDE_DIR) 
[WARNING] -- Performing Test THREAD_LOCAL_SUPPORTED
[WARNING] -- Performing Test THREAD_LOCAL_SUPPORTED - Success
[WARNING] -- Performing Test PROTOC_IS_COMPATIBLE
[WARNING] -- Performing Test PROTOC_IS_COMPATIBLE - Failed
[WARNING] CMake Warning at main/native/libhdfspp/CMakeLists.txt:86 (message):
[WARNING]   WARNING: the Protocol Buffers Library and the Libhdfs++ Library must both
[WARNING]   be compiled with the same (or compatible) compiler.  Normally only the same
[WARNING]   major versions of the same compiler are compatible with each other.
[WARNING] 
[WARNING] 
[WARNING] -- valgrind location: MEMORYCHECK_COMMAND-NOTFOUND
[WARNING] -- Using Cyrus SASL; link with /usr/lib64/libsasl2.so

这是因为没有按照要求安装依赖，导致缺失 protobuf-devel ，手动补充安装。

附录

参考链接

本文撰写于一年前，如出现图片失效或有任何问题，请在下方留言。博主看到后将及时修正，谢谢！

本文由 柒 创作，采用 知识共享署名4.0 国际许可协议进行许可。
转载本站文章前请注明出处，文章作者保留所有权限。
最后编辑时间： 2022-12-19 12:13 PM

← → ↓ ↑
禁用 / 当前已拒绝评论，仅可查看「历史评论」。