大数据集群报错 SemanticException Unable to determine 解决

后知后觉 现有 4 评论

因特殊需要修改了大数据库集群的主机名或修改 HDFS 分布式存储的端口后,重启服务后,获取数据时就会看到 FAILED: SemanticException Unable to determine if hdfs://... 报错。

报错

具体的报错类如:

FAILED: SemanticException Unable to determine if hdfs://localhost:9000/user/hive/warehouse/datax.db/s_lo_p_f is encrypted: 
org.apache.hadoop.hive.ql.metadata.HiveException: java.net.ConnectException: 
Call From hadoop/172.16.1.102 to localhost:9000 failed on connection exception:
 java.net.ConnectException: Connection refused; 
For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

因修改了主机名,但是报错中访问的地址还是老的主机名地址,这是因为什么呢?经过查阅文档后发现,在 Hive 中,所有的 HDFS 区块都会被写入 Metadata 中(后端可以是多种数据库引擎),一般情况下使用的是 MySQL,那么就需要修正 Metadata 中记录的区块位置,才能正常执行命令。

处理

记录了 HDFS 区块信息的表在库中名为 DBSSDS,可以用命令进行查询:

SELECT * FROM hive.DBS;

输出类如:

+-------+-----------------------+--------------------------------------------------------+--------------+------------+------------+-----------+
| DB_ID | DESC                  | DB_LOCATION_URI                                        | NAME         | OWNER_NAME | OWNER_TYPE | CTLG_NAME |
+-------+-----------------------+--------------------------------------------------------+--------------+------------+------------+-----------+
|     1 | Default Hive database | hdfs://localhost:9000/user/hive/warehouse              | default      | public     | ROLE       | hive      |
|     6 | NULL                  | hdfs://localhost:9000/user/hive/warehouse/dba.db       | dba          | root       | USER       | hive      |
|     7 | NULL                  | hdfs://localhost:9000/user/hive/warehouse/dbb.db       | dbb          | root       | USER       | hive      |
+-------+-----------------------+--------------------------------------------------------+--------------+------------+------------+-----------+

例如修改了主机名或端口为 hadoop:9001,可以直接执行 SQL:

UPDATE DBS SET DB_LOCATION_URI=REPLACE(DB_LOCATION_URI,'localhost:9000','hadoop:9001');
UPDATE SDS SET LOCATION=REPLACE(LOCATION,'localhost:9000','hadoop:9001');

HDFS 的端口可以用命令 hdfs getconf -confKey fs.defaultFS 查询。


附录

参考链接

如果遇到问题或者对文章内容存疑,请在下方留言,博主看到后将及时回复,谢谢!
禁用 / 当前已拒绝评论,仅可查看「历史评论」。
  1. avatarImg вот здесь

    Many thanks, Numerous information!

    YaBrowser 22.7 macOS 12.5
    IP 属地 未知
  2. avatarImg sitemap

    Your mmode off describing everythng inn this pos is genuinely pleasant, evvery one caan withoit difficuulty kow it, Thajks a lot.

    Firefox 114.0 Ubuntu
    IP 属地 未知
  3. avatarImg sitemap.xml

    Goood day! I knkw this iis kinda off topi but I
    waas wonmdering if you knew wherte I could find a captcdha plugin forr my commebt form?
    I'm using the saame blog platform as yours andd I'm haing
    propblems finding one? Thanks a lot!

    Opera 89.0 Windows 10
    IP 属地 未知
    1. avatarImg
      @sitemap.xml

      It's a private project, You can make a plugin with Google recaptcha docs.

      Firefox 120.0 macOS Catalina
      IP 属地 未知