Hive 集群安装配置

前言

接Zookeeper 安装配置 和 Hadoop HA 集群安装配置，本文继续总结 Hive 集群安装配置。其实之前的文章中已经总结过部分，但是不全面：

前提

主机名修改、SSH互信、JDK安装等。

版本

Hive 3.1.3

安装包

https://archive.apache.org/dist/hive/hive-3.1.3/apache-hive-3.1.3-bin.tar.gz

more >>

展开全文 >>

Ray 集群安装配置

前言

作为新手，学习总结一下 Ray 的集群安装配置及简单使用。

官网

Ray 的核心特点

  • 简单的分布式编程模型：通过装饰器（@ray.remote）可以轻松将函数或类转换为分布式版本，无需深入了解分布式系统细节。
  • 自动任务调度：Ray 拥有高效的任务调度器，能够自动在集群中分配任务，优化资源利用率。
  • 支持状态 ful 计算：除了无状态的任务，Ray 还支持有状态的 Actor 模型，适合需要维护状态的场景（如训练模型、服务部署）。
  • 丰富的生态系统：包含多个上层库，如用于超参数调优的 Ray Tune、用于强化学习的 Ray RLlib、用于分布式训练的 Ray Train 等。
  • 灵活的部署方式：可在单机、集群、云环境（AWS、GCP、Azure 等）中部署，支持动态资源扩展。 more >>

展开全文 >>

异常解决记录 | Yarn NodeManager 注册异常

前言

该异常发生背景：

  • 项目上的同事之前已部署了8个节点均正常
  • 项目上的同事后面又扩容了8个节点均异常
  • Yarn ResourceManager 配置了 HA
  • Hadoop 版本： 3.1.1

具体异常

1
2
3
4
5
6
7
8
9
10
11
12
2025-09-17 09:30:00,869 INFO  client.ConfiguredRMFailoverProxyProvider (ConfiguredRMFailoverProxyProvider.java:performFailover(100)) - Failing over to rm2
2025-09-17 09:30:01,043 INFO  retry.RetryInvocationHandler (RetryInvocationHandler.java:log(411)) - org.apache.hadoop.security.authorize.AuthorizationException: User nm/indata-192-168-1-3.indata.com@INDATA.COM  (auth:KERBEROS) is not authorized for protocol interface org.apache.hadoop.yarn.server.api.ResourceTrackerPB: this service is only accessible by nm/192.168.1.3@INDATA.COM, while invoking ResourceTrackerPBClientImpl.registerNodeManager over rm2 after 1 failover attempts. Trying to failover after sleeping for 21498ms.
2025-09-17 09:30:22,541 INFO  client.ConfiguredRMFailoverProxyProvider (ConfiguredRMFailoverProxyProvider.java:performFailover(100)) - Failing over to rm1
2025-09-17 09:30:22,545 INFO  retry.RetryInvocationHandler (RetryInvocationHandler.java:log(411)) - java.net.ConnectException: Call From indata-192-168-1-3.indata.com/192.168.1.3 to indata-192-168-1-1.indata.com:8031 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused, while invoking ResourceTrackerPBClientImpl.registerNodeManager over rm1 after 2 failover attempts. Trying to failover after sleeping for 33407ms.
2025-09-17 09:30:55,953 INFO  client.ConfiguredRMFailoverProxyProvider (ConfiguredRMFailoverProxyProvider.java:performFailover(100)) - Failing over to rm2
2025-09-17 09:30:55,992 INFO  retry.RetryInvocationHandler (RetryInvocationHandler.java:log(411)) - org.apache.hadoop.security.authorize.AuthorizationException: User nm/indata-192-168-1-3.indata.com@INDATA.COM  (auth:KERBEROS) is not authorized for protocol interface org.apache.hadoop.yarn.server.api.ResourceTrackerPB: this service is only accessible by nm/192.168.1.3@INDATA.COM, while invoking ResourceTrackerPBClientImpl.registerNodeManager over rm2 after 3 failover attempts. Trying to failover after sleeping for 44974ms.
2025-09-17 09:31:40,967 INFO  client.ConfiguredRMFailoverProxyProvider (ConfiguredRMFailoverProxyProvider.java:performFailover(100)) - Failing over to rm1
2025-09-17 09:31:40,971 INFO  retry.RetryInvocationHandler (RetryInvocationHandler.java:log(411)) - java.net.ConnectException: Call From indata-192-168-1-3.indata.com/192.168.1.3 to indata-192-168-1-1.indata.com:8031 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused, while invoking ResourceTrackerPBClientImpl.registerNodeManager over rm1 after 4 failover attempts. Trying to failover after sleeping for 15164ms.
2025-09-17 09:31:56,136 INFO  client.ConfiguredRMFailoverProxyProvider (ConfiguredRMFailoverProxyProvider.java:performFailover(100)) - Failing over to rm2
2025-09-17 09:31:56,181 INFO  retry.RetryInvocationHandler (RetryInvocationHandler.java:log(411)) - org.apache.hadoop.security.authorize.AuthorizationException: User nm/indata-192-168-1-3.indata.com@INDATA.COM  (auth:KERBEROS) is not authorized for protocol interface org.apache.hadoop.yarn.server.api.ResourceTrackerPB: this service is only accessible by nm/192.168.1.3@INDATA.COM, while invoking ResourceTrackerPBClientImpl.registerNodeManager over rm2 after 5 failover attempts. Trying to failover after sleeping for 27554ms.
2025-09-17 09:32:23,741 INFO  client.ConfiguredRMFailoverProxyProvider (ConfiguredRMFailoverProxyProvider.java:performFailover(100)) - Failing over to rm1
2025-09-17 09:32:23,749 INFO  retry.RetryInvocationHandler (RetryInvocationHandler.java:log(411)) - java.net.ConnectException: Call From indata-192-168-1-3.indata.com/192.168.1.3 to indata-192-168-1-1.indata.com:8031 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused, while invoking ResourceTrackerPBClientImpl.registerNodeManager over rm1 after 6 failover attempts. Trying to failover after sleeping for 35229ms.
more >>

展开全文 >>

tag:

    缺失模块。
    1、请确保node版本大于6.2
    2、在博客根目录（注意不是yilia根目录）执行以下命令：
    npm i hexo-generator-json-content --save

    3、在根目录_config.yml里添加配置： 

      jsonContent:
    meta: false
    pages: false
    posts:
      title: true
      date: true
      path: true
      text: false
      raw: false
      content: false
      slug: false
      updated: false
      comments: false
      link: false
      permalink: false
      excerpt: false
      categories: false
      tags: true

2016年本科毕业于合肥工业大学<br><br>信息与计算科学专业<br><br>目前在济南一家公司从事大数据的工作~~