Spark Thrift Server HA 解决方案

2022-02-11

前些天发现了一个巨牛的人工智能学习网站，通俗易懂，风趣幽默，忍不住给大家分享一下。点击跳转到网站：https://www.captainai.net/dongkelun

欢迎关注我的公众号

前言

本文为组内同事整理,这里稍作改动记录。本文主要修改Spark源码，实现Spark Spark Thrift Server 注册到到ZK，通过ZK连接实现负载均衡；另外可以通过使用Kyuubi实现HA，这里不做详细描述

背景

Spark ThriftServer 不支持Zookeeper连接，不能实现负载均衡。

解决方案:修改Spark ThriftServer源码，使其支持Zookeeper连接

版本信息：Spark3.1.2 ；Hive2.3.7

Spark ThriftServer 启动命令

nohup spark-submit --master yarn --deploy-mode client --num-executors 2 --executor-cores 1 --executor-memory 1G --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name SparkThriftServer_1 spark-internal --hiveconf hive.server2.thrift.http.port=20003 >>  /var/log/spark2/20003.log 2>&1 < /dev/null &

Spark 源码debug

./bin/spark-submit --master yarn --deploy-mode client --num-executors 2 --executor-cores 1 --executor-memory 1G --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name SparkThriftServer_1 spark-internal --hiveconf hive.server2.thrift.http.port=20003  --driver-java-options "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005"

Spark 连接

单节点

1	!connect jdbc:hive2://indata-1192-168-44-128:20003/default;principal=HTTP/indata-1192-168-44-128@INDATA.COM?hive.server2.transport.mode=http;hive.server2.thrift.http.path=cliservice;

Zookeeper

1	!connect jdbc:hive2://192-168-44-129.indata.com:2181,indata-1192-168-44-128:2181,indata-192-168-44-130.indata.com:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2_zk

Zookeeper 使用

连接：
/usr/hdp/3.1.0.0-78/zookeeper/bin/zkCli.sh -server indata-1192-168-44-128:2181
使用：
[zk: indata-1192-168-44-128:2181(CONNECTED) 2] ls /hiveserver2_zk
[serverUri=indata-1192-168-44-128:20004;version=2.3.7;sequence=0000000022]
[zk: indata-1192-168-44-128:2181(CONNECTED) 3] get  /hiveserver2_zk/serverUri=indata-1192-168-44-128:20004;version=2.3.7;sequence=0000000022
hive.server2.authentication=KERBEROS;hive.server2.transport.mode=http;hive.server2.thrift.http.path=cliservice;hive.server2.thrift.http.port=20004;hive.server2.thrift.bind.host=indata-1192-168-44-128;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=spark/_HOST@INDATA.COM
cZxid = 0x200001d9d
ctime = Fri Feb 11 08:37:57 CST 2022
mZxid = 0x200001d9d
mtime = Fri Feb 11 08:37:57 CST 2022
pZxid = 0x200001d9d
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x37ed86adc050206
dataLength = 306
numChildren = 0

Spark 源码修改

pom

添加 sql/hive-thriftserver 模块，使其编译，便于修改代码

1	<module>sql/hive-thriftserver</module>

HiveThriftServer2

修改包路径：sql\hive-thriftserver\src\main\scala\org\apache\spark\sql\hive\thriftserver\HiveThriftServer2.scala 源码
添加zookeeper支持，增加hiveConf,如果开启zookeeper，通过反射调用 addServerInstanceToZooKeeper、removeServerInstanceFromZooKeeper 方法

private[hive] class HiveThriftServer2(sqlContext: SQLContext)
  extends HiveServer2
  with ReflectedCompositeService {
  // state is tracked internally so that the server only attempts to shut down if it successfully
  // started, and then once only.
  private val started = new AtomicBoolean(false)

  var hiveConf: HiveConf = _
  override def init(hiveConf: HiveConf): Unit = {
    this.hiveConf = hiveConf

    val sparkSqlCliService = new SparkSQLCLIService(this, sqlContext)
    setSuperField(this, "cliService", sparkSqlCliService)
    addService(sparkSqlCliService)
    val thriftCliService = if (isHTTPTransportMode(hiveConf)) {
      new ThriftHttpCLIService(sparkSqlCliService)
    } else {
      new ThriftBinaryCLIService(sparkSqlCliService)
    }
    setSuperField(this, "thriftCLIService", thriftCliService)
    addService(thriftCliService)
    initCompositeService(hiveConf)
  }

  private def isHTTPTransportMode(hiveConf: HiveConf): Boolean = {
    val transportMode = hiveConf.getVar(ConfVars.HIVE_SERVER2_TRANSPORT_MODE)
    transportMode.toLowerCase(Locale.ROOT).equals("http")
  }


  override def start(): Unit = {
    super.start()
    started.set(true)
    
    hiveConf.set(HiveConf.ConfVars.HIVE_SERVER2_THRIFT_BIND_HOST.varname, getServerHost)
    if (this.hiveConf.getBoolVar(
      ConfVars.HIVE_SERVER2_SUPPORT_DYNAMIC_SERVICE_DISCOVERY)) {
      invoke(classOf[HiveServer2], this, "addServerInstanceToZooKeeper",
        classOf[HiveConf] -> this.hiveConf)
    }
  }

  override def stop(): Unit = {
    if (started.getAndSet(false)) {
      if (this.hiveConf.getBoolVar(
        ConfVars.HIVE_SERVER2_SUPPORT_DYNAMIC_SERVICE_DISCOVERY)) {
        invoke(classOf[HiveServer2], this, "removeServerInstanceFromZooKeeper")
      }
      super.stop()
    }
  }
}

HiveServer2

包路径：spark\sql\hive-thriftserver\src\main\java\org\apache\hive\service\server\HiveServer2.java
由于spark不支持zookeeper，需要添加 addServerInstanceToZooKeeper、removeServerInstanceFromZooKeeper方法



/**
   * StartOptionExecutor: starts HiveServer2.
   * This is the default executor, when no option is specified.
   */
  static class StartOptionExecutor implements ServerOptionsExecutor {
    @Override
    public void execute() {
      try {
        startHiveServer2();
      } catch (Throwable t) {
        LOG.error("Error starting HiveServer2", t);
        System.exit(-1);
      }
    }
  }

  private String getServerInstanceURI() throws Exception {
    if ((thriftCLIService == null) || (thriftCLIService.getServerIPAddress() == null)) {
      throw new Exception("Unable to get the server address; it hasn't been initialized yet.");
    }
    return thriftCLIService.getServerIPAddress().getHostName() + ":"
            + thriftCLIService.getPortNumber();
  }
  /**
   * For a kerberized cluster, we dynamically set up the client's JAAS conf.
   *
   * @param hiveConf
   * @return
   * @throws Exception
   */
  private void setUpZooKeeperAuth(HiveConf hiveConf) throws Exception {
    if (UserGroupInformation.isSecurityEnabled()) {
      String principal = hiveConf.getVar(ConfVars.HIVE_SERVER2_KERBEROS_PRINCIPAL);
      if (principal.isEmpty()) {
        throw new IOException("HiveServer2 Kerberos principal is empty");
      }
      String keyTabFile = hiveConf.getVar(ConfVars.HIVE_SERVER2_KERBEROS_KEYTAB);
      if (keyTabFile.isEmpty()) {
        throw new IOException("HiveServer2 Kerberos keytab is empty");
      }
      // Install the JAAS Configuration for the runtime
      Utils.setZookeeperClientKerberosJaasConfig(principal, keyTabFile);
    }
  }
  /**
   * ACLProvider for providing appropriate ACLs to CuratorFrameworkFactory
   */
  private final ACLProvider zooKeeperAclProvider = new ACLProvider() {

    @Override
    public List<ACL> getDefaultAcl() {
      List<ACL> nodeAcls = new ArrayList<ACL>();
      if (UserGroupInformation.isSecurityEnabled()) {
        // Read all to the world
        nodeAcls.addAll(Ids.READ_ACL_UNSAFE);
        // Create/Delete/Write/Admin to the authenticated user
        nodeAcls.add(new ACL(Perms.ALL, Ids.AUTH_IDS));
      } else {
        // ACLs for znodes on a non-kerberized cluster
        // Create/Read/Delete/Write/Admin to the world
        nodeAcls.addAll(Ids.OPEN_ACL_UNSAFE);
      }
      return nodeAcls;
    }

    @Override
    public List<ACL> getAclForPath(String path) {
      return getDefaultAcl();
    }
  };
  /**
   * Adds a server instance to ZooKeeper as a znode.
   *
   * @param hiveConf
   * @throws Exception
   */
  private void addServerInstanceToZooKeeper(HiveConf hiveConf) throws Exception {
    String zooKeeperEnsemble = ZooKeeperHiveHelper.getQuorumServers(hiveConf);
    String rootNamespace = hiveConf.getVar(HiveConf.ConfVars.HIVE_SERVER2_ZOOKEEPER_NAMESPACE);
    String instanceURI = getServerInstanceURI();
    setUpZooKeeperAuth(hiveConf);
    int sessionTimeout =
            (int) hiveConf.getTimeVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_SESSION_TIMEOUT,
                    TimeUnit.MILLISECONDS);
    int baseSleepTime =
            (int) hiveConf.getTimeVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_CONNECTION_BASESLEEPTIME,
                    TimeUnit.MILLISECONDS);
    int maxRetries = hiveConf.getIntVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_CONNECTION_MAX_RETRIES);
    // Create a CuratorFramework instance to be used as the ZooKeeper client
    // Use the zooKeeperAclProvider to create appropriate ACLs
    zooKeeperClient =
            CuratorFrameworkFactory.builder().connectString(zooKeeperEnsemble)
                    .sessionTimeoutMs(sessionTimeout).aclProvider(zooKeeperAclProvider)
                    .retryPolicy(new ExponentialBackoffRetry(baseSleepTime, maxRetries)).build();
    zooKeeperClient.start();
    // Create the parent znodes recursively; ignore if the parent already exists.
    try {
      zooKeeperClient.create().creatingParentsIfNeeded().withMode(CreateMode.PERSISTENT)
              .forPath(ZooKeeperHiveHelper.ZOOKEEPER_PATH_SEPARATOR + rootNamespace);
      LOG.info("Created the root name space: " + rootNamespace + " on ZooKeeper for HiveServer2");
    } catch (KeeperException e) {
      if (e.code() != KeeperException.Code.NODEEXISTS) {
        LOG.error("Unable to create HiveServer2 namespace: " + rootNamespace + " on ZooKeeper", e);
        throw e;
      }
    }
    // Create a znode under the rootNamespace parent for this instance of the server
    // Znode name: serverUri=host:port;version=versionInfo;sequence=sequenceNumber
    try {
      String pathPrefix =
              ZooKeeperHiveHelper.ZOOKEEPER_PATH_SEPARATOR + rootNamespace
                      + ZooKeeperHiveHelper.ZOOKEEPER_PATH_SEPARATOR + "serverUri=" + instanceURI + ";"
                      + "version=" + HiveVersionInfo.getVersion() + ";" + "sequence=";
      String znodeData = "";
      if (hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_SERVER2_ZOOKEEPER_PUBLISH_CONFIGS)) {
        // HiveServer2 configs that this instance will publish to ZooKeeper,
        // so that the clients can read these and configure themselves properly.
        Map<String, String> confsToPublish = new HashMap<String, String>();
        addConfsToPublish(hiveConf, confsToPublish);
        // Publish configs for this instance as the data on the node
        znodeData = Joiner.on(';').withKeyValueSeparator("=").join(confsToPublish);
      } else {
        znodeData = instanceURI;
      }
      byte[] znodeDataUTF8 = znodeData.getBytes(Charset.forName("UTF-8"));
      znode =
              new PersistentEphemeralNode(zooKeeperClient,
                      PersistentEphemeralNode.Mode.EPHEMERAL_SEQUENTIAL, pathPrefix, znodeDataUTF8);
      znode.start();
      // We'll wait for 120s for node creation
      long znodeCreationTimeout = 120;
      if (!znode.waitForInitialCreate(znodeCreationTimeout, TimeUnit.SECONDS)) {
        throw new Exception("Max znode creation wait time: " + znodeCreationTimeout + "s exhausted");
      }
      setDeregisteredWithZooKeeper(false);
      znodePath = znode.getActualPath();
      // Set a watch on the znode
      if (zooKeeperClient.checkExists().usingWatcher(new DeRegisterWatcher()).forPath(znodePath) == null) {
        // No node exists, throw exception
        throw new Exception("Unable to create znode for this HiveServer2 instance on ZooKeeper.");
      }
      LOG.info("Created a znode on ZooKeeper for HiveServer2 uri: " + instanceURI);
    } catch (Exception e) {
      LOG.error("Unable to create a znode for this server instance", e);
      if (znode != null) {
        znode.close();
      }
      throw (e);
    }
  }
  /**
   * The watcher class which sets the de-register flag when the znode corresponding to this server
   * instance is deleted. Additionally, it shuts down the server if there are no more active client
   * sessions at the time of receiving a 'NodeDeleted' notification from ZooKeeper.
   */
  private class DeRegisterWatcher implements Watcher {
    @Override
    public void process(WatchedEvent event) {
      if (event.getType().equals(Watcher.Event.EventType.NodeDeleted)) {
        if (znode != null) {
          try {
            znode.close();
            LOG.warn("This HiveServer2 instance is now de-registered from ZooKeeper. "
                    + "The server will be shut down after the last client sesssion completes.");
          } catch (IOException e) {
            LOG.error("Failed to close the persistent ephemeral znode", e);
          } finally {
            HiveServer2.this.setDeregisteredWithZooKeeper(true);
            // If there are no more active client sessions, stop the server
            if (cliService.getSessionManager().getOpenSessionCount() == 0) {
              LOG.warn("This instance of HiveServer2 has been removed from the list of server "
                      + "instances available for dynamic service discovery. "
                      + "The last client session has ended - will shutdown now.");
              HiveServer2.this.stop();
            }
          }
        }
      }
    }
  }

  private void removeServerInstanceFromZooKeeper() throws Exception {
    setDeregisteredWithZooKeeper(true);

    if (znode != null) {
      znode.close();
    }
    zooKeeperClient.close();
    LOG.info("Server instance removed from ZooKeeper.");
  }
  private void setDeregisteredWithZooKeeper(boolean deregisteredWithZooKeeper) {
    this.deregisteredWithZooKeeper = deregisteredWithZooKeeper;
  }
  public String getServerHost() throws Exception {
    if ((thriftCLIService == null) || (thriftCLIService.getServerIPAddress() == null)) {
      throw new Exception("Unable to get the server address; it hasn't been initialized yet.");
    }
    return thriftCLIService.getServerIPAddress().getHostName();
  }
  /**
   * Add conf keys, values that HiveServer2 will publish to ZooKeeper.
   * @param hiveConf
   */
  private void addConfsToPublish(HiveConf hiveConf, Map<String, String> confsToPublish) {
    // Hostname
    confsToPublish.put(ConfVars.HIVE_SERVER2_THRIFT_BIND_HOST.varname,
            hiveConf.getVar(ConfVars.HIVE_SERVER2_THRIFT_BIND_HOST));
    // Transport mode
    confsToPublish.put(ConfVars.HIVE_SERVER2_TRANSPORT_MODE.varname,
            hiveConf.getVar(ConfVars.HIVE_SERVER2_TRANSPORT_MODE));
    // Transport specific confs
    if (isHTTPTransportMode(hiveConf)) {
      confsToPublish.put(ConfVars.HIVE_SERVER2_THRIFT_HTTP_PORT.varname,
              Integer.toString(hiveConf.getIntVar(ConfVars.HIVE_SERVER2_THRIFT_HTTP_PORT)));
      confsToPublish.put(ConfVars.HIVE_SERVER2_THRIFT_HTTP_PATH.varname,
              hiveConf.getVar(ConfVars.HIVE_SERVER2_THRIFT_HTTP_PATH));
    } else {
      confsToPublish.put(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname,
              Integer.toString(hiveConf.getIntVar(ConfVars.HIVE_SERVER2_THRIFT_PORT)));
      confsToPublish.put(ConfVars.HIVE_SERVER2_THRIFT_SASL_QOP.varname,
              hiveConf.getVar(ConfVars.HIVE_SERVER2_THRIFT_SASL_QOP));
    }
    // Auth specific confs
    confsToPublish.put(ConfVars.HIVE_SERVER2_AUTHENTICATION.varname,
            hiveConf.getVar(ConfVars.HIVE_SERVER2_AUTHENTICATION));
    if (isKerberosAuthMode(hiveConf)) {
      confsToPublish.put(ConfVars.HIVE_SERVER2_KERBEROS_PRINCIPAL.varname,
              hiveConf.getVar(ConfVars.HIVE_SERVER2_KERBEROS_PRINCIPAL));
    }
    // SSL conf
    confsToPublish.put(ConfVars.HIVE_SERVER2_USE_SSL.varname,
            Boolean.toString(hiveConf.getBoolVar(ConfVars.HIVE_SERVER2_USE_SSL)));
  }
  public static boolean isKerberosAuthMode(HiveConf hiveConf) {
    String authMode = hiveConf.getVar(HiveConf.ConfVars.HIVE_SERVER2_AUTHENTICATION);
    if (authMode != null && (authMode.equalsIgnoreCase("KERBEROS"))) {
      return true;
    }
    return false;
  }

Spark 编译

apache-maven-3.6.3
scala-2.12.15

全部编译

1	./build/mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.4 -Phive -Phive-thriftserver -Dscala-2.12 -DskipTests clean package

指定模块编译

1	./build/mvn -Pyarn -Phive -Phive-thriftserver -DskipTests -pl sql/hive-thriftserver -am clean package

编译过程中遇到的问题

[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.3.0:compile (scala-compile-first) on project spark-hive-thriftserver_2.12: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.3.0:compile failed: java.lang.AssertionError: assertion failed: Expected protocol to be 'file' or empty in URI jar:file:/D:/Repositories/Maven/org/apache/zookeeper/zookeeper/3.4.14/zookeeper-3.4.14.jar!/org/apache/zookeeper/ZooDefs$Ids.class -> [Help 1]

解决方式：
修改 scala-maven-plugin 版本：4.3.0 改为 4.5.6

 <groupId>net.alchim31.maven</groupId>
 <artifactId>scala-maven-plugin</artifactId>
 <version>4.5.6</version>

验证

替换 spark-hive-thriftserver_2.12-3.1.2.jar

将编译好的包 spark-hive-thriftserver_2.12-3.1.2.jar，替换到 Spark 安装路径 jars/ 下，启动 Spark

修改 Spark conf 下 hive-site 配置

添加：

<property>
  <name>hive.server2.support.dynamic.service.discovery</name>
  <value>true</value>
</property>

<property>
  <name>hive.server2.zookeeper.namespace</name>
  <value>hiveserver2_zk</value>
</property>

<property>
  <name>hive.zookeeper.quorum</name>
  <value>192-168-44-129.indata.com:2181,indata-1192-168-44-128:2181,indata-192-168-44-130.indata.com:2181</value>
</property>

<property>
  <name>hive.zookeeper.client.port</name>
  <value>2181</value>
</property>

启动 Spark ThriftServer

spark-submit --master yarn --deploy-mode client --num-executors 2 --executor-cores 1 --executor-memory 1G --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name SparkThriftServer_1 spark-internal --hiveconf hive.server2.thrift.http.port=20004

连接验证

[root@indata-192-168-44-128 spark-3.1.2]./bin/beeline
beeline> !connect jdbc:hive2://192-168-44-129.indata.com:2181,indata-1192-168-44-128:2181,indata-192-168-44-130.indata.com:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2_zk
22/02/11 08:57:31 INFO ConnectionStateManager: State change: CONNECTED
22/02/11 08:57:31 INFO ZooKeeper: Session: 0x27ed86adc0401d2 closed
22/02/11 08:57:31 INFO Utils: Resolved authority: indata-1192-168-44-128:20004
22/02/11 08:57:31 INFO ClientCnxn: EventThread shut down for session: 0x27ed86adc0401d2
22/02/11 08:57:31 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Connected to: Spark SQL (version 3.1.2)
Driver: Hive JDBC (version 2.3.7)
Transaction isolation: TRANSACTION_REPEATABLE_READ

验证过程中遇到的问题

zookeeper解析不到IP

22/02/11 09:08:46 WARN HiveConnection: Failed to connect to :20004
22/02/11 09:08:46 INFO CuratorFrameworkImpl: Starting
22/02/11 09:08:46 INFO ZooKeeper: Initiating client connection, connectString=192-168-44-129.indata.com:2181,indata-1192-168-44-128:2181,indata-192-168-44-130.indata.com:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@3da30852
22/02/11 09:08:46 INFO ClientCnxn: Opening socket connection to server indata-192-168-44-130.indata.com/192.168.44.130:2181. Will not attempt to authenticate using SASL (unknown error)
22/02/11 09:08:46 INFO ClientCnxn: Socket connection established to indata-192-168-44-130.indata.com/192.168.44.130:2181, initiating session
22/02/11 09:08:46 INFO ClientCnxn: Session establishment complete on server indata-192-168-44-130.indata.com/192.168.44.130:2181, sessionid = 0x37ed86adc050208, negotiated timeout = 60000
22/02/11 09:08:46 INFO ConnectionStateManager: State change: CONNECTED
22/02/11 09:08:46 INFO ZooKeeper: Session: 0x37ed86adc050208 closed
22/02/11 09:08:46 INFO ClientCnxn: EventThread shut down for session: 0x37ed86adc050208
22/02/11 09:08:46 ERROR Utils: Unable to read HiveServer2 configs from ZooKeeper
Error: Could not open client transport for any of the Server URI's in ZooKeeper: Host name may not be empty (state=08S01,code=0)

解决方案

通过 Hive JDBC Connector 代码连接zookeeper地址进行debug，发现 hive.server2.thrift.bind.host 值为空，
同时，zookeeper命令查看节点值：

1
2

[zk: indata-1192-168-44-128:2181(CONNECTED) 5] get  /hiveserver2_zk/serverUri=indata-1192-168-44-128:20004;version=2.3.7;sequence=0000000023
hive.server2.authentication=KERBEROS;hive.server2.transport.mode=http;hive.server2.thrift.http.path=cliservice;hive.server2.thrift.http.port=20004;hive.server2.thrift.bind.host=;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=spark/_HOST@INDATA.COM

hive.server2.thrift.bind.host值同样为空，定位到ThriftServer注册到zookeeper时产生问题
zookeeper注册代码：

private void addConfsToPublish(HiveConf hiveConf, Map<String, String> confsToPublish) {
    // Hostname
    confsToPublish.put(ConfVars.HIVE_SERVER2_THRIFT_BIND_HOST.varname,
    hiveConf.getVar(ConfVars.HIVE_SERVER2_THRIFT_BIND_HOST));

//hive-site没有 HIVE_SERVER2_THRIFT_BIND_HOST 配置项，因此为空。
//通过getServerHost方法动态获取该配置
hiveConf.set(HiveConf.ConfVars.HIVE_SERVER2_THRIFT_BIND_HOST.varname, getServerHost)

代码

完整代码已提交到：https://gitee.com/dongkelun/spark/tree/3.1.2-STS-HA/

本文由 董可伦 发表于伦少的博客 ,采用署名-非商业性使用-禁止演绎 3.0进行许可。

非商业转载请注明作者及出处。商业转载请联系作者本人。

本文标题：Spark Thrift Server HA 解决方案

本文链接：https://dongkelun.com/2022/02/11/sparkThriftServerHA/

欢迎关注我的公众号