

生态对接常见问题总结¶

适用场景¶

生态对接常见问题总结 1.0 ↔ FusionInsight HD 6.5 (HDFS)

Kerberos认证相关问题¶

`kinit: Cannot contact any KDC for realm 'HADOOP.COM' while getting initial credentials`¶

报错原因：集群环境双平面部署，使用在管理平面网段的跳板机尝试对接集群会报类似问题
解决办法：将跳板机转接到业务平面网段上进行对接
经验分享：对接一定是在业务平面网段上同集群各组件进行交互，测试开始前请 务必先检查网络平面状况 ，再进行对接测试

`Cannot locate KDC`¶

报错片段：

Caused by: KrbException: Cannot locate KDC
        at sun.security.krb5.Config.getKDCList(Config.java:1084)
        at sun.security.krb5.KdcComm.send(KdcComm.java:218)
        at sun.security.krb5.KdcComm.send(KdcComm.java:200)
        at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:316)
        at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:361)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:776)
        ... 24 more
Caused by: KrbException: Generic error (description in e-text) (60) - Unable to locate KDC for realm HADOOP.COM
        at sun.security.krb5.Config.getKDCFromDNS(Config.java:1181)
        at sun.security.krb5.Config.getKDCList(Config.java:1057)

报错原因：跳板机krb5.conf配置文件配置问题导致跳板机找不到集群KDC服务器地址，无法进行认证
解决办法：在正确的配置路径上配置好krb5.conf文件
经验分享：跳板机根据系统不同分为windwos，Linux。
- 默认情况下windows主机从C:\Windows\krb5.ini文件获取KDC服务器信息（krb5.ini文件内容和krb5.conf文件内容一致，只是后缀名不同），对接测试前 务必检查该文件内容
- 默认情况下linux主机从/etc/krb5.conf文件获取KDC服务器信息，对接测试前 务必检查该文件内容
- 对接过程如果配置了JVM启动参数-Djava.security.krb5.conf，一般会以改启动参数配置的krb5.conf具体路径为准，对接测试前 务必检查该文件内容

`Clock skew too great`¶

报错片段

GSSException: No valid credentials provided
(Mechanism level: Attempt to obtain new INITIATE credentials failed! (null))
 . . . Caused by: javax.security.auth.login.LoginException: Clock skew too great

GSSException: No valid credentials provided (Mechanism level: Clock skew too great (37) - PROCESS_TGS

kinit: krb5_get_init_creds: time skew (343) larger than max (300)

报错原因：跳板机时间与集群时间相差太多（FusionInsight 默认5分钟以内）
解决办法：修改跳板机时间使与集群时间相差在5分钟以内
经验分享：该报错关键词能很快定位出报错原因，但有时并不会直接以Clock skew too great为关键词报错，而是以GSSException: No valid credentials provided其他形式报错，就会比较容易忽略检查时间、时区是否正确。对接测试前请 务必检查跳板机与集群时间是否在相差范围以内（5分钟）
错误来源： FI三方AD服务配置完成后使用hdfs dfs -ls命令检查结果报错

`GSS initiate failed` 没有其他报错关键词¶

报错片段

WARN  ipc.Client (Client.java:run(676)) - Couldn't setup connection for rm@EXAMPLE.COM to /172.22.97.127:8020
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
  at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:375)
  at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:558)

该报错关键词只是表明认证失败的结果，并看不出来任何连接失败的原因，需要在报错日志附近找到Caused by等关键词定位到具体连接失败的原因，或者打开JVM报错日志开关-Dsun.security.krb5.debug=false，结合集群kdc服务日志/var/log/Bigdata/kerberos/krb5kdc.log分析原因
错误来源：tensorflow对接时有过改报错，打开JVM报错日志开关检查出的报错原因

`No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)`¶

报错片段

javax.security.sasl.SaslException:
GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

报错原因：简言之认证失败
- 可能没使用kinit预先认证
- 可能kinit预先认证后，缓存的认证票据过期
- 可能使用的user.keytab认证文件错误，和认证用户不匹配，或者该认证文件不存在
- 可能缓存的票据被其他程序更改
- 可能认证用户的域名（Realm）和集群的域名不一致
解决办法：确认认证用户，密码，user.keytab的正确性，确保认证过程中没有被其他的任务干扰
经验分享：对于keytab文件的正确性验证，可以在跳板机上安装客户端，使用命令kinit -kt 认证keytab文件认证principal的方式来验证

`No JAAS configuration section named 'Client' was found in specified JAAS configuration file`¶

报错片段

WARN  [2020-08-05 06:54:19,089] org.apache.zookeeper.ClientCnxn: SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/opt/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.

报错原因： jaas.conf配置文件内容缺少Client片段
解决办法：检查jaas.conf配置文件内容正确
经验分享： jaas.conf配置文件中的Client片段是连接Zookeeper组件的，KafkaClient片段是连接Kafka安全模式的，请根据报错内容确认相关片段的内容正确

`KeeperErrorCode = XXX`¶

报错片段1(hive).

Caused by: org.apache.hive.jdbc.ZooKeeperHiveClientException: Unable to read HiveServer2 configs from ZooKeeper
    at org.apache.hive.jdbc.ZooKeeperHiveClientHelper.configureConnParams(ZooKeeperHiveClientHelper.java:100)
    at org.apache.hive.jdbc.Utils.configureConnParams(Utils.java:514)
    at org.apache.hive.jdbc.Utils.parseURL(Utils.java:435)
    at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:141)
    ... 34 common frames omitted
Caused by: org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /hiveserver2
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1590)

报错片段2(hbase).

 org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = Session expired for /hbase
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:157)
        at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4212)
        at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:515)

报错片段3(hetu).

Caused by: java.sql.SQLException: KeeperErrorCode = ConnectionLoss for /hetuserver/hsbroker/registry
  at g5.qry.sql.impl.SQLConnBuilder.build0(SQLConnBuilder.java:69) ~

报错片段4(kafka).

Causing: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /brokers/topics/hdfs_audit_log_sandbox/partitions
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[eagle-topology-0.5.0-assembly.jar:na]
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[eagle-topology-0.5.0-assembly.jar:na]
 at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1590) ~[eagle-topology-0.5.0-assembly.jar:na]

报错原因：尝试连接Zookeeper组件失败，可能发生在连接hive，hbase，hetu，kafka的过程中
- JVM启动参数未配置或者内容有问题
- JVM启动参数并未在三方软件启动时成功导入
- JVM参数导入的jaas.conf配置文件不存在
- jaas.conf配置文件中配置的keytab文件，principal配置不正确
- 三方软件匹配的zookeeper依赖jar包版本较老，需更换为华为的zookeeper依赖jar包，比如zookeeper-3.5.1.jar（651版本）
解决办法：
- 检查JVM启动参数内容，比如：-Djava.security.auth.login.config=/opt/jaas.conf -Dzookeeper.server.principal=zookeeper/hadoop.hadoop.com -Djava.security.krb5.conf=/etc/krb5.conf
- 检查上述配置相关的配置文件jaas.conf，krb5.conf文件内容无误
- 确保三方软件启动时上述JVM参数成功导入，linux可使用ps -ef命令查看，windows可使用visualvm工具进行监控
- 确保对接相关zookeeper依赖jar包替换为华为的zookeeper依赖jar包，比如zookeeper-3.5.1.jar（651版本）
经验分享：该问题为对接中最为常见的错误，请参考解决办法的要点，逐条分析

`GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]`¶

报错片段

ERROR [AdminClient clientId=adminclient-1] Connection to node -2 failed authentication due to: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state. (org.apache.kafka.clients.NetworkClient:607)
[2019-06-26 18:05:06,410] WARN [Principal=developuser@HADOOP.COM]: TGT renewal thread has been interrupted and will exit. (org.apache.kafka.common.security.kerberos.KerberosLogin:192)
[2019-06-26 18:05:06,412] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed:112)
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
        at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
        at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:45)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)

报错原因：FusionInsight的KDC服务器没有对应的服务端票据，该错误最常发生在连接kafka安全模式（21007端口），以及一些HTTP请求中（SPENGO），以及有的时候连接zookeeper的时候
解决办法：
- 对接Kafka安全模式请务必更换对应的kafka client依赖jar包为kafka-clients-1.1.0.jar（651版本），不要使用开源的该jar包，华为对该jar包有修改
- 如果是HTTP请求比如SPENGO认证，可以考虑在KDC服务器中手动添加相关的服务票
- 对接zookeeper的是有jvm参数需要引入: -Dzookeeper.server.principal=zookeeper/hadoop.hadoop.com
经验分享：该问题为对接kafka安全模式中最常见的问题，关键点就在于需要替换华为kafka-clients-1.1.0.jar（651版本）依赖包。并且如果遇到集群域名更改的情况，需要想办法在三方软件中把域名参数kerberos.domain.name输入进去，比如kerberos.domain.name=hadoop.hadoopss.com

定位该问题最好的办法是在jvm参数中加入kerberos debug日志开启参数-Dsun.security.krb5.debug=true，并且登陆集群KDC服务器，查看相关日志/var/log/Bigdata/kerberos/krb5kdc.log，找出错误原因

`ICMP Port Unreachable`¶

问题原因：三方软件根据开源的默认规则，使用udp 88端口进行kerberos认证，而FusionInsight 使用udp 21732端口进行kerberos认证
解决办法：使用udp端口更改工具（https://github.com/troglobit/uredir ），将该工具部署在KDC服务器上，执行命令./uredir IP:88 IP:21732进行端口绑定
经验分享：一般而言开源kerberos对接使用的是88端口，可以使用抓包工具进行检查，比如 windows可使用Microsoft Network Monitor工具检查，linux可使用tcpdump工具进行检查

`javax.security.auth.login.LoginException: No password provided`¶

报错片段

2015-12-15 17:16:23,517 - WARN  [main:SaslServerCallbackHandler@105] - No password found for user: null
2015-12-15 17:16:23,536 - ERROR [main:ZooKeeperServerMain@63] - Unexpected exception, exiting abnormally
java.io.IOException: Could not configure server because SASL configuration did
not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: No password provided
  at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207)
  at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87)

问题原因：
- keytab文件配置错误
- keytab文件使用的principal同认证用户不匹配
- keytab文件并没有配置进去，三方软件没有读到
解决办法：参考问题原因检查，配置正确

`javax.security.auth.login.LoginException: Unable to obtain password from user`¶

报错片段

Exception in thread "main" java.io.IOException:
   Login failure for xxx@REALM from keytab /opt/xx.keytab:
  javax.security.auth.login.LoginException: Unable to obtain password from user
  at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:962)
  at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:564)
  at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:154)
  at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
  at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
  at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856)
  at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)

问题原因：跟上一个问题类似
- keytab文件配置错误
- keytab文件使用的principal同认证用户不匹配
- keytab文件并没有配置进去，三方软件没有读到
解决办法：参考问题原因检查，配置正确

`SIMPLE authentication is not enabled. Available:[TOKEN]"`¶

问题原因：改报错一般出现在客户端，客户端尝试使用普通模式（SIMPLE）进行认证，但是服务端只支持Kerberos认证（TOKEN）
解决办法：在客户端配置文件中把hadoop.security.authentication改成kerberos，或者通过别的配置方式，让三方软件使用Kerberos进行认证

集群配置相关问题¶

`java.lang.ClassNotFoundException`¶

报错片段

java.lang.IllegalArgumentException: Compression codec com.huawei.hadoop.datasight.io.compress.lzc.ZCodec not found.
        at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:139)
        at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:180)
        at org.apache.nifi.processors.hadoop.GetHDFS.processBatchOfFiles(GetHDFS.java:353)

问题原因：三方软件对应组件依赖的jar包确实
解决办法：在华为FusionInsight客户端找到对应类的jar包，在将依赖的jar包引入
经验分享：一般通过命令grep -R "缺失的类名" /opt/hadoopclient在客户端中查找对应jar包，比如该片段缺少的jar包为hadoop-plugins-1.0.jar（651版本）

`java.lang.NoSuchMethodError`¶

报错片段

java.lang.NoSuchMethodError: org.apache.kafka.common.acl.AclBindingFilter.<init>(Lorg/apache/kafka/common/resource/ResourcePatternFilter;Lorg/apache/kafka/common/acl/AccessControlEntryFilter;)V
        at org.apache.kafka.connect.mirror.MirrorSourceConnector.<clinit>(MirrorSourceConnector.java:67)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

问题原因：三方软件相关组件的依赖库同华为FusionInsight对应组件的依赖库不匹配
解决办法：如果遇到类似报错，要具体定位到是哪些依赖jar包错误是比较困难的，没有具体的解决办法。在对接过程中尝试加入自定义的依赖库，或者选取和FusionInsight现网版本相近的开源版本依赖库进行对接
经验分享：该报错原因就是依赖jar包的不匹配，遇到该报错是很难定位到具体依赖jar包的，需要根据解决办法的思路尝试从其他的思路来解决

`java.net.UnknownHostException: hacluster`¶

报错片段

java.lang.IllegalArgumentException: java.net.UnknownHostException: hacluster
        at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:444)
        at org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:132)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:355)

问题原因: 三方软件不支持HDFS高可用性，或者高可用性没有配置正确
解决办法： - 如果三方软件不支持高可用性配置，修改导入的hdfs配置文件core-site.xml文件中的配置项如下：
```
<property>
<name>fs.defaultFS</name>
<value>hdfs://主namenode节点ip:21005</value>
</property>
```
- 参考三方软件文档，配置高可用性
经验分享：该报错原因就是配置文件中的配置项有问题，比如core-site.xml配置文件中的fs.defaultFS，有时在对接hbase的时候也会有类似问题出现，可以在hbase-site.xml配置文件中查找相关配置项进行更改

`Cannot modify XXX at runtime. It is not in list of params that are allowed to be modified at runtime`¶

问题原因: hive白名单没有开通该相关参数
解决办法：登陆FusionInsight Manager -> 集群 -> Hive -> 全部配置 -> hive.security.authorization.sqlstd.confwhitelist.append -> 添加报错缺失的白名单

例如：报错提示 Cannot modify mapred.job.name at runtime.

对应的在 hive.security.authorization.sqlstd.confwhitelist.append 配置项的末尾添加内容： |mapred\.job\.name

说明：hive.security.authorization.sqlstd.confwhitelist.append这个参数中的内容以|分隔，填写的内容需要用转义字符\

`Permission denied`¶

报错片段

HiveAccessControlException Permission denied: Principal [name=developuser, type=USER] does not have following privileges for operation CREATETABLE [[OBJECT OWNERSHIP] on Object [type=DFS_URI, name=hdfs://hacluster/user/hive/warehouse/sdc_drift_example03]]

问题原因：认证用户在FusionInsight上没有对应的权限
解决办法：仔细查看报错，明确哪个用户，在什么对象上，没有什么权限，再去FusionInsight Manager上配置相关权限。

具体参考《产品文档》 -> 《应用开发指南》 -> 《安全模式》 -> 《安全认证》 -> 《准备开发用户》相关章节

`Failed to connect to driver at xxx:端口 retrying`¶

说明：该报错出现的场景为同spark2x组件对接中任务失败，报错出现的位置在FusionInsight Yarn上查看相关失败任务的container日志stout中

问题原因：使用yarn-client模式提交spark任务，集群不知道driver的主机名
解决办法：在对应集群worker的节点上，修改/etc/hosts配置文件，将提交任务的主机名加进去

生态对接常见问题总结¶

适用场景¶

Kerberos认证相关问题¶

kinit: Cannot contact any KDC for realm 'HADOOP.COM' while getting initial credentials¶

Cannot locate KDC¶

Clock skew too great¶

GSS initiate failed 没有其他报错关键词¶

No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)¶

No JAAS configuration section named 'Client' was found in specified JAAS configuration file¶

KeeperErrorCode = XXX¶

GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]¶

ICMP Port Unreachable¶

javax.security.auth.login.LoginException: No password provided¶

javax.security.auth.login.LoginException: Unable to obtain password from user¶

SIMPLE authentication is not enabled. Available:[TOKEN]"¶

集群配置相关问题¶

java.lang.ClassNotFoundException¶

java.lang.NoSuchMethodError¶

java.net.UnknownHostException: hacluster¶

Cannot modify XXX at runtime. It is not in list of params that are allowed to be modified at runtime¶

Permission denied¶

Failed to connect to driver at xxx:端口 retrying¶

`kinit: Cannot contact any KDC for realm 'HADOOP.COM' while getting initial credentials`¶

`Cannot locate KDC`¶

`Clock skew too great`¶

`GSS initiate failed` 没有其他报错关键词¶

`No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)`¶

`No JAAS configuration section named 'Client' was found in specified JAAS configuration file`¶

`KeeperErrorCode = XXX`¶

`GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]`¶

`ICMP Port Unreachable`¶

`javax.security.auth.login.LoginException: No password provided`¶

`javax.security.auth.login.LoginException: Unable to obtain password from user`¶

`SIMPLE authentication is not enabled. Available:[TOKEN]"`¶

`java.lang.ClassNotFoundException`¶

`java.lang.NoSuchMethodError`¶

`java.net.UnknownHostException: hacluster`¶

`Cannot modify XXX at runtime. It is not in list of params that are allowed to be modified at runtime`¶

`Permission denied`¶

`Failed to connect to driver at xxx:端口 retrying`¶