使用hive3.x 问题记录

本文主要是介绍使用hive3.x 问题记录，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

使用 hive3.1.2 问题记录

搭建步骤省略 ...

一、HIVE ON MR

1、第二次以上插入值 insert into table tablename values("xx")；出现错误:

1)、metastore.RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to reconnect (1 of 1) after 1s. setPartitionColumnStatistics
org.apache.thrift.transport.TTransportException: null

解决：当hiveserver2和 metastore 服务在一个机器会出现这种错误分两台机器起就会报2错误这才是本质

2)、 Error occurred during processing of message.
java.lang.ClassCastException: org.apache.hadoop.hive.metastore.api.StringColumnStatsData cannot be cast to org.apache.hadoop.hive.metastore.columnstats.cache.StringColumnStatsDataInspector

解释一：网上搜索是一种Bug
https://issues.apache.org/jira/browse/HIVE-20151
https://issues.apache.org/jira/browse/HIVE-19316
解释二:
列状态为True会触发这种问题

临时解决:
1、建表之前设置set hive.stats.autogather=false；
2、或者建表之后执行load data inpath “/xxx.txt” into table tal_name;
1、2 方法可以去掉表的列状态为True的情况即可重复执行INSERT INTO tb_name values

一、HIVE ON TEZ

说明：使用的是hive3.1.2 + tez0.9.2

1、错误信息
java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V

网上有人说将hadoop /share/hadoop/common/lib/guava-27.0-jre.jar
路径下的jar替换 tez/lib下的低版本guava jar就可以
我试了还是报这种错误
于是我将tez升级到0.10版本解决这个问题

2、执行卡住是申请的资源不足导致
没有进度一直为0
通过设置set hive.tez.container.size=512解决

<property>
<name>hive.tez.container.size</name>
<value>-1</value>
<description>By default Tez will spawn containers of the size of a mapper. This can be used to overwrite.</description>   </property>
其他影响参数可以尝试设置解决 set hive.tez.java.opts=-Xmx1640m; set
tez.runtime.io.sort.mb=820; set
tez.runtime.unordered.output.buffer.size-mb=205;

其余相关配置参考如下：
yarn-site配置

   <property><name>yarn.nodemanager.env-whitelist</name><value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value></property><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><property><name>yarn.resourcemanager.hostname</name><value>hadoop1</value></property><!-- 是否开启聚合日志 --><property><name>yarn.log-aggregation-enable</name><value>true</value></property><property><name>yarn.scheduler.minimum-allocation-mb</name><value>512</value></property><property><name>yarn.scheduler.maximum-allocation-mb</name><value>8192</value></property><property><name>yarn.nodemanager.resource.memory-mb</name><value>4096</value></property><property><name>yarn.nodemanager.pmem-check-enabled</name><value>false</value></property><property><name>yarn.nodemanager.vmem-check-enabled</name><value>false</value></property><property><name>yarn.nodemanager.resource.cpu-vcores</name><value>8</value></property><!-- 聚合日志在hdfs上的url --><property><name>yarn.log.server.url</name><value>http://hadoop1:19888/jobhistory/logs</value></property><!-- 参数说明：聚合后的日志在HDFS上保存多长时间，单位为s。--><property><name>yarn.log-aggregation.retain-seconds</name><value>604800</value></property><property><name>yarn.resourcemanager.webapp.address.rm1</name><value>hadoop0:8088</value></property><property><name>yarn.resourcemanager.webapp.address.rm2</name><value>hadoop1:8088</value></property><!--启用resourcemanager ha--><property><name>yarn.resourcemanager.ha.enabled</name><value>true</value></property><!--声明两台resourcemanager的地址--><property><name>yarn.resourcemanager.cluster-id</name><value>cluster-yarn1</value></property><property><name>yarn.resourcemanager.ha.rm-ids</name><value>rm1,rm2</value></property><property><name>yarn.resourcemanager.hostname.rm1</name><value>hadoop0</value></property><property><name>yarn.resourcemanager.hostname.rm2</name><value>hadoop1</value></property><!--指定zookeeper集群的地址--><property><name>yarn.resourcemanager.zk-address</name><value>hadoop0:2181,hadoop1:2181,hadoop2:2181</value></property><!--启用自动恢复--><property><name>yarn.resourcemanager.recovery.enabled</name><value>true</value></property><!--指定resourcemanager的状态信息存储在zookeeper集群--><property><name>yarn.resourcemanager.store.class</name><value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value></property>
</configuration>

tez-site配置

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property><name>tez.lib.uris</name><value>${fs.defaultFS}/tez/tez.tar.gz</value>
</property>
<property><name>tez.use.cluster.hadoop-libs</name><value>false</value>
</property>
<property><name>tez.history.logging.service.class</name><value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
</property><property><name>tez.am.resource.memory.mb</name><value>1024</value>
</property><property><name>tez.am.resource.cpu.vcores</name><value>1</value>
</property>
<property><name>tez.task.resource.memory.mb</name><value>1024</value>
</property>
</configuration>

hive-site配置

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration><property><name>hive.metastore.uris</name><value>thrift://192.168.187.192:9083</value></property><property><name>hive.execution.engine</name><value>tez</value></property><property><name>hive.tez.container.size</name><value>1024</value>  </property>
</configuration>