We noticed a problem the last couple of days where Hive was hanging while doing a “LOAD DATA” command. This would be a symptom of Hive being unable to write to HDFS in general. So, a “CREATE TABLE AS SELECT” would fail, also.
I did an strace on the hive command:
strace -f hive -e "LOAD DATA ..."
I found that hive was trying to connect to the ZooKeeper server, but that the connections were timing out.
I looked into the ZooKeeper settings to make sure that it was allowed to run enough threads to meet everyone’s needs, and it was.
However, I also found a setting that was limiting the number of connections per IP address:
maxClientCnxns=60
Because this server was also running Hue for all of our users, it was running out of connections allowed. I increased this number significantly, restarted ZooKeeper, and we haven’t had any problems since.