Hive on Spark parameters

The timeout for the job monitor to get a Spark job state (in seconds)

60

When set to true, enables dynamic partition pruning (DPP) for the Spark engine so that joins on partition keys are processed by writing to a temporary HDFS file, and read later for removing unnecessary partitions

false

Similar to hive.spark.dynamic.partition.pruning, but enables DPP only if the join on the partitioned table can be converted to a MapJoin

false

The maximum data size for the dimension table that generates partition pruning information (in megabytes). If a table reaches this limit, the optimization is disabled

100

Updates Spark job execution progress in-place in the terminal

true

hive.spark.use.ts.stats.for.mapjoin

If set to true, the MapJoin optimization in Hive/Spark uses statistics from TableScan operators at the root of the operator tree, instead of parent ReduceSink operators of the Join operator

false

Defines whether to show EXPLAIN result at the user level for Hive-on-Spark queries. When enabled, logs EXPLAIN output for the query at the user level

false

Time to wait to finish pre-warming Spark executors when hive.prewarm.enabled=true (in milliseconds)

5000

hive.spark.optimize.shuffle.serde

If set to true, Hive on Spark registers custom serializers for data types in shuffle. This should result in less shuffled data

false

hive.merge.sparkfiles

Merges small files at the end of a Spark DAG Transformation

false

Defines whether to use operator stats to determine reducer parallelism for Hive on Spark. If set to false, Hive will use source table stats to determine reducer parallelism for all first level reduce tasks, and the maximum reducer parallelism from all parents for all the rest (second level and onward) reducer tasks. Setting to false triggers an alternative algorithm for calculating the number of partitions per Spark shuffle. This new algorithm typically results in an increased number of partitions per shuffle

true

If set to true, MapJoin optimization in Hive/Spark will use statistics from TableScan operators at the root of operator tree, instead of parent ReduceSink operators of the Join operator. Setting this to true is useful when the operator statistics used for a common join-to-map join conversion are inaccurate

false

When set to true, uses Spark’s RDD#groupByKey() to perform groupings. When set to false, use Spark’s RDD#repartitionAndSortWithinPartitions() to perform groupings. While groupByKey() has better performance when running grouping, it can use an excessive amount of memory. Setting this to false may reduce memory usage, but will affect the performance

true