org.apache.hadoop.hive.metastore.txn

Class CompactionTxnHandler

java.lang.Object

org.apache.hadoop.hive.metastore.txn.TxnHandler

org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler

public class CompactionTxnHandler
extends TxnHandler

Extends the transaction handler with methods needed only by the compactor threads. These methods are not available through the thrift interface.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.metastore.txn.TxnHandler

TxnHandler.DatabaseProduct, TxnHandler.RetryException

Field Summary

Fields inherited from class org.apache.hadoop.hive.metastore.txn.TxnHandler

CLEANING_RESPONSE, conf, dbProduct, deadlockCnt, INITIATED_RESPONSE, INITIATED_STATE, LOCK_ACQUIRED, LOCK_EXCLUSIVE, LOCK_SEMI_SHARED, LOCK_SHARED, LOCK_WAITING, MAJOR_TYPE, MINOR_TYPE, READY_FOR_CLEANING, TXN_ABORTED, TXN_OPEN, WORKING_RESPONSE, WORKING_STATE

Constructor Summary

Constructors

Constructor and Description
CompactionTxnHandler(org.apache.hadoop.hive.conf.HiveConf conf)

Method Summary

Methods

Modifier and Type	Method and Description
void	cleanEmptyAbortedTxns() Clean up aborted transactions from txns that have no components in txn_components.
static org.apache.hadoop.hive.common.ValidTxnList	createValidCompactTxnList(GetOpenTxnsInfoResponse txns) Transform a GetOpenTxnsInfoResponse to a ValidTxnList.
List<String>	findColumnsWithStats(CompactionInfo ci) Queries metastore DB directly to find columns in the table which have statistics information.
CompactionInfo	findNextToCompact(String workerId) This will grab the next compaction request off of the queue, and assign it to the worker.
Set<CompactionInfo>	findPotentialCompactions(int maxAborted) This will look through the completed_txn_components table and look for partitions or tables that may be ready for compaction.
List<CompactionInfo>	findReadyToClean() Find entries in the queue that are ready to be cleaned.
void	markCleaned(CompactionInfo info) This will remove an entry from the queue after it has been compacted.
void	markCompacted(CompactionInfo info) This will mark an entry in the queue as compacted and put it in the ready to clean state.
void	revokeFromLocalWorkers(String hostname) This will take all entries assigned to workers on a host return them to INITIATED state.
void	revokeTimedoutWorkers(long timeout) This call will return all compaction queue entries assigned to a worker but over the timeout back to the initiated state.
void	setRunAs(long cq_id, String user) Sets the user to run as.

Methods inherited from class org.apache.hadoop.hive.metastore.txn.TxnHandler

abortTxn, checkLock, checkRetryable, closeDbConn, closeStmt, commitTxn, compact, createValidReadTxnList, determineDatabaseProduct, getDbConn, getDbTime, getIdentifierQuoteString, getOpenTxns, getOpenTxnsInfo, heartbeat, heartbeatTxnRange, lock, lockNoWait, openTxns, showCompact, showLocks, unlock

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CompactionTxnHandler

public CompactionTxnHandler(org.apache.hadoop.hive.conf.HiveConf conf)

Method Detail

findPotentialCompactions

public Set<CompactionInfo> findPotentialCompactions(int maxAborted)
                                             throws MetaException

This will look through the completed_txn_components table and look for partitions or tables that may be ready for compaction. Also, look through txns and txn_components tables for aborted transactions that we should add to the list.

Parameters:

maxAborted - Maximum number of aborted queries to allow before marking this as a potential compaction.

Returns:

list of CompactionInfo structs. These will not have id, type, or runAs set since these are only potential compactions not actual ones.

Throws:

MetaException

setRunAs

public void setRunAs(long cq_id,
            String user)
              throws MetaException

Sets the user to run as. This is for the case where the request was generated by the user and so the worker must set this value later.

Parameters:

cq_id - id of this entry in the queue

user - user to run the jobs as

Throws:

MetaException

findNextToCompact

public CompactionInfo findNextToCompact(String workerId)
                                 throws MetaException

This will grab the next compaction request off of the queue, and assign it to the worker.

Parameters:

workerId - id of the worker calling this, will be recorded in the db

Returns:

an info element for this compaction request, or null if there is no work to do now.

Throws:

MetaException

markCompacted

public void markCompacted(CompactionInfo info)
                   throws MetaException

This will mark an entry in the queue as compacted and put it in the ready to clean state.

Parameters:

info - info on the compaction entry to mark as compacted.

Throws:

MetaException

findReadyToClean

public List<CompactionInfo> findReadyToClean()
                                      throws MetaException

Find entries in the queue that are ready to be cleaned.

Returns:

information on the entry in the queue.

Throws:

MetaException

markCleaned

public void markCleaned(CompactionInfo info)
                 throws MetaException

This will remove an entry from the queue after it has been compacted.

Parameters:

info - info on the compaction entry to remove

Throws:

MetaException

cleanEmptyAbortedTxns

public void cleanEmptyAbortedTxns()
                           throws MetaException

Clean up aborted transactions from txns that have no components in txn_components.

Throws:

MetaException

revokeFromLocalWorkers

public void revokeFromLocalWorkers(String hostname)
                            throws MetaException

This will take all entries assigned to workers on a host return them to INITIATED state. The initiator should use this at start up to clean entries from any workers that were in the middle of compacting when the metastore shutdown. It does not reset entries from worker threads on other hosts as those may still be working.

Parameters:

hostname - Name of this host. It is assumed this prefixes the thread's worker id, so that like hostname% will match the worker id.

Throws:

MetaException

revokeTimedoutWorkers

public void revokeTimedoutWorkers(long timeout)
                           throws MetaException

This call will return all compaction queue entries assigned to a worker but over the timeout back to the initiated state. This should be called by the initiator on start up and occasionally when running to clean up after dead threads. At start up revokeFromLocalWorkers(String) should be called first.

Parameters:

timeout - number of milliseconds since start time that should elapse before a worker is declared dead.

Throws:

MetaException

findColumnsWithStats

public List<String> findColumnsWithStats(CompactionInfo ci)
                                  throws MetaException

Queries metastore DB directly to find columns in the table which have statistics information. If ci includes partition info then per partition stats info is examined, otherwise table level stats are examined.

Throws:

MetaException

createValidCompactTxnList

public static org.apache.hadoop.hive.common.ValidTxnList createValidCompactTxnList(GetOpenTxnsInfoResponse txns)

Transform a GetOpenTxnsInfoResponse to a ValidTxnList. This assumes that the caller intends to compact the files, and thus treats only open transactions as invalid.

Parameters:

txns - txn list from the metastore

Returns:

a valid txn list.