parquet.hadoop

Class MemoryManager

java.lang.Object

parquet.hadoop.MemoryManager

public class MemoryManager
extends Object

Implements a memory manager that keeps a global context of how many Parquet writers there are and manages the memory between them. For use cases with dynamic partitions, it is easy to end up with many writers in the same task. By managing the size of each allocation, we try to cut down the size of each allocation and keep the task from running out of memory. This class balances the allocation size of each writer by resize them averagely. When the sum of each writer's allocation size is less than total memory pool, keep them original value. When the sum exceeds, decrease each writer's allocation size by a ratio.

Constructor Summary

Constructors

Constructor and Description
MemoryManager(float ratio, long minAllocation)

Method Summary

Methods

Modifier and Type	Method and Description
void	registerScaleCallBack(String callBackName, Runnable callBack) Register callback and deduplicate it if any.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MemoryManager

public MemoryManager(float ratio,
             long minAllocation)

Method Detail

registerScaleCallBack

public void registerScaleCallBack(String callBackName,
                         Runnable callBack)

Register callback and deduplicate it if any.

Parameters:

callBackName - the name of callback. It should be identical.

callBack - the callback passed in from upper layer, such as Hive.