parquet.hadoop

Class ParquetWriter<T>

java.lang.Object

parquet.hadoop.ParquetWriter<T>

All Implemented Interfaces:

Closeable, AutoCloseable

Direct Known Subclasses:

ExampleParquetWriter

public class ParquetWriter<T>
extends Object
implements Closeable

Write records to a Parquet file.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ParquetWriter.Builder<T,SELF extends ParquetWriter.Builder<T,SELF>> An abstract builder class for ParquetWriter instances.

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_BLOCK_SIZE
static CompressionCodecName	DEFAULT_COMPRESSION_CODEC_NAME
static boolean	DEFAULT_IS_DICTIONARY_ENABLED
static boolean	DEFAULT_IS_VALIDATING_ENABLED
static int	DEFAULT_PAGE_SIZE
static ParquetProperties.WriterVersion	DEFAULT_WRITER_VERSION
static int	MAX_PADDING_SIZE_DEFAULT

Constructor Summary

Constructors

Constructor and Description
ParquetWriter(org.apache.hadoop.fs.Path file, org.apache.hadoop.conf.Configuration conf, WriteSupport<T> writeSupport) Deprecated.
ParquetWriter(org.apache.hadoop.fs.Path file, ParquetFileWriter.Mode mode, WriteSupport<T> writeSupport, CompressionCodecName compressionCodecName, int blockSize, int pageSize, int dictionaryPageSize, boolean enableDictionary, boolean validating, ParquetProperties.WriterVersion writerVersion, org.apache.hadoop.conf.Configuration conf) Deprecated.
ParquetWriter(org.apache.hadoop.fs.Path file, WriteSupport<T> writeSupport) Deprecated.
ParquetWriter(org.apache.hadoop.fs.Path file, WriteSupport<T> writeSupport, CompressionCodecName compressionCodecName, int blockSize, int pageSize) Deprecated.
ParquetWriter(org.apache.hadoop.fs.Path file, WriteSupport<T> writeSupport, CompressionCodecName compressionCodecName, int blockSize, int pageSize, boolean enableDictionary, boolean validating) Deprecated.
ParquetWriter(org.apache.hadoop.fs.Path file, WriteSupport<T> writeSupport, CompressionCodecName compressionCodecName, int blockSize, int pageSize, int dictionaryPageSize, boolean enableDictionary, boolean validating) Deprecated.
ParquetWriter(org.apache.hadoop.fs.Path file, WriteSupport<T> writeSupport, CompressionCodecName compressionCodecName, int blockSize, int pageSize, int dictionaryPageSize, boolean enableDictionary, boolean validating, org.apache.hadoop.conf.Configuration conf) Create a new ParquetWriter.
ParquetWriter(org.apache.hadoop.fs.Path file, WriteSupport<T> writeSupport, CompressionCodecName compressionCodecName, int blockSize, int pageSize, int dictionaryPageSize, boolean enableDictionary, boolean validating, ParquetProperties.WriterVersion writerVersion) Deprecated.
ParquetWriter(org.apache.hadoop.fs.Path file, WriteSupport<T> writeSupport, CompressionCodecName compressionCodecName, int blockSize, int pageSize, int dictionaryPageSize, boolean enableDictionary, boolean validating, ParquetProperties.WriterVersion writerVersion, org.apache.hadoop.conf.Configuration conf) Deprecated.

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
long	getDataSize()
void	write(T object)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_BLOCK_SIZE

public static final int DEFAULT_BLOCK_SIZE

See Also:

Constant Field Values

DEFAULT_PAGE_SIZE

public static final int DEFAULT_PAGE_SIZE

See Also:

Constant Field Values

DEFAULT_COMPRESSION_CODEC_NAME

public static final CompressionCodecName DEFAULT_COMPRESSION_CODEC_NAME

DEFAULT_IS_DICTIONARY_ENABLED

public static final boolean DEFAULT_IS_DICTIONARY_ENABLED

See Also:

Constant Field Values

DEFAULT_IS_VALIDATING_ENABLED

public static final boolean DEFAULT_IS_VALIDATING_ENABLED

See Also:

Constant Field Values

DEFAULT_WRITER_VERSION

public static final ParquetProperties.WriterVersion DEFAULT_WRITER_VERSION

MAX_PADDING_SIZE_DEFAULT

public static final int MAX_PADDING_SIZE_DEFAULT

See Also:

Constant Field Values

Constructor Detail

ParquetWriter

@Deprecated
public ParquetWriter(org.apache.hadoop.fs.Path file,
                        WriteSupport<T> writeSupport,
                        CompressionCodecName compressionCodecName,
                        int blockSize,
                        int pageSize)
              throws IOException

Deprecated.

Create a new ParquetWriter. (with dictionary encoding enabled and validation off)

Parameters:

file - the file to create

writeSupport - the implementation to write a record to a RecordConsumer

compressionCodecName - the compression codec to use

blockSize - the block size threshold

pageSize - the page size threshold

Throws:

IOException

See Also:

ParquetWriter(Path, WriteSupport, CompressionCodecName, int, int, boolean, boolean)

ParquetWriter

@Deprecated
public ParquetWriter(org.apache.hadoop.fs.Path file,
                        WriteSupport<T> writeSupport,
                        CompressionCodecName compressionCodecName,
                        int blockSize,
                        int pageSize,
                        boolean enableDictionary,
                        boolean validating)
              throws IOException

Deprecated.

Create a new ParquetWriter.

Parameters:

file - the file to create

writeSupport - the implementation to write a record to a RecordConsumer

compressionCodecName - the compression codec to use

blockSize - the block size threshold

pageSize - the page size threshold (both data and dictionary)

enableDictionary - to turn dictionary encoding on

validating - to turn on validation using the schema

Throws:

IOException

See Also:

ParquetWriter(Path, WriteSupport, CompressionCodecName, int, int, int, boolean, boolean)

ParquetWriter

@Deprecated
public ParquetWriter(org.apache.hadoop.fs.Path file,
                        WriteSupport<T> writeSupport,
                        CompressionCodecName compressionCodecName,
                        int blockSize,
                        int pageSize,
                        int dictionaryPageSize,
                        boolean enableDictionary,
                        boolean validating)
              throws IOException

Deprecated.

Create a new ParquetWriter.

Parameters:

file - the file to create

writeSupport - the implementation to write a record to a RecordConsumer

compressionCodecName - the compression codec to use

blockSize - the block size threshold

pageSize - the page size threshold

dictionaryPageSize - the page size threshold for the dictionary pages

enableDictionary - to turn dictionary encoding on

validating - to turn on validation using the schema

Throws:

IOException

See Also:

#ParquetWriter(Path, WriteSupport, CompressionCodecName, int, int, int, boolean, boolean, WriterVersion)

ParquetWriter

@Deprecated
public ParquetWriter(org.apache.hadoop.fs.Path file,
                        WriteSupport<T> writeSupport,
                        CompressionCodecName compressionCodecName,
                        int blockSize,
                        int pageSize,
                        int dictionaryPageSize,
                        boolean enableDictionary,
                        boolean validating,
                        ParquetProperties.WriterVersion writerVersion)
              throws IOException

Deprecated.

Create a new ParquetWriter. Directly instantiates a Hadoop Configuration which reads configuration from the classpath.

Parameters:

file - the file to create

writeSupport - the implementation to write a record to a RecordConsumer

compressionCodecName - the compression codec to use

blockSize - the block size threshold

pageSize - the page size threshold

dictionaryPageSize - the page size threshold for the dictionary pages

enableDictionary - to turn dictionary encoding on

validating - to turn on validation using the schema

writerVersion - version of parquetWriter from ParquetProperties.WriterVersion

Throws:

IOException

See Also:

#ParquetWriter(Path, WriteSupport, CompressionCodecName, int, int, int, boolean, boolean, WriterVersion, Configuration)

ParquetWriter

public ParquetWriter(org.apache.hadoop.fs.Path file,
             WriteSupport<T> writeSupport,
             CompressionCodecName compressionCodecName,
             int blockSize,
             int pageSize,
             int dictionaryPageSize,
             boolean enableDictionary,
             boolean validating,
             org.apache.hadoop.conf.Configuration conf)
              throws IOException

Create a new ParquetWriter.

Parameters:

file - the file to create

writeSupport - the implementation to write a record to a RecordConsumer

compressionCodecName - the compression codec to use

blockSize - the block size threshold

pageSize - the page size threshold

dictionaryPageSize - the page size threshold for the dictionary pages

enableDictionary - to turn dictionary encoding on

validating - to turn on validation using the schema

conf - Hadoop configuration to use while accessing the filesystem

Throws:

IOException

ParquetWriter

@Deprecated
public ParquetWriter(org.apache.hadoop.fs.Path file,
                        WriteSupport<T> writeSupport,
                        CompressionCodecName compressionCodecName,
                        int blockSize,
                        int pageSize,
                        int dictionaryPageSize,
                        boolean enableDictionary,
                        boolean validating,
                        ParquetProperties.WriterVersion writerVersion,
                        org.apache.hadoop.conf.Configuration conf)
              throws IOException

Deprecated.

Create a new ParquetWriter.

Parameters:

file - the file to create

writeSupport - the implementation to write a record to a RecordConsumer

compressionCodecName - the compression codec to use

blockSize - the block size threshold

pageSize - the page size threshold

dictionaryPageSize - the page size threshold for the dictionary pages

enableDictionary - to turn dictionary encoding on

validating - to turn on validation using the schema

writerVersion - version of parquetWriter from ParquetProperties.WriterVersion

conf - Hadoop configuration to use while accessing the filesystem

Throws:

IOException

ParquetWriter

@Deprecated
public ParquetWriter(org.apache.hadoop.fs.Path file,
                        ParquetFileWriter.Mode mode,
                        WriteSupport<T> writeSupport,
                        CompressionCodecName compressionCodecName,
                        int blockSize,
                        int pageSize,
                        int dictionaryPageSize,
                        boolean enableDictionary,
                        boolean validating,
                        ParquetProperties.WriterVersion writerVersion,
                        org.apache.hadoop.conf.Configuration conf)
              throws IOException

Deprecated.

Create a new ParquetWriter.

Parameters:

file - the file to create

mode - file creation mode

writeSupport - the implementation to write a record to a RecordConsumer

compressionCodecName - the compression codec to use

blockSize - the block size threshold

pageSize - the page size threshold

dictionaryPageSize - the page size threshold for the dictionary pages

enableDictionary - to turn dictionary encoding on

validating - to turn on validation using the schema

writerVersion - version of parquetWriter from ParquetProperties.WriterVersion

conf - Hadoop configuration to use while accessing the filesystem

Throws:

IOException

ParquetWriter

@Deprecated
public ParquetWriter(org.apache.hadoop.fs.Path file,
                        WriteSupport<T> writeSupport)
              throws IOException

Deprecated.

Create a new ParquetWriter. The default block size is 50 MB.The default page size is 1 MB. Default compression is no compression. Dictionary encoding is disabled.

Parameters:

file - the file to create

writeSupport - the implementation to write a record to a RecordConsumer

Throws:

IOException

ParquetWriter

@Deprecated
public ParquetWriter(org.apache.hadoop.fs.Path file,
                        org.apache.hadoop.conf.Configuration conf,
                        WriteSupport<T> writeSupport)
              throws IOException

Deprecated.

Throws:

IOException

Method Detail

write

public void write(T object)
           throws IOException

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

getDataSize

public long getDataSize()

Returns:

the total size of data written to the file and buffered in memory