parquet.hadoop

Class ParquetRecordWriter<T>

java.lang.Object

org.apache.hadoop.mapreduce.RecordWriter<Void,T>

parquet.hadoop.ParquetRecordWriter<T>

Type Parameters:

T - the type of the materialized records

public class ParquetRecordWriter<T>
extends org.apache.hadoop.mapreduce.RecordWriter<Void,T>

Writes records to a Parquet file

Author:

Julien Le Dem

See Also:

ParquetOutputFormat

Constructor Summary

Constructors

Constructor and Description
ParquetRecordWriter(ParquetFileWriter w, WriteSupport<T> writeSupport, MessageType schema, Map<String,String> extraMetaData, int blockSize, int pageSize, CodecFactory.BytesCompressor compressor, int dictionaryPageSize, boolean enableDictionary, boolean validating, ParquetProperties.WriterVersion writerVersion) Deprecated.
ParquetRecordWriter(ParquetFileWriter w, WriteSupport<T> writeSupport, MessageType schema, Map<String,String> extraMetaData, long blockSize, int pageSize, CodecFactory.BytesCompressor compressor, int dictionaryPageSize, boolean enableDictionary, boolean validating, ParquetProperties.WriterVersion writerVersion, MemoryManager memoryManager)

Method Summary

Methods

Modifier and Type	Method and Description
void	close(org.apache.hadoop.mapreduce.TaskAttemptContext context)
void	write(Void key, T value)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ParquetRecordWriter

@Deprecated
public ParquetRecordWriter(ParquetFileWriter w,
                              WriteSupport<T> writeSupport,
                              MessageType schema,
                              Map<String,String> extraMetaData,
                              int blockSize,
                              int pageSize,
                              CodecFactory.BytesCompressor compressor,
                              int dictionaryPageSize,
                              boolean enableDictionary,
                              boolean validating,
                              ParquetProperties.WriterVersion writerVersion)

Deprecated.

Parameters:

w - the file to write to

writeSupport - the class to convert incoming records

schema - the schema of the records

extraMetaData - extra meta data to write in the footer of the file

blockSize - the size of a block in the file (this will be approximate)

compressor - the compressor used to compress the pages

dictionaryPageSize - the threshold for dictionary size

enableDictionary - to enable the dictionary

validating - if schema validation should be turned on

ParquetRecordWriter

public ParquetRecordWriter(ParquetFileWriter w,
                   WriteSupport<T> writeSupport,
                   MessageType schema,
                   Map<String,String> extraMetaData,
                   long blockSize,
                   int pageSize,
                   CodecFactory.BytesCompressor compressor,
                   int dictionaryPageSize,
                   boolean enableDictionary,
                   boolean validating,
                   ParquetProperties.WriterVersion writerVersion,
                   MemoryManager memoryManager)

Parameters:

w - the file to write to

writeSupport - the class to convert incoming records

schema - the schema of the records

extraMetaData - extra meta data to write in the footer of the file

blockSize - the size of a block in the file (this will be approximate)

compressor - the compressor used to compress the pages

dictionaryPageSize - the threshold for dictionary size

enableDictionary - to enable the dictionary

validating - if schema validation should be turned on

Method Detail

close

public void close(org.apache.hadoop.mapreduce.TaskAttemptContext context)
           throws IOException,
                  InterruptedException

Specified by:

close in class org.apache.hadoop.mapreduce.RecordWriter<Void,T>

Throws:

IOException

InterruptedException

write

public void write(Void key,
         T value)
           throws IOException,
                  InterruptedException

Specified by:

write in class org.apache.hadoop.mapreduce.RecordWriter<Void,T>

Throws:

IOException

InterruptedException