parquet.hadoop

Class ParquetFileReader

java.lang.Object

parquet.hadoop.ParquetFileReader

All Implemented Interfaces:

Closeable, AutoCloseable

public class ParquetFileReader
extends Object
implements Closeable

Internal implementation of the Parquet file reader as a block container

Author:

Julien Le Dem

Field Summary

Fields

Modifier and Type	Field and Description
static String	PARQUET_READ_PARALLELISM

Constructor Summary

Constructors

Constructor and Description
ParquetFileReader(org.apache.hadoop.conf.Configuration configuration, FileMetaData fileMetaData, org.apache.hadoop.fs.Path filePath, List<BlockMetaData> blocks, List<ColumnDescriptor> columns)
ParquetFileReader(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath, List<BlockMetaData> blocks, List<ColumnDescriptor> columns) Deprecated. use @link{ParquetFileReader(Configuration configuration, FileMetaData fileMetaData, Path filePath, List blocks, List columns)} instead

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
static List<Footer>	readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus fileStatus) Read the footers of all the files under that path (recursively) not using summary files.
static List<Footer>	readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration, List<org.apache.hadoop.fs.FileStatus> partFiles) Deprecated.
static List<Footer>	readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration, List<org.apache.hadoop.fs.FileStatus> partFiles, boolean skipRowGroups) read all the footers of the files provided (not using summary files)
static List<Footer>	readAllFootersInParallelUsingSummaryFiles(org.apache.hadoop.conf.Configuration configuration, Collection<org.apache.hadoop.fs.FileStatus> partFiles, boolean skipRowGroups) for files provided, check if there's a summary file.
static List<Footer>	readAllFootersInParallelUsingSummaryFiles(org.apache.hadoop.conf.Configuration configuration, List<org.apache.hadoop.fs.FileStatus> partFiles) Deprecated.
static ParquetMetadata	readFooter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus file) Deprecated. use ParquetFileReader#readFooter(Configuration, FileStatus, MetadataFilter)
static ParquetMetadata	readFooter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus file, ParquetMetadataConverter.MetadataFilter filter) Reads the meta data block in the footer of the file
static ParquetMetadata	readFooter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path file) Deprecated.
static ParquetMetadata	readFooter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path file, ParquetMetadataConverter.MetadataFilter filter) Reads the meta data in the footer of the file.
static List<Footer>	readFooters(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus pathStatus) Deprecated.
static List<Footer>	readFooters(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus pathStatus, boolean skipRowGroups) Read the footers of all the files under that path (recursively) using summary files if possible
static List<Footer>	readFooters(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path path) Deprecated.
PageReadStore	readNextRowGroup() Reads all the columns requested from the row group at the current file position.
static List<Footer>	readSummaryFile(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus summaryStatus) Specifically reads a given summary file

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PARQUET_READ_PARALLELISM

public static String PARQUET_READ_PARALLELISM

Constructor Detail

ParquetFileReader

public ParquetFileReader(org.apache.hadoop.conf.Configuration configuration,
                 org.apache.hadoop.fs.Path filePath,
                 List<BlockMetaData> blocks,
                 List<ColumnDescriptor> columns)
                  throws IOException

Deprecated. use @link{ParquetFileReader(Configuration configuration, FileMetaData fileMetaData, Path filePath, List blocks, List columns)} instead

Throws:

IOException

ParquetFileReader

public ParquetFileReader(org.apache.hadoop.conf.Configuration configuration,
                 FileMetaData fileMetaData,
                 org.apache.hadoop.fs.Path filePath,
                 List<BlockMetaData> blocks,
                 List<ColumnDescriptor> columns)
                  throws IOException

Parameters:

configuration - the Hadoop conf

fileMetaData - fileMetaData for parquet file

blocks - the blocks to read

columns - the columns to read (their path)

Throws:

IOException - if the file can not be opened

Method Detail

readAllFootersInParallelUsingSummaryFiles

@Deprecated
public static List<Footer> readAllFootersInParallelUsingSummaryFiles(org.apache.hadoop.conf.Configuration configuration,
                                                                List<org.apache.hadoop.fs.FileStatus> partFiles)
                                                              throws IOException

Deprecated.

for files provided, check if there's a summary file. If a summary file is found it is used otherwise the file footer is used.

Parameters:

configuration - the hadoop conf to connect to the file system;

partFiles - the part files to read

Returns:

the footers for those files using the summary file if possible.

Throws:

IOException

readAllFootersInParallelUsingSummaryFiles

public static List<Footer> readAllFootersInParallelUsingSummaryFiles(org.apache.hadoop.conf.Configuration configuration,
                                                     Collection<org.apache.hadoop.fs.FileStatus> partFiles,
                                                     boolean skipRowGroups)
                                                              throws IOException

for files provided, check if there's a summary file. If a summary file is found it is used otherwise the file footer is used.

Parameters:

configuration - the hadoop conf to connect to the file system;

partFiles - the part files to read

skipRowGroups - to skipRowGroups in the footers

Returns:

the footers for those files using the summary file if possible.

Throws:

IOException

readAllFootersInParallel

@Deprecated
public static List<Footer> readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration,
                                               List<org.apache.hadoop.fs.FileStatus> partFiles)
                                             throws IOException

Deprecated.

Throws:

IOException

readAllFootersInParallel

public static List<Footer> readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration,
                                    List<org.apache.hadoop.fs.FileStatus> partFiles,
                                    boolean skipRowGroups)
                                             throws IOException

read all the footers of the files provided (not using summary files)

Parameters:

configuration - the conf to access the File System

partFiles - the files to read

skipRowGroups - to skip the rowGroup info

Returns:

the footers

Throws:

IOException

readAllFootersInParallel

public static List<Footer> readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration,
                                    org.apache.hadoop.fs.FileStatus fileStatus)
                                             throws IOException

Read the footers of all the files under that path (recursively) not using summary files. rowGroups are not skipped

Parameters:

configuration - the configuration to access the FS

fileStatus - the root dir

Returns:

all the footers

Throws:

IOException

readFooters

@Deprecated
public static List<Footer> readFooters(org.apache.hadoop.conf.Configuration configuration,
                                  org.apache.hadoop.fs.Path path)
                                throws IOException

Deprecated.

Throws:

IOException

readFooters

@Deprecated
public static List<Footer> readFooters(org.apache.hadoop.conf.Configuration configuration,
                                  org.apache.hadoop.fs.FileStatus pathStatus)
                                throws IOException

Deprecated.

this always returns the row groups

Parameters:

configuration -

pathStatus -

Returns:

Throws:

IOException

readFooters

public static List<Footer> readFooters(org.apache.hadoop.conf.Configuration configuration,
                       org.apache.hadoop.fs.FileStatus pathStatus,
                       boolean skipRowGroups)
                                throws IOException

Read the footers of all the files under that path (recursively) using summary files if possible

Parameters:

configuration - the configuration to access the FS

fileStatus - the root dir

Returns:

all the footers

Throws:

IOException

readSummaryFile

public static List<Footer> readSummaryFile(org.apache.hadoop.conf.Configuration configuration,
                           org.apache.hadoop.fs.FileStatus summaryStatus)
                                    throws IOException

Specifically reads a given summary file

Parameters:

configuration -

summaryStatus -

Returns:

the metadata translated for each file

Throws:

IOException

readFooter

@Deprecated
public static final ParquetMetadata readFooter(org.apache.hadoop.conf.Configuration configuration,
                                    org.apache.hadoop.fs.Path file)
                                        throws IOException

Deprecated.

Reads the meta data block in the footer of the file

Parameters:

configuration -

file - the parquet File

Returns:

the metadata blocks in the footer

Throws:

IOException - if an error occurs while reading the file

readFooter

public static ParquetMetadata readFooter(org.apache.hadoop.conf.Configuration configuration,
                         org.apache.hadoop.fs.Path file,
                         ParquetMetadataConverter.MetadataFilter filter)
                                  throws IOException

Reads the meta data in the footer of the file. Skipping row groups (or not) based on the provided filter

Parameters:

configuration -

file - the Parquet File

filter - the filter to apply to row groups

Returns:

the metadata with row groups filtered.

Throws:

IOException - if an error occurs while reading the file

readFooter

@Deprecated
public static final ParquetMetadata readFooter(org.apache.hadoop.conf.Configuration configuration,
                                    org.apache.hadoop.fs.FileStatus file)
                                        throws IOException

Deprecated. use ParquetFileReader#readFooter(Configuration, FileStatus, MetadataFilter)

Throws:

IOException

readFooter

public static final ParquetMetadata readFooter(org.apache.hadoop.conf.Configuration configuration,
                         org.apache.hadoop.fs.FileStatus file,
                         ParquetMetadataConverter.MetadataFilter filter)
                                        throws IOException

Reads the meta data block in the footer of the file

Parameters:

configuration -

file - the parquet File

filter - the filter to apply to row groups

Returns:

the metadata blocks in the footer

Throws:

IOException - if an error occurs while reading the file

readNextRowGroup

public PageReadStore readNextRowGroup()
                               throws IOException

Reads all the columns requested from the row group at the current file position.

Returns:

the PageReadStore which can provide PageReaders for each column.

Throws:

IOException - if an error occurs while reading

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException