# Design

## Non-nested types

Let's start with the design used for non-nested arrays. The (private) entry point of this module for
non-nested arrays is `simple::page_iter_to_arrays`.

This function expects

- a (fallible) streaming iterator of decompressed and encoded pages, `Pages`
- the source (parquet) column type, including its logical information
- the target (arrow) `DataType`
- the chunk size

and returns an iterator of `Array`, `ArrayIter`.

This design is shared among _all_ `(parquet, arrow)` implemented tuples. Their main difference is
how they are deserialized, which depends on the source and target types.

When the array iterator is pulled the first time, the following happens:

- a page from `Pages` is pulled
- a `PageState<'a>` is built from the page
- the `PageState` is consumed into a mutable array:
  - if `chunk_size` is larger than the number of rows in the page, the mutable array state is
    preserved and a new page is pulled and the process repeated until we fill a chunk.
  - if `chunk_size` is smaller than the number of rows in the page, the mutable array state is
    returned and the remaining of the page is consumed into multiple mutable arrays of length
    `chunk_size` into a FIFO queue.

Subsequent pulls of arrays will first try to pull from the FIFO queue. Once the queue is empty, the
a new page is pulled.

### `PageState`

As mentioned above, the iterator leverages the idea that we attach a state to a page. Recall that a
page is essentially `[header][data]`. The `data` part contains encoded
`[rep levels][def levels][non-null values]`. Some pages have an associated dictionary page, in which
case the `non-null values` represent the indices.

Irrespectively of the physical type, the main idea is to split the page in two iterators:

- An iterator over `def levels`
- An iterator over `non-null values`

and progress the iterators as needed. In particular, for non-nested types, `def levels` is a bitmap
with the same representation as Arrow, in which case the validity is extended directly.

The `non-null values` are "expanded" by filling null values with the default value of each physical
type.

## Nested types

For nested type with N+1 levels (1 is the primitive), we need to build the nest information of each
N levels + the non-nested Arrow array.

This is done by first transversing the parquet types and using it to initialize, per chunk, the N
levels.

The per-chunk execution is then similar but `chunk_size` only drives the number of retrieved rows
from the outermost parquet group (the field). Each of these pulls knows how many items need to be
pulled from the inner groups, all the way to the primitive type. This works because in parquet a row
cannot be split between two pages and thus each page is guaranteed to contain a full row.

The `PageState` of nested types is composed by 4 iterators:

- A (zipped) iterator over `rep levels` and `def levels`
- An iterator over `def levels`
- An iterator over `non-null values`

The idea is that an iterator of `rep, def` contain all the information to decode the nesting
structure of an arrow array. The other two iterators are equivalent to the non-nested types with the
exception that `def levels` are no equivalent to arrow bitmaps.
