Thomas Peiselt ea84099b07

Lazy load delta: Support for large tables (#71) ...

* Allow for delta tables to be directly backed by storage.

Enables experimental support for delta tables that are too large to be
stored in memory. We directly expose `DeltaTable` instead of copying the
data into a datafusion::Memtable.

Disadvantages:
- in the new mode, no support for S3
- as we're relying on datafusion to handle the parquet files directly,
  nested schemas and certain data types may not work properly.

2021-09-06 00:56:55 -07:00

48 B

Raw Permalink History

View Raw