Commit Graph

1 Commits

Author SHA1 Message Date
Thomas Peiselt
ea84099b07
Lazy load delta: Support for large tables (#71)
* Allow for delta tables to be directly backed by storage.

Enables experimental support for delta tables that are too large to be
stored in memory. We directly expose `DeltaTable` instead of copying the
data into a datafusion::Memtable.

Disadvantages:
- in the new mode, no support for S3
- as we're relying on datafusion to handle the parquet files directly,
  nested schemas and certain data types may not work properly.
2021-09-06 00:56:55 -07:00