Commit Graph

5 Commits

Author SHA1 Message Date
Thomas Peiselt
ea84099b07
Lazy load delta: Support for large tables (#71)
* Allow for delta tables to be directly backed by storage.

Enables experimental support for delta tables that are too large to be
stored in memory. We directly expose `DeltaTable` instead of copying the
data into a datafusion::Memtable.

Disadvantages:
- in the new mode, no support for S3
- as we're relying on datafusion to handle the parquet files directly,
  nested schemas and certain data types may not work properly.
2021-09-06 00:56:55 -07:00
Erwin Kroon
ff2d06b0e4
add support for all Arrow IPC formats in roapi-http (#67)
* add support for all arrow IPC formats in roapi-http

* refactor: schema inferrence and partitions in 1 loop
2021-09-05 14:27:09 -07:00
Thomas Peiselt
5ace8b8695
Lazy load parquet (#63)
* PoC for parquet: reading a table by registering parquet directly.

* Adding config flag and restoring existing _in-memory_ code path.

* Addressing review comment: separate `to_mem_table()`.

* Addressing review comment: default-able `LoadOptionParquet`.

* Adding test: make sure we instantiated `datafusion::datasource::ParquetTable`
2021-09-03 19:36:18 +00:00
Qingping Hou
95f9ced23d support deriving table name from table uri 2021-07-22 23:38:28 -07:00
Qingping Hou
e0f213c316 split project into roapi-http and columnq crates 2021-02-17 13:46:20 -08:00