Thomas Peiselt
ea84099b07
Lazy load delta: Support for large tables ( #71 )
...
* Allow for delta tables to be directly backed by storage.
Enables experimental support for delta tables that are too large to be
stored in memory. We directly expose `DeltaTable` instead of copying the
data into a datafusion::Memtable.
Disadvantages:
- in the new mode, no support for S3
- as we're relying on datafusion to handle the parquet files directly,
nested schemas and certain data types may not work properly.
2021-09-06 00:56:55 -07:00
Erwin Kroon
ff2d06b0e4
add support for all Arrow IPC formats in roapi-http ( #67 )
...
* add support for all arrow IPC formats in roapi-http
* refactor: schema inferrence and partitions in 1 loop
2021-09-05 14:27:09 -07:00
Thomas Peiselt
5ace8b8695
Lazy load parquet ( #63 )
...
* PoC for parquet: reading a table by registering parquet directly.
* Adding config flag and restoring existing _in-memory_ code path.
* Addressing review comment: separate `to_mem_table()`.
* Addressing review comment: default-able `LoadOptionParquet`.
* Adding test: make sure we instantiated `datafusion::datasource::ParquetTable`
2021-09-03 19:36:18 +00:00
Qingping Hou
0e8690d509
use reexported arrow from datafusion in columnq
2021-08-14 16:33:27 -07:00
Qingping Hou
0a2a965072
update arrow to 5.x and datafusion to latest
2021-08-08 17:42:12 -07:00
Qingping Hou
98110356d1
fix test
2021-07-22 23:38:28 -07:00
Qingping Hou
95f9ced23d
support deriving table name from table uri
2021-07-22 23:38:28 -07:00
Qingping Hou
d7fab6369c
bump deltalake version to 0.4
2021-07-18 17:59:32 -07:00
Qingping Hou
d5359b4e56
lots of goodies
...
* add in memory io source support
* add sql command to columnq cli
* support passing table format as columnq cli option
2021-07-18 00:06:59 -07:00
Qingping Hou
4c78d1e728
update cargo install command to use lock file
2021-06-13 17:40:42 -07:00
Qingping Hou
3b415b2a5b
bump version to 0.1.3
2021-04-11 23:29:47 -07:00
Qingping Hou
9afd69b95b
fix clippy error
2021-03-28 12:22:11 -07:00
Qingping Hou
edefae4310
feat: add delta table support
2021-03-11 23:10:06 -08:00
Qingping Hou
8fd2f96ab8
add ndjson support
2021-03-07 13:41:18 -08:00
Qingping Hou
f33d496c76
add s3 support
2021-03-07 12:11:37 -08:00
QP Hou
3631c2db14
update arrow and datafusion to latest master ( #13 )
2021-02-28 19:50:27 +00:00
QP Hou
717de08e67
fix clippy warnings ( #7 )
2021-02-21 09:24:49 +00:00
QP Hou
2c39fb13e7
support loading multiple partitions ( #6 )
2021-02-21 09:09:39 +00:00
QP Hou
2f4646bbb4
feat: support http store for parquet data source ( #5 )
2021-02-20 22:42:22 +00:00
QP Hou
f6dfc1faaf
feat: support https transport for csv data source ( #4 )
...
close #1
2021-02-20 09:47:11 +00:00
Qingping Hou
e0f213c316
split project into roapi-http and columnq crates
2021-02-17 13:46:20 -08:00
