Commit Graph

278 Commits

Author SHA1 Message Date
QP Hou
c44becdff7
fix disable-read-only mode parsing (#369) 2025-01-04 00:46:36 -08:00
QP Hou
529ba782d6
fix reload-interval cli config parsing (#368)
fixes https://github.com/roapi/roapi/issues/367
2025-01-04 00:03:32 -08:00
QP Hou
1570c7d4bb
update minor dependency versions & fix release (#366)
Some checks failed
build / build (push) Has been cancelled
build / database_test (push) Has been cancelled
build / object_store_memory_test (push) Has been cancelled
build / object_store_direct_test (push) Has been cancelled
build / openssl_build (push) Has been cancelled
build / mac_cross_build (push) Has been cancelled
build / Docker Image Build (push) Has been cancelled
columnq-cli release / Validate git tag (push) Has been cancelled
roapi release / Validate git tag (push) Has been cancelled
columnq-cli release / macos (push) Has been cancelled
columnq-cli release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
columnq-cli release / PyPI Release (push) Has been cancelled
roapi release / macos (push) Has been cancelled
roapi release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
roapi release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
roapi release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
roapi release / PyPI Release (push) Has been cancelled
roapi release / Docker Image Release (push) Has been cancelled
2025-01-01 01:54:02 -08:00
QP Hou
a9317f73eb
cleanup & update dependencies tree (#365)
update datafusion to 43, deltalake to 0.22, axum to 0.7 and arrow to 53

---------

Co-authored-by: hozan23 <hozan23@karyontech.net>
2024-12-31 19:06:49 -08:00
QP Hou
6e6b315ca3
add header override support for http backed sources (#364)
closes #358
2024-12-31 01:22:23 -08:00
QP Hou
03771341fb
ci: remove deprecated action args (#363)
Some checks failed
build / build (push) Has been cancelled
build / database_test (push) Has been cancelled
build / object_store_memory_test (push) Has been cancelled
build / object_store_direct_test (push) Has been cancelled
build / openssl_build (push) Has been cancelled
build / mac_cross_build (push) Has been cancelled
build / Docker Image Build (push) Has been cancelled
columnq-cli release / Validate git tag (push) Has been cancelled
roapi release / Validate git tag (push) Has been cancelled
columnq-cli release / macos (push) Has been cancelled
columnq-cli release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
columnq-cli release / PyPI Release (push) Has been cancelled
roapi release / macos (push) Has been cancelled
roapi release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
roapi release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
roapi release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
roapi release / PyPI Release (push) Has been cancelled
roapi release / Docker Image Release (push) Has been cancelled
2024-12-28 23:13:09 -08:00
QP Hou
3600e4b660
ci: use actions-rust-lang/setup-rust-toolchain to manage cargo cache (#362) 2024-12-28 22:55:29 -08:00
QP Hou
49da15e4c0
refactor: make table refresher a required field (#360)
other drive by changes:
* add nix flake for quick dev env setup
2024-12-28 22:44:34 -08:00
QP Hou
10bdbfa314
ci: fix cargo cache management (#361) 2024-12-28 22:08:56 -08:00
Sami Airo
62ad38fd03
columnq - issue with the argument types (#359)
Thank you for the project!

I was using the columnq v0.5.2 and getting the following error:

```
thread 'main' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/clap_builder-4.5.2/src/parser/error.rs:32:9:
Mismatch between definition and access of `table`. Could not downcast to TypeId { t: 337271291176647618489964333718931667905 }, need to downcast to TypeId { t: 150318335506304927533756938025983212585 }
```

Looks like the argument types needed to be updated to String. The roapi
source was already using String - like the following:

```
    if let Some(tables) = matches.get_many::<String>("table") {
        for v in tables {
            config.tables.push(whatever!(
                parse_table_uri_arg(v),
                "Failed to parse table uri: {v}"
            ));
        }
    }
```
2024-12-28 21:51:37 -08:00
QP Hou
c1e26a84d1
enable continuous background refresh for delta tables (#352)
Some checks failed
build / build (push) Has been cancelled
build / database_test (push) Has been cancelled
build / object_store_memory_test (push) Has been cancelled
build / object_store_direct_test (push) Has been cancelled
build / openssl_build (push) Has been cancelled
build / mac_cross_build (push) Has been cancelled
build / Docker Image Build (push) Has been cancelled
columnq-cli release / Validate git tag (push) Has been cancelled
roapi release / Validate git tag (push) Has been cancelled
columnq-cli release / macos (push) Has been cancelled
columnq-cli release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
columnq-cli release / PyPI Release (push) Has been cancelled
roapi release / macos (push) Has been cancelled
roapi release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
roapi release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
roapi release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
roapi release / PyPI Release (push) Has been cancelled
roapi release / Docker Image Release (push) Has been cancelled
2024-12-02 06:49:16 -08:00
Rui Chen
85c55c9951
fix: add missing explicit lifetime in ExcelSubrange return type for rust 1.83 build (#353)
seeing some regression build failure when building roapi 0.12.0 against
rust 1.83.0

```
  error: elided lifetime has a name
    --> columnq/src/table/excel.rs:48:10
     |
  41 | impl<'a> ExcelSubrange<'a> {
     |      -- lifetime `'a` declared here
  ...
  48 |     ) -> ExcelSubrange {
     |          ^^^^^^^^^^^^^ this elided lifetime gets resolved as `'a`
     |
  note: the lint level is defined here
    --> columnq/src/lib.rs:1:9
     |
  1  | #![deny(warnings)]
     |         ^^^^^^^^
     = note: `#[deny(elided_named_lifetimes)]` implied by `#[deny(warnings)]`
```

relates to https://github.com/Homebrew/homebrew-core/pull/199379

Signed-off-by: Rui Chen <rui@chenrui.dev>
2024-12-02 08:51:29 +00:00
QP Hou
322bd203bd
ci(chore): update to action/checkout@v4 (#350)
Some checks failed
build / build (push) Has been cancelled
build / database_test (push) Has been cancelled
build / object_store_memory_test (push) Has been cancelled
build / object_store_direct_test (push) Has been cancelled
build / openssl_build (push) Has been cancelled
build / mac_cross_build (push) Has been cancelled
build / Docker Image Build (push) Has been cancelled
columnq-cli release / Validate git tag (push) Has been cancelled
roapi release / Validate git tag (push) Has been cancelled
columnq-cli release / macos (push) Has been cancelled
columnq-cli release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
columnq-cli release / PyPI Release (push) Has been cancelled
roapi release / macos (push) Has been cancelled
roapi release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
roapi release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
roapi release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
roapi release / PyPI Release (push) Has been cancelled
roapi release / Docker Image Release (push) Has been cancelled
2024-10-23 21:50:29 -07:00
Qingping Hou
b16072b1dc ci(fix): use unique name for pypi linux release artifacts 2024-10-23 21:26:26 -07:00
Qingping Hou
be4fb9d359 ci(fix): use unique name for upload-artifact jobs
Some checks failed
build / build (push) Has been cancelled
build / database_test (push) Has been cancelled
build / object_store_memory_test (push) Has been cancelled
build / object_store_direct_test (push) Has been cancelled
build / openssl_build (push) Has been cancelled
build / mac_cross_build (push) Has been cancelled
build / Docker Image Build (push) Has been cancelled
columnq-cli release / Validate git tag (push) Has been cancelled
roapi release / Validate git tag (push) Has been cancelled
columnq-cli release / macos (push) Has been cancelled
columnq-cli release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
columnq-cli release / PyPI Release (push) Has been cancelled
roapi release / macos (push) Has been cancelled
roapi release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
roapi release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
roapi release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
roapi release / PyPI Release (push) Has been cancelled
roapi release / Docker Image Release (push) Has been cancelled
2024-10-20 19:02:00 -07:00
Qingping Hou
f4dacd9fa7 ci(fix): update upload-artifacts action to v4 2024-10-20 17:52:58 -07:00
hozan23
daabd84418
columnq: add apply_query() and exec_query_with_df() to query::rest module (#349)
Following this PR #348 

This PR exposes two functions from the columnq::query::rest module:

    apply_query()
    exec_query_with_df()

This PR also includes breaking changes to ensure consistency with the
function names in the `columnq::query::graphql` module. I renamed the
function `query_table` to `exec_table_query`.
2024-10-20 17:14:01 -07:00
hozan23
e282c6c1d1
columnq: add apply_query() and exec_query_with_df() to query::graphql module (#348)
This PR exposes two functions from the `columnq::query::graphql` module:

- `apply_query()`
- `exec_query_with_df()`

These functions are similar to `query_to_df()` and `exec_query()`, but
they accept a `DataFrame` instead of a `SessionContext`.

This is useful for defining rules directly within a `DataFrame` without
modifying the table schema in DataFusion's session context.
2024-10-20 17:09:06 -07:00
Brendan Ryback
b776dd34ce
Support for Dynamic Table Deletion #340 (#343)
Some checks failed
build / build (push) Has been cancelled
build / database_test (push) Has been cancelled
build / object_store_memory_test (push) Has been cancelled
build / object_store_direct_test (push) Has been cancelled
build / openssl_build (push) Has been cancelled
build / mac_cross_build (push) Has been cancelled
build / Docker Image Build (push) Has been cancelled
columnq-cli release / Validate git tag (push) Has been cancelled
roapi release / Validate git tag (push) Has been cancelled
columnq-cli release / macos (push) Has been cancelled
columnq-cli release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
columnq-cli release / PyPI Release (push) Has been cancelled
roapi release / macos (push) Has been cancelled
roapi release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
roapi release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
roapi release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
roapi release / PyPI Release (push) Has been cancelled
roapi release / Docker Image Release (push) Has been cancelled
closes #340
2024-08-25 21:02:56 +00:00
QP Hou
08ec089a84
bump object_store and, arrow (52) and datafusion (39) (#341)
Some checks failed
build / build (push) Has been cancelled
build / database_test (push) Has been cancelled
build / object_store_memory_test (push) Has been cancelled
build / object_store_direct_test (push) Has been cancelled
build / openssl_build (push) Has been cancelled
build / mac_cross_build (push) Has been cancelled
build / Docker Image Build (push) Has been cancelled
columnq-cli release / Validate git tag (push) Has been cancelled
roapi release / Validate git tag (push) Has been cancelled
columnq-cli release / macos (push) Has been cancelled
columnq-cli release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
columnq-cli release / PyPI Release (push) Has been cancelled
roapi release / macos (push) Has been cancelled
roapi release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
roapi release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
roapi release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
roapi release / PyPI Release (push) Has been cancelled
roapi release / Docker Image Release (push) Has been cancelled
close https://github.com/roapi/roapi/issues/337
2024-08-04 20:18:35 -07:00
QP Hou
7f0d4bf941
build with stable rust toolchain (#335) 2024-05-27 00:12:55 -07:00
QP Hou
3bfcb29bcd
bump datafusion to 37 and arrow to 51 (#334)
closes #324 #325
2024-05-26 23:32:56 -07:00
QP Hou
9eba0281c6
support http for delta tables (#333) 2024-05-26 00:33:01 -07:00
QP Hou
a46351c4d4
bump snmalloc to 0.3 (#332) 2024-05-19 22:10:57 -07:00
QP Hou
1938fe6c2b
fix remote storage support for deltalake (#331) 2024-05-19 20:56:46 -07:00
QP Hou
8168d93859
add api to get schema by table name (#330) 2024-05-18 20:29:55 -07:00
QP Hou
5a0b68de6d
fix: exclude folder in object_store s3 list content (#329) 2024-05-05 17:26:06 -07:00
Qingping Hou
76400b1d60 bump columnq-cli version 2024-04-28 23:27:27 -07:00
Qingping Hou
40682d733a bump datafusion for partition filter fix 2024-04-28 23:09:32 -07:00
Qingping Hou
98b46f2ad5 fix build cache export 2024-04-28 16:30:42 -07:00
Qingping Hou
5ff5ef76d0 bump columnq-cli version 2024-04-28 15:49:01 -07:00
Qingping Hou
9fbbd7fd35 fix roapi release job change rule 2024-04-28 12:39:39 -07:00
QP Hou
44b7b57b2d
fix macos python wheel release job (#328) 2024-04-28 12:17:43 -07:00
QP Hou
4c4999db68
update rust toolchain github action (#327) 2024-04-28 01:03:05 -07:00
Qingping Hou
a553521d61 fix python release for real 2024-04-28 00:16:44 -07:00
Qingping Hou
be9b93c9d8 fix pypi release 2024-04-28 00:10:24 -07:00
QP Hou
ffc794c43b
support partition filter pushdown (#326) 2024-04-28 00:03:34 -07:00
QP Hou
fd022b57bb fix docker release 2024-03-24 20:29:25 -07:00
QP Hou
9ec4eb4523 columnq-cli: fix table argument parsing 2024-03-24 16:01:33 -07:00
QP Hou
199f9d8e0f
bump datafusion to 35 and arrow to 50 (#320)
also bump rustc version
2024-03-24 15:58:36 -07:00
QP Hou
c2a3268d44
add token & basic auth support for flightsql endpoint (#318)
required to enable SSL in grafana's flightsql plugin
2024-03-10 22:48:01 -07:00
Maksym Dovhal
81a25205b7
Update xlsx table (#316)
This PR is a fix for issue https://github.com/roapi/roapi/issues/259

List of updates/fixes:
* module xlsx renamed to excel.
* Allow reading not only xlsx format but also xls, ods, xlsb
* Allow Excel DateTime format and transform it to arrow
Timestamp(Seconds, None)
* Allow using NULLs in any data types and use null value instead of
string "null"
* Fix issue with incorrect data type inference when multiple data types
are detected.
* Add possibility to specify data schema in config.
* Add new options: -
rows_range_start
 - rows_range_end
 - columns_range_start
 - columns_range_end
 - schema_inference_lines
* Make sheet_name optional and if it is not specified than use first
sheet by default

* Bump calamine crate to version 0.23.1 and add feature "dates"
(supporting for DateTime column format)

Documentation updates: https://github.com/roapi/docs/pull/20
2024-01-31 20:16:34 -08:00
QP Hou
9377b7ccad
support infer schema by list of files (#310) 2023-11-19 20:14:26 -08:00
QP Hou
b2454f48e5
build docker image with cargo chef cache (#309) 2023-11-19 11:31:39 -08:00
QP Hou
a322d1b443
support defining nested enum data types in schema config (#308) 2023-11-18 19:01:53 -08:00
QP Hou
b70c3bcb0b
enable aws feature in object_store crate by default 2023-10-31 01:03:22 -07:00
QP Hou
d531f8ae46
support partitioned tables (#307) 2023-10-20 16:00:07 -07:00
QP Hou
ea01ffe0d3
upgrade clap to 4 (#304) 2023-10-14 21:09:21 -07:00
Qingping Hou
51584dd3a0 update readme 2023-10-11 00:19:45 -07:00
Qingping Hou
8df946f68a update readme 2023-10-11 00:16:39 -07:00