CodeCow/roapi
1
0
Fork 0
mirror of https://github.com/roapi/roapi.git synced 2026-06-11 21:00:56 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
314 Commits 3 Branches 47 Tags 2.1 MiB
main
Commit Graph

82 Commits

Author SHA1 Message Date
QP Hou
c5efa9716d
upgrade dependencies (#425) 2026-03-24 00:51:44 -07:00
QP Hou
083a31d6cc
bump deltalake, datafusion and arrow to latest (#396) 2025-05-04 23:32:59 -07:00
Mike Shauneu
6fa208fce4
Add HTTP object_store to be able to connect without caching in memory… (#393) 
Before: 
```
roapi -t taxi=https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-01.parquet,use_memory_table=false,format=parquet
[2025-04-23T22:43:25Z INFO  roapi::context] loading `uri(https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-01.parquet)` as table `taxi`
error: Failed to build context: Failed to load table: Table error: Failed to infer table schema: Internal error: No suitable object store found for https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-01.parquet. See `RuntimeEnv::register_object_store`.
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
```

After:
```
roapi -t 'taxi=https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-01.parquet,use_memory_table=false,format=parquet'`
[2025-04-23T22:47:00Z INFO  roapi::context] loading `uri(https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-01.parquet)` as table `taxi`
[2025-04-23T22:47:00Z INFO  roapi::context] registered `uri(https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-01.parquet)` as table `taxi`
[2025-04-23T22:47:00Z INFO  roapi::startup] Running in read-only mode.
[2025-04-23T22:47:00Z INFO  roapi::startup] 🚀 Listening on 127.0.0.1:5432 for Postgres traffic...
[2025-04-23T22:47:00Z INFO  roapi::startup] 🚀 Listening on 127.0.0.1:32010 for FlightSQL traffic...
[2025-04-23T22:47:00Z INFO  roapi::startup] 🚀 Listening on 127.0.0.1:8080 for HTTP traffic...
```
 2025-04-23 22:19:50 -07:00
QP Hou
9ebc252011
remove lazy_static dep (#376)
Some checks failed
build / build (push) Has been cancelled
Details
build / database_test (push) Has been cancelled
Details
build / object_store_memory_test (push) Has been cancelled
Details
build / object_store_direct_test (push) Has been cancelled
Details
build / openssl_build (push) Has been cancelled
Details
build / mac_cross_build (push) Has been cancelled
Details
build / Docker Image Build (push) Has been cancelled
Details
columnq-cli release / Validate git tag (push) Has been cancelled
Details
roapi release / Validate git tag (push) Has been cancelled
Details
columnq-cli release / macos (push) Has been cancelled
Details
columnq-cli release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
Details
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
Details
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
Details
columnq-cli release / PyPI Release (push) Has been cancelled
Details
roapi release / macos (push) Has been cancelled
Details
roapi release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
Details
roapi release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
Details
roapi release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
Details
roapi release / PyPI Release (push) Has been cancelled
Details
roapi release / Docker Image Release (push) Has been cancelled
Details
2025-01-21 22:46:17 -08:00
QP Hou
1570c7d4bb
update minor dependency versions & fix release (#366)
Some checks failed
build / build (push) Has been cancelled
Details
build / database_test (push) Has been cancelled
Details
build / object_store_memory_test (push) Has been cancelled
Details
build / object_store_direct_test (push) Has been cancelled
Details
build / openssl_build (push) Has been cancelled
Details
build / mac_cross_build (push) Has been cancelled
Details
build / Docker Image Build (push) Has been cancelled
Details
columnq-cli release / Validate git tag (push) Has been cancelled
Details
roapi release / Validate git tag (push) Has been cancelled
Details
columnq-cli release / macos (push) Has been cancelled
Details
columnq-cli release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
Details
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
Details
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
Details
columnq-cli release / PyPI Release (push) Has been cancelled
Details
roapi release / macos (push) Has been cancelled
Details
roapi release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
Details
roapi release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
Details
roapi release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
Details
roapi release / PyPI Release (push) Has been cancelled
Details
roapi release / Docker Image Release (push) Has been cancelled
Details
2025-01-01 01:54:02 -08:00
QP Hou
a9317f73eb
cleanup & update dependencies tree (#365) 
update datafusion to 43, deltalake to 0.22, axum to 0.7 and arrow to 53

---------

Co-authored-by: hozan23 <hozan23@karyontech.net>
 2024-12-31 19:06:49 -08:00
QP Hou
c1e26a84d1
enable continuous background refresh for delta tables (#352)
Some checks failed
build / build (push) Has been cancelled
Details
build / database_test (push) Has been cancelled
Details
build / object_store_memory_test (push) Has been cancelled
Details
build / object_store_direct_test (push) Has been cancelled
Details
build / openssl_build (push) Has been cancelled
Details
build / mac_cross_build (push) Has been cancelled
Details
build / Docker Image Build (push) Has been cancelled
Details
columnq-cli release / Validate git tag (push) Has been cancelled
Details
roapi release / Validate git tag (push) Has been cancelled
Details
columnq-cli release / macos (push) Has been cancelled
Details
columnq-cli release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
Details
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
Details
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
Details
columnq-cli release / PyPI Release (push) Has been cancelled
Details
roapi release / macos (push) Has been cancelled
Details
roapi release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
Details
roapi release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
Details
roapi release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
Details
roapi release / PyPI Release (push) Has been cancelled
Details
roapi release / Docker Image Release (push) Has been cancelled
Details
2024-12-02 06:49:16 -08:00
QP Hou
08ec089a84
bump object_store and, arrow (52) and datafusion (39) (#341)
Some checks failed
build / build (push) Has been cancelled
Details
build / database_test (push) Has been cancelled
Details
build / object_store_memory_test (push) Has been cancelled
Details
build / object_store_direct_test (push) Has been cancelled
Details
build / openssl_build (push) Has been cancelled
Details
build / mac_cross_build (push) Has been cancelled
Details
build / Docker Image Build (push) Has been cancelled
Details
columnq-cli release / Validate git tag (push) Has been cancelled
Details
roapi release / Validate git tag (push) Has been cancelled
Details
columnq-cli release / macos (push) Has been cancelled
Details
columnq-cli release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
Details
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
Details
columnq-cli release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
Details
columnq-cli release / PyPI Release (push) Has been cancelled
Details
roapi release / macos (push) Has been cancelled
Details
roapi release / windows (map[features:database-sqlite python-architecture:x64 target:x86_64-pc-windows-msvc]) (push) Has been cancelled
Details
roapi release / linux (map[features:rustls,database-sqlite image_tag:aarch64-musl manylinux:2014 name_suffix: rustflags: target:aarch64-unknown-linux-musl upload:true]) (push) Has been cancelled
Details
roapi release / linux (map[features:rustls,database-sqlite image_tag:x86_64-musl manylinux:2010 name_suffix: rustflags:-C target-cpu=skylake target:x86_64-unknown-linux-musl upload:true]) (push) Has been cancelled
Details
roapi release / PyPI Release (push) Has been cancelled
Details
roapi release / Docker Image Release (push) Has been cancelled
Details 
close https://github.com/roapi/roapi/issues/337
 2024-08-04 20:18:35 -07:00
QP Hou
3bfcb29bcd
bump datafusion to 37 and arrow to 51 (#334) 
closes #324 #325
 2024-05-26 23:32:56 -07:00
QP Hou
1938fe6c2b
fix remote storage support for deltalake (#331) 2024-05-19 20:56:46 -07:00
QP Hou
5a0b68de6d
fix: exclude folder in object_store s3 list content (#329) 2024-05-05 17:26:06 -07:00
QP Hou
ffc794c43b
support partition filter pushdown (#326) 2024-04-28 00:03:34 -07:00
QP Hou
199f9d8e0f
bump datafusion to 35 and arrow to 50 (#320) 
also bump rustc version
 2024-03-24 15:58:36 -07:00
Maksym Dovhal
81a25205b7
Update xlsx table (#316) 
This PR is a fix for issue https://github.com/roapi/roapi/issues/259

List of updates/fixes:
* module xlsx renamed to excel.
* Allow reading not only xlsx format but also xls, ods, xlsb
* Allow Excel DateTime format and transform it to arrow
Timestamp(Seconds, None)
* Allow using NULLs in any data types and use null value instead of
string "null"
* Fix issue with incorrect data type inference when multiple data types
are detected.
* Add possibility to specify data schema in config.
* Add new options: -
rows_range_start
 - rows_range_end
 - columns_range_start
 - columns_range_end
 - schema_inference_lines
* Make sheet_name optional and if it is not specified than use first
sheet by default

* Bump calamine crate to version 0.23.1 and add feature "dates"
(supporting for DateTime column format)

Documentation updates: https://github.com/roapi/docs/pull/20
 2024-01-31 20:16:34 -08:00
QP Hou
a322d1b443
support defining nested enum data types in schema config (#308) 2023-11-18 19:01:53 -08:00
QP Hou
b70c3bcb0b
enable aws feature in object_store crate by default 2023-10-31 01:03:22 -07:00
QP Hou
ea01ffe0d3
upgrade clap to 4 (#304) 2023-10-14 21:09:21 -07:00
QP Hou
b10afb0116
enable resolve 2 in cargo.toml (#300) 2023-10-03 01:53:06 -07:00
QP Hou
eef379f9bb
switch to snafu for error handling in columnq (#299) 2023-10-03 00:27:04 -07:00
QP Hou
2baf669841
update datafusion to 31 (#297) 2023-09-30 19:24:47 -07:00
Joe
ff3617767d
feat: support postgresql (#286) 2023-08-29 09:06:11 -07:00
Charles Swanberg
01c5422206
Cloud support deltalake (#280) 
* fix: enable cloud support for deltalake

* fix clippy issues

* formatting fixes
 2023-06-20 00:43:35 -07:00
QP Hou
df39a13d3f
upgrade datafusion and arrow (#279) 
* update arrow, datafusion, delta, convergence, object_store, sqlparser

* fix breaking code

* update to datafusion 23

* update connectorx fork

---------

Co-authored-by: Chitral Verma <chitralverma@gmail.com>
 2023-05-29 23:25:45 -07:00
Jiayu Liu
ae13d3bc9e
support toml as config lang (#277) 
* support toml as config lang

* update readme
 2023-05-22 22:45:39 -07:00
Robert Gieseke
411d727263 Update connector-x to latest commit of original repo 
Closes #271
 2023-04-14 08:27:30 -07:00
Rich
51e01ef968
Upgrade object_store to 0.5.4 (#262) 
object_stoer 0.5.4 introduce more authentication support for GCP and Azure, including Workload Identity
 2023-02-28 00:03:36 -08:00
Rich
f9b17888cb
fix url decoding of space (#257) 
* Add two test case of space

* decode percent in url

* remove query of space
 2023-01-30 23:06:01 -08:00
QP Hou
c8e7b596ec
ci: add sqlite tests (#253) 2023-01-28 23:55:54 -08:00
QP Hou
90e79dc7f0
fix(postgres): ignore non-select queries (#252) 2023-01-28 21:59:34 -05:00
Rich
2ec39ddba8
support Native Azure Storage (#250) 
* support native Azure Storage

* query azure_blogs in CI

* check Azure Storage support in readme

* remove debug print

* Handle url.host_str error
 2023-01-23 18:25:56 -08:00
Rich
fffc8558a8
Native GCS support (#248) 
* Set object_store minimal version and turn on aws_profile feature

* Add native GCS support with GOOGLE_APPLICATION_CREDENTIALS

* hacky: create /tmp/gcs.json for test

* use GoogleCloudStorageBuilder::from_env

* create temp file inside test for GCS

Co-authored-by: QP Hou <dave2008713@gmail.com>
 2023-01-22 07:42:47 +00:00
Rich
e4f00489d1
#240 Set object_store minimal version and turn on aws_profile feature (#247) 
* Set object_store minimal version and turn on aws_profile feature

* remove unnecessary with_endpoint after upgrade to object_store 0.5.3
 2023-01-21 23:20:11 -08:00
Rich
bd168d92d8
replace rusoto_s3: part 2 (last part) (#245) 
* replace rusoto_s3 in columnq::table::delta

* remove rusoto crate
 2023-01-18 22:15:12 -08:00
Rich
1e3644ec36
#240 support s3 direct with ColumnQObjectStoreProvider (#241) 
* Add test case for s3 direct query

* draft: add ColumnQObjectStoreProvider and global Datafusion SessionContext

* fix a few tests of load params change

* fix database tests of load params change

* do not overwrite endpoint if AWS_ENDPOINT_URL is not provided

* ObjectStoreProvider trait change after upgrade from Datafusion 12 to 14

* add comment for difference of test_s3_memory and test_s3_direct

* add unit test for ColumnQObjectStoreProvider

* fix s3 unit test with aws region in CI
 2023-01-16 22:03:11 -08:00
QP Hou
3d4da9a810
update datafusion to 14 and arrow to 26 (#242) 2023-01-15 22:47:13 -08:00
Rich
eed91fdbbe
#235: upgrade rusoto and force http1 for rustls (#236) 
* upgrade rusoto and force http1 for rustls

* add dependency for  openssl_build
 2023-01-01 15:33:25 -08:00
QP Hou
1467739e25
replace dotenv with dotenvy (#216) 2022-11-21 22:15:58 -08:00
Y Togami
35549c0ddf
feat: support .xlsx (#218) 2022-10-28 18:46:09 -07:00
Y Togami
e9d0a03b89
dep: replace tempdir to tempfile (#202) 2022-10-15 17:53:42 -07:00
Qingping Hou
4e812b9995 split database feature flag into database-mysql and database-sqlite 2022-09-25 18:58:05 -07:00
Qingping Hou
37e5d5b03a bump datafusion to 12 and sqlparser to 0.23 2022-09-25 18:58:05 -07:00
Qingping Hou
200abe6185 support bitint in json serialization 2022-05-30 17:32:51 -07:00
Qingping Hou
ebc43c6fd9 display binding addresses on start up 2022-05-30 16:45:05 -07:00
QP Hou
6dc3b77a62
bump datafusion and sqlparser dep (#168) 2022-05-28 19:41:42 -07:00
zemel leong
3ace6078fa
Add MySQL, Sqlite support. (#162) 
* added MySQL and Sqlite datasource support
* updated arrow, datafusion and deltalake to latest version
* cleared simd ci test cache to workaround nightly compiler bug
 2022-04-07 00:55:10 -07:00
Daniël Heres
5d41a28417
Update axum, sqlparser (#160) 2022-04-02 10:43:50 -07:00
QP Hou
a2676c288f
update dependency versions for roapi-http 0.6.0 (#159) 2022-03-19 12:59:10 -07:00
Tiphaine Ruy
646c467916
chore: bump version for axum, dirs, sqlparser, yup-auth2, (#145) 
* chore: bump version for axum, dirs, sqlparser, yup-auth2,

* chore: Bump datafusion to 7 and Delta-rs to f5e4b5f94393b517d3d88c200d7169cce11b304b

* fix(test): query are now case insensitive in datafusion

* fix(test): Remove println! statements
 2022-02-24 22:57:03 -08:00
QP Hou
cd271c65ac
support empty cells in google spreadsheet (#141) 2022-02-14 22:23:17 -08:00
QP Hou
528f89efc9
fix json table partition loading (#133) 2022-01-27 20:14:11 -08:00