feat: add corex-storage crate for unified storage abstraction
- Introduced corex-storage crate with support for local disk, S3-compatible, and Google Cloud Storage backends. - Implemented StorageDriver trait for various storage backends. - Added LocalFileStorage for local disk operations. - Added S3Storage for S3-compatible object storage with multipart upload support. - Added GcsStorage for Google Cloud Storage operations. - Included error handling for storage operations. - Added tests for each storage backend to ensure functionality. - Created README.md for documentation and usage examples. - Added Apache and MIT licenses for open-source compliance.
This commit is contained in:
1 parent
6a4b2b8f54
commit
db4f277336
47 files changed
No files matched your search
@@ -0,0 +1,51 @@
|
||||
//! # corex-storage
|
||||
//!
|
||||
//! A unified storage & file system abstraction so file handling doesn't get
|
||||
//! locked to one physical storage location.
|
||||
//!
|
||||
//! - **Local disk** ([`local::LocalFileStorage`], feature `local`, default).
|
||||
//! - **S3-compatible object storage** ([`s3::S3Storage`], feature `s3`) —
|
||||
//! works with Amazon S3 and any S3-compatible endpoint, including MinIO
|
||||
//! (pass a custom endpoint via [`s3::S3Storage::connect_with_endpoint`]).
|
||||
//! - **Google Cloud Storage** ([`gcs::GcsStorage`], feature `gcs`).
|
||||
//!
|
||||
//! All operations are stream-based (an [`traits::ObjectStream`] is a boxed
|
||||
//! [`tokio::io::AsyncRead`]), so uploading/downloading a huge object never
|
||||
//! requires holding it entirely in memory.
|
||||
//!
|
||||
//! ## Example
|
||||
//!
|
||||
//! ```
|
||||
//! use corex_storage::{LocalFileStorage, StorageDriver, bytes_stream, read_to_vec};
|
||||
//! # #[tokio::main] async fn main() {
|
||||
//! # let dir = tempfile::tempdir().unwrap();
|
||||
//! let storage = LocalFileStorage::new(dir.path());
|
||||
//! storage.put("hello.txt", bytes_stream(b"hi".to_vec())).await.unwrap();
|
||||
//! let data = read_to_vec(storage.get("hello.txt").await.unwrap()).await.unwrap();
|
||||
//! assert_eq!(data, b"hi");
|
||||
//! # }
|
||||
//! ```
|
||||
|
||||
pub mod traits;
|
||||
|
||||
#[cfg(feature = "local")]
|
||||
pub mod local;
|
||||
|
||||
#[cfg(feature = "s3")]
|
||||
pub mod s3;
|
||||
|
||||
#[cfg(feature = "gcs")]
|
||||
pub mod gcs;
|
||||
|
||||
pub use traits::{
|
||||
bytes_stream, read_to_vec, ObjectMeta, ObjectStream, StorageDriver, StorageError,
|
||||
};
|
||||
|
||||
#[cfg(feature = "local")]
|
||||
pub use local::LocalFileStorage;
|
||||
|
||||
#[cfg(feature = "s3")]
|
||||
pub use s3::S3Storage;
|
||||
|
||||
#[cfg(feature = "gcs")]
|
||||
pub use gcs::GcsStorage;
|
||||
Reference in new issue
Block a user