Compare commits

...

935 Commits

Author SHA1 Message Date
安正超
ce1f7cfdcb chore(skills): add repository-local workflow skills (#2190) 2026-03-17 22:13:46 +08:00
马登山
c66c6d97ec fix(lifecycle): respect Filter.Prefix and safe delete marker expiry (#2185)
Signed-off-by: likewu <likewu@126.com>
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: likewu <likewu@126.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-17 18:45:38 +08:00
dependabot[bot]
be89b5fc6a build(deps): bump lz4_flex from 0.12.0 to 0.12.1 (#2181)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-17 10:10:41 +08:00
houseme
94cdb89e29 feat(obs): add init_obs_with_config API and signature guard test (#2175)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
2026-03-16 18:17:55 +08:00
heihutu
06dff96c09 chore(deps): update flake.lock (#2173) 2026-03-16 16:01:36 +08:00
安正超
c1d5106acc feat(ci): allow selecting build platforms in build workflow (#2171) 2026-03-15 22:01:44 +08:00
heihutu
0a2411f59c chore(deps): update flake.lock (#2169)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-15 16:10:12 +08:00
houseme
1ede71b881 chore: update nix-flake-update.yml to use FLAKE_UPDATE_TOKEN for user… (#2168) 2026-03-15 14:49:38 +08:00
github-actions[bot]
4fb7059e6f chore(deps): update flake.lock (#2165) 2026-03-15 10:26:05 +08:00
安正超
2ad275ecc3 fix(helm): quote obs stdout configmap value (#2166)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 马登山 <Cxymds@qq.com>
2026-03-15 10:11:25 +08:00
Philip Schmid
9179fd5608 fix(helm): merge customAnnotations with class-specific ingress annotations (#2161)
Signed-off-by: Philip Schmid <philip.schmid@protonmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-15 09:22:12 +08:00
LeonWang0735
7f1cdaedad feat(replication): add bandwidth-aware reporting for bucket replication metrics (#2141) 2026-03-15 09:03:10 +08:00
houseme
7f3459f5a8 fix(obs): fixed unresolved import super::local::ensure_dir_permissions (#2164) 2026-03-15 00:33:06 +08:00
yxrxy
d3cff7d033 feat(webdav): add WebDAV protocol gateway (#2158)
Signed-off-by: yxrxy <1532529704@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 马登山 <Cxymds@qq.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-03-14 23:06:53 +08:00
majinghe
f66a90c1b2 fix: fix github action error caused by oltp modification (#2163)
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-14 22:15:35 +08:00
majinghe
afcaaf66fc feat: add support for obs enpoint support in helm chart (#2160) 2026-03-14 21:44:44 +08:00
安正超
a1104b45f6 fix(obs): honor target-only rust_log directives (#2159)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-03-14 11:14:46 +08:00
bcdax110
82d9452736 docs: fix incorrect UID in Docker Quick Start of README_ZH (#2149)
Signed-off-by: bcdax110 <1711382287@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-14 09:26:07 +08:00
houseme
6e0f034ad1 refactor(obs): enhance log rotation robustness and refine filter logic (#2155)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-03-14 09:20:35 +08:00
houseme
593a58c161 refactor(obs): optimize logging with custom RollingAppender and improved cleanup (#2151)
Signed-off-by: houseme <housemecn@gmail.com>
Signed-off-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-03-13 13:20:27 +08:00
houseme
f83bf95b04 feat(ecstore): Skip rustls provider install if already present (#2145)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
2026-03-12 18:02:19 +08:00
安正超
aa88b1976a fix(ecstore): avoid warm tier init panics (#2144) 2026-03-12 13:52:49 +08:00
安正超
e2f741d41f fix(helm): use canonical scanner start delay env (#2142) 2026-03-12 10:06:42 +08:00
安正超
ad54293d7e fix(admin): propagate heal handler background errors (#2124) 2026-03-12 10:06:12 +08:00
安正超
83fb530609 refactor(config): normalize scanner env naming (#2129) 2026-03-11 22:41:41 +08:00
安正超
aa84d34bf8 fix(auth): preserve IAMAuth clone and correct missing-key error (#2123) 2026-03-11 21:59:12 +08:00
安正超
df57f0c033 fix(workers): clamp worker release count (#2122) 2026-03-11 21:59:00 +08:00
安正超
c47dec8549 fix(signer): avoid panics in v2 signing for missing data (#2121) 2026-03-11 21:58:40 +08:00
安正超
fdbe12ec95 fix(scanner): respect configured scan start delay (#2119) 2026-03-11 21:56:48 +08:00
安正超
b2e8078971 fix(policy): avoid unicode panic in variable resolver (#2115) 2026-03-11 21:56:32 +08:00
安正超
ac43a44a00 [codex] fix scanner first cycle startup delay (#2137) 2026-03-11 20:02:01 +08:00
安正超
5625f04697 fix(common): remove panic paths in runtime helpers (#2116)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
2026-03-11 18:12:37 +08:00
安正超
e1f24f764d fix(credentials): harden masked debug output (#2114)
Signed-off-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
2026-03-11 15:40:37 +08:00
安正超
7d7e0b2654 fix(utils): harden panic-prone paths (#2113)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
2026-03-11 15:16:03 +08:00
安正超
9908a44c38 fix(protocols): return errors instead of panics for sync signatures (#2120)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
2026-03-11 11:22:20 +08:00
evan slack
4b480727d6 feat(perf): Add configurable bitrot skip for reads (#2110)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-03-11 10:59:00 +08:00
simon-escapecode
f00d01ec2d fix: resolve silent failure in MQTT bucket event notifications (#2112)
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-11 10:08:30 +08:00
dependabot[bot]
7e8c7fa2b2 build(deps): bump quinn-proto from 0.11.13 to 0.11.14 (#2127)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-11 09:30:53 +08:00
安正超
845ad1fa16 fix(obs): avoid panic in telemetry init and clamp sampler boundaries (#2118)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-11 01:32:46 +08:00
安正超
bb4fbf5ae2 fix(notify): ignore disabled targets when sending events (#2117)
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-11 00:37:30 +08:00
安正超
3df7105dae fix(server): init event notifier when partial notify configured (#2125) 2026-03-10 23:52:40 +08:00
evan slack
b3da8ae269 feat(scanner): Add dynamic throttling presets (#2095)
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: GatewayJ <835269233@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: weisd <im@weisd.in>
2026-03-10 16:12:56 +08:00
majinghe
67e5f5e3c3 feat: add metrics support in helm chart (#2109)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-10 12:08:36 +08:00
majinghe
296efea42f change ghcr username and password name due to github restrict (#2108) 2026-03-09 21:44:35 +08:00
GatewayJ
16946c5a54 fix: allow root to bypass bucket policy deny for policy management APIs (#2102)
Co-authored-by: GatewayJ <8352692332qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-03-09 20:36:29 +08:00
majinghe
73d29e95dd feat:add docker image support for quay.io and ghcr.io (#2107)
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-09 16:22:28 +08:00
dependabot[bot]
e930c5c281 build(deps): bump libc from 0.2.182 to 0.2.183 in the dependencies group (#2106)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-09 16:03:17 +08:00
安正超
9d03029959 fix(iam): sync user cache on load-user notifications (#2104) 2026-03-09 09:36:02 +08:00
loverustfs
a02c354ef5 Fix image url error
Fix image url error

Signed-off-by: loverustfs <hello@rustfs.com>
2026-03-08 23:39:50 +08:00
houseme
60aa47bf61 feat(storage): integrate S3Operation into OperationHelper for unified metrics and audit (#2103) 2026-03-08 17:57:33 +08:00
houseme
8e4a1ef917 refactor(protocols): replace tar with astral-tokio-tar for async processing (#2099)
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
2026-03-08 15:18:15 +08:00
Peter Hamilton
b035d10abb fix(metrics): Remove high cardinality labels causing memory leak (#2098)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-03-08 13:01:11 +08:00
github-actions[bot]
2180e9e7a1 chore(deps): update flake.lock (#2097)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-08 10:51:32 +08:00
evan slack
57e49e6737 feat(obs): Add metric to count all s3 operations (#2088)
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-08 10:19:20 +08:00
Senol Colak
b07383760f Add OpenStack Swift API Support (#2066)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <noreply@github.com>
2026-03-08 01:11:35 +08:00
evan slack
7c94be4e8c fix(obs): Remove high cardinality label on rustfs_api_requests_total (#2087)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-03-07 20:46:33 +08:00
evan slack
d52a10c5fb chore(obs): Improve tracing instrumentation (#2086)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-03-07 20:03:20 +08:00
安正超
8c4735ff88 docs: scope AGENTS instructions by directory (#2083) 2026-03-05 17:25:37 +08:00
LeonWang0735
a0503168d4 fix(heal):heal failed replication via must_replicate instead of check replicate_delete (#2072) 2026-03-05 15:47:36 +08:00
安正超
b73059dcf2 fix(admin): allow non-consoleAdmin self password update (#2082) 2026-03-05 15:47:21 +08:00
weisd
ed18b3da75 Fix data usage cache and scanner (#2074) 2026-03-04 19:55:01 +08:00
houseme
05032cf887 chore: update dependencies and workspace resolver (#2073) 2026-03-04 19:22:54 +08:00
唐小鸭
f89cdfe5b3 update s3s 0.14.0-dev (#2070)
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-04 01:07:24 +08:00
houseme
f4b523c236 build(deps): bump the dependencies group with 7 updates (#2069) 2026-03-04 00:42:03 +08:00
安正超
c6209ba59d ci: optimize workflow runtime and remove redundant pipeline work (#2065) 2026-03-03 20:56:37 +08:00
houseme
5e7495a042 build(obs): restrict pyroscope dependency to unix targets (#2064) 2026-03-03 20:41:37 +08:00
evan slack
ac4b13def1 feat(obs): Optional continuous CPU profiling with grafana pyroscope (#2035)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
2026-03-03 15:28:58 +08:00
安正超
08e1f4670b fix: restore default CORS fallback and STS object ACL ownership (#2053)
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-03 01:08:50 +08:00
唐小鸭
fff96a0921 fix sse-options (#2056) 2026-03-03 01:08:37 +08:00
唐小鸭
f17725a2ea fix(sse): allow PUT/GET without KMS when no SSE or bucket default (#2054)
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-03 00:44:23 +08:00
houseme
bf957e3523 remove rustflags target cpu (#2052) 2026-03-02 23:59:34 +08:00
Rafael Herrero
a6090b98dc fix(iam): remove incorrect trailing slash from OIDC issuer URL (#2050) 2026-03-02 19:48:12 +08:00
houseme
2ac07c95a8 refactor(obs): enhance log cleanup and rotation (#2040) 2026-03-02 16:28:32 +08:00
安正超
e157a88f09 fix: support query-only presigned URL access (#2046) 2026-03-02 15:46:50 +08:00
安正超
01a75b5f58 Add env variable alias compatibility warnings (#2044) 2026-03-02 15:34:19 +08:00
GatewayJ
2cb8db36a5 fix(iam): user group policy and delete group (fixes #2028) (#2043)
Co-authored-by: GatewayJ <8352692332qq.com>
2026-03-02 14:19:01 +08:00
weisd
e3815aa101 fix(ecstore): add etag fallback when mod_time unavailable in metadata (#2042) 2026-03-02 13:50:17 +08:00
dependabot[bot]
fd32507ce5 build(deps): bump datafusion from 52.1.0 to 52.2.0 in the dependencies group (#2037)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-02 11:37:24 +08:00
GatewayJ
ba32fd9d96 fix(s3): allow anonymous access when PublicAccessBlock config is miss… (#2039)
Co-authored-by: GatewayJ <8352692332qq.com>
2026-03-02 11:37:00 +08:00
安正超
273dbc9c38 feat(s3): return 409 BucketAlreadyExists when non-owner creates existing bucket (#2034) 2026-03-01 22:53:41 +08:00
安正超
f0c5d762f3 feat(s3): enforce RestrictPublicBuckets for anonymous access (#2033)
Signed-off-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-01 20:50:19 +08:00
houseme
c452f24487 Optimize log cleanup and rotation, update dependencies (#2032)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-03-01 20:09:52 +08:00
安正超
798e620088 fix(s3): add x-amz-grant-* headers to policy condition values (#2031) 2026-03-01 19:00:04 +08:00
安正超
e5e1010c31 fix(s3): return InvalidRange when CopySourceRange exceeds source object size (#2029) 2026-03-01 17:46:31 +08:00
安正超
8aecc7267b fix(s3): implement S3-compliant CORS and bucket existence checks (#2026) 2026-03-01 16:02:02 +08:00
安正超
f42b155f59 fix(s3): allow Object Lock on versioned buckets and reject invalid checksums (#2024) 2026-03-01 14:19:02 +08:00
Smig
d13c423d50 Bump Helm Chart version to 0.0.83 and appVersion to 1.0.0-alpha.83 (#2019)
Signed-off-by: Smig <89040888+smiggiddy@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-01 13:15:50 +08:00
安正超
fbb162d8bb test(s3): promote 145 passing tests to implemented list (#2023) 2026-03-01 12:55:49 +08:00
github-actions[bot]
7c52af22e5 chore(deps): update flake.lock (#2020)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-01 12:43:12 +08:00
安正超
7a83b818b8 fix(policy): address review feedback from #2018 (#2021)
Signed-off-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
2026-03-01 11:05:20 +08:00
heihutu
595f916ba3 build(deps): bump the dependencies group with 2 updates (#2017)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
2026-03-01 10:04:05 +08:00
安正超
7eb136faf0 feat(policy): add Service principal, ArnLike/IfExists conditions, and logging error ordering (#2018) 2026-03-01 08:44:42 +08:00
heihutu
2c01b8c49d feat(obs): add advanced log management configuration (#2016)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
2026-03-01 03:23:48 +08:00
安正超
e7466eb1cc fix: policy StringNotEquals double negation and delete_objects version mapping (#2015) 2026-03-01 03:13:52 +08:00
evan slack
fd1b903531 fix(obs): Update observability docker compose stack (#2010) 2026-03-01 03:03:50 +08:00
安正超
fe884eabfc fix(s3): improve S3 API compatibility for versioning, SSE, and policy (#2013) 2026-03-01 02:21:13 +08:00
安正超
0701e1c35f chore(s3-tests): promote 42 passing tests to implemented list (#2011) 2026-03-01 01:39:08 +08:00
唐小鸭
568c07ced9 fix: implement handling for "aws-chunked" Content-Encoding (#2009)
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-03-01 01:22:12 +08:00
安正超
f93db578df fix(s3): reject invalid SSE algorithm (aes:kms) in PutObject (#2008) 2026-03-01 00:56:26 +08:00
安正超
1872bdcedd fix(s3): reject SSE-C with partial headers per S3 spec (#2007) 2026-02-28 22:56:35 +08:00
安正超
27ff35e574 fix: avoid region fallback panic paths (#2006) 2026-02-28 22:35:56 +08:00
安正超
aa3f960b3d Fix: validate SSE headers in object read/write paths (#2005) 2026-02-28 18:07:56 +08:00
安正超
212b7ae8e1 Update logo image link in README.md (#2004)
Signed-off-by: 安正超 <anzhengchao@gmail.com>
2026-02-28 16:29:11 +08:00
安正超
b4a633ebc6 fix(ecstore): set expiration header for put object via lifecycle prediction (#2003) 2026-02-28 16:21:58 +08:00
安正超
3f5ccb20fc fix(s3): normalize GetObjectAttributes ETag XML response (#2002) 2026-02-28 14:53:53 +08:00
houseme
274b6f8bc7 build(deps): bump the dependencies group with 4 updates (#2001) 2026-02-28 12:01:17 +08:00
安正超
a24cbbb7a6 fix(s3): return proper HTTP 400 for SSE-C validation errors (#1998) 2026-02-28 10:24:46 +08:00
安正超
af6c32efac refactor: improve code quality with safer error handling, trait decomposition, and dead code cleanup (#1997) 2026-02-28 01:19:47 +08:00
Brayan Jules
7ce23c6b54 fix(ecstore): allow trailing slash in object names to match S3 behavior (#1996)
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-02-27 22:46:42 +08:00
evan slack
dcbc67eb91 perf(lock): Use global lock manager, instead of one per request (#1848)
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-27 22:26:32 +08:00
LoganZ2
e73b17aff6 fix(scanner): skip recent IO-error objects (#1860)
Signed-off-by: LoganZ2 <103290230+LoganZ2@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-02-27 22:25:52 +08:00
GatewayJ
55396f13d4 feat: policy add object tag (#1908)
Co-authored-by: GatewayJ <8352692332qq.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-02-27 22:24:57 +08:00
Senol Colak
b69183aadf Openstack Keystone integration - v1 keeps the same mechanism as (#1961)
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-02-27 22:23:35 +08:00
houseme
d17d2083d4 feat(targets): enhance webhook TLS support with custom CA and skip-verify (#1994)
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-02-27 21:24:49 +08:00
evan slack
bdb2a9e9b7 fix(dashboard): Rename grafana dashboard rustfs.yaml -> rustfs.json (#1987)
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-02-27 15:49:34 +08:00
安正超
9d2b8822cf refactor: stabilize heal format recovery integration tests (#1984)
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-27 15:26:19 +08:00
houseme
3433dfa88e feat(config): refine defaults and improve region handling (#1990)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-27 15:02:29 +08:00
weisd
68cc0a5df7 chore(heal): remove unused global response broadcast (#1991) 2026-02-27 14:12:03 +08:00
weisd
368bba3345 fix(ecstore): adjust unformatted disk error mapping (#1988) 2026-02-27 13:45:33 +08:00
安正超
b23a1a4ff9 refactor(app): remove dead objects/ code and migrate put_object_extract to usecase layer (#1980)
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-27 10:24:48 +08:00
LeonWang0735
10140be6d8 fix(replication): handle TLS CA trust and force-delete replication edge cases (#1983) 2026-02-27 08:40:39 +08:00
heihutu
c32b6f2f37 refactor region parsing (#1981) 2026-02-27 02:34:09 +08:00
heihutu
d983638391 build: update docker config and refine s3s region handling (#1976)
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-27 01:21:12 +08:00
安正超
eb07f084cb refactor(app): complete phase 5 gate and equivalence guards (#1979) 2026-02-26 23:05:24 +08:00
安正超
09aa6d9f6f refactor(app): remove remaining global access in main init (#1978) 2026-02-26 22:14:51 +08:00
安正超
a3c76618f3 refactor(ci): add layered dependency guard baseline (#1977) 2026-02-26 21:55:45 +08:00
安正超
2c85721654 refactor(app): centralize context resolvers for admin/server paths (#1975) 2026-02-26 20:41:11 +08:00
安正超
dafb31d208 refactor(rpc): use node name accessor in health handlers (#1972) 2026-02-26 19:37:16 +08:00
安正超
40903ec2af refactor(admin): move KMS management handlers (#1971) 2026-02-26 15:29:51 +08:00
安正超
49579129c1 refactor(app): decouple AppContext adapters from GLOBAL statics (#1970) 2026-02-26 14:54:45 +08:00
安正超
40692f18ed fix(iam): address PR 1875 review issues for OIDC STS flows (#1969) 2026-02-26 14:38:57 +08:00
Jeff Poegel
c35ef84a8c feat(iam): add OpenID Connect SSO with claim-based policy resolution (#1875)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-02-26 14:03:17 +08:00
安正超
0f8bc461d6 refactor(admin): route kms handlers through app context (#1967)
Signed-off-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-26 13:31:59 +08:00
evan slack
ae6eacd7e3 fix(typo): change dang_ling to dangling (#1968) 2026-02-26 13:03:36 +08:00
安正超
4b82cc20bb refactor(admin): route kms handlers via app context (#1965) 2026-02-26 10:32:16 +08:00
安正超
1c01c3d73a refactor(app): route buffer config through AppContext (#1964) 2026-02-26 09:19:59 +08:00
安正超
1a549d78ca refactor(storage): converge put-object quota metadata context (#1963) 2026-02-26 09:18:07 +08:00
安正超
7909a57634 refactor(server): route config access through AppContext (#1960)
Signed-off-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-26 00:04:03 +08:00
安正超
fd86d0bd0f refactor(admin): route tier config manager through AppContext (#1959) 2026-02-25 23:11:00 +08:00
安正超
4c08e18812 refactor(app): converge lower-priority global reads via AppContext (#1958) 2026-02-25 22:30:53 +08:00
安正超
dc795a494a refactor(app): remove multipart metadata global fallback (#1957)
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
2026-02-25 21:47:35 +08:00
houseme
024a3107d8 build(deps): bump the dependencies group with 4 updates (#1955) 2026-02-25 19:21:20 +08:00
安正超
aea7f41149 refactor(app): route admin/object globals through AppContext (#1954) 2026-02-25 18:01:52 +08:00
majinghe
52090d72d6 fix: add liveness and readiness probe (#1953) 2026-02-25 15:43:46 +08:00
安正超
d774d6821b refactor(app): route metadata/endpoints access through AppContext (#1949) 2026-02-25 15:07:09 +08:00
安正超
672c255567 fix: restore SSE baseline on latest main (#1951) 2026-02-25 14:19:04 +08:00
GatewayJ
62b51b5649 feat: admin permission check (#1783)
Signed-off-by: GatewayJ <835269233@qq.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-25 11:58:30 +08:00
安正超
0d9e5f1e93 refactor(app): add iam and notify interface boundaries (#1948) 2026-02-25 09:49:02 +08:00
安正超
b48f273c7d refactor(filemeta): split filemeta into focused submodules (#1946) 2026-02-25 08:38:32 +08:00
安正超
7f132a290c refactor(ecstore): split set_disk.rs into submodules (#1945) 2026-02-25 07:41:08 +08:00
安正超
095b77795d refactor(ecstore): split store.rs into store submodules (#1942) 2026-02-25 06:35:27 +08:00
安正超
aac4a6c25f refactor(storage): split tonic_service into rpc modules (#1939) 2026-02-24 23:04:38 +08:00
安正超
5ed4772ed8 refactor(ecstore): split store_api into focused modules (#1938) 2026-02-24 22:31:46 +08:00
安正超
f4874ec89d refactor(storage): extract remaining s3_api response builders (#1937) 2026-02-24 21:57:43 +08:00
安正超
1b1fd6295d test(admin): cover kms list-keys route registration (#1936) 2026-02-24 21:31:19 +08:00
安正超
c864d14c9e [codex] Refactor P1-07: slim KMS handler ownership (#1935) 2026-02-24 21:12:03 +08:00
yxrxy
deb1dbedbb fix(ftps): Fix basename extraction and implement recursive delete (#1920)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
2026-02-24 20:49:57 +08:00
mkrueger92
3b024a9dc5 rustfs#1916 Allow existing secrets to be used for tls certs in ingress (#1918)
Signed-off-by: mkrueger92 <7305571+mkrueger92@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-24 20:34:08 +08:00
安正超
c692777ead refactor(app): migrate delete-objects and listing orchestration (#1933) 2026-02-24 20:09:01 +08:00
安正超
c10084867a refactor(app): migrate multipart list and copy-part orchestration (#1932) 2026-02-24 19:27:41 +08:00
0xdx2
17b3054a77 feat(s3select): add JSON handling and flattening for EcObjectStore (#1930)
Signed-off-by: 0xdx2 <xuedamon2@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-24 18:05:34 +08:00
LeonWang0735
06d12a8ec8 feat(replication):add replication bandwidth throttle monitor and reader (#1885)
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-24 15:21:45 +08:00
Niraj Yadav
8f00d1fbb0 feat(admin): implement handler for delete group (#1901)
Signed-off-by: Niraj Yadav <niryadav@redhat.com>
Signed-off-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: yxrxy <yxrxytrigger@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-02-24 13:00:46 +08:00
安正超
f9da807bdf test: add regression coverage for access action mapping (#1928) 2026-02-24 12:27:58 +08:00
cxymds
49eda934d9 fix: policy-action-1903 (#1927)
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-02-24 11:21:35 +08:00
安正超
e556b64996 refactor(app): migrate bucket ACL/location/list-buckets orchestration (#1924) 2026-02-24 11:10:21 +08:00
安正超
bc026b746e refactor(app): migrate object lock and attributes flows to usecase (#1922) 2026-02-23 22:15:24 +08:00
安正超
588631b02a refactor(app): migrate object ACL/tagging flows to usecase (#1921) 2026-02-23 21:19:57 +08:00
安正超
0f631e6dd2 refactor(app): migrate bucket config flows to usecase (#1919) 2026-02-23 20:11:18 +08:00
dependabot[bot]
045988e062 build(deps): bump the dependencies group with 15 updates (#1912)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-23 17:57:53 +08:00
安正超
1614cc1b2c refactor(app): migrate restore/select and admin info orchestration (#1917) 2026-02-23 14:20:07 +08:00
安正超
5b8cbaf7c7 refactor: migrate multipart orchestration to usecase (#1915) 2026-02-23 13:10:58 +08:00
安正超
3cdd2b313b refactor(app): migrate bucket sub-operation flows (#1914) 2026-02-23 12:15:30 +08:00
安正超
d1768aa1c3 refactor(app): migrate create/delete/head bucket flows (#1913) 2026-02-23 11:05:56 +08:00
安正超
d9c97c5c52 refactor(app): migrate copy/delete/head object flows (#1911) 2026-02-23 10:32:53 +08:00
安正超
cf1d109bb9 refactor(app): route put/get/listv2 through usecases (#1910) 2026-02-22 23:36:48 +08:00
安正超
84053484e6 refactor(app): add AppContext skeleton wiring (#1909) 2026-02-22 22:41:42 +08:00
安正超
4211652991 refactor(app): add application layer module entry (#1907) 2026-02-22 22:15:37 +08:00
安正超
4a6e81d427 refactor(storage): extract object-lock response builders from ecfs (#1906) 2026-02-22 12:27:07 +08:00
安正超
094e6a7319 refactor(storage): extract tagging helpers from ecfs (#1881) 2026-02-22 11:20:41 +08:00
github-actions[bot]
6972a7b4b2 chore(deps): update flake.lock (#1905)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-02-22 10:08:24 +08:00
evan slack
23f7ffe36b fix(startup): Only monitor disk health after format loading (#1854)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2026-02-21 20:46:14 +08:00
LeonWang0735
f31cd4b716 fix(replication): replicate delete all versions to targets (#1898)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-02-21 20:12:05 +08:00
loverustfs
da63b5e562 Fix/x86 64 compat drop target cpu native (#1895) 2026-02-21 10:24:14 +08:00
loverustfs
5d737eaeb7 fix(ecstore): invalidate GlobalFileCache after write_all_private to fix DeleteMarker visibility (#1890) 2026-02-20 22:47:45 +08:00
cui
cde66e0a46 fix: uncompress -> compress (#1855)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2026-02-20 17:20:32 +08:00
etak64n
1a4a84bebe fix: remove deprecated darwin.apple_sdk references from flake.nix (#1884)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-02-20 16:22:06 +08:00
Burak Bozacı
db70a2bed0 Feature/deployment probe override (#1876)
Co-authored-by: capitansec <capitansec@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-20 16:19:51 +08:00
安正超
583377d2a5 refactor(storage): extract ACL response builders into s3_api (#1880) 2026-02-19 22:57:35 +08:00
安正超
a4e8e1fd5e refactor(storage): extract ListBuckets response assembly (#1879) 2026-02-19 22:35:47 +08:00
LeonWang0735
c7211c9df7 fix:correctly handle replicate delete (#1850)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-02-19 13:23:29 +08:00
Miguel Caballer Fernandez
7ae0415715 Increase ACCESS_KEY_MAX_LEN to 128 (#1870)
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-18 22:00:16 +08:00
Rohmilchkaese
3f4cb6883e fix(helm): apply traefikAnnotations and gate TLS secret on certManager.enabled (#1864)
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-18 18:59:41 +08:00
houseme
d345ace326 perf(obs): optimize metrics recorder and telemetry initialization (#1859) 2026-02-18 07:05:43 +08:00
evan slack
9da332c47d perf(metrics): Cache metric handles instead of creating each call (#1852)
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
2026-02-18 01:03:35 +08:00
evan slack
8010284aa3 obs(export): Add env vars to selectivly disable exporting traces/metrics/logs (#1853)
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-18 00:38:30 +08:00
loverustfs
cf633569a2 fix: remove duplicate common prefixes for slash delimiter (#1797) (#1841)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
2026-02-17 21:36:31 +08:00
evan slack
229f0f89c8 perf(read): Remove unecessary allocation in read_xl_meta_no_data (#1846)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2026-02-16 19:08:12 +08:00
安正超
d19edd9a2c refactor(storage): use named params for multipart list APIs (#1833) 2026-02-16 11:50:07 +08:00
dependabot[bot]
e6c032cc92 build(deps): bump the dependencies group with 5 updates (#1845) 2026-02-16 10:18:23 +08:00
唐小鸭
4413878739 fix(compress): downgrade non-compressible log level and expand excluded file types (#1780) 2026-02-16 03:36:22 +08:00
evan slack
9786d9b004 metrics(scanner): Add metrics to scanner (#1823)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
2026-02-15 18:36:40 +08:00
Jasmine Lowen 🦁
bffeacf1d2 chore(nix): update flake lock & fix devshell+package (#1805) 2026-02-15 18:01:58 +08:00
houseme
c7f1a18cc5 chore(deps): bump zip from 7.4.0 to 8.0.0 (#1837) 2026-02-15 17:40:22 +08:00
heihutu
f4028a4641 chore(deps): update flake.lock (#1835)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-15 16:58:01 +08:00
Jasmine Lowen 🦁
21ef6d505e feat(config): allow specifying keys via files (key files) (#1814)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
2026-02-15 16:28:52 +08:00
majinghe
da15d622a0 fix: gateway api listener name hardcode issue (#1834)
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-15 16:01:40 +08:00
majinghe
2d4d240508 feat: add existing pvc claim for standalone (#1829)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2026-02-15 15:49:37 +08:00
evan slack
2093a13308 logging(disks): Propogate storage disk init error, improve logging (#1825)
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-15 15:28:07 +08:00
loverustfs
e41ddad003 docs: reformat CLA.md for better readability 2026-02-15 14:40:22 +08:00
loverustfs
8bc2db750f docs: update contributor license agreement 2026-02-15 14:36:01 +08:00
安正超
21ade0aaa7 refactor(storage): use named params for ListObjectVersions (#1832) 2026-02-15 13:12:00 +08:00
安正超
2debc14e4d refactor(storage): extract ListObjectsV2 parameter parsing (#1831) 2026-02-15 12:55:07 +08:00
安正超
2fadb16365 refactor(storage): extract list object versions helpers (#1830) 2026-02-15 12:32:26 +08:00
安正超
339a5db668 refactor(storage): extract multipart list param parser (#1817)
Signed-off-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-15 11:50:19 +08:00
realyashnegi
715cf33b89 fix(admin): return 503 when health deps are not ready (#1824) 2026-02-15 10:26:47 +08:00
houseme
4895c180e1 ci(Flake): optimize nix-flake-update workflow (#1827)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
2026-02-15 09:21:43 +08:00
houseme
1554e7e76a ci: optimize and translate nix workflow (#1821) 2026-02-15 00:37:57 +08:00
houseme
8512a38f68 build(deps): bump the dependencies group with 16 updates (#1820) 2026-02-14 23:43:09 +08:00
evan slack
9fe3d5621e feat(observability): Add additional metric panels to grafana dashboard (#1778)
Signed-off-by: evan slack <51209817+evanofslack@users.noreply.github.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-14 21:41:07 +08:00
安正超
d3ff6ff36a refactor(storage): centralize S3 response/error helpers (#1818) 2026-02-14 21:20:57 +08:00
安正超
546485a8ee refactor(storage): extract list_parts parameter parsing (#1816) 2026-02-14 20:55:14 +08:00
安正超
c701f30bd3 refactor(storage): extract ListMultipartUploads response builder (#1815) 2026-02-14 20:44:23 +08:00
安正超
6d6a2b7ed6 refactor(storage): extract ListParts response assembly helper (#1812) 2026-02-14 13:58:18 +08:00
安正超
257e31a4b4 refactor(storage): extract ListObjectsV2 response assembly helper (#1811) 2026-02-14 12:31:10 +08:00
安正超
22ae004205 refactor(storage): extract list_objects v1 response builder (#1810) 2026-02-14 11:50:15 +08:00
GatewayJ
fb0267981d fix(iam): STS parent groups fallback and session policy debug for #1423 (#1804)
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-02-14 11:40:36 +08:00
安正超
eaeb83aa1c refactor(storage): add s3_api facade and extract read helper (#1803)
Signed-off-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-14 11:23:17 +08:00
shadow1runner
8e1fcd4792 fix(helm): add {{ .Release.Namespace }} for kustomize v5.8 compat, closes #1808 (#1809)
Co-authored-by: Helmut Wolf <3902045+shadow1runner@users.noreply.github.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2026-02-14 10:25:13 +08:00
yxrxy
23f79ae88f fix: improve IAM and quota authorization (#1781)
Signed-off-by: yxrxy <yxrxytrigger@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2026-02-14 10:09:47 +08:00
安正超
b3daa80e72 refactor(admin): extract kms management route registration (#1801) 2026-02-13 23:21:04 +08:00
安正超
986a259a9b refactor(admin): move kms key route registration (#1799) 2026-02-13 19:18:15 +08:00
安正超
53d601e6ec refactor(admin): move kms dynamic route registration (#1798) 2026-02-13 18:34:02 +08:00
安正超
6bf4fd1273 refactor(admin): extract user policy binding route registration (#1796) 2026-02-13 17:11:48 +08:00
安正超
c4a68d3efe refactor(admin): extract user lifecycle route registration (#1795) 2026-02-13 13:42:25 +08:00
安正超
cbb4329428 refactor(admin): extract user IAM route registration (#1794) 2026-02-13 12:35:59 +08:00
安正超
2fc36bb52e fix: restore s3 compatibility regressions and CI coverage (#1793) 2026-02-13 12:26:52 +08:00
安正超
921cfb849c refactor(admin): move accountinfo route registration (#1790) 2026-02-13 10:36:54 +08:00
安正超
bfc924a70b refactor(admin): move route registration into handler modules (#1789) 2026-02-12 23:27:35 +08:00
安正超
7d8f7a12ba refactor(admin): modularize handlers and route registration (#1787) 2026-02-12 21:28:48 +08:00
安正超
4203adaac1 refactor(admin): split remaining handlers into modules (#1782) 2026-02-12 20:56:52 +08:00
evan slack
c60be70d4d tool(agents): AGENTS.md instructs to respond in english, not chinese (#1775) 2026-02-12 10:48:32 +08:00
安正超
2edf0ed747 refactor(admin): extract health check handler module (#1777) 2026-02-12 10:46:07 +08:00
evan slack
9824171995 feat(observability): Add grafana dashboard, observability changes (#1770)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-02-11 15:55:08 +08:00
heihutu
ecceb8fd1c build(deps): bump the dependencies group with 2+ updates (#1769)
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-11 09:39:47 +08:00
Tyooughtul
1184806c3f Fix/resolve pr 1710 (#1743) 2026-02-11 08:24:55 +08:00
houseme
4411c625e2 feat(metrics): async collection with configurable intervals & graceful shutdown (#1768) 2026-02-10 21:37:24 +08:00
Md. Amdadul Bari Imad
c07ed61989 fix(entrypoint): remove dead HTTP URL check in volume filtering (#1761)
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-10 12:07:08 +08:00
yxrxy
da58f8e291 fix: Allow non-admin users to read bucket quota configuration. (#1759)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
2026-02-10 11:23:21 +08:00
mengyu-sxyz
aa011ade19 fix: improve part size calculation in optimal_part_info function (#1532)
Signed-off-by: mengyu-sxyz <mengyu@sentio.xyz>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
2026-02-10 10:56:47 +08:00
LoganZ2
ccf3b29df5 fix: stabilize head metadata responses and heal tests (#1732)
Signed-off-by: LoganZ2 <103290230+LoganZ2@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-10 09:44:14 +08:00
evan slack
682b5bbb2f perf(scanner): Change DataUseageEntry from clone to borrow (#1757)
Signed-off-by: evan slack <51209817+evanofslack@users.noreply.github.com>
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-10 08:53:26 +08:00
LeonWang0735
f4e9ef2edc fix(replication): avoid re-replication loop in Active-Active replication (#1751)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-02-09 14:11:30 +08:00
dependabot[bot]
ff8c1c782a build(deps): bump libunftp from 0.21.0 to 0.22.0 in the dependencies group across 1 directory (#1756)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-09 13:43:47 +08:00
houseme
ca6076fe18 build(deps): bump the dependencies group with 5 updates (#1755) 2026-02-09 11:31:43 +08:00
evan slack
58ee140324 perf(regex): Compile bucket validation regex once (#1753) 2026-02-09 10:50:17 +08:00
thorntonmc
927f3a57d7 perf(quota): Skip expensive usage checks when no quota configured (#1749)
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-08 22:37:53 +08:00
majinghe
a574285ab2 feat: add support for mtls with kubernetes installation (#1741) 2026-02-08 09:31:58 +08:00
LeonWang0735
60793c17d7 fix: persist replication status and timestamp after replicate_object (#1747)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-07 12:50:37 +08:00
houseme
0b870d6301 build(deps): bump the dependencies group with 19 updates (#1745) 2026-02-07 12:22:14 +08:00
Isaac Mills
d635ee8d2e Propogate tracing context from HTTP requests into spans (#1739)
Signed-off-by: Isaac Mills <57533634+StratusFearMe21@users.noreply.github.com>
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-07 11:52:31 +08:00
yxrxy
5c2eda356e feat: migrate FTP/SFTP to protocols crate and update dependencies (#1580)
Signed-off-by: yxrxy <yxrxytrigger@gmail.com>
Signed-off-by: houseme <housemecn@gmail.com>
Signed-off-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2026-02-06 20:58:42 +08:00
LeonWang0735
a2b88a79ec test(e2e_test): add automated cluster environment for conditional PUT race test (#1673)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2026-02-06 15:01:59 +08:00
LeonWang0735
6eb90e7df9 fix: fetch_owner set to true when calling list_objects_v2 in the list_objects function (#1730)
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-05 22:48:32 +08:00
majinghe
466429b958 feat: add contour as ingress controller with http proxy (#1729)
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-05 21:04:18 +08:00
GatewayJ
c8411fd62c fix: bucket policy id field serde (#1726) 2026-02-05 18:38:57 +08:00
houseme
6bba41f11f Fix/fix issues #1564 (#1708)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-05 13:45:14 +08:00
majinghe
e30781654d enhancement: add support for http to https redirect for traefik gatew… (#1712) 2026-02-04 20:21:42 +08:00
唐小鸭
7a42af922d Refactor: refactor SSE layer and KMS subsystem (#1703)
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-04 16:10:33 +08:00
weisd
4d19b069c3 fix: replication delete (#1714) 2026-02-04 13:39:35 +08:00
loverustfs
a4563f7b41 fix: return null versionId when suspended (#1066) (#1709) 2026-02-04 09:22:52 +08:00
dependabot[bot]
174e12bf66 build(deps): bump bytes from 1.11.0 to 1.11.1 (#1711)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-04 08:41:30 +08:00
majinghe
f03034b99e feat: add glibc based docker image support (#1705)
Signed-off-by: majinghe <42570491+majinghe@users.noreply.github.com>
2026-02-03 21:27:20 +08:00
LeonWang0735
36f14acbe9 fix: object lock compliance mode allows deletion (#1687)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-02-03 17:06:24 +08:00
majinghe
2f66f15524 feat: add obs log rotations environment variables (#1702) 2026-02-03 10:35:30 +08:00
houseme
cb468fb32f Refactor trusted-proxies: modernize utils, improve safety, and fix clippy lints (#1693)
Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
Co-authored-by: GatewayJ <835269233@qq.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
2026-02-03 01:06:22 +08:00
houseme
d1a70176a2 fix: Fixed that account_id returns normal value (#1695) 2026-02-02 20:15:54 +08:00
LeonWang0735
ec4458f846 Fix/correctly handle terraform s3 backend with versioned bucket (#1686)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-02-02 12:43:07 +08:00
yxrxy
00ccc19e27 Revert "fix: resolve Issue #1465 - IAM credential change crash (#1535)" (#1685) 2026-02-02 07:14:53 +08:00
houseme
07cf2feaad fix(pprof): Fixed the problem that pprof crate does not support the window platform (#1681)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-01 00:12:55 +08:00
likewu
087f58b7c8 fix(lifecycle): lifecycle fixes (#1625)
Signed-off-by: likewu <likewu@126.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2026-01-31 19:14:47 +08:00
moechs
6fc35e442c fix: add gatewayApi.enabled check to TraefikService template (#1679)
Signed-off-by: moechs <68768084+moechs@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-31 17:42:27 +08:00
majinghe
a798b20308 fix: init container security hardened for operation permission error (#1680) 2026-01-31 17:00:42 +08:00
houseme
38b779b924 feat(profiling): support cross-platform memory profiling with mimalloc and pprof (#1674) 2026-01-30 22:23:49 +08:00
weisd
1aba8c10b9 refactor: Remove unused data usage collection code (#1672)
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-30 19:08:39 +08:00
houseme
90ed75a3dc refactor(utils/os): Optimize Windows OS utilities and add safety comments (#1671)
Co-authored-by: weisd <weishidavip@163.com>
2026-01-30 16:13:42 +08:00
weisd
dce117840c refactor: NamespaceLock (nslock), AHM→Heal Crate, and Lock/Clippy Fixes (#1664)
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: weisd <2057561+weisd@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-30 13:13:41 +08:00
majinghe
1c085590ca fix: traefik gateway api support with sticky session (#1660)
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-30 12:34:31 +08:00
houseme
2ee81496b0 fix: deduplicate disks in capacity calculation to prevent inflation (#1656) 2026-01-30 00:03:21 +08:00
LeonWang0735
022e3dfc21 fix:s3 tests fix (#1652)
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-29 19:35:03 +08:00
majinghe
6b15e727f5 feat: add virtual host mode support for kubernetes installation (#1655) 2026-01-29 17:55:24 +08:00
zhangwenlong
ab84da24ef fix: build error on loongarch64 (#904)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: 0xdx2 <xuedamon2@gmail.com>
2026-01-29 17:41:56 +08:00
houseme
e377c7e7f9 fix(head): clearer NoSuchKey for prefix keys and validate part numbers (#1638) 2026-01-29 15:40:51 +08:00
houseme
7c497a30b2 build(deps): bump the dependencies group with 12 updates (#1650) 2026-01-29 14:21:35 +08:00
evan slack
51e8a4820f fix: map unversioned destination replication error correctly (#1645) 2026-01-29 10:13:36 +08:00
loverustfs
a81bbed551 Fix align 2026-01-28 22:03:42 +08:00
loverustfs
072de6b025 Update README to streamline badge links (#1643)
Signed-off-by: loverustfs <hello@rustfs.com>
2026-01-28 21:58:48 +08:00
loverustfs
9269cb779b Add badge to readme
Add badge to readme

Signed-off-by: loverustfs <hello@rustfs.com>
2026-01-28 21:57:22 +08:00
安正超
fff175dcdd chore: remove skills and docs dir. (#1631) 2026-01-27 22:43:22 +08:00
LeonWang0735
26c0230e8f fix:use RFC1123 format for last-modified header in 304 responses (#1627)
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-27 19:06:31 +08:00
LeonWang0735
8edb1affc0 Fix:s3 compatibility (#1617)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
2026-01-27 16:56:31 +08:00
heihutu
74759b6e99 fix: missing object.key in S3 event notifications for multipart uploads #1609 (#1624)
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-27 13:32:56 +08:00
yxrxy
db29c0cae2 fix: missing object.key in S3 event notifications for multipart uploads (#1621)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-01-27 11:48:45 +08:00
weisd
6ab7b75fd4 fix bug (#1615)
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-27 09:54:13 +08:00
houseme
2108c4ad28 fix: remove plaintext credential logging (#1619) 2026-01-27 01:47:39 +08:00
houseme
d251b9fb35 fix: unify path handling to use S3-standard forward slashes on all platforms (#1555)
Signed-off-by: houseme <housemecn@gmail.com>
Signed-off-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: LeonWang0735 <wlywly0735@126.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-01-26 18:49:21 +08:00
dependabot[bot]
172bed0ff2 build(deps): bump the dependencies group with 3 updates (#1612)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-26 11:00:56 +08:00
安正超
1db7bac2dc fix: readme list. (#1608) 2026-01-25 09:35:12 +08:00
安正超
4890fb25c1 fix: listobjects v2 pagination (#1607) 2026-01-25 09:30:08 +08:00
Dat Truong
3838a13606 Update README with docker-buildx.sh features (#1585)
Signed-off-by: Dat Truong <truongminhdat07@gmail.com>
2026-01-25 09:25:26 +08:00
houseme
c28134c957 Add support for success_action_status and success_action_redirect in AWS S3 POST object uploads (#1606) 2026-01-25 02:50:29 +08:00
安正超
173dad27d1 fix: preserve exact JSON format in bucket policy GET response (#1598)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-01-24 23:02:01 +08:00
GatewayJ
9285acba06 feat: object retention (#1589) 2026-01-24 22:12:45 +08:00
LeonWang0735
db5e72e475 Fix/correctly handle compression (#1594) 2026-01-24 10:55:10 +08:00
安正超
16160b7b84 fix: use main user for s3tests tenant to prevent teardown failures (#1597) 2026-01-24 10:05:10 +08:00
安正超
461ba3aeba fix: handle duplicate part numbers in CompleteMultipartUpload (#1584)
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-23 22:41:46 +08:00
yxrxy
65de487eba fix: resolve Issue #1465 - IAM credential change crash (#1535)
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2026-01-23 15:11:01 +08:00
houseme
e5284a85ed fix: Fixed detection warnings in rust v1.93.0 (#1591) 2026-01-23 13:09:41 +08:00
安正超
fd08be7be2 chore: update README. (#1586) 2026-01-22 22:44:28 +08:00
安正超
43bf846633 fix: correct max_keys field in list_object_versions response (#1576)
Signed-off-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-01-22 20:58:03 +08:00
heihutu
db253c01a9 refactor: replace chrono with jiff for time handling (#1582)
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-22 17:24:56 +08:00
weisd
6631407416 feat: Add RustFS Scanner Module and Multiple Bug Fixes (#1579) 2026-01-22 13:39:38 +08:00
LeonWang0735
6c5f8e591a Fix/correctly handle get object ifmatch&ifnonematch (#1563)
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
2026-01-22 10:34:05 +08:00
LeonWang0735
3b5f7fb3ff Fix/correctly handle object lock (#1556)
Signed-off-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-01-22 00:15:21 +08:00
安正超
87ead2bea3 fix: implement get_object_torrent to return 404 NoSuchKey (#1575)
Signed-off-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-01-21 21:49:22 +08:00
安正超
0320508f8d feat: add comprehensive skills and agents for Rust development (#1573) 2026-01-21 20:45:39 +08:00
安正超
47ec125589 fix(s3): return NoSuchUpload for abort_multipart_upload when upload_id not found (#1569)
Signed-off-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-01-21 20:45:18 +08:00
安正超
9fc1c264b0 fix(s3): add x-amz-tagging-count header to HEAD object response (#1568)
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-21 20:45:06 +08:00
安正超
ae50760fcc fix(s3): return NoSuchTagSet for get_bucket_tagging when tags not set (#1567) 2026-01-21 00:10:37 +08:00
majinghe
60d54af749 enhancement: add podman installation support and static files generating (#1565) 2026-01-20 20:53:59 +08:00
houseme
f59380ae17 docs: remove deprecated RUSTFS_EXTERNAL_ADDRESS and RUST_LOG variables (#1561) 2026-01-20 17:34:33 +08:00
houseme
7c8fd8518f feat(admin): make capacity calculation resilient when backend info is missing (#1560)
Signed-off-by: heihutu <30542132+heihutu@users.noreply.github.com>
Signed-off-by: majinghe <42570491+majinghe@users.noreply.github.com>
Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
2026-01-20 13:03:09 +08:00
majinghe
14ce251e3b enhancement: unify logger level setting using obs env instead of RUST_LOG (#1529)
Signed-off-by: heihutu <30542132+heihutu@users.noreply.github.com>
Signed-off-by: majinghe <42570491+majinghe@users.noreply.github.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
2026-01-20 12:09:41 +08:00
Peter Olds
d578707f95 Helm: Add ability to enable Virtual Hosting paths (#1559) 2026-01-20 10:39:17 +08:00
houseme
46126ade81 upgrade s3s version (#1558) 2026-01-19 23:14:23 +08:00
heihutu
51bfb9c4f2 perf: optimize transport layer (TCP/TLS/H2) for S3 traffic (#1551)
Signed-off-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-19 13:43:02 +08:00
houseme
99be71e4c2 feat(http): Enable dynamic window adjustment (#1549) 2026-01-19 09:53:17 +08:00
houseme
a9f499282c fix: Increase lock acquire timeout for network storage reliability (#1548) 2026-01-19 01:14:36 +08:00
houseme
c9e2d7da2a Dependabot/cargo/dep 0117 (#1547) 2026-01-18 12:02:53 +08:00
Juri Malinovski
e52a60e64e helm: disable default resources, fix poddisruptionbudget condition (#1539)
Signed-off-by: Juri Malinovski <juri.malinovski@coolbet.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-17 21:35:01 +08:00
Michele Zanotti
28e2af0829 helm: use values in test connection pod image (#1536)
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-17 21:14:55 +08:00
LeonWang0735
3012119b81 optimize:replace size magic number -1 with SIZE_TRANSFORMED constant (#1542) 2026-01-17 19:24:44 +08:00
heihutu
76fa86fdc5 feat(server): optimize http transport and socket configuration for S3… (#1537)
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-17 02:53:24 +08:00
LeonWang0735
2ab6f8c029 fix:correctly handle compress object when put object (#1534) 2026-01-16 23:11:48 +08:00
weisd
0927f937a7 fix: Fix BitrotWriter encode writer implementation (#1531) 2026-01-16 17:11:54 +08:00
Audric
548a39ffe7 fix: return error instead of silently ignoring invalid ARNs in notification config (#1528)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-16 16:12:55 +08:00
LeonWang0735
ed4329d50c fix:correctly handle copy object (#1512)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-01-16 10:07:48 +08:00
LeonWang0735
18b22eedd9 Fix:correctly handle versioning obj (#1521) 2026-01-16 08:12:05 +08:00
GatewayJ
55e4cdec5d feat: add Cors (#1496)
Signed-off-by: GatewayJ <835269233@qq.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-15 20:03:26 +08:00
houseme
dceb7aac8a upgrade s3s from 0.13.0-alpha.1 to 0.13.0-alpha.2 (#1518) 2026-01-15 17:18:54 +08:00
GatewayJ
e3a7eb2d3d fix: standart policy format (#1508) 2026-01-15 15:33:22 +08:00
majinghe
1e683f12ef fix: change health check statement to fix unhealthy issue for docker … (#1515) 2026-01-15 11:29:45 +08:00
houseme
6a63fba5c2 chore(deps): bump crc-fast, chrono, aws-smithy-types, ssh-key (#1513) 2026-01-15 10:51:14 +08:00
houseme
df502f2ac6 chore(deps): bump multiple dependencies (#1510) 2026-01-15 00:57:04 +08:00
安正超
cb53ee13cd fix: handle copy_source_if_match in copy_object for S3 compatibility (#1408)
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-14 21:09:13 +08:00
Arthur Darcet
6928221b56 In the PVC definition, skip the storageClassName attr if null/empty (#1498)
Signed-off-by: Arthur Darcet <arthur.darcet@mistral.ai>
Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-14 20:18:00 +08:00
houseme
2d58eea702 fix: exclude matching key from ListObjects results when using marker/startAfter (#1506) 2026-01-14 19:21:51 +08:00
houseme
109ca7a100 perf(utils): optimize User-Agent generation and platform detection (#1504) 2026-01-14 18:08:02 +08:00
Jasper Weyne
15e6d4dbd0 feat: add support for existing gateways in helm chart (#1469)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-01-14 17:54:37 +08:00
Jan S
68c5c0b834 Use POSIX statvfs, since statfs is not designed to be portable (#1495) 2026-01-14 16:03:32 +08:00
houseme
27480f7625 Refactor Event Admin Handlers and Parallelize Target Status Probes (#1501) 2026-01-14 14:18:02 +08:00
houseme
f795299d53 Optimization and collation of dependencies introduction processing (#1493) 2026-01-13 15:02:54 +08:00
houseme
650fae71fb Remove the rustfs/console/config.json route (#1487) 2026-01-13 10:15:41 +08:00
houseme
dc76e4472e Fix object tagging functionality issues #1415 (#1485) 2026-01-13 01:11:50 +08:00
houseme
b5140f0098 build(deps): bump tracing-opentelemetry and flate2 version (#1484)
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-12 23:53:31 +08:00
LeonWang0735
5f2e594480 fix:handle null version ID in delete and return version_id in get_object (#1479)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-12 22:02:09 +08:00
houseme
bec51bb783 fix: return 404 for HEAD requests on non-existent objects in TLS (#1480) 2026-01-12 19:30:59 +08:00
houseme
1fad8167af dependency name ignore for object_store (#1481) 2026-01-12 19:13:37 +08:00
weisd
f0da8ce216 fix: avoid unwrap() panic in delete_prefix parsing (#1476)
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-12 13:26:01 +08:00
houseme
f9d3a908f0 Refactor:replace jsonwebtoken feature from rust_crypto to aws_lc_rs (#1474)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-12 12:25:02 +08:00
yxrxy
29d86036b1 feat: implement bucket quota system (#1461)
Signed-off-by: yxrxy <1532529704@qq.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2026-01-12 11:42:07 +08:00
weisd
78b13f3ff2 fix: add delete prefix option support (#1471) 2026-01-12 11:19:09 +08:00
houseme
760cb1d734 Fix Windows Path Separator Handling in rustfs_utils (#1464)
Co-authored-by: reatang <tangtang1251@qq.com>
2026-01-11 19:53:51 +08:00
houseme
6b2eebee1d fix: Remove secret and signature from the log (#1466) 2026-01-11 17:45:16 +08:00
houseme
ddaa9e35ea fix(http): Fix console bucket management functionality failure caused by RUSTFS_SERVER_DOMAINS (#1467) 2026-01-11 16:47:51 +08:00
loverustfs
703d961168 fix: honor bucket policy for authenticated users (#1460)
Co-authored-by: GatewayJ <835269233@qq.com>
2026-01-10 20:01:28 +08:00
loverustfs
e614e530cf Modify ahead images url 2026-01-10 16:12:40 +08:00
loverustfs
00119548d2 Ahead 2026-01-10 16:11:11 +08:00
GatewayJ
d532c7c972 feat: object-list access (#1457)
Signed-off-by: loverustfs <github@rustfs.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: loverustfs <github@rustfs.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-01-10 10:11:08 +08:00
houseme
04f441361e replace winapi to windows crate (#1455) 2026-01-10 02:15:08 +08:00
mkrueger92
9e162b6e9e Default to helm chart version for docker image and not latest (#1385)
Signed-off-by: mkrueger92 <7305571+mkrueger92@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-08 21:16:00 +08:00
majinghe
900f7724b8 add gateway api support due to ingress nginx retirement (#1432)
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-08 20:57:55 +08:00
majinghe
4f5653e656 add upgrade strategy for standalone mode (#1431) 2026-01-08 20:44:16 +08:00
houseme
a95e549430 Fix/fix improve for audit (#1418) 2026-01-07 18:05:52 +08:00
weisd
00f3275603 rm online check (#1416) 2026-01-07 13:42:03 +08:00
weisd
359c9d2d26 Enhance Object Version Management and Replication Status Handling (#1413) 2026-01-07 10:44:35 +08:00
weisd
3ce99939a3 fix: improve memory ordering for disk health tracker (#1412) 2026-01-06 23:59:08 +08:00
Jan S
02f809312b Fix windows missing default backlog (#1405)
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-06 23:41:12 +08:00
GatewayJ
356dc7e0c2 feat: Add permission verification for account creation (#1401)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-01-06 21:47:18 +08:00
安正超
e4ad86ada6 test(s3): add 9 delimiter list tests to implemented tests (#1410) 2026-01-06 21:13:39 +08:00
GatewayJ
b95bee64b2 fix: Correct import permissions (#1402) 2026-01-06 14:53:26 +08:00
Jan S
18fb920fa4 Remove the sysctl crate and use libc's sysctl call interface (#1396)
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-06 10:26:09 +08:00
Jan S
5f19eef945 fix: OpenBSD does not support TCPKeepalive intervals (#1382)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-01-06 00:41:39 +08:00
houseme
40ad2a6ea9 Remove unused crates (#1394) 2026-01-05 23:18:08 +08:00
安正超
e7a3129be4 feat: s3 tests classification (#1392)
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-05 22:24:35 +08:00
weisd
b142563127 fix rpc client (#1393) 2026-01-05 21:52:04 +08:00
weisd
5660208e89 Refactor RPC Authentication System for Improved Maintainability (#1391) 2026-01-05 19:51:51 +08:00
安正超
0b6f3302ce fix: improve s3-tests readiness detection and Python package installation (#1390) 2026-01-05 17:56:42 +08:00
安正超
60103f0f72 fix: s3 api compatibility (#1370)
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-05 16:54:16 +08:00
weisd
ab752458ce Fix Path Traversal and Enhance Object Validation (#1387) 2026-01-05 15:57:15 +08:00
dependabot[bot]
1d6c8750e7 build(deps): bump the dependencies group with 2 updates (#1383)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-05 15:33:57 +08:00
loverustfs
9c44f71a0a Revise security vulnerability reporting instructions
Updated the reporting process for security vulnerabilities.

Signed-off-by: loverustfs <hello@rustfs.com>
2026-01-05 15:05:33 +08:00
loverustfs
9c432fc963 Enhance security policy with philosophy and reporting updates
Added a security philosophy section emphasizing transparency and community contributions. Updated the reporting process for vulnerabilities to ensure responsible disclosure.

Signed-off-by: loverustfs <hello@rustfs.com>
2026-01-05 14:09:48 +08:00
LeonWang0735
f86761fae9 fix:allow NotResource-only policies in statement validation (#1364)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-01-05 13:07:42 +08:00
mkrueger92
377ed507c5 Enable the possibility to freely configure request and limit (#1374)
Signed-off-by: mkrueger92 <7305571+mkrueger92@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-05 09:22:53 +08:00
loverustfs
e063306ac3 Delete the non-existent CLA section.
Delete the non-existent CLA section.

Signed-off-by: loverustfs <hello@rustfs.com>
2026-01-05 07:11:39 +08:00
Dominik Gašparić
8009ad5692 Fix event object structure according to AWS rules (#1379)
Signed-off-by: Dominik Gašparić <56818232+codedoga@users.noreply.github.com>
2026-01-05 01:51:14 +08:00
houseme
fb89a16086 dep: upgrade tokio 1.49.0 (#1378) 2026-01-05 00:07:38 +08:00
Andreas Nussberger
666c0a9a38 helm: add nodeSelector to standalone deployment (#1367)
Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
2026-01-04 20:52:16 +08:00
majinghe
486a4b58e6 add node selector for standalone deployment (#1368) 2026-01-04 20:49:58 +08:00
GatewayJ
f5f6ea4a5c feat:policy Resources support string and array modes. (#1346)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-01-04 19:21:37 +08:00
yxrxy
38c2d74d36 fix: fix FTPS/SFTP download issues and optimize S3Client caching (#1353)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2026-01-04 17:28:18 +08:00
yxrxy
ffbcd3852f fix: fix bucket policy principal parsing to support * and {AWS: *} fo… (#1354)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-01-04 15:53:10 +08:00
houseme
75b144b7d4 Fixing URL output format in IPv6 environments #1343 and Incorrect time in UI #1350 (#1363) 2026-01-04 14:56:54 +08:00
Jan S
d06397cf4a fix: try casting available blocks to a u64 on FreeBSD and OpenBSD (#1360)
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-04 11:06:14 +08:00
Jan S
f995943832 fix: do not hardcode bash path (#1358)
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-04 10:39:59 +08:00
LeonWang0735
de4a3fa766 fix:correct RemoteAddr extension type to enable IP-based policy evaluation (#1356) 2026-01-04 10:13:27 +08:00
loverustfs
4d0045ff18 Add workflow to mark stale issues automatically
Add workflow to mark stale issues automatically

Signed-off-by: loverustfs <hello@rustfs.com>
2026-01-03 11:42:12 +08:00
usernameisnull
d96e04a579 fix: remove nginx-ingress default body size limit (#1335)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
2026-01-02 20:39:16 +08:00
GatewayJ
cc916926ff feat:Permission verification for deleting versions (#1341)
Signed-off-by: GatewayJ <835269233@qq.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-01-02 18:19:34 +08:00
houseme
134e7e237c chore: upgrade GitHub Actions artifact actions (#1339) 2026-01-02 12:29:59 +08:00
yxrxy
cf53a9d84a chore: replace native-tls with pure rustls for FTPS/SFTP e2e tests (#1334)
Signed-off-by: yxrxy <1532529704@qq.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-01-02 11:08:28 +08:00
houseme
8d7cd4cb1b chore: upgrade dependencies and migrate to aws-lc-rs (#1333) 2026-01-02 00:02:34 +08:00
安正超
61b3100260 fix: s3 list object versions next marker (#1328) 2026-01-01 23:26:32 +08:00
0xdx2
b19e8070a2 fix(tagging): fix e2e test_object_tagging failure (#1327) 2026-01-01 17:38:37 +08:00
yxrxy
b8aa8214e2 Feat/ftps&sftp (#1308)
[feat] ftp / sftp
2025-12-31 09:01:15 +08:00
yxrxy
3c14947878 fix(iam): preserve decrypt-failed credentials instead of deleting them (#1312)
Signed-off-by: loverustfs <github@rustfs.com>
Co-authored-by: loverustfs <github@rustfs.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-30 22:41:10 +08:00
houseme
2924b4e463 Restore globals and add unified TLS/mTLS loading from RUSTFS_TLS_PATH (#1309)
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
2025-12-30 21:55:43 +08:00
loverustfs
b4ba62fa33 fix: correctly handle aws:SourceIp in policy evaluation (#1301) (#1306)
Signed-off-by: loverustfs <github@rustfs.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-30 16:54:48 +08:00
loverustfs
a5b3522880 Add trendshift 2025-12-30 13:03:15 +08:00
安正超
056a0ee62b feat: add local s3-tests script with configurable options and improvements (#1300) 2025-12-29 23:48:32 +08:00
Juri Malinovski
4603ece708 helm: add enableServiceLinks, poddisruptionbudget (#1293)
Signed-off-by: Juri Malinovski <juri.malinovski@coolbet.com>
2025-12-29 09:31:18 +08:00
houseme
eb33e82b56 fix: Prevent panic in GetMetrics gRPC handler on invalid input (#1291)
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
2025-12-29 03:10:23 +08:00
Ali Mehraji
c7e2b4d8e7 Modular Makefile (#1288)
Signed-off-by: Ali Mehraji <a.mehraji75@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-12-28 21:57:44 +08:00
LeonWang0735
71c59d1187 fix:ListObjects and ListObjectV2 correctly handles unordered and delimiter (#1285) 2025-12-28 16:18:42 +08:00
loverustfs
e3a0a07495 fix: ensure version_id is returned in S3 response headers (#1272)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2025-12-28 09:41:32 +08:00
0xdx2
136db7e0c9 feat: add function to extract user-defined metadata keys and integrat… (#1281)
Signed-off-by: 0xdx2 <xuedamon2@gmail.com>
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-12-27 22:18:16 +08:00
Juri Malinovski
2e3c5f695a helm: update default Chart.yaml, appVersion version bump, add appVersion as a default image tag (#1247)
Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
2025-12-27 20:50:22 +08:00
bbb4aaa
fe9609fd17 fix:affinity.podAntiAffinity.enabled value not taking effect (#1280)
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-27 20:46:25 +08:00
bbb4aaa
f2d79b485e fix: prevent PV/PVC deletion during rustfs uninstallation (#1279) 2025-12-27 20:45:43 +08:00
Copilot
3d6681c9e5 chore: remove e2e-mint workflow (#1274)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>
2025-12-26 21:55:04 +08:00
lgpseu
07a26fadad opt: store IoLoadMetrics records with circular vector (#1265)
Co-authored-by: houseme <housemecn@gmail.com>
2025-12-26 12:59:40 +08:00
majinghe
a083fca17a delete -R parameter in init container step (#1264) 2025-12-25 18:14:50 +08:00
houseme
89c3ae77a4 feat: Add TONIC_PREFIX prefix matching in ReadinessGateService (#1261) 2025-12-25 14:28:07 +08:00
houseme
82a6e78845 Inject GlobalReadiness into HTTP server pipeline and gate traffic until FullReady (#1255) 2025-12-25 00:19:03 +08:00
houseme
7e75c9b1f5 remove unlinked file (#1258) 2025-12-24 23:37:43 +08:00
weisd
8bdff3fbcb fix: Add retry mechanism for GLOBAL_CONFIG_SYS initialization (#1252) 2025-12-24 16:38:28 +08:00
Andrea Manzi
65d32e693f add ca-certificates in mcp-server Dockerfile (#1248)
Signed-off-by: Andrea Manzi <andrea.manzi@gmail.com>
2025-12-24 08:36:14 +08:00
Michele Zanotti
1ff28b3157 helm: expose init container parameters as helm values (#1232)
Co-authored-by: houseme <housemecn@gmail.com>
2025-12-23 21:31:28 +08:00
Juri Malinovski
2186f46ea3 helm: fix service/containers ports, fix podAntiAffinity (#1230)
Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
2025-12-23 20:36:33 +08:00
唐小鸭
add6453aea feat: add seek support for small objects in rustfs (#1231)
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-23 20:27:34 +08:00
yxrxy
4418c882ad Revert "fix(iam): store previous credentials in .rustfs.sys bucket to… (#1238)
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-23 19:37:39 +08:00
Muhammed Hussain Karimi
00c607b5ce 🧑‍💻 Fix nix develop problem with Git-Based dependecies on nix develop shell (#1243)
Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev>
2025-12-23 19:26:50 +08:00
majinghe
79585f98e0 delete userless helm chart file (#1245) 2025-12-23 19:15:29 +08:00
majinghe
2a3517f1d5 Custom annotation (#1242) 2025-12-23 17:31:01 +08:00
tryao
3942e07487 console port is 9001 (#1235)
Signed-off-by: tryao <yaotairan@gmail.com>
2025-12-23 13:36:38 +08:00
houseme
04811c0006 update s3s version (#1237) 2025-12-23 13:09:57 +08:00
Ali Mehraji
73c15d6be1 Add: rust installation in Makefile (#1188)
Signed-off-by: Ali Mehraji <a.mehraji75@gmail.com>
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-23 08:51:04 +08:00
loverustfs
af5c0b13ef fix: HeadObject returns 404 for deleted objects with versioning enabled (#1229)
Co-authored-by: houseme <housemecn@gmail.com>
2025-12-22 20:43:00 +08:00
Juri Malinovski
f17990f746 helm: allow to define additional config variables (#1220)
Signed-off-by: Juri Malinovski <juri.malinovski@coolbet.com>
2025-12-22 20:25:23 +08:00
weisd
80cfb4feab Add Disk Timeout and Health Check Functionality (#1196)
Signed-off-by: weisd <im@weisd.in>
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-22 17:15:19 +08:00
houseme
08f1a31f3f Fix notification event stream cleanup, add bounded send concurrency, and reduce overhead (#1224) 2025-12-22 00:57:05 +08:00
loverustfs
1c51e204ab ci: reduce cargo build jobs to 2 for standard-2 runner 2025-12-21 23:54:40 +08:00
loverustfs
958f054123 ci: update all workflows to use ubicloud-standard-2 runner 2025-12-21 23:43:12 +08:00
0xdx2
3e2252e4bb fix(config):Update argument parsing for volumes and server_domains to support del… (#1209)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-21 17:54:23 +08:00
loverustfs
f3a1431fa5 fix: resolve TLS handshake failure in inter-node communication (#1201) (#1222)
Co-authored-by: houseme <housemecn@gmail.com>
2025-12-21 16:11:55 +08:00
yxrxy
3bd96bcf10 fix: resolve event target deletion issue (#1219) 2025-12-21 12:43:48 +08:00
majinghe
20ea591049 add custom nodeport support (#1217) 2025-12-20 22:02:21 +08:00
GatewayJ
cc31e88c91 fix: expiration time (#1215) 2025-12-20 20:25:52 +08:00
yxrxy
b5535083de fix(iam): store previous credentials in .rustfs.sys bucket to preserv… (#1213) 2025-12-20 19:15:49 +08:00
loverustfs
1e35edf079 chore(ci): restore workflows before 8e0aeb4 (#1212) 2025-12-20 07:50:49 +08:00
Copilot
8dd3e8b534 fix: decode form-urlencoded object names in webhook/mqtt Key field (#1210)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-12-20 01:31:09 +08:00
loverustfs
8e0aeb4fdc Optimize ci ubicloud (#1208) 2025-12-19 23:22:45 +08:00
majinghe
abe8a50b5a add cert manager and ingress annotations support (#1206) 2025-12-19 21:50:23 +08:00
loverustfs
61f4d307b5 Modify latest version tips to console 2025-12-19 14:57:19 +08:00
loverustfs
3eafeb0ff0 Modify to accelerate 2025-12-19 13:01:17 +08:00
houseme
4abfc9f554 Fix/fix event 1216 (#1191)
Signed-off-by: loverustfs <hello@rustfs.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-19 12:07:07 +08:00
唐小鸭
1057953052 fix: Remove the compression check that has already been handled by tower-http::CompressionLayer. (#1190)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-19 10:15:52 +08:00
loverustfs
889c67f359 Modify to ubicloud 2025-12-19 09:42:21 +08:00
loverustfs
1d111464f9 Return to GitHub hosting
Return to GitHub hosting

Signed-off-by: loverustfs <hello@rustfs.com>
2025-12-19 09:15:26 +08:00
loverustfs
a0b2f5a232 self-host
self-host

Signed-off-by: loverustfs <hello@rustfs.com>
2025-12-18 22:23:25 +08:00
Muhammed Hussain Karimi
46557cddd1 🧑‍💻 Improve shebang compatibility (#1180)
Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev>
2025-12-18 20:13:24 +08:00
安正超
443947e1ac fix: improve S3 API compatibility for ListObjects operations (#1173)
Signed-off-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-17 21:50:03 +08:00
yxrxy
8821fcc1e7 feat: Replace LRU cache with Moka async cache in policy variables (#1166)
Co-authored-by: houseme <housemecn@gmail.com>
2025-12-17 00:19:31 +08:00
houseme
17828ec2a8 Dependabot/cargo/s3s df2434d 1216 (#1170)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-12-16 21:21:43 +08:00
mythrnr
94d5b1c1e4 fix: format of bucket event notifications (#1138) 2025-12-16 20:44:57 +08:00
GatewayJ
0bca1fbd56 fix: the method for correcting judgment headers (#1159)
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-16 19:30:50 +08:00
唐小鸭
52c2d15a4b feat: Implement whitelist-based HTTP response compression configuration (#1136)
Signed-off-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-16 15:05:40 +08:00
yxrxy
352035a06f feat: Implement AWS policy variables support (#1131)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-16 13:32:01 +08:00
yihong
fe4fabb195 fix: other two memory leak in the code base (#1160)
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-12-16 11:45:45 +08:00
GatewayJ
07c5e7997a list object version Interface returns storage_class (#1133)
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-16 07:09:05 +08:00
yihong
0007b541cd feat: add pre-commit file (#1155)
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2025-12-15 22:23:43 +08:00
dependabot[bot]
0f2e4d124c build(deps): bump the dependencies group with 3 updates (#1148)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-15 20:39:04 +08:00
Christian Simon
2e4ce6921b helm: Mount /tmp as emptyDir (#1105)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-15 16:59:28 +08:00
Juri Malinovski
7178a94792 helm: refactor helm chart (#1122)
Signed-off-by: Juri Malinovski <juri.malinovski@coolbet.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-15 13:05:43 +08:00
sunfkny
e8fe9731fd Fix memory leak in Cache update method (#1143) 2025-12-15 10:04:14 +08:00
Jörg Thalheim
3ba415740e Add docs for using Nix flake (#1103)
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: 0xdx2 <xuedamon2@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-12-14 09:44:13 +08:00
Lazar
aeccd14d99 Replace placeholder content in SECURITY.md (#1140)
Signed-off-by: Lazar <66002359+WauHundeland@users.noreply.github.com>
2025-12-14 09:31:27 +08:00
Jörg Thalheim
89a155a35d flake: add Nix flake for reproducible builds (#1096)
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: 0xdx2 <xuedamon2@gmail.com>
2025-12-13 23:54:54 +08:00
yihong
67095c05f9 fix: update tool chain make everything happy (#1134)
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2025-12-13 20:32:42 +08:00
czaloumis
1229fddb5d render imagePullSecrets in Deployment/StatefulSet (#1130)
Signed-off-by: czaloumis <80974398+czaloumis@users.noreply.github.com>
2025-12-13 11:23:35 +08:00
majinghe
08be8f5472 add image pull secret support (#1127)
Co-authored-by: houseme <housemecn@gmail.com>
2025-12-12 20:25:25 +08:00
Sebastian Wolf
0bf25fdefa feat: Be able to set region from Helm chart (#1119)
Co-authored-by: houseme <housemecn@gmail.com>
2025-12-12 12:30:35 +08:00
houseme
9e2fa148ee Fix type errors in ecfs.rs and apply clippy fixes for Rust 1.92.0 (#1121) 2025-12-12 00:49:21 +08:00
安正超
cb3e496b17 Feat/e2e s3tests (#1120)
Signed-off-by: 安正超 <anzhengchao@gmail.com>
2025-12-11 22:32:07 +08:00
YGoetschel
997f54e700 Fix Docker-based Development Workflow (#1031)
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-12-11 19:48:14 +08:00
houseme
1a4e95e940 chore: remove unused dependencies to optimize build (#1117) 2025-12-11 18:13:26 +08:00
Christian Simon
a3006ab407 helm: Use service.type from Values (#1106)
Co-authored-by: houseme <housemecn@gmail.com>
2025-12-11 17:32:15 +08:00
houseme
e197486c8c upgrade action checkout version from v5 to v6 (#1067)
Co-authored-by: 0xdx2 <xuedamon2@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-11 15:39:20 +08:00
dependabot[bot]
0da943a6a4 build(deps): bump s3s from 0.12.0-rc.4 to 0.12.0-rc.5 in the s3s group (#1046)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
2025-12-11 15:20:36 +08:00
guojidan
fba201df3d fix: harden data usage aggregation and cache handling (#1102)
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-11 09:55:25 +08:00
yxrxy
ccbab3232b fix: ListObjectsV2 correctly handles repeated folder names in prefixes (#1104)
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-11 09:38:52 +08:00
loverustfs
421f66ea18 Disable codeql 2025-12-11 09:29:46 +08:00
yxrxy
ede2fa9d0b fix: is-admin api (For STS/temporary credentials, we need to check the… (#1101)
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-11 08:55:41 +08:00
tennisleng
978845b555 fix(lifecycle): Fix ObjectInfo fields and mod_time error handling (#1088)
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-11 07:17:35 +08:00
Jacob
53c126d678 fix: decode percent-encoded paths in get_file_path() (#1072)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-10 22:30:02 +08:00
0xdx2
9f12a7678c feat(ci): add codeql to scanner code (#1076) 2025-12-10 21:48:18 +08:00
Jörg Thalheim
2c86fe30ec Content encoding (#1089)
Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-10 15:21:51 +08:00
tennisleng
ac0c34e734 fix(lifecycle): Return NoSuchLifecycleConfiguration error for missing lifecycle config (#1087)
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-10 12:35:22 +08:00
majinghe
ae46ea4bd3 fix github action security found by github CodeQL (#1091) 2025-12-10 12:07:28 +08:00
majinghe
8b3d4ea59b enhancement logs output for container deployment (#1090) 2025-12-10 11:14:05 +08:00
houseme
ef261deef6 improve code for is admin (#1082) 2025-12-09 17:34:47 +08:00
Copilot
20961d7c91 Add comprehensive special character handling with validation refactoring and extensive test coverage (#1078)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-12-09 13:40:29 +08:00
shiro.lee
8de8172833 fix: the If-None-Match error handling in the complete_multipart_uploa… (#1065)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-08 23:10:20 +08:00
orbisai0security
7c98c62d60 [Security] Fix HIGH vulnerability: yaml.docker-compose.security.writable-filesystem-service.writable-filesystem-service (#1005)
Co-authored-by: orbisai0security <orbisai0security@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-12-08 22:05:10 +08:00
Ali Mehraji
15c75b9d36 simple deployment via docker-compose (#1043)
Signed-off-by: Ali Mehraji <a.mehraji75@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-12-08 21:25:11 +08:00
yxrxy
af650716da feat: add is-admin user api (#1063) 2025-12-08 21:15:04 +08:00
shiro.lee
552e95e368 fix: the If-None-Match error handling in the put_object method when t… (#1034)
Co-authored-by: 0xdx2 <xuedamon2@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-08 15:36:31 +08:00
dependabot[bot]
619cc69512 build(deps): bump the dependencies group with 3 updates (#1052)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-12-08 14:31:53 +08:00
Jitter
76d25d9a20 Fix/issue #1001 dead node detection (#1054)
Co-authored-by: weisd <im@weisd.in>
Co-authored-by: Jitterx69 <mohit@example.com>
2025-12-08 12:29:46 +08:00
yihong
834025d9e3 docs: fix some dead link (#1053)
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2025-12-08 11:23:24 +08:00
houseme
e2d8e9e3d3 Feature/improve profiling (#1038)
Co-authored-by: Jitter <jitterx69@gmail.com>
Co-authored-by: weisd <im@weisd.in>
2025-12-07 22:39:47 +08:00
Jitter
cd6a26bc3a fix(net): resolve 1GB upload hang and macos build (Issue #1001 regression) (#1035) 2025-12-07 18:05:51 +08:00
tennisleng
5f256249f4 fix: correct ARN parsing for notification targets (#1010)
Co-authored-by: Andrew Leng <work@Andrews-MacBook-Air.local>
Co-authored-by: houseme <housemecn@gmail.com>
2025-12-06 23:12:58 +08:00
Jitter
b10d80cbb6 fix: detect dead nodes via HTTP/2 keepalives (Issue #1001) (#1025)
Co-authored-by: weisd <im@weisd.in>
2025-12-06 21:45:42 +08:00
0xdx2
7c6cbaf837 feat: enhance error handling and add precondition checks for object o… (#1008) 2025-12-06 20:39:03 +08:00
Hunter Wu
72930b1e30 security: Fix timing attack vulnerability in credential comparison (#1014)
Co-authored-by: Copilot AI <copilot@github.com>
2025-12-06 15:13:27 +08:00
LemonDouble
6ca8945ca7 feat(helm): split storageSize into data and log storage parameters (#1018) 2025-12-06 14:01:49 +08:00
majinghe
0d0edc22be update helm package ci file and helm values file (#1004) 2025-12-05 22:13:00 +08:00
weisd
030d3c9426 fix filemeta nil versionid (#1002) 2025-12-05 20:30:08 +08:00
majinghe
b8b905be86 add helm package ci file (#994) 2025-12-05 15:09:53 +08:00
Damien Degois
ace58fea0d feat(helm): add existingSecret handling and support for extra manifests (#992) 2025-12-05 14:14:59 +08:00
唐小鸭
3a79242133 feat: The observability module can be set separately. (#993) 2025-12-05 13:46:06 +08:00
Andrew Steurer
63d846ed14 Fix link to CONTRIBUTING.md in README (#991) 2025-12-05 09:23:26 +08:00
shiro.lee
3a79fcfe73 fix: add the is_truncated field to the return of the list_object_vers… (#985) 2025-12-04 22:26:31 +08:00
weisd
b3c80ae362 fix: listdir rpc (#979)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-04 16:12:10 +08:00
Hey Martin
3fd003b21d Delete duplicate titles in the README (#977) 2025-12-04 15:55:26 +08:00
houseme
1d3f622922 console: add version_handler and improve comments (#975) 2025-12-04 13:41:06 +08:00
loverustfs
e31b4303ed fix link error 2025-12-04 08:26:41 +08:00
houseme
5b0a3a0764 upgrade crate version and improve heal config (#963) 2025-12-03 18:49:11 +08:00
weisd
a8b7b28fd0 Fix Admin Heal API and Add Pagination Support for Large Buckets (#933)
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-12-03 18:10:46 +08:00
loverustfs
e355d3db80 Modify readme 2025-12-03 17:18:53 +08:00
weisd
4d7bf98c82 add logs (#962) 2025-12-03 13:17:47 +08:00
shiro.lee
699164e05e fix: add the is_truncated field to the return of the list_objects int… (#958) 2025-12-03 03:14:17 +08:00
dependabot[bot]
d35ceac441 build(deps): bump criterion in the dependencies group (#947)
Bumps the dependencies group with 1 update: [criterion](https://github.com/criterion-rs/criterion.rs).


Updates `criterion` from 0.7.0 to 0.8.0
- [Release notes](https://github.com/criterion-rs/criterion.rs/releases)
- [Changelog](https://github.com/criterion-rs/criterion.rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/criterion-rs/criterion.rs/compare/criterion-plot-v0.7.0...criterion-v0.8.0)

---
updated-dependencies:
- dependency-name: criterion
  dependency-version: 0.8.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-12-02 00:16:28 +08:00
houseme
93982227ac Improve health check handlers for endpoint and console (GET/HEAD, safer error handling) (#942)
* Improve health check handlers for endpoint and console

- Add unified GET/HEAD handling for `/health` and `/rustfs/console/health`
- Implement proper method filtering and 405 with `Allow: GET, HEAD`
- Avoid panics by removing `unwrap()` in health check logic
- Add safe fallbacks for JSON serialization and uptime calculation
- Ensure HEAD requests return only status and headers (empty body)
- Keep response format backward compatible for monitoring systems

* fix
2025-11-30 02:43:59 +08:00
Copilot
fdcdb30d28 Optimize concurrent GetObject performance with Moka cache, comprehensive metrics, complete test suite, cache writeback, and comprehensive documentation (#916)
* Initial plan

* feat: add concurrency-aware buffer sizing and hot object caching for GetObject

- Implement adaptive buffer sizing based on concurrent request load
- Add per-request tracking with automatic cleanup using RAII guards
- Implement hot object cache (LRU) for frequently accessed small files (<= 10MB)
- Add disk I/O semaphore to prevent saturation under extreme load
- Integrate concurrency module into GetObject implementation
- Buffer sizes now adapt: low concurrency uses large buffers for throughput,
  high concurrency uses smaller buffers for fairness and memory efficiency
- Add comprehensive metrics collection for monitoring performance

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* docs: add comprehensive documentation and tests for concurrent GetObject optimization

- Add detailed technical documentation explaining the solution
- Document root cause analysis and solution architecture
- Include performance expectations and testing recommendations
- Add integration tests for concurrency tracking and buffer sizing
- Add cache behavior tests
- Include benchmark tests for concurrent request handling

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix: address code review issues in concurrency module

- Fix race condition in cache size tracking by using consistent atomic operations within lock
- Correct buffer sizing logic: 1-2 requests use 100%, 3-4 use 75%, 5-8 use 50%, >8 use 40%
- Improve error message for semaphore acquire failure
- Document limitation of streaming cache implementation (not yet implemented)
- Add TODO for proper streaming cache with suggested approaches
- Update tests to match corrected buffer sizing thresholds

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* docs: add comprehensive implementation summary for concurrent GetObject optimization

- Executive summary of problem and solution
- Detailed architecture documentation for each optimization
- Integration points and code examples
- Complete testing strategy and performance validation steps
- Monitoring and observability guidelines with Prometheus queries
- Deployment considerations and rollout strategy
- Future enhancement roadmap
- Success criteria and quantitative metrics

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* fix

* refactor: optimize cache with lru 0.16.2 read-first pattern and add advanced features

- Implement optimized read-first cache access using peek() to reduce write lock contention
- Add batch cache operations: get_cached_batch() for efficient multi-object retrieval
- Add cache utility methods: is_cached(), remove_cached(), get_hot_keys()
- Implement warm_cache() for pre-populating cache on startup
- Add get_advanced_buffer_size() with file size and sequential read optimization
- Enhance test suite with 8 new comprehensive tests covering:
  - Batch operations and cache warming
  - Hot keys tracking and analysis
  - Cache removal and LRU behavior verification
  - Concurrent cache access performance
  - Advanced buffer sizing strategies
- Improve documentation and code comments in English throughout

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* docs: add final comprehensive optimization summary

- Complete technical documentation of all optimizations
- Detailed performance analysis and metrics
- Production deployment guide with examples
- Comprehensive API reference and usage patterns
- Migration guide and future enhancement roadmap
- All documentation in professional English

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* fix

* add moka crate for cache

* feat: replace LRU with Moka cache and add comprehensive metrics

- Replace lru crate with moka 0.12.11 for superior concurrent performance
- Implement lock-free cache with automatic TTL/TTI expiration
- Add size-based eviction using Moka's weigher function
- Integrate comprehensive metrics collection throughout GetObject flow:
  * Cache hit/miss tracking with per-key access counts
  * Request concurrency gauges
  * Disk permit wait time histograms
  * Total request duration tracking
  * Response size and buffer size histograms
- Deep integration with ecfs.rs GetObject operation
- Add hit rate calculation method
- Enhanced CacheStats with hit/miss counters
- Lock-free concurrent reads for better scalability

Moka advantages over LRU:
- True lock-free concurrent access
- Built-in TTL and TTI support
- Automatic size-based eviction
- Better performance under high concurrency
- Native async support

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* docs: add comprehensive Moka cache migration and metrics documentation

- Complete technical documentation of LRU to Moka migration
- Detailed performance comparison and benchmarks
- Comprehensive metrics catalog with 15+ Prometheus metrics
- Prometheus query examples for monitoring
- Dashboard and alerting recommendations
- Migration guide with code examples
- Troubleshooting guide for common issues
- Future enhancement roadmap

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* fix

* refactor: update tests for Moka cache implementation

- Completely refactor test suite to align with Moka-based concurrency.rs
- Add Clone derive to ConcurrencyManager for test convenience
- Update all tests to handle Moka's async behavior with proper delays
- Add new tests:
  * test_cache_hit_rate - validate hit rate calculation
  * test_ttl_expiration - verify TTL configuration
  * test_is_cached_no_side_effects - ensure contains doesn't affect LRU
  * bench_concurrent_cache_performance - benchmark concurrent access
- Updated existing tests:
  * test_moka_cache_operations - renamed and updated for Moka API
  * test_moka_cache_eviction - validate automatic eviction
  * test_hot_keys_tracking - improved assertions for sorted results
  * test_concurrent_cache_access - validate lock-free performance
- All tests now include appropriate sleep delays for Moka's async processing
- Enhanced documentation and assertions for better test clarity
- Total: 18 comprehensive integration tests

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* docs: add comprehensive Moka test suite documentation

- Complete test suite documentation for all 18 tests
- Detailed test patterns and best practices for Moka
- Debugging guide for common test failures
- Performance expectations and CI integration
- 100% API coverage validation
- Future enhancement roadmap

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* fmt

* docs: enhance code comments and test documentation throughout

- Significantly improve module-level documentation in concurrency.rs
- Add detailed explanations of performance characteristics and expected improvements
- Document the solution to issue #911 with before/after metrics
- Enhance constant documentation with purpose and impact explanations
- Add comprehensive RAII guard documentation with thread safety notes
- Improve test documentation with detailed test strategy and rationale
- Add Moka-specific test patterns and timing guidance
- Include performance expectations for each test category
- Document why each test matters for solving the original issue
- All documentation written in professional English
- Follow Rust documentation best practices with examples

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* remove lru crate

* upgrade version

* fix: resolve test errors by correcting module structure and test assertions

- Fix test import paths to use crate:: instead of rustfs:: (binary-only crate)
- Keep test file in src/storage/ instead of tests/ (no lib.rs exists)
- Add #[cfg(test)] guard to mod declaration in storage/mod.rs
- Fix Arc type annotations for Moka's ConcurrencyManager in concurrent tests
- Correct test_buffer_size_bounds assertions to match actual implementation:
  * Minimum buffer is 32KB for files <100KB, 64KB otherwise
  * Maximum buffer respects base_buffer_size when concurrency is low
  * Buffer sizing doesn't cap at file size, only at min/max constraints
- All 17 integration tests now pass successfully

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix: modify `TimeoutLayer::new` to `TimeoutLayer::with_status_code` and improve docker health check

* fix

* feat: implement cache writeback for small objects in GetObject

- Add cache writeback logic for objects meeting caching criteria:
  * No range/part request (full object retrieval)
  * Object size known and <= 10MB (max_object_size threshold)
  * Not encrypted (SSE-C or managed encryption)
- Read eligible objects into memory and cache via background task
- Serve response from in-memory data for immediate client response
- Add metrics counter for cache writeback operations
- Add 3 new tests for cache writeback functionality:
  * test_cache_writeback_flow - validates round-trip caching
  * test_cache_writeback_size_limit - ensures large objects aren't cached
  * test_cache_writeback_concurrent - validates thread-safe concurrent writes
- Update test suite documentation (now 20 comprehensive tests)

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* improve code for const

* cargo clippy

* feat: add cache enable/disable configuration via environment variable

- Add is_cache_enabled() method to ConcurrencyManager
- Read RUSTFS_OBJECT_CACHE_ENABLE env var (default: false) at startup
- Update ecfs.rs to check is_cache_enabled() before cache lookup and writeback
- Cache lookup and writeback now respect the enable flag
- Add test_cache_enable_configuration test
- Constants already exist in rustfs_config:
  * ENV_OBJECT_CACHE_ENABLE = "RUSTFS_OBJECT_CACHE_ENABLE"
  * DEFAULT_OBJECT_CACHE_ENABLE = false
- Total: 21 comprehensive tests passing

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* fmt

* fix

* fix

* feat: implement comprehensive CachedGetObject response cache with metadata

- Add CachedGetObject struct with full response metadata fields:
  * body, content_length, content_type, e_tag, last_modified
  * expires, cache_control, content_disposition, content_encoding
  * storage_class, version_id, delete_marker, tag_count, etc.
- Add dual cache architecture in HotObjectCache:
  * Legacy simple byte cache for backward compatibility
  * New response cache for complete GetObject responses
- Add ConcurrencyManager methods for response caching:
  * get_cached_object() - retrieve cached response with metadata
  * put_cached_object() - store complete response
  * invalidate_cache() - invalidate on write operations
  * invalidate_cache_versioned() - invalidate both version and latest
  * make_cache_key() - generate cache keys with version support
  * max_object_size() - get cache threshold
- Add builder pattern for CachedGetObject construction
- Add 6 new tests for response cache functionality (27 total):
  * test_cached_get_object_basic - basic operations
  * test_cached_get_object_versioned - version key handling
  * test_cache_invalidation - write operation invalidation
  * test_cache_invalidation_versioned - versioned invalidation
  * test_cached_get_object_size_limit - size enforcement
  * test_max_object_size - threshold accessor

All 27 tests pass successfully.

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: integrate CachedGetObject cache in ecfs.rs with full metadata and cache invalidation

Integration of CachedGetObject response cache in ecfs.rs:
1. get_object: Cache lookup uses get_cached_object() with full metadata
   - Returns complete response with e_tag, last_modified, content_type, etc.
   - Parses last_modified from RFC3339 string
   - Supports versioned cache keys via make_cache_key()

2. get_object: Cache writeback uses put_cached_object() with metadata
   - Stores content_type, e_tag, last_modified in CachedGetObject
   - Background writeback via tokio::spawn()

3. Cache invalidation added to write operations:
   - put_object: invalidate_cache_versioned() after store.put_object()
   - put_object_extract: invalidate_cache_versioned() after each file extraction
   - copy_object: invalidate_cache_versioned() after store.copy_object()
   - delete_object: invalidate_cache_versioned() after store.delete_object()
   - delete_objects: invalidate_cache_versioned() for each deleted object
   - complete_multipart_upload: invalidate_cache_versioned() after completion

4. Fixed test_adaptive_buffer_sizing to be more robust with parallel tests

All 27 tests pass.

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix: add error logging for time parsing in cache lookup and writeback

- Add warning log when RFC3339 parsing fails in cache lookup
- Add warning log when time formatting fails in cache writeback
- Improves debugging for cache-related issues

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* fix

* upgrade version

* fmt

* add http: 4138 and add max_object_size key

* fix

* fix

* fix

* modify metrics key

* add

* upgrade crates version and improve docker observability

* feat: implement adaptive I/O strategy based on disk permit wait time

Implements adaptive I/O strategy for GetObject operations based on disk permit
wait times, as requested in issue #911.

Key changes:

1. New types in concurrency.rs:
   - IoLoadLevel enum: Low (<10ms), Medium (10-50ms), High (50-200ms), Critical (>200ms)
   - IoStrategy struct: contains optimized I/O parameters based on load level
   - IoLoadMetrics: rolling window metrics for load tracking

2. New ConcurrencyManager methods:
   - calculate_io_strategy(): calculates adaptive strategy from permit wait duration
   - record_permit_wait(): records observation for load tracking
   - smoothed_load_level(): returns averaged load level for stability
   - io_load_stats(): returns (avg_wait, p95_wait, max_wait, count) for monitoring
   - adaptive_buffer_size(): convenience method for buffer sizing

3. Integration in ecfs.rs get_object:
   - Calculate IoStrategy after acquiring disk permit
   - Use strategy buffer_size instead of static sizing
   - Consider strategy.cache_writeback_enabled in cache decision
   - Record new metrics: io.load.level gauge, io.buffer.multiplier gauge,
     io.strategy.selected counter by load level

4. New tests (4 added, 31 total):
   - test_io_load_level_classification: validates load level thresholds
   - test_io_strategy_buffer_sizing: validates buffer multipliers
   - test_calculate_io_strategy: validates manager strategy calculation
   - test_io_load_stats: validates stats tracking

All 31 tests pass. Clippy clean. Formatted.

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* fix

* docs: add comprehensive architecture design and fix cache hit event notification

1. Added CONCURRENCY_ARCHITECTURE.md with complete design documentation:
   - Executive summary and problem statement
   - Architecture overview with request flow diagram
   - Detailed module analysis for concurrency.rs and ecfs.rs
   - Critical analysis of helper.complete() for cache hits
   - Adaptive I/O strategy design with algorithm
   - Cache architecture with CachedGetObject structure
   - Metrics and monitoring with Prometheus queries
   - Performance characteristics and future enhancements

2. Fixed critical issue: Cache hit path now calls helper.complete()
   - S3 bucket notifications (s3:GetObject events) now trigger for cache hits
   - Event-driven workflows (Lambda, SNS) work correctly for all object access
   - Maintains audit trail for both cache hits and misses

All 31 tests pass.

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix: set object info and version_id on helper before complete() for cache hits

When serving from cache, properly configure the OperationHelper before
calling complete() to ensure S3 bucket notifications include complete
object metadata:

1. Build ObjectInfo from cached metadata:
   - bucket, name, size, actual_size
   - etag, mod_time, version_id, delete_marker
   - storage_class, content_type, content_encoding
   - user_metadata (user_defined)

2. Set helper.object(event_info).version_id(version_id_str) before complete()

3. Updated CONCURRENCY_ARCHITECTURE.md with:
   - Complete code example for cache hit event notification
   - Explanation of why ObjectInfo is required
   - Documentation of version_id handling

This ensures:
- Lambda triggers receive proper object metadata for cache hits
- SNS/SQS notifications include complete information
- Audit logs contain accurate object details
- Version-specific event routing works correctly

All 31 tests pass.

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* improve code

* fmt

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-11-30 01:16:55 +08:00
Serhiy Novoseletskiy
a6cf0740cb Updated RUSTFS_VOLUMES (#922)
1. Removed .rustfs.svc.cluster.local as all pods for statefulset are running in the same namespace
2. used "rustfs.fullname" as it's used in statefulset services and statefull set names

Co-authored-by: houseme <housemecn@gmail.com>
2025-11-29 23:50:18 +08:00
loverustfs
a2e3a719d3 Improve reading experience 2025-11-28 16:03:41 +08:00
loverustfs
76efee37fa fix error 2025-11-28 15:23:26 +08:00
loverustfs
fd7c0964a0 Modify Readme 2025-11-28 15:16:59 +08:00
唐小鸭
701960dd81 fix out of range for slice (#931) 2025-11-27 15:57:38 +08:00
Shyim
ee04cc77a0 remove debug (#912)
* remove debug

* Refactor get_global_encryption_service function

* Refactor get_global_encryption_service function

---------

Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-11-26 11:56:01 +08:00
houseme
069194f553 Fix/getobjectlength (#920)
* fix getobject content length resp

* Fix regression in exception handling for non-existent key with enhanced compression predicate and metadata improvements (#915)

* Initial plan

* Fix GetObject regression by excluding error responses from compression

The issue was that CompressionLayer was attempting to compress error responses,
which could cause Content-Length header mismatches. By excluding 4xx and 5xx
responses from compression, we ensure error responses (like NoSuchKey) are sent
correctly without body truncation.

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Enhance NoSuchKey fix with improved compression predicate and comprehensive tests

- Enhanced ShouldCompress predicate with size-based exclusion (< 256 bytes)
- Added detailed documentation explaining the compression logic
- Added debug logging for better observability
- Created comprehensive test suite with 4 test cases:
  - test_get_deleted_object_returns_nosuchkey
  - test_head_deleted_object_returns_nosuchkey
  - test_get_nonexistent_object_returns_nosuchkey
  - test_multiple_gets_deleted_object
- Added extensive inline documentation and comments
- Created docs/fix-nosuchkey-regression.md with full analysis

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Add compression best practices documentation

Added comprehensive guide covering:
- Best practices for HTTP response compression
- Common pitfalls and solutions
- Performance considerations and trade-offs
- Testing guidelines and examples
- Monitoring and alerting recommendations
- Migration guide for existing services

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* fmt

* fmt

* Fix/objectdelete (#917)

* fix getobject content length resp

* fix delete object

---------

Co-authored-by: houseme <housemecn@gmail.com>

* Add comprehensive analysis of NoSuchKey fix and related improvements

Created detailed documentation analyzing:
- HTTP compression layer fix (primary issue)
- Content-length calculation fix from PR #917
- Delete object metadata fixes from PR #917
- How all components work together
- Complete scenario walkthrough
- Performance impact analysis
- Testing strategy and deployment checklist

This ties together all the changes in the PR branch including the merged
improvements from PR #917.

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* replace `once_cell` to `std`

* fmt

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: weisd <im@weisd.in>

* fmt

---------

Co-authored-by: weisd <weishidavip@163.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: weisd <im@weisd.in>
2025-11-24 18:56:34 +08:00
weisd
fce4e64da4 Fix/objectdelete (#917)
* fix getobject content length resp

* fix delete object

---------

Co-authored-by: houseme <housemecn@gmail.com>
2025-11-24 16:35:51 +08:00
houseme
44bdebe6e9 build(deps): bump the dependencies group with 10 updates (#914)
* build(deps): bump the dependencies group with 10 updates

* build(deps): bump the dependencies group with 8 updates (#913)

Bumps the dependencies group with 8 updates:

| Package | From | To |
| --- | --- | --- |
| [bytesize](https://github.com/bytesize-rs/bytesize) | `2.2.0` | `2.3.0` |
| [aws-config](https://github.com/smithy-lang/smithy-rs) | `1.8.10` | `1.8.11` |
| [aws-credential-types](https://github.com/smithy-lang/smithy-rs) | `1.2.9` | `1.2.10` |
| [aws-sdk-s3](https://github.com/awslabs/aws-sdk-rust) | `1.113.0` | `1.115.0` |
| [convert_case](https://github.com/rutrum/convert-case) | `0.9.0` | `0.10.0` |
| [hashbrown](https://github.com/rust-lang/hashbrown) | `0.16.0` | `0.16.1` |
| [rumqttc](https://github.com/bytebeamio/rumqtt) | `0.25.0` | `0.25.1` |
| [starshard](https://github.com/houseme/starshard) | `0.5.0` | `0.6.0` |


Updates `bytesize` from 2.2.0 to 2.3.0
- [Release notes](https://github.com/bytesize-rs/bytesize/releases)
- [Changelog](https://github.com/bytesize-rs/bytesize/blob/master/CHANGELOG.md)
- [Commits](https://github.com/bytesize-rs/bytesize/compare/bytesize-v2.2.0...bytesize-v2.3.0)

Updates `aws-config` from 1.8.10 to 1.8.11
- [Release notes](https://github.com/smithy-lang/smithy-rs/releases)
- [Changelog](https://github.com/smithy-lang/smithy-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/smithy-lang/smithy-rs/commits)

Updates `aws-credential-types` from 1.2.9 to 1.2.10
- [Release notes](https://github.com/smithy-lang/smithy-rs/releases)
- [Changelog](https://github.com/smithy-lang/smithy-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/smithy-lang/smithy-rs/commits)

Updates `aws-sdk-s3` from 1.113.0 to 1.115.0
- [Release notes](https://github.com/awslabs/aws-sdk-rust/releases)
- [Commits](https://github.com/awslabs/aws-sdk-rust/commits)

Updates `convert_case` from 0.9.0 to 0.10.0
- [Commits](https://github.com/rutrum/convert-case/commits)

Updates `hashbrown` from 0.16.0 to 0.16.1
- [Release notes](https://github.com/rust-lang/hashbrown/releases)
- [Changelog](https://github.com/rust-lang/hashbrown/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/hashbrown/compare/v0.16.0...v0.16.1)

Updates `rumqttc` from 0.25.0 to 0.25.1
- [Release notes](https://github.com/bytebeamio/rumqtt/releases)
- [Changelog](https://github.com/bytebeamio/rumqtt/blob/main/CHANGELOG.md)
- [Commits](https://github.com/bytebeamio/rumqtt/compare/rumqttc-0.25.0...rumqttc-0.25.1)

Updates `starshard` from 0.5.0 to 0.6.0
- [Commits](https://github.com/houseme/starshard/compare/0.5.0...0.6.0)

---
updated-dependencies:
- dependency-name: bytesize
  dependency-version: 2.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: aws-config
  dependency-version: 1.8.11
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: aws-credential-types
  dependency-version: 1.2.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: aws-sdk-s3
  dependency-version: 1.115.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: convert_case
  dependency-version: 0.10.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: hashbrown
  dependency-version: 0.16.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: rumqttc
  dependency-version: 0.25.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: starshard
  dependency-version: 0.6.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-24 11:03:35 +08:00
majinghe
2b268fdd7f update tls configuration in helm chart (#900)
* update tls configuration in helm chart

* typo fix
2025-11-20 22:20:11 +08:00
houseme
18cd9a8b46 build(deps): bump the dependencies group with 5 updates (#896) 2025-11-20 13:04:24 +08:00
loverustfs
e14809ee04 Revise data sovereignty and compliance details in README
Updated the comparison between RustFS and other object storage solutions to clarify data sovereignty and compliance aspects.
2025-11-20 09:11:15 +08:00
loverustfs
390d051ddd Update README.md
Correcting inaccurate expressions
2025-11-20 08:55:14 +08:00
weisd
b73ca0220e add ifmatch support (#885)
Co-authored-by: loverustfs <hello@rustfs.com>
2025-11-20 08:40:47 +08:00
loverustfs
b4071d493c Modify README 2025-11-20 08:24:07 +08:00
Copilot
277d80de13 Fix: Implement priority-based heal queue with comprehensive diagnostic logging (#884)
* Initial plan

* Implement priority-based heal queue with deduplication

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Apply cargo fmt formatting fixes

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Add queue monitoring, better error handling, and adaptive processing

- Add priority-based queue statistics tracking
- Implement queue capacity warnings (>80% full)
- Process multiple tasks per cycle when capacity allows
- Add proper error logging for failed heal request submissions
- Add Hash trait to HealPriority for HashMap support
- Improve observability with detailed queue status logs

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Add comprehensive diagnostic logging to heal flow

- Add detailed logging for shard availability checks during reads
- Log successful heal request submissions
- Add detailed logging in heal_object for disk checking
- Log each disk's heal decision with reason and endpoint
- Add logging for heal writing and renaming phases
- Improve variable naming (nil_count -> available_shards)
- Add success confirmation messages for renamed healed data

These logs will help diagnose where the heal flow is failing.

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* feat(heal): enhance robustness, safety, and observability of heal process

- **Logging & Observability**:
  - Upgrade critical heal logs from [info](/crates/e2e_test/src/reliant/node_interact_test.rs:196:0-213:1) to `warn` for better visibility.
  - Implement structured logging with `tracing` fields for machine readability.
  - Add `#[tracing::instrument]` to [HealTask](c/crates/ahm/src/heal/task.rs:182:0-205:1) and [SetDisks](/crates/ecstore/src/set_disk.rs:120:0-131:1) methods for automatic context propagation.

- **Robustness**:
  - Add exponential backoff retry (3 attempts) for acquiring write locks in [heal_object](/crates/ahm/src/heal/storage.rs:438:4-460:5) to handle contention.
  - Handle [rename_data](/crates/ecstore/src/set_disk.rs:392:4-516:5) failures gracefully by preserving temporary files instead of forcing deletion, preventing potential data loss.

- **Data Safety**:
  - Fix [object_exists](/crates/ahm/src/heal/storage.rs:395:4-412:5) to propagate IO errors instead of treating them as "object not found".
  - Update [ErasureSetHealer](/crates/ahm/src/heal/erasure_healer.rs:28:0-33:1) to mark objects as failed rather than skipped when existence checks error, ensuring they are tracked for retry.

* fix

* fmt

* improve code for heal_object

* fix

* fix

* fix

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-11-20 00:36:25 +08:00
shiro.lee
9b9bbb662b fix: removing the Limit on the Number of Object Versions (#819) (#892)
removing the Limit on the Number of Object Versions (#819)
2025-11-19 22:34:26 +08:00
majinghe
44f3f3d070 add standalone mode support (#881)
* add standalone mode support

* update readme file

* change non-root from 1000 to 10001

* delete self sign crt content

* modify security content

* fix synatx error for readme file.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* update image repository and tag info.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix helm chart syntax issue.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix helm chart syntax issue.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2025-11-19 13:56:21 +08:00
weisd
a13ce08590 fix listbucket sign (#890) 2025-11-19 11:08:16 +08:00
weisd
55d44622ed list object include deleted support (#882)
Co-authored-by: houseme <housemecn@gmail.com>
2025-11-18 21:51:10 +08:00
Copilot
6da5766ea2 Implement adaptive buffer sizing with workload profiles - Production-Ready Implementation (#878)
* Initial plan

* Add workload profiles module with adaptive buffer sizing

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Add comprehensive adaptive buffer sizing documentation

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Add implementation summary documentation

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Implement Phase 2: Opt-In Usage for buffer profiles

- Add RUSTFS_BUFFER_PROFILE_ENABLE and RUSTFS_BUFFER_PROFILE config options
- Implement global buffer configuration with enable/disable flag
- Add WorkloadProfile::from_name() for parsing profile names
- Update put_object, put_object_extract, and upload_part to use opt-in function
- Add init_buffer_profile_system() in main.rs for initialization
- Add comprehensive tests for opt-in functionality
- Update documentation with Phase 2 usage examples

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Implement Phase 3: Default Enablement for buffer profiles

- Change default to enabled with GeneralPurpose profile
- Replace --buffer-profile-enable with --buffer-profile-disable for opt-out
- Update init_buffer_profile_system() to reflect Phase 3 logic
- Add comprehensive Phase 3 documentation
- Create MIGRATION_PHASE3.md guide for smooth transition
- Add test for Phase 3 default behavior
- Update IMPLEMENTATION_SUMMARY.md to mark Phase 3 as complete

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Implement Phase 4: Full Integration with deprecated legacy and metrics

- Deprecate get_adaptive_buffer_size() function (backward compatible)
- Implement profile-only buffer sizing in get_buffer_size_opt_in()
- Add performance metrics collection (optional, via metrics feature)
- Even disabled mode now uses GeneralPurpose profile (no hardcoded values)
- Add comprehensive Phase 4 documentation and guide
- Add test_phase4_full_integration() test
- Update IMPLEMENTATION_SUMMARY.md to mark Phase 4 as complete

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Finalize adaptive buffer sizing: remove deprecated code and improve documentation

English:
- Remove deprecated get_adaptive_buffer_size() function
- Remove deprecated test cases (test_adaptive_buffer_size, Phase 3 legacy comparisons)
- Improve all code comments to be clear and professional English
- Add comprehensive bilingual implementation summary (COMPLETE_SUMMARY.md)
- Update buffer sizing comments in put_object, put_object_extract, upload_part
- Enhance init_buffer_profile_system documentation
- Clean up phase-specific references in comments

Chinese (中文):
- 移除已弃用的 get_adaptive_buffer_size() 函数
- 移除已弃用的测试用例(test_adaptive_buffer_size、Phase 3 旧版比较)
- 改进所有代码注释,使用清晰专业的英文
- 添加全面的双语实现摘要(COMPLETE_SUMMARY.md)
- 更新 put_object、put_object_extract、upload_part 中的缓冲区调整注释
- 增强 init_buffer_profile_system 文档
- 清理注释中的特定阶段引用

This commit completes the adaptive buffer sizing implementation by:
1. Removing all deprecated legacy code and tests
2. Improving code documentation quality
3. Providing comprehensive bilingual summary

本提交完成自适应缓冲区大小实现:
1. 移除所有已弃用的旧代码和测试
2. 提高代码文档质量
3. 提供全面的双语摘要

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fmt

* fix

* fix

* fix

* fix

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-11-18 13:32:02 +08:00
weisd
85bc0ce2d5 fix: filemeta version handling and delete operations (#879)
* fix filemeta version

* fix clippy

* fix delete version

* fix clippy/test
2025-11-18 09:24:22 +08:00
Copilot
601f3456bc Fix large file upload freeze with adaptive buffer sizing (#869)
* Initial plan

* Fix large file upload freeze by increasing StreamReader buffer size

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Add comprehensive documentation for large file upload freeze fix

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* upgrade s3s version

* Fix compilation error: use BufReader instead of non-existent StreamReader::with_capacity

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Update documentation with correct BufReader implementation

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* add tokio feature `io-util`

* Implement adaptive buffer sizing based on file size

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Constants are managed uniformly and fmt code

* fix

* Fix: Trigger self-heal on read when shards missing from rejoined nodes (#871)

* Initial plan

* Fix: Trigger self-heal when missing shards detected during read

- Added proactive heal detection in get_object_with_fileinfo
- When reading an object, now checks if any shards are missing even if read succeeds
- Sends low-priority heal request to reconstruct missing shards on rejoined nodes
- This fixes the issue where data written during node outage is not healed when node rejoins

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* Unify CRC implementations to crc-fast (#873)

* Initial plan

* Replace CRC libraries with unified crc-fast implementation

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* fix: replace low to Normal

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-11-17 23:15:20 +08:00
weisd
1279baa72b fix replication (#875) 2025-11-17 17:37:41 +08:00
weisd
acdefb6703 fix read lock (#866) 2025-11-16 11:44:13 +08:00
Copilot
b7964081ce Fix KMS configuration synchronization across cluster nodes (#855)
* Initial plan

* Add KMS configuration persistence to cluster storage

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Apply code formatting to KMS configuration changes

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* add comment

* fix fmt

* fix

* Fix overlapping dependabot cargo configurations

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* improve code for comment and replace  `Once_Cell` to `std::sync::OnceLock`

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2025-11-16 00:05:03 +08:00
Nugine
f73fa59bf6 ci: fix dependabot (#860) 2025-11-15 22:35:59 +08:00
Nugine
0b1b7832fe ci: update s3s weekly (#858) 2025-11-15 22:05:03 +08:00
houseme
c242957c6f build(deps): bump the dependencies group with 8 updates (#857) 2025-11-15 19:51:07 +08:00
houseme
55e3a1f7e0 fix(audit): prevent state transition when no targets exist (#854)
Avoid setting AuditSystemState::Starting when target list is empty.
Now checks target availability before state transition, keeping the
system in Stopped state if no enabled targets are found.

- Check targets.is_empty() before setting Starting state
- Return early with Ok(()) when no targets exist
- Maintain consistent state machine behavior
- Prevent transient "Starting" state with no actual targets

Resolves issue where audit system would incorrectly enter Starting
state even when configuration contained no enabled targets.
2025-11-14 20:23:21 +08:00
majinghe
3cf565e847 delete sink file path env and update readme file with container user change (#852)
* update container user change in readme file

* delete sink file path env vars

---------

Co-authored-by: houseme <housemecn@gmail.com>
2025-11-14 13:00:15 +08:00
houseme
9d553620cf remove linux dep and upgrade Protocol Buffers and FlatBuffers (#853) 2025-11-14 12:50:55 +08:00
houseme
51584986e1 feat(obs): unify metrics initialization and fix exporter move error (#851)
* feat(obs): unify metrics initialization and fix exporter move error

- Fix Rust E0382 (use after move) by removing duplicate MetricExporter consumption.
- Consolidate MeterProvider construction into single Recorder builder path.
- Remove redundant Recorder::builder(...).install_global() call.
- Ensure PeriodicReader setup is performed only once (HTTP + optional stdout).
- Set global meter provider and metrics recorder exactly once.
- Preserve existing behavior for stdout/file vs HTTP modes.
- Minor cleanup: consistent resource reuse and interval handling.

* update telemetry.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix

* fix

* fix

* fix: modify logger level from error to event

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-11-14 00:50:07 +08:00
majinghe
93090adf7c enhance security context part for k8s deployment (#850) 2025-11-13 18:18:19 +08:00
houseme
d4817a4bea fix: modify logger from warn to info (#842)
* fix: modify logger from warn to info

* upgrade version
2025-11-12 11:13:38 +08:00
houseme
7e1a9e2ede 🔒 Upgrade Cryptography Libraries to Latest RC Versions (#837)
* fix

* chore: upgrade cryptography libraries to RC versions

- Upgrade aes-gcm to 0.11.0-rc.2 with rand_core support
- Upgrade chacha20poly1305 to 0.11.0-rc.2
- Upgrade argon2 to 0.6.0-rc.2 with std features
- Upgrade hmac to 0.13.0-rc.3
- Upgrade pbkdf2 to 0.13.0-rc.2
- Upgrade rsa to 0.10.0-rc.10
- Upgrade sha1 and sha2 to 0.11.0-rc.3
- Upgrade md-5 to 0.11.0-rc.3

These upgrades provide enhanced security features and performance
improvements while maintaining backward compatibility with existing
encryption workflows.

* add

* improve code

* fix
2025-11-11 21:10:03 +08:00
安正超
8a020ec4d9 wip (#830) 2025-11-11 09:34:58 +08:00
weisd
77a3489ed2 fix list object err (#831)
fix list object err (#831)

#827
#815
#635
#752
2025-11-10 23:42:15 +08:00
weisd
5941062909 fix (#828) 2025-11-10 19:22:58 +08:00
houseme
98be7df0f5 feat(storage): refactor audit and notification with OperationHelper (#825)
* improve code for audit

* improve code ecfs.rs

* improve code

* improve code for ecfs.rs

* feat(storage): refactor audit and notification with OperationHelper

This commit introduces a significant refactoring of the audit logging and event notification mechanisms within `ecfs.rs`.

The core of this change is the new `OperationHelper` struct, which encapsulates and simplifies the logic for both concerns. It replaces the previous `AuditHelper` and manual event dispatching.

Key improvements include:

- **Unified Handling**: `OperationHelper` manages both audit and notification builders, providing a single, consistent entry point for S3 operations.
- **RAII for Automation**: By leveraging the `Drop` trait, the helper automatically dispatches logs and notifications when it goes out of scope. This simplifies S3 method implementations and ensures cleanup even on early returns.
- **Fluent API**: A builder-like pattern with methods such as `.object()`, `.version_id()`, and `.suppress_event()` makes the code more readable and expressive.
- **Context-Aware Logic**: The helper's `.complete()` method intelligently populates log details based on the operation's `S3Result` and only triggers notifications on success.
- **Modular Design**: All helper logic is now isolated in `rustfs/src/storage/helper.rs`, improving separation of concerns and making `ecfs.rs` cleaner.

This refactoring significantly enhances code clarity, reduces boilerplate, and improves the robustness of logging and notification handling across the storage layer.

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* improve code for audit and notify

* fix

* fix

* fix
2025-11-10 17:30:50 +08:00
houseme
b26aad4129 improve code for logger (#822)
* improve code for logger

* fix
2025-11-08 22:36:24 +08:00
Alex Bykov
5989589c3e Update configuration.md (#812)
Escaping Pipe Character in the table "CLI Flags..."

Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2025-11-08 10:56:14 +08:00
majinghe
4716454faa add non root user support for container deployment (#817) 2025-11-08 10:00:14 +08:00
houseme
29056a767a Refactor Telemetry Initialization and Environment Utilities (#811)
* improve code for metrics

* improve code for metrics

* fix

* fix

* Refactor telemetry initialization and environment functions ordering

- Reorder functions in envs.rs by type size (8-bit to 64-bit, signed before unsigned) and add missing variants like get_env_opt_u16.
- Optimize init_telemetry to support three modes: stdout logging (default error level with span tracing), file rolling logs (size-based with retention), and HTTP-based observability with sub-endpoints (trace, metric, log) falling back to unified endpoint.
- Fix stdout logging issue by retaining WorkerGuard in OtelGuard to prevent premature release of async writer threads.
- Enhance observability mode with HTTP protocol, compression, and proper resource management.
- Update OtelGuard to include tracing_guard for stdout and flexi_logger_handles for file logging.
- Improve error handling and configuration extraction in OtelConfig.

* fix

* up

* fix

* fix

* improve code for obs

* fix

* fix
2025-11-07 20:01:54 +08:00
weisd
e823922654 feat:add api error message (#801)
* feat:add api error message
* fix: check input
* fix: test
2025-11-07 09:53:49 +08:00
shiro.lee
8203f9ff6f fix: when the Object Lock configuration does not exist, an error message should be returned (#771) (#798)
fix: when the Object Lock configuration does not exist, an error message should be returned (#771) (#798)
2025-11-05 23:48:54 +08:00
houseme
1b22a1e078 Refactor modify stdout (#797)
* fix

* fix
2025-11-05 20:04:28 +08:00
weisd
461d5dff86 fix list max keys (#795) 2025-11-05 15:30:32 +08:00
houseme
38f26b7c94 improve import,crate version,and copyright (#790) 2025-11-05 09:10:06 +08:00
安正超
eb7eb9c5a1 fix: resolve logic errors in ahm heal module (#788)
* fix: resolve logic errors in ahm heal module

- Fix response publishing logic in HealChannelProcessor to properly handle errors
- Fix negative index handling in DiskStatusChange event to fail fast instead of silently converting to 0
- Enhance timeout control in heal_erasure_set Step 3 loop to immediately respond to cancellation/timeout
- Add proper error propagation for task cancellation and timeout in bucket healing loop

* fix: stabilize performance impact measurement test

- Increase measurement count from 3 to 5 runs for better stability
- Increase workload from 5000 to 10000 operations for more accurate timing
- Use median of 5 measurements instead of single measurement
- Ensure with_scanner duration is at least baseline to avoid negative overhead
- Increase wait time for scanner state stabilization

* wip

* Update crates/ahm/src/heal/channel.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* refactor: remove redundant ok_or_else + expect in event.rs

Replace redundant ok_or_else() + expect() pattern with
unwrap_or_else() + panic!() to avoid creating unnecessary Error
type when the value will panic anyway. This also defers error
message formatting until the error actually occurs.

* Update crates/ahm/src/heal/task.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix(ahm): fix logic errors and add unit tests

- Fix panic in HealEvent::to_heal_request for invalid indices
- Replace unwrap() calls with proper error handling in resume.rs
- Fix race conditions and timeout calculation in task.rs
- Fix semaphore acquisition error handling in erasure_healer.rs
- Improve error message for large objects in storage.rs
- Add comprehensive unit tests for progress, event, and channel modules
- Fix clippy warning: move test module to end of file in heal_channel.rs

* style: apply cargo fmt formatting

* refactor(ahm): address copilot review suggestions

- Add comment to check_control_flags explaining why return value is discarded
- Fix hardcoded median index in performance test using constant and dynamic calculation

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-11-05 08:15:23 +08:00
houseme
d934e3905b Refactor telemetry initialization for non-production environments (#789)
* add dep `scopeguard`

* improve for tracing

* fix

* fix

* improve code for import

* add logger trace id

* fix

* fix

* fix

* fix

* fix
2025-11-05 00:55:08 +08:00
weisd
6617372b33 fix rmdir versionid (#784) 2025-11-03 18:23:16 +08:00
weisd
769778e565 fix iam (#783) 2025-11-03 17:39:51 +08:00
houseme
a7f5c4af46 fix windows response (#781) 2025-11-03 12:49:39 +08:00
dependabot[bot]
a9d5fbac54 build(deps): bump the dependencies group with 6 updates (#777)
* build(deps): bump the dependencies group with 6 updates

Bumps the dependencies group with 6 updates:

| Package | From | To |
| --- | --- | --- |
| [axum-extra](https://github.com/tokio-rs/axum) | `0.10.3` | `0.12.0` |
| [aws-config](https://github.com/smithy-lang/smithy-rs) | `1.8.8` | `1.8.10` |
| [aws-sdk-s3](https://github.com/awslabs/aws-sdk-rust) | `1.109.0` | `1.110.0` |
| [aws-smithy-types](https://github.com/smithy-lang/smithy-rs) | `1.3.3` | `1.3.4` |
| [clap](https://github.com/clap-rs/clap) | `4.5.50` | `4.5.51` |
| [matchit](https://github.com/ibraheemdev/matchit) | `0.8.4` | `0.9.0` |


Updates `axum-extra` from 0.10.3 to 0.12.0
- [Release notes](https://github.com/tokio-rs/axum/releases)
- [Changelog](https://github.com/tokio-rs/axum/blob/main/CHANGELOG.md)
- [Commits](https://github.com/tokio-rs/axum/compare/axum-extra-v0.10.3...axum-extra-v0.12.0)

Updates `aws-config` from 1.8.8 to 1.8.10
- [Release notes](https://github.com/smithy-lang/smithy-rs/releases)
- [Changelog](https://github.com/smithy-lang/smithy-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/smithy-lang/smithy-rs/commits)

Updates `aws-sdk-s3` from 1.109.0 to 1.110.0
- [Release notes](https://github.com/awslabs/aws-sdk-rust/releases)
- [Commits](https://github.com/awslabs/aws-sdk-rust/commits)

Updates `aws-smithy-types` from 1.3.3 to 1.3.4
- [Release notes](https://github.com/smithy-lang/smithy-rs/releases)
- [Changelog](https://github.com/smithy-lang/smithy-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/smithy-lang/smithy-rs/commits)

Updates `clap` from 4.5.50 to 4.5.51
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/clap_complete-v4.5.50...clap_complete-v4.5.51)

Updates `matchit` from 0.8.4 to 0.9.0
- [Release notes](https://github.com/ibraheemdev/matchit/releases)
- [Commits](https://github.com/ibraheemdev/matchit/compare/v0.8.4...v0.9.0)

---
updated-dependencies:
- dependency-name: axum-extra
  dependency-version: 0.12.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: aws-config
  dependency-version: 1.8.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: aws-sdk-s3
  dependency-version: 1.110.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: aws-smithy-types
  dependency-version: 1.3.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: clap
  dependency-version: 4.5.51
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: matchit
  dependency-version: 0.9.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* upgrade crates version

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-11-03 00:34:54 +08:00
houseme
281e68c9bf fix (#776) 2025-11-01 09:28:46 +08:00
houseme
d30c42f85a feat(admin): Add admin v3 API routes and profiling endpoints for RustFS (#774)
* add Jemalloc

* feat: optimize AI rules with unified .rules.md  (#401)

* feat: optimize AI rules with unified .rules.md and entry points

- Create .rules.md as the central AI coding rules file
- Add .copilot-rules.md as GitHub Copilot entry point
- Add CLAUDE.md as Claude AI entry point
- Incorporate principles from rustfs.com project
- Add three critical rules:
  1. Use English for all code comments and documentation
  2. Clean up temporary scripts after use
  3. Only make confident modifications

* Update CLAUDE.md

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* feat: translate chinese to english (#402)

* Checkpoint before follow-up message

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

* Translate project documentation and comments from Chinese to English

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

* Fix typo: "unparseable" to "unparsable" in version test comment

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

* Refactor compression test code with minor syntax improvements

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* fix: the automatic logout issue and user list display failure on Windows systems (#353) (#343) (#403)

Co-authored-by: 安正超 <anzhengchao@gmail.com>

* upgrade version

* improve code for profiling

* fix

* Initial plan

* feat: Implement layered DNS resolver with caching and validation

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: Integrate DNS resolver into main application and fix formatting

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: Implement enhanced DNS resolver with Moka cache and layered fallback

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: Implement hickory-resolver with TLS support for enhanced DNS resolution

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* upgrade

* add .gitignore config

* fix

* add

* add

* up

* improve linux profiling

* fix

* fix

* fix

* feat(admin): Refactor profiling endpoints

Replaces the existing pprof profiling endpoints with new trigger-based APIs for CPU and memory profiling. This change simplifies the handler logic by moving the profiling implementation to a dedicated module.

A new handler file `admin/handlers/profile.rs` is created to contain the logic for these new endpoints. The core profiling functions are now expected to be in the `profiling` module, which the new handlers call to generate and save profile data.

* cargo shear --fix

* fix

* fix

* fix

---------

Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: shiro.lee <69624924+shiroleeee@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
2025-11-01 03:16:37 +08:00
Niklas Mollenhauer
79012be2c8 Add default storage class to ListObjectsV2 (#765)
* Add InvalidRangeSpec error

* Add EntityTooSmall to from_u32

* Add InvalidRangeSpec to from_u32

* Map InvalidRangeSpec to correct S3ErrorCode

* Return Error::InvalidRangeSpec

* Use auto implementation

* Add default storage class to ListObjectsV2

Resolves #764

* Add storage_class to response

* Make storage class optional so default won't be an empty string

---------

Co-authored-by: houseme <housemecn@gmail.com>
2025-10-31 19:32:25 +08:00
loverustfs
325ff62684 Issue 762 (#763)
* Add InvalidRangeSpec error

* Add EntityTooSmall to from_u32

* Add InvalidRangeSpec to from_u32

* Map InvalidRangeSpec to correct S3ErrorCode

* Return Error::InvalidRangeSpec

* Use auto implementation

---------

Co-authored-by: Niklas Mollenhauer <nikeee@outlook.com>
2025-10-31 17:20:18 +08:00
安正超
f0c2ede7a7 Remove unnecessary tools folder in CI workflow (#770) 2025-10-31 16:44:08 +08:00
安正超
b9fd66c1cd Delete deploy/build/rustfs.run-zh.md (#757) 2025-10-30 13:56:26 +08:00
安正超
c43b11fb92 Delete deploy/build/rustfs-zh.service (#756) 2025-10-30 13:55:51 +08:00
安正超
d737a439d5 Delete deploy/config/rustfs-zh.env (#755) 2025-10-30 13:54:53 +08:00
houseme
0714c7a9ca modify logger level from info to error (#744)
* modify logger level from `info` to `error`

* fix test

* improve tokio runtime config

* add rustfs helm chart files (#747)

* add rustfs helm chart files

* update readme file with helm chart

* delete helm chart license file

* fix typo in readme file

* fix: restore localized samples in tests (#749)

* fix: restore required localized examples

* style: fix formatting issues

* improve code for Observability

* upgrade crates version

* fix

* up

* fix

---------

Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2025-10-29 19:20:53 +08:00
loverustfs
2ceb65adb4 replace rustfs pic 2025-10-29 15:50:18 +08:00
安正超
dd47fcf2a8 fix: restore localized samples in tests (#749)
* fix: restore required localized examples

* style: fix formatting issues
2025-10-29 13:16:31 +08:00
majinghe
64ba52bc1e add rustfs helm chart files (#747)
* add rustfs helm chart files

* update readme file with helm chart

* delete helm chart license file

* fix typo in readme file
2025-10-29 12:23:21 +08:00
shiro.lee
d2ced233e5 fix: when the error returned by make_bucket is BucketExists, replace … (#735)
* fix: when the error returned by make_bucket is BucketExists, replace BucketAlreadyExists with BucketAlreadyOwnedByYou (#719)

* test: In the test_api_error_from_storage_error_mappings test method, modify the corresponding mapping relationships

---------

Co-authored-by: weisd <im@weisd.in>
2025-10-28 15:26:34 +08:00
weisd
40660e7b80 fix: scandir object (#733)
* fix: scandir object count

* fix: base64 list continuation_token
2025-10-28 15:02:43 +08:00
likewu
2aca1f77af Fix/ilm (#721)
* fix tip remote tier error
* fix transitioned_object
* fix filemeta
* add GCS R2
* add aliyun tencent huaweicloud azure gcs r2 backend tier
* fix signer
* change azure to s3
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2025-10-27 20:23:50 +08:00
Ben Scholzen
6f3d2885cd fix: take content type from PutObjectInput instead of headers (#718)
fixes #716

Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2025-10-26 21:44:54 +08:00
shiro.lee
6ab7619023 fix: The issue of multi-level objects created in Windows not being displayed has been fixed (#661) (#723) 2025-10-26 12:00:13 +08:00
weisd
ed73e2b782 fix:add favicon.ico route (#713) 2025-10-25 16:11:18 +08:00
weisd
6a59c0a474 fix: multipart upload checksum validation (#712)
* fix multipart upload checksum
2025-10-24 18:23:32 +08:00
houseme
c5264f9703 improve code for metrics and switch tokio-tar to astral-tokio-tar (#705)
* improve code for metrics and switch tokio-tar to astral-tokio-tar

* remove log

* fix
2025-10-24 13:07:56 +08:00
DamonXue
b47765b4c0 docs: add Star History section to README files (#696)
Co-authored-by: 0xdx2 <xuedamon2@gmail.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2025-10-24 08:58:58 +08:00
houseme
e22b24684f chore: bump dependencies, add metrics support, remove DNS resolver (#699)
* upgrade version

* add metrics

* remove dns resolver

* add metrics counter for create bucket

* fix

* fix

* fix
2025-10-24 00:16:17 +08:00
weisd
1d069fd351 Improve the peer client (#693) 2025-10-23 17:21:55 +08:00
houseme
416d3ad5b7 Refactor: Add observability enable flag, improve comments, remove unused config params, and enhance run function error logging. (#689)
* improve code for dns log

* fix

* Improve comments, remove unused parameters in config.rs (opt), add observability enable flag, and enhance error logging in run function execution.
2025-10-23 13:59:57 +08:00
weisd
f30698ec7f Refactor Console Server Architecture (#685)
* todo

* fix console server

* fix console server

* fix console server

* fix console server

* fix console server
2025-10-23 00:06:09 +08:00
houseme
7dcf01f127 feat: adjust metrics push interval to 3 seconds (#686)
- Reduce metrics push frequency from default to 3s for better performance
- Optimize resource utilization during metrics collection
- Improve real-time monitoring responsiveness

Related to admin metrics optimization on fix/admin-metrics branch
2025-10-22 23:47:11 +08:00
weisd
e524a106c5 add make bucket error logs (#683)
* add make bucket error logs
2025-10-22 16:23:08 +08:00
weisd
d9e5f5d2e3 fix (#682) 2025-10-22 10:35:40 +08:00
livelycode36
684e832530 fix: prevent duplicate data volumes in entrypoint.sh (#681) 2025-10-22 09:04:04 +08:00
weisd
a65856bdf4 Fix CRC32C Checksum Implementation and Enhance Authentication System (#678)
* fix: get_condition_values

* fix checksum crc32c

* fix clippy
2025-10-21 21:28:00 +08:00
weisd
2edb2929b2 fix: DataUsageInfo add list bucket permission (#674) 2025-10-21 10:05:54 +08:00
majinghe
14bc55479b fix docker healthcheck unhealthy issue (#672) 2025-10-21 09:39:15 +08:00
weisd
cd1e244c68 Refactor: Introduce content checksums and improve multipart/object metadata handling (#671)
* feat:  adapt to s3s typed etag support

* refactor: move replication struct to rustfs_filemeta, fix filemeta transition bug

* add head_object checksum, filter object metadata output

* fix multipart checksum

* fix multipart checksum

* add content md5,sha256 check

* fix test

* fix cargo

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2025-10-20 23:46:13 +08:00
songhahaha66
46797dc815 fix(export): fix the policy and service account export (#665)
* fix(export): fix the policy export mechanism

* fix: correct service account check logic in IamSys
2025-10-20 19:40:54 +08:00
Nugine
7f24dbda19 build(deps): upgrade s3s (#667)
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2025-10-19 18:32:01 +08:00
loverustfs
ef11d3a2eb fix words error 2025-10-19 18:13:58 +08:00
loverustfs
d1398cb3ab fix error 2025-10-19 18:10:45 +08:00
majinghe
95019c4cb5 add ansible installation with mnmd (#664)
* add ansible installation with mnmd

* change script install dir name
2025-10-18 22:20:17 +08:00
houseme
4168e6c180 chore(docs): move root examples to docs/examples/docker and update README (#663)
* chore(docs): move root `examples` to `docs/examples/docker` and update README

- Move root `examples/` contents into `docs/examples/docker/`.
- Update `docs/examples/README.md` to add migration note, new `docker/` entry and usage examples.
- Replace references from `examples/` to `docs/examples/docker/` where applicable.
- Reminder: verify CI and external links still point to the correct paths.

* fix
2025-10-17 17:17:36 +08:00
houseme
42d3645d6f fix(targets): make target removal and reload transactional; prevent reappearing entries (#662)
* feat: improve code for notify

* upgrade starshard version

* upgrade version

* Fix ETag format to comply with HTTP standards by wrapping with quotes (#592)

* Initial plan

* Fix ETag format to comply with HTTP standards by wrapping with quotes

Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>

* bufigx

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>

* Improve lock (#596)

* improve lock

Signed-off-by: Mu junxiang <1948535941@qq.com>

* feat(tests): add wait_for_object_absence helper and improve lifecycle test reliability

Signed-off-by: Mu junxiang <1948535941@qq.com>

* chore: remove dirty docs

Signed-off-by: Mu junxiang <1948535941@qq.com>

---------

Signed-off-by: Mu junxiang <1948535941@qq.com>

* feat(append): implement object append operations with state tracking (#599)

* feat(append): implement object append operations with state tracking

Signed-off-by: junxiang Mu <1948535941@qq.com>

* chore: rebase

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>

* build(deps): upgrade s3s (#595)

Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>

* fix: validate mqtt broker

* improve code for `import`

* fix

* improve

* remove logger from `rustfs-obs` crate

* remove code for config Observability

* fix

* improve code

* fix comment

* up

* up

* upgrade version

* fix

* fmt

* upgrade tokio version to 1.48.0

* upgrade `datafusion` and `reed-solomon-simd` version

* fix

* fmt

* improve code for notify webhook example

* improve code

* fix

* fix

* fmt

---------

Signed-off-by: Mu junxiang <1948535941@qq.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: guojidan <63799833+guojidan@users.noreply.github.com>
Co-authored-by: Nugine <nugine@foxmail.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2025-10-17 15:34:53 +08:00
安正超
30e7f00b02 fix: update ahm integration test fixture (#659) 2025-10-17 09:13:56 +08:00
overtrue
58f8a8f46b fix: correct HTTP range suffix handling 2025-10-16 21:39:21 +08:00
gatewayJ
aae768f446 feat: Simple OPA support (#644)
* opa-feature

* Update crates/policy/src/policy/opa.rs

* add the content related to 'Copyright'

---------

Co-authored-by: root <root@debian.localdomain>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-16 10:35:26 +08:00
安正超
d447b3e426 feat: adapt to s3s typed etag support (#653)
* feat:  adapt to s3s typed etag support

* refactor: streamline etag handling
2025-10-15 21:27:20 +08:00
安正超
8f310cd4a8 test: allow mocking dns resolver (#656) 2025-10-15 21:24:03 +08:00
majinghe
8ed01a3e06 Refactor mnmd docker compose for extendence (#652) 2025-10-15 03:48:05 +08:00
loverustfs
9e1739ed8d chore(docs): update README and README_ZH (#649) 2025-10-13 18:49:34 +08:00
loverustfs
7abbfc9c2c RustFS trending images
RustFS trending
2025-10-13 17:45:54 +08:00
安正超
639bf0c233 Revert "feat(append): implement object append operations with state tracking (#599)" (#646)
This reverts commit 4f73760a45.
2025-10-12 23:47:51 +08:00
Copilot
ad99019749 Add complete MNMD Docker deployment example with startup coordination and VolumeNotFound fix (#642)
* Initial plan

* Add MNMD Docker deployment example with 4 nodes x 4 drives

- Create docs/examples/mnmd/ directory structure
- Add docker-compose.yml with proper disk indexing (1..4)
- Add wait-and-start.sh for startup coordination
- Add README.md with usage instructions and alternatives
- Add CHECKLIST.md with step-by-step verification
- Fixes VolumeNotFound issue by using correct volume paths
- Implements health checks and startup ordering
- Uses service names for stable inter-node addressing

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Add docs/examples README as index for deployment examples

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Add automated test script for MNMD deployment

- Add test-deployment.sh with comprehensive validation
- Test container status, health, endpoints, connectivity
- Update README to reference test script
- Make script executable

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* improve code

* improve code

* improve dep crates `cargo shear --fix`

* upgrade aws-sdk-s3

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-10-12 13:15:14 +08:00
houseme
aac9b1edb7 chore: improve event and docker-compose ,Improve the permissions of the endpoint health interface, upgrade otel from 0.30.0 to 0.31.0 (#620)
* feat: improve code for notify

* upgrade starshard version

* upgrade version

* Fix ETag format to comply with HTTP standards by wrapping with quotes (#592)

* Initial plan

* Fix ETag format to comply with HTTP standards by wrapping with quotes

Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>

* bufigx

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>

* Improve lock (#596)

* improve lock

Signed-off-by: Mu junxiang <1948535941@qq.com>

* feat(tests): add wait_for_object_absence helper and improve lifecycle test reliability

Signed-off-by: Mu junxiang <1948535941@qq.com>

* chore: remove dirty docs

Signed-off-by: Mu junxiang <1948535941@qq.com>

---------

Signed-off-by: Mu junxiang <1948535941@qq.com>

* feat(append): implement object append operations with state tracking (#599)

* feat(append): implement object append operations with state tracking

Signed-off-by: junxiang Mu <1948535941@qq.com>

* chore: rebase

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>

* build(deps): upgrade s3s (#595)

Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>

* fix: validate mqtt broker

* improve code for `import`

* upgrade otel relation crates version

* fix:dep("jsonwebtoken") feature = 'rust_crypto'

* fix

* fix

* fix

* upgrade version

* improve code for ecfs

* chore: improve event and docker-compose ,Improve the permissions of the `endpoint` health interface

* fix

* fix

* fix

* fix

* improve code

* fix

---------

Signed-off-by: Mu junxiang <1948535941@qq.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: guojidan <63799833+guojidan@users.noreply.github.com>
Co-authored-by: Nugine <nugine@foxmail.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2025-10-11 09:08:25 +08:00
weisd
5689311cff fix:#630 (#633) 2025-10-10 15:16:28 +08:00
安正超
007d9c0b21 fix: normalize ETag comparison in multipart upload and replication (#627)
- Normalize ETags by removing quotes before comparison in complete_multipart_upload
- Fix ETag comparison in replication logic to handle quoted ETags from API responses
- Fix ETag comparison in transition object logic
- Add unit tests for trim_etag function

This fixes the ETag mismatch error when uploading large files (5GB+) via multipart upload,
which was caused by PR #592 adding quotes to ETag responses while internal storage remains unquoted.

Fixes #625
2025-10-08 21:19:57 +08:00
Nugine
626c7ed34a fix: CompleteMultipartUpload encryption (#626) 2025-10-08 20:27:40 +08:00
houseme
0e680eae31 fix typos and bump the dependencies group with 9 updates (#614)
* fix typos

* build(deps): bump the dependencies group with 9 updates (#613)

Bumps the dependencies group with 9 updates:

| Package | From | To |
| --- | --- | --- |
| [axum](https://github.com/tokio-rs/axum) | `0.8.4` | `0.8.6` |
| [axum-extra](https://github.com/tokio-rs/axum) | `0.10.1` | `0.10.3` |
| [regex](https://github.com/rust-lang/regex) | `1.11.2` | `1.11.3` |
| [serde](https://github.com/serde-rs/serde) | `1.0.226` | `1.0.228` |
| [shadow-rs](https://github.com/baoyachi/shadow-rs) | `1.3.0` | `1.4.0` |
| [sysinfo](https://github.com/GuillaumeGomez/sysinfo) | `0.37.0` | `0.37.1` |
| [thiserror](https://github.com/dtolnay/thiserror) | `2.0.16` | `2.0.17` |
| [tokio-rustls](https://github.com/rustls/tokio-rustls) | `0.26.3` | `0.26.4` |
| [zeroize](https://github.com/RustCrypto/utils) | `1.8.1` | `1.8.2` |


Updates `axum` from 0.8.4 to 0.8.6
- [Release notes](https://github.com/tokio-rs/axum/releases)
- [Changelog](https://github.com/tokio-rs/axum/blob/main/CHANGELOG.md)
- [Commits](https://github.com/tokio-rs/axum/compare/axum-v0.8.4...axum-v0.8.6)

Updates `axum-extra` from 0.10.1 to 0.10.3
- [Release notes](https://github.com/tokio-rs/axum/releases)
- [Changelog](https://github.com/tokio-rs/axum/blob/main/CHANGELOG.md)
- [Commits](https://github.com/tokio-rs/axum/compare/axum-extra-v0.10.1...axum-extra-v0.10.3)

Updates `regex` from 1.11.2 to 1.11.3
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.11.2...1.11.3)

Updates `serde` from 1.0.226 to 1.0.228
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.226...v1.0.228)

Updates `shadow-rs` from 1.3.0 to 1.4.0
- [Release notes](https://github.com/baoyachi/shadow-rs/releases)
- [Commits](https://github.com/baoyachi/shadow-rs/compare/1.3.0...v1.4.0)

Updates `sysinfo` from 0.37.0 to 0.37.1
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GuillaumeGomez/sysinfo/compare/v0.37.0...v0.37.1)

Updates `thiserror` from 2.0.16 to 2.0.17
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/2.0.16...2.0.17)

Updates `tokio-rustls` from 0.26.3 to 0.26.4
- [Release notes](https://github.com/rustls/tokio-rustls/releases)
- [Commits](https://github.com/rustls/tokio-rustls/compare/v/0.26.3...v/0.26.4)

Updates `zeroize` from 1.8.1 to 1.8.2
- [Commits](https://github.com/RustCrypto/utils/compare/zeroize-v1.8.1...zeroize-v1.8.2)

---
updated-dependencies:
- dependency-name: axum
  dependency-version: 0.8.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: axum-extra
  dependency-version: 0.10.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: regex
  dependency-version: 1.11.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: serde
  dependency-version: 1.0.228
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: shadow-rs
  dependency-version: 1.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: sysinfo
  dependency-version: 0.37.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: thiserror
  dependency-version: 2.0.17
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: tokio-rustls
  dependency-version: 0.26.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: zeroize
  dependency-version: 1.8.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-10-02 23:29:18 +08:00
weisd
7622b37f7b add iam notification (#604)
move tonic service to rustfs
2025-09-30 17:32:23 +08:00
Nugine
f1dd3a982e build(deps): upgrade s3s (#595)
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2025-09-28 21:10:42 +08:00
guojidan
4f73760a45 feat(append): implement object append operations with state tracking (#599)
* feat(append): implement object append operations with state tracking

Signed-off-by: junxiang Mu <1948535941@qq.com>

* chore: rebase

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-09-27 20:06:26 -07:00
guojidan
be66cf8bd3 Improve lock (#596)
* improve lock

Signed-off-by: Mu junxiang <1948535941@qq.com>

* feat(tests): add wait_for_object_absence helper and improve lifecycle test reliability

Signed-off-by: Mu junxiang <1948535941@qq.com>

* chore: remove dirty docs

Signed-off-by: Mu junxiang <1948535941@qq.com>

---------

Signed-off-by: Mu junxiang <1948535941@qq.com>
2025-09-27 17:57:56 -07:00
Copilot
23b40d398f Fix ETag format to comply with HTTP standards by wrapping with quotes (#592)
* Initial plan

* Fix ETag format to comply with HTTP standards by wrapping with quotes

Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>

* bufigx

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2025-09-27 10:03:05 +08:00
weisd
90f21a9102 refactor: Reimplement bucket replication system with enhanced architecture (#590)
* feat:refactor replication

* use aws sdk for replication client

* refactor/replication

* merge main

* fix lifecycle test
2025-09-26 14:27:53 +08:00
guojidan
9b029d18b2 feat(lock): enhance lock management with timeout and ownership tracking (#589)
- Add lock timeout support and track acquisition time in lock state
- Improve lock conflict handling with detailed error messages
- Optimize lock reuse when already held by same owner
- Refactor lock state to store owner info and timeout duration
- Update all lock operations to handle new state structure

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-09-25 20:21:53 -07:00
houseme
9b7f4d477a Fix Tokio Runtime Initialization: Remove Private API Usage and Ensure IO Enabled (#587)
* fix: remove code

* improve code for tokio runtime config

* improve code for main

* fix: add tokio enable_all

* upgrade version

* improve for Cargo.toml
2025-09-24 22:23:31 +08:00
guojidan
12ecb36c6d Fix collect (#586)
* fix: fix datausageinfo

Signed-off-by: junxiang Mu <1948535941@qq.com>

* feat(data-usage): implement local disk snapshot aggregation for data usage statistics

Signed-off-by: junxiang Mu <1948535941@qq.com>

* feat(scanner): improve data usage collection with local scan aggregation

Signed-off-by: junxiang Mu <1948535941@qq.com>

* refactor: improve object existence check and code style

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-09-24 02:48:23 -07:00
guojidan
ef0dbaaeb5 feat(encryption): add managed encryption support for SSE-S3 and SSE-KMS (#583)
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-09-24 02:09:04 -07:00
Copilot
29b0935be7 RustFS rustfs-audit Complete Implementation with Enterprise Observability (#557)
* Initial plan

* Implement core audit system with multi-target fan-out and configuration management

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Changes before error encountered

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Complete audit system with comprehensive observability and test coverage

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* improve code

* fix

* improve code

* fix test

* fix test

* fix

* add `rustfs-audit` to `rustfs`

* upgrade crate version

* fmt

* fmt

* fix

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-09-24 08:23:46 +08:00
安正超
08aeca89ef feat: Allow alpha versions to create latest Docker tag (#577)
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2025-09-23 19:39:00 +08:00
gatewayJ
d39ce6d8e9 fix: correct DeleteObjectVersionAction (#574)
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2025-09-23 09:49:41 +08:00
guojidan
9ddf6a011d feature: support kms && encryt (#573)
* feat(kms): implement key management service with local and vault backends

Signed-off-by: junxiang Mu <1948535941@qq.com>

* feat(kms): enhance security with zeroize for sensitive data and improve key management

Signed-off-by: junxiang Mu <1948535941@qq.com>

* remove Hashi word

Signed-off-by: junxiang Mu <1948535941@qq.com>

* refactor: remove unused request structs from kms handlers

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-09-22 17:53:05 +08:00
houseme
f7e188eee7 feat: upgrade datafusion to v50.0.0 and update related dependencies f… (#563)
* feat: upgrade datafusion to v50.0.0 and update related dependencies for compatibility

* fix

* fmt
2025-09-18 23:30:25 +08:00
houseme
4b9cb512f2 remove crate rustfs-audit-logger (#562) 2025-09-18 17:46:46 +08:00
Copilot
e5f0760009 Fix entrypoint.sh incorrectly passing logs directory as data volume with improved separation (#561)
* Initial plan

* Fix entrypoint.sh: separate log directory from data volumes

Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>

* Improve separation: use functions and RUSTFS_OBS_LOG_DIRECTORY env var

Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>
2025-09-18 17:05:14 +08:00
houseme
a6c211f4ea Feature/add dns logs (#558)
* add logs

* improve code for dns and logger
2025-09-18 12:00:43 +08:00
shiro.lee
f049c656d9 fix: list_objects does not return common_prefixes field. (#543) (#554)
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2025-09-18 07:27:37 +08:00
majinghe
65dd947350 add tls support for docker compose (#553)
* add tls support for docker compose

* update docker compose file with comment
2025-09-17 22:45:23 +08:00
0xdx2
57f082ee2b fix: enforce max-keys limit to 1000 in S3 implementation (#549)
Co-authored-by: damon <damonxue2@gmail.com>
2025-09-16 18:02:24 +08:00
weisd
ae7e86d7ef refactor: simplify initialization flow and modernize string formatting (#548) 2025-09-16 15:44:50 +08:00
houseme
a12a3bedc3 feat(obs): optimize WriteMode selection logic in init_telemetry (#546)
- Refactor WriteMode selection to ensure all variables moved into thread closures are owned types, preventing lifetime issues.
- Simplify and clarify WriteMode assignment for production and non-production environments.
- Improve code readability and maintainability for logger initialization.
2025-09-16 08:25:37 +08:00
Copilot
cafec06b7e [Optimization] Enhance obs module telemetry.rs with environment-aware logging and production security (#539)
* Initial plan

* Implement environment-aware logging with production stdout auto-disable

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* add mimalloc crate

* fix

* improve code

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2025-09-15 14:52:20 +08:00
Parm Gill
1770679e66 Adding a toggle for update check (#532) 2025-09-14 22:26:48 +08:00
jon
a4fbf596e6 add startup logo (#528)
* add startup logo

* Replace logo ASCII art in main.rs

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2025-09-14 12:04:00 +08:00
houseme
3f717292bf feat(console): support accessing console endpoint via port 9000 (#533)
* fix(main): update dns_init startup logic and remove unused code in http.rs

- Refactored the DNS resolver initialization logic in `main.rs` for improved startup reliability and error handling.
- Removed unused code from `http.rs` to keep the codebase clean and maintainable.

* feat(console): support accessing console endpoint via port 9000

- Added compatibility to allow console access through port 9000.
- Improved endpoint detection and routing for console service on standard and custom ports.
- Enhanced user experience for environments using port 9000 as the default access point.
2025-09-14 01:14:14 +08:00
houseme
73f0ecbf8f fix(main): update dns_init startup logic and remove unused code in http.rs (#531)
- Refactored the DNS resolver initialization logic in `main.rs` for improved startup reliability and error handling.
- Removed unused code from `http.rs` to keep the codebase clean and maintainable.
2025-09-13 23:43:25 +08:00
houseme
0c3079ae5e remove deps (#529) 2025-09-13 21:06:40 +08:00
majinghe
ebf30b0db5 update docker compose usage guidence in READEME file (#526) 2025-09-13 16:24:36 +08:00
Copilot
29c004d935 feat: enhance console separation with enterprise-grade security, monitoring, and advanced tower-http integration (#513)
* Initial plan

* feat: implement console service separation from endpoint

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: add console separation documentation and tests

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: enhance console separation with configurable CORS and improved Docker support

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: implement enhanced console separation with security hardening and monitoring

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* refactor: implement console TLS following endpoint logic and improve configuration

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* add tower-http feature "timeout|limit"

* add dependencies crates `axum-server`

* refactor: reconstruct console server with enhanced tower-http features and environment variables

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* upgrade dep

* improve code for dns and console port `:9001`

* improve code

* fix

* docs: comprehensive improvement of console separation documentation and Docker deployment standards

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fmt

* add logs

* improve code for Config handler

* remove logs

* fix

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-09-13 14:48:14 +08:00
majinghe
4595bf7db6 fix docker compose running with no such file error (#519)
* fix docker compose running with no such file error

* fix observability docker compose
2025-09-13 13:04:06 +08:00
guojidan
f372ccf4a8 disable pprof on win (#524)
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-09-12 18:43:45 +08:00
guojidan
9ce867f585 feat(lock): Optimize lock management performance in high-concurrency scenarios (#523)
Increase the size of the notification pool to reduce the thundering herd effect under high concurrency
Implement an adaptive timeout mechanism that dynamically adjusts based on system load and priority
Add a lock protection mechanism to prevent premature cleanup of active locks
Add lock acquisition methods for high-priority and critical-priority locks
Improve the cleanup strategy to be more conservative under high load
Add detailed debug logs to assist in diagnosing lock issues

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-09-12 18:17:07 +08:00
guojidan
124c31a68b refactor(profiling): Remove performance profiling support for Windows and optimize dependency management (#518)
Remove the pprof performance profiling functionality on the Windows platform, as this platform does not support the relevant features
Move the pprof dependency to the platform-specific configuration for non-Windows systems
Update the performance profiling endpoint handling logic to distinguish between platform support statuses
Add the CLAUDE.md document to explain project build and architecture information

Signed-off-by: RustFS Developer <dandan@rustfs.com>
Co-authored-by: RustFS Developer <dandan@rustfs.com>
2025-09-12 09:11:44 +08:00
guojidan
62a01f3801 Performance: improve (#514)
* Performance: improve

Signed-off-by: junxiang Mu <1948535941@qq.com>

* remove dirty

Signed-off-by: junxiang Mu <1948535941@qq.com>

* fix some err

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-09-11 19:48:28 +08:00
weisd
70e6bec2a4 feat:admin auth (#512)
* feat:admin auth

* fix:#509
2025-09-11 16:49:07 +08:00
guojidan
cf863ba059 feat(lock): Add support for disabling lock manager (#511)
* feat(lock): Add support for disabling lock manager
Implement control of lock system activation and deactivation via environment variables
Add DisabledLockManager for lock-free operation scenarios
Introduce LockManager trait to uniformly manage different lock managers

Signed-off-by: junxiang Mu <1948535941@qq.com>

* refactor(lock): Optimize implementation of global lock manager and parsing of boolean environment variables
Refactor the implementation of the global lock manager: wrap FastObjectLockManager with Arc and add the as_fast_lock_manager method
Extract the boolean environment variable parsing logic into an independent function parse_bool_env_var

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-09-11 13:46:06 +08:00
guojidan
d4beb1cc0b Fix lock (#510)
* Refactor: reimplement lock

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Fix: fix test case failed

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Improve: lock pref

Signed-off-by: junxiang Mu <1948535941@qq.com>

* fix(lock): Fix resource cleanup issue when batch lock acquisition fails
Ensure that the locks already acquired are properly released when batch lock acquisition fails to avoid memory leaks
Improve the lock protection mechanism to prevent double release issues
Add complete Apache license declarations to all files

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-09-11 12:10:35 +08:00
0xdx2
971e74281c fix:Fix some errors tested in mint (#507)
* refactor: replace new_object_layer_fn with get_validated_store for bucket validation

* feat: add validation for object tagging limits and uniqueness

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* feat: add EntityTooSmall error for multipart uploads and update error handling

* feat: validate max_parts input range for S3 multipart uploads

* Update rustfs/src/storage/ecfs.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: optimize tag key and value length validation checks

---------

Co-authored-by: damon <damonxue2@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-09-10 22:22:29 +08:00
Copilot
ca9a2b6ab9 feat: Implement enhanced DNS resolver with hickory-resolver, TLS support, and layered fallback for Kubernetes environments (#505)
* Initial plan

* feat: Implement layered DNS resolver with caching and validation

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: Integrate DNS resolver into main application and fix formatting

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: Implement enhanced DNS resolver with Moka cache and layered fallback

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: Implement hickory-resolver with TLS support for enhanced DNS resolution

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* upgrade

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-09-10 21:16:33 +08:00
houseme
4e00110bfe add bucket notification configuration (#502) 2025-09-10 00:56:27 +08:00
安正超
9c97524c3b feat: consolidate AI rules into unified AGENTS.md (#501)
- Merge all AI rules from .rules.md, .cursorrules, and CLAUDE.md into AGENTS.md
- Add competitor keyword prohibition rules (minio, ceph, swift, etc.)
- Simplify rules by removing overly detailed code examples
- Integrate new development principles as highest priority
- Remove old tool-specific rule files
- Fix clippy warnings for format string improvements
2025-09-09 21:36:34 +08:00
guojidan
14a8802ce7 Fix: fix collect usage data (#500)
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-09-09 18:39:51 +08:00
guojidan
9d5ed1acac Feature/scanner performance optimization (#498)
* Refactor: reimplement scanner

Signed-off-by: RustFS Developer <dandan@rustfs.com>

* comment lock

Signed-off-by: junxiang Mu <1948535941@qq.com>

* remove dirty file

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Fix: fix rebase

* fix(scanner): Improve error handling and logging

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: RustFS Developer <dandan@rustfs.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: RustFS Developer <dandan@rustfs.com>
2025-09-08 18:35:45 +08:00
0xdx2
44f3eb7244 Fix: add support for additional AWS S3 storage classes and validation logic (#487)
* Fix: add pagination fields to S3 response

* Fix: add support for additional AWS S3 storage classes and validation logic

* Fix: improve handling of optional fields in S3 response

---------

Co-authored-by: DamonXue <damonxue2@gmail.com>
2025-09-05 09:50:41 +08:00
weisd
01b2623f66 Fix/response (#485)
* fix:list_parts response

* fix:list_objects skip delete_marker
2025-09-03 17:52:31 +08:00
dependabot[bot]
cf4d63795f build(deps): bump crc-fast from 1.4.0 to 1.5.0 in the dependencies group (#481)
Bumps the dependencies group with 1 update: [crc-fast](https://github.com/awesomized/crc-fast-rust).


Updates `crc-fast` from 1.4.0 to 1.5.0
- [Release notes](https://github.com/awesomized/crc-fast-rust/releases)
- [Changelog](https://github.com/awesomized/crc-fast-rust/blob/main/CHANGELOG.md)
- [Commits](https://github.com/awesomized/crc-fast-rust/compare/1.4.0...1.5.0)

---
updated-dependencies:
- dependency-name: crc-fast
  dependency-version: 1.5.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: weisd <im@weisd.in>
2025-09-03 17:30:08 +08:00
WenTao
0efc818635 Fix Windows path separator issue using PathBuf (#482)
* Update mod.rs

The following code uses a separator that is not compatible with Windows:

format!("{}/{}", file_config.path.clone(), rustfs_config::DEFAULT_SINK_FILE_LOG_FILE)


Change it to the following code:


std::path::Path::new(&file_config.path)
    .join(rustfs_config::DEFAULT_SINK_FILE_LOG_FILE)
    .to_string_lossy()
    .to_string()

* Replaced format! macro with PathBuf::join to fix path separator issue on Windows.Tested on Windows 10 with Rust 1.85.0, paths now correctly use \ separator.
2025-09-03 15:25:08 +08:00
weisd
c9d26c6e88 Fix/delete version (#484)
* fix:delete_version

* fix:test_lifecycle_expiry_basic

---------

Co-authored-by: likewu <likewu@126.com>
2025-09-03 15:12:58 +08:00
likewu
087df484a3 Fix/ilm (#478) 2025-09-02 18:18:26 +08:00
houseme
04bf4b0f98 feat: add S3 object legal hold and retention management APIs (#476)
* add bucket rule

* translation

* improve code for event notice add rule
2025-09-02 00:14:10 +08:00
likewu
7462be983a Feature up/ilm (#470)
* fix delete-marker expiration. add api_restore.

* time retry object upload

* lock file

* make fmt

* restore object

* serde-rs-xml -> quick-xml

* scanner_item prefix object_name

* object_path

* object_name

* fi version_purge_status

* old_dir None

Co-authored-by: houseme <housemecn@gmail.com>
2025-09-01 16:11:28 +08:00
houseme
5264503e47 build(deps): bump aws-config and clap upgrade version (#472) 2025-08-30 20:30:46 +08:00
dependabot[bot]
3b8cb0df41 build(deps): bump tracing-subscriber in the cargo group (#471)
Bumps the cargo group with 1 update: [tracing-subscriber](https://github.com/tokio-rs/tracing).


Updates `tracing-subscriber` from 0.3.19 to 0.3.20
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.19...tracing-subscriber-0.3.20)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-version: 0.3.20
  dependency-type: direct:production
  dependency-group: cargo
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-30 19:02:26 +08:00
houseme
9aebef31ff refactor(admin/event): optimize notification target routing and logic handling (#463)
* add

* fix

* add target arns list

* improve code for arns

* upgrade crates version

* fix

* improve import code mod.rs

* fix

* improve

* improve code

* improve code

* fix

* fmt
2025-08-27 09:39:25 +08:00
zzhpro
c2d782bed1 feat: support conditional writes (#409)
* feat: support conditional writes

* refactor: avoid using unwrap

* fix: obtain lock before check in CompleteMultiPartUpload

* refactor: do not obtain a lock when getting object meta

* fix: avoid using unwrap and modifying incoming arguments

* test: add e2e tests for conditional writes

---------

Co-authored-by: guojidan <63799833+guojidan@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2025-08-25 18:35:24 -07:00
likewu
e00f5be746 Fix/addtier (#454)
* fix retry

* fmt

* fix

* fix

* fix

---------

Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2025-08-25 10:24:48 +08:00
shiro.lee
e23297f695 fix: add the default port number to the given server domains (#373) (#458) 2025-08-25 07:49:36 +08:00
0xdx2
d6840a6e04 feat: add support for range requests in upload_part_copy and implement parse_copy_source_range function (#453)
* feat: add support for range requests in upload_part_copy and implement parse_copy_source_range function

* style: format debug and error logging for improved readability

* feat: implement parse_copy_source_range function and improve error handling in range requests

* Update rustfs/src/storage/ecfs.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: correct return type in parse_copy_source_range function

* fix: remove unnecessary unwrap in parse_copy_source_range tests

* fix: simplify etag comparison in copy condition validation

---------

Co-authored-by: DamonXue <damonxue2@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2025-08-24 10:54:48 +08:00
houseme
3557a52dc4 Potential fix for code scanning alert no. 7: Workflow does not contain permissions (#457)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-08-24 10:10:04 +08:00
houseme
fd2aab2bd9 fix:revet #443 #446 (#452)
* fix: revet #443 #446

* fix
2025-08-23 17:30:06 +08:00
houseme
f1c50fcb74 fix:Workflow does not contain permissions (#451) 2025-08-23 12:35:23 +08:00
houseme
bdcba3460e Potential fix for code scanning alert no. 13: Code injection (#447)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2025-08-23 10:05:00 +08:00
houseme
8857f31b07 Comment out error log for missing subscribers (#448) 2025-08-22 21:15:46 +08:00
loverustfs
5b85bf7a00 lock: dedicate unlock worker to thread runtime; robust fallback in Drop (#446)
* lock: dedicate unlock worker to thread runtime; robust fallback in Drop

* Update crates/lock/src/guard.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update crates/lock/src/guard.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update crates/lock/src/guard.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Refactor logging in UNLOCK_TX error handling

Removed redundant logging of lock_id in warning message.

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-08-22 16:51:56 +08:00
loverustfs
46bd75c0f8 ahm(scanner): throttle scanning, skip recently-modified objects, and … (#443)
* ahm(scanner): throttle scanning, skip recently-modified objects, and gate missing-object heals to deep mode; adjust conservative defaults

Signed-off-by: loverustfs <hello@rustfs.com>

* ecstore: enable virtual-host AUTO heuristics and URL building; signer: fix SigV2 canonical resource for vhost; add unit tests

* ecstore: AUTO virtual-host style URL selection; signer: SigV2 canonical resource fixes for vhost; tests added.\nahm: fix clippy drop_non_drop; integration tests robust to existing bucket; ignore flaky lifecycle test.\nMakefile: test target falls back to cargo test when nextest missing.\npre-commit: all checks green.

---------

Signed-off-by: loverustfs <hello@rustfs.com>
2025-08-22 16:03:29 +08:00
houseme
5fc5dd0fd9 Remove rustfs-gui module (#445)
This commit completely removes the rustfs-gui module from the project. The deletion includes:

- All source code files (*.rs) and associated resources
- GUI-specific dependencies from Cargo.toml
- Build scripts and configuration files specific to the GUI module
- Documentation and assets related to the graphical interface

The removal is performed because:
- The GUI component is no longer maintained
- Focus is shifting to core functionality and CLI interface
- Limited resources available for GUI development and maintenance

The core filesystem functionality remains available through the rustfs library and CLI interface.
2025-08-22 09:15:22 +08:00
houseme
adc07e5209 feat(targets): extract targets module into a standalone crate (#441)
* init audit logger module

* add audit webhook default config kvs

* feat: Add comprehensive tests for authentication module (#309)

* feat: add comprehensive tests for authentication module

- Add 33 unit tests covering all public functions in auth.rs
- Test IAMAuth struct creation and secret key validation
- Test check_claims_from_token with various credential types and scenarios
- Test session token extraction from headers and query parameters
- Test condition values generation for different user types
- Test query parameter parsing with edge cases
- Test Credentials helper methods (is_expired, is_temp, is_service_account)
- Ensure tests handle global state dependencies gracefully
- All tests pass successfully with 100% coverage of testable functions

* style: fix code formatting issues

* Add verification script for checking PR branch statuses and tests

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

* fix: resolve clippy uninlined format args warning

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* feat: add basic tests for core storage module (#313)

* feat: add basic tests for core storage module

- Add 6 unit tests for FS struct and basic functionality
- Test FS creation, Debug and Clone trait implementations
- Test RUSTFS_OWNER constant definition and values
- Test S3 error code creation and handling
- Test compression format detection for common file types
- Include comprehensive documentation about integration test needs

Note: Full S3 API testing requires complex setup with storage backend,
global configuration, and network infrastructure - better suited for
integration tests rather than unit tests.

* style: fix code formatting issues

* fix: resolve clippy warnings in storage tests

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* feat: add tests for admin handlers module (#314)

* feat: add tests for admin handlers module

- Add 5 new unit tests for admin handler functionality
- Test AccountInfo struct creation, serialization and default values
- Test creation of all admin handler structs (13 handlers)
- Test HealOpts JSON serialization and deserialization
- Test HealOpts URL encoding/decoding with proper field types
- Maintain existing test while adding comprehensive coverage
- Include documentation about integration test requirements

All tests pass successfully with proper error handling for complex dependencies.

* style: fix code formatting issues

* fix: resolve clippy warnings in admin handlers tests

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* build(deps): bump the dependencies group with 3 updates (#326)

* perf: avoid transmitting parity shards when the object is good (#322)

* upgrade version

* Fix: fix data integrity check

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Fix: Separate Clippy's fix and check commands into two commands.

Signed-off-by: junxiang Mu <1948535941@qq.com>

* fix: miss inline metadata (#345)

* Update dependabot.yml

* fix: Fixed an issue where the list_objects_v2 API did not return dire… (#352)

* fix: Fixed an issue where the list_objects_v2 API did not return directory names when they conflicted with file names in the same bucket (e.g., test/ vs. test.txt, aaa/ vs. aaa.csv) (#335)

* fix: adjusted the order of directory listings

* init

* fix

* fix

* feat: add docker usage for rustfs mcp (#365)

* feat: enhance metadata extraction with object name for MIME type detection

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Feature: lock support auto release

Signed-off-by: junxiang Mu <1948535941@qq.com>

* improve lock

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Fix: fix scanner detect

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Fix: clippy && fmt

Signed-off-by: junxiang Mu <1948535941@qq.com>

* refactor(ecstore): Optimize memory usage for object integrity verification

Change the object integrity verification from reading all data to streaming processing to avoid memory overflow caused by large objects.

Modify the TLS key log check to use environment variables directly instead of configuration constants.

Add memory limits for object data reading in the AHM module.

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Chore: reduce PR template checklist

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Chore: remove comment code (#376)

Signed-off-by: junxiang Mu <1948535941@qq.com>

* chore: upgrade actions/checkout from v4 to v5 (#381)

* chore: upgrade actions/checkout from v4 to v5

- Update GitHub Actions checkout action version
- Ensure compatibility with latest workflow features
- Maintain existing checkout behavior and configuration

* upgrade version

* fix

* add and improve code for notify

* feat: extend rustfs mcp with bucket creation and deletion (#416)

* feat: extend rustfs mcp with bucket creation and deletion

* update file to fix pipeline error

* change variable name to fix pipeline error

* fix(ecstore): add async-recursion to resolve nightly trait solver reg… (#415)

* fix(ecstore): add async-recursion to resolve nightly trait solver regression

The newest nightly compiler switched to the new trait solver, which
currently rejects async recursive functions that were previously accepted.
This causes the following compilation failures:

- `LocalDisk::delete_file()`
- `LocalDisk::scan_dir()`

Add `async-recursion` as a workspace dependency and annotate both functions with `#[async_recursion]` so that the crate compiles cleanly with the latest nightly and will continue to build once the new solver lands in stable.

Signed-off-by: reigadegr <2722688642@qq.com>

* fix: resolve duplicate bound error in scan_dir function

Replaced inline trait bounds with where clause to avoid duplication caused by macro expansion.

Signed-off-by: reigadegr <2722688642@qq.com>

---------

Signed-off-by: reigadegr <2722688642@qq.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>

* fix:make bucket exists (#428)

* feat: include user-defined metadata in S3 response (#431)

* fix: simplify Docker entrypoint following efficient user switching pattern (#421)

* fix: simplify Docker entrypoint following efficient user switching pattern

- Remove ALL file permission modifications (no chown at all)
- Use chroot --userspec or gosu to switch user context
- Extremely simple and fast implementation
- Zero filesystem modifications for permissions

Fixes #388

* Update entrypoint.sh

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update entrypoint.sh

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update entrypoint.sh

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* wip

* wip

* wip

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* docs: update doc/docker-data-dir README.md (#432)

* add targets crates

* feat(targets): extract targets module into a standalone crate

- Move all target-related code (MQTT, Webhook, etc.) into a new `targets` crate
- Update imports and dependencies to reference the new crate
- Refactor interfaces to ensure compatibility with the new crate structure
- Adjust Cargo.toml and workspace configuration accordingly

* fix

* fix

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>
Signed-off-by: reigadegr <2722688642@qq.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: zzhpro <56196563+zzhpro@users.noreply.github.com>
Co-authored-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: weisd <im@weisd.in>
Co-authored-by: shiro.lee <69624924+shiroleeee@users.noreply.github.com>
Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
Co-authored-by: guojidan <63799833+guojidan@users.noreply.github.com>
Co-authored-by: reigadegr <103645642+reigadegr@users.noreply.github.com>
Co-authored-by: 0xdx2 <xuedamon2@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-08-21 22:33:07 +08:00
安正超
357cced49c Replace prints with logs and fix grammar (#437)
* refactor: replace print statements with proper logging and fix grammar

- Fix English grammar errors in existing log messages
- Add tracing imports where needed
- Improve log message clarity and consistency
- Follow project logging best practices using tracing crate

* fix: resolve clippy warnings and format code

- Fix unused import warnings by making test imports conditional with #[cfg(test)]
- Fix unused variable warning by prefixing with underscore
- Run cargo fmt to fix formatting issues
- Ensure all code passes clippy checks with -D warnings flag

* refactor: move tracing::debug import into test module

Move the tracing::debug import from file-level #[cfg(test)] into the test module itself for better code organization and consistency with other test modules

* Checkpoint before follow-up message

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

* refactor: move tracing::debug import into test module in user_agent.rs

Complete the refactoring by moving the tracing::debug import from file-level #[cfg(test)] into the test module for consistency across all test files

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2025-08-21 05:49:40 +08:00
Csrayz
a104c33974 [FEAT] add error message (#435)
When the bucket is not found, return a helpful message
2025-08-20 23:49:59 +08:00
安正超
516e00f15f fix: Dockerfile with error permission change. (#436)
* fix: dockerfile and permission error.

* fix: dockerfile and permission error.
2025-08-20 23:32:03 +08:00
安正超
a64c3c28b8 docs: update doc/docker-data-dir README.md (#432) 2025-08-20 00:00:20 +08:00
安正超
e9c9a2d1f2 fix: simplify Docker entrypoint following efficient user switching pattern (#421)
* fix: simplify Docker entrypoint following efficient user switching pattern

- Remove ALL file permission modifications (no chown at all)
- Use chroot --userspec or gosu to switch user context
- Extremely simple and fast implementation
- Zero filesystem modifications for permissions

Fixes #388

* Update entrypoint.sh

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update entrypoint.sh

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update entrypoint.sh

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* wip

* wip

* wip

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-08-19 22:58:54 +08:00
0xdx2
3ebab98d2d feat: include user-defined metadata in S3 response (#431) 2025-08-19 22:09:50 +08:00
weisd
10c949af62 fix:make bucket exists (#428) 2025-08-19 16:14:59 +08:00
reigadegr
4a3325276d fix(ecstore): add async-recursion to resolve nightly trait solver reg… (#415)
* fix(ecstore): add async-recursion to resolve nightly trait solver regression

The newest nightly compiler switched to the new trait solver, which
currently rejects async recursive functions that were previously accepted.
This causes the following compilation failures:

- `LocalDisk::delete_file()`
- `LocalDisk::scan_dir()`

Add `async-recursion` as a workspace dependency and annotate both functions with `#[async_recursion]` so that the crate compiles cleanly with the latest nightly and will continue to build once the new solver lands in stable.

Signed-off-by: reigadegr <2722688642@qq.com>

* fix: resolve duplicate bound error in scan_dir function

Replaced inline trait bounds with where clause to avoid duplication caused by macro expansion.

Signed-off-by: reigadegr <2722688642@qq.com>

---------

Signed-off-by: reigadegr <2722688642@qq.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2025-08-18 20:58:05 +08:00
majinghe
c5f6c66f72 feat: extend rustfs mcp with bucket creation and deletion (#416)
* feat: extend rustfs mcp with bucket creation and deletion

* update file to fix pipeline error

* change variable name to fix pipeline error
2025-08-18 09:06:55 +08:00
shiro.lee
c7c149975b fix: the automatic logout issue and user list display failure on Windows systems (#353) (#343) (#403)
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2025-08-14 00:20:27 +08:00
安正超
d552210b59 feat: translate chinese to english (#402)
* Checkpoint before follow-up message

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

* Translate project documentation and comments from Chinese to English

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

* Fix typo: "unparseable" to "unparsable" in version test comment

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

* Refactor compression test code with minor syntax improvements

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2025-08-14 00:19:01 +08:00
安正超
581607da6a feat: optimize AI rules with unified .rules.md (#401)
* feat: optimize AI rules with unified .rules.md and entry points

- Create .rules.md as the central AI coding rules file
- Add .copilot-rules.md as GitHub Copilot entry point
- Add CLAUDE.md as Claude AI entry point
- Incorporate principles from rustfs.com project
- Add three critical rules:
  1. Use English for all code comments and documentation
  2. Clean up temporary scripts after use
  3. Only make confident modifications

* Update CLAUDE.md

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2025-08-14 00:18:09 +08:00
安正超
e95107f7d6 fix: separate RELEASE tag and VERSION in Docker build (#399)
- RELEASE: GitHub release tag without 'v' prefix (e.g., 1.0.0-alpha.42)
- VERSION: filename version with 'v' prefix (e.g., v1.0.0-alpha.42)
- Download URL uses RELEASE for path, VERSION for filename
- Fixes incorrect URL generation that was adding extra 'v' prefix

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2025-08-13 22:49:48 +08:00
安正超
a693cb52f3 feat: change Docker build to download from GitHub releases instead of dl.rustfs.com (#398)
- Modified Dockerfile to download pre-built binaries from GitHub releases
- For latest releases, use GitHub API to find the correct download URL
- For specific versions, construct the GitHub release URL directly
- Updated docker-buildx.sh script messages to reflect new download source
- This change addresses security concerns about potential tampering with binaries from dl.rustfs.com

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2025-08-13 22:00:41 +08:00
houseme
2c7366038e modify protobuf version from to 2025-08-13 01:01:50 +08:00
houseme
1cc6dfde87 modify protobuf version from 31.1 to 31.0 2025-08-13 00:58:22 +08:00
weisd
387f4faf78 fix:rm object versions (#385) 2025-08-12 15:33:47 +08:00
houseme
0f7093c5f9 chore: upgrade actions/checkout from v4 to v5 (#381)
* chore: upgrade actions/checkout from v4 to v5

- Update GitHub Actions checkout action version
- Ensure compatibility with latest workflow features
- Maintain existing checkout behavior and configuration

* upgrade version
2025-08-12 11:17:58 +08:00
guojidan
6a5c0055e7 Chore: remove comment code (#376)
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-08-11 08:57:33 +08:00
guojidan
76288f2501 Merge pull request #372 from guojidan/fix-scanner
refactor(ecstore): Optimize memory usage for object integrity verification
2025-08-10 06:44:05 -07:00
junxiang Mu
3497ccfada Chore: reduce PR template checklist
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-08-10 21:29:30 +08:00
junxiang Mu
24e3d3a2ce refactor(ecstore): Optimize memory usage for object integrity verification
Change the object integrity verification from reading all data to streaming processing to avoid memory overflow caused by large objects.

Modify the TLS key log check to use environment variables directly instead of configuration constants.

Add memory limits for object data reading in the AHM module.

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-08-10 21:24:15 +08:00
guojidan
ebad748cdc Merge pull request #368 from guojidan/fix-sql
Fix scanner && lock
2025-08-09 06:37:36 -07:00
junxiang Mu
b7e56ed92c Fix: clippy && fmt
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-08-09 21:16:56 +08:00
junxiang Mu
4811632751 Fix: fix scanner detect
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-08-09 21:06:17 +08:00
junxiang Mu
374a702f04 improve lock
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-08-09 21:05:46 +08:00
junxiang Mu
e369e9f481 Feature: lock support auto release
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-08-09 17:52:08 +08:00
guojidan
fe2e4a2274 Merge pull request #367 from guojidan/fix-sql
feat: enhance metadata extraction with object name for MIME type dete…
2025-08-08 21:53:12 -07:00
junxiang Mu
b391272e94 feat: enhance metadata extraction with object name for MIME type detection
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-08-09 12:29:04 +08:00
majinghe
c55c7a6373 feat: add docker usage for rustfs mcp (#365) 2025-08-08 17:18:20 +08:00
houseme
67f1c371a9 upgrade version 2025-08-08 11:33:32 +08:00
guojidan
d987686c14 feat(lifecycle): Implement object lifecycle management functionality (#358)
* feat(lifecycle): Implement object lifecycle management functionality

Add a lifecycle module to automatically handle object expiration and transition during scanning
Modify the file metadata cache module to be publicly visible to support lifecycle operations
Adjust the scanning interval to a shorter time for testing lifecycle rules
Implement the parsing and execution logic for S3 lifecycle configurations
Add integration tests to verify the lifecycle expiration functionality
Update dependencies to support the new lifecycle features

Signed-off-by: junxiang Mu <1948535941@qq.com>

* fix cargo dependencies

Signed-off-by: junxiang Mu <1948535941@qq.com>

* fix fmt

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
2025-08-08 10:51:02 +08:00
houseme
48a9707110 fix: add tokio-test (#363)
* fix: add tokio-test

* fix: "called `unwrap` on `v` after checking its variant with `is_some`"

    = help: try using `if let` or `match`
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_unwrap
    = note: `-D clippy::unnecessary-unwrap` implied by `-D warnings`
    = help: to override `-D warnings` add `#[allow(clippy::unnecessary_unwrap)]`

* fmt

* set toolchain 1.88.0

* fmt

* fix: cliip
2025-08-08 10:23:22 +08:00
bestgopher
b89450f54d replace make with just (#349) 2025-08-07 22:37:05 +08:00
houseme
e0c99bced4 chore: add tls log and removing unused crates (#359)
* chore: add tls log

* improve code for http

* improve code dependencies for `cargo.toml` and removing unused crates

* modify name

* improve code

* fix

* Update crates/config/src/constants/env.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* improve code

* fix

* add `is_enabled` and `is_disabled`

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-08-07 19:02:09 +08:00
houseme
130f85a575 chore: add tls log (#357) 2025-08-07 17:33:57 +08:00
shiro.lee
c42fbed3d2 fix: Fixed an issue where the list_objects_v2 API did not return dire… (#352)
* fix: Fixed an issue where the list_objects_v2 API did not return directory names when they conflicted with file names in the same bucket (e.g., test/ vs. test.txt, aaa/ vs. aaa.csv) (#335)

* fix: adjusted the order of directory listings
2025-08-07 11:05:05 +08:00
安正超
fd539f0f0a Update dependabot.yml 2025-08-06 22:55:52 +08:00
weisd
9aba89a12c fix: miss inline metadata (#345) 2025-08-06 11:45:23 +08:00
guojidan
7b27b29e3a Merge pull request #344 from guojidan/bug-fix
Fix: fix data integrity check
2025-08-05 20:31:10 -07:00
junxiang Mu
7ef014a433 Fix: Separate Clippy's fix and check commands into two commands.
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-08-06 11:22:08 +08:00
junxiang Mu
1b88714d27 Fix: fix data integrity check
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-08-06 11:03:29 +08:00
zzhpro
b119894425 perf: avoid transmitting parity shards when the object is good (#322) 2025-08-02 14:37:43 +08:00
dependabot[bot]
a37aa664f5 build(deps): bump the dependencies group with 3 updates (#326) 2025-08-02 06:44:16 +08:00
安正超
9b8abbb009 feat: add tests for admin handlers module (#314)
* feat: add tests for admin handlers module

- Add 5 new unit tests for admin handler functionality
- Test AccountInfo struct creation, serialization and default values
- Test creation of all admin handler structs (13 handlers)
- Test HealOpts JSON serialization and deserialization
- Test HealOpts URL encoding/decoding with proper field types
- Maintain existing test while adding comprehensive coverage
- Include documentation about integration test requirements

All tests pass successfully with proper error handling for complex dependencies.

* style: fix code formatting issues

* fix: resolve clippy warnings in admin handlers tests

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2025-08-02 06:38:35 +08:00
安正超
3e5a48af65 feat: add basic tests for core storage module (#313)
* feat: add basic tests for core storage module

- Add 6 unit tests for FS struct and basic functionality
- Test FS creation, Debug and Clone trait implementations
- Test RUSTFS_OWNER constant definition and values
- Test S3 error code creation and handling
- Test compression format detection for common file types
- Include comprehensive documentation about integration test needs

Note: Full S3 API testing requires complex setup with storage backend,
global configuration, and network infrastructure - better suited for
integration tests rather than unit tests.

* style: fix code formatting issues

* fix: resolve clippy warnings in storage tests

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2025-08-02 06:37:31 +08:00
安正超
d5aef963f9 feat: Add comprehensive tests for authentication module (#309)
* feat: add comprehensive tests for authentication module

- Add 33 unit tests covering all public functions in auth.rs
- Test IAMAuth struct creation and secret key validation
- Test check_claims_from_token with various credential types and scenarios
- Test session token extraction from headers and query parameters
- Test condition values generation for different user types
- Test query parameter parsing with edge cases
- Test Credentials helper methods (is_expired, is_temp, is_service_account)
- Ensure tests handle global state dependencies gracefully
- All tests pass successfully with 100% coverage of testable functions

* style: fix code formatting issues

* Add verification script for checking PR branch statuses and tests

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

* fix: resolve clippy uninlined format args warning

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2025-08-02 06:36:45 +08:00
houseme
6c37e1cb2a refactor: replace lazy_static with LazyLock (#318)
* refactor: replace `lazy_static` with `LazyLock`

Replace `lazy_static` with `LazyLock`.

Compile time may reduce a little.

See https://github.com/rust-lang-nursery/lazy-static.rs/issues/214

* fmt

* fix
2025-07-31 14:25:39 +08:00
0xdx2
e9d7e211b9 fix:Add etag to get object response
fix:Add etag to  get object response
2025-07-31 11:31:15 +08:00
0xdx2
45bbd1e5c4 Add etag to get object response
Add etag to  get object response
2025-07-31 11:20:10 +08:00
0xdx2
57d196771a Merge pull request #312 from rustfs/0xdx2-s3s_xmlns
fix: update s3s version to solve xml namespace type attribute bug.
2025-07-30 23:53:56 +08:00
0xdx2
6202f50e15 fix: update s3s version to solve xml namespace type attribute bug.
update s3s version to solve xml namespace type attribute bug.
2025-07-30 23:40:43 +08:00
houseme
c5df1f92c2 refactor: replace lazy_static with LazyLock and notify crate registry create_targets_from_config (#311)
* improve code for notify

* improve code for logger and fix typo (#272)

* Add GNU to  build.yml (#275)

* fix unzip error

* fix url change error

fix url change error

* Simplify user experience and integrate console and endpoint

Simplify user experience and integrate console and endpoint

* Add gnu to  build.yml

* upgrade version

* feat: add `cargo clippy --fix --allow-dirty` to pre-commit command (#282)

Resolves #277

- Add --fix flag to automatically fix clippy warnings
- Add --allow-dirty flag to run on dirty Git trees
- Improves code quality in pre-commit workflow

* fix: the issue where preview fails when the path length exceeds 255 characters (#280)

* fix

* fix: improve Windows build support and CI/CD workflow (#283)

- Fix Windows zip command issue by using PowerShell Compress-Archive
- Add Windows support for OSS upload with ossutil
- Replace Chinese comments with English in build.yml
- Fix bash syntax error in package_zip function
- Improve code formatting and consistency
- Update various configuration files for better cross-platform support

Resolves Windows build failures in GitHub Actions.

* fix: update link in README.md leading to a 404 error (#285)

* add rustfs.spec for rustfs (#103)

add support on loongarch64

* improve cargo.lock

* build(deps): bump the dependencies group with 5 updates (#289)

Bumps the dependencies group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [hyper-util](https://github.com/hyperium/hyper-util) | `0.1.15` | `0.1.16` |
| [rand](https://github.com/rust-random/rand) | `0.9.1` | `0.9.2` |
| [serde_json](https://github.com/serde-rs/json) | `1.0.140` | `1.0.141` |
| [strum](https://github.com/Peternator7/strum) | `0.27.1` | `0.27.2` |
| [sysinfo](https://github.com/GuillaumeGomez/sysinfo) | `0.36.0` | `0.36.1` |


Updates `hyper-util` from 0.1.15 to 0.1.16
- [Release notes](https://github.com/hyperium/hyper-util/releases)
- [Changelog](https://github.com/hyperium/hyper-util/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/hyper-util/compare/v0.1.15...v0.1.16)

Updates `rand` from 0.9.1 to 0.9.2
- [Release notes](https://github.com/rust-random/rand/releases)
- [Changelog](https://github.com/rust-random/rand/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-random/rand/compare/rand_core-0.9.1...rand_core-0.9.2)

Updates `serde_json` from 1.0.140 to 1.0.141
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.140...v1.0.141)

Updates `strum` from 0.27.1 to 0.27.2
- [Release notes](https://github.com/Peternator7/strum/releases)
- [Changelog](https://github.com/Peternator7/strum/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Peternator7/strum/compare/v0.27.1...v0.27.2)

Updates `sysinfo` from 0.36.0 to 0.36.1
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GuillaumeGomez/sysinfo/compare/v0.36.0...v0.36.1)

---
updated-dependencies:
- dependency-name: hyper-util
  dependency-version: 0.1.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: rand
  dependency-version: 0.9.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: serde_json
  dependency-version: 1.0.141
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: strum
  dependency-version: 0.27.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: sysinfo
  dependency-version: 0.36.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* improve code for logger

* improve

* upgrade

* refactor: 优化构建工作流,统一 latest 文件处理和简化制品上传 (#293)

* Refactor: DatabaseManagerSystem as global

Signed-off-by: junxiang Mu <1948535941@qq.com>

* fix: fmt

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Test: add e2e_test for s3select

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Test: add test script for e2e

Signed-off-by: junxiang Mu <1948535941@qq.com>

* improve code for registry and intergation

* improve code for registry `create_targets_from_config`

* fix

* Feature up/ilm (#305)

* fix

* fix

* fix

* fix delete-marker expiration. add api_restore.

* fix

* time retry object upload

* lock file

* make fmt

* fix

* restore object

* fix

* fix

* serde-rs-xml -> quick-xml

* fix

* checksum

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* transfer lang to english

* upgrade clap version from 4.5.41 to 4.5.42

* refactor: replace `lazy_static` with `LazyLock`

* add router

* fix: modify comment

* improve code

* fix typos

* fix

* fix: modify name and fmt

* improve code for registry

* fix test

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: shiro.lee <69624924+shiroleeee@users.noreply.github.com>
Co-authored-by: Marco Orlandin <mipnamic@mipnamic.net>
Co-authored-by: zhangwenlong <zhangwenlong@loongson.cn>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: likewu <likewu@126.com>
2025-07-30 19:02:10 +08:00
wangsl
4f1770d3fe feat:add mcp integration (#300)
* add list_buckets mcp server

* add list_objects mcp

* add upload object mcp

* add get object mcp

* add list_buckets mcp server

* fix: resolve clippy warnings in rustfs-mcp-server

* fix: rename mcp package

* fix

* fix:remove useless comment

* feat:add mcp doc
2025-07-30 14:25:01 +08:00
likewu
d56cee26db Feature up/ilm (#305)
* fix

* fix

* fix

* fix delete-marker expiration. add api_restore.

* fix

* time retry object upload

* lock file

* make fmt

* fix

* restore object

* fix

* fix

* serde-rs-xml -> quick-xml

* fix

* checksum

* fix

* fix

* fix

* fix

* fix

* fix

* fix
2025-07-29 14:21:19 +08:00
weisd
56fd8132e9 fix:#303 returns empty when querying an empty or not dir (#304) 2025-07-28 16:17:40 +08:00
guojidan
35daa74430 Merge pull request #302 from guojidan/lock
Lock: add transactional
2025-07-28 12:00:44 +08:00
junxiang Mu
dc156fb4cd Fix: clippy
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-28 11:38:42 +08:00
junxiang Mu
de905a878c Cargo: use workspace dependence
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-28 11:02:40 +08:00
junxiang Mu
f3252f989b Test: Add e2e test case for lock transactional
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-28 11:00:10 +08:00
junxiang Mu
01a2afca9a lock: Add transactional
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-28 10:59:43 +08:00
guojidan
a4fe68ad21 Merge pull request #301 from guojidan/improve-sql
s3Select: add unit test case
2025-07-28 09:56:10 +08:00
junxiang Mu
c03f86b23c s3Select: add unit test case
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-28 09:19:47 +08:00
guojidan
5667f324ae Merge pull request #297 from guojidan/improve-sql
Test: Add e2e_test case for sql && add script for e2e_test
2025-07-25 17:16:41 +08:00
junxiang Mu
bcd806796f Test: add test script for e2e
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-25 16:52:06 +08:00
junxiang Mu
612404c47f Test: add e2e_test for s3select
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-25 15:07:44 +08:00
guojidan
85388262b3 Merge pull request #294 from guojidan/improve-sql
Refactor: DatabaseManagerSystem as global
2025-07-25 08:33:54 +08:00
junxiang Mu
25a4503285 fix: fmt
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-25 08:18:14 +08:00
安正超
526c4d5a61 refactor: 优化构建工作流,统一 latest 文件处理和简化制品上传 (#293) 2025-07-25 01:10:04 +08:00
junxiang Mu
addc964d56 Refactor: DatabaseManagerSystem as global
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 17:12:51 +08:00
loverustfs
371119f733 GNU to MUSL modify Dockerfile 2025-07-24 16:36:15 +08:00
guojidan
021abc0398 Merge pull request #292 from guojidan/Arc
Chore: remove dirty file(cache.rs)
2025-07-24 16:32:20 +08:00
junxiang Mu
0672b6dd3e Chore: remove dirty file(cache.rs)
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 14:57:48 +08:00
guojidan
1372dc2857 Merge pull request #288 from guojidan/scanner
Refactor: Scanner
2025-07-24 14:42:54 +08:00
houseme
77bc9af109 Update Cargo.toml 2025-07-24 14:14:12 +08:00
junxiang Mu
91b1c84430 rebase
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:18:05 +08:00
junxiang Mu
b667927216 fix fmt
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:28 +08:00
junxiang Mu
29795fac51 fix Cargo.toml
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:28 +08:00
junxiang Mu
2ce7e01f55 Chore: remove dirty file(heal)
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:27 +08:00
junxiang Mu
4fefd63a5b rebase
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu
2a8c46874d fix: auto heal when xl.meta lose
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu
b8b5511b68 fix: heal data part lose
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu
bdaee228db fix(ahm): adjust test expectations for missing xl.meta recovery scenario
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu
d562620e99 fix: implement uses_data_dir method
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu
69b0c828c9 fix: scanner add heal bucket
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu
2bfd1efb9b Fix: fix add heal_manager into scanner when scanner start
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu
0854e6b921 Chore: rename init_heal_manager_with_channel
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu
b907f4e61b refactor(ahm): remove obsolete scanner/data_usage.rs after data usage refactor
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu
6ec568459c chore: update admin handlers, lockfile, and minor fixes
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu
ea210d52dc refactor(heal): unify heal request interface, add disk field, update ahm/ecstore/common for erasure set healing
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:03 +08:00
junxiang Mu
3d3c6e4e06 chore(protos): update proto definitions, remove ns_scanner, fix codegen and formatting
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
junxiang Mu
e7d0a8d4b9 feat: integrate global metrics system into AHM scanner
- Add global metrics system to common crate for cross-module usage
- Integrate global metrics collection into AHM scanner operations
- Update ECStore to use common metrics system instead of local implementation
- Add chrono dependency to AHM crate for timestamp handling
- Re-export IlmAction from common metrics in ECStore lifecycle module
- Update scanner methods to use global metrics for cycle, disk, and volume scans
- Maintain backward compatibility with local metrics collector
- Fix clippy warnings and ensure proper code formatting

This change enables unified metrics collection across the entire RustFS system,
allowing better monitoring and observability of scanner operations.

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
junxiang Mu
7d3b2b774c fix heal disk
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
junxiang Mu
aed8f52423 refactor: integrate disk healing into erasure set healing
- Remove HealType::Disk and related disk-specific healing methods
- Integrate disk format healing into heal_erasure_set with include_format_heal option
- Update auto disk scanner to use ErasureSet heal type instead of Disk heal
- Fix disk status change event handling to use ErasureSet heal requests
- Add proper bucket list retrieval for auto healing scenarios
- Update data scanner to submit ErasureSet heal tasks for offline disks
- Remove duplicate healing logic between Disk and ErasureSet types
- Ensure all healing operations go through unified ErasureSet healing path
2025-07-24 12:12:49 +08:00
junxiang Mu
c49414f6ac fix: resolve test conflicts and improve data scanner functionality
- Fix multi-threaded test conflicts in AHM heal integration tests
- Remove global environment sharing to prevent test state pollution
- Fix test_all_disk_method by clearing global disk map before test
- Improve data scanner and cache value implementations
- Update dependencies and resolve clippy warnings

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
junxiang Mu
8e766b90cd feat: implement heal channel mechanism for admin-ahm communication
- Add global unbounded channel in common crate for heal requests
- Implement channel processor in ahm to handle heal commands
- Add Start/Query/Cancel commands support via channel
- Integrate heal manager initialization in main.rs
- Replace direct MRF calls with channel-based heal requests in ecstore
- Support advanced heal options including pool_index and set_index
- Enable admin handlers to send heal requests via channel
2025-07-24 12:12:49 +08:00
junxiang Mu
3409cd8dff feat(ahm): add HealingTracker support & complete fresh-disk healing
• Introduce ecstore HealingTracker into ahm crate; load/init/save tracker
• Re-implement heal_fresh_disk to use heal_erasure_set with tracker
• Enhance auto-disk scanner: detect unformatted disks via get_disk_id()
• Remove DataUsageCache handling for now
• Refactor imports & types, clean up duplicate constants
2025-07-24 12:12:49 +08:00
junxiang Mu
f4973a681c feat: implement complete ahm heal system with ecstore integration
- Add comprehensive heal storage API with ECStore integration
- Implement heal object, bucket, disk, metadata, and EC decode operations
- Add heal task management with progress tracking and statistics
- Optimize heal manager by removing unnecessary workers
- Add integration tests for core heal functionality (heal_object, heal_bucket, heal_format)
- Integrate with ecstore's native heal commands for actual repair operations

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
junxiang Mu
4fb3d187d0 feat: implement heal subsystem for automatic data repair
- Add heal module with core types (HealType, HealRequest, HealTask)
- Implement HealManager for task scheduling and execution
- Add HealStorageAPI trait and ECStoreHealStorage implementation
- Integrate heal capabilities into scanner for automatic repair
- Support multiple heal types: object, bucket, disk, metadata, MRF, EC decode
- Add progress tracking and event system for heal operations
- Merge heal and scanner error types for unified error handling
- Include comprehensive logging and metrics for heal operations

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
dandan
0aff736efd Chore: fix ref and fix comment
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
dandan
2aa7a631ef feat: refactor scanner module and add data usage statistics
- Move scanner code to scanner/ subdirectory for better organization
- Add data usage statistics collection and persistence
- Implement histogram support for size and version distribution
- Add global cancel token management for scanner operations
- Integrate scanner with ECStore for comprehensive data analysis
- Update error handling and improve test isolation
- Add data usage API endpoints and backend integration

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
dandan
b40ef147a9 refact: step 2
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
junxiang Mu
1f11a3167b fix: Refact heal and scanner design
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
guojidan
18b0134ddf Merge pull request #290 from guojidan/feat/complete-lock-implementation
refactor: reimplement lock
2025-07-24 12:11:19 +08:00
junxiang Mu
b48a5fdc94 fix fmt
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 11:52:57 +08:00
junxiang Mu
168a07a670 add api into ecstore
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 11:52:57 +08:00
junxiang Mu
cad005bc21 refactor(lock): unify NamespaceLock client model and LockRequest API
- Refactor NamespaceLock to use a unified client vector and quorum mechanism,
  removing legacy local/distributed lock split and related code.
- Update LockRequest to split timeout into acquire_timeout and ttl, and add
  builder methods for both.
- Adjust all batch lock APIs to accept ttl and use new LockRequest fields.
- Update all affected tests and documentation for the new API.

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 11:52:57 +08:00
root
dc44cde081 tmp
Signed-off-by: root <root@PC.localdomain>
2025-07-24 11:52:57 +08:00
dandan
4ccdeb9d2a refactor(lock): restructure lock crate, remove unused modules and clarify directory layout
- Remove unused core/rwlock.rs and manager/ modules (ManagerFactory, LifecycleManager, NamespaceManager)
- Move all lock-related code into crates/lock/src with clear submodules: client, core, utils, etc.
- Ensure only necessary files and APIs are exposed, improve maintainability
- No functional logic change, pure structure and cleanup refactor

Signed-off-by: dandan <dandan@dandandeMac-Studio.local>
2025-07-24 11:52:55 +08:00
dependabot[bot]
1b48934f47 build(deps): bump the dependencies group with 5 updates (#289)
Bumps the dependencies group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [hyper-util](https://github.com/hyperium/hyper-util) | `0.1.15` | `0.1.16` |
| [rand](https://github.com/rust-random/rand) | `0.9.1` | `0.9.2` |
| [serde_json](https://github.com/serde-rs/json) | `1.0.140` | `1.0.141` |
| [strum](https://github.com/Peternator7/strum) | `0.27.1` | `0.27.2` |
| [sysinfo](https://github.com/GuillaumeGomez/sysinfo) | `0.36.0` | `0.36.1` |


Updates `hyper-util` from 0.1.15 to 0.1.16
- [Release notes](https://github.com/hyperium/hyper-util/releases)
- [Changelog](https://github.com/hyperium/hyper-util/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/hyper-util/compare/v0.1.15...v0.1.16)

Updates `rand` from 0.9.1 to 0.9.2
- [Release notes](https://github.com/rust-random/rand/releases)
- [Changelog](https://github.com/rust-random/rand/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-random/rand/compare/rand_core-0.9.1...rand_core-0.9.2)

Updates `serde_json` from 1.0.140 to 1.0.141
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.140...v1.0.141)

Updates `strum` from 0.27.1 to 0.27.2
- [Release notes](https://github.com/Peternator7/strum/releases)
- [Changelog](https://github.com/Peternator7/strum/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Peternator7/strum/compare/v0.27.1...v0.27.2)

Updates `sysinfo` from 0.36.0 to 0.36.1
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GuillaumeGomez/sysinfo/compare/v0.36.0...v0.36.1)

---
updated-dependencies:
- dependency-name: hyper-util
  dependency-version: 0.1.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: rand
  dependency-version: 0.9.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: serde_json
  dependency-version: 1.0.141
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: strum
  dependency-version: 0.27.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: sysinfo
  dependency-version: 0.36.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-07-24 11:50:52 +08:00
zhangwenlong
25fa645184 add rustfs.spec for rustfs (#103)
add support on loongarch64
2025-07-24 11:39:09 +08:00
Marco Orlandin
3a3bb880f2 fix: update link in README.md leading to a 404 error (#285) 2025-07-24 09:15:04 +08:00
安正超
affe27298c fix: improve Windows build support and CI/CD workflow (#283)
- Fix Windows zip command issue by using PowerShell Compress-Archive
- Add Windows support for OSS upload with ossutil
- Replace Chinese comments with English in build.yml
- Fix bash syntax error in package_zip function
- Improve code formatting and consistency
- Update various configuration files for better cross-platform support

Resolves Windows build failures in GitHub Actions.
2025-07-22 23:55:57 +08:00
shiro.lee
629db6218e fix: the issue where preview fails when the path length exceeds 255 characters (#280) 2025-07-22 22:10:57 +08:00
安正超
aa1a3ce4e8 feat: add cargo clippy --fix --allow-dirty to pre-commit command (#282)
Resolves #277

- Add --fix flag to automatically fix clippy warnings
- Add --allow-dirty flag to run on dirty Git trees
- Improves code quality in pre-commit workflow
2025-07-22 22:10:53 +08:00
houseme
693db59fcc fix 2025-07-21 20:45:59 +08:00
houseme
0a7df4ef26 fix 2025-07-21 19:03:15 +08:00
houseme
9dcdc44718 fix 2025-07-21 18:03:01 +08:00
houseme
2a0c618f8b fix: windows build 2025-07-21 17:45:56 +08:00
loverustfs
bebd78fbbb Add GNU to build.yml (#275)
* fix unzip error

* fix url change error

fix url change error

* Simplify user experience and integrate console and endpoint

Simplify user experience and integrate console and endpoint

* Add gnu to  build.yml
2025-07-21 16:58:29 +08:00
houseme
3f095e75cb improve code for logger and fix typo (#272) 2025-07-21 15:20:36 +08:00
houseme
f7d30da9e0 fix typo (#267)
* fix typo

* cargo fmt
2025-07-20 00:11:15 +08:00
Chrislearn Young
823d4b6f79 Add typos github actions and fix typos (#265)
* Add typo github actions and fix typos

* cargo fmt
2025-07-19 22:08:50 +08:00
安正超
051ea7786f fix: ossutil install command. (#263) 2025-07-19 18:21:31 +08:00
安正超
42b645e355 fix: robust Dockerfile version logic for v prefix handling (#262)
* fix: robust Dockerfile version logic for v prefix handling

* wip
2025-07-19 15:50:15 +08:00
安正超
f27ee96014 feat: enhance entrypoint and Dockerfiles for flexible volume and permission management (#260)
* feat: enhance entrypoint and Dockerfiles for flexible volume and permission management\n\n- Support batch mount and permission fix in entrypoint.sh\n- Add coreutils/shadow (alpine) and coreutils/passwd (ubuntu) for UID/GID/ownership\n- Use ENTRYPOINT for unified startup\n- Make local dev and prod Dockerfile behavior consistent\n- Improve security and user experience\n\nBREAKING CHANGE: entrypoint.sh and Dockerfile now require additional packages for permission management, and support batch volume mount via RUSTFS_VOLUMES.

* chore: update Dockerfile comments to English only

* fix(entrypoint): improve local/remote volume detection and permission logic in entrypoint.sh

* Update entrypoint.sh

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update entrypoint.sh

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update Dockerfile

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-19 11:48:46 +08:00
houseme
20cd117aa6 improve code for dockerfile (#256)
* improve code for dockerfile

* Update Dockerfile

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* improve code for file name

* improve code for dockerfile

* fix

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-18 15:53:00 +08:00
houseme
fc8931d69f improve code for dockerfile (#253)
* improve code for dockerfile

* Update Dockerfile

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-18 11:05:00 +08:00
weisd
0167b2decd fix: optimize RPC connection management and prevent race conditions (#252) 2025-07-18 10:41:00 +08:00
weisd
e67980ff3c Fix/range content length (#251)
* fix:getobject range length
2025-07-17 23:25:21 +08:00
weisd
96760bba5a fix:getobject range length (#250) 2025-07-17 23:14:19 +08:00
overtrue
2501d7d241 fix: remove branch restriction from Docker workflow_run trigger
The Docker workflow was not triggering for tag-based releases because it had
'branches: [main]' restriction in the workflow_run configuration. When pushing
tags, the triggering workflow runs on the tag, not on main branch.

Changes:
- Remove 'branches: [main]' from workflow_run trigger
- Simplify tag detection using github.event.workflow_run context instead of API calls
- Use official workflow_run event properties (head_branch, event) for reliable detection
- Support both 'refs/tags/VERSION' and direct 'VERSION' formats
- Add better logging for debugging workflow trigger issues

This fixes the issue where Docker images were not built for tagged releases.
2025-07-17 08:13:34 +08:00
overtrue
55b84262b5 fix: use GitHub API for reliable tag detection in Docker workflow
- Replace git commands with GitHub API calls for tag detection
- Add proper commit checkout for workflow_run events
- Use gh CLI and curl fallback for better reliability
- Add debug output to help troubleshoot tag detection issues

This should fix the issue where Docker builds were not triggered for tagged releases
due to missing tag information in the workflow_run environment.
2025-07-17 08:01:33 +08:00
overtrue
ce4252eb1a fix: correct Docker workflow trigger logic for tag-based releases
BREAKING CHANGE: Fixed Docker workflow that was incorrectly skipping builds for tagged releases
- Fix logic to detect tag pushes using git refs instead of branch names
- Properly identify tag pushes vs branch pushes using git show-ref
- Support both v-prefixed and bare version formats
- Ensure Docker images are built for all tagged releases including prereleases
2025-07-17 07:46:54 +08:00
overtrue
db708917b4 docs: update .docker/README.md to reflect simplified Makefile commands
- Add new Makefile Commands section with simplified docker-dev* commands
- Update Development Workflow to use new dev-env-* commands
- Update directory structure (remove deleted alpine/ directory)
- Reorganize build instructions to prioritize Makefile over direct scripts
- Add Common Development Tasks section with make help commands
2025-07-17 07:30:13 +08:00
overtrue
8ddb45627d refactor: simplify Docker build commands and fix version matching
- Remove obsolete .docker/alpine/Dockerfile.protoc (superseded by Dockerfile.source)
- Simplify Makefile commands by removing backward compatibility aliases
  * Replace docker-buildx-source* with shorter docker-dev* commands
  * Replace start/stop with explicit dev-env-start/dev-env-stop commands
- Fix Docker workflow version matching logic to correctly distinguish:
  * 1.0.0 vs 1.0.0-alpha.11 (prerelease detection)
  * Support both v1.0.0 and 1.0.0 formats (with/without v prefix)
  * Reorder case patterns to match prereleases before releases

BREAKING CHANGE: Removed legacy command aliases
- Use 'make docker-dev-local' instead of 'make docker-buildx-source-local'
- Use 'make dev-env-start' instead of 'make start'
2025-07-17 07:29:00 +08:00
overtrue
550c225b79 wip 2025-07-17 07:07:02 +08:00
overtrue
0d46b550a8 refactor: merge release workflow into build workflow and clean up
- Merge release logic into build.yml to avoid cross-workflow artifact access issues
- Add release jobs (create-release, upload-release-assets, update-latest-version, publish-release) that run only for tag pushes
- Use standard actions/download-artifact@v4 within the same workflow (no cross-workflow limitations)
- Deprecate standalone release.yml workflow with warning job and confirmation requirement
- Remove references to deleted release-notes-template.md file from both workflows
- Update build summary messages to reflect integrated release process

This resolves the 'Prepare release assets' failure by eliminating the need for cross-workflow artifact access.
2025-07-17 07:06:51 +08:00
overtrue
0693cca1a4 fix: resolve workflow_run artifact access issue in release pipeline
- Replace actions/download-artifact@v4 with GitHub API calls to access artifacts from triggering workflow
- Add proper permissions (contents: read, actions: read) to prepare-assets job
- Handle both workflow_run and workflow_dispatch trigger scenarios
- Fix the root cause: workflow_run events cannot access artifacts from triggering workflows using standard download-artifact action

Fixes the 'Prepare release assets' step failure by implementing cross-workflow artifact access through GitHub API.
2025-07-17 06:58:09 +08:00
安正超
0d9f9e381a refactor: use workflow_run trigger for release workflow to eliminate timing issues (#241)
* fix: use correct tag reference in release workflow wait-for-artifacts step

- Change ref from github.ref to needs.release-check.outputs.tag
- Fix issue where wait-on-check-action receives full git reference (refs/tags/1.0.0-alpha.21)
  instead of clean tag name (1.0.0-alpha.21)
- This resolves timeout errors when waiting for build artifacts during release process

Fixes the release workflow failure for tag 1.0.0-alpha.21

* refactor: use workflow_run trigger for release workflow instead of push

- Replace push trigger with workflow_run to eliminate timing issues
- Release workflow now triggers only after Build workflow completes successfully
- Remove wait-for-artifacts step completely (no longer needed)
- Add should_release condition to control release execution
- Support both tag pushes and manual releases via workflow_dispatch
- Align with docker.yml pattern for better reliability

This completely resolves the release workflow timeout issues by ensuring
build artifacts are always available before the release process starts.

Fixes the fundamental timing issue where release.yml and build.yml
were racing against each other when triggered by the same tag push.
2025-07-17 06:48:09 +08:00
安正超
6c7aa5a7ae fix: use correct tag reference in release workflow wait-for-artifacts step (#240)
- Change ref from github.ref to needs.release-check.outputs.tag
- Fix issue where wait-on-check-action receives full git reference (refs/tags/1.0.0-alpha.21)
  instead of clean tag name (1.0.0-alpha.21)
- This resolves timeout errors when waiting for build artifacts during release process

Fixes the release workflow failure for tag 1.0.0-alpha.21
2025-07-17 06:36:57 +08:00
overtrue
a27d935925 wip 2025-07-17 06:31:25 +08:00
安正超
b4f87a4fee feat: disable Docker builds for development versions (#239)
* feat: disable Docker builds for development versions

- Remove dev-latest, main-latest, and dev-* version options from manual triggers
- Skip Docker builds for development versions in workflow_run events
- Only build Docker images for releases (v1.0.0) and prereleases (v1.0.0-alpha1)
- Simplify tags generation logic by removing development branch handling
- Update workflow documentation to reflect release-only Docker strategy

BREAKING CHANGE: Development Docker images are no longer built automatically

* feat: remove dev channel support from Dockerfile

- Remove CHANNEL build argument (no longer needed)
- Simplify download logic to only support release channel
- Remove dev-specific package download paths
- Update BASE_URL to point directly to release directory
- Remove channel label from Docker image metadata
- Streamline version handling (latest vs specific release)

This aligns with the workflow changes that disabled dev Docker builds.
2025-07-17 06:06:40 +08:00
安正超
ee5f94a2e2 fix: use consistent short SHA generation across workflows (#238)
- Replace manual cut -c1-7 with git rev-parse --short in docker.yml
- Ensures consistent short SHA length between build.yml and docker.yml
- Git automatically adjusts length for uniqueness, preventing conflicts
2025-07-17 05:48:30 +08:00
安正超
9c3cf554d3 fix: correct Docker build logic for dev version downloads (#237) 2025-07-17 05:36:15 +08:00
安正超
addbfa5487 fix: resolve Docker workflow manual build parameter issues (#236)
- Remove unsupported 'scopes' parameter from docker/login-action@v3
  * Fixes 'Unexpected input(s) scopes' error during Docker Hub login

- Add version format conversion for Dockerfile compatibility
  * main-latest/dev-latest → RELEASE=latest + CHANNEL=dev
  * latest → RELEASE=latest + CHANNEL=release
  * dev-* → RELEASE=dev-* + CHANNEL=dev
  * v* → RELEASE={version without v} + CHANNEL=release

- Fix Docker build parameter passing
  * Use converted docker_release and docker_channel values
  * Ensures correct binary download URLs in Dockerfile

Resolves manual Docker build failures reported in:
https://github.com/rustfs/rustfs/actions/runs/16330398463/job/46131302262
2025-07-17 05:21:06 +08:00
安正超
5eb461d7b7 refactor: remove redundant linux_builds_success logic in docker workflow (#235)
- Remove linux_builds_success output and related variables
- Simplify build-docker condition to only check should_build
- The should_build check already includes workflow success verification
- Reduce code complexity while maintaining the same functionality
2025-07-17 05:09:41 +08:00
安正超
1ea45afcd7 feat: Implement precise Docker build triggering using workflow_run event (#233)
* fix: correct YAML indentation error in docker workflow

- Fix incorrect indentation at line 237 in .github/workflows/docker.yml
- Step 'Extract metadata and generate tags' had 12 spaces instead of 6
- This was causing YAML syntax validation to fail

* fix: restore unified build-rustfs task with correct YAML syntax

- Revert complex job separation back to single build-rustfs task
- Maintain Linux and macOS builds in unified matrix
- Fix YAML indentation and syntax issues
- Docker builds will use only Linux binaries as designed in Dockerfile

* feat: implement precise Docker build triggering using workflow_run

- Use workflow_run event to trigger Docker builds independently
- Add precise Linux build status checking via GitHub API
- Only trigger Docker builds when both Linux architectures succeed
- Remove coupling between build.yml and docker.yml workflows
- Improve TARGETPLATFORM consistency in Dockerfile

This resolves the issue where Docker builds would trigger even if
Linux ARM64 builds failed, causing missing binary artifacts during
multi-architecture Docker image creation.
2025-07-17 04:51:08 +08:00
安正超
dbd86f6aee fix: correct YAML indentation error in docker workflow (#232)
- Fix incorrect indentation at line 237 in .github/workflows/docker.yml
- Step 'Extract metadata and generate tags' had 12 spaces instead of 6
- This was causing YAML syntax validation to fail
2025-07-17 04:28:31 +08:00
overtrue
af693f7b3f refactor: restructure Docker build pipeline to depend on binary builds
- Change docker.yml to use workflow_call triggered by build.yml
- Remove redundant force_build parameter from build.yml
- Simplify build_docker parameter (build implies push in CI/CD)
- Add proper dependency chain: build.yml -> docker.yml -> registry
- Update documentation to reflect new architecture
- Mark Dockerfile.source as local development only
2025-07-17 04:19:20 +08:00
安正超
3be5ee6445 fix: simplify Dockerfile.source and resolve build issues (#231)
- Remove complex dependency caching to fix workspace structure issues
- Remove sccache to eliminate rustc wrapper errors
- Ensure target installation in build step for cross-compilation
- Add debug output and error handling for unsupported platforms
- Use simple COPY . . approach for more reliable builds
2025-07-16 23:53:28 +08:00
overtrue
0acc8fe26a fix: docker build from source 2025-07-16 23:46:30 +08:00
overtrue
ecf40eb86c fix: docker build from source 2025-07-16 23:43:34 +08:00
overtrue
48ce7055f8 fix: remove dockerhub username 2025-07-16 22:35:14 +08:00
weisd
749f55d688 feat: enhance version function with automatic version increment (#227) 2025-07-16 18:09:43 +08:00
loverustfs
e5d17f5382 Disable Dockerfile.source 2025-07-16 18:03:09 +08:00
weisd
982cc66c74 fix: Refactor session policy handling and fix owner permission check (#226) 2025-07-16 16:40:51 +08:00
loverustfs
74bf4909c8 Modify docker source file 2025-07-15 23:17:39 +08:00
loverustfs
9c956b4445 Disable other docker mode 2025-07-15 22:10:00 +08:00
weisd
4c1fc9317e fix: content-range (#216) 2025-07-15 17:23:33 +08:00
weisd
a9d77a618f feat: implement list_parts API for S3 multipart upload compatibility (#209)
* feat: add list_parts api
2025-07-15 16:04:03 +08:00
overtrue
38cdc87e93 fix 2025-07-15 02:36:07 +08:00
安正超
f5ff93b65e fix: restore working build configuration by removing cargo.config.toml (#206)
- Remove cargo.config.toml file that was causing build issues
- Restore .github/workflows/build.yml to working state from commit 2e9792577f
- These changes ensure the build system works correctly again
2025-07-15 02:24:13 +08:00
安正超
6ef6f188e5 fix: Restore working build configuration from 4fb4b353 (#204)
* fix: Resolve zstd-sys Zig compilation issues

- Remove specific Zig version constraint in action.yml to use default version
- Clean up duplicate environment variable settings in build-rustfs.sh
- Add CARGO_TARGET_*_LINKER environment variables for better cross-compilation support
- Optimize build configuration for consistent cross-platform compilation

Fixes compilation issues with zstd-sys when using Zig cross-compilation.
Aligns with previously working configuration that uses default Zig version.

* fix: Restore working build configuration from 4fb4b353

- Restore matrix.cross parameter to differentiate cross-compilation
- Use simple cargo zigbuild instead of complex build-rustfs.sh script
- Remove unnecessary zstd dependencies from action.yml
- Restore console asset download step
- Use correct target directory path for packaging
- Align with known working configuration from commit 4fb4b353

This reverts to the proven working build approach that successfully
performed cross-platform compilation.

* fix: Align build-rustfs.sh with working version logic

- Simplify build logic to match working version 4fb4b353
- Use exact same build commands as the working build.yml:
  * cargo build for native compilation
  * cargo zigbuild for Linux ARM64 cross-compilation
  * cross build for Windows ARM64 cross-compilation
- Remove complex environment variable setup that caused conflicts
- Add touch rustfs/build.rs to match working version
- Use -p rustfs --bins flag consistent with working version

This ensures build-rustfs.sh (if used) follows the proven working approach.
2025-07-14 20:22:29 +08:00
安正超
ccad91a4a9 fix: resolve zstd-sys compilation issues with zig cross-compilation (#203)
- Update to mlugg/setup-zig@v2 for better stability and features
- Use Zig 0.13.0 for improved musl target support
- Add system zstd libraries (libzstd-dev, zstd) to Ubuntu dependencies
- Configure environment variables for zstd-sys to use pkg-config
- Enable pkg-config feature for zstd dependency to prefer system library
- Add proper C/C++ compiler configuration for musl targets

Fixes the 'error: unable to parse target query x86_64-unknown-linux-musl: UnknownOperatingSystem'
compilation error in zstd-sys during cross-compilation.
2025-07-14 20:01:52 +08:00
安正超
63b79ae151 fix: add cross-platform SHA256 checksum generation (#202)
- Add generate_sha256() function to handle cross-platform SHA256 generation
- Use shasum -a 256 on macOS instead of sha256sum
- Use sha256sum on Linux with shasum as fallback
- Replace direct sha256sum usage in build script with new function
- Fixes 'sha256sum: command not found' error on macOS builds
2025-07-14 19:45:42 +08:00
安正超
9284f64e2a fix: resolve aarch64-unknown-linux-musl build issue with cargo-zigbuild integration (#201)
- Enable install-cross-tools in GitHub Actions build workflow
- Add cargo-zigbuild support for Linux targets in build-rustfs.sh
- Prioritize cargo-zigbuild over cross tool for better glibc compatibility
- Add musl-specific environment variables for proper static linking
- Update error messages with Linux-specific build suggestions
- Configure Zig compiler environment for musl targets
2025-07-14 19:31:30 +08:00
安正超
b9bbae27de fix: resolve macOS build issue by disabling cross tool for apple-darwin targets (#200) 2025-07-14 19:25:01 +08:00
安正超
36e3efb5a5 feat: implement Docker improvements and binary build scripts (#191)
* feat: implement Docker improvements and binary build scripts

This commit transforms the RustFS Docker build system to follow MinIO's best practices:

## 🏗️ Binary Build Script (build-rustfs.sh)
- Create independent binary compilation script for multi-platform builds
- Support x86_64 and aarch64 Linux musl targets
- Include checksum generation and optional binary signing
- Support cross-compilation and upload functionality
- Automated target installation and environment setup

## 🐳 Docker Improvements
- Rewrite Dockerfiles to download precompiled binaries instead of building from source
- Follow MinIO's approach for security and binary verification
- Add comprehensive LABEL metadata (version, build-date, vcs-ref)
- Implement proper environment variable management
- Add signature verification with minisign (commented for future use)
- Include static curl download for minimal runtime dependencies

## 🚀 Enhanced Build Script (docker-buildx.sh)
- Inspired by MinIO's docker-buildx.sh for consistency and reliability
- Support multiple platforms with proper build arguments
- Auto-detect git versions and pass metadata to containers
- Improved error messages with helpful troubleshooting hints
- Cleanup and cache management between builds

## 🛠️ Supporting Scripts
- scripts/download-static-curl.sh: Download statically compiled curl
- scripts/setup-test-binaries.sh: Create test binaries for local development

## 📋 Key Benefits
- Faster Docker builds (download vs compile)
- Better security with signature verification
- Consistent with industry standards (MinIO approach)
- Proper multi-platform support
- Enhanced metadata and traceability
- Independent binary distribution capability

* feat: update Docker files to use Aliyun OSS for binary downloads

* feat: merge stash with OSS binary download improvements

- Remove old build_rustfs.sh script
- Keep Aliyun OSS download URLs for binary retrieval
- Maintain Docker build improvements from stash
- Resolve merge conflicts between stash and OSS updates

* feat: improve build-rustfs.sh with auto platform detection

- Auto-detect current platform using uname (like old build_rustfs.sh)
- Default to building for current platform only
- Add --all-platforms flag for cross-compilation to Linux musl targets
- Support macOS (darwin) and Linux platforms
- Auto-enable cross compilation when needed
- Provide better usage examples and platform detection info

This makes the script much more user-friendly by default while
maintaining flexibility for cross-compilation scenarios.

* refactor: simplify build-rustfs.sh for CI/CD pipeline usage

- Remove cross-compilation complexity (each CI runner builds natively)
- Focus on single platform builds per runner
- Remove --all-platforms and --cross options
- Simplify to match CI/CD workflow where:
  * Linux x86_64 runner builds Linux x86_64 binary
  * Linux ARM64 runner builds Linux ARM64 binary
  * macOS x86_64 runner builds macOS x86_64 binary
  * macOS ARM64 runner builds macOS ARM64 binary
- Keep signing and upload functionality for release CI
- Make the script's purpose and usage clearer

This aligns with the user's understanding that build scripts should
focus on native compilation for the current platform only.

* feat: update download server domain to dl.rustfs.com

- Update Dockerfile to use dl.rustfs.com/dev/ for development binaries
- Update Dockerfile.release to use dl.rustfs.com/release/ for release binaries
- Update docker-buildx.sh error messages with new URLs
- Update build-rustfs.sh upload target to dl.rustfs.com
- Update test scripts to reference new domain
- Clean up remaining git conflict markers

This centralizes all binary downloads through the official
dl.rustfs.com domain instead of direct OSS access.

* fix: correct dl.rustfs.com path structure to include /artifacts/rustfs/

- Update all download URLs to use correct path structure:
  * Dev: https://dl.rustfs.com/artifacts/rustfs/dev/
  * Release: https://dl.rustfs.com/artifacts/rustfs/release/
- Test confirmed both paths return HTTP 200 with application/zip content-type
- Update Dockerfile, Dockerfile.release, docker-buildx.sh, and build-rustfs.sh
- Update test scripts with correct base path

The dl.rustfs.com domain requires the /artifacts/rustfs/ prefix
to access the binary files correctly.

* feat: refactor Dockerfile to download binaries from GitHub Releases

- Changed binary download source from dl.rustfs.com to GitHub Releases
- Added support for latest release auto-detection via GitHub API
- Enhanced error handling with detailed messages and helpful links
- Added optional checksum verification using SHA256SUMS
- Improved architecture support for amd64 and arm64
- Removed unnecessary minisign installation
- Added jq dependency for JSON parsing

* feat: consolidate Docker build to use single Dockerfile

- Removed Dockerfile.release and use unified Dockerfile instead
- Updated docker-buildx.sh to use single Dockerfile with build args
- Both latest and release variants now use GitHub Releases
- Simplified build process and reduced maintenance overhead
- Updated error messages to point to GitHub releases

* chore: remove unused Dockerfile.obs

- Removed Dockerfile.obs as it's no longer needed
- Simplified Docker build configuration

* feat: unify Docker prebuild variants to use GitHub Releases

- Updated .docker/alpine/Dockerfile.prebuild to download from GitHub Releases
- Updated .docker/ubuntu/Dockerfile.prebuild to download from GitHub Releases
- All prebuild variants now consistently use GitHub Releases as binary source
- Added checksum verification for all prebuild variants
- Updated .docker/README.md to reflect unified GitHub Releases approach
- Improved error handling and user guidance in all prebuild Dockerfiles

* feat: major Docker structure simplification and consolidation

## 🎯 Simplified Docker Structure

Moved from complex multi-directory structure to clean root-level organization:

### Before:
- Dockerfile (production)
- .docker/alpine/Dockerfile.prebuild (duplicate)
- .docker/alpine/Dockerfile.source
- .docker/ubuntu/Dockerfile.prebuild (duplicate)
- .docker/ubuntu/Dockerfile.source
- .docker/ubuntu/Dockerfile.dev

### After:
- Dockerfile (production - Alpine + GitHub Releases)
- Dockerfile.source (source build - Ubuntu + cross-compilation)
- Dockerfile.dev (development - Ubuntu + full toolchain)

## 🔧 Key Changes

- **Eliminated Duplicates**: Removed redundant prebuild variants
- **Moved Core Files**: Dockerfile.{source,dev} now in root directory
- **Unified Configuration**: cargo.config.toml moved to root
- **Updated References**: Fixed all GitHub Actions and docker-compose paths
- **Simplified CI Matrix**: Reduced from 5 to 3 Docker variants

## 📦 Preserved Valuable Diversity

- **Production**: Alpine-based for minimal size
- **Source**: Ubuntu-based with cross-compilation support
- **Development**: Ubuntu-based with full development tools

## 🚀 Benefits

-  Cleaner project structure
-  Easier maintenance and navigation
-  Reduced CI/CD complexity
-  Faster build matrix execution
-  Maintained functionality and flexibility

* chore: remove duplicate cargo.config.toml from .docker directory

The file is now in the root directory and no longer needed in .docker/

* fix: update all references to removed Dockerfile files

- Updated .docker/compose/README.md to reference Dockerfile.source instead of Dockerfile.obs
- Updated docker-compose.yml to use Dockerfile.source instead of Dockerfile.dev
- Updated scripts/build-docker-multiarch.sh to use Dockerfile.source for devenv builds
- Updated .github/workflows/docker.yml to use Dockerfile.source for dev builds
- Updated Makefile to use Dockerfile.source for init-devenv target
- Updated .docker/README.md to remove references to non-existent Dockerfile.dev
- Ensured all Docker configurations consistently use the unified Dockerfile structure

* chore: remove unnecessary console static assets download

- Remove obsolete download steps from build.yml and performance.yml
- Console static assets are already embedded via rust-embed in rustfs/static/
- The download from dl.rustfs.com is no longer needed as project contains complete console assets
- This improves build reliability and reduces external dependencies
- Replaced with verification steps that confirm embedded assets are present

* feat: update Makefile and README.md for new Docker build system

- Updated Makefile to use unified Docker build system:
  - Replace references to non-existent Dockerfile.ubuntu22.04 and Dockerfile.rockylinux9.3
  - Add new docker-buildx targets using docker-buildx.sh script
  - Deprecate old docker-build-multiarch targets with warnings
  - Add docker-build-production and docker-build-source targets
  - Update help-docker with new command structure

- Updated README.md with docker-buildx.sh usage:
  - Add comprehensive Docker build from source section
  - Document multi-architecture build capabilities
  - Include both script and Make target examples
  - Show registry flexibility and build optimization features
  - Update step numbers in quickstart guide

- Improve developer experience with clear documentation and updated tooling
- Maintain backward compatibility with deprecation warnings

* feat: integrate console assets download into build-rustfs.sh

- Added console download functionality to build-rustfs.sh:
  - New flags: --download-console, --force-console-update, --console-version
  - Intelligent detection of existing console assets
  - Retry logic with fallback error handling
  - Consistent with Docker build asset management

- Updated scripts to use unified build process:
  - scripts/static.sh: Now uses build-rustfs.sh for console downloads
  - scripts/run.sh: Uses build-rustfs.sh instead of direct curl
  - scripts/run.ps1: Updated with guidance for Windows users

- Benefits:
  - Unified asset management across all build processes
  - Consistent version handling and retry logic
  - Eliminates duplicate download logic
  - Better error handling and user feedback
  - Preparation for CI/CD integration

- Removed unused download-static-curl.sh script

This change centralizes console asset management and prepares for
streamlined CI/CD processes where build-rustfs.sh becomes the
single point of truth for binary and asset builds.

* fix: update PowerShell script to use unified console asset management

- Updated scripts/run.ps1 to use build-rustfs.sh for console asset downloads
- Added guidance for Windows users to use the unified build script
- Maintains consistency across all platform-specific scripts

* feat: add binary verification to build script

- Add verify_binary function to test built binaries
- Test --help and --version commands
- Verify binary structure with readelf/otool
- Add --skip-verification option for cross-compilation
- Include verification status in build output
- Automatic error handling if verification fails

* feat: add platform selection support to build script

- Add --platform parameter to build-rustfs.sh for target platform selection
- Implement cross-compilation support with automatic 'cross' tool detection
- Auto-enable --skip-verification for cross-compilation scenarios
- Update all Makefile build targets to use unified build-rustfs.sh script
- Add helpful error messages and suggestions for cross-compilation failures
- Update help documentation with platform selection examples
- Improve build consistency across different architectures

* feat: modernize CI/CD build process with build-rustfs.sh

- Replace manual cargo build commands with unified build-rustfs.sh script
- Simplify matrix configuration by removing cross-compilation flags
- Ensure consistency between local and CI/CD builds
- Automatic cross-compilation tool detection and installation
- Built-in binary verification for quality assurance
- Unified console asset management
- Better error handling and suggestions

Benefits:
- Consistent build process across all environments
- Automatic detection and handling of cross-compilation scenarios
- Built-in quality checks with binary verification
- Reduced CI/CD configuration complexity
- Better maintainability with single source of truth for build logic

* feat: optimize CI/CD workspace path management

- Add WORKSPACE_DIR environment variable to cache github.workspace
- Set default working-directory at job level for consistency
- Use explicit workspace paths in critical operations
- Improve reliability and maintainability of CI/CD paths
- Ensure consistent behavior across different GitHub Actions environments

Benefits:
- More explicit and reliable path handling
- Better maintainability with centralized workspace reference
- Reduced risk of path-related issues in CI/CD
- Consistent working directory across all job steps

* refactor: simplify CI/CD path management - remove redundant workspace references

- Remove unnecessary WORKSPACE_DIR environment variable
- Remove redundant defaults.run.working-directory setting
- Use relative paths since GITHUB_WORKSPACE is the default working directory
- Follow GitHub Actions best practices by leveraging default behavior

As per GitHub Actions documentation, GITHUB_WORKSPACE is already the default
working directory, so explicit specification is unnecessary in most cases.

* docs: update Docker README to reflect current project state

- Fix directory structure: remove non-existent nginx/ directory
- Correct base OS: Dockerfile.source uses Debian Bookworm, not Ubuntu 22.04
- Add docker-buildx.sh script documentation
- Update Docker tag examples to match actual CI/CD workflows
- Add CI/CD integration section explaining automated builds
- Document build variants and manual build options
- Reflect current project architecture and tooling

These updates ensure the documentation accurately represents the current
Docker build system and CI/CD workflows.

* fix: update Docker command in rustfs README

- Replace quay.io registry with Docker Hub (rustfs/rustfs:latest)
- Remove separate console port 9001, console now runs on main port 9000
- Add both Docker and Podman examples for user choice
- Fix console access URL to use unified port

This aligns with the recent console port consolidation changes
and the project's move to Docker Hub as the primary registry.

* wip

* fix: remove unnecessary entrypoint.sh and fix Docker paths

* Update Dockerfile

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* cleanup: remove unused DOCKERFILE_PATH variable from Makefile

* feat: update Docker build to use dl.rustfs.com for binary downloads

- Replace GitHub releases download with dl.rustfs.com
- Add CHANNEL parameter support (release/dev)
- Update docker-buildx.sh to support channel-specific builds
- Improve error messages with new download URLs
- Support both latest and specific version downloads
- Add channel validation in build script

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-14 19:15:46 +08:00
Nugine
04d1c8724d build: upgrade s3s (#193) 2025-07-14 15:19:01 +08:00
houseme
4fb4b353f8 improve code for cargo.toml 2025-07-13 23:13:08 +08:00
houseme
564a02f344 feat(obs, net): Add Tempo service and enable dual-stack listener (#192)
This commit introduces two key enhancements: the integration of Grafana Tempo for distributed tracing and the implementation of a dual-stack TCP listener for improved network compatibility.

- **Observability**:
  - Adds the `tempo` service to the `docker-compose.yml` observability stack.
  - Tempo is configured to collect and store traces, integrating with the existing OpenTelemetry setup.
  - A custom `tempo-entrypoint.sh` script is included to manage volume permissions on startup.

- **Networking**:
  - Modifies `http.rs` to support dual-stack (IPv4/IPv6) connections on a single socket.
  - By setting the `IPV6_V6ONLY` socket option to `false`, the server can now accept both IPv6 and IPv4-mapped IPv6 traffic, enhancing cross-platform support.
2025-07-13 20:22:46 +08:00
安正超
5b582a4234 feat: disable GitHub Packages uploads in Docker workflow (#189) 2025-07-12 19:07:56 +08:00
安正超
2e9792577f fix: correct SHA length matching in GitHub Actions workflow (#188)
* feat: enhance Docker build system with advanced version selection

## New Features
- Add force_rebuild parameter for Docker workflow manual triggers
- Improve version pattern matching with better regex validation
- Add comprehensive Docker Build Guide documentation
- Enhanced logging and error reporting for build process
- Support for prerelease version detection (alpha, beta, rc)

## Improvements
- Better version pattern validation for releases and dev builds
- More detailed build logs with context and warnings
- Clear documentation for all Docker image variants and use cases
- Updated README with Docker version examples and guide reference

## Documentation
- New comprehensive Docker Build Guide (docs/DOCKER_BUILD_GUIDE.md)
- Updated README with version-specific Docker examples
- Workflow dependency diagram and troubleshooting guide
- Complete reference for all supported version patterns

This enhancement provides a robust, well-documented Docker build system
that supports flexible version selection while maintaining deterministic
build behavior without fallback mechanisms.

* fix: simplify dev version regex pattern in docker workflow

* fix: simplify version number regex pattern in docker workflow

* feat: remove docs directory

* fix: correct SHA length matching in main-latest filename generation

* refactor: use bash string operations instead of sed for main-latest filename generation

* refactor: simplify filename generation by removing redundant intermediate variables

* feat: add dev-latest version generation for all development builds

* feat: add dev-latest support to Docker workflow
2025-07-12 18:46:37 +08:00
安正超
2066e0a03b fix: correct SHA length matching in main-latest filename generation (#187)
* feat: enhance Docker build system with advanced version selection

## New Features
- Add force_rebuild parameter for Docker workflow manual triggers
- Improve version pattern matching with better regex validation
- Add comprehensive Docker Build Guide documentation
- Enhanced logging and error reporting for build process
- Support for prerelease version detection (alpha, beta, rc)

## Improvements
- Better version pattern validation for releases and dev builds
- More detailed build logs with context and warnings
- Clear documentation for all Docker image variants and use cases
- Updated README with Docker version examples and guide reference

## Documentation
- New comprehensive Docker Build Guide (docs/DOCKER_BUILD_GUIDE.md)
- Updated README with version-specific Docker examples
- Workflow dependency diagram and troubleshooting guide
- Complete reference for all supported version patterns

This enhancement provides a robust, well-documented Docker build system
that supports flexible version selection while maintaining deterministic
build behavior without fallback mechanisms.

* fix: simplify dev version regex pattern in docker workflow

* fix: simplify version number regex pattern in docker workflow

* feat: remove docs directory

* fix: correct SHA length matching in main-latest filename generation
2025-07-12 11:43:17 +08:00
安正超
a4d49a500f feat: Enhanced Docker Build System with Advanced Version Selection (#186)
* feat: enhance Docker build system with advanced version selection

## New Features
- Add force_rebuild parameter for Docker workflow manual triggers
- Improve version pattern matching with better regex validation
- Add comprehensive Docker Build Guide documentation
- Enhanced logging and error reporting for build process
- Support for prerelease version detection (alpha, beta, rc)

## Improvements
- Better version pattern validation for releases and dev builds
- More detailed build logs with context and warnings
- Clear documentation for all Docker image variants and use cases
- Updated README with Docker version examples and guide reference

## Documentation
- New comprehensive Docker Build Guide (docs/DOCKER_BUILD_GUIDE.md)
- Updated README with version-specific Docker examples
- Workflow dependency diagram and troubleshooting guide
- Complete reference for all supported version patterns

This enhancement provides a robust, well-documented Docker build system
that supports flexible version selection while maintaining deterministic
build behavior without fallback mechanisms.

* fix: simplify dev version regex pattern in docker workflow

* fix: simplify version number regex pattern in docker workflow

* feat: remove docs directory
2025-07-12 11:31:00 +08:00
安正超
a8fbced928 feat: improve Docker build with version selection and remove fallback mechanism (#185)
- Add version input parameter to docker.yml workflow_dispatch
- Support main-latest, latest, dev-xxx, and specific version patterns
- Remove complex fallback mechanism from all Dockerfile variants
- Add clear error handling with helpful user guidance
- Create main-latest versions for development builds
- Ensure Docker builds require explicit VERSION parameter
- Update all Docker variants (production, alpine, ubuntu) consistently

This change solves the build dependency issue where Docker builds
could fail when expected binary artifacts don't exist, by providing
a clean version selection mechanism without unpredictable fallbacks.
2025-07-12 11:09:44 +08:00
安正超
99ca405279 feat: enhance cursor rules with strict main branch protection (#184) 2025-07-12 10:59:17 +08:00
overtrue
2e1d1018aa refactor: simplify version handling by removing unnecessary CLEAN_VERSION variable 2025-07-12 10:42:47 +08:00
overtrue
c57b4be1c7 refactor: use bash variable expansion for dev- prefix handling 2025-07-12 10:41:08 +08:00
overtrue
238a016242 chore: ignore .secrets 2025-07-12 10:30:37 +08:00
shiro.lee
2c0c7fafa3 fix: Optimized io::ErrorKind::NotFound error handling during Windows system startup (#181) 2025-07-12 06:54:21 +08:00
overtrue
ee4962fe31 fix: docker build 2025-07-11 23:42:46 +08:00
安正超
55895d0a10 fix: resolve Docker Hub authentication issues in multi-platform builds (#180) 2025-07-11 23:36:37 +08:00
overtrue
676897d389 fix: docker 2025-07-11 23:29:47 +08:00
loverustfs
5205ff6695 Add hellogithub icon 2025-07-11 23:24:55 +08:00
overtrue
15cf3ce92b fix: docker 2025-07-11 23:22:48 +08:00
安正超
c0441b2412 fix: resolve GitHub Actions workflow validation errors in docker.yml (#179)
* fix: resolve GitHub Actions workflow validation errors in docker.yml

- Fix usage of secrets context in conditional expressions
- Add environment variables to build-docker and create-manifest jobs
- Replace 'secrets.DOCKERHUB_USERNAME' with 'env.DOCKERHUB_USERNAME' in if conditions
- Maintain secure handling of Docker Hub credentials through proper env context

* Update .github/workflows/docker.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-11 23:12:58 +08:00
安正超
6267872ddb feat: add latest version support for release builds (#178)
- Add automatic creation of latest version files for release and prerelease builds
- Simplify installation script by providing direct latest URLs
- Support rustfs-linux-{arch}-latest.zip naming convention
- Improve build artifact management and user experience
2025-07-11 23:01:36 +08:00
安正超
618779a89d feat: implement multi-channel release system with artifact naming (#176)
* feat: implement multi-channel release system with artifact naming

- Add dedicated release.yml workflow for handling GitHub releases
- Refactor build.yml to support dev/release/prerelease artifact naming
- Update docker.yml to support version-specific image tagging
- Implement artifact naming rules:
  - Dev: rustfs-{platform}-{arch}-dev-{sha}.zip
  - Release: rustfs-{platform}-{arch}-v{version}.zip
  - Prerelease: rustfs-{platform}-{arch}-v{version}.zip
- Add OSS upload directory separation (dev/ vs release/)
- Only stable releases update latest.json and create latest tags
- Separate GitHub Release creation from build workflow
- Add comprehensive build summaries and status reporting

This enables proper multi-channel distribution with clear artifact
identification and prevents confusion between dev and stable releases.

* fix: support version tags without v prefix (1.0.0 instead of v1.0.0)

- Update trigger patterns from 'v*.*.*' to '*.*.*' in all workflows
- Fix version extraction logic to handle tags without v prefix
- Maintain backward compatibility with existing logic

Note: Artifact naming still includes 'v' prefix for clarity
(e.g., tag '1.0.0' creates 'rustfs-linux-x86_64-v1.0.0.zip')

* feat: update Dockerfile to support multi-channel release system

- Add build arguments for VERSION, BUILD_TYPE, and TARGETARCH
- Support dynamic artifact download based on build type:
  - Development: downloads from artifacts/rustfs/dev/
  - Release: downloads from artifacts/rustfs/release/
- Auto-generate correct filenames based on new naming convention:
  - Dev: rustfs-linux-{arch}-dev-{sha}.zip
  - Release: rustfs-linux-{arch}-v{version}.zip
- Add architecture mapping for multi-platform builds
- Pass BUILD_TYPE parameter from docker.yml workflow
- Improve error handling with helpful download path suggestions

This ensures Docker images use the correct pre-built binaries
from the new multi-channel release system.

* feat: optimize and consolidate Dockerfile structure

## Major Improvements:

###  Created Missing Files
- Add .docker/Dockerfile.alpine for lightweight Alpine-based builds
- Support both pre-built binary download and source compilation

### 🔧 Fixed Critical Issues
- Fix Dockerfile.obs: ubuntu:latest → ubuntu:22.04 (stable version)
- Add proper security practices (non-root user, health checks)
- Add proper error handling and environment variables

### 🗑️ Eliminated Redundancy
- Remove .docker/Dockerfile.ubuntu22.04 (duplicate of devenv)
- Update docker.yml workflow to use devenv for ubuntu variant
- Consolidate similar functionality into fewer, better files

### 🚀 Enhanced Functionality
- Make devenv Dockerfile dual-purpose (dev environment + runtime)
- Add VERSION/BUILD_TYPE support for dynamic binary downloads
- Improve security with proper user management
- Add comprehensive health checks and error handling

### 📊 Final Dockerfile Structure:
1. Dockerfile (production, Alpine-based, pre-built binaries)
2. Dockerfile.multi-stage (full source builds, Ubuntu-based)
3. Dockerfile.obs (observability builds, Ubuntu-based)
4. .docker/Dockerfile.alpine (lightweight Alpine variant)
5. .docker/Dockerfile.devenv (development + ubuntu variant)
6. .docker/Dockerfile.rockylinux9.3 (RockyLinux variant)

This reduces redundancy while maintaining all necessary build variants
and improving maintainability across the entire container ecosystem.

* refactor: streamline Dockerfile structure and remove unused files

## 🎯 Major Cleanup:

### 🗑️ Removed Unused Files (2 files)
- Delete Dockerfile.obs (not referenced anywhere)
- Delete .docker/Dockerfile.rockylinux9.3 (not referenced anywhere)

### 📁 Reorganized File Layout
- Move Dockerfile.multi-stage → .docker/Dockerfile.multi-stage
- Update docker-compose.yml to use new path
- Keep main Dockerfile in root (production use)
- Consolidate variants in .docker/ directory

###  Final Clean Structure:

### 📊 Before vs After:
- **Before**: 7 files (1 missing, 2 unused, scattered layout)
- **After**: 4 files (all used, organized layout)
- **Reduction**: 43% fewer files, 100% utilization

This eliminates confusion and reduces maintenance overhead while
keeping all actually needed functionality intact.

* refactor: implement comprehensive Docker tag strategy with production variant

- Restore production variant as default with explicit naming
- Add support for prerelease channels (alpha, beta, rc)
- Implement rolling development tags (dev, dev-variant)
- Support semantic versioning with variant combinations
- Update documentation with complete tag strategy examples
- Align with GPT-suggested comprehensive tagging approach

Tag examples:
- rustfs/rustfs:1.2.3 (main production)
- rustfs/rustfs:1.2.3-production (explicit production)
- rustfs/rustfs:1.2.3-alpine (Alpine variant)
- rustfs/rustfs:alpha (latest alpha)
- rustfs/rustfs:dev (latest development)
- rustfs/rustfs:dev-13e4a0b (specific commit)

* perf: optimize Docker build speed with comprehensive caching and compilation improvements

- Add dual caching strategy: GitHub Actions + Registry cache
- Implement sccache for Rust compilation caching across builds
- Configure parallel compilation with all available CPU cores
- Add optimized cargo configuration for faster builds
- Enable sparse registry protocol for dependency resolution
- Configure LLD linker for faster linking
- Add BuildKit optimizations with inline cache
- Disable provenance/SBOM generation for faster builds
- Document build performance improvements and timings

Performance improvements:
- Source builds: ~40-50% faster with cache hits
- Pre-built binaries: ~30-40% faster
- Parallel matrix builds reduce total CI time significantly
- Registry cache provides persistent cross-run benefits

* refactor: consolidate Docker variants and eliminate duplication

- Replace root Dockerfile with enhanced Alpine prebuild version
- Remove redundant alpine variant from build matrix
- Root Dockerfile now includes:
  - Non-root user security
  - Health checks
  - Better error handling
  - protoc/flatc tool support
- Update documentation to reflect simplified 4-variant strategy
- Remove duplicate .docker/alpine/Dockerfile.prebuild

Build matrix now:
- production (root Dockerfile - Alpine prebuild)
- alpine-source (Alpine source build)
- ubuntu (Ubuntu prebuild)
- ubuntu-source (Ubuntu source build)

Benefits:
- Eliminates functional duplication
- Improves security with non-root execution
- Maintains same image variants with better quality
- Simplifies maintenance

* fix: restore alpine variant for better user choice

- Restore alpine variant (rustfs/rustfs:1.2.3-alpine)
- Re-add .docker/alpine/Dockerfile.prebuild
- Update build matrix to include 5 variants again:
  - production (default)
  - alpine (explicit Alpine choice)
  - alpine-source (Alpine source build)
  - ubuntu (Ubuntu pre-built)
  - ubuntu-source (Ubuntu source build)
- Update documentation to reflect restored alpine tags
- Fix build performance table to include all variants

User feedback: Alpine variant provides explicit choice even if
similar to production variant. Better UX with clear options.

* fix: remove redundant rustup target add commands in Alpine Dockerfiles

- Remove 'rustup target add x86_64-unknown-linux-musl' from Alpine source build
- Remove redundant target add from Alpine prebuild fallback path
- Remove redundant target add from root Dockerfile fallback path

Reason: rust:alpine base image already has x86_64-unknown-linux-musl
as the default target since Alpine uses musl libc by default.

Thanks to @houseme for spotting this redundancy in code review.

* fix: add missing RUSTFS_VOLUMES environment variable in Dockerfiles

- Add RUSTFS_VOLUMES=/data to all Dockerfile variants
- This fixes the issue where CMD ['/app/rustfs'] was used without providing the required volumes parameter
- The volumes parameter is required by the application and can be provided via command line or RUSTFS_VOLUMES environment variable

* fix: update docker-compose configurations to ensure all environments work correctly

- Added missing access key and secret key environment variables to docker-compose.yaml
- This ensures the distributed test environment has proper authentication credentials
- Complementary fix to the previous Dockerfile updates for consistent configuration

* fix: recreate missing Dockerfile.obs with complete content

- The file was accidentally left empty after initial creation
- Now contains proper Ubuntu-based configuration for observability environment
- Includes all necessary environment variables including RUSTFS_VOLUMES
- Supports docker-compose-obs.yaml configuration

* refactor: organize Docker Compose configurations and eliminate duplication

- Move specialized configurations to .docker/compose/ directory
- Rename docker-compose.yaml → docker-compose.cluster.yaml (distributed testing)
- Rename docker-compose-obs.yaml → docker-compose.observability.yaml (observability testing)
- Keep docker-compose.yml as the main production configuration
- Add comprehensive README explaining different configuration purposes
- Eliminates confusion between similar filenames
- Provides clear guidance on when to use each configuration

* fix: correct relative paths in moved Docker Compose configurations

- Fix binary volume mount paths in docker-compose.cluster.yaml (./target → ../../target)
- Fix Dockerfile.obs context path in docker-compose.observability.yaml (. → ../..)
- Fix observability config file paths (./.docker → ../../.docker)
- Update README.md with correct usage instructions for new locations
- All configurations now correctly reference files relative to their new positions

* refactor: move Dockerfile.obs to .docker/compose/ directory for better organization

- Move Dockerfile.obs from root to .docker/compose/ directory
- Update all dockerfile references in docker-compose.observability.yaml
- Keep related files (Dockerfile.obs + docker-compose.observability.yaml) together
- Clean up root directory by removing specialized-purpose Dockerfile
- Update README.md to document new file organization
- Improves project structure and file discoverability

* refactor: improve Docker build configuration for better clarity

- Move Dockerfile.obs back to project root for simpler build context
- Update docker-compose.observability.yaml to use cleaner dockerfile reference
- Change from '.docker/compose/Dockerfile.obs' to simply 'Dockerfile.obs'
- Maintain context as '../..' for access to project files
- Remove redundant Dockerfile.obs documentation from compose README
- This follows Docker best practices: simple context + Dockerfile at context root

* wip
2025-07-11 22:18:33 +08:00
houseme
b3ec2325ed improve docker comprose config file and remove docs dir (#174)
* refactor(config): Unify S3 API and Console ports

This commit streamlines the server configuration by unifying the S3 API and the WebUI (Console) to serve on a single port.

Previously, the console was managed by separate configuration options (`RUSTFS_CONSOLE_ENABLE` and `RUSTFS_CONSOLE_ADDRESS`), requiring a distinct port. This added complexity to deployment and configuration.

With this change:
- The `RUSTFS_CONSOLE_ADDRESS` and `RUSTFS_CONSOLE_FS_ENDPOINT` environment variables are removed.
- The WebUI is now always available and served directly from the main application port defined by `RUSTFS_ADDRESS`.
- This simplifies setup, reduces the number of exposed ports, and makes the application easier to manage and deploy, especially in containerized environments.

Users should update their startup scripts and remove the deprecated `RUSTFS_CONSOLE_*` variables.

* improve docker comprose config file and remove docs dir
2025-07-11 16:55:24 +08:00
houseme
49a5643e76 refactor(config): Unify S3 API and Console ports (#173)
This commit streamlines the server configuration by unifying the S3 API and the WebUI (Console) to serve on a single port.

Previously, the console was managed by separate configuration options (`RUSTFS_CONSOLE_ENABLE` and `RUSTFS_CONSOLE_ADDRESS`), requiring a distinct port. This added complexity to deployment and configuration.

With this change:
- The `RUSTFS_CONSOLE_ADDRESS` and `RUSTFS_CONSOLE_FS_ENDPOINT` environment variables are removed.
- The WebUI is now always available and served directly from the main application port defined by `RUSTFS_ADDRESS`.
- This simplifies setup, reduces the number of exposed ports, and makes the application easier to manage and deploy, especially in containerized environments.

Users should update their startup scripts and remove the deprecated `RUSTFS_CONSOLE_*` variables.
2025-07-11 14:20:22 +08:00
loverustfs
657395af8a fix docker quickstart 2025-07-11 10:59:11 +08:00
loverustfs
4de62ed77e fix quickstart 2025-07-11 10:58:22 +08:00
houseme
505f493729 chore: bump workspace dependencies versions (#168)
* upgrade package version

# Conflicts:
#	crates/rio/Cargo.toml

* fix

* upgrade version

* upgrade version

* cargo fmt
2025-07-11 10:35:27 +08:00
weisd
be05b704b0 feat: add Content-Length headers to admin API responses (#169) 2025-07-11 09:40:57 +08:00
安正超
b33c2fa3cf Update build.yml 2025-07-11 09:00:06 +08:00
安正超
98674c60d4 Update README.md 2025-07-11 08:44:50 +08:00
安正超
e39eb86967 fix: remove unused command 2025-07-11 08:03:29 +08:00
1094 changed files with 214709 additions and 62315 deletions

View File

@@ -0,0 +1,78 @@
---
name: code-change-verification
description: Verify code changes by identifying correctness, regression, security, and performance risks from diffs or patches, then produce prioritized findings with file/line evidence and concrete fixes. Use when reviewing commits, PRs, and merged patches before/after release.
---
# Code Change Verification
Use this skill to review code changes consistently before merge, before release, and during incident follow-up.
## Quick Start
1. Read the scope: commit, PR, patch, or file list.
2. Map each changed area by risk and user impact.
3. Inspect each risky change in context.
4. Report findings first, ordered by severity.
5. Close with residual risks and verification recommendations.
## Core Workflow
### 1) Scope and assumptions
- Confirm change source (diff, commit, PR, files), target branch, language/runtime, and version.
- If context is missing, state assumptions before deeper analysis.
- Focus only on requested scope; avoid reviewing unrelated files.
### 2) Risk map
- Prioritize in this order:
- Data correctness and user-visible behavior
- API/contract compatibility
- Security and authz/authn boundaries
- Concurrency and lifecycle correctness
- Performance and resource usage
- Give higher priority to stateful paths, migration logic, defaults, and error handling.
### 3) Evidence-based inspection
- Read each modified hunk with neighboring context.
- Trace call paths and call-site expectations.
- Check for:
- invariant breaks and missing guards
- unchecked assumptions and null/empty/error-path handling
- stale tests, fixtures, and configs
- hidden coupling to shared helpers/constants/features
- If a point is uncertain, mark it as an open question instead of guessing.
### 4) Findings-first output
- Order findings by severity:
- P0: critical failure, security breach, or data loss risk
- P1: high-impact regression
- P2: medium risk correctness gap
- P3: low risk/quality debt
- For each finding include:
- Severity
- `path:line` reference
- concise issue statement
- impact and likely failure mode
- specific fix or mitigation
- validation step to confirm
- If no issues exist, explicitly state `No findings` and why.
### 5) Close
- Report assumptions and unknowns.
- Suggest targeted checks (tests, canary checks, logs/metrics, migration validation).
## Output Template
1. Findings
2. No findings (if applicable)
3. Assumptions / Unknowns
4. Recommended verification steps
## Finding Template
- `[P1] Missing timeout for downstream call`
- Location: `path/to/file.rs:123`
- Issue: ...
- Impact: ...
- Fix suggestion: ...
- Validation: ...

View File

@@ -0,0 +1,4 @@
interface:
display_name: "Code Change Verification"
short_description: "Prioritize risks and verify code changes before merge."
default_prompt: "Inspect a patch or diff, identify correctness/security/regression risks, and return prioritized findings with file/line evidence and fixes."

View File

@@ -0,0 +1,88 @@
---
name: pr-creation-checker
description: Prepare PR-ready diffs by validating scope, checking required verification steps, drafting a compliant English PR title/body, and surfacing blockers before opening or updating a pull request in RustFS.
---
# PR Creation Checker
Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whether a branch is ready for PR.
## Read sources of truth first
- Read `AGENTS.md`.
- Read `.github/pull_request_template.md`.
- Use `Makefile` and `.config/make/` for local quality commands.
- Use `.github/workflows/ci.yml` for CI expectations.
- Do not restate long command matrices or template sections from memory when the files exist.
## Workflow
1. Collect PR context
- Confirm base branch, current branch, change goal, and scope.
- Confirm whether the task is: draft a new PR, update an existing PR, or preflight-check readiness.
- Confirm whether the branch includes only intended changes.
2. Inspect change scope
- Review the diff and summarize what changed.
- Call out unrelated edits, generated artifacts, logs, or secrets as blockers.
- Mark risky areas explicitly: auth, storage, config, network, migrations, breaking changes.
3. Verify readiness requirements
- Require `make pre-commit` before marking the PR ready.
- If `make` is unavailable, use the equivalent commands from `.config/make/`.
- Add scope-specific verification commands when the changed area needs more than the baseline.
- If required checks fail, stop and return `BLOCKED`.
4. Draft PR metadata
- Write the PR title in English using Conventional Commits and keep it within 72 characters.
- If a generic PR workflow suggests a different title format, ignore it and follow the repository rule instead.
- In RustFS, do not use tool-specific prefixes such as `[codex]` when the repository requires Conventional Commits.
- Keep the PR body in English.
- Use the exact section headings from `.github/pull_request_template.md`.
- Fill non-applicable sections with `N/A`.
- Include verification commands in the PR description.
- Do not include local filesystem paths in the PR body unless the user explicitly asks for them.
- Prefer repo-relative paths, command names, and concise summaries over machine-specific paths such as `/Users/...`.
5. Prepare reviewer context
- Summarize why the change exists.
- Summarize what was verified.
- Call out risks, rollout notes, config impact, and rollback notes when applicable.
- Mention assumptions or missing context instead of guessing.
6. Prepare CLI-safe output
- When proposing `gh pr create` or `gh pr edit`, use `--body-file`, never inline `--body` for multiline markdown.
- Return a ready-to-save PR body plus a short title.
- If not ready, return blockers first and list the minimum steps needed to unblock.
## Output format
### Status
- `READY` or `BLOCKED`
### Title
- `<type>(<scope>): <summary>`
### PR Body
- Reproduce the repository template headings exactly.
- Fill every section.
- Omit local absolute paths unless explicitly required.
### Verification
- List each command run.
- State pass/fail.
### Risks
- List breaking changes, config changes, migration impact, or `N/A`.
## Blocker rules
- Return `BLOCKED` if `make pre-commit` has not passed.
- Return `BLOCKED` if the diff contains unrelated changes that are not acknowledged.
- Return `BLOCKED` if required template sections are missing.
- Return `BLOCKED` if the title/body is not in English.
- Return `BLOCKED` if the title does not follow the repository's Conventional Commit rule.
## Reference
- Use [pr-readiness-checklist.md](references/pr-readiness-checklist.md) for a short final pass before opening or editing the PR.

View File

@@ -0,0 +1,4 @@
interface:
display_name: "PR Creation Checker"
short_description: "Draft RustFS-ready PRs with checks, template, and blockers."
default_prompt: "Inspect a branch or diff, verify required PR checks, and produce a compliant English PR title/body plus blockers or readiness status."

View File

@@ -0,0 +1,14 @@
# PR Readiness Checklist
- Confirm the branch is based on current `main`.
- Confirm the diff matches the stated scope.
- Confirm no secrets, logs, temp files, or unrelated refactors are included.
- Confirm `make pre-commit` passed, or document why it could not run.
- Confirm extra verification commands are listed for risky changes.
- Confirm the PR title uses Conventional Commits and stays within 72 characters.
- Confirm the PR title does not use tool-specific prefixes such as `[codex]`.
- Confirm the PR body is in English.
- Confirm the PR body keeps the exact headings from `.github/pull_request_template.md`.
- Confirm non-applicable sections are filled with `N/A`.
- Confirm the PR body does not include local absolute paths unless explicitly required.
- Confirm multiline GitHub CLI commands use `--body-file`.

View File

@@ -0,0 +1,66 @@
---
name: test-coverage-improver
description: Run project coverage checks, rank high-risk gaps, and propose high-impact tests to improve regression confidence for changed and critical code paths before release.
---
# Test Coverage Improver
Use this skill when you need a prioritized, risk-aware plan to improve tests from coverage results.
## Usage assumptions
- Focus scope is either changed lines/files, a module, or the whole repository.
- Coverage artifact must be generated or provided in a supported format.
- If required context is missing, call out assumptions explicitly before proposing work.
## Workflow
1. Define scope and baseline
- Confirm target language, framework, and branch.
- Confirm whether the scope is changed files only or full-repo.
2. Produce coverage snapshot
- Rust: `cargo llvm-cov` (or `cargo tarpaulin`) with existing repo config.
- JavaScript/TypeScript: `npm test -- --coverage` and read `coverage/coverage-final.json`.
- Python: `pytest --cov=<pkg> --cov-report=json` and read `coverage.json`.
- Collect total, per-file, and changed-line coverage.
3. Rank highest-risk gaps
- Prioritize changed code, branch coverage gaps, and low-confidence boundaries.
- Apply the risk rubric in [coverage-prioritization.md](references/coverage-prioritization.md).
- Keep shortlist to 58 gaps.
- For each gap, capture: file, lines, uncovered branches, and estimated risk score.
4. Propose high-impact tests
- For each shortlisted gap, output:
- Intent and expected behavior.
- Normal, edge, and failure scenarios.
- Assertions and side effects to verify.
- Setup needs (fixtures, mocks, integration dependencies).
- Estimated effort (`S/M/L`).
5. Close with validation plan
- State which gaps remain after proposals.
- Provide concrete verification command and acceptance threshold.
- List assumptions or blockers (environment, fixtures, flaky dependencies).
## Output template
### Coverage Snapshot
- total / branch coverage
- changed-file coverage
- top missing regions by size
### Top Gaps (ranked)
- `path:line-range` | risk score | why critical
### Test Proposals
- `path:line-range`
- Test name
- scenarios
- assertions
- effort
### Validation Plan
- command
- pass criteria
- remaining risk

View File

@@ -0,0 +1,4 @@
interface:
display_name: "Test Coverage Improver"
short_description: "Find top uncovered risk areas and propose high-impact tests."
default_prompt: "Run coverage checks, identify largest gaps, and recommend highest-impact test cases to improve risk coverage."

View File

@@ -0,0 +1,25 @@
# Coverage Gap Prioritization Guide
Use this rubric for each uncovered area.
Score = (Criticality × 2) + CoverageDebt + (Volatility × 0.5)
- Criticality:
- 5: authz/authn, data-loss, payment/consistency path
- 4: state mutation, cache invalidation, scheduling
- 3: error handling + fallbacks in user-visible flows
- 2: parsing/format conversion paths
- 1: logging-only or low-impact utilities
- CoverageDebt:
- 0: 05 uncovered lines
- 1: 620 uncovered lines
- 2: 2140 uncovered lines
- 3: 41+ uncovered lines
- Volatility:
- 1: stable legacy code with few recent edits
- 2: changed in last 2 releases
- 3: touched in last 30 days or currently in active PR
Sort by score descending, then by business impact.

View File

@@ -0,0 +1,64 @@
## —— Development/Source builds using direct buildx commands ---------------------------------------
.PHONY: docker-dev
docker-dev: ## Build dev multi-arch image (cannot load locally)
@echo "🏗️ Building multi-architecture development Docker images with buildx..."
@echo "💡 This builds from source code and is intended for local development and testing"
@echo "⚠️ Multi-arch images cannot be loaded locally, use docker-dev-push to push to registry"
$(DOCKER_CLI) buildx build \
--platform linux/amd64,linux/arm64 \
--file $(DOCKERFILE_SOURCE) \
--tag rustfs:source-latest \
--tag rustfs:dev-latest \
.
.PHONY: docker-dev-local
docker-dev-local: ## Build dev single-arch image (local load)
@echo "🏗️ Building single-architecture development Docker image for local use..."
@echo "💡 This builds from source code for the current platform and loads locally"
$(DOCKER_CLI) buildx build \
--file $(DOCKERFILE_SOURCE) \
--tag rustfs:source-latest \
--tag rustfs:dev-latest \
--load \
.
.PHONY: docker-dev-push
docker-dev-push: ## Build and push multi-arch development image # e.g (make docker-dev-push REGISTRY=xxx)
@if [ -z "$(REGISTRY)" ]; then \
echo "❌ Error: Please specify registry, example: make docker-dev-push REGISTRY=ghcr.io/username"; \
exit 1; \
fi
@echo "🚀 Building and pushing multi-architecture development Docker images..."
@echo "💡 Pushing to registry: $(REGISTRY)"
$(DOCKER_CLI) buildx build \
--platform linux/amd64,linux/arm64 \
--file $(DOCKERFILE_SOURCE) \
--tag $(REGISTRY)/rustfs:source-latest \
--tag $(REGISTRY)/rustfs:dev-latest \
--push \
.
.PHONY: dev-env-start
dev-env-start: ## Start development container environment
@echo "🚀 Starting development environment..."
$(DOCKER_CLI) buildx build \
--file $(DOCKERFILE_SOURCE) \
--tag rustfs:dev \
--load \
.
$(DOCKER_CLI) stop $(CONTAINER_NAME) 2>/dev/null || true
$(DOCKER_CLI) rm $(CONTAINER_NAME) 2>/dev/null || true
$(DOCKER_CLI) run -d --name $(CONTAINER_NAME) \
-p 9010:9010 -p 9000:9000 \
-v $(shell pwd):/workspace \
-it rustfs:dev
.PHONY: dev-env-stop
dev-env-stop: ## Stop development container environment
@echo "🛑 Stopping development environment..."
$(DOCKER_CLI) stop $(CONTAINER_NAME) 2>/dev/null || true
$(DOCKER_CLI) rm $(CONTAINER_NAME) 2>/dev/null || true
.PHONY: dev-env-restart
dev-env-restart: dev-env-stop dev-env-start ## Restart development container environment

View File

@@ -0,0 +1,41 @@
## —— Production builds using docker buildx (for CI/CD and production) -----------------------------
.PHONY: docker-buildx
docker-buildx: ## Build production multi-arch image (no push)
@echo "🏗️ Building multi-architecture production Docker images with buildx..."
./docker-buildx.sh
.PHONY: docker-buildx-push
docker-buildx-push: ## Build and push production multi-arch image
@echo "🚀 Building and pushing multi-architecture production Docker images with buildx..."
./docker-buildx.sh --push
.PHONY: docker-buildx-version
docker-buildx-version: ## Build and version production multi-arch image # e.g (make docker-buildx-version VERSION=v1.0.0)
@if [ -z "$(VERSION)" ]; then \
echo "❌ Error: Please specify version, example: make docker-buildx-version VERSION=v1.0.0"; \
exit 1; \
fi
@echo "🏗️ Building multi-architecture production Docker images (version: $(VERSION))..."
./docker-buildx.sh --release $(VERSION)
.PHONY: docker-buildx-push-version
docker-buildx-push-version: ## Build and version and push production multi-arch image # e.g (make docker-buildx-push-version VERSION=v1.0.0)
@if [ -z "$(VERSION)" ]; then \
echo "❌ Error: Please specify version, example: make docker-buildx-push-version VERSION=v1.0.0"; \
exit 1; \
fi
@echo "🚀 Building and pushing multi-architecture production Docker images (version: $(VERSION))..."
./docker-buildx.sh --release $(VERSION) --push
.PHONY: docker-buildx-production-local
docker-buildx-production-local: ## Build production single-arch image locally
@echo "🏗️ Building single-architecture production Docker image locally..."
@echo "💡 Alternative to docker-buildx.sh for local testing"
$(DOCKER_CLI) buildx build \
--file $(DOCKERFILE_PRODUCTION) \
--tag rustfs:production-latest \
--tag rustfs:latest \
--load \
--build-arg RELEASE=latest \
.

View File

@@ -0,0 +1,16 @@
## —— Single Architecture Docker Builds (Traditional) ----------------------------------------------
.PHONY: docker-build-production
docker-build-production: ## Build single-arch production image
@echo "🏗️ Building single-architecture production Docker image..."
@echo "💡 Consider using 'make docker-buildx-production-local' for multi-arch support"
$(DOCKER_CLI) build -f $(DOCKERFILE_PRODUCTION) -t rustfs:latest .
.PHONY: docker-build-source
docker-build-source: ## Build single-arch source image
@echo "🏗️ Building single-architecture source Docker image..."
@echo "💡 Consider using 'make docker-dev-local' for multi-arch support"
DOCKER_BUILDKIT=1 $(DOCKER_CLI) build \
--build-arg BUILDKIT_INLINE_CACHE=1 \
-f $(DOCKERFILE_SOURCE) -t rustfs:source .

View File

@@ -0,0 +1,22 @@
## —— Docker-based build (alternative approach) ----------------------------------------------------
# Usage: make BUILD_OS=ubuntu22.04 build-docker
# Output: target/ubuntu22.04/release/rustfs
.PHONY: build-docker
build-docker: SOURCE_BUILD_IMAGE_NAME = rustfs-$(BUILD_OS):v1
build-docker: SOURCE_BUILD_CONTAINER_NAME = rustfs-$(BUILD_OS)-build
build-docker: BUILD_CMD = /root/.cargo/bin/cargo build --release --bin rustfs --target-dir /root/s3-rustfs/target/$(BUILD_OS)
build-docker: ## Build using Docker container # e.g (make build-docker BUILD_OS=ubuntu22.04)
@echo "🐳 Building RustFS using Docker ($(BUILD_OS))..."
$(DOCKER_CLI) buildx build -t $(SOURCE_BUILD_IMAGE_NAME) -f $(DOCKERFILE_SOURCE) .
$(DOCKER_CLI) run --rm --name $(SOURCE_BUILD_CONTAINER_NAME) -v $(shell pwd):/root/s3-rustfs -it $(SOURCE_BUILD_IMAGE_NAME) $(BUILD_CMD)
.PHONY: docker-inspect-multiarch
docker-inspect-multiarch: ## Check image architecture support
@if [ -z "$(IMAGE)" ]; then \
echo "❌ Error: Please specify image, example: make docker-inspect-multiarch IMAGE=rustfs/rustfs:latest"; \
exit 1; \
fi
@echo "🔍 Inspecting multi-architecture image: $(IMAGE)"
docker buildx imagetools inspect $(IMAGE)

55
.config/make/build.mak Normal file
View File

@@ -0,0 +1,55 @@
## —— Local Native Build using build-rustfs.sh script (Recommended) --------------------------------
.PHONY: build
build: ## Build RustFS binary (includes console by default)
@echo "🔨 Building RustFS using build-rustfs.sh script..."
./build-rustfs.sh
.PHONY: build-dev
build-dev: ## Build RustFS in Development mode
@echo "🔨 Building RustFS in development mode..."
./build-rustfs.sh --dev
.PHONY: build-musl
build-musl: ## Build x86_64 musl version
@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
./build-rustfs.sh --platform x86_64-unknown-linux-musl
.PHONY: build-gnu
build-gnu: ## Build x86_64 GNU version
@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
./build-rustfs.sh --platform x86_64-unknown-linux-gnu
.PHONY: build-musl-arm64
build-musl-arm64: ## Build aarch64 musl version
@echo "🔨 Building rustfs for aarch64-unknown-linux-musl..."
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
./build-rustfs.sh --platform aarch64-unknown-linux-musl
.PHONY: build-gnu-arm64
build-gnu-arm64: ## Build aarch64 GNU version
@echo "🔨 Building rustfs for aarch64-unknown-linux-gnu..."
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
./build-rustfs.sh --platform aarch64-unknown-linux-gnu
.PHONY: build-cross-all
build-cross-all: core-deps ## Build binaries for all architectures
@echo "🔧 Building all target architectures..."
@echo "💡 On macOS/Windows, use 'make docker-dev' for reliable multi-arch builds"
@echo "🔨 Generating protobuf code..."
cargo run --bin gproto || true
@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
./build-rustfs.sh --platform x86_64-unknown-linux-musl
@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
./build-rustfs.sh --platform x86_64-unknown-linux-gnu
@echo "🔨 Building rustfs for aarch64-unknown-linux-musl..."
./build-rustfs.sh --platform aarch64-unknown-linux-musl
@echo "🔨 Building rustfs for aarch64-unknown-linux-gnu..."
./build-rustfs.sh --platform aarch64-unknown-linux-gnu

24
.config/make/check.mak Normal file
View File

@@ -0,0 +1,24 @@
## —— Check and Inform Dependencies ----------------------------------------------------------------
# Fatal check
# Checks all required dependencies and exits with error if not found
# (e.g., cargo, rustfmt)
check-%:
@command -v $* >/dev/null 2>&1 || { \
echo >&2 "❌ '$*' is not installed."; \
exit 1; \
}
# Warning-only check
# Checks for optional dependencies and issues a warning if not found
# (e.g., cargo-nextest for enhanced testing)
warn-%:
@command -v $* >/dev/null 2>&1 || { \
echo >&2 "⚠️ '$*' is not installed."; \
}
# For checking dependencies use check-<dep-name> or warn-<dep-name>
.PHONY: core-deps fmt-deps test-deps
core-deps: check-cargo ## Check core dependencies
fmt-deps: check-rustfmt ## Check lint and formatting dependencies
test-deps: warn-cargo-nextest ## Check tests dependencies

6
.config/make/deploy.mak Normal file
View File

@@ -0,0 +1,6 @@
## —— Deploy using dev_deploy.sh script ------------------------------------------------------------
.PHONY: deploy-dev
deploy-dev: build-musl ## Deploy to dev server
@echo "🚀 Deploying to dev server: $${IP}"
./scripts/dev_deploy.sh $${IP}

38
.config/make/help.mak Normal file
View File

@@ -0,0 +1,38 @@
## —— Help, Help Build and Help Docker -------------------------------------------------------------
.PHONY: help
help: ## Shows This Help Menu
echo -e "$$HEADER"
grep -E '(^[a-zA-Z0-9_-]+:.*?## .*$$)|(^## )' $(MAKEFILE_LIST) | sed 's/^[^:]*://g' | awk 'BEGIN {FS = ":.*?## | #"} ; {printf "${cyan}%-30s${reset} ${white}%s${reset} ${green}%s${reset}\n", $$1, $$2, $$3}' | sed -e 's/\[36m##/\n[32m##/'
.PHONY: help-build
help-build: ## Shows RustFS build help
@echo ""
@echo "💡 build-rustfs.sh script provides more options, smart detection and binary verification"
@echo ""
@echo "🔧 Direct usage of build-rustfs.sh script:"
@echo ""
@echo " ./build-rustfs.sh --help # View script help"
@echo " ./build-rustfs.sh --no-console # Build without console resources"
@echo " ./build-rustfs.sh --force-console-update # Force update console resources"
@echo " ./build-rustfs.sh --dev # Development mode build"
@echo " ./build-rustfs.sh --sign # Sign binary files"
@echo " ./build-rustfs.sh --platform x86_64-unknown-linux-gnu # Specify target platform"
@echo " ./build-rustfs.sh --skip-verification # Skip binary verification"
@echo ""
.PHONY: help-docker
help-docker: ## Shows docker environment and suggestion help
@echo ""
@echo "📋 Environment Variables:"
@echo " REGISTRY Image registry address (required for push)"
@echo " DOCKERHUB_USERNAME Docker Hub username"
@echo " DOCKERHUB_TOKEN Docker Hub access token"
@echo " GITHUB_TOKEN GitHub access token"
@echo ""
@echo "💡 Suggestions:"
@echo " Production use: Use docker-buildx* commands (based on precompiled binaries)"
@echo " Local development: Use docker-dev* commands (build from source)"
@echo " Development environment: Use dev-env-* commands to manage dev containers"
@echo ""

22
.config/make/lint-fmt.mak Normal file
View File

@@ -0,0 +1,22 @@
## —— Code quality and Formatting ------------------------------------------------------------------
.PHONY: fmt
fmt: core-deps fmt-deps ## Format code
@echo "🔧 Formatting code..."
cargo fmt --all
.PHONY: fmt-check
fmt-check: core-deps fmt-deps ## Check code formatting
@echo "📝 Checking code formatting..."
cargo fmt --all --check
.PHONY: clippy-check
clippy-check: core-deps ## Run clippy checks
@echo "🔍 Running clippy checks..."
cargo clippy --fix --allow-dirty
cargo clippy --all-targets --all-features -- -D warnings
.PHONY: compilation-check
compilation-check: core-deps ## Run compilation check
@echo "🔨 Running compilation check..."
cargo check --all-targets

View File

@@ -0,0 +1,11 @@
## —— Pre Commit Checks ----------------------------------------------------------------------------
.PHONY: setup-hooks
setup-hooks: ## Set up git hooks
@echo "🔧 Setting up git hooks..."
chmod +x .git/hooks/pre-commit
@echo "✅ Git hooks setup complete!"
.PHONY: pre-commit
pre-commit: fmt clippy-check compilation-check test ## Run pre-commit checks
@echo "✅ All pre-commit checks passed!"

22
.config/make/tests.mak Normal file
View File

@@ -0,0 +1,22 @@
## —— Tests and e2e test ---------------------------------------------------------------------------
TEST_THREADS ?= 1
.PHONY: test
test: core-deps test-deps ## Run all tests
@echo "🧪 Running tests..."
@if command -v cargo-nextest >/dev/null 2>&1; then \
cargo nextest run --all --exclude e2e_test; \
else \
echo " cargo-nextest not found; falling back to 'cargo test'"; \
cargo test --workspace --exclude e2e_test -- --nocapture --test-threads="$(TEST_THREADS)"; \
fi
cargo test --all --doc
.PHONY: e2e-server
e2e-server: ## Run e2e-server tests
sh $(shell pwd)/scripts/run.sh
.PHONY: probe-e2e
probe-e2e: ## Probe e2e tests
sh $(shell pwd)/scripts/probe.sh

View File

@@ -1,910 +0,0 @@
# RustFS Project Cursor Rules
## ⚠️ CRITICAL DEVELOPMENT RULES ⚠️
### 🚨 NEVER COMMIT DIRECTLY TO MASTER/MAIN BRANCH 🚨
- **This is the most important rule - NEVER modify code directly on main or master branch**
- **ALL CHANGES MUST GO THROUGH PULL REQUESTS - NO EXCEPTIONS**
- **Always work on feature branches and use pull requests for all changes**
- **Any direct commits to master/main branch are strictly forbidden**
- **Pull requests are the ONLY way to merge code to main branch**
- Before starting any development, always:
1. `git checkout main` (switch to main branch)
2. `git pull` (get latest changes)
3. `git checkout -b feat/your-feature-name` (create and switch to feature branch)
4. Make your changes on the feature branch
5. Commit and push to the feature branch
6. **Create a pull request for review - THIS IS MANDATORY**
7. **Wait for PR approval and merge through GitHub interface only**
## Project Overview
RustFS is a high-performance distributed object storage system written in Rust, compatible with S3 API. The project adopts a modular architecture, supporting erasure coding storage, multi-tenant management, observability, and other enterprise-level features.
## Core Architecture Principles
### 1. Modular Design
- Project uses Cargo workspace structure, containing multiple independent crates
- Core modules: `rustfs` (main service), `ecstore` (erasure coding storage), `common` (shared components)
- Functional modules: `iam` (identity management), `madmin` (management interface), `crypto` (encryption), etc.
- Tool modules: `cli` (command line tool), `crates/*` (utility libraries)
### 2. Asynchronous Programming Pattern
- Comprehensive use of `tokio` async runtime
- Prioritize `async/await` syntax
- Use `async-trait` for async methods in traits
- Avoid blocking operations, use `spawn_blocking` when necessary
### 3. Error Handling Strategy
- **Use modular, type-safe error handling with `thiserror`**
- Each module should define its own error type using `thiserror::Error` derive macro
- Support error chains and context information through `#[from]` and `#[source]` attributes
- Use `Result<T>` type aliases for consistency within each module
- Error conversion between modules should use explicit `From` implementations
- Follow the pattern: `pub type Result<T> = core::result::Result<T, Error>`
- Use `#[error("description")]` attributes for clear error messages
- Support error downcasting when needed through `other()` helper methods
- Implement `Clone` for errors when required by the domain logic
- **Current module error types:**
- `ecstore::error::StorageError` - Storage layer errors
- `ecstore::disk::error::DiskError` - Disk operation errors
- `iam::error::Error` - Identity and access management errors
- `policy::error::Error` - Policy-related errors
- `crypto::error::Error` - Cryptographic operation errors
- `filemeta::error::Error` - File metadata errors
- `rustfs::error::ApiError` - API layer errors
- Module-specific error types for specialized functionality
## Code Style Guidelines
### 1. Formatting Configuration
```toml
max_width = 130
fn_call_width = 90
single_line_let_else_max_width = 100
```
### 2. **🔧 MANDATORY Code Formatting Rules**
**CRITICAL**: All code must be properly formatted before committing. This project enforces strict formatting standards to maintain code consistency and readability.
#### Pre-commit Requirements (MANDATORY)
Before every commit, you **MUST**:
1. **Format your code**:
```bash
cargo fmt --all
```
2. **Verify formatting**:
```bash
cargo fmt --all --check
```
3. **Pass clippy checks**:
```bash
cargo clippy --all-targets --all-features -- -D warnings
```
4. **Ensure compilation**:
```bash
cargo check --all-targets
```
#### Quick Commands
Use these convenient Makefile targets for common tasks:
```bash
# Format all code
make fmt
# Check if code is properly formatted
make fmt-check
# Run clippy checks
make clippy
# Run compilation check
make check
# Run tests
make test
# Run all pre-commit checks (format + clippy + check + test)
make pre-commit
# Setup git hooks (one-time setup)
make setup-hooks
```
#### 🔒 Automated Pre-commit Hooks
This project includes a pre-commit hook that automatically runs before each commit to ensure:
- ✅ Code is properly formatted (`cargo fmt --all --check`)
- ✅ No clippy warnings (`cargo clippy --all-targets --all-features -- -D warnings`)
- ✅ Code compiles successfully (`cargo check --all-targets`)
**Setting Up Pre-commit Hooks** (MANDATORY for all developers):
Run this command once after cloning the repository:
```bash
make setup-hooks
```
Or manually:
```bash
chmod +x .git/hooks/pre-commit
```
#### 🚫 Commit Prevention
If your code doesn't meet the formatting requirements, the pre-commit hook will:
1. **Block the commit** and show clear error messages
2. **Provide exact commands** to fix the issues
3. **Guide you through** the resolution process
Example output when formatting fails:
```
❌ Code formatting check failed!
💡 Please run 'cargo fmt --all' to format your code before committing.
🔧 Quick fix:
cargo fmt --all
git add .
git commit
```
### 3. Naming Conventions
- Use `snake_case` for functions, variables, modules
- Use `PascalCase` for types, traits, enums
- Constants use `SCREAMING_SNAKE_CASE`
- Global variables prefix `GLOBAL_`, e.g., `GLOBAL_Endpoints`
- Use meaningful and descriptive names for variables, functions, and methods
- Avoid meaningless names like `temp`, `data`, `foo`, `bar`, `test123`
- Choose names that clearly express the purpose and intent
### 4. Type Declaration Guidelines
- **Prefer type inference over explicit type declarations** when the type is obvious from context
- Let the Rust compiler infer types whenever possible to reduce verbosity and improve maintainability
- Only specify types explicitly when:
- The type cannot be inferred by the compiler
- Explicit typing improves code clarity and readability
- Required for API boundaries (function signatures, public struct fields)
- Needed to resolve ambiguity between multiple possible types
**Good examples (prefer these):**
```rust
// Compiler can infer the type
let items = vec![1, 2, 3, 4];
let config = Config::default();
let result = process_data(&input);
// Iterator chains with clear context
let filtered: Vec<_> = items.iter().filter(|&&x| x > 2).collect();
```
**Avoid unnecessary explicit types:**
```rust
// Unnecessary - type is obvious
let items: Vec<i32> = vec![1, 2, 3, 4];
let config: Config = Config::default();
let result: ProcessResult = process_data(&input);
```
**When explicit types are beneficial:**
```rust
// API boundaries - always specify types
pub fn process_data(input: &[u8]) -> Result<ProcessResult, Error> { ... }
// Ambiguous cases - explicit type needed
let value: f64 = "3.14".parse().unwrap();
// Complex generic types - explicit for clarity
let cache: HashMap<String, Arc<Mutex<CacheEntry>>> = HashMap::new();
```
### 5. Documentation Comments
- Public APIs must have documentation comments
- Use `///` for documentation comments
- Complex functions add `# Examples` and `# Parameters` descriptions
- Error cases use `# Errors` descriptions
- Always use English for all comments and documentation
- Avoid meaningless comments like "debug 111" or placeholder text
### 6. Import Guidelines
- Standard library imports first
- Third-party crate imports in the middle
- Project internal imports last
- Group `use` statements with blank lines between groups
## Asynchronous Programming Guidelines
### 1. Trait Definition
```rust
#[async_trait::async_trait]
pub trait StorageAPI: Send + Sync {
async fn get_object(&self, bucket: &str, object: &str) -> Result<ObjectInfo>;
}
```
### 2. Error Handling
```rust
// Use ? operator to propagate errors
async fn example_function() -> Result<()> {
let data = read_file("path").await?;
process_data(data).await?;
Ok(())
}
```
### 3. Concurrency Control
- Use `Arc` and `Mutex`/`RwLock` for shared state management
- Prioritize async locks from `tokio::sync`
- Avoid holding locks for long periods
## Logging and Tracing Guidelines
### 1. Tracing Usage
```rust
#[tracing::instrument(skip(self, data))]
async fn process_data(&self, data: &[u8]) -> Result<()> {
info!("Processing {} bytes", data.len());
// Implementation logic
}
```
### 2. Log Levels
- `error!`: System errors requiring immediate attention
- `warn!`: Warning information that may affect functionality
- `info!`: Important business information
- `debug!`: Debug information for development use
- `trace!`: Detailed execution paths
### 3. Structured Logging
```rust
info!(
counter.rustfs_api_requests_total = 1_u64,
key_request_method = %request.method(),
key_request_uri_path = %request.uri().path(),
"API request processed"
);
```
## Error Handling Guidelines
### 1. Error Type Definition
```rust
// Use thiserror for module-specific error types
#[derive(thiserror::Error, Debug)]
pub enum MyError {
#[error("IO error: {0}")]
Io(#[from] std::io::Error),
#[error("Storage error: {0}")]
Storage(#[from] ecstore::error::StorageError),
#[error("Custom error: {message}")]
Custom { message: String },
#[error("File not found: {path}")]
FileNotFound { path: String },
#[error("Invalid configuration: {0}")]
InvalidConfig(String),
}
// Provide Result type alias for the module
pub type Result<T> = core::result::Result<T, MyError>;
```
### 2. Error Helper Methods
```rust
impl MyError {
/// Create error from any compatible error type
pub fn other<E>(error: E) -> Self
where
E: Into<Box<dyn std::error::Error + Send + Sync>>,
{
MyError::Io(std::io::Error::other(error))
}
}
```
### 3. Error Conversion Between Modules
```rust
// Convert between different module error types
impl From<ecstore::error::StorageError> for MyError {
fn from(e: ecstore::error::StorageError) -> Self {
match e {
ecstore::error::StorageError::FileNotFound => {
MyError::FileNotFound { path: "unknown".to_string() }
}
_ => MyError::Storage(e),
}
}
}
// Provide reverse conversion when needed
impl From<MyError> for ecstore::error::StorageError {
fn from(e: MyError) -> Self {
match e {
MyError::FileNotFound { .. } => ecstore::error::StorageError::FileNotFound,
MyError::Storage(e) => e,
_ => ecstore::error::StorageError::other(e),
}
}
}
```
### 4. Error Context and Propagation
```rust
// Use ? operator for clean error propagation
async fn example_function() -> Result<()> {
let data = read_file("path").await?;
process_data(data).await?;
Ok(())
}
// Add context to errors
fn process_with_context(path: &str) -> Result<()> {
std::fs::read(path)
.map_err(|e| MyError::Custom {
message: format!("Failed to read {}: {}", path, e)
})?;
Ok(())
}
```
### 5. API Error Conversion (S3 Example)
```rust
// Convert storage errors to API-specific errors
use s3s::{S3Error, S3ErrorCode};
#[derive(Debug)]
pub struct ApiError {
pub code: S3ErrorCode,
pub message: String,
pub source: Option<Box<dyn std::error::Error + Send + Sync>>,
}
impl From<ecstore::error::StorageError> for ApiError {
fn from(err: ecstore::error::StorageError) -> Self {
let code = match &err {
ecstore::error::StorageError::BucketNotFound(_) => S3ErrorCode::NoSuchBucket,
ecstore::error::StorageError::ObjectNotFound(_, _) => S3ErrorCode::NoSuchKey,
ecstore::error::StorageError::BucketExists(_) => S3ErrorCode::BucketAlreadyExists,
ecstore::error::StorageError::InvalidArgument(_, _, _) => S3ErrorCode::InvalidArgument,
ecstore::error::StorageError::MethodNotAllowed => S3ErrorCode::MethodNotAllowed,
ecstore::error::StorageError::StorageFull => S3ErrorCode::ServiceUnavailable,
_ => S3ErrorCode::InternalError,
};
ApiError {
code,
message: err.to_string(),
source: Some(Box::new(err)),
}
}
}
impl From<ApiError> for S3Error {
fn from(err: ApiError) -> Self {
let mut s3e = S3Error::with_message(err.code, err.message);
if let Some(source) = err.source {
s3e.set_source(source);
}
s3e
}
}
```
### 6. Error Handling Best Practices
#### Pattern Matching and Error Classification
```rust
// Use pattern matching for specific error handling
async fn handle_storage_operation() -> Result<()> {
match storage.get_object("bucket", "key").await {
Ok(object) => process_object(object),
Err(ecstore::error::StorageError::ObjectNotFound(bucket, key)) => {
warn!("Object not found: {}/{}", bucket, key);
create_default_object(bucket, key).await
}
Err(ecstore::error::StorageError::BucketNotFound(bucket)) => {
error!("Bucket not found: {}", bucket);
Err(MyError::Custom {
message: format!("Bucket {} does not exist", bucket)
})
}
Err(e) => {
error!("Storage operation failed: {}", e);
Err(MyError::Storage(e))
}
}
}
```
#### Error Aggregation and Reporting
```rust
// Collect and report multiple errors
pub fn validate_configuration(config: &Config) -> Result<()> {
let mut errors = Vec::new();
if config.bucket_name.is_empty() {
errors.push("Bucket name cannot be empty");
}
if config.region.is_empty() {
errors.push("Region must be specified");
}
if !errors.is_empty() {
return Err(MyError::Custom {
message: format!("Configuration validation failed: {}", errors.join(", "))
});
}
Ok(())
}
```
#### Contextual Error Information
```rust
// Add operation context to errors
#[tracing::instrument(skip(self))]
async fn upload_file(&self, bucket: &str, key: &str, data: Vec<u8>) -> Result<()> {
self.storage
.put_object(bucket, key, data)
.await
.map_err(|e| MyError::Custom {
message: format!("Failed to upload {}/{}: {}", bucket, key, e)
})
}
```
## Performance Optimization Guidelines
### 1. Memory Management
- Use `Bytes` instead of `Vec<u8>` for zero-copy operations
- Avoid unnecessary cloning, use reference passing
- Use `Arc` for sharing large objects
### 2. Concurrency Optimization
```rust
// Use join_all for concurrent operations
let futures = disks.iter().map(|disk| disk.operation());
let results = join_all(futures).await;
```
### 3. Caching Strategy
- Use `LazyLock` for global caching
- Implement LRU cache to avoid memory leaks
## Testing Guidelines
### 1. Unit Tests
```rust
#[cfg(test)]
mod tests {
use super::*;
use test_case::test_case;
#[tokio::test]
async fn test_async_function() {
let result = async_function().await;
assert!(result.is_ok());
}
#[test_case("input1", "expected1")]
#[test_case("input2", "expected2")]
fn test_with_cases(input: &str, expected: &str) {
assert_eq!(function(input), expected);
}
#[test]
fn test_error_conversion() {
use ecstore::error::StorageError;
let storage_err = StorageError::BucketNotFound("test-bucket".to_string());
let api_err: ApiError = storage_err.into();
assert_eq!(api_err.code, S3ErrorCode::NoSuchBucket);
assert!(api_err.message.contains("test-bucket"));
assert!(api_err.source.is_some());
}
#[test]
fn test_error_types() {
let io_err = std::io::Error::new(std::io::ErrorKind::NotFound, "file not found");
let my_err = MyError::Io(io_err);
// Test error matching
match my_err {
MyError::Io(_) => {}, // Expected
_ => panic!("Unexpected error type"),
}
}
#[test]
fn test_error_context() {
let result = process_with_context("nonexistent_file.txt");
assert!(result.is_err());
let err = result.unwrap_err();
match err {
MyError::Custom { message } => {
assert!(message.contains("Failed to read"));
assert!(message.contains("nonexistent_file.txt"));
}
_ => panic!("Expected Custom error"),
}
}
}
```
### 2. Integration Tests
- Use `e2e_test` module for end-to-end testing
- Simulate real storage environments
### 3. Test Quality Standards
- Write meaningful test cases that verify actual functionality
- Avoid placeholder or debug content like "debug 111", "test test", etc.
- Use descriptive test names that clearly indicate what is being tested
- Each test should have a clear purpose and verify specific behavior
- Test data should be realistic and representative of actual use cases
## Cross-Platform Compatibility Guidelines
### 1. CPU Architecture Compatibility
- **Always consider multi-platform and different CPU architecture compatibility** when writing code
- Support major architectures: x86_64, aarch64 (ARM64), and other target platforms
- Use conditional compilation for architecture-specific code:
```rust
#[cfg(target_arch = "x86_64")]
fn optimized_x86_64_function() { /* x86_64 specific implementation */ }
#[cfg(target_arch = "aarch64")]
fn optimized_aarch64_function() { /* ARM64 specific implementation */ }
#[cfg(not(any(target_arch = "x86_64", target_arch = "aarch64")))]
fn generic_function() { /* Generic fallback implementation */ }
```
### 2. Platform-Specific Dependencies
- Use feature flags for platform-specific dependencies
- Provide fallback implementations for unsupported platforms
- Test on multiple architectures in CI/CD pipeline
### 3. Endianness Considerations
- Use explicit byte order conversion when dealing with binary data
- Prefer `to_le_bytes()`, `from_le_bytes()` for consistent little-endian format
- Use `byteorder` crate for complex binary format handling
### 4. SIMD and Performance Optimizations
- Use portable SIMD libraries like `wide` or `packed_simd`
- Provide fallback implementations for non-SIMD architectures
- Use runtime feature detection when appropriate
## Security Guidelines
### 1. Memory Safety
- Disable `unsafe` code (workspace.lints.rust.unsafe_code = "deny")
- Use `rustls` instead of `openssl`
### 2. Authentication and Authorization
```rust
// Use IAM system for permission checks
let identity = iam.authenticate(&access_key, &secret_key).await?;
iam.authorize(&identity, &action, &resource).await?;
```
## Configuration Management Guidelines
### 1. Environment Variables
- Use `RUSTFS_` prefix
- Support both configuration files and environment variables
- Provide reasonable default values
### 2. Configuration Structure
```rust
#[derive(Debug, Deserialize, Clone)]
pub struct Config {
pub address: String,
pub volumes: String,
#[serde(default)]
pub console_enable: bool,
}
```
## Dependency Management Guidelines
### 1. Workspace Dependencies
- Manage versions uniformly at workspace level
- Use `workspace = true` to inherit configuration
### 2. Feature Flags
```rust
[features]
default = ["file"]
gpu = ["dep:nvml-wrapper"]
kafka = ["dep:rdkafka"]
```
## Deployment and Operations Guidelines
### 1. Containerization
- Provide Dockerfile and docker-compose configuration
- Support multi-stage builds to optimize image size
### 2. Observability
- Integrate OpenTelemetry for distributed tracing
- Support Prometheus metrics collection
- Provide Grafana dashboards
### 3. Health Checks
```rust
// Implement health check endpoint
async fn health_check() -> Result<HealthStatus> {
// Check component status
}
```
## Code Review Checklist
### 1. **Code Formatting and Quality (MANDATORY)**
- [ ] **Code is properly formatted** (`cargo fmt --all --check` passes)
- [ ] **All clippy warnings are resolved** (`cargo clippy --all-targets --all-features -- -D warnings` passes)
- [ ] **Code compiles successfully** (`cargo check --all-targets` passes)
- [ ] **Pre-commit hooks are working** and all checks pass
- [ ] **No formatting-related changes** mixed with functional changes (separate commits)
### 2. Functionality
- [ ] Are all error cases properly handled?
- [ ] Is there appropriate logging?
- [ ] Is there necessary test coverage?
### 3. Performance
- [ ] Are unnecessary memory allocations avoided?
- [ ] Are async operations used correctly?
- [ ] Are there potential deadlock risks?
### 4. Security
- [ ] Are input parameters properly validated?
- [ ] Are there appropriate permission checks?
- [ ] Is information leakage avoided?
### 5. Cross-Platform Compatibility
- [ ] Does the code work on different CPU architectures (x86_64, aarch64)?
- [ ] Are platform-specific features properly gated with conditional compilation?
- [ ] Is byte order handling correct for binary data?
- [ ] Are there appropriate fallback implementations for unsupported platforms?
### 6. Code Commits and Documentation
- [ ] Does it comply with [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)?
- [ ] Are commit messages concise and under 72 characters for the title line?
- [ ] Commit titles should be concise and in English, avoid Chinese
- [ ] Is PR description provided in copyable markdown format for easy copying?
## Common Patterns and Best Practices
### 1. Resource Management
```rust
// Use RAII pattern for resource management
pub struct ResourceGuard {
resource: Resource,
}
impl Drop for ResourceGuard {
fn drop(&mut self) {
// Clean up resources
}
}
```
### 2. Dependency Injection
```rust
// Use dependency injection pattern
pub struct Service {
config: Arc<Config>,
storage: Arc<dyn StorageAPI>,
}
```
### 3. Graceful Shutdown
```rust
// Implement graceful shutdown
async fn shutdown_gracefully(shutdown_rx: &mut Receiver<()>) {
tokio::select! {
_ = shutdown_rx.recv() => {
info!("Received shutdown signal");
// Perform cleanup operations
}
_ = tokio::time::sleep(SHUTDOWN_TIMEOUT) => {
warn!("Shutdown timeout reached");
}
}
}
```
## Domain-Specific Guidelines
### 1. Storage Operations
- All storage operations must support erasure coding
- Implement read/write quorum mechanisms
- Support data integrity verification
### 2. Network Communication
- Use gRPC for internal service communication
- HTTP/HTTPS support for S3-compatible API
- Implement connection pooling and retry mechanisms
### 3. Metadata Management
- Use FlatBuffers for serialization
- Support version control and migration
- Implement metadata caching
These rules should serve as guiding principles when developing the RustFS project, ensuring code quality, performance, and maintainability.
### 4. Code Operations
#### Branch Management
- **🚨 CRITICAL: NEVER modify code directly on main or master branch - THIS IS ABSOLUTELY FORBIDDEN 🚨**
- **⚠️ ANY DIRECT COMMITS TO MASTER/MAIN WILL BE REJECTED AND MUST BE REVERTED IMMEDIATELY ⚠️**
- **🔒 ALL CHANGES MUST GO THROUGH PULL REQUESTS - NO DIRECT COMMITS TO MAIN UNDER ANY CIRCUMSTANCES 🔒**
- **Always work on feature branches - NO EXCEPTIONS**
- Always check the .cursorrules file before starting to ensure you understand the project guidelines
- **MANDATORY workflow for ALL changes:**
1. `git checkout main` (switch to main branch)
2. `git pull` (get latest changes)
3. `git checkout -b feat/your-feature-name` (create and switch to feature branch)
4. Make your changes ONLY on the feature branch
5. Test thoroughly before committing
6. Commit and push to the feature branch
7. **Create a pull request for code review - THIS IS THE ONLY WAY TO MERGE TO MAIN**
8. **Wait for PR approval before merging - NEVER merge your own PRs without review**
- Use descriptive branch names following the pattern: `feat/feature-name`, `fix/issue-name`, `refactor/component-name`, etc.
- **Double-check current branch before ANY commit: `git branch` to ensure you're NOT on main/master**
- **Pull Request Requirements:**
- All changes must be submitted via PR regardless of size or urgency
- PRs must include comprehensive description and testing information
- PRs must pass all CI/CD checks before merging
- PRs require at least one approval from code reviewers
- Even hotfixes and emergency changes must go through PR process
- **Enforcement:**
- Main branch should be protected with branch protection rules
- Direct pushes to main should be blocked by repository settings
- Any accidental direct commits to main must be immediately reverted via PR
#### Development Workflow
## 🎯 **Core Development Principles**
- **🔴 Every change must be precise - don't modify unless you're confident**
- Carefully analyze code logic and ensure complete understanding before making changes
- When uncertain, prefer asking users or consulting documentation over blind modifications
- Use small iterative steps, modify only necessary parts at a time
- Evaluate impact scope before changes to ensure no new issues are introduced
- **🚀 GitHub PR creation prioritizes gh command usage**
- Prefer using `gh pr create` command to create Pull Requests
- Avoid having users manually create PRs through web interface
- Provide clear and professional PR titles and descriptions
- Using `gh` commands ensures better integration and automation
## 📝 **Code Quality Requirements**
- Use English for all code comments, documentation, and variable names
- Write meaningful and descriptive names for variables, functions, and methods
- Avoid meaningless test content like "debug 111" or placeholder values
- Before each change, carefully read the existing code to ensure you understand the code structure and implementation, do not break existing logic implementation, do not introduce new issues
- Ensure each change provides sufficient test cases to guarantee code correctness
- Do not arbitrarily modify numbers and constants in test cases, carefully analyze their meaning to ensure test case correctness
- When writing or modifying tests, check existing test cases to ensure they have scientific naming and rigorous logic testing, if not compliant, modify test cases to ensure scientific and rigorous testing
- **Before committing any changes, run `cargo clippy --all-targets --all-features -- -D warnings` to ensure all code passes Clippy checks**
- After each development completion, first git add . then git commit -m "feat: feature description" or "fix: issue description", ensure compliance with [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
- **Keep commit messages concise and under 72 characters** for the title line, use body for detailed explanations if needed
- After each development completion, first git push to remote repository
- After each change completion, summarize the changes, do not create summary files, provide a brief change description, ensure compliance with [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
- Provide change descriptions needed for PR in the conversation, ensure compliance with [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
- **Always provide PR descriptions in English** after completing any changes, including:
- Clear and concise title following Conventional Commits format
- Detailed description of what was changed and why
- List of key changes and improvements
- Any breaking changes or migration notes if applicable
- Testing information and verification steps
- **Provide PR descriptions in copyable markdown format** enclosed in code blocks for easy one-click copying
## 🚫 AI 文档生成限制
### 禁止生成总结文档
- **严格禁止创建任何形式的AI生成总结文档**
- **不得创建包含大量表情符号、详细格式化表格和典型AI风格的文档**
- **不得在项目中生成以下类型的文档:**
- 基准测试总结文档BENCHMARK*.md
- 实现对比分析文档IMPLEMENTATION_COMPARISON*.md
- 性能分析报告文档
- 架构总结文档
- 功能对比文档
- 任何带有大量表情符号和格式化内容的文档
- **如果需要文档,请只在用户明确要求时创建,并保持简洁实用的风格**
- **文档应当专注于实际需要的信息,避免过度格式化和装饰性内容**
- **任何发现的AI生成总结文档都应该立即删除**
### 允许的文档类型
- README.md项目介绍保持简洁
- 技术文档(仅在明确需要时创建)
- 用户手册(仅在明确需要时创建)
- API文档从代码生成
- 变更日志CHANGELOG.md

View File

@@ -1,27 +0,0 @@
FROM ubuntu:22.04
ENV LANG C.UTF-8
RUN sed -i s@http://.*archive.ubuntu.com@http://repo.huaweicloud.com@g /etc/apt/sources.list
RUN apt-get clean && apt-get update && apt-get install wget git curl unzip gcc pkg-config libssl-dev lld libdbus-1-dev libwayland-dev libwebkit2gtk-4.1-dev libxdo-dev -y
# install protoc
RUN wget https://github.com/protocolbuffers/protobuf/releases/download/v31.1/protoc-31.1-linux-x86_64.zip \
&& unzip protoc-31.1-linux-x86_64.zip -d protoc3 \
&& mv protoc3/bin/* /usr/local/bin/ && chmod +x /usr/local/bin/protoc \
&& mv protoc3/include/* /usr/local/include/ && rm -rf protoc-31.1-linux-x86_64.zip protoc3
# install flatc
RUN wget https://github.com/google/flatbuffers/releases/download/v25.2.10/Linux.flatc.binary.g++-13.zip \
&& unzip Linux.flatc.binary.g++-13.zip \
&& mv flatc /usr/local/bin/ && chmod +x /usr/local/bin/flatc && rm -rf Linux.flatc.binary.g++-13.zip
# install rust
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
COPY .docker/cargo.config.toml /root/.cargo/config.toml
WORKDIR /root/s3-rustfs
CMD [ "bash", "-c", "while true; do sleep 1; done" ]

View File

@@ -1,32 +0,0 @@
FROM rockylinux:9.3 AS builder
ENV LANG C.UTF-8
RUN sed -e 's|^mirrorlist=|#mirrorlist=|g' \
-e 's|^#baseurl=http://dl.rockylinux.org/$contentdir|baseurl=https://mirrors.ustc.edu.cn/rocky|g' \
-i.bak \
/etc/yum.repos.d/rocky-extras.repo \
/etc/yum.repos.d/rocky.repo
RUN dnf makecache
RUN yum install wget git unzip gcc openssl-devel pkgconf-pkg-config -y
# install protoc
RUN wget https://github.com/protocolbuffers/protobuf/releases/download/v31.1/protoc-31.1-linux-x86_64.zip \
&& unzip protoc-31.1-linux-x86_64.zip -d protoc3 \
&& mv protoc3/bin/* /usr/local/bin/ && chmod +x /usr/local/bin/protoc \
&& mv protoc3/include/* /usr/local/include/ && rm -rf protoc-31.1-linux-x86_64.zip protoc3
# install flatc
RUN wget https://github.com/google/flatbuffers/releases/download/v25.2.10/Linux.flatc.binary.g++-13.zip \
&& unzip Linux.flatc.binary.g++-13.zip \
&& mv flatc /usr/local/bin/ && chmod +x /usr/local/bin/flatc \
&& rm -rf Linux.flatc.binary.g++-13.zip
# install rust
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
COPY .docker/cargo.config.toml /root/.cargo/config.toml
WORKDIR /root/s3-rustfs

View File

@@ -1,25 +0,0 @@
FROM ubuntu:22.04
ENV LANG C.UTF-8
RUN sed -i s@http://.*archive.ubuntu.com@http://repo.huaweicloud.com@g /etc/apt/sources.list
RUN apt-get clean && apt-get update && apt-get install wget git curl unzip gcc pkg-config libssl-dev lld libdbus-1-dev libwayland-dev libwebkit2gtk-4.1-dev libxdo-dev -y
# install protoc
RUN wget https://github.com/protocolbuffers/protobuf/releases/download/v31.1/protoc-31.1-linux-x86_64.zip \
&& unzip protoc-31.1-linux-x86_64.zip -d protoc3 \
&& mv protoc3/bin/* /usr/local/bin/ && chmod +x /usr/local/bin/protoc \
&& mv protoc3/include/* /usr/local/include/ && rm -rf protoc-31.1-linux-x86_64.zip protoc3
# install flatc
RUN wget https://github.com/google/flatbuffers/releases/download/v25.2.10/Linux.flatc.binary.g++-13.zip \
&& unzip Linux.flatc.binary.g++-13.zip \
&& mv flatc /usr/local/bin/ && chmod +x /usr/local/bin/flatc && rm -rf Linux.flatc.binary.g++-13.zip
# install rust
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
COPY .docker/cargo.config.toml /root/.cargo/config.toml
WORKDIR /root/s3-rustfs

131
.docker/README.md Normal file
View File

@@ -0,0 +1,131 @@
# RustFS Docker Infrastructure
This directory contains the complete Docker infrastructure for building, deploying, and monitoring RustFS. It provides ready-to-use configurations for development, testing, and production-grade observability.
## 📂 Directory Structure
| Directory | Description | Status |
| :--- | :--- | :--- |
| **[`observability/`](observability/README.md)** | **[RECOMMENDED]** Full-stack observability (Prometheus, Grafana, Tempo, Loki). | ✅ Production-Ready |
| **[`compose/`](compose/README.md)** | Specialized setups (e.g., 4-node distributed cluster testing). | ⚠️ Testing Only |
| **[`mqtt/`](mqtt/README.md)** | EMQX Broker configuration for MQTT integration testing. | 🧪 Development |
| **[`openobserve-otel/`](openobserve-otel/README.md)** | Alternative lightweight observability stack using OpenObserve. | 🔄 Alternative |
---
## 📄 Root Directory Files
The following files in the project root are essential for Docker operations:
### Build Scripts & Dockerfiles
| File | Description | Usage |
| :--- | :--- | :--- |
| **`docker-buildx.sh`** | **Multi-Arch Build Script**<br>Automates building and pushing Docker images for `amd64` and `arm64`. Supports release and dev channels. | `./docker-buildx.sh --push` |
| **`Dockerfile`** | **Production Image (Alpine)**<br>Lightweight image using musl libc. Downloads pre-built binaries from GitHub Releases. | `docker build -t rustfs:latest .` |
| **`Dockerfile.glibc`** | **Production Image (Ubuntu)**<br>Standard image using glibc. Useful if you need specific dynamic libraries. | `docker build -f Dockerfile.glibc .` |
| **`Dockerfile.source`** | **Development Image**<br>Builds RustFS from source code. Includes build tools. Ideal for local development and CI. | `docker build -f Dockerfile.source .` |
### Docker Compose Configurations
| File | Description | Usage |
| :--- | :--- | :--- |
| **`docker-compose.yml`** | **Main Development Setup**<br>Comprehensive setup with profiles for development, observability, and proxying. | `docker compose up -d`<br>`docker compose --profile observability up -d` |
| **`docker-compose-simple.yml`** | **Quick Start Setup**<br>Minimal configuration running a single RustFS instance with 4 volumes. Perfect for first-time users. | `docker compose -f docker-compose-simple.yml up -d` |
---
## 🌟 Observability Stack (Recommended)
Located in: [`.docker/observability/`](observability/README.md)
We provide a comprehensive, industry-standard observability stack designed for deep insights into RustFS performance. This is the recommended setup for both development and production monitoring.
### Components
- **Metrics**: Prometheus (Collection) + Grafana (Visualization)
- **Traces**: Tempo (Storage) + Jaeger (UI)
- **Logs**: Loki
- **Ingestion**: OpenTelemetry Collector
### Key Features
- **Full Persistence**: All metrics, logs, and traces are saved to Docker volumes, ensuring no data loss on restarts.
- **Correlation**: Seamlessly jump between Logs, Traces, and Metrics in Grafana.
- **High Performance**: Optimized configurations for batching, compression, and memory management.
### Quick Start
```bash
cd .docker/observability
docker compose up -d
```
---
## 🧪 Specialized Environments
Located in: [`.docker/compose/`](compose/README.md)
These configurations are tailored for specific testing scenarios that require complex topologies.
### Distributed Cluster (4-Nodes)
Simulates a real-world distributed environment with 4 RustFS nodes running locally.
```bash
docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
```
### Integrated Observability Test
A self-contained environment running 4 RustFS nodes alongside the full observability stack. Useful for end-to-end testing of telemetry.
```bash
docker compose -f .docker/compose/docker-compose.observability.yaml up -d
```
---
## 📡 MQTT Integration
Located in: [`.docker/mqtt/`](mqtt/README.md)
Provides an EMQX broker for testing RustFS MQTT features.
### Quick Start
```bash
cd .docker/mqtt
docker compose up -d
```
- **Dashboard**: [http://localhost:18083](http://localhost:18083) (Default: `admin` / `public`)
- **MQTT Port**: `1883`
---
## 👁️ Alternative: OpenObserve
Located in: [`.docker/openobserve-otel/`](openobserve-otel/README.md)
For users preferring a lightweight, all-in-one solution, we support OpenObserve. It combines logs, metrics, and traces into a single binary and UI.
### Quick Start
```bash
cd .docker/openobserve-otel
docker compose up -d
```
---
## 🔧 Common Operations
### Cleaning Up
To stop all containers and remove volumes (**WARNING**: deletes all persisted data):
```bash
docker compose down -v
```
### Viewing Logs
To follow logs for a specific service:
```bash
docker compose logs -f [service_name]
```
### Checking Status
To see the status of all running containers:
```bash
docker compose ps
```

44
.docker/compose/README.md Normal file
View File

@@ -0,0 +1,44 @@
# Specialized Docker Compose Configurations
This directory contains specialized Docker Compose configurations for specific testing scenarios.
## ⚠️ Important Note
**For Observability:**
We **strongly recommend** using the new, fully integrated observability stack located in `../observability/`. It provides a production-ready setup with Prometheus, Grafana, Tempo, Loki, and OpenTelemetry Collector, all with persistent storage and optimized configurations.
The `docker-compose.observability.yaml` in this directory is kept for legacy reference or specific minimal testing needs but is **not** the primary recommended setup.
## 📁 Configuration Files
### Cluster Testing
- **`docker-compose.cluster.yaml`**
- **Purpose**: Simulates a 4-node RustFS distributed cluster.
- **Use Case**: Testing distributed storage logic, consensus, and failover.
- **Nodes**: 4 RustFS instances.
- **Storage**: Uses local HTTP endpoints.
### Legacy / Minimal Observability
- **`docker-compose.observability.yaml`**
- **Purpose**: A minimal observability setup.
- **Status**: **Deprecated**. Please use `../observability/docker-compose.yml` instead.
## 🚀 Usage Examples
### Cluster Testing
To start a 4-node cluster for distributed testing:
```bash
# From project root
docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
```
### (Deprecated) Minimal Observability
```bash
# From project root
docker compose -f .docker/compose/docker-compose.observability.yaml up -d
```

View File

@@ -14,70 +14,69 @@
services:
node0:
image: rustfs:v1 # 替换为你的镜像名称和标签
image: rustfs/rustfs:latest # Replace with your image name and label
container_name: node0
hostname: node0
environment:
- RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9002
- RUSTFS_ACCESS_KEY=rustfsadmin
- RUSTFS_SECRET_KEY=rustfsadmin
platform: linux/amd64
ports:
- "9000:9000" # 映射宿主机的 9001 端口到容器的 9000 端口
- "8000:9001" # 映射宿主机的 9001 端口到容器的 9000 端口
- "9000:9000" # Map port 9001 of the host to port 9000 of the container
volumes:
- ./target/x86_64-unknown-linux-musl/release/rustfs:/app/rustfs
# - ./data/node0:/data # 将当前路径挂载到容器内的 /root/data
- ../../target/x86_64-unknown-linux-gnu/release/rustfs:/app/rustfs
command: "/app/rustfs"
node1:
image: rustfs:v1
image: rustfs/rustfs:latest
container_name: node1
hostname: node1
environment:
- RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9002
- RUSTFS_ACCESS_KEY=rustfsadmin
- RUSTFS_SECRET_KEY=rustfsadmin
platform: linux/amd64
ports:
- "9001:9000" # 映射宿主机的 9002 端口到容器的 9000 端口
- "9001:9000" # Map port 9002 of the host to port 9000 of the container
volumes:
- ./target/x86_64-unknown-linux-musl/release/rustfs:/app/rustfs
# - ./data/node1:/data
- ../../target/x86_64-unknown-linux-gnu/release/rustfs:/app/rustfs
command: "/app/rustfs"
node2:
image: rustfs:v1
image: rustfs/rustfs:latest
container_name: node2
hostname: node2
environment:
- RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9002
- RUSTFS_ACCESS_KEY=rustfsadmin
- RUSTFS_SECRET_KEY=rustfsadmin
platform: linux/amd64
ports:
- "9002:9000" # 映射宿主机的 9003 端口到容器的 9000 端口
- "9002:9000" # Map port 9003 of the host to port 9000 of the container
volumes:
- ./target/x86_64-unknown-linux-musl/release/rustfs:/app/rustfs
# - ./data/node2:/data
- ../../target/x86_64-unknown-linux-gnu/release/rustfs:/app/rustfs
command: "/app/rustfs"
node3:
image: rustfs:v1
image: rustfs/rustfs:latest
container_name: node3
hostname: node3
environment:
- RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9002
- RUSTFS_ACCESS_KEY=rustfsadmin
- RUSTFS_SECRET_KEY=rustfsadmin
platform: linux/amd64
ports:
- "9003:9000" # 映射宿主机的 9004 端口到容器的 9000 端口
- "9003:9000" # Map port 9004 of the host to port 9000 of the container
volumes:
- ./target/x86_64-unknown-linux-musl/release/rustfs:/app/rustfs
# - ./data/node3:/data
- ../../target/x86_64-unknown-linux-gnu/release/rustfs:/app/rustfs
command: "/app/rustfs"

View File

@@ -0,0 +1,224 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
services:
# --- Observability Stack ---
tempo-init:
image: busybox:latest
command: [ "sh", "-c", "chown -R 10001:10001 /var/tempo" ]
volumes:
- tempo-data:/var/tempo
user: root
networks:
- rustfs-network
restart: "no"
tempo:
image: grafana/tempo:latest
user: "10001"
command: [ "-config.file=/etc/tempo.yaml" ]
volumes:
- ../../.docker/observability/tempo.yaml:/etc/tempo.yaml:ro
- tempo-data:/var/tempo
ports:
- "3200:3200" # tempo
- "4317" # otlp grpc
- "4318" # otlp http
restart: unless-stopped
networks:
- rustfs-network
otel-collector:
image: otel/opentelemetry-collector-contrib:latest
environment:
- TZ=Asia/Shanghai
volumes:
- ../../.docker/observability/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
ports:
- "1888:1888" # pprof
- "8888:8888" # Prometheus metrics for Collector
- "8889:8889" # Prometheus metrics for application indicators
- "13133:13133" # health check
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
- "55679:55679" # zpages
networks:
- rustfs-network
depends_on:
- tempo
- jaeger
- prometheus
- loki
jaeger:
image: jaegertracing/jaeger:latest
environment:
- TZ=Asia/Shanghai
- SPAN_STORAGE_TYPE=badger
- BADGER_EPHEMERAL=false
- BADGER_DIRECTORY_VALUE=/badger/data
- BADGER_DIRECTORY_KEY=/badger/key
- COLLECTOR_OTLP_ENABLED=true
volumes:
- jaeger-data:/badger
ports:
- "16686:16686" # Web UI
- "14269:14269" # Admin/Metrics
networks:
- rustfs-network
prometheus:
image: prom/prometheus:latest
environment:
- TZ=Asia/Shanghai
volumes:
- ../../.docker/observability/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus-data:/prometheus
ports:
- "9090:9090"
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--web.enable-otlp-receiver'
- '--web.enable-remote-write-receiver'
- '--enable-feature=promql-experimental-functions'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/usr/share/prometheus/console_libraries'
- '--web.console.templates=/usr/share/prometheus/consoles'
networks:
- rustfs-network
loki:
image: grafana/loki:latest
environment:
- TZ=Asia/Shanghai
volumes:
- ../../.docker/observability/loki.yaml:/etc/loki/local-config.yaml:ro
- loki-data:/loki
ports:
- "3100:3100"
command: -config.file=/etc/loki/local-config.yaml
networks:
- rustfs-network
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000" # Web UI
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
- GF_SECURITY_ADMIN_USER=admin
- TZ=Asia/Shanghai
- GF_INSTALL_PLUGINS=grafana-pyroscope-datasource
- GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/home.json
networks:
- rustfs-network
volumes:
- ../../.docker/observability/grafana/provisioning:/etc/grafana/provisioning:ro
- ../../.docker/observability/grafana/dashboards:/var/lib/grafana/dashboards:ro
depends_on:
- prometheus
- tempo
- loki
# --- RustFS Cluster ---
node1:
build:
context: ../..
dockerfile: Dockerfile.source
container_name: node1
environment:
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
- RUSTFS_ADDRESS=:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
- RUSTFS_OBS_LOGGER_LEVEL=debug
platform: linux/amd64
ports:
- "9001:9000"
networks:
- rustfs-network
depends_on:
- otel-collector
node2:
build:
context: ../..
dockerfile: Dockerfile.source
container_name: node2
environment:
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
- RUSTFS_ADDRESS=:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
- RUSTFS_OBS_LOGGER_LEVEL=debug
platform: linux/amd64
ports:
- "9002:9000"
networks:
- rustfs-network
depends_on:
- otel-collector
node3:
build:
context: ../..
dockerfile: Dockerfile.source
container_name: node3
environment:
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
- RUSTFS_ADDRESS=:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
- RUSTFS_OBS_LOGGER_LEVEL=debug
platform: linux/amd64
ports:
- "9003:9000"
networks:
- rustfs-network
depends_on:
- otel-collector
node4:
build:
context: ../..
dockerfile: Dockerfile.source
container_name: node4
environment:
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
- RUSTFS_ADDRESS=:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
- RUSTFS_OBS_LOGGER_LEVEL=debug
platform: linux/amd64
ports:
- "9004:9000"
networks:
- rustfs-network
depends_on:
- otel-collector
volumes:
prometheus-data:
tempo-data:
loki-data:
jaeger-data:
networks:
rustfs-network:
driver: bridge
name: "network_rustfs_config"
driver_opts:
com.docker.network.enable_ipv6: "true"

30
.docker/mqtt/README.md Normal file
View File

@@ -0,0 +1,30 @@
# MQTT Broker (EMQX)
This directory contains the configuration for running an EMQX MQTT broker, which can be used for testing RustFS's MQTT integration.
## 🚀 Quick Start
To start the EMQX broker:
```bash
docker compose up -d
```
## 📊 Access
- **Dashboard**: [http://localhost:18083](http://localhost:18083)
- **Default Credentials**: `admin` / `public`
- **MQTT Port**: `1883`
- **WebSocket Port**: `8083`
## 🛠️ Configuration
The `docker-compose.yml` file sets up a single-node EMQX instance.
- **Persistence**: Data is not persisted by default (for testing).
- **Network**: Uses the default bridge network.
## 📝 Notes
- This setup is intended for development and testing purposes.
- For production deployments, please refer to the official [EMQX Documentation](https://www.emqx.io/docs/en/latest/).

82
.docker/nginx/nginx.conf Normal file
View File

@@ -0,0 +1,82 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
worker_processes auto;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log warn;
sendfile on;
keepalive_timeout 65;
# RustFS Server Block
server {
listen 80;
server_name localhost;
# Redirect HTTP to HTTPS (optional, uncomment if SSL is configured)
# return 301 https://$host$request_uri;
location / {
proxy_pass http://rustfs:9000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# S3 specific headers
proxy_set_header X-Amz-Date $http_x_amz_date;
proxy_set_header Authorization $http_authorization;
# Disable buffering for large uploads
proxy_request_buffering off;
client_max_body_size 0;
}
location /rustfs/console {
proxy_pass http://rustfs:9001;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
# SSL Configuration (Example)
# server {
# listen 443 ssl;
# server_name localhost;
#
# ssl_certificate /etc/nginx/ssl/server.crt;
# ssl_certificate_key /etc/nginx/ssl/server.key;
#
# location / {
# proxy_pass http://rustfs:9000;
# ...
# }
# }
}

0
.docker/nginx/ssl/.keep Normal file
View File

5
.docker/observability/.gitignore vendored Normal file
View File

@@ -0,0 +1,5 @@
jaeger-data/*
loki-data/*
prometheus-data/*
tempo-data/*
grafana-data/*

View File

@@ -1,109 +1,85 @@
# Observability
# RustFS Observability Stack
This directory contains the observability stack for the application. The stack is composed of the following components:
This directory contains the comprehensive observability stack for RustFS, designed to provide deep insights into application performance, logs, and traces.
- Prometheus v3.2.1
- Grafana 11.6.0
- Loki 3.4.2
- Jaeger 2.4.0
- Otel Collector 0.120.0 # 0.121.0 remove loki
## Components
## Prometheus
The stack is composed of the following best-in-class open-source components:
Prometheus is a monitoring and alerting toolkit. It scrapes metrics from instrumented jobs, either directly or via an
intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to
either aggregate and record new time series from existing data or generate alerts. Grafana or other API consumers can be
used to visualize the collected data.
- **Prometheus** (v2.53.1): The industry standard for metric collection and alerting.
- **Grafana** (v11.1.0): The leading platform for observability visualization.
- **Loki** (v3.1.0): A horizontally-scalable, highly-available, multi-tenant log aggregation system.
- **Tempo** (v2.5.0): A high-volume, minimal dependency distributed tracing backend.
- **Jaeger** (v1.59.0): Distributed tracing system (configured as a secondary UI/storage).
- **OpenTelemetry Collector** (v0.104.0): A vendor-agnostic implementation for receiving, processing, and exporting telemetry data.
## Grafana
## Architecture
Grafana is a multi-platform open-source analytics and interactive visualization web application. It provides charts,
graphs, and alerts for the web when connected to supported data sources.
1. **Telemetry Collection**: Applications send OTLP (OpenTelemetry Protocol) data (Metrics, Logs, Traces) to the **OpenTelemetry Collector**.
2. **Processing & Exporting**: The Collector processes the data (batching, memory limiting) and exports it to the respective backends:
- **Traces** -> **Tempo** (Primary) & **Jaeger** (Secondary/Optional)
- **Metrics** -> **Prometheus** (via scraping the Collector's exporter)
- **Logs** -> **Loki**
3. **Visualization**: **Grafana** connects to all backends (Prometheus, Tempo, Loki, Jaeger) to provide a unified dashboard experience.
## Loki
## Features
Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is
designed to be very cost-effective and easy to operate. It does not index the contents of the logs, but rather a set of
labels for each log stream.
- **Full Persistence**: All data (Metrics, Logs, Traces) is persisted to Docker volumes, ensuring no data loss on restart.
- **Correlation**: Seamless navigation between Metrics, Logs, and Traces in Grafana.
- Jump from a Metric spike to relevant Traces.
- Jump from a Trace to relevant Logs.
- **High Performance**: Optimized configurations for batching, compression, and memory management.
- **Standardized Protocols**: Built entirely on OpenTelemetry standards.
## Jaeger
## Quick Start
Jaeger is a distributed tracing system released as open source by Uber Technologies. It is used for monitoring and
troubleshooting microservices-based distributed systems, including:
### Prerequisites
- Distributed context propagation
- Distributed transaction monitoring
- Root cause analysis
- Service dependency analysis
- Performance / latency optimization
- Docker
- Docker Compose
## Otel Collector
### Deploy
The OpenTelemetry Collector offers a vendor-agnostic implementation on how to receive, process, and export telemetry
data. It removes the need to run, operate, and maintain multiple agents/collectors in order to support open-source
observability data formats (e.g. Jaeger, Prometheus, etc.) sending to one or more open-source or commercial back-ends.
## How to use
To deploy the observability stack, run the following command:
- docker latest version
Run the following command to start the entire stack:
```bash
docker compose -f docker-compose.yml -f docker-compose.override.yml up -d
docker compose up -d
```
- docker compose v2.0.0 or before
### Access Dashboards
| Service | URL | Credentials | Description |
| :------------- | :----------------------------------------------- | :---------------- | :----------------------------- |
| **Grafana** | [http://localhost:3000](http://localhost:3000) | `admin` / `admin` | Main visualization hub. |
| **Prometheus** | [http://localhost:9090](http://localhost:9090) | - | Metric queries and status. |
| **Jaeger UI** | [http://localhost:16686](http://localhost:16686) | - | Secondary trace visualization. |
| **Tempo** | [http://localhost:3200](http://localhost:3200) | - | Tempo status/metrics. |
## Configuration
### Data Persistence
Data is stored in the following Docker volumes:
- `prometheus-data`: Prometheus metrics
- `tempo-data`: Tempo traces (WAL and Blocks)
- `loki-data`: Loki logs (Chunks and Rules)
- `jaeger-data`: Jaeger traces (Badger DB)
To clear all data:
```bash
docke-compose -f docker-compose.yml -f docker-compose.override.yml up -d
docker compose down -v
```
To access the Grafana dashboard, navigate to `http://localhost:3000` in your browser. The default username and password
are `admin` and `admin`, respectively.
To access the Jaeger dashboard, navigate to `http://localhost:16686` in your browser.
To access the Prometheus dashboard, navigate to `http://localhost:9090` in your browser.
## How to stop
To stop the observability stack, run the following command:
```bash
docker compose -f docker-compose.yml -f docker-compose.override.yml down
```
## How to remove data
To remove the data generated by the observability stack, run the following command:
```bash
docker compose -f docker-compose.yml -f docker-compose.override.yml down -v
```
## How to configure
To configure the observability stack, modify the `docker-compose.override.yml` file. The file contains the following
```yaml
services:
prometheus:
environment:
- PROMETHEUS_CONFIG_FILE=/etc/prometheus/prometheus.yml
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
grafana:
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
volumes:
- ./grafana/provisioning:/etc/grafana/provisioning
```
The `prometheus` service mounts the `prometheus.yml` file to `/etc/prometheus/prometheus.yml`. The `grafana` service
mounts the `grafana/provisioning` directory to `/etc/grafana/provisioning`. You can modify these files to configure the
observability stack.
### Customization
- **Prometheus**: Edit `prometheus.yml` to add scrape targets or alerting rules.
- **Grafana**: Dashboards and datasources are provisioned from the `grafana/` directory.
- **Collector**: Edit `otel-collector-config.yaml` to modify pipelines, processors, or exporters.
## Troubleshooting
- **Service Health**: Check the health of services using `docker compose ps`.
- **Logs**: View logs for a specific service using `docker compose logs -f <service_name>`.
- **Otel Collector**: Check `http://localhost:13133` for health status and `http://localhost:1888/debug/pprof/` for profiling.

View File

@@ -1,27 +1,85 @@
## 部署可观测性系统
# RustFS 可观测性技术栈
OpenTelemetry Collector 提供了一个厂商中立的遥测数据处理方案,用于接收、处理和导出遥测数据。它消除了为支持多种开源可观测性数据格式(如
Jaeger、Prometheus 等)而需要运行和维护多个代理/收集器的必要性。
本目录包含 RustFS 的全面可观测性技术栈,旨在提供对应用程序性能、日志和追踪的深入洞察。
### 快速部署
## 组件
1. 进入 `.docker/observability` 目录
2. 执行以下命令启动服务:
该技术栈由以下一流的开源组件组成:
- **Prometheus** (v2.53.1): 行业标准的指标收集和告警工具。
- **Grafana** (v11.1.0): 领先的可观测性可视化平台。
- **Loki** (v3.1.0): 水平可扩展、高可用、多租户的日志聚合系统。
- **Tempo** (v2.5.0): 高吞吐量、最小依赖的分布式追踪后端。
- **Jaeger** (v1.59.0): 分布式追踪系统(配置为辅助 UI/存储)。
- **OpenTelemetry Collector** (v0.104.0): 接收、处理和导出遥测数据的供应商无关实现。
## 架构
1. **遥测收集**: 应用程序将 OTLP (OpenTelemetry Protocol) 数据(指标、日志、追踪)发送到 **OpenTelemetry Collector**
2. **处理与导出**: Collector 处理数据(批处理、内存限制)并将其导出到相应的后端:
- **追踪** -> **Tempo** (主要) & **Jaeger** (辅助/可选)
- **指标** -> **Prometheus** (通过抓取 Collector 的导出器)
- **日志** -> **Loki**
3. **可视化**: **Grafana** 连接到所有后端Prometheus, Tempo, Loki, Jaeger提供统一的仪表盘体验。
## 特性
- **完全持久化**: 所有数据(指标、日志、追踪)都持久化到 Docker 卷,确保重启后无数据丢失。
- **关联性**: 在 Grafana 中实现指标、日志和追踪之间的无缝导航。
- 从指标峰值跳转到相关追踪。
- 从追踪跳转到相关日志。
- **高性能**: 针对批处理、压缩和内存管理进行了优化配置。
- **标准化协议**: 完全基于 OpenTelemetry 标准构建。
## 快速开始
### 前置条件
- Docker
- Docker Compose
### 部署
运行以下命令启动整个技术栈:
```bash
docker compose -f docker-compose.yml up -d
docker compose up -d
```
### 访问监控面板
### 访问仪表盘
服务启动后,可通过以下地址访问各个监控面板:
| 服务 | URL | 凭据 | 描述 |
| :--- | :--- | :--- | :--- |
| **Grafana** | [http://localhost:3000](http://localhost:3000) | `admin` / `admin` | 主要可视化中心。 |
| **Prometheus** | [http://localhost:9090](http://localhost:9090) | - | 指标查询和状态。 |
| **Jaeger UI** | [http://localhost:16686](http://localhost:16686) | - | 辅助追踪可视化。 |
| **Tempo** | [http://localhost:3200](http://localhost:3200) | - | Tempo 状态/指标。 |
- Grafana: `http://localhost:3000` (默认账号/密码:`admin`/`admin`)
- Jaeger: `http://localhost:16686`
- Prometheus: `http://localhost:9090`
## 配置
## 配置可观测性
### 数据持久化
```shell
export RUSTFS_OBS_ENDPOINT="http://localhost:4317" # OpenTelemetry Collector 地址
数据存储在以下 Docker 卷中:
- `prometheus-data`: Prometheus 指标
- `tempo-data`: Tempo 追踪 (WAL 和 Blocks)
- `loki-data`: Loki 日志 (Chunks 和 Rules)
- `jaeger-data`: Jaeger 追踪 (Badger DB)
要清除所有数据:
```bash
docker compose down -v
```
### 自定义
- **Prometheus**: 编辑 `prometheus.yml` 以添加抓取目标或告警规则。
- **Grafana**: 仪表盘和数据源从 `grafana/` 目录预置。
- **Collector**: 编辑 `otel-collector-config.yaml` 以修改管道、处理器或导出器。
## 故障排除
- **服务健康**: 使用 `docker compose ps` 检查服务健康状况。
- **日志**: 使用 `docker compose logs -f <service_name>` 查看特定服务的日志。
- **Otel Collector**: 检查 `http://localhost:13133` 获取健康状态,检查 `http://localhost:1888/debug/pprof/` 进行性能分析。

View File

@@ -0,0 +1,270 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
services:
rustfs:
security_opt:
- "no-new-privileges:true"
image: rustfs/rustfs:latest
container_name: rustfs-server
ports:
- "9000:9000" # S3 API port
- "9001:9001" # Console port
environment:
- RUSTFS_VOLUMES=/data/rustfs
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_CORS_ALLOWED_ORIGINS=*
- RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
- RUSTFS_ACCESS_KEY=rustfsadmin
- RUSTFS_SECRET_KEY=rustfsadmin
- RUSTFS_OBS_LOGGER_LEVEL=info
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
- RUSTFS_OBS_PROFILING_ENDPOINT=http://pyroscope:4040
volumes:
- rustfs-data:/data/rustfs
networks:
- otel-network
restart: unless-stopped
healthcheck:
test:
[
"CMD",
"sh",
"-c",
"curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health",
]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
depends_on:
otel-collector:
condition: service_started
rustfs-init:
image: alpine
container_name: rustfs-init
volumes:
- rustfs-data:/data
networks:
- otel-network
command: >
sh -c "
chown -R 10001:10001 /data &&
echo 'Volume Permissions fixed' &&
exit 0
"
restart: no
# --- Tracing ---
tempo:
image: grafana/tempo:latest
container_name: tempo
command: [ "-config.file=/etc/tempo.yaml" ]
volumes:
- ./tempo.yaml:/etc/tempo.yaml:ro
- tempo-data:/var/tempo
ports:
- "3200:3200" # tempo
- "4317" # otlp grpc
- "4318" # otlp http
networks:
- otel-network
restart: unless-stopped
depends_on:
- redpanda
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
interval: 10s
timeout: 5s
retries: 3
start_period: 15s
redpanda:
image: redpandadata/redpanda:latest # for tempo ingest
container_name: redpanda
ports:
- "9092:9092"
networks:
- otel-network
restart: unless-stopped
command: >
redpanda start --overprovisioned
--mode=dev-container
--kafka-addr=PLAINTEXT://0.0.0.0:9092
--advertise-kafka-addr=PLAINTEXT://redpanda:9092
jaeger:
image: jaegertracing/jaeger:latest
container_name: jaeger
environment:
- SPAN_STORAGE_TYPE=badger
- BADGER_EPHEMERAL=false
- BADGER_DIRECTORY_VALUE=/badger/data
- BADGER_DIRECTORY_KEY=/badger/key
- COLLECTOR_OTLP_ENABLED=true
volumes:
- ./jaeger.yaml:/etc/jaeger/config.yml
- jaeger-data:/badger
ports:
- "16686:16686" # Web UI
- "14269:14269" # Admin/Metrics
- "4317" # otlp grpc
- "4318" # otlp http
command: [ "--config", "/etc/jaeger/config.yml" ]
networks:
- otel-network
restart: unless-stopped
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
interval: 10s
timeout: 5s
retries: 3
start_period: 15s
# --- Metrics ---
prometheus:
image: prom/prometheus:latest
container_name: prometheus
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus-data:/prometheus
ports:
- "9090:9090"
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--web.enable-otlp-receiver" # Enable OTLP
- "--web.enable-remote-write-receiver" # Enable remote write
- "--enable-feature=promql-experimental-functions" # Enable info()
- "--storage.tsdb.retention.time=30d"
restart: unless-stopped
networks:
- otel-network
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
interval: 10s
timeout: 5s
retries: 3
# --- Logging ---
loki:
image: grafana/loki:latest
container_name: loki
volumes:
- ./loki.yaml:/etc/loki/loki.yaml:ro
- loki-data:/loki
ports:
- "3100:3100"
command: -config.file=/etc/loki/loki.yaml
networks:
- otel-network
restart: unless-stopped
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
interval: 15s
timeout: 10s
retries: 5
start_period: 60s
# --- Collection ---
otel-collector:
image: otel/opentelemetry-collector-contrib:latest
volumes:
- ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
ports:
- "1888:1888" # pprof
- "8888:8888" # Prometheus metrics for Collector
- "8889:8889" # Prometheus metrics for application indicators
- "13133:13133" # health check
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
- "55679:55679" # zpages
networks:
- otel-network
restart: unless-stopped
depends_on:
- tempo
- jaeger
- prometheus
- loki
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:13133" ]
interval: 10s
timeout: 5s
retries: 3
# --- Profiles ---
pyroscope:
image: grafana/pyroscope:latest
container_name: pyroscope
ports:
- "4040:4040"
command:
- -self-profiling.disable-push=true
networks:
- otel-network
restart: unless-stopped
# --- Visualization ---
grafana:
image: grafana/grafana:latest
container_name: grafana
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
- GF_SECURITY_ADMIN_USER=admin
volumes:
- ./grafana/provisioning:/etc/grafana/provisioning:ro
- ./grafana/dashboards:/etc/grafana/dashboards:ro
- grafana-data:/var/lib/grafana
networks:
- otel-network
restart: unless-stopped
depends_on:
- prometheus
- tempo
- loki
healthcheck:
test:
[ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
interval: 10s
timeout: 5s
retries: 3
volumes:
rustfs-data:
tempo-data:
jaeger-data:
prometheus-data:
loki-data:
grafana-data:
networks:
otel-network:
driver: bridge
name: "network_otel"
ipam:
config:
- subnet: 172.28.0.0/16
driver_opts:
com.docker.network.enable_ipv6: "true"

View File

@@ -0,0 +1,62 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Docker Compose override file for High Availability Tempo setup
#
# Usage:
# docker-compose -f docker-compose-example-for-rustfs.yml \
# -f docker-compose-tempo-ha-override.yml up
services:
# Override Tempo to use high-availability configuration
tempo:
volumes:
- ./tempo-ha.yaml:/etc/tempo.yaml:ro
- tempo-data:/var/tempo
ports:
- "3200:3200" # Tempo HTTP
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
- "7946:7946" # Memberlist
- "14250:14250" # Jaeger gRPC
- "14268:14268" # Jaeger Thrift HTTP
- "9411:9411" # Zipkin
environment:
- TEMPO_MEMBERLIST_BIND_PORT=7946
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
interval: 10s
timeout: 5s
retries: 5
start_period: 30s
depends_on:
- redpanda
volumes:
tempo-data:
driver: local
driver_opts:
type: tmpfs
device: tmpfs
o: "size=4g" # Allocate 4GB tmpfs for Tempo data (adjust based on your needs)
# Network configuration remains the same
# networks:
# otel-network:
# driver: bridge
# name: "network_otel"
# ipam:
# config:
# - subnet: 172.28.0.0/16

View File

@@ -13,67 +13,187 @@
# limitations under the License.
services:
otel-collector:
image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.127.0
environment:
- TZ=Asia/Shanghai
# --- Tracing ---
tempo:
image: grafana/tempo:latest
container_name: tempo
command: [ "-config.file=/etc/tempo.yaml" ]
volumes:
- ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
- ./tempo.yaml:/etc/tempo.yaml:ro
- tempo-data:/var/tempo
ports:
- 1888:1888
- 8888:8888
- 8889:8889
- 13133:13133
- 4317:4317
- 4318:4318
- 55679:55679
- "3200:3200" # tempo
- "4317" # otlp grpc
- "4318" # otlp http
- "7946" # memberlist
networks:
- otel-network
restart: unless-stopped
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
interval: 10s
timeout: 5s
retries: 3
start_period: 15s
jaeger:
image: jaegertracing/jaeger:2.7.0
image: jaegertracing/jaeger:latest
container_name: jaeger
environment:
- TZ=Asia/Shanghai
- SPAN_STORAGE_TYPE=badger
- BADGER_EPHEMERAL=false
- BADGER_DIRECTORY_VALUE=/badger/data
- BADGER_DIRECTORY_KEY=/badger/key
- COLLECTOR_OTLP_ENABLED=true
volumes:
- ./jaeger.yaml:/etc/jaeger/config.yml
- jaeger-data:/badger
ports:
- "16686:16686"
- "14317:4317"
- "14318:4318"
- "16686:16686" # Web UI
- "14269:14269" # Admin/Metrics
- "4317" # otlp grpc
- "4318" # otlp http
command: [ "--config", "/etc/jaeger/config.yml" ]
networks:
- otel-network
restart: unless-stopped
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
interval: 10s
timeout: 5s
retries: 3
start_period: 15s
# --- Metrics ---
prometheus:
image: prom/prometheus:v3.4.1
environment:
- TZ=Asia/Shanghai
image: prom/prometheus:latest
container_name: prometheus
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus-data:/prometheus
ports:
- "9090:9090"
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--web.enable-otlp-receiver" # Enable OTLP
- "--web.enable-remote-write-receiver" # Enable remote write
- "--enable-feature=promql-experimental-functions" # Enable info()
- "--storage.tsdb.retention.time=30d"
restart: unless-stopped
networks:
- otel-network
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
interval: 10s
timeout: 5s
retries: 3
# --- Logging ---
loki:
image: grafana/loki:3.5.1
environment:
- TZ=Asia/Shanghai
image: grafana/loki:latest
container_name: loki
volumes:
- ./loki-config.yaml:/etc/loki/local-config.yaml
- ./loki.yaml:/etc/loki/loki.yaml:ro
- loki-data:/loki
ports:
- "3100:3100"
command: -config.file=/etc/loki/local-config.yaml
command: -config.file=/etc/loki/loki.yaml
networks:
- otel-network
grafana:
image: grafana/grafana:12.0.2
restart: unless-stopped
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
interval: 15s
timeout: 10s
retries: 5
start_period: 60s
# --- Collection ---
otel-collector:
image: otel/opentelemetry-collector-contrib:latest
volumes:
- ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
ports:
- "3000:3000" # Web UI
- "1888:1888" # pprof
- "8888:8888" # Prometheus metrics for Collector
- "8889:8889" # Prometheus metrics for application indicators
- "13133:13133" # health check
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
- "55679:55679" # zpages
networks:
- otel-network
restart: unless-stopped
depends_on:
- tempo
- jaeger
- prometheus
- loki
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:13133" ]
interval: 10s
timeout: 5s
retries: 3
# --- Profiles ---
pyroscope:
image: grafana/pyroscope:latest
container_name: pyroscope
ports:
- "4040:4040"
command:
- -self-profiling.disable-push=true
networks:
- otel-network
restart: unless-stopped
# --- Visualization ---
grafana:
image: grafana/grafana:latest
container_name: grafana
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
- TZ=Asia/Shanghai
- GF_SECURITY_ADMIN_USER=admin
volumes:
- ./grafana/provisioning:/etc/grafana/provisioning:ro
- ./grafana/dashboards:/etc/grafana/dashboards:ro
- grafana-data:/var/lib/grafana
networks:
- otel-network
restart: unless-stopped
depends_on:
- prometheus
- tempo
- loki
healthcheck:
test:
[ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
interval: 10s
timeout: 5s
retries: 3
volumes:
tempo-data:
jaeger-data:
prometheus-data:
loki-data:
grafana-data:
networks:
otel-network:
driver: bridge
name: "network_otel_config"
name: "network_otel"
ipam:
config:
- subnet: 172.28.0.0/16
driver_opts:
com.docker.network.enable_ipv6: "true"
com.docker.network.enable_ipv6: "true"

File diff suppressed because it is too large Load Diff

View File

@@ -12,8 +12,14 @@
# See the License for the specific language governing permissions and
# limitations under the License.
[source.crates-io]
registry = "https://github.com/rust-lang/crates.io-index"
apiVersion: 1
[net]
git-fetch-with-cli = true
providers:
- name: "default"
orgId: 1
folder: ""
type: file
disableDeletion: false
updateIntervalSeconds: 10
options:
path: /etc/grafana/dashboards

View File

@@ -0,0 +1,97 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
uid: prometheus
access: proxy
orgId: 1
url: http://prometheus:9090
isDefault: true
version: 1
editable: false
jsonData:
httpMethod: GET
exemplarTraceIdDestinations:
- name: trace_id
datasourceUid: tempo
- name: Tempo
type: tempo
uid: tempo
access: proxy
orgId: 1
url: http://tempo:3200
isDefault: false
version: 1
editable: false
jsonData:
httpMethod: GET
serviceMap:
datasourceUid: prometheus
tracesToLogs:
datasourceUid: loki
tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
mappedTags: [ { key: 'service.name', value: 'app' } ]
spanStartTimeShift: '1s'
spanEndTimeShift: '-1s'
filterByTraceID: true
filterBySpanID: false
tracesToMetrics:
datasourceUid: prometheus
tags: [ { key: 'service.name' }, { key: 'job' } ]
queries:
- name: 'Service-Level Latency'
query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m])) by (le)'
- name: 'Service-Level Calls'
query: 'sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))'
- name: 'Service-Level Errors'
query: 'sum(rate(traces_spanmetrics_calls_total{status_code="ERROR", $$__tags}[5m]))'
nodeGraph:
enabled: true
- name: Loki
type: loki
uid: loki
orgId: 1
url: http://loki:3100
isDefault: false
version: 1
editable: false
jsonData:
derivedFields:
- datasourceUid: tempo
matcherRegex: 'trace_id=(\w+)'
name: 'TraceID'
url: '$${__value.raw}'
- name: Jaeger
type: jaeger
uid: jaeger
url: http://jaeger:16686
access: proxy
isDefault: false
editable: false
jsonData:
tracesToLogs:
datasourceUid: loki
tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
mappedTags: [ { key: 'service.name', value: 'app' } ]
spanStartTimeShift: '1s'
spanEndTimeShift: '-1s'
filterByTraceID: true
filterBySpanID: false

View File

@@ -0,0 +1,98 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
uid: prometheus
url: http://prometheus:9090
access: proxy
isDefault: true
editable: false
jsonData:
httpMethod: GET
exemplarTraceIdDestinations:
- name: trace_id
datasourceUid: tempo
- name: Tempo
type: tempo
uid: tempo
access: proxy
url: http://tempo:3200
isDefault: false
editable: false
jsonData:
httpMethod: GET
serviceMap:
datasourceUid: prometheus
tracesToLogs:
datasourceUid: loki
tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
mappedTags: [ { key: 'service.name', value: 'app' } ]
spanStartTimeShift: '-1h'
spanEndTimeShift: '1h'
filterByTraceID: true
filterBySpanID: false
tracesToMetrics:
datasourceUid: prometheus
tags: [ { key: 'service.name' }, { key: 'job' } ]
queries:
- name: 'Service-Level Latency'
query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m])) by (le)'
- name: 'Service-Level Calls'
query: 'sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))'
- name: 'Service-Level Errors'
query: 'sum(rate(traces_spanmetrics_calls_total{status_code="ERROR", $$__tags}[5m]))'
nodeGraph:
enabled: true
- name: Loki
type: loki
uid: loki
url: http://loki:3100
basicAuth: false
isDefault: false
editable: false
jsonData:
derivedFields:
- datasourceUid: tempo
matcherRegex: 'trace_id=(\w+)'
name: 'TraceID'
url: '$${__value.raw}'
- name: Jaeger
type: jaeger
uid: jaeger
url: http://jaeger:16686
access: proxy
isDefault: false
editable: false
jsonData:
tracesToLogs:
datasourceUid: loki
tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
mappedTags: [ { key: 'service.name', value: 'app' } ]
spanStartTimeShift: '1s'
spanEndTimeShift: '-1s'
filterByTraceID: true
filterBySpanID: false
- name: Pyroscope
type: grafana-pyroscope-datasource
url: http://pyroscope:4040
jsonData:
minStep: '15s'

View File

@@ -1,112 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
service:
extensions: [ jaeger_storage, jaeger_query, remote_sampling, healthcheckv2 ]
pipelines:
traces:
receivers: [ otlp, jaeger, zipkin ]
processors: [ batch, adaptive_sampling ]
exporters: [ jaeger_storage_exporter ]
telemetry:
resource:
service.name: jaeger
metrics:
level: detailed
readers:
- pull:
exporter:
prometheus:
host: 0.0.0.0
port: 8888
logs:
level: debug
# TODO Initialize telemetry tracer once OTEL released new feature.
# https://github.com/open-telemetry/opentelemetry-collector/issues/10663
extensions:
healthcheckv2:
use_v2: true
http:
# pprof:
# endpoint: 0.0.0.0:1777
# zpages:
# endpoint: 0.0.0.0:55679
jaeger_query:
storage:
traces: some_store
traces_archive: another_store
ui:
config_file: ./cmd/jaeger/config-ui.json
log_access: true
# The maximum duration that is considered for clock skew adjustments.
# Defaults to 0 seconds, which means it's disabled.
max_clock_skew_adjust: 0s
grpc:
endpoint: 0.0.0.0:16685
http:
endpoint: 0.0.0.0:16686
jaeger_storage:
backends:
some_store:
memory:
max_traces: 1000000
another_store:
memory:
max_traces: 1000000
metric_backends:
some_metrics_storage:
prometheus:
endpoint: http://prometheus:9090
normalize_calls: true
normalize_duration: true
remote_sampling:
# You can either use file or adaptive sampling strategy in remote_sampling
# file:
# path: ./cmd/jaeger/sampling-strategies.json
adaptive:
sampling_store: some_store
initial_sampling_probability: 0.1
http:
grpc:
receivers:
otlp:
protocols:
grpc:
http:
jaeger:
protocols:
grpc:
thrift_binary:
thrift_compact:
thrift_http:
zipkin:
processors:
batch:
# Adaptive Sampling Processor is required to support adaptive sampling.
# It expects remote_sampling extension with `adaptive:` config to be enabled.
adaptive_sampling:
exporters:
jaeger_storage_exporter:
trace_storage: some_store

View File

@@ -0,0 +1,74 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
service:
extensions: [jaeger_storage, jaeger_query]
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [jaeger_storage_exporter, spanmetrics]
metrics/spanmetrics:
receivers: [spanmetrics]
exporters: [prometheus]
telemetry:
resource:
service.name: jaeger
metrics:
level: detailed
readers:
- pull:
exporter:
prometheus:
host: 0.0.0.0
port: 8888
logs:
level: DEBUG
extensions:
jaeger_query:
storage:
traces: some_storage
metrics: some_metrics_storage
jaeger_storage:
backends:
some_storage:
memory:
max_traces: 100000
metric_backends:
some_metrics_storage:
prometheus:
endpoint: http://prometheus:9090
normalize_calls: true
normalize_duration: true
connectors:
spanmetrics:
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
http:
endpoint: "0.0.0.0:4318"
processors:
batch:
exporters:
jaeger_storage_exporter:
trace_storage: some_storage
prometheus:
endpoint: "0.0.0.0:8889"

View File

@@ -11,22 +11,21 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
log_level: debug
grpc_listen_port: 9095
log_level: info
grpc_server_max_concurrent_streams: 1000
common:
instance_addr: 127.0.0.1
path_prefix: /tmp/loki
path_prefix: /loki
storage:
filesystem:
chunks_directory: /tmp/loki/chunks
rules_directory: /tmp/loki/rules
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
@@ -39,9 +38,6 @@ query_range:
enabled: true
max_size_mb: 100
limits_config:
metric_aggregation_enabled: true
schema_config:
configs:
- from: 2020-10-24
@@ -52,26 +48,16 @@ schema_config:
prefix: index_
period: 24h
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
allow_structured_metadata: true
max_line_size: 256KB
pattern_ingester:
enabled: true
metric_aggregation:
loki_address: localhost:3100
ruler:
alertmanager_url: http://localhost:9093
frontend:
encoding: protobuf
# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
#analytics:
# reporting_enabled: false

View File

@@ -15,57 +15,102 @@
receivers:
otlp:
protocols:
grpc: # OTLP gRPC 接收器
grpc:
endpoint: 0.0.0.0:4317
http: # OTLP HTTP 接收器
http:
endpoint: 0.0.0.0:4318
processors:
batch: # 批处理处理器,提升吞吐量
timeout: 5s
send_batch_size: 1000
batch:
timeout: 1s
send_batch_size: 1024
memory_limiter:
check_interval: 1s
limit_mib: 512
limit_mib: 1024
spike_limit_mib: 256
transform/logs:
log_statements:
- context: log
statements:
- set(attributes["message"], body.string)
- set(attributes["log.body"], body.string)
exporters:
otlp/traces: # OTLP 导出器,用于跟踪数据
endpoint: "jaeger:4317" # Jaeger 的 OTLP gRPC 端点
tls:
insecure: true # 开发环境禁用 TLS生产环境需配置证书
prometheus: # Prometheus 导出器,用于指标数据
endpoint: "0.0.0.0:8889" # Prometheus 刮取端点
namespace: "rustfs" # 指标前缀
send_timestamps: true # 发送时间戳
# enable_open_metrics: true
loki: # Loki 导出器,用于日志数据
# endpoint: "http://loki:3100/otlp/v1/logs"
endpoint: "http://loki:3100/loki/api/v1/push"
otlp/tempo:
endpoint: "tempo:4317"
tls:
insecure: true
compression: gzip
retry_on_failure:
enabled: true
initial_interval: 1s
max_interval: 30s
max_elapsed_time: 300s
sending_queue:
enabled: true
num_consumers: 10
queue_size: 5000
otlp/jaeger:
endpoint: "jaeger:4317"
tls:
insecure: true
compression: gzip
retry_on_failure:
enabled: true
initial_interval: 1s
max_interval: 30s
max_elapsed_time: 300s
sending_queue:
enabled: true
num_consumers: 10
queue_size: 5000
prometheus:
endpoint: "0.0.0.0:8889"
send_timestamps: true
metric_expiration: 5m
resource_to_telemetry_conversion:
enabled: true
otlphttp/loki:
endpoint: "http://loki:3100/otlp"
tls:
insecure: true
compression: gzip
extensions:
health_check:
endpoint: 0.0.0.0:13133
pprof:
endpoint: 0.0.0.0:1888
zpages:
endpoint: 0.0.0.0:55679
service:
extensions: [ health_check, pprof, zpages ] # 启用扩展
extensions: [ health_check, pprof, zpages ]
pipelines:
traces:
receivers: [ otlp ]
processors: [ memory_limiter,batch ]
exporters: [ otlp/traces ]
processors: [ memory_limiter, batch ]
exporters: [ otlp/tempo, otlp/jaeger ]
metrics:
receivers: [ otlp ]
processors: [ batch ]
exporters: [ prometheus ]
logs:
receivers: [ otlp ]
processors: [ batch ]
exporters: [ loki ]
processors: [ batch, transform/logs ]
exporters: [ otlphttp/loki ]
telemetry:
logs:
level: "info" # Collector 日志级别
level: "info"
encoding: "json"
metrics:
address: "0.0.0.0:8888" # Collector 自身指标暴露
level: "normal"
readers:
- pull:
exporter:
prometheus:
host: '0.0.0.0'
port: 8888

View File

@@ -13,13 +13,64 @@
# limitations under the License.
global:
scrape_interval: 5s # 刮取间隔
scrape_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
evaluation_interval: 15s
external_labels:
cluster: 'rustfs-dev' # Label to identify the cluster
replica: '1' # Replica identifier
scrape_configs:
- job_name: 'otel-collector'
static_configs:
- targets: ['otel-collector:8888'] # Collector 刮取指标
- job_name: 'otel-metrics'
- targets: [ 'otel-collector:8888' ] # Scrape metrics from Collector
scrape_interval: 10s
- job_name: 'rustfs-app-metrics'
static_configs:
- targets: ['otel-collector:8889'] # 应用指标
- targets: [ 'otel-collector:8889' ] # Application indicators
scrape_interval: 15s
metric_relabel_configs:
- source_labels: [ __name__ ]
regex: 'go_.*'
action: drop # Drop Go runtime metrics if not needed
- job_name: 'tempo'
static_configs:
- targets: [ 'tempo:3200' ] # Scrape metrics from Tempo
- job_name: 'jaeger'
static_configs:
- targets: [ 'jaeger:14269' ] # Jaeger admin port (14269 is standard for admin/metrics)
- job_name: 'loki'
static_configs:
- targets: [ 'loki:3100' ]
- job_name: 'prometheus'
static_configs:
- targets: [ 'localhost:9090' ]
otlp:
promote_resource_attributes:
- service.instance.id
- service.name
- service.namespace
- cloud.availability_zone
- cloud.region
- container.name
- deployment.environment.name
- k8s.cluster.name
- k8s.container.name
- k8s.cronjob.name
- k8s.daemonset.name
- k8s.deployment.name
- k8s.job.name
- k8s.namespace.name
- k8s.pod.name
- k8s.replicaset.name
- k8s.statefulset.name
translation_strategy: NoUTF8EscapingWithSuffixes
storage:
tsdb:
out_of_order_time_window: 30m

View File

@@ -0,0 +1,286 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# High Availability Tempo Configuration for docker-compose-example-for-rustfs.yml
# Features:
# - Distributed architecture with multiple components
# - Kafka-based ingestion for fault tolerance
# - Replication factor of 3 for data resilience
# - Query frontend for load balancing
# - Metrics generation from traces
# - WAL for durability
partition_ring_live_store: true
stream_over_http_enabled: true
server:
http_listen_port: 3200
http_server_read_timeout: 30s
http_server_write_timeout: 30s
grpc_server_max_recv_msg_size: 4194304 # 4MB
grpc_server_max_send_msg_size: 4194304
log_level: info
log_format: json
# Memberlist configuration for distributed mode
memberlist:
node_name: tempo
bind_port: 7946
join_members:
- tempo:7946
retransmit_factor: 4
node_timeout: 15s
retransmit_interval: 300ms
dead_node_reclaim_time: 30s
# Distributor configuration - receives traces and routes to ingesters
distributor:
ingester_write_path_enabled: true
kafka_write_path_enabled: true
rate_limit_bytes: 10MB
rate_limit_enabled: true
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
max_concurrent_streams: 0
max_receive_message_size: 4194304
http:
endpoint: "0.0.0.0:4318"
cors:
allowed_origins:
- "*"
max_age: 86400
jaeger:
protocols:
grpc:
endpoint: "0.0.0.0:14250"
thrift_http:
endpoint: "0.0.0.0:14268"
zipkin:
endpoint: "0.0.0.0:9411"
ring:
kvstore:
store: memberlist
heartbeat_timeout: 5s
replication_factor: 3
heartbeat_interval: 5s
# Ingester configuration - stores traces and querying
ingester:
lifecycler:
address: tempo
ring:
kvstore:
store: memberlist
replication_factor: 3
max_cache_freshness_per_sec: 10s
heartbeat_interval: 5s
heartbeat_timeout: 5s
num_tokens: 128
tokens_file_path: /var/tempo/tokens.json
claim_on_rollout: true
trace_idle_period: 20s
max_block_bytes: 10_000_000
max_block_duration: 10m
chunk_size_bytes: 1_000_000
chunk_encoding: snappy
wal:
checkpoint_duration: 5s
max_wal_blocks: 4
metrics:
enabled: true
level: block
target_info_duration: 15m
# WAL configuration for data durability
wal:
checkpoint_duration: 5s
flush_on_shutdown: true
path: /var/tempo/wal
# Kafka ingestion configuration - for high throughput scenarios
ingest:
enabled: true
kafka:
brokers: [ redpanda:9092 ]
topic: tempo-ingest
encoding: protobuf
consumer_group: tempo-ingest-consumer
session_timeout: 10s
rebalance_timeout: 1m
partition: auto
verbosity: 2
# Query frontend configuration - distributed querying
query_frontend:
compression: gzip
downstream_url: http://localhost:3200
log_queries_longer_than: 5s
cache_uncompressed_bytes: 100MB
max_outstanding_requests_per_tenant: 100
max_query_length: 48h
max_query_lookback: 30d
default_result_cache_ttl: 1m
result_cache:
cache:
enable_fifocache: true
default_validity: 1m
rf1_after: "1999-01-01T00:00:00Z"
mcp_server:
enabled: true
# Querier configuration - queries traces
querier:
frontend_worker:
frontend_address: localhost:3200
grpc_client_config:
max_recv_msg_size: 104857600
max_concurrent_queries: 20
max_metric_bytes_per_trace: 1MB
# Query scheduler configuration - for distributed querying
query_scheduler:
use_scheduler_ring: false
# Metrics generator configuration - generates metrics from traces
metrics_generator:
enabled: true
registry:
enabled: true
external_labels:
source: tempo
cluster: rustfs-docker-ha
environment: production
storage:
path: /var/tempo/generator/wal
remote_write:
- url: http://prometheus:9090/api/v1/write
send_exemplars: true
resource_to_telemetry_conversion:
enabled: true
processor:
batch:
timeout: 10s
send_batch_size: 1024
memory_limiter:
check_interval: 5s
limit_mib: 512
spike_limit_mib: 128
processors:
- span-metrics
- local-blocks
- service-graphs
generate_native_histograms: both
# Backend worker configuration
backend_worker:
backend_scheduler_addr: localhost:3200
compaction:
block_retention: 24h
compacted_block_retention: 1h
ring:
kvstore:
store: memberlist
# Backend scheduler configuration
backend_scheduler:
enabled: true
provider:
compaction:
compaction:
block_retention: 24h
compacted_block_retention: 1h
concurrency: 25
v2_out_path: /var/tempo/blocks/compaction
# Storage configuration - local backend with proper retention
storage:
trace:
backend: local
wal:
path: /var/tempo/wal
checkpoint_duration: 5s
flush_on_shutdown: true
local:
path: /var/tempo/blocks
bloom_filter_false_positive: 0.05
bloom_shift: 4
index:
downsample_bytes: 1000000
page_size_bytes: 0
cache_size_bytes: 0
pool:
max_workers: 400
queue_depth: 10000
# Compactor configuration - manages block compaction
compactor:
compaction:
block_retention: 168h # 7 days
compacted_block_retention: 1h
concurrency: 25
v2_out_path: /var/tempo/blocks/compaction
shard_count: 32
max_block_bytes: 107374182400 # 100GB
max_compaction_objects: 6000000
max_time_per_tenant: 5m
block_size_bytes: 107374182400
ring:
kvstore:
store: memberlist
heartbeat_interval: 5s
heartbeat_timeout: 5s
# Limits configuration - rate limiting and quotas
limits:
max_traces_per_user: 10000
max_bytes_per_trace: 10485760 # 10MB
max_search_bytes_per_trace: 0
forgiving_oversize_traces: true
rate_limit_bytes: 10MB
rate_limit_enabled: true
ingestion_burst_size_bytes: 20MB
ingestion_rate_limit_bytes: 10MB
max_bytes_per_second: 10485760
metrics_generator_max_active_series: 10000
metrics_generator_max_churned_series: 10000
metrics_generator_forta_out_of_order_ttl: 5m
# Override configuration
overrides:
defaults:
metrics_generator:
processors:
- span-metrics
- local-blocks
- service-graphs
generate_native_histograms: both
max_active_series: 10000
max_churned_series: 10000
# Usage reporting configuration
usage_report:
reporting_enabled: false
# Tracing configuration for debugging
tracing:
enabled: true
jaeger:
sampler:
name: probabilistic
param: 0.1
reporter_log_spans: false

View File

@@ -0,0 +1,124 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
partition_ring_live_store: true
stream_over_http_enabled: true
server:
http_listen_port: 3200
log_level: info
memberlist:
node_name: tempo
bind_port: 7946
join_members:
- tempo:7946
# Distributor configuration - receives traces and writes directly to ingesters
distributor:
ingester_write_path_enabled: true
kafka_write_path_enabled: false
receivers:
otlp:
protocols:
grpc:
endpoint: "tempo:4317"
http:
endpoint: "tempo:4318"
ring:
kvstore:
store: memberlist
# Ingester configuration - consumes from Kafka and stores traces
ingester:
lifecycler:
ring:
kvstore:
store: memberlist
replication_factor: 1
tokens_file_path: /var/tempo/tokens.json
trace_idle_period: 10s
max_block_bytes: 1_000_000
max_block_duration: 5m
backend_scheduler:
provider:
compaction:
compaction:
block_retention: 1h
backend_worker:
backend_scheduler_addr: localhost:3200
compaction:
block_retention: 1h
ring:
kvstore:
store: memberlist
querier:
frontend_worker:
frontend_address: tempo:3200
metrics_generator:
registry:
external_labels:
source: tempo
cluster: docker-compose
storage:
path: /var/tempo/generator/wal
remote_write:
- url: http://prometheus:9090/api/v1/write
send_exemplars: true
query_frontend:
rf1_after: "1999-01-01T00:00:00Z"
mcp_server:
enabled: true
storage:
trace:
backend: local
wal:
path: /var/tempo/wal
local:
path: /var/tempo/blocks
overrides:
defaults:
metrics_generator:
processors: [ "span-metrics", "service-graphs", "local-blocks" ]
generate_native_histograms: both
ingest:
enabled: false
# Disabled because using direct ingester write path
# If you want Kafka path, enable this and set:
# kafka:
# brokers: [redpanda:9092]
# topic: tempo-ingest
# encoding: protobuf
# consumer_group: tempo-ingest-consumer
block_builder:
consume_cycle_duration: 30s
compactor:
compaction:
block_retention: 168h # 7 days
ring:
kvstore:
store: memberlist
usage_report:
reporting_enabled: false

View File

@@ -5,71 +5,57 @@
English | [中文](README_ZH.md)
This directory contains the configuration files for setting up an observability stack with OpenObserve and OpenTelemetry
Collector.
This directory contains the configuration for an **alternative** observability stack using OpenObserve.
### Overview
## ⚠️ Note
This setup provides a complete observability solution for your applications:
For the **recommended** observability stack (Prometheus, Grafana, Tempo, Loki), please see `../observability/`.
- **OpenObserve**: A modern, open-source observability platform for logs, metrics, and traces.
- **OpenTelemetry Collector**: Collects and processes telemetry data before sending it to OpenObserve.
## 🌟 Overview
### Setup Instructions
OpenObserve is a lightweight, all-in-one observability platform that handles logs, metrics, and traces in a single binary. This setup is ideal for:
- Resource-constrained environments.
- Quick setup and testing.
- Users who prefer a unified UI.
1. **Prerequisites**:
- Docker and Docker Compose installed
- Sufficient memory resources (minimum 2GB recommended)
## 🚀 Quick Start
2. **Starting the Services**:
```bash
cd .docker/openobserve-otel
docker compose -f docker-compose.yml up -d
```
3. **Accessing the Dashboard**:
- OpenObserve UI: http://localhost:5080
- Default credentials:
- Username: root@rustfs.com
- Password: rustfs123
### Configuration
#### OpenObserve Configuration
The OpenObserve service is configured with:
- Root user credentials
- Data persistence through a volume mount
- Memory cache enabled
- Health checks
- Exposed ports:
- 5080: HTTP API and UI
- 5081: OTLP gRPC
#### OpenTelemetry Collector Configuration
The collector is configured to:
- Receive telemetry data via OTLP (HTTP and gRPC)
- Collect logs from files
- Process data in batches
- Export data to OpenObserve
- Manage memory usage
### Integration with Your Application
To send telemetry data from your application, configure your OpenTelemetry SDK to send data to:
- OTLP gRPC: `localhost:4317`
- OTLP HTTP: `localhost:4318`
For example, in a Rust application using the `rustfs-obs` library:
### 1. Start Services
```bash
export RUSTFS_OBS_ENDPOINT=http://localhost:4317
export RUSTFS_OBS_SERVICE_NAME=yourservice
export RUSTFS_OBS_SERVICE_VERSION=1.0.0
export RUSTFS_OBS_ENVIRONMENT=development
cd .docker/openobserve-otel
docker compose up -d
```
### 2. Access Dashboard
- **URL**: [http://localhost:5080](http://localhost:5080)
- **Username**: `root@rustfs.com`
- **Password**: `rustfs123`
## 🛠️ Configuration
### OpenObserve
- **Persistence**: Data is persisted to a Docker volume.
- **Ports**:
- `5080`: HTTP API and UI
- `5081`: OTLP gRPC
### OpenTelemetry Collector
- **Receivers**: OTLP (gRPC `4317`, HTTP `4318`)
- **Exporters**: Sends data to OpenObserve.
## 🔗 Integration
Configure your application to send OTLP data to the collector:
- **Endpoint**: `http://localhost:4318` (HTTP) or `localhost:4317` (gRPC)
Example for RustFS:
```bash
export RUSTFS_OBS_ENDPOINT=http://localhost:4318
export RUSTFS_OBS_SERVICE_NAME=rustfs-node-1
```

View File

@@ -5,71 +5,57 @@
[English](README.md) | 中文
## 中文
本目录包含使用 OpenObserve 的**替代**可观测性技术栈配置。
本目录包含搭建 OpenObserve 和 OpenTelemetry Collector 可观测性栈的配置文件。
## ⚠️ 注意
### 概述
对于**推荐**的可观测性技术栈Prometheus, Grafana, Tempo, Loki请参阅 `../observability/`
此设置为应用程序提供了完整的可观测性解决方案:
## 🌟 概览
- **OpenObserve**:现代化、开源的可观测性平台,用于日志、指标和追踪。
- **OpenTelemetry Collector**:收集和处理遥测数据,然后将其发送到 OpenObserve
OpenObserve 是一个轻量级、一体化的可观测性平台,在一个二进制文件中处理日志、指标和追踪。此设置非常适合:
- 资源受限的环境
- 快速设置和测试。
- 喜欢统一 UI 的用户。
### 设置说明
## 🚀 快速开始
1. **前提条件**
- 已安装 Docker 和 Docker Compose
- 足够的内存资源(建议至少 2GB
2. **启动服务**
```bash
cd .docker/openobserve-otel
docker compose -f docker-compose.yml up -d
```
3. **访问仪表板**
- OpenObserve UIhttp://localhost:5080
- 默认凭据:
- 用户名root@rustfs.com
- 密码rustfs123
### 配置
#### OpenObserve 配置
OpenObserve 服务配置:
- 根用户凭据
- 通过卷挂载实现数据持久化
- 启用内存缓存
- 健康检查
- 暴露端口:
- 5080HTTP API 和 UI
- 5081OTLP gRPC
#### OpenTelemetry Collector 配置
收集器配置为:
- 通过 OTLPHTTP 和 gRPC接收遥测数据
- 从文件中收集日志
- 批处理数据
- 将数据导出到 OpenObserve
- 管理内存使用
### 与应用程序集成
要从应用程序发送遥测数据,将 OpenTelemetry SDK 配置为发送数据到:
- OTLP gRPC:`localhost:4317`
- OTLP HTTP:`localhost:4318`
例如,在使用 `rustfs-obs` 库的 Rust 应用程序中:
### 1. 启动服务
```bash
export RUSTFS_OBS_ENDPOINT=http://localhost:4317
export RUSTFS_OBS_SERVICE_NAME=yourservice
export RUSTFS_OBS_SERVICE_VERSION=1.0.0
export RUSTFS_OBS_ENVIRONMENT=development
```
cd .docker/openobserve-otel
docker compose up -d
```
### 2. 访问仪表盘
- **URL**: [http://localhost:5080](http://localhost:5080)
- **用户名**: `root@rustfs.com`
- **密码**: `rustfs123`
## 🛠️ 配置
### OpenObserve
- **持久化**: 数据持久化到 Docker 卷。
- **端口**:
- `5080`: HTTP API 和 UI
- `5081`: OTLP gRPC
### OpenTelemetry Collector
- **接收器**: OTLP (gRPC `4317`, HTTP `4318`)
- **导出器**: 将数据发送到 OpenObserve。
## 🔗 集成
配置您的应用程序将 OTLP 数据发送到收集器:
- **端点**: `http://localhost:4318` (HTTP) 或 `localhost:4317` (gRPC)
RustFS 示例:
```bash
export RUSTFS_OBS_ENDPOINT=http://localhost:4318
export RUSTFS_OBS_SERVICE_NAME=rustfs-node-1
```

1
.dockerignore Normal file
View File

@@ -0,0 +1 @@
target

1
.envrc Normal file
View File

@@ -0,0 +1 @@
use flake

30
.github/AGENTS.md vendored Normal file
View File

@@ -0,0 +1,30 @@
# GitHub Workflow Instructions
Applies to `.github/` and repository pull-request operations.
## Pull Requests
- PR titles and descriptions must be in English.
- Use `.github/pull_request_template.md` for every PR body.
- Keep all template section headings.
- Use `N/A` for non-applicable sections.
- Include verification commands in the PR details.
- For `gh pr create` and `gh pr edit`, always write markdown body to a file and pass `--body-file`.
- Do not use multiline inline `--body`; backticks and shell expansion can corrupt content or trigger unintended commands.
- Recommended pattern:
- `cat > /tmp/pr_body.md <<'EOF'`
- `...markdown...`
- `EOF`
- `gh pr create ... --body-file /tmp/pr_body.md`
## CI Alignment
When changing CI-sensitive behavior, keep local validation aligned with `.github/workflows/ci.yml`.
Current `test-and-lint` gate includes:
- `cargo nextest run --all --exclude e2e_test`
- `cargo test --all --doc`
- `cargo fmt --all --check`
- `cargo clippy --all-targets --all-features -- -D warnings`
- `./scripts/check_layer_dependencies.sh`

View File

@@ -52,24 +52,19 @@ runs:
sudo apt-get install -y \
musl-tools \
build-essential \
lld \
libdbus-1-dev \
libwayland-dev \
libwebkit2gtk-4.1-dev \
libxdo-dev \
pkg-config \
libssl-dev
- name: Install protoc
uses: arduino/setup-protoc@v3
with:
version: "31.1"
version: "33.1"
repo-token: ${{ inputs.github-token }}
- name: Install flatc
uses: Nugine/setup-flatc@v1
with:
version: "25.2.10"
version: "25.9.23"
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable

1
.github/copilot-instructions.md vendored Symbolic link
View File

@@ -0,0 +1 @@
../AGENTS.md

View File

@@ -22,8 +22,23 @@ updates:
- package-ecosystem: "cargo" # See documentation for possible values
directory: "/" # Location of package manifests
schedule:
interval: "monthly"
interval: "weekly"
day: "monday"
timezone: "Asia/Shanghai"
time: "08:00"
ignore:
- dependency-name: "object_store"
versions: [ "0.13.x" ]
- dependency-name: "libunftp"
versions: [ "0.23.x" ]
groups:
s3s:
update-types:
- "minor"
- "patch"
patterns:
- "s3s"
- "s3s-*"
dependencies:
patterns:
- "*"
- "*"

View File

@@ -19,9 +19,7 @@ Pull Request Template for RustFS
## Checklist
- [ ] I have read and followed the [CONTRIBUTING.md](CONTRIBUTING.md) guidelines
- [ ] Code is formatted with `cargo fmt --all`
- [ ] Passed `cargo clippy --all-targets --all-features -- -D warnings`
- [ ] Passed `cargo check --all-targets`
- [ ] Passed `make pre-commit`
- [ ] Added/updated necessary tests
- [ ] Documentation updated (if needed)
- [ ] CI/CD passed (if applicable)

103
.github/s3tests/README.md vendored Normal file
View File

@@ -0,0 +1,103 @@
# S3 Compatibility Tests Configuration
This directory contains the configuration for running [Ceph S3 compatibility tests](https://github.com/ceph/s3-tests) against RustFS.
## Configuration File
The `s3tests.conf` file is based on the official `s3tests.conf.SAMPLE` from the ceph/s3-tests repository. It uses environment variable substitution via `envsubst` to configure the endpoint and credentials.
### Key Configuration Points
- **Host**: Set via `${S3_HOST}` environment variable (e.g., `rustfs-single` for single-node, `lb` for multi-node)
- **Port**: 9000 (standard RustFS port)
- **Credentials**: Uses `${S3_ACCESS_KEY}` and `${S3_SECRET_KEY}` from workflow environment
- **TLS**: Disabled (`is_secure = False`)
## Test Execution Strategy
### Network Connectivity Fix
Tests run inside a Docker container on the `rustfs-net` network, which allows them to resolve and connect to the RustFS container hostnames. This fixes the "Temporary failure in name resolution" error that occurred when tests ran on the GitHub runner host.
### Performance Optimizations
1. **Parallel Execution**: Uses `pytest-xdist` with `-n 4` to run tests in parallel across 4 workers
2. **Load Distribution**: Uses `--dist=loadgroup` to distribute test groups across workers
3. **Fail-Fast**: Uses `--maxfail=50` to stop after 50 failures, saving time on catastrophic failures
### Feature Filtering
Tests are filtered using pytest markers (`-m`) to skip features not yet supported by RustFS:
- `lifecycle` - Bucket lifecycle policies
- `versioning` - Object versioning
- `s3website` - Static website hosting
- `bucket_logging` - Bucket logging
- `encryption` / `sse_s3` - Server-side encryption
- `cloud_transition` / `cloud_restore` - Cloud storage transitions
- `lifecycle_expiration` / `lifecycle_transition` - Lifecycle operations
This filtering:
1. Reduces test execution time significantly (from 1+ hour to ~10-15 minutes)
2. Focuses on features RustFS currently supports
3. Avoids hundreds of expected failures
## Running Tests Locally
### Single-Node Test
```bash
# Set credentials
export S3_ACCESS_KEY=rustfsadmin
export S3_SECRET_KEY=rustfsadmin
# Start RustFS container
docker run -d --name rustfs-single \
--network rustfs-net \
-e RUSTFS_ADDRESS=0.0.0.0:9000 \
-e RUSTFS_ACCESS_KEY=$S3_ACCESS_KEY \
-e RUSTFS_SECRET_KEY=$S3_SECRET_KEY \
-e RUSTFS_VOLUMES="/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3" \
rustfs-ci
# Generate config
export S3_HOST=rustfs-single
envsubst < .github/s3tests/s3tests.conf > /tmp/s3tests.conf
# Run tests
docker run --rm \
--network rustfs-net \
-v /tmp/s3tests.conf:/etc/s3tests.conf:ro \
python:3.12-slim \
bash -c '
apt-get update -qq && apt-get install -y -qq git
git clone --depth 1 https://github.com/ceph/s3-tests.git /s3-tests
cd /s3-tests
pip install -q -r requirements.txt pytest-xdist
S3TEST_CONF=/etc/s3tests.conf pytest -v -n 4 \
s3tests/functional/test_s3.py \
-m "not lifecycle and not versioning and not s3website and not bucket_logging and not encryption and not sse_s3"
'
```
## Test Results Interpretation
- **PASSED**: Test succeeded, feature works correctly
- **FAILED**: Test failed, indicates a potential bug or incompatibility
- **ERROR**: Test setup failed (e.g., network issues, missing dependencies)
- **SKIPPED**: Test skipped due to marker filtering
## Adding New Feature Support
When adding support for a new S3 feature to RustFS:
1. Remove the corresponding marker from the filter in `.github/workflows/e2e-s3tests.yml`
2. Run the tests to verify compatibility
3. Fix any failing tests
4. Update this README to reflect the newly supported feature
## References
- [Ceph S3 Tests Repository](https://github.com/ceph/s3-tests)
- [S3 API Compatibility](https://docs.aws.amazon.com/AmazonS3/latest/API/)
- [pytest-xdist Documentation](https://pytest-xdist.readthedocs.io/)

193
.github/s3tests/s3tests.conf vendored Normal file
View File

@@ -0,0 +1,193 @@
# RustFS s3-tests configuration
# Based on: https://github.com/ceph/s3-tests/blob/master/s3tests.conf.SAMPLE
#
# Usage:
# Single-node: S3_HOST=rustfs-single envsubst < s3tests.conf > /tmp/s3tests.conf
# Multi-node: S3_HOST=lb envsubst < s3tests.conf > /tmp/s3tests.conf
[DEFAULT]
## this section is just used for host, port and bucket_prefix
# host set for RustFS - will be substituted via envsubst
host = ${S3_HOST}
# port for RustFS
port = 9000
## say "False" to disable TLS
is_secure = False
## say "False" to disable SSL Verify
ssl_verify = False
[fixtures]
## all the buckets created will start with this prefix;
## {random} will be filled with random characters to pad
## the prefix to 30 characters long, and avoid collisions
bucket prefix = rustfs-{random}-
# all the iam account resources (users, roles, etc) created
# will start with this name prefix
iam name prefix = s3-tests-
# all the iam account resources (users, roles, etc) created
# will start with this path prefix
iam path prefix = /s3-tests/
[s3 main]
# main display_name
display_name = RustFS Tester
# main user_id
user_id = rustfsadmin
# main email
email = tester@rustfs.local
# zonegroup api_name for bucket location
api_name = default
## main AWS access key
access_key = ${S3_ACCESS_KEY}
## main AWS secret key
secret_key = ${S3_SECRET_KEY}
## replace with key id obtained when secret is created, or delete if KMS not tested
#kms_keyid = 01234567-89ab-cdef-0123-456789abcdef
## Storage classes
#storage_classes = "LUKEWARM, FROZEN"
## Lifecycle debug interval (default: 10)
#lc_debug_interval = 20
## Restore debug interval (default: 100)
#rgw_restore_debug_interval = 60
#rgw_restore_processor_period = 60
[s3 alt]
# alt display_name
display_name = RustFS Alt Tester
## alt email
email = alt@rustfs.local
# alt user_id
user_id = rustfsalt
# alt AWS access key (must be different from s3 main for many tests)
access_key = ${S3_ALT_ACCESS_KEY}
# alt AWS secret key
secret_key = ${S3_ALT_SECRET_KEY}
#[s3 cloud]
## to run the testcases with "cloud_transition" for transition
## and "cloud_restore" for restore attribute.
## Note: the waiting time may have to tweaked depending on
## the I/O latency to the cloud endpoint.
## host set for cloud endpoint
# host = localhost
## port set for cloud endpoint
# port = 8001
## say "False" to disable TLS
# is_secure = False
## cloud endpoint credentials
# access_key = 0555b35654ad1656d804
# secret_key = h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==
## storage class configured as cloud tier on local rgw server
# cloud_storage_class = CLOUDTIER
## Below are optional -
## Above configured cloud storage class config options
# retain_head_object = false
# allow_read_through = false # change it to enable read_through
# read_through_restore_days = 2
# target_storage_class = Target_SC
# target_path = cloud-bucket
## another regular storage class to test multiple transition rules,
# storage_class = S1
[s3 tenant]
# tenant display_name
display_name = RustFS Tenant Tester
# tenant user_id
# Note: Using same user_id as main to avoid teardown failures.
# RustFS does not currently support multi-tenancy, so the tenant client
# effectively operates as the main user. This ensures nuke_prefixed_buckets()
# in s3-tests teardown can successfully clean up resources.
user_id = rustfsadmin
# tenant AWS access key
access_key = ${S3_ACCESS_KEY}
# tenant AWS secret key
secret_key = ${S3_SECRET_KEY}
# tenant email
email = tenant@rustfs.local
# tenant name
# Note: Empty tenant name to avoid multi-tenant path issues during teardown.
# When s3-tests calls get_tenant_client(), it uses this tenant value in requests.
# An empty value makes the tenant client behave like the main client, preventing
# "bucket not found" errors when teardown tries to clean up test buckets.
tenant =
#following section needs to be added for all sts-tests
[iam]
#used for iam operations in sts-tests
#email
email = s3@rustfs.local
#user_id
user_id = rustfsiam
#access_key
access_key = ${S3_ACCESS_KEY}
#secret_key
secret_key = ${S3_SECRET_KEY}
#display_name
display_name = RustFS IAM User
# iam account root user for iam_account tests
[iam root]
access_key = ${S3_ACCESS_KEY}
secret_key = ${S3_SECRET_KEY}
user_id = RGW11111111111111111
email = account1@rustfs.local
# iam account root user in a different account than [iam root]
[iam alt root]
access_key = ${S3_ACCESS_KEY}
secret_key = ${S3_SECRET_KEY}
user_id = RGW22222222222222222
email = account2@rustfs.local
#following section needs to be added when you want to run Assume Role With Webidentity test
[webidentity]
#used for assume role with web identity test in sts-tests
#all parameters will be obtained from ceph/qa/tasks/keycloak.py
#token=<access_token>
#aud=<obtained after introspecting token>
#sub=<obtained after introspecting token>
#azp=<obtained after introspecting token>
#user_token=<access token for a user, with attribute Department=[Engineering, Marketing>]
#thumbprint=<obtained from x509 certificate>
#KC_REALM=<name of the realm>

View File

@@ -16,13 +16,13 @@ name: Security Audit
on:
push:
branches: [main]
branches: [ main ]
paths:
- '**/Cargo.toml'
- '**/Cargo.lock'
- '.github/workflows/audit.yml'
pull_request:
branches: [main]
branches: [ main ]
paths:
- '**/Cargo.toml'
- '**/Cargo.lock'
@@ -31,17 +31,20 @@ on:
- cron: '0 0 * * 0' # Weekly on Sunday at midnight UTC
workflow_dispatch:
permissions:
contents: read
env:
CARGO_TERM_COLOR: always
jobs:
security-audit:
name: Security Audit
runs-on: ubuntu-latest
runs-on: ubicloud-standard-2
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@v6
- name: Install cargo-audit
uses: taiki-e/install-action@v2
@@ -54,7 +57,7 @@ jobs:
- name: Upload audit results
if: always()
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v6
with:
name: security-audit-results-${{ github.run_number }}
path: audit-results.json
@@ -62,14 +65,14 @@ jobs:
dependency-review:
name: Dependency Review
runs-on: ubuntu-latest
runs-on: ubicloud-standard-2
if: github.event_name == 'pull_request'
permissions:
contents: read
pull-requests: write
steps:
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@v6
- name: Dependency Review
uses: actions/dependency-review-action@v4

View File

@@ -12,28 +12,25 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Build and Release Workflow
#
# This workflow builds RustFS binaries and automatically triggers Docker image builds.
#
# Flow:
# 1. Build binaries for multiple platforms
# 2. Upload binaries to OSS storage
# 3. Trigger docker.yml to build and push images using the uploaded binaries
#
# Manual Parameters:
# - build_docker: Build and push Docker images (default: true)
# - platforms: Comma-separated platform IDs or 'all' (default: all)
name: Build and Release
on:
push:
tags: ["*"]
branches: [main]
paths-ignore:
- "**.md"
- "**.txt"
- ".github/**"
- "docs/**"
- "deploy/**"
- "scripts/dev_*.sh"
- "LICENSE*"
- "README*"
- "**/*.png"
- "**/*.jpg"
- "**/*.svg"
- ".gitignore"
- ".dockerignore"
pull_request:
branches: [main]
tags: [ "*.*.*" ]
branches: [ main ]
paths-ignore:
- "**.md"
- "**.txt"
@@ -52,11 +49,19 @@ on:
- cron: "0 0 * * 0" # Weekly on Sunday at midnight UTC
workflow_dispatch:
inputs:
force_build:
description: "Force build even without changes"
build_docker:
description: "Build and push Docker images after binary build"
required: false
default: false
default: true
type: boolean
platforms:
description: "Comma-separated targets or 'all' (e.g. linux-x86_64-musl,macos-aarch64)"
required: false
default: "all"
type: string
permissions:
contents: read
env:
CARGO_TERM_COLOR: always
@@ -65,83 +70,147 @@ env:
CARGO_INCREMENTAL: 0
jobs:
# Second layer: Business logic level checks (handling build strategy)
# Build strategy check - determine build type based on trigger
build-check:
name: Build Strategy Check
runs-on: ubuntu-latest
runs-on: ubicloud-standard-2
outputs:
should_build: ${{ steps.check.outputs.should_build }}
build_type: ${{ steps.check.outputs.build_type }}
version: ${{ steps.check.outputs.version }}
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
steps:
- name: Checkout repository
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Determine build strategy
id: check
run: |
should_build=false
build_type="none"
version=""
short_sha=""
is_prerelease=false
# Business logic: when we need to build
if [[ "${{ github.event_name }}" == "schedule" ]] || \
[[ "${{ github.event_name }}" == "workflow_dispatch" ]] || \
[[ "${{ github.event.inputs.force_build }}" == "true" ]] || \
[[ "${{ contains(github.event.head_commit.message, '--build') }}" == "true" ]]; then
# Get short SHA for all builds
short_sha=$(git rev-parse --short HEAD)
# Determine build type based on trigger
if [[ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]]; then
# Tag push - release or prerelease
should_build=true
tag_name="${GITHUB_REF#refs/tags/}"
version="${tag_name}"
# Check if this is a prerelease
if [[ "$tag_name" == *"alpha"* ]] || [[ "$tag_name" == *"beta"* ]] || [[ "$tag_name" == *"rc"* ]]; then
build_type="prerelease"
is_prerelease=true
echo "🚀 Prerelease build detected: $tag_name"
else
build_type="release"
echo "📦 Release build detected: $tag_name"
fi
elif [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
# Main branch push - development build
should_build=true
build_type="development"
fi
# Always build for tag pushes (version releases)
if [[ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]]; then
version="dev-${short_sha}"
echo "🛠️ Development build detected"
elif [[ "${{ github.event_name }}" == "schedule" ]] || \
[[ "${{ github.event_name }}" == "workflow_dispatch" ]] || \
[[ "${{ contains(github.event.head_commit.message, '--build') }}" == "true" ]]; then
# Scheduled or manual build
should_build=true
build_type="release"
echo "🏷️ Tag detected: forcing release build"
build_type="development"
version="dev-${short_sha}"
echo "⚡ Manual/scheduled build detected"
fi
echo "should_build=$should_build" >> $GITHUB_OUTPUT
echo "build_type=$build_type" >> $GITHUB_OUTPUT
echo "Build needed: $should_build (type: $build_type)"
echo "version=$version" >> $GITHUB_OUTPUT
echo "short_sha=$short_sha" >> $GITHUB_OUTPUT
echo "is_prerelease=$is_prerelease" >> $GITHUB_OUTPUT
echo "📊 Build Summary:"
echo " - Should build: $should_build"
echo " - Build type: $build_type"
echo " - Version: $version"
echo " - Short SHA: $short_sha"
echo " - Is prerelease: $is_prerelease"
# Build RustFS binaries
prepare-platform-matrix:
name: Prepare Platform Matrix
runs-on: ubicloud-standard-2
outputs:
matrix: ${{ steps.select.outputs.matrix }}
selected: ${{ steps.select.outputs.selected }}
steps:
- name: Select target platforms
id: select
shell: bash
run: |
set -euo pipefail
selected="${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}"
selected="$(echo "${selected}" | tr -d '[:space:]')"
if [[ -z "${selected}" ]]; then
selected="all"
fi
all='{"include":[
{"target_id":"linux-x86_64-musl","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":""},
{"target_id":"linux-aarch64-musl","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-musl","cross":true,"platform":"linux","rustflags":""},
{"target_id":"linux-x86_64-gnu","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
{"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
{"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
{"target_id":"macos-x86_64","os":"macos-latest","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
{"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":""}
]}'
if [[ "${selected}" == "all" ]]; then
matrix="$(jq -c . <<<"${all}")"
else
unknown="$(jq -rn --arg selected "${selected}" --argjson all "${all}" '
($selected | split(",") | map(select(length > 0))) as $req
| ($all.include | map(.target_id)) as $known
| [$req[] | select(( $known | index(.) ) == null)]
')"
if [[ "$(jq 'length' <<<"${unknown}")" -gt 0 ]]; then
echo "Unknown platforms: $(jq -r 'join(\",\")' <<<"${unknown}")" >&2
echo "Allowed: $(jq -r '.include[].target_id' <<<"${all}" | paste -sd ',' -)" >&2
exit 1
fi
matrix="$(jq -c --arg selected "${selected}" '
($selected | split(",") | map(select(length > 0))) as $req
| .include |= map(select(.target_id as $id | ($req | index($id))))
' <<<"${all}")"
fi
echo "selected=${selected}" >> "$GITHUB_OUTPUT"
echo "matrix=${matrix}" >> "$GITHUB_OUTPUT"
echo "Selected platforms: ${selected}"
build-rustfs:
name: Build RustFS
needs: [build-check]
if: needs.build-check.outputs.should_build == 'true'
needs: [ build-check, prepare-platform-matrix ]
if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
runs-on: ${{ matrix.os }}
timeout-minutes: 60
env:
RUSTFLAGS: ${{ matrix.cross == 'false' && '-C target-cpu=native' || '' }}
RUSTFLAGS: ${{ matrix.rustflags }}
strategy:
fail-fast: false
matrix:
include:
# Linux builds
- os: ubuntu-latest
target: x86_64-unknown-linux-musl
cross: false
platform: linux
- os: ubuntu-latest
target: aarch64-unknown-linux-musl
cross: true
platform: linux
# macOS builds
- os: macos-latest
target: aarch64-apple-darwin
cross: false
platform: macos
- os: macos-latest
target: x86_64-apple-darwin
cross: false
platform: macos
# # Windows builds (temporarily disabled)
# - os: windows-latest
# target: x86_64-pc-windows-msvc
# cross: false
# platform: windows
# - os: windows-latest
# target: aarch64-pc-windows-msvc
# cross: true
# platform: windows
matrix: ${{ fromJson(needs.prepare-platform-matrix.outputs.matrix) }}
steps:
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@v6
with:
fetch-depth: 0
@@ -156,9 +225,9 @@ jobs:
install-cross-tools: ${{ matrix.cross }}
- name: Download static console assets
shell: bash
run: |
mkdir -p ./rustfs/static
rm -rf ./rustfs/static/*
if [[ "${{ matrix.platform }}" == "windows" ]]; then
curl.exe -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" -o console.zip --retry 3 --retry-delay 5 --max-time 300
if [[ $? -eq 0 ]]; then
@@ -170,7 +239,6 @@ jobs:
fi
else
chmod +w ./rustfs/static/LICENSE || true
rm -f ./rustfs/static/LICENSE
curl -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" \
-o console.zip --retry 3 --retry-delay 5 --max-time 300
if [[ $? -eq 0 ]]; then
@@ -183,6 +251,7 @@ jobs:
fi
- name: Build RustFS
shell: bash
run: |
# Force rebuild by touching build.rs
touch rustfs/build.rs
@@ -193,7 +262,7 @@ jobs:
cargo install cross --git https://github.com/cross-rs/cross
cross build --release --target ${{ matrix.target }} -p rustfs --bins
else
# Use zigbuild for Linux ARM64
# Use zigbuild for other cross-compilation
cargo zigbuild --release --target ${{ matrix.target }} -p rustfs --bins
fi
else
@@ -204,7 +273,63 @@ jobs:
id: package
shell: bash
run: |
PACKAGE_NAME="rustfs-${{ matrix.target }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
SHORT_SHA="${{ needs.build-check.outputs.short_sha }}"
# Extract platform and arch from target
TARGET="${{ matrix.target }}"
PLATFORM="${{ matrix.platform }}"
# Map target to architecture and variant
case "$TARGET" in
*x86_64*musl*)
ARCH="x86_64"
VARIANT="musl"
;;
*x86_64*gnu*)
ARCH="x86_64"
VARIANT="gnu"
;;
*x86_64*)
ARCH="x86_64"
VARIANT=""
;;
*aarch64*musl*|*arm64*musl*)
ARCH="aarch64"
VARIANT="musl"
;;
*aarch64*gnu*|*arm64*gnu*)
ARCH="aarch64"
VARIANT="gnu"
;;
*aarch64*|*arm64*)
ARCH="aarch64"
VARIANT=""
;;
*armv7*)
ARCH="armv7"
VARIANT=""
;;
*)
ARCH="unknown"
VARIANT=""
;;
esac
# Generate package name based on build type
if [[ -n "$VARIANT" ]]; then
ARCH_WITH_VARIANT="${ARCH}-${VARIANT}"
else
ARCH_WITH_VARIANT="${ARCH}"
fi
if [[ "$BUILD_TYPE" == "development" ]]; then
# Development build: rustfs-${platform}-${arch}-${variant}-dev-${short_sha}.zip
PACKAGE_NAME="rustfs-${PLATFORM}-${ARCH_WITH_VARIANT}-dev-${SHORT_SHA}"
else
# Release/Prerelease build: rustfs-${platform}-${arch}-${variant}-v${version}.zip
PACKAGE_NAME="rustfs-${PLATFORM}-${ARCH_WITH_VARIANT}-v${VERSION}"
fi
# Create zip packages for all platforms
# Ensure zip is available
@@ -215,27 +340,132 @@ jobs:
fi
cd target/${{ matrix.target }}/release
zip "../../../${PACKAGE_NAME}.zip" rustfs
# Determine the binary name based on platform
if [[ "${{ matrix.platform }}" == "windows" ]]; then
BINARY_NAME="rustfs.exe"
else
BINARY_NAME="rustfs"
fi
# Verify the binary exists before packaging
if [[ ! -f "$BINARY_NAME" ]]; then
echo "❌ Binary $BINARY_NAME not found in $(pwd)"
if [[ "${{ matrix.platform }}" == "windows" ]]; then
dir
else
ls -la
fi
exit 1
fi
# Universal packaging function
package_zip() {
local src=$1
local dst=$2
if [[ "${{ matrix.platform }}" == "windows" ]]; then
# Windows uses PowerShell Compress-Archive
powershell -Command "Compress-Archive -Path '$src' -DestinationPath '$dst' -Force"
elif command -v zip &> /dev/null; then
# Unix systems use zip command
zip "$dst" "$src"
else
echo "❌ No zip utility available"
exit 1
fi
}
# Create the zip package
echo "Start packaging: $BINARY_NAME -> ../../../${PACKAGE_NAME}.zip"
package_zip "$BINARY_NAME" "../../../${PACKAGE_NAME}.zip"
cd ../../..
# Verify the package was created
if [[ -f "${PACKAGE_NAME}.zip" ]]; then
echo "✅ Package created successfully: ${PACKAGE_NAME}.zip"
if [[ "${{ matrix.platform }}" == "windows" ]]; then
dir
else
ls -lh ${PACKAGE_NAME}.zip
fi
else
echo "❌ Failed to create package: ${PACKAGE_NAME}.zip"
exit 1
fi
# Create latest version files right after the main package
LATEST_FILES=""
if [[ "$BUILD_TYPE" == "release" ]] || [[ "$BUILD_TYPE" == "prerelease" ]]; then
# Create latest version filename
# Convert from rustfs-linux-x86_64-musl-v1.0.0 to rustfs-linux-x86_64-musl-latest
LATEST_FILE="${PACKAGE_NAME%-v*}-latest.zip"
echo "🔄 Creating latest version: ${PACKAGE_NAME}.zip -> $LATEST_FILE"
cp "${PACKAGE_NAME}.zip" "$LATEST_FILE"
if [[ -f "$LATEST_FILE" ]]; then
echo "✅ Latest version created: $LATEST_FILE"
LATEST_FILES="$LATEST_FILE"
fi
elif [[ "$BUILD_TYPE" == "development" ]]; then
# Development builds (only main branch triggers development builds)
# Create main-latest version filename
# Convert from rustfs-linux-x86_64-dev-abc123 to rustfs-linux-x86_64-main-latest
MAIN_LATEST_FILE="${PACKAGE_NAME%-dev-*}-main-latest.zip"
echo "🔄 Creating main-latest version: ${PACKAGE_NAME}.zip -> $MAIN_LATEST_FILE"
cp "${PACKAGE_NAME}.zip" "$MAIN_LATEST_FILE"
if [[ -f "$MAIN_LATEST_FILE" ]]; then
echo "✅ Main-latest version created: $MAIN_LATEST_FILE"
LATEST_FILES="$MAIN_LATEST_FILE"
# Also create a generic main-latest for Docker builds (Linux only)
if [[ "${{ matrix.platform }}" == "linux" ]]; then
DOCKER_MAIN_LATEST_FILE="rustfs-linux-${ARCH_WITH_VARIANT}-main-latest.zip"
echo "🔄 Creating Docker main-latest version: ${PACKAGE_NAME}.zip -> $DOCKER_MAIN_LATEST_FILE"
cp "${PACKAGE_NAME}.zip" "$DOCKER_MAIN_LATEST_FILE"
if [[ -f "$DOCKER_MAIN_LATEST_FILE" ]]; then
echo "✅ Docker main-latest version created: $DOCKER_MAIN_LATEST_FILE"
LATEST_FILES="$LATEST_FILES $DOCKER_MAIN_LATEST_FILE"
fi
fi
fi
fi
echo "package_name=${PACKAGE_NAME}" >> $GITHUB_OUTPUT
echo "package_file=${PACKAGE_NAME}.zip" >> $GITHUB_OUTPUT
echo "Package created: ${PACKAGE_NAME}.zip"
echo "latest_files=${LATEST_FILES}" >> $GITHUB_OUTPUT
echo "build_type=${BUILD_TYPE}" >> $GITHUB_OUTPUT
echo "version=${VERSION}" >> $GITHUB_OUTPUT
- name: Upload artifacts
uses: actions/upload-artifact@v4
echo "📦 Package created: ${PACKAGE_NAME}.zip"
if [[ -n "$LATEST_FILES" ]]; then
echo "📦 Latest files created: $LATEST_FILES"
fi
echo "🔧 Build type: ${BUILD_TYPE}"
echo "📊 Version: ${VERSION}"
- name: Upload to GitHub artifacts
uses: actions/upload-artifact@v6
with:
name: ${{ steps.package.outputs.package_name }}
path: ${{ steps.package.outputs.package_file }}
path: "rustfs-*.zip"
retention-days: ${{ startsWith(github.ref, 'refs/tags/') && 30 || 7 }}
- name: Upload to Aliyun OSS
if: needs.build-check.outputs.build_type == 'release' && env.OSS_ACCESS_KEY_ID != ''
if: env.OSS_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
env:
OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
OSS_REGION: cn-beijing
OSS_ENDPOINT: https://oss-cn-beijing.aliyuncs.com
OSS_ENDPOINT: https://oss-accelerate.aliyuncs.com
shell: bash
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
# Install ossutil (platform-specific)
OSSUTIL_VERSION="2.1.1"
case "${{ matrix.platform }}" in
@@ -271,150 +501,357 @@ jobs:
chmod +x /usr/local/bin/ossutil
OSSUTIL_BIN=ossutil
;;
windows)
OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-windows-amd64.zip"
OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-windows-amd64"
curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
unzip "$OSSUTIL_ZIP"
mv "${OSSUTIL_DIR}/ossutil.exe" ./ossutil.exe
rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
OSSUTIL_BIN=./ossutil.exe
;;
esac
# Upload the package file directly to OSS
echo "Uploading ${{ steps.package.outputs.package_file }} to OSS..."
$OSSUTIL_BIN cp "${{ steps.package.outputs.package_file }}" oss://rustfs-artifacts/artifacts/rustfs/ --force
# Create latest.json (only for the first Linux build to avoid duplication)
if [[ "${{ matrix.target }}" == "x86_64-unknown-linux-musl" ]]; then
VERSION="${GITHUB_REF#refs/tags/v}"
echo "{\"version\":\"${VERSION}\",\"release_date\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}" > latest.json
$OSSUTIL_BIN cp latest.json oss://rustfs-version/latest.json --force
# Determine upload path based on build type
if [[ "$BUILD_TYPE" == "development" ]]; then
OSS_PATH="oss://rustfs-artifacts/artifacts/rustfs/dev/"
echo "📤 Uploading development build to OSS dev directory"
else
OSS_PATH="oss://rustfs-artifacts/artifacts/rustfs/release/"
echo "📤 Uploading release build to OSS release directory"
fi
# Release management
release:
name: GitHub Release
needs: [build-check, build-rustfs]
if: always() && needs.build-check.outputs.build_type == 'release'
runs-on: ubuntu-latest
# Upload all rustfs zip files to OSS using glob pattern
echo "📤 Uploading all rustfs-*.zip files to $OSS_PATH..."
for zip_file in rustfs-*.zip; do
if [[ -f "$zip_file" ]]; then
echo "Uploading: $zip_file to $OSS_PATH..."
$OSSUTIL_BIN cp "$zip_file" "$OSS_PATH" --force
echo "✅ Uploaded: $zip_file"
fi
done
echo "✅ Upload completed successfully"
# Build summary
build-summary:
name: Build Summary
needs: [ build-check, build-rustfs ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubicloud-standard-2
steps:
- name: Build completion summary
shell: bash
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
echo "🎉 Build completed successfully!"
echo "📦 Build type: $BUILD_TYPE"
echo "🔢 Version: $VERSION"
echo ""
# Check build status
BUILD_STATUS="${{ needs.build-rustfs.result }}"
echo "📊 Build Results:"
echo " 📦 All platforms: $BUILD_STATUS"
echo ""
case "$BUILD_TYPE" in
"development")
echo "🛠️ Development build artifacts have been uploaded to OSS dev directory"
echo "⚠️ This is a development build - not suitable for production use"
;;
"release")
echo "🚀 Release build artifacts have been uploaded to OSS release directory"
echo "✅ This build is ready for production use"
echo "🏷️ GitHub Release will be created in this workflow"
;;
"prerelease")
echo "🧪 Prerelease build artifacts have been uploaded to OSS release directory"
echo "⚠️ This is a prerelease build - use with caution"
echo "🏷️ GitHub Release will be created in this workflow"
;;
esac
echo ""
echo "🐳 Docker Images:"
if [[ "${{ github.event.inputs.build_docker }}" == "false" ]]; then
echo "⏭️ Docker image build was skipped (binary only build)"
elif [[ "$BUILD_STATUS" == "success" ]]; then
echo "🔄 Docker images will be built and pushed automatically via workflow_run event"
else
echo "❌ Docker image build will be skipped due to build failure"
fi
# Create GitHub Release (only for tag pushes)
create-release:
name: Create GitHub Release
needs: [ build-check, build-rustfs ]
if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
runs-on: ubicloud-standard-2
permissions:
contents: write
outputs:
release_id: ${{ steps.create.outputs.release_id }}
release_url: ${{ steps.create.outputs.release_url }}
steps:
- name: Checkout repository
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Create GitHub Release
id: create
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
VERSION="${{ needs.build-check.outputs.version }}"
IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
# Determine release type for title
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
if [[ "$TAG" == *"alpha"* ]]; then
RELEASE_TYPE="alpha"
elif [[ "$TAG" == *"beta"* ]]; then
RELEASE_TYPE="beta"
elif [[ "$TAG" == *"rc"* ]]; then
RELEASE_TYPE="rc"
else
RELEASE_TYPE="prerelease"
fi
else
RELEASE_TYPE="release"
fi
# Check if release already exists
if gh release view "$TAG" >/dev/null 2>&1; then
echo "Release $TAG already exists"
RELEASE_ID=$(gh release view "$TAG" --json databaseId --jq '.databaseId')
RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
else
# Get release notes from tag message
RELEASE_NOTES=$(git tag -l --format='%(contents)' "${TAG}")
if [[ -z "$RELEASE_NOTES" || "$RELEASE_NOTES" =~ ^[[:space:]]*$ ]]; then
if [[ "$IS_PRERELEASE" == "true" ]]; then
RELEASE_NOTES="Pre-release ${VERSION} (${RELEASE_TYPE})"
else
RELEASE_NOTES="Release ${VERSION}"
fi
fi
# Create release title
if [[ "$IS_PRERELEASE" == "true" ]]; then
TITLE="RustFS $VERSION (${RELEASE_TYPE})"
else
TITLE="RustFS $VERSION"
fi
# Create the release
PRERELEASE_FLAG=""
if [[ "$IS_PRERELEASE" == "true" ]]; then
PRERELEASE_FLAG="--prerelease"
fi
gh release create "$TAG" \
--title "$TITLE" \
--notes "$RELEASE_NOTES" \
$PRERELEASE_FLAG \
--draft
RELEASE_ID=$(gh release view "$TAG" --json databaseId --jq '.databaseId')
RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
fi
echo "release_id=$RELEASE_ID" >> $GITHUB_OUTPUT
echo "release_url=$RELEASE_URL" >> $GITHUB_OUTPUT
echo "Created release: $RELEASE_URL"
# Prepare and upload release assets
upload-release-assets:
name: Upload Release Assets
needs: [ build-check, build-rustfs, create-release ]
if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
runs-on: ubicloud-standard-2
permissions:
contents: write
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Download all build artifacts
uses: actions/download-artifact@v7
with:
path: ./artifacts
pattern: rustfs-*
merge-multiple: true
- name: Prepare release assets
id: prepare
shell: bash
run: |
VERSION="${{ needs.build-check.outputs.version }}"
TAG="${{ needs.build-check.outputs.version }}"
mkdir -p ./release-assets
# Copy and verify artifacts (including latest files created during build)
ASSETS_COUNT=0
for file in ./artifacts/*.zip; do
if [[ -f "$file" ]]; then
cp "$file" ./release-assets/
ASSETS_COUNT=$((ASSETS_COUNT + 1))
fi
done
if [[ $ASSETS_COUNT -eq 0 ]]; then
echo "❌ No artifacts found!"
exit 1
fi
cd ./release-assets
# Generate checksums for all files (including latest versions)
if ls *.zip >/dev/null 2>&1; then
sha256sum *.zip > SHA256SUMS
sha512sum *.zip > SHA512SUMS
fi
# Create signature placeholder files
for file in *.zip; do
echo "# Signature for $file" > "${file}.asc"
echo "# GPG signature will be added in future versions" >> "${file}.asc"
done
echo "📦 Prepared assets:"
ls -la
echo "🔢 Total asset count: $ASSETS_COUNT"
- name: Upload to GitHub Release
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
cd ./release-assets
# Upload all files
for file in *; do
if [[ -f "$file" ]]; then
echo "📤 Uploading $file..."
gh release upload "$TAG" "$file" --clobber
fi
done
echo "✅ All assets uploaded successfully"
# Update latest.json for stable releases only
update-latest-version:
name: Update Latest Version
needs: [ build-check, upload-release-assets ]
if: startsWith(github.ref, 'refs/tags/')
runs-on: ubicloud-standard-2
steps:
- name: Update latest.json
env:
OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
OSS_REGION: cn-beijing
OSS_ENDPOINT: https://oss-cn-beijing.aliyuncs.com
shell: bash
run: |
if [[ -z "$OSS_ACCESS_KEY_ID" ]]; then
echo "⚠️ OSS credentials not available, skipping latest.json update"
exit 0
fi
VERSION="${{ needs.build-check.outputs.version }}"
TAG="${{ needs.build-check.outputs.version }}"
# Install ossutil
OSSUTIL_VERSION="2.1.1"
OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-linux-amd64.zip"
OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-linux-amd64"
curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
unzip "$OSSUTIL_ZIP"
mv "${OSSUTIL_DIR}/ossutil" /usr/local/bin/
rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
chmod +x /usr/local/bin/ossutil
# Create latest.json
cat > latest.json << EOF
{
"version": "${VERSION}",
"tag": "${TAG}",
"release_date": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
"release_type": "stable",
"download_url": "https://github.com/${{ github.repository }}/releases/tag/${TAG}"
}
EOF
# Upload to OSS
ossutil cp latest.json oss://rustfs-version/latest.json --force
echo "✅ Updated latest.json for stable release $VERSION"
# Publish release (remove draft status)
publish-release:
name: Publish Release
needs: [ build-check, create-release, upload-release-assets ]
if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
runs-on: ubicloud-standard-2
permissions:
contents: write
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
uses: actions/checkout@v6
- name: Download all artifacts
uses: actions/download-artifact@v4
with:
path: ./release-artifacts
- name: Prepare release assets
id: release_prep
run: |
VERSION="${GITHUB_REF#refs/tags/}"
VERSION_CLEAN="${VERSION#v}"
echo "version=${VERSION}" >> $GITHUB_OUTPUT
echo "version_clean=${VERSION_CLEAN}" >> $GITHUB_OUTPUT
# Organize artifacts
mkdir -p ./release-files
# Copy all artifacts (.zip files)
find ./release-artifacts -name "*.zip" -exec cp {} ./release-files/ \;
# Generate checksums for all files
cd ./release-files
if ls *.zip >/dev/null 2>&1; then
sha256sum *.zip >> SHA256SUMS
sha512sum *.zip >> SHA512SUMS
fi
cd ..
# Display what we're releasing
echo "=== Release Files ==="
ls -la ./release-files/
- name: Create GitHub Release
- name: Update release notes and publish
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
VERSION="${{ steps.release_prep.outputs.version }}"
VERSION_CLEAN="${{ steps.release_prep.outputs.version_clean }}"
TAG="${{ needs.build-check.outputs.version }}"
VERSION="${{ needs.build-check.outputs.version }}"
IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
# Check if release already exists
if gh release view "$VERSION" >/dev/null 2>&1; then
echo "Release $VERSION already exists, skipping creation"
# Determine release type
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
if [[ "$TAG" == *"alpha"* ]]; then
RELEASE_TYPE="alpha"
elif [[ "$TAG" == *"beta"* ]]; then
RELEASE_TYPE="beta"
elif [[ "$TAG" == *"rc"* ]]; then
RELEASE_TYPE="rc"
else
RELEASE_TYPE="prerelease"
fi
else
# Get release notes from tag message
RELEASE_NOTES=$(git tag -l --format='%(contents)' "${VERSION}")
if [[ -z "$RELEASE_NOTES" || "$RELEASE_NOTES" =~ ^[[:space:]]*$ ]]; then
RELEASE_NOTES="Release ${VERSION_CLEAN}"
fi
# Determine if this is a prerelease
PRERELEASE_FLAG=""
if [[ "$VERSION" == *"alpha"* ]] || [[ "$VERSION" == *"beta"* ]] || [[ "$VERSION" == *"rc"* ]]; then
PRERELEASE_FLAG="--prerelease"
fi
# Create the release only if it doesn't exist
gh release create "$VERSION" \
--title "RustFS $VERSION_CLEAN" \
--notes "$RELEASE_NOTES" \
$PRERELEASE_FLAG
RELEASE_TYPE="release"
fi
- name: Upload release assets
env:
GH_TOKEN: ${{ github.token }}
run: |
VERSION="${{ steps.release_prep.outputs.version }}"
cd ./release-files
# Upload all binary files
for file in *.zip; do
if [[ -f "$file" ]]; then
echo "Uploading $file..."
gh release upload "$VERSION" "$file" --clobber
# Get original release notes from tag
ORIGINAL_NOTES=$(git tag -l --format='%(contents)' "${TAG}")
if [[ -z "$ORIGINAL_NOTES" || "$ORIGINAL_NOTES" =~ ^[[:space:]]*$ ]]; then
if [[ "$IS_PRERELEASE" == "true" ]]; then
ORIGINAL_NOTES="Pre-release ${VERSION} (${RELEASE_TYPE})"
else
ORIGINAL_NOTES="Release ${VERSION}"
fi
done
# Upload checksum files
if [[ -f "SHA256SUMS" ]]; then
echo "Uploading SHA256SUMS..."
gh release upload "$VERSION" "SHA256SUMS" --clobber
fi
if [[ -f "SHA512SUMS" ]]; then
echo "Uploading SHA512SUMS..."
gh release upload "$VERSION" "SHA512SUMS" --clobber
fi
# Publish the release (remove draft status)
gh release edit "$TAG" --draft=false
- name: Update release notes
env:
GH_TOKEN: ${{ github.token }}
run: |
VERSION="${{ steps.release_prep.outputs.version }}"
VERSION_CLEAN="${{ steps.release_prep.outputs.version_clean }}"
# Check if release already has custom notes (not auto-generated)
EXISTING_NOTES=$(gh release view "$VERSION" --json body --jq '.body' 2>/dev/null || echo "")
# Only update if release notes are empty or auto-generated
if [[ -z "$EXISTING_NOTES" ]] || [[ "$EXISTING_NOTES" == *"Release ${VERSION_CLEAN}"* ]]; then
echo "Updating release notes for $VERSION"
# Get original release notes from tag
ORIGINAL_NOTES=$(git tag -l --format='%(contents)' "${VERSION}")
if [[ -z "$ORIGINAL_NOTES" || "$ORIGINAL_NOTES" =~ ^[[:space:]]*$ ]]; then
ORIGINAL_NOTES="Release ${VERSION_CLEAN}"
fi
# Use external template file and substitute variables
sed -e "s/\${VERSION}/$VERSION/g" \
-e "s/\${VERSION_CLEAN}/$VERSION_CLEAN/g" \
-e "s/\${ORIGINAL_NOTES}/$(echo "$ORIGINAL_NOTES" | sed 's/[[\.*^$()+?{|]/\\&/g')/g" \
.github/workflows/release-notes-template.md > enhanced_notes.md
# Update the release with enhanced notes
gh release edit "$VERSION" --notes-file enhanced_notes.md
else
echo "Release $VERSION already has custom notes, skipping update to preserve manual edits"
fi
echo "🎉 Released $TAG successfully!"
echo "📄 Release URL: ${{ needs.create-release.outputs.release_url }}"

View File

@@ -4,7 +4,7 @@
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
@@ -16,10 +16,9 @@ name: Continuous Integration
on:
push:
branches: [main]
branches: [ main ]
paths-ignore:
- "**.md"
- "**.txt"
- "docs/**"
- "deploy/**"
- "scripts/dev_*.sh"
@@ -36,10 +35,9 @@ on:
- ".github/workflows/audit.yml"
- ".github/workflows/performance.yml"
pull_request:
branches: [main]
branches: [ main ]
paths-ignore:
- "**.md"
- "**.txt"
- "docs/**"
- "deploy/**"
- "scripts/dev_*.sh"
@@ -59,17 +57,26 @@ on:
- cron: "0 0 * * 0" # Weekly on Sunday at midnight UTC
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
CARGO_BUILD_JOBS: 2
jobs:
skip-check:
name: Skip Duplicate Actions
permissions:
actions: write
contents: read
runs-on: ubuntu-latest
runs-on: ubicloud-standard-2
outputs:
should_skip: ${{ steps.skip_check.outputs.should_skip }}
steps:
@@ -80,18 +87,28 @@ jobs:
concurrent_skipping: "same_content_newer"
cancel_others: true
paths_ignore: '["*.md", "docs/**", "deploy/**"]'
# Never skip release events and tag pushes
do_not_skip: '["workflow_dispatch", "schedule", "merge_group", "release", "push"]'
typos:
name: Typos
needs: skip-check
if: needs.skip-check.outputs.should_skip != 'true'
runs-on: ubicloud-standard-2
steps:
- uses: actions/checkout@v6
- uses: dtolnay/rust-toolchain@stable
- name: Typos check with custom config file
uses: crate-ci/typos@master
test-and-lint:
name: Test and Lint
needs: skip-check
if: needs.skip-check.outputs.should_skip != 'true'
runs-on: ubuntu-latest
runs-on: ubicloud-standard-4
timeout-minutes: 60
steps:
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@v6
- name: Setup Rust environment
uses: ./.github/actions/setup
@@ -112,35 +129,73 @@ jobs:
- name: Run clippy lints
run: cargo clippy --all-targets --all-features -- -D warnings
e2e-tests:
name: End-to-End Tests
- name: Check layered dependencies
run: ./scripts/check_layer_dependencies.sh
build-rustfs-debug-binary:
name: Build RustFS Debug Binary
needs: skip-check
if: needs.skip-check.outputs.should_skip != 'true'
runs-on: ubuntu-latest
runs-on: ubicloud-standard-4
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@v6
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-e2e-${{ hashFiles('**/Cargo.lock') }}
cache-shared-key: ci-rustfs-debug-binary-${{ hashFiles('**/Cargo.lock') }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Build debug binary
run: |
touch rustfs/build.rs
cargo build -p rustfs --bins --jobs 2
- name: Upload debug binary
uses: actions/upload-artifact@v6
with:
name: rustfs-debug-binary
path: target/debug/rustfs
if-no-files-found: error
retention-days: 1
e2e-tests:
name: End-to-End Tests
needs: [ skip-check, build-rustfs-debug-binary ]
if: needs.skip-check.outputs.should_skip != 'true'
runs-on: ubicloud-standard-2
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Clean up previous test run
run: |
rm -rf /tmp/rustfs
rm -f /tmp/rustfs.log
- name: Download debug binary
uses: actions/download-artifact@v7
with:
name: rustfs-debug-binary
path: target/debug
- name: Make binary executable
run: chmod +x ./target/debug/rustfs
- name: Setup Rust toolchain for s3s-e2e installation
uses: dtolnay/rust-toolchain@stable
- name: Install s3s-e2e test tool
uses: taiki-e/cache-cargo-install-action@v2
with:
tool: s3s-e2e
git: https://github.com/Nugine/s3s.git
rev: b7714bfaa17ddfa9b23ea01774a1e7bbdbfc2ca3
- name: Build debug binary
run: |
touch rustfs/build.rs
cargo build -p rustfs --bins
git: https://github.com/s3s-project/s3s.git
rev: 4a04a670cf41274d9be9ab65dc36f4aa3f92fbad
- name: Run end-to-end tests
run: |
@@ -149,8 +204,44 @@ jobs:
- name: Upload test logs
if: failure()
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v6
with:
name: e2e-test-logs-${{ github.run_number }}
path: /tmp/rustfs.log
retention-days: 3
s3-implemented-tests:
name: S3 Implemented Tests
needs: [ skip-check, build-rustfs-debug-binary ]
if: needs.skip-check.outputs.should_skip != 'true'
runs-on: ubicloud-standard-4
timeout-minutes: 60
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Download debug binary
uses: actions/download-artifact@v7
with:
name: rustfs-debug-binary
path: target/debug
- name: Make binary executable
run: chmod +x ./target/debug/rustfs
- name: Run implemented s3-tests
run: |
DEPLOY_MODE=binary \
RUSTFS_BINARY=./target/debug/rustfs \
TEST_MODE=single \
MAXFAIL=1 \
./scripts/s3-tests/run.sh
- name: Upload s3 test artifacts
if: always()
uses: actions/upload-artifact@v6
with:
name: s3tests-implemented-${{ github.run_number }}
path: artifacts/s3tests-single/**
if-no-files-found: ignore
retention-days: 3

View File

@@ -12,42 +12,33 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Docker Images Workflow
#
# This workflow builds Docker images using pre-built binaries from the build workflow.
#
# Trigger Types:
# 1. workflow_run: Automatically triggered when "Build and Release" workflow completes
# 2. workflow_dispatch: Manual trigger for standalone Docker builds
#
# Key Features:
# - Only triggers when Linux builds (x86_64 + aarch64) are successful
# - Independent of macOS/Windows build status
# - Uses workflow_run event for precise control
# - Only builds Docker images for releases and prereleases (development builds are skipped)
name: Docker Images
# Permissions needed for workflow_run event and Docker registry access
permissions:
contents: read
packages: write
on:
push:
tags: ["*"]
branches: [main]
paths-ignore:
- "**.md"
- "**.txt"
- ".github/**"
- "docs/**"
- "deploy/**"
- "scripts/dev_*.sh"
- "LICENSE*"
- "README*"
- "**/*.png"
- "**/*.jpg"
- "**/*.svg"
- ".gitignore"
- ".dockerignore"
pull_request:
branches: [main]
paths-ignore:
- "**.md"
- "**.txt"
- ".github/**"
- "docs/**"
- "deploy/**"
- "scripts/dev_*.sh"
- "LICENSE*"
- "README*"
- "**/*.png"
- "**/*.jpg"
- "**/*.svg"
- ".gitignore"
- ".dockerignore"
# Automatically triggered when build workflow completes
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
# Manual trigger with same parameters for consistency
workflow_dispatch:
inputs:
push_images:
@@ -55,156 +46,403 @@ on:
required: false
default: true
type: boolean
version:
description: "Version to build (latest for stable release, or specific version like v1.0.0, v1.0.0-alpha1)"
required: false
default: "latest"
type: string
force_rebuild:
description: "Force rebuild even if binary exists (useful for testing)"
required: false
default: false
type: boolean
env:
CONCLUSION: ${{ github.event.workflow_run.conclusion }}
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
TRIGGERING_EVENT: ${{ github.event.workflow_run.event }}
DOCKERHUB_USERNAME: rustfs
CARGO_TERM_COLOR: always
REGISTRY_DOCKERHUB: rustfs/rustfs
REGISTRY_GHCR: ghcr.io/${{ github.repository }}
REGISTRY_QUAY: quay.io/${{ secrets.QUAY_USERNAME }}/rustfs
DOCKER_PLATFORMS: linux/amd64,linux/arm64
jobs:
# Check if we should build
# Check if we should build Docker images
build-check:
name: Build Check
runs-on: ubuntu-latest
name: Docker Build Check
runs-on: ubicloud-standard-2
outputs:
should_build: ${{ steps.check.outputs.should_build }}
should_push: ${{ steps.check.outputs.should_push }}
build_type: ${{ steps.check.outputs.build_type }}
version: ${{ steps.check.outputs.version }}
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
create_latest: ${{ steps.check.outputs.create_latest }}
steps:
- name: Checkout repository
uses: actions/checkout@v6
with:
fetch-depth: 0
# For workflow_run events, checkout the specific commit that triggered the workflow
ref: ${{ github.event.workflow_run.head_sha || github.sha }}
- name: Check build conditions
id: check
run: |
should_build=false
should_push=false
build_type="none"
version=""
short_sha=""
is_prerelease=false
create_latest=false
# Always build on workflow_dispatch or when changes detected
if [[ "${{ github.event_name }}" == "workflow_dispatch" ]] || \
[[ "${{ github.event_name }}" == "push" ]] || \
[[ "${{ github.event_name }}" == "pull_request" ]]; then
if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
# Triggered by build workflow completion
echo "🔗 Triggered by build workflow completion"
# Check if the triggering workflow was successful
# If the workflow succeeded, it means ALL builds (including Linux x86_64 and aarch64) succeeded
if [[ "$CONCLUSION" == "success" ]]; then
echo "✅ Build workflow succeeded, all builds including Linux are successful"
should_build=true
should_push=true
else
echo "❌ Build workflow failed (conclusion: $CONCLUSION), skipping Docker build"
should_build=false
fi
# Extract version info from commit message or use commit SHA
# Use Git to generate consistent short SHA (ensures uniqueness like build.yml)
short_sha=$(git rev-parse --short "$HEAD_SHA")
# Determine build type based on triggering workflow event and ref
triggering_event="$TRIGGERING_EVENT"
head_branch="$HEAD_BRANCH"
echo "🔍 Analyzing triggering workflow:"
echo " 📋 Event: $triggering_event"
echo " 🌿 Head branch: $head_branch"
echo " 📎 Head SHA: $HEAD_SHA"
# Check if this was triggered by a tag push
if [[ "$triggering_event" == "push" ]]; then
# For tag pushes, head_branch will be like "refs/tags/v1.0.0" or just "v1.0.0"
if [[ "$head_branch" == refs/tags/* ]]; then
# Extract tag name from refs/tags/TAG_NAME
tag_name="${head_branch#refs/tags/}"
version="$tag_name"
elif [[ "$head_branch" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+ ]]; then
# Direct tag name like "v1.0.0" or "1.0.0-alpha.1"
version="$head_branch"
elif [[ "$head_branch" == "main" ]]; then
# Regular branch push to main
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (main branch push)"
else
# Other branch push
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (branch: $head_branch)"
fi
# If we extracted a version (tag), determine release type
if [[ -n "$version" ]] && [[ "$version" != "dev-${short_sha}" ]]; then
# Remove 'v' prefix if present for consistent version format
if [[ "$version" == v* ]]; then
version="${version#v}"
fi
if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
build_type="prerelease"
is_prerelease=true
echo "🧪 Building Docker image for prerelease: $version"
else
build_type="release"
create_latest=true
echo "🚀 Building Docker image for release: $version"
fi
fi
else
# Non-push events
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (event: $triggering_event)"
fi
echo "🔄 Build triggered by workflow_run:"
echo " 📋 Conclusion: $CONCLUSION"
echo " 🌿 Branch: $HEAD_BRANCH"
echo " 📎 SHA: $HEAD_SHA"
echo " 🎯 Event: $TRIGGERING_EVENT"
elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
# Manual trigger
input_version="${{ github.event.inputs.version }}"
version="${input_version}"
should_push="${{ github.event.inputs.push_images }}"
should_build=true
fi
# Push only on main branch, tags, or manual trigger
if [[ "${{ github.ref }}" == "refs/heads/main" ]] || \
[[ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]] || \
[[ "${{ github.event.inputs.push_images }}" == "true" ]]; then
should_push=true
# Get short SHA
short_sha=$(git rev-parse --short HEAD)
echo "🎯 Manual Docker build triggered:"
echo " 📋 Requested version: $input_version"
echo " 🔧 Force rebuild: ${{ github.event.inputs.force_rebuild }}"
echo " 🚀 Push images: $should_push"
case "$input_version" in
"latest")
build_type="release"
create_latest=true
echo "🚀 Building with latest stable release version"
;;
# Prerelease versions (must match first, more specific)
v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
build_type="prerelease"
is_prerelease=true
echo "🧪 Building with prerelease version: $input_version"
;;
# Release versions (match after prereleases, more general)
v[0-9]*|[0-9]*.*.*)
build_type="release"
create_latest=true
echo "📦 Building with specific release version: $input_version"
;;
*)
# Invalid version for Docker build
should_build=false
echo "❌ Invalid version for Docker build: $input_version"
echo "⚠️ Only release versions (latest, v1.0.0, 1.0.0) and prereleases (v1.0.0-alpha1, 1.0.0-beta2) are supported"
;;
esac
fi
echo "should_build=$should_build" >> $GITHUB_OUTPUT
echo "should_push=$should_push" >> $GITHUB_OUTPUT
echo "Build: $should_build, Push: $should_push"
echo "build_type=$build_type" >> $GITHUB_OUTPUT
echo "version=$version" >> $GITHUB_OUTPUT
echo "short_sha=$short_sha" >> $GITHUB_OUTPUT
echo "is_prerelease=$is_prerelease" >> $GITHUB_OUTPUT
echo "create_latest=$create_latest" >> $GITHUB_OUTPUT
echo "🐳 Docker Build Summary:"
echo " - Should build: $should_build"
echo " - Should push: $should_push"
echo " - Build type: $build_type"
echo " - Version: $version"
echo " - Short SHA: $short_sha"
echo " - Is prerelease: $is_prerelease"
echo " - Create latest: $create_latest"
# Build multi-arch Docker images
# Strategy: Build images using pre-built binaries from dl.rustfs.com
# Supports both release and dev channel binaries based on build context
# Only runs when should_build is true (which includes workflow success check)
build-docker:
name: Build Docker Images
needs: build-check
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
runs-on: ubicloud-standard-2
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
variant:
- name: production
dockerfile: Dockerfile
platforms: linux/amd64,linux/arm64
- name: ubuntu
dockerfile: .docker/Dockerfile.ubuntu22.04
platforms: linux/amd64,linux/arm64
- name: alpine
dockerfile: .docker/Dockerfile.alpine
platforms: linux/amd64,linux/arm64
fail-fast: false
matrix:
include:
- variant: musl
file: Dockerfile
suffix: ""
- variant: glibc
file: Dockerfile.glibc
suffix: "-glibc"
steps:
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@v6
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ env.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- name: Login to Quay.io
uses: docker/login-action@v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Login to Docker Hub
if: needs.build-check.outputs.should_push == 'true' && secrets.DOCKERHUB_USERNAME != ''
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to GitHub Container Registry
if: needs.build-check.outputs.should_push == 'true'
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata
- name: Extract metadata and generate tags
id: meta
uses: docker/metadata-action@v5
with:
images: |
${{ env.REGISTRY_DOCKERHUB }}
${{ env.REGISTRY_GHCR }}
tags: |
type=ref,event=branch,suffix=-${{ matrix.variant.name }}
type=ref,event=pr,suffix=-${{ matrix.variant.name }}
type=semver,pattern={{version}},suffix=-${{ matrix.variant.name }}
type=semver,pattern={{major}}.{{minor}},suffix=-${{ matrix.variant.name }}
type=raw,value=latest,suffix=-${{ matrix.variant.name }},enable={{is_default_branch}}
flavor: |
latest=false
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
SHORT_SHA="${{ needs.build-check.outputs.short_sha }}"
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
VARIANT_SUFFIX="${{ matrix.suffix }}"
# Convert version format for Dockerfile compatibility
case "$VERSION" in
"latest")
# For stable latest, use RELEASE=latest + release CHANNEL
DOCKER_RELEASE="latest"
DOCKER_CHANNEL="release"
;;
v*)
# For versioned releases (v1.0.0), remove 'v' prefix for Dockerfile
DOCKER_RELEASE="${VERSION#v}"
DOCKER_CHANNEL="release"
;;
*)
# For other versions, pass as-is
DOCKER_RELEASE="${VERSION}"
DOCKER_CHANNEL="release"
;;
esac
echo "docker_release=$DOCKER_RELEASE" >> $GITHUB_OUTPUT
echo "docker_channel=$DOCKER_CHANNEL" >> $GITHUB_OUTPUT
echo "🐳 Docker build parameters:"
echo " - Original version: $VERSION"
echo " - Docker RELEASE: $DOCKER_RELEASE"
echo " - Docker CHANNEL: $DOCKER_CHANNEL"
# Generate tags based on build type
# Only support release and prerelease builds (no development builds)
TAG_BASE="${VERSION}${VARIANT_SUFFIX}"
TAGS="${{ env.REGISTRY_DOCKERHUB }}:$TAG_BASE,${{ env.REGISTRY_GHCR }}:$TAG_BASE,${{ env.REGISTRY_QUAY }}:$TAG_BASE"
# Add channel tags for prereleases and latest for stable
if [[ "$CREATE_LATEST" == "true" ]]; then
# TODO: Temporary change - the current alpha version will also create the latest tag
# After the version is stabilized, the logic here remains unchanged, but the upstream CREATE_LATEST setting needs to be restored.
# Stable release (and temporary alpha versions)
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:latest${VARIANT_SUFFIX}"
elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
# Prerelease channel tags (alpha, beta, rc)
if [[ "$VERSION" == *"alpha"* ]]; then
CHANNEL="alpha"
elif [[ "$VERSION" == *"beta"* ]]; then
CHANNEL="beta"
elif [[ "$VERSION" == *"rc"* ]]; then
CHANNEL="rc"
fi
if [[ -n "$CHANNEL" ]]; then
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:${CHANNEL}${VARIANT_SUFFIX}"
fi
fi
# Output tags
echo "tags=$TAGS" >> $GITHUB_OUTPUT
# Generate labels
LABELS="org.opencontainers.image.title=RustFS"
LABELS="$LABELS,org.opencontainers.image.description=RustFS distributed object storage system"
LABELS="$LABELS,org.opencontainers.image.version=$VERSION"
LABELS="$LABELS,org.opencontainers.image.revision=${{ github.sha }}"
LABELS="$LABELS,org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}"
LABELS="$LABELS,org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
LABELS="$LABELS,org.opencontainers.image.build-type=$BUILD_TYPE"
echo "labels=$LABELS" >> $GITHUB_OUTPUT
echo "🐳 Generated Docker tags:"
echo "$TAGS" | tr ',' '\n' | sed 's/^/ - /'
echo "📋 Build type: $BUILD_TYPE"
echo "🔖 Version: $VERSION"
- name: Build and push Docker image
uses: docker/build-push-action@v5
uses: docker/build-push-action@v6
with:
context: .
file: ${{ matrix.variant.dockerfile }}
platforms: ${{ matrix.variant.platforms }}
file: ${{ matrix.file }}
platforms: ${{ env.DOCKER_PLATFORMS }}
push: ${{ needs.build-check.outputs.should_push == 'true' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha,scope=docker-${{ matrix.variant.name }}
cache-to: type=gha,mode=max,scope=docker-${{ matrix.variant.name }}
cache-from: |
type=gha,scope=docker-${{ matrix.variant }}
cache-to: |
type=gha,mode=max,scope=docker-${{ matrix.variant }}
build-args: |
BUILDTIME=${{ fromJSON(steps.meta.outputs.json).labels['org.opencontainers.image.created'] }}
VERSION=${{ fromJSON(steps.meta.outputs.json).labels['org.opencontainers.image.version'] }}
REVISION=${{ fromJSON(steps.meta.outputs.json).labels['org.opencontainers.image.revision'] }}
BUILDTIME=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
VERSION=${{ needs.build-check.outputs.version }}
BUILD_TYPE=${{ needs.build-check.outputs.build_type }}
REVISION=${{ github.sha }}
RELEASE=${{ steps.meta.outputs.docker_release }}
CHANNEL=${{ steps.meta.outputs.docker_channel }}
BUILDKIT_INLINE_CACHE=1
# Enable advanced BuildKit features for better performance
provenance: false
sbom: false
# Add retry mechanism by splitting the build process
no-cache: false
pull: true
# Create manifest for main production image
create-manifest:
name: Create Manifest
needs: [build-check, build-docker]
if: needs.build-check.outputs.should_push == 'true' && startsWith(github.ref, 'refs/tags/')
runs-on: ubuntu-latest
# Note: Manifest creation is no longer needed as we only build one variant
# Multi-arch manifests are automatically created by docker/build-push-action
# Docker build summary
docker-summary:
name: Docker Build Summary
needs: [ build-check, build-docker ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubicloud-standard-2
steps:
- name: Login to Docker Hub
if: secrets.DOCKERHUB_USERNAME != ''
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Create and push manifest
- name: Docker build completion summary
run: |
VERSION=${GITHUB_REF#refs/tags/}
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
# Create main image tag (without variant suffix)
if [[ -n "${{ secrets.DOCKERHUB_USERNAME }}" ]]; then
docker buildx imagetools create \
-t ${{ env.REGISTRY_DOCKERHUB }}:${VERSION} \
-t ${{ env.REGISTRY_DOCKERHUB }}:latest \
${{ env.REGISTRY_DOCKERHUB }}:${VERSION}-production
fi
echo "🐳 Docker build completed successfully!"
echo "📦 Build type: $BUILD_TYPE"
echo "🔢 Version: $VERSION"
echo "🚀 Strategy: Images using pre-built binaries (release channel only)"
echo ""
docker buildx imagetools create \
-t ${{ env.REGISTRY_GHCR }}:${VERSION} \
-t ${{ env.REGISTRY_GHCR }}:latest \
${{ env.REGISTRY_GHCR }}:${VERSION}-production
case "$BUILD_TYPE" in
"release")
echo "🚀 Release Docker image has been built with ${VERSION} tags"
echo "✅ This image is ready for production use"
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been created for stable release"
fi
;;
"prerelease")
echo "🧪 Prerelease Docker image has been built with ${VERSION} tags"
echo "⚠️ This is a prerelease image - use with caution"
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been explicitly created for prerelease"
else
echo "🚫 Latest tag NOT created for prerelease"
fi
;;
*)
echo "❌ Unexpected build type: $BUILD_TYPE"
;;
esac

443
.github/workflows/e2e-s3tests.yml vendored Normal file
View File

@@ -0,0 +1,443 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: e2e-s3tests
on:
workflow_dispatch:
inputs:
test-mode:
description: "Test mode to run"
required: true
type: choice
default: "single"
options:
- single
- multi
xdist:
description: "Enable pytest-xdist (parallel). '0' to disable."
required: false
default: "0"
maxfail:
description: "Stop after N failures (debug friendly)"
required: false
default: "1"
markexpr:
description: "pytest -m expression (feature filters)"
required: false
default: "not lifecycle and not versioning and not s3website and not bucket_logging and not encryption"
env:
# main user
S3_ACCESS_KEY: rustfsadmin
S3_SECRET_KEY: rustfsadmin
# alt user (must be different from main for many s3-tests)
S3_ALT_ACCESS_KEY: rustfsalt
S3_ALT_SECRET_KEY: rustfsalt
S3_REGION: us-east-1
RUST_LOG: info
PLATFORM: linux/amd64
BUILDX_CACHE_SCOPE: rustfs-e2e-s3tests-source
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event.inputs['test-mode'] || 'single' }}
cancel-in-progress: true
defaults:
run:
shell: bash
jobs:
s3tests-single:
if: github.event.inputs['test-mode'] == 'single'
runs-on: ubicloud-standard-2
timeout-minutes: 120
steps:
- uses: actions/checkout@v6
- name: Cache pip downloads
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-e2e-s3tests-${{ hashFiles('.github/workflows/e2e-s3tests.yml') }}
restore-keys: |
${{ runner.os }}-pip-e2e-s3tests-
- name: Install Python tools
run: |
python3 -m pip install --user --upgrade pip awscurl tox
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Enable buildx
uses: docker/setup-buildx-action@v3
- name: Build RustFS image (source, cached)
run: |
DOCKER_BUILDKIT=1 docker buildx build --load \
--platform ${PLATFORM} \
--cache-from type=gha,scope=${BUILDX_CACHE_SCOPE} \
--cache-to type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE} \
-t rustfs-ci \
-f Dockerfile.source .
- name: Create network
run: docker network inspect rustfs-net >/dev/null 2>&1 || docker network create rustfs-net
- name: Remove existing rustfs-single (if any)
run: docker rm -f rustfs-single >/dev/null 2>&1 || true
- name: Start single RustFS
run: |
docker run -d --name rustfs-single \
--network rustfs-net \
-p 9000:9000 \
-e RUSTFS_ADDRESS=0.0.0.0:9000 \
-e RUSTFS_ACCESS_KEY=$S3_ACCESS_KEY \
-e RUSTFS_SECRET_KEY=$S3_SECRET_KEY \
-e RUSTFS_VOLUMES="/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3" \
-v /tmp/rustfs-single:/data \
rustfs-ci
- name: Wait for RustFS ready
run: |
for i in {1..60}; do
if curl -sf http://127.0.0.1:9000/health >/dev/null 2>&1; then
echo "RustFS is ready"
exit 0
fi
if [ "$(docker inspect -f '{{.State.Running}}' rustfs-single 2>/dev/null)" != "true" ]; then
echo "RustFS container not running" >&2
docker logs rustfs-single || true
exit 1
fi
sleep 2
done
echo "Health check timed out" >&2
docker logs rustfs-single || true
exit 1
- name: Generate s3tests config
run: |
export S3_HOST=127.0.0.1
envsubst < .github/s3tests/s3tests.conf > s3tests.conf
- name: Provision s3-tests alt user (required by suite)
run: |
# Admin API requires AWS SigV4 signing. awscurl is used by RustFS codebase as well.
awscurl \
--service s3 \
--region "${S3_REGION}" \
--access_key "${S3_ACCESS_KEY}" \
--secret_key "${S3_SECRET_KEY}" \
-X PUT \
-H 'Content-Type: application/json' \
-d '{"secretKey":"'"${S3_ALT_SECRET_KEY}"'","status":"enabled","policy":"readwrite"}' \
"http://127.0.0.1:9000/rustfs/admin/v3/add-user?accessKey=${S3_ALT_ACCESS_KEY}"
# Explicitly attach built-in policy via policy mapping.
# s3-tests relies on alt client being able to ListBuckets during setup cleanup.
awscurl \
--service s3 \
--region "${S3_REGION}" \
--access_key "${S3_ACCESS_KEY}" \
--secret_key "${S3_SECRET_KEY}" \
-X PUT \
"http://127.0.0.1:9000/rustfs/admin/v3/set-user-or-group-policy?policyName=readwrite&userOrGroup=${S3_ALT_ACCESS_KEY}&isGroup=false"
# Sanity check: alt user can list buckets (should not be AccessDenied).
awscurl \
--service s3 \
--region "${S3_REGION}" \
--access_key "${S3_ALT_ACCESS_KEY}" \
--secret_key "${S3_ALT_SECRET_KEY}" \
-X GET \
"http://127.0.0.1:9000/" >/dev/null
- name: Prepare s3-tests
run: |
git clone --depth 1 https://github.com/ceph/s3-tests.git s3-tests
- name: Run ceph s3-tests (debug friendly)
run: |
export PATH="$HOME/.local/bin:$PATH"
mkdir -p artifacts/s3tests-single
cd s3-tests
set -o pipefail
MAXFAIL="${{ github.event.inputs.maxfail }}"
if [ -z "$MAXFAIL" ]; then MAXFAIL="1"; fi
MARKEXPR="${{ github.event.inputs.markexpr }}"
if [ -z "$MARKEXPR" ]; then MARKEXPR="not lifecycle and not versioning and not s3website and not bucket_logging and not encryption"; fi
XDIST="${{ github.event.inputs.xdist }}"
if [ -z "$XDIST" ]; then XDIST="0"; fi
XDIST_ARGS=""
if [ "$XDIST" != "0" ]; then
# Add pytest-xdist to requirements.txt so tox installs it inside
# its virtualenv. Installing outside tox does NOT work.
echo "pytest-xdist" >> requirements.txt
XDIST_ARGS="-n $XDIST --dist=loadgroup"
fi
# Run tests from s3tests/functional (boto2+boto3 combined directory).
S3TEST_CONF=${GITHUB_WORKSPACE}/s3tests.conf \
tox -- \
-vv -ra --showlocals --tb=long \
--maxfail="$MAXFAIL" \
--junitxml=${GITHUB_WORKSPACE}/artifacts/s3tests-single/junit.xml \
$XDIST_ARGS \
s3tests/functional/test_s3.py \
-m "$MARKEXPR" \
2>&1 | tee ${GITHUB_WORKSPACE}/artifacts/s3tests-single/pytest.log
- name: Collect RustFS logs
if: always()
run: |
mkdir -p artifacts/rustfs-single
docker logs rustfs-single > artifacts/rustfs-single/rustfs.log 2>&1 || true
docker inspect rustfs-single > artifacts/rustfs-single/inspect.json || true
- name: Upload artifacts
if: always() && env.ACT != 'true'
uses: actions/upload-artifact@v6
with:
name: s3tests-single
path: artifacts/**
s3tests-multi:
if: github.event_name == 'workflow_dispatch' && github.event.inputs['test-mode'] == 'multi'
runs-on: ubicloud-standard-2
timeout-minutes: 150
steps:
- uses: actions/checkout@v6
- name: Cache pip downloads
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-e2e-s3tests-${{ hashFiles('.github/workflows/e2e-s3tests.yml') }}
restore-keys: |
${{ runner.os }}-pip-e2e-s3tests-
- name: Install Python tools
run: |
python3 -m pip install --user --upgrade pip awscurl tox
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Enable buildx
uses: docker/setup-buildx-action@v3
- name: Build RustFS image (source, cached)
run: |
DOCKER_BUILDKIT=1 docker buildx build --load \
--platform ${PLATFORM} \
--cache-from type=gha,scope=${BUILDX_CACHE_SCOPE} \
--cache-to type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE} \
-t rustfs-ci \
-f Dockerfile.source .
- name: Prepare cluster compose
run: |
cat > compose.yml <<'EOF'
services:
rustfs1:
image: rustfs-ci
hostname: rustfs1
networks: [rustfs-net]
environment:
RUSTFS_ADDRESS: "0.0.0.0:9000"
RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
volumes:
- rustfs1-data:/data
rustfs2:
image: rustfs-ci
hostname: rustfs2
networks: [rustfs-net]
environment:
RUSTFS_ADDRESS: "0.0.0.0:9000"
RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
volumes:
- rustfs2-data:/data
rustfs3:
image: rustfs-ci
hostname: rustfs3
networks: [rustfs-net]
environment:
RUSTFS_ADDRESS: "0.0.0.0:9000"
RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
volumes:
- rustfs3-data:/data
rustfs4:
image: rustfs-ci
hostname: rustfs4
networks: [rustfs-net]
environment:
RUSTFS_ADDRESS: "0.0.0.0:9000"
RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
volumes:
- rustfs4-data:/data
lb:
image: haproxy:2.9
hostname: lb
networks: [rustfs-net]
ports:
- "9000:9000"
volumes:
- ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
networks:
rustfs-net:
name: rustfs-net
volumes:
rustfs1-data:
rustfs2-data:
rustfs3-data:
rustfs4-data:
EOF
cat > haproxy.cfg <<'EOF'
defaults
mode http
timeout connect 5s
timeout client 30s
timeout server 30s
frontend fe_s3
bind *:9000
default_backend be_s3
backend be_s3
balance roundrobin
server s1 rustfs1:9000 check
server s2 rustfs2:9000 check
server s3 rustfs3:9000 check
server s4 rustfs4:9000 check
EOF
- name: Launch cluster
run: docker compose -f compose.yml up -d
- name: Wait for LB ready
run: |
for i in {1..90}; do
if curl -sf http://127.0.0.1:9000/health >/dev/null 2>&1; then
echo "Load balancer is ready"
exit 0
fi
sleep 2
done
echo "LB or backend not ready" >&2
docker compose -f compose.yml logs --tail=200 || true
exit 1
- name: Generate s3tests config
run: |
export S3_HOST=127.0.0.1
envsubst < .github/s3tests/s3tests.conf > s3tests.conf
- name: Provision s3-tests alt user (required by suite)
run: |
awscurl \
--service s3 \
--region "${S3_REGION}" \
--access_key "${S3_ACCESS_KEY}" \
--secret_key "${S3_SECRET_KEY}" \
-X PUT \
-H 'Content-Type: application/json' \
-d '{"secretKey":"'"${S3_ALT_SECRET_KEY}"'","status":"enabled","policy":"readwrite"}' \
"http://127.0.0.1:9000/rustfs/admin/v3/add-user?accessKey=${S3_ALT_ACCESS_KEY}"
awscurl \
--service s3 \
--region "${S3_REGION}" \
--access_key "${S3_ACCESS_KEY}" \
--secret_key "${S3_SECRET_KEY}" \
-X PUT \
"http://127.0.0.1:9000/rustfs/admin/v3/set-user-or-group-policy?policyName=readwrite&userOrGroup=${S3_ALT_ACCESS_KEY}&isGroup=false"
awscurl \
--service s3 \
--region "${S3_REGION}" \
--access_key "${S3_ALT_ACCESS_KEY}" \
--secret_key "${S3_ALT_SECRET_KEY}" \
-X GET \
"http://127.0.0.1:9000/" >/dev/null
- name: Prepare s3-tests
run: |
git clone --depth 1 https://github.com/ceph/s3-tests.git s3-tests
- name: Run ceph s3-tests (multi, debug friendly)
run: |
export PATH="$HOME/.local/bin:$PATH"
mkdir -p artifacts/s3tests-multi
cd s3-tests
set -o pipefail
MAXFAIL="${{ github.event.inputs.maxfail }}"
if [ -z "$MAXFAIL" ]; then MAXFAIL="1"; fi
MARKEXPR="${{ github.event.inputs.markexpr }}"
if [ -z "$MARKEXPR" ]; then MARKEXPR="not lifecycle and not versioning and not s3website and not bucket_logging and not encryption"; fi
XDIST="${{ github.event.inputs.xdist }}"
if [ -z "$XDIST" ]; then XDIST="0"; fi
XDIST_ARGS=""
if [ "$XDIST" != "0" ]; then
# Add pytest-xdist to requirements.txt so tox installs it inside
# its virtualenv. Installing outside tox does NOT work.
echo "pytest-xdist" >> requirements.txt
XDIST_ARGS="-n $XDIST --dist=loadgroup"
fi
# Run tests from s3tests/functional (boto2+boto3 combined directory).
S3TEST_CONF=${GITHUB_WORKSPACE}/s3tests.conf \
tox -- \
-vv -ra --showlocals --tb=long \
--maxfail="$MAXFAIL" \
--junitxml=${GITHUB_WORKSPACE}/artifacts/s3tests-multi/junit.xml \
$XDIST_ARGS \
s3tests/functional/test_s3.py \
-m "$MARKEXPR" \
2>&1 | tee ${GITHUB_WORKSPACE}/artifacts/s3tests-multi/pytest.log
- name: Collect logs
if: always()
run: |
mkdir -p artifacts/cluster
docker compose -f compose.yml logs --no-color > artifacts/cluster/cluster.log 2>&1 || true
- name: Upload artifacts
if: always() && env.ACT != 'true'
uses: actions/upload-artifact@v6
with:
name: s3tests-multi
path: artifacts/**

94
.github/workflows/helm-package.yml vendored Normal file
View File

@@ -0,0 +1,94 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Publish helm chart to artifacthub
on:
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
permissions:
contents: read
env:
new_version: ${{ github.event.workflow_run.head_branch }}
jobs:
build-helm-package:
runs-on: ubicloud-standard-2
# Only run on successful builds triggered by tag pushes (version format: x.y.z or x.y.z-suffix)
if: |
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
contains(github.event.workflow_run.head_branch, '.')
steps:
- name: Checkout helm chart repo
uses: actions/checkout@v6
- name: Replace chart app version
run: |
set -e
set -x
old_version=$(grep "^appVersion:" helm/rustfs/Chart.yaml | awk '{print $2}')
sed -i "s/$old_version/$new_version/g" helm/rustfs/Chart.yaml
- name: Set up Helm
uses: azure/setup-helm@v4.3.0
- name: Package Helm Chart
run: |
cp helm/README.md helm/rustfs/
package_version=$(echo $new_version | awk -F '-' '{print $2}' | awk -F '.' '{print $NF}')
helm package ./helm/rustfs --destination helm/rustfs/ --version "0.0.$package_version"
- name: Upload helm package as artifact
uses: actions/upload-artifact@v6
with:
name: helm-package
path: helm/rustfs/*.tgz
retention-days: 1
publish-helm-package:
runs-on: ubicloud-standard-2
needs: [ build-helm-package ]
steps:
- name: Checkout helm package repo
uses: actions/checkout@v6
with:
repository: rustfs/helm
token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
- name: Download helm package
uses: actions/download-artifact@v7
with:
name: helm-package
path: ./
- name: Set up helm
uses: azure/setup-helm@v4.3.0
- name: Generate index
run: helm repo index . --url https://charts.rustfs.com
- name: Push helm package and index file
run: |
git config --global user.name "${{ secrets.USERNAME }}"
git config --global user.email "${{ secrets.EMAIL_ADDRESS }}"
git status .
git add .
git commit -m "Update rustfs helm package with $new_version."
git push origin main

View File

@@ -1,18 +1,36 @@
name: 'issue-translator'
on:
issue_comment:
types: [created]
issues:
types: [opened]
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: "issue-translator"
on:
issue_comment:
types: [ created ]
issues:
types: [ opened ]
permissions:
contents: read
issues: write
jobs:
build:
runs-on: ubuntu-latest
runs-on: ubicloud-standard-4
steps:
- uses: usthe/issues-translate-action@v2.7
with:
IS_MODIFY_TITLE: false
# not require, default false, . Decide whether to modify the issue title
# if true, the robot account @Issues-translate-bot must have modification permissions, invite @Issues-translate-bot to your project or use your custom bot.
CUSTOM_BOT_NOTE: Bot detected the issue body's language is not English, translate it automatically.
CUSTOM_BOT_NOTE: Bot detected the issue body's language is not English, translate it automatically.
# not require. Customize the translation robot prefix message.

65
.github/workflows/nix-flake-update.yml vendored Normal file
View File

@@ -0,0 +1,65 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Update Nix Flake
on:
workflow_dispatch:
schedule:
- cron: '0 0 * * 0' # Weekly on Sundays
permissions:
contents: write
pull-requests: write
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
update-flake:
name: Update flake.lock
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Install Nix
uses: DeterminateSystems/determinate-nix-action@v3
- name: Check Nix flake inputs
uses: DeterminateSystems/flake-checker-action@v12
- name: Update flake.lock
id: update
uses: DeterminateSystems/update-flake-lock@main
with:
git-author-name: heihutu
git-author-email: heihutu@gmail.com
git-committer-name: heihutu
git-committer-email: heihutu@gmail.com
pr-title: "chore(deps): update flake.lock"
pr-labels: |
dependencies
nix
automated
commit-msg: "chore(deps): update flake.lock"
pr-reviewers: houseme, overtrue, majinghe
token: ${{ secrets.FLAKE_UPDATE_TOKEN }}
- name: Log PR details
if: steps.update.outputs.pull-request-number
run: |
echo "Pull Request created: ${{ steps.update.outputs.pull-request-number }}"

84
.github/workflows/nix.yml vendored Normal file
View File

@@ -0,0 +1,84 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Nix CI
on:
push:
branches: [ "main" ]
paths:
- 'flake.nix'
- 'flake.lock'
- 'Cargo.toml'
- 'Cargo.lock'
- '.github/workflows/nix.yml'
pull_request:
branches: [ "main" ]
paths:
- 'flake.nix'
- 'flake.lock'
- 'Cargo.toml'
- 'Cargo.lock'
- '.github/workflows/nix.yml'
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
nix-validation:
name: Nix Build & Check
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Install Nix
uses: cachix/install-nix-action@v31
with:
github_access_token: ${{ secrets.GITHUB_TOKEN }}
extra_nix_config: |
experimental-features = nix-command flakes
- name: Setup Magic Nix Cache
uses: DeterminateSystems/magic-nix-cache-action@v13
- name: Setup Flake Checker
uses: DeterminateSystems/flake-checker-action@v12
- name: Verify Flake
run: |
echo "Checking flake structure and evaluation..."
nix flake show
nix flake check --print-build-logs
- name: Build RustFS
run: |
echo "Building the default package..."
nix build .#default --print-build-logs
- name: Test Binary
run: |
echo "Verifying the built binary..."
if [ -x "./result/bin/rustfs" ]; then
./result/bin/rustfs --help
echo "Binary verification successful."
else
echo "Error: Binary not found or not executable at ./result/bin/rustfs"
exit 1
fi

View File

@@ -16,12 +16,12 @@ name: Performance Testing
on:
push:
branches: [main]
branches: [ main ]
paths:
- '**/*.rs'
- '**/Cargo.toml'
- '**/Cargo.lock'
- '.github/workflows/performance.yml'
- "**/*.rs"
- "**/Cargo.toml"
- "**/Cargo.lock"
- ".github/workflows/performance.yml"
workflow_dispatch:
inputs:
profile_duration:
@@ -30,6 +30,9 @@ on:
default: "120"
type: string
permissions:
contents: read
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
@@ -37,11 +40,11 @@ env:
jobs:
performance-profile:
name: Performance Profiling
runs-on: ubuntu-latest
runs-on: ubicloud-standard-2
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@v6
- name: Setup Rust environment
uses: ./.github/actions/setup
@@ -73,12 +76,11 @@ jobs:
echo "RUSTFS_VOLUMES=./target/volume/test{0...4}" >> $GITHUB_ENV
echo "RUST_LOG=rustfs=info,ecstore=info,s3s=info,iam=info,rustfs-obs=info" >> $GITHUB_ENV
- name: Download static files
- name: Verify console static assets
run: |
curl -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" \
-o tempfile.zip --retry 3 --retry-delay 5
unzip -o tempfile.zip -d ./rustfs/static
rm tempfile.zip
# Console static assets are already embedded in the repository
echo "Console static assets size: $(du -sh rustfs/static/)"
echo "Console static assets are embedded via rust-embed, no external download needed"
- name: Build with profiling optimizations
run: |
@@ -105,7 +107,7 @@ jobs:
- name: Upload profile data
if: steps.profiling.outputs.profile_generated == 'true'
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v6
with:
name: performance-profile-${{ github.run_number }}
path: samply-profile.json
@@ -113,11 +115,11 @@ jobs:
benchmark:
name: Benchmark Tests
runs-on: ubuntu-latest
runs-on: ubicloud-standard-2
timeout-minutes: 45
steps:
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@v6
- name: Setup Rust environment
uses: ./.github/actions/setup
@@ -133,7 +135,7 @@ jobs:
tee benchmark-results.json
- name: Upload benchmark results
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v6
with:
name: benchmark-results-${{ github.run_number }}
path: benchmark-results.json

View File

@@ -1,78 +0,0 @@
## RustFS ${VERSION_CLEAN}
${ORIGINAL_NOTES}
---
### 🚀 Quick Download
**Linux (Static Binaries - No Dependencies):**
```bash
# x86_64 (Intel/AMD)
curl -LO https://github.com/rustfs/rustfs/releases/download/${VERSION}/rustfs-x86_64-unknown-linux-musl.zip
unzip rustfs-x86_64-unknown-linux-musl.zip
sudo mv rustfs /usr/local/bin/
# ARM64 (Graviton, Apple Silicon VMs)
curl -LO https://github.com/rustfs/rustfs/releases/download/${VERSION}/rustfs-aarch64-unknown-linux-musl.zip
unzip rustfs-aarch64-unknown-linux-musl.zip
sudo mv rustfs /usr/local/bin/
```
**macOS:**
```bash
# Apple Silicon (M1/M2/M3)
curl -LO https://github.com/rustfs/rustfs/releases/download/${VERSION}/rustfs-aarch64-apple-darwin.zip
unzip rustfs-aarch64-apple-darwin.zip
sudo mv rustfs /usr/local/bin/
# Intel
curl -LO https://github.com/rustfs/rustfs/releases/download/${VERSION}/rustfs-x86_64-apple-darwin.zip
unzip rustfs-x86_64-apple-darwin.zip
sudo mv rustfs /usr/local/bin/
```
### 📁 Available Downloads
| Platform | Architecture | File | Description |
|----------|-------------|------|-------------|
| Linux | x86_64 | `rustfs-x86_64-unknown-linux-musl.zip` | Static binary, no dependencies |
| Linux | ARM64 | `rustfs-aarch64-unknown-linux-musl.zip` | Static binary, no dependencies |
| macOS | Apple Silicon | `rustfs-aarch64-apple-darwin.zip` | Native binary, ZIP archive |
| macOS | Intel | `rustfs-x86_64-apple-darwin.zip` | Native binary, ZIP archive |
### 🔐 Verification
Download checksums and verify your download:
```bash
# Download checksums
curl -LO https://github.com/rustfs/rustfs/releases/download/${VERSION}/SHA256SUMS
# Verify (Linux)
sha256sum -c SHA256SUMS --ignore-missing
# Verify (macOS)
shasum -a 256 -c SHA256SUMS --ignore-missing
```
### 🛠️ System Requirements
- **Linux**: Any distribution with glibc 2.17+ (CentOS 7+, Ubuntu 16.04+)
- **macOS**: 10.15+ (Catalina or later)
- **Windows**: Windows 10 version 1809 or later
### 📚 Documentation
- [Installation Guide](https://github.com/rustfs/rustfs#installation)
- [Quick Start](https://github.com/rustfs/rustfs#quick-start)
- [Configuration](https://github.com/rustfs/rustfs/blob/main/docs/)
- [API Documentation](https://docs.rs/rustfs)
### 🆘 Support
- 🐛 [Report Issues](https://github.com/rustfs/rustfs/issues)
- 💬 [Community Discussions](https://github.com/rustfs/rustfs/discussions)
- 📖 [Documentation](https://github.com/rustfs/rustfs/tree/main/docs)

20
.github/workflows/stale.yml vendored Normal file
View File

@@ -0,0 +1,20 @@
name: "Mark stale issues"
on:
schedule:
- cron: "30 1 * * *"
jobs:
stale:
runs-on: ubuntu-latest
steps:
- uses: actions/stale@v9
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
stale-issue-message: 'This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.'
stale-issue-label: 'stale'
## Mark if there is no activity for more than 7 days
days-before-stale: 7
# If no one responds after 3 days, the tag will be closed.
days-before-close: 3
# These tags are exempt and will not close automatically.
exempt-issue-labels: 'pinned,security'

29
.gitignore vendored
View File

@@ -2,9 +2,13 @@
.DS_Store
.idea
.vscode
.cursor
.direnv/
/test
/logs
/data
/docs
/rustfs-data/
.devcontainer
rustfs/static/*
!rustfs/static/.gitkeep
@@ -19,3 +23,28 @@ deploy/certs/*
profile.json
.docker/openobserve-otel/data
*.zst
.secrets
*.go
*.pb
*.svg
deploy/logs/*.log.*
artifacts/
# s3-tests local artifacts (root directory only)
/s3-tests/
/s3-tests-local/
/s3tests.conf
/s3tests.conf.*
*.events
*.audit
*.snappy
PR_DESCRIPTION.md
IMPLEMENTATION_PLAN.md
scripts/s3-tests/selected_tests.txt
docs
# nix stuff
result*
*.gz
rustfs-webdav.code-workspace
.aiexclude

32
.pre-commit-config.yaml Normal file
View File

@@ -0,0 +1,32 @@
# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
- repo: local
hooks:
- id: cargo-fmt
name: cargo fmt
entry: cargo fmt --all --check
language: system
types: [rust]
pass_filenames: false
- id: cargo-clippy
name: cargo clippy
entry: cargo clippy --all-targets --all-features -- -D warnings
language: system
types: [rust]
pass_filenames: false
- id: cargo-check
name: cargo check
entry: cargo check --all-targets
language: system
types: [rust]
pass_filenames: false
- id: cargo-test
name: cargo test
entry: bash -c 'cargo test --workspace --exclude e2e_test && cargo test --all --doc'
language: system
types: [rust]
pass_filenames: false

192
.vscode/launch.json vendored
View File

@@ -1,9 +1,31 @@
{
// 使用 IntelliSense 了解相关属性。
// 悬停以查看现有属性的描述。
// 欲了解更多信息,请访问: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"type": "lldb",
"request": "launch",
"name": "Debug(only) executable 'rustfs'",
"env": {
"RUST_LOG": "rustfs=info,ecstore=info,s3s=info,iam=info",
"RUSTFS_SKIP_BACKGROUND_TASK": "on"
//"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
// "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
// "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"
},
"program": "${workspaceFolder}/target/debug/rustfs",
"args": [
"--access-key",
"rustfsadmin",
"--secret-key",
"rustfsadmin",
"--address",
"0.0.0.0:9010",
"--server-domains",
"127.0.0.1:9010",
"./target/volume/test{1...4}"
],
"cwd": "${workspaceFolder}"
},
{
"type": "lldb",
"request": "launch",
@@ -20,18 +42,22 @@
}
},
"env": {
"RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug"
"RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug,iam=debug",
"RUSTFS_SKIP_BACKGROUND_TASK": "on",
//"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
// "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
// "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"
},
"args": [
"--access-key",
"AKEXAMPLERUSTFS",
"rustfsadmin",
"--secret-key",
"SKEXAMPLERUSTFS",
"rustfsadmin",
"--address",
"0.0.0.0:9010",
"--domain-name",
"--server-domains",
"127.0.0.1:9010",
"./target/volume/test{0...4}"
"./target/volume/test{1...4}"
],
"cwd": "${workspaceFolder}"
},
@@ -63,12 +89,8 @@
"test",
"--no-run",
"--lib",
"--package=ecstore"
],
"filter": {
"name": "ecstore",
"kind": "lib"
}
"--package=rustfs-ecstore"
]
},
"args": [],
"cwd": "${workspaceFolder}"
@@ -77,14 +99,152 @@
"name": "Debug executable target/debug/rustfs",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/target/debug/rustfs",
"cargo": {
"args": [
"run",
"--bin",
"rustfs",
"-j",
"1",
"--profile",
"dev"
]
},
"args": [],
"cwd": "${workspaceFolder}",
//"stopAtEntry": false,
//"preLaunchTask": "cargo build",
"env": {
"RUSTFS_ACCESS_KEY": "rustfsadmin",
"RUSTFS_SECRET_KEY": "rustfsadmin",
//"RUSTFS_VOLUMES": "./target/volume/test{1...4}",
"RUSTFS_ADDRESS": ":9000",
"RUSTFS_CONSOLE_ENABLE": "true",
// "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
// "RUSTFS_OBS_METRIC_ENDPOINT": "http://127.0.0.1:4318/v1/metrics", // default otlp http endpoint
// "RUSTFS_OBS_LOG_ENDPOINT": "http://127.0.0.1:4318/v1/logs", // default otlp http endpoint
// "RUSTFS_COMPRESS_ENABLE": "true",
"RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
"RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
"RUST_LOG":"rustfs=debug,ecstore=debug,s3s=debug,iam=debug",
},
"sourceLanguages": [
"rust"
],
},
{
"type": "lldb",
"request": "launch",
"name": "Debug test_lifecycle_transition_basic",
"cargo": {
"args": [
"test",
"-p",
"rustfs-scanner",
"--test",
"lifecycle_integration_test",
"serial_tests::test_lifecycle_transition_basic",
"-j",
"1"
]
},
"args": [],
"cwd": "${workspaceFolder}"
},
{
"name": "Debug executable target/debug/test",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/target/debug/deps/lifecycle_integration_test-5915cbfcab491b3b",
"args": [
"--skip",
"test_lifecycle_expiry_basic",
"--skip",
"test_lifecycle_expiry_deletemarker",
//"--skip",
//"test_lifecycle_transition_basic",
],
"cwd": "${workspaceFolder}",
//"stopAtEntry": false,
//"preLaunchTask": "cargo build",
"sourceLanguages": [
"rust"
],
}
},
{
"name": "Debug executable target/debug/rustfs with sse",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/target/debug/rustfs",
"args": [],
"cwd": "${workspaceFolder}",
//"stopAtEntry": false,
//"preLaunchTask": "cargo build",
"env": {
"RUSTFS_ACCESS_KEY": "rustfsadmin",
"RUSTFS_SECRET_KEY": "rustfsadmin",
"RUSTFS_VOLUMES": "./target/volumes/test{1...4}",
"RUSTFS_ADDRESS": ":9000",
"RUSTFS_CONSOLE_ENABLE": "true",
"RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
"RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
// "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
// "RUSTFS_OBS_METRIC_ENDPOINT": "http://127.0.0.1:4318/v1/metrics", // default otlp http endpoint
// "RUSTFS_OBS_LOG_ENDPOINT": "http://127.0.0.1:4318/v1/logs", // default otlp http endpoint
// "RUSTFS_COMPRESS_ENABLE": "true",
// 1. simple sse test key (no kms system)
// "__RUSTFS_SSE_SIMPLE_CMK": "2dfNXGHlsEflGVCxb+5DIdGEl1sIvtwX+QfmYasi5QM=",
// 2. kms local backend test key
"RUSTFS_KMS_ENABLE": "true",
"RUSTFS_KMS_BACKEND": "local",
"RUSTFS_KMS_KEY_DIR": "./target/kms-key-dir",
"RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
"RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
// 3. kms vault backend test key
// "RUSTFS_KMS_ENABLE": "true",
// "RUSTFS_KMS_BACKEND": "vault",
// "RUSTFS_KMS_VAULT_ADDRESS": "http://127.0.0.1:8200",
// "RUSTFS_KMS_VAULT_TOKEN": "Dev Token",
// "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
},
"sourceLanguages": [
"rust"
],
},
{
"name": "E2E test executable target/debug/rustfs",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/target/debug/rustfs",
"args": [],
"cwd": "${workspaceFolder}",
//"stopAtEntry": false,
//"preLaunchTask": "cargo build",
"env": {
"RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug,iam=debug",
"RUST_BACKTRACE": "full",
"RUSTFS_ACCESS_KEY": "rustfsadmin",
"RUSTFS_SECRET_KEY": "rustfsadmin",
"RUSTFS_VOLUMES": "./target/e2e-test/test{1...4}",
"RUSTFS_REGION": "us-east-1",
"RUSTFS_ADDRESS": ":9000",
"RUSTFS_CONSOLE_ENABLE": "true",
"RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
"RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
"RUSTFS_KMS_ENABLE": "true",
"RUSTFS_KMS_BACKEND": "local",
"RUSTFS_KMS_KEY_DIR": "./target/e2e-key-dir",
"RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
"RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
},
"sourceLanguages": [
"rust"
],
},
]
}

67
AGENTS.md Normal file
View File

@@ -0,0 +1,67 @@
# RustFS Agent Instructions (Global)
This root file keeps repository-wide rules only.
Use the nearest subdirectory `AGENTS.md` for path-specific guidance.
## Rule Precedence
1. System/developer instructions.
2. This file (global defaults).
3. The nearest `AGENTS.md` in the current path (more specific scope wins).
If repo-level instructions conflict, follow the nearest file and keep behavior aligned with CI.
## Communication and Language
- Respond in the same language used by the requester.
- Keep source code, comments, commit messages, and PR title/body in English.
## Sources of Truth
- Workspace layout and crate membership: `Cargo.toml` (`[workspace].members`)
- Local quality commands: `Makefile` and `.config/make/`
- CI quality gates: `.github/workflows/ci.yml`
- PR template: `.github/pull_request_template.md`
Avoid duplicating long crate lists or command matrices in instruction files.
Reference the source files above instead.
## Mandatory Before Commit
Run and pass:
```bash
make pre-commit
```
If `make` is unavailable, run the equivalent checks defined under `.config/make/`.
Do not commit when required checks fail.
## Git and PR Baseline
- Use feature branches based on the latest `main`.
- Follow Conventional Commits, with subject length <= 72 characters.
- Keep PR title and description in English.
- Use `.github/pull_request_template.md` and keep all section headings.
- Use `N/A` for non-applicable template sections.
- Include verification commands in the PR description.
- When using `gh pr create`/`gh pr edit`, use `--body-file` instead of inline `--body` for multiline markdown.
## Security Baseline
- Never commit secrets, credentials, or key material.
- Use environment variables or vault tooling for sensitive configuration.
- For localhost-sensitive tests, verify proxy settings to avoid traffic leakage.
## Scoped Guidance in This Repository
- `.github/AGENTS.md`
- `crates/AGENTS.md`
- `crates/config/AGENTS.md`
- `crates/ecstore/AGENTS.md`
- `crates/e2e_test/AGENTS.md`
- `crates/iam/AGENTS.md`
- `crates/kms/AGENTS.md`
- `crates/policy/AGENTS.md`
- `rustfs/src/admin/AGENTS.md`
- `rustfs/src/storage/AGENTS.md`

79
CHANGELOG.md Normal file
View File

@@ -0,0 +1,79 @@
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Fixed
- **Helm Ingress**: `customAnnotations` are now merged with class-specific annotations (nginx/traefik) instead of being ignored when `ingress.className` is set.
### Added
- **OpenStack Keystone Authentication Integration**: Full support for OpenStack Keystone authentication via X-Auth-Token headers
- Tower-based middleware (`KeystoneAuthLayer`) self-contained within `rustfs-keystone` crate
- Task-local storage for async-safe credential passing between middleware and auth handlers
- Automatic detection of Keystone credentials (access keys prefixed with `keystone:`)
- Role-based permission mapping (admin/reseller_admin roles grant owner permissions)
- Token caching for high-performance validation with configurable cache size and TTL
- Dual authentication support: Keystone and standard AWS Signature v4 work simultaneously
- Immediate 401 response for invalid tokens (no fallback to local auth)
- XML-formatted error responses compatible with S3 API
- Comprehensive integration documentation with manual testing guide
- **32 unit and integration tests** covering middleware, auth handlers, task-local storage, and role detection
### Changed
- **HTTP Server Stack**: Integrated `KeystoneAuthLayer` middleware from `rustfs-keystone` crate into service stack (positioned after ReadinessGateLayer)
- **IAMAuth**: Enhanced `get_secret_key()` to return empty secret for Keystone credentials (bypasses signature validation)
- **Auth Module**: Modified `check_key_valid()` to retrieve Keystone credentials from task-local storage and determine admin status
### Technical Details
- Middleware is self-contained in `rustfs-keystone` crate following the trusted-proxies pattern for integration-specific middleware
- Uses `BoxBody` pattern for Hyper 1.x compatibility
- Task-local storage provides request-scoped credential passing without modifying HTTP request/response types
- Integration preserves existing S3 authentication flow while adding Keystone support
- Zero breaking changes to existing functionality
- No new top-level directories in main binary crate (middleware lives in integration crate)
### Documentation
- Updated `crates/keystone/README.md` with complete integration architecture and workflow
- Added detailed manual testing guide with 10 test scenarios
- Updated main `README.md` to list Keystone authentication as available feature
- Added troubleshooting section for common integration issues
### Configuration
New environment variables:
- `RUSTFS_KEYSTONE_ENABLE` - Enable/disable Keystone authentication (default: false)
- `RUSTFS_KEYSTONE_AUTH_URL` - Keystone API endpoint URL
- `RUSTFS_KEYSTONE_VERSION` - Keystone API version (v3)
- `RUSTFS_KEYSTONE_ADMIN_USER` - Admin username for privileged operations
- `RUSTFS_KEYSTONE_ADMIN_PASSWORD` - Admin password
- `RUSTFS_KEYSTONE_ADMIN_PROJECT` - Admin project name
- `RUSTFS_KEYSTONE_ADMIN_DOMAIN` - Admin domain name (default: Default)
- `RUSTFS_KEYSTONE_CACHE_SIZE` - Token cache size (default: 10000)
- `RUSTFS_KEYSTONE_CACHE_TTL` - Token cache TTL in seconds (default: 300)
- `RUSTFS_KEYSTONE_VERIFY_SSL` - Verify SSL certificates (default: true)
### Files Modified
- `crates/keystone/src/middleware.rs` - Created Keystone authentication middleware (self-contained in keystone crate)
- `crates/keystone/src/lib.rs` - Exported middleware module and KEYSTONE_CREDENTIALS
- `crates/keystone/Cargo.toml` - Added Tower/HTTP dependencies for middleware functionality
- `rustfs/src/server/http.rs` - Integrated KeystoneAuthLayer from rustfs-keystone crate
- `rustfs/src/auth.rs` - Enhanced IAMAuth and check_key_valid for Keystone support, imported KEYSTONE_CREDENTIALS from rustfs-keystone
- `crates/keystone/README.md` - Comprehensive integration documentation
- `README.md` - Added Keystone as available feature
### Testing
- 16 unit tests in rustfs-keystone crate (config, auth, middleware, identity)
- 10 integration tests in rustfs-keystone crate (task-local storage, middleware layer, scope isolation)
- 6 auth unit tests in rustfs crate (role detection, task-local storage, Keystone credential handling)
- **Total: 32 tests** passing with zero compilation errors
- Manual testing guide provided for end-to-end validation
- All tests passing with `cargo test --all --exclude e2e_test`
---
## Previous Releases
See [GitHub Releases](https://github.com/rustfs/rustfs/releases) for previous version history.

61
CLA.md
View File

@@ -1,39 +1,60 @@
RustFS Individual Contributor License Agreement
# RustFS Individual Contributor License Agreement
Thank you for your interest in contributing documentation and related software code to a project hosted or managed by RustFS. In order to clarify the intellectual property license granted with Contributions from any person or entity, RustFS must have a Contributor License Agreement (CLA) on file that has been signed by each Contributor, indicating agreement to the license terms below. This version of the Contributor License Agreement allows an individual to submit Contributions to the applicable project. If you are making a submission on behalf of a legal entity, then you should sign the separate Corporate Contributor License Agreement.
Thank you for your interest in contributing to RustFS. In order to clarify the intellectual property license granted with Contributions from any person or entity, RustFS, Inc. ("RustFS") must have a Contributor License Agreement ("CLA") on file that has been signed by each Contributor, indicating agreement to the license terms below. This license is for your protection as a Contributor as well as the protection of RustFS and its users; it does not change your rights to use your own Contributions for any other purpose.
You accept and agree to the following terms and conditions for Your present and future Contributions submitted to RustFS. You hereby irrevocably assign and transfer to RustFS all right, title, and interest in and to Your Contributions, including all copyrights and other intellectual property rights therein.
You accept and agree to the following terms and conditions for Your present and future Contributions submitted to RustFS. Except for the license granted herein to RustFS and recipients of software distributed by RustFS, You reserve all right, title, and interest in and to Your Contributions.
Definitions
**You understand and agree that You will not receive any royalty or other compensation for Your Contributions.**
“You” (or “Your”) shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with RustFS. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, “control” means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
## 1. Definitions
“Contribution” shall mean any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You to RustFS for inclusion in, or documentation of, any of the products or projects owned or managed by RustFS (the “Work”), including without limitation any Work described in Schedule A. For the purposes of this definition, “submitted” means any form of electronic or written communication sent to RustFS or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, RustFS for the purpose of discussing and improving the Work.
* **"You"** (or **"Your"**) shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with RustFS. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
Assignment of Copyright
* **"Contribution"** shall mean any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You to RustFS for inclusion in, or documentation of, any of the products or projects owned or managed by RustFS (the "Work"). For the purposes of this definition, "submitted" means any form of electronic or written communication sent to RustFS or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, RustFS for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution."
Subject to the terms and conditions of this Agreement, You hereby irrevocably assign and transfer to RustFS all right, title, and interest in and to Your Contributions, including all copyrights and other intellectual property rights therein, for the entire term of such rights, including all renewals and extensions. You agree to execute all documents and take all actions as may be reasonably necessary to vest in RustFS the ownership of Your Contributions and to assist RustFS in perfecting, maintaining, and enforcing its rights in Your Contributions.
## 2. Grant of Copyright License
Grant of Patent License
Subject to the terms and conditions of this Agreement, You hereby grant to RustFS and to recipients of software distributed by RustFS a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works under any license, including proprietary or commercial licenses and open-source licenses, at RustFS's sole discretion.
Subject to the terms and conditions of this Agreement, You hereby grant to RustFS and to recipients of documentation and software distributed by RustFS a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed.
## 3. Grant of Patent License
You represent that you are legally entitled to grant the above assignment and license.
Subject to the terms and conditions of this Agreement, You hereby grant to RustFS and to recipients of software distributed by RustFS a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed.
You represent that each of Your Contributions is Your original creation (see section 7 for submissions on behalf of others). You represent that Your Contribution submissions include complete details of any third-party license or other restriction (including, but not limited to, related patents and trademarks) of which you are personally aware and which are associated with any part of Your Contributions.
## 4. Representations
You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON- INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE.
You represent that you are legally entitled to grant the above license. If your employer(s) has rights to intellectual property that you create that includes your Contributions, you represent that you have received permission to make Contributions on behalf of that employer, that your employer has waived such rights for your Contributions to RustFS, or that your employer has executed a separate Corporate CLA with RustFS.
Should You wish to submit work that is not Your original creation, You may submit it to RustFS separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously marking the work as “Submitted on behalf of a third-party: [named here]”.
You represent that each of Your Contributions is Your original creation. You represent that Your Contribution submissions include complete details of any third-party license or other restriction (including, but not limited to, related patents and trademarks) of which you are personally aware and which are associated with any part of Your Contributions.
You agree to notify RustFS of any facts or circumstances of which you become aware that would make these representations inaccurate in any respect.
## 5. Support and Warranty
Modification of CLA
You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of **TITLE**, **NON-INFRINGEMENT**, **MERCHANTABILITY**, or **FITNESS FOR A PARTICULAR PURPOSE**.
RustFS reserves the right to update or modify this CLA in the future. Any updates or modifications to this CLA shall apply only to Contributions made after the effective date of the revised CLA. Contributions made prior to the update shall remain governed by the version of the CLA that was in effect at the time of submission. It is not necessary for all Contributors to re-sign the CLA when the CLA is updated or modified.
## 6. Third-Party Work
Governing Law and Dispute Resolution
Should You wish to submit work that is not Your original creation, You may submit it to RustFS separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously marking the work as "Submitted on behalf of a third-party: [named here]".
This Agreement will be governed by and construed in accordance with the laws of the Peoples Republic of China excluding that body of laws known as conflict of laws. The parties expressly agree that the United Nations Convention on Contracts for the International Sale of Goods will not apply. Any legal action or proceeding arising under this Agreement will be brought exclusively in the courts located in Beijing, China, and the parties hereby irrevocably consent to the personal jurisdiction and venue therein.
## 7. Governing Law and Jurisdiction
For your reading convenience, this Agreement is written in parallel English and Chinese sections. To the extent there is a conflict between the English and Chinese sections, the English sections shall govern.
This Agreement shall be governed by and construed in accordance with the laws of the State of Delaware, United States of America, without regard to its conflict of laws principles. The parties expressly agree that the United Nations Convention on Contracts for the International Sale of Goods will not apply. Any legal action or proceeding arising under or in connection with this Agreement shall be brought exclusively in the state or federal courts located in the State of Delaware, United States of America, and the parties hereby irrevocably consent to the personal jurisdiction and venue therein.
## 8. Severability
If any provision of this Agreement is found to be invalid or unenforceable, the remaining provisions will continue in full force and effect.
## 9. Entire Agreement and Assignment
This Agreement constitutes the entire agreement between the parties with respect to the subject matter hereof and supersedes all prior agreements, understandings, negotiations, and discussions, whether written or oral. RustFS may assign its rights and obligations under this Agreement to any third party. You may not assign Your rights or obligations under this Agreement without the prior written consent of RustFS.
---
**Please read the terms of this Agreement carefully. By submitting a Contribution to RustFS, You agree to be bound by the terms and conditions of this Agreement.**
| | |
|---|---|
| **Full Name** | __________________________________ |
| **GitHub Username** | __________________________________ |
| **Email Address** | __________________________________ |
| **Date** | __________________________________ |
*(Electronic signature or acknowledgement via GitHub commit/Pull Request constitutes valid acceptance of this Agreement).*

View File

@@ -2,6 +2,8 @@
## 📋 Code Quality Requirements
For instructions on setting up and running the local development environment, please see [Development Guide](docs/DEVELOPMENT.md).
### 🔧 Code Formatting Rules
**MANDATORY**: All code must be properly formatted before committing. This project enforces strict formatting standards to maintain code consistency and readability.
@@ -184,6 +186,39 @@ cargo clippy --all-targets --all-features -- -D warnings
cargo clippy --fix --all-targets --all-features
```
## 📝 Pull Request Guidelines
### Language Requirements
**All Pull Request titles and descriptions MUST be written in English.**
This ensures:
- Consistency across all contributions
- Accessibility for international contributors
- Better integration with automated tools and CI/CD systems
- Clear communication in a globally understood language
#### PR Description Requirements
When creating a Pull Request, ensure:
1. **Title**: Use English and follow Conventional Commits format (e.g., `fix: improve s3-tests readiness detection`)
2. **Description**: Write in English, following the PR template format
3. **Code Comments**: Must be in English (as per coding standards)
4. **Commit Messages**: Must be in English (as per commit guidelines)
#### PR Template
Always use the PR template (`.github/pull_request_template.md`) and fill in all sections:
- Type of Change
- Related Issues
- Summary of Changes
- Checklist
- Impact
- Additional Notes
**Note**: While you may communicate with reviewers in Chinese during discussions, the PR itself (title, description, and all formal documentation) must be in English.
---
Following these guidelines ensures high code quality and smooth collaboration across the RustFS project! 🚀

9690
Cargo.lock generated

File diff suppressed because it is too large Load Diff

View File

@@ -15,41 +15,54 @@
[workspace]
members = [
"rustfs", # Core file system implementation
"cli/rustfs-gui", # Graphical user interface client
"crates/appauth", # Application authentication and authorization
"crates/audit", # Audit target management system with multi-target fan-out
"crates/checksums", # client checksums
"crates/common", # Shared utilities and data structures
"crates/config", # Configuration management
"crates/credentials", # Credential management system
"crates/crypto", # Cryptography and security features
"crates/ecstore", # Erasure coding storage implementation
"crates/e2e_test", # End-to-end test suite
"crates/filemeta", # File metadata management
"crates/heal", # Erasure set and object healing
"crates/iam", # Identity and Access Management
"crates/keystone", # OpenStack Keystone integration
"crates/kms", # Key Management Service
"crates/lock", # Distributed locking implementation
"crates/madmin", # Management dashboard and admin API interface
"crates/mcp", # MCP server for S3 operations
"crates/metrics", # Metrics collection and reporting
"crates/notify", # Notification system for events
"crates/obs", # Observability utilities
"crates/policy", # Policy management
"crates/protocols", # Protocol implementations (FTPS, SFTP, etc.)
"crates/protos", # Protocol buffer definitions
"crates/rio", # Rust I/O utilities and abstractions
"crates/s3-common", # Common utilities and data structures for S3 compatibility
"crates/s3select-api", # S3 Select API interface
"crates/s3select-query", # S3 Select query engine
"crates/scanner", # Scanner for data integrity checks and health monitoring
"crates/signer", # client signer
"crates/targets", # Target-specific configurations and utilities
"crates/trusted-proxies", # Trusted proxies management
"crates/utils", # Utility functions and helpers
"crates/workers", # Worker thread pools and task scheduling
"crates/zip", # ZIP file handling and compression
"crates/ahm",
]
resolver = "2"
resolver = "3"
[workspace.package]
edition = "2024"
license = "Apache-2.0"
repository = "https://github.com/rustfs/rustfs"
rust-version = "1.85"
rust-version = "1.93.0"
version = "0.0.5"
homepage = "https://rustfs.com"
description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
categories = ["web-programming", "development-tools", "filesystem", "network-programming"]
authors = ["RustFS Team"]
[workspace.lints.rust]
unsafe_code = "deny"
@@ -57,223 +70,244 @@ unsafe_code = "deny"
[workspace.lints.clippy]
all = "warn"
[patch.crates-io]
rustfs-utils = { path = "crates/utils" }
rustfs-filemeta = { path = "crates/filemeta" }
rustfs-rio = { path = "crates/rio" }
[workspace.dependencies]
rustfs-ahm = { path = "crates/ahm", version = "0.0.3" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "0.0.5" }
# RustFS Internal Crates
rustfs = { path = "./rustfs", version = "0.0.5" }
rustfs-heal = { path = "crates/heal", version = "0.0.5" }
rustfs-appauth = { path = "crates/appauth", version = "0.0.5" }
rustfs-audit = { path = "crates/audit", version = "0.0.5" }
rustfs-checksums = { path = "crates/checksums", version = "0.0.5" }
rustfs-common = { path = "crates/common", version = "0.0.5" }
rustfs-config = { path = "./crates/config", version = "0.0.5" }
rustfs-credentials = { path = "crates/credentials", version = "0.0.5" }
rustfs-crypto = { path = "crates/crypto", version = "0.0.5" }
rustfs-ecstore = { path = "crates/ecstore", version = "0.0.5" }
rustfs-filemeta = { path = "crates/filemeta", version = "0.0.5" }
rustfs-iam = { path = "crates/iam", version = "0.0.5" }
rustfs-keystone = { path = "crates/keystone", version = "0.0.5" }
rustfs-kms = { path = "crates/kms", version = "0.0.5" }
rustfs-lock = { path = "crates/lock", version = "0.0.5" }
rustfs-madmin = { path = "crates/madmin", version = "0.0.5" }
rustfs-mcp = { path = "crates/mcp", version = "0.0.5" }
rustfs-metrics = { path = "crates/metrics", version = "0.0.5" }
rustfs-notify = { path = "crates/notify", version = "0.0.5" }
rustfs-obs = { path = "crates/obs", version = "0.0.5" }
rustfs-policy = { path = "crates/policy", version = "0.0.5" }
rustfs-protos = { path = "crates/protos", version = "0.0.5" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "0.0.5" }
rustfs = { path = "./rustfs", version = "0.0.5" }
rustfs-zip = { path = "./crates/zip", version = "0.0.5" }
rustfs-config = { path = "./crates/config", version = "0.0.5" }
rustfs-obs = { path = "crates/obs", version = "0.0.5" }
rustfs-notify = { path = "crates/notify", version = "0.0.5" }
rustfs-utils = { path = "crates/utils", version = "0.0.5" }
rustfs-rio = { path = "crates/rio", version = "0.0.5" }
rustfs-filemeta = { path = "crates/filemeta", version = "0.0.5" }
rustfs-s3-common = { path = "crates/s3-common", version = "0.0.5" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "0.0.5" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "0.0.5" }
rustfs-scanner = { path = "crates/scanner", version = "0.0.5" }
rustfs-signer = { path = "crates/signer", version = "0.0.5" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "0.0.5" }
rustfs-targets = { path = "crates/targets", version = "0.0.5" }
rustfs-utils = { path = "crates/utils", version = "0.0.5" }
rustfs-workers = { path = "crates/workers", version = "0.0.5" }
aes-gcm = { version = "0.10.3", features = ["std"] }
arc-swap = "1.7.1"
argon2 = { version = "0.5.3", features = ["std"] }
atoi = "2.0.0"
async-channel = "2.4.0"
rustfs-zip = { path = "./crates/zip", version = "0.0.5" }
rustfs-protocols = { path = "crates/protocols", version = "0.0.5" }
# Async Runtime and Networking
async-channel = "2.5.0"
async-compression = { version = "0.4.41" }
async-recursion = "1.1.1"
async-trait = "0.1.88"
async-compression = { version = "0.4.0" }
atomic_enum = "0.3.0"
aws-sdk-s3 = "1.96.0"
axum = "0.8.4"
axum-extra = "0.10.1"
axum-server = { version = "0.7.2", features = ["tls-rustls"] }
base64-simd = "0.8.0"
base64 = "0.22.1"
brotli = "8.0.1"
bytes = { version = "1.10.1", features = ["serde"] }
bytesize = "2.0.1"
async-trait = "0.1.89"
axum = "0.8.8"
futures = "0.3.32"
futures-core = "0.3.32"
futures-util = "0.3.32"
pollster = "0.4.0"
hyper = { version = "1.8.1", features = ["http2", "http1", "server"] }
hyper-rustls = { version = "0.27.7", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs", "webpki-roots"] }
hyper-util = { version = "0.1.20", features = ["tokio", "server-auto", "server-graceful", "tracing"] }
http = "1.4.0"
http-body = "1.0.1"
http-body-util = "0.1.3"
reqwest = { version = "0.13.2", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
socket2 = { version = "0.6.3", features = ["all"] }
tokio = { version = "1.50.0", features = ["fs", "rt-multi-thread"] }
tokio-rustls = { version = "0.26.4", default-features = false, features = ["logging", "tls12", "aws-lc-rs"] }
tokio-stream = { version = "0.1.18" }
tokio-test = "0.4.5"
tokio-util = { version = "0.7.18", features = ["io", "compat"] }
tonic = { version = "0.14.5", features = ["gzip"] }
tonic-prost = { version = "0.14.5" }
tonic-prost-build = { version = "0.14.5" }
tower = { version = "0.5.3", features = ["timeout"] }
tower-http = { version = "0.6.8", features = ["cors"] }
# Serialization and Data Formats
bytes = { version = "1.11.1", features = ["serde"] }
bytesize = "2.3.1"
byteorder = "1.5.0"
cfg-if = "1.0.1"
chacha20poly1305 = { version = "0.10.1" }
chrono = { version = "0.4.41", features = ["serde"] }
clap = { version = "4.5.40", features = ["derive", "env"] }
const-str = { version = "0.6.2", features = ["std", "proc"] }
crc32fast = "1.4.2"
criterion = { version = "0.5", features = ["html_reports"] }
dashmap = "6.1.0"
datafusion = "46.0.1"
flatbuffers = "25.12.19"
form_urlencoded = "1.2.2"
prost = "0.14.3"
quick-xml = "0.39.2"
rmcp = { version = "1.2.0" }
rmp = { version = "0.8.15" }
rmp-serde = { version = "1.3.1" }
serde = { version = "1.0.228", features = ["derive"] }
serde_json = { version = "1.0.149", features = ["raw_value"] }
serde_urlencoded = "0.7.1"
schemars = "1.2.1"
# Cryptography and Security
aes-gcm = { version = "0.11.0-rc.3", features = ["rand_core"] }
argon2 = { version = "0.6.0-rc.7" }
blake3 = { version = "1.8.3", features = ["rayon", "mmap"] }
chacha20poly1305 = { version = "0.11.0-rc.3" }
crc-fast = "1.9.0"
hmac = { version = "0.13.0-rc.5" }
jsonwebtoken = { version = "10.3.0", features = ["aws_lc_rs"] }
openidconnect = { version = "4.0", default-features = false }
pbkdf2 = "0.13.0-rc.9"
rsa = { version = "0.10.0-rc.17" }
rustls = { version = "0.23.37", default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
rustls-pki-types = "1.14.0"
sha1 = "0.11.0-rc.5"
sha2 = "0.11.0-rc.5"
subtle = "2.6"
zeroize = { version = "1.8.2", features = ["derive"] }
# Time and Date
chrono = { version = "0.4.44", features = ["serde"] }
humantime = "2.3.0"
jiff = { version = "0.2.23", features = ["serde"] }
time = { version = "0.3.47", features = ["std", "parsing", "formatting", "macros", "serde"] }
# Utilities and Tools
anyhow = "1.0.102"
arc-swap = "1.8.2"
astral-tokio-tar = "0.5.6"
atoi = "2.0.0"
atomic_enum = "0.3.0"
aws-config = { version = "1.8.15" }
aws-credential-types = { version = "1.2.14" }
aws-sdk-s3 = { version = "1.126.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
aws-smithy-http-client = { version = "1.1.12", default-features = false, features = ["default-client", "rustls-aws-lc"] }
aws-smithy-types = { version = "1.4.6" }
backtrace = "0.3.76"
base64 = "0.22.1"
base64-simd = "0.8.0"
brotli = "8.0.2"
cfg-if = "1.0.4"
clap = { version = "4.6.0", features = ["derive", "env"] }
const-str = { version = "1.1.0", features = ["std", "proc"] }
convert_case = "0.11.0"
criterion = { version = "0.8", features = ["html_reports"] }
crossbeam-queue = "0.3.12"
crossbeam-channel = "0.5.15"
crossbeam-deque = "0.8.6"
crossbeam-utils = "0.8.21"
datafusion = "52.3.0"
derive_builder = "0.20.2"
dioxus = { version = "0.6.3", features = ["router"] }
dirs = "6.0.0"
enumset = "1.1.6"
flatbuffers = "25.2.10"
flate2 = "1.1.2"
flexi_logger = { version = "0.31.2", features = ["trc", "dont_minimize_extra_stacks"] }
form_urlencoded = "1.2.1"
futures = "0.3.31"
futures-core = "0.3.31"
futures-util = "0.3.31"
glob = "0.3.2"
enumset = "1.1.10"
faster-hex = "0.10.0"
flate2 = "1.1.9"
glob = "0.3.3"
google-cloud-storage = "1.9.0"
google-cloud-auth = "1.7.0"
hashbrown = { version = "0.16.1", features = ["serde", "rayon"] }
hex = "0.4.3"
hex-simd = "0.8.0"
highway = { version = "1.3.0" }
hmac = "0.12.1"
hyper = "1.6.0"
hyper-util = { version = "0.1.14", features = [
"tokio",
"server-auto",
"server-graceful",
] }
hyper-rustls = "0.27.7"
http = "1.3.1"
http-body = "1.0.1"
humantime = "2.2.0"
ipnetwork = { version = "0.21.1", features = ["serde"] }
jsonwebtoken = "9.3.1"
keyring = { version = "3.6.2", features = [
"apple-native",
"windows-native",
"sync-secret-service",
] }
lazy_static = "1.5.0"
libsystemd = { version = "0.7.2" }
local-ip-address = "0.6.5"
libc = "0.2.183"
libsystemd = "0.7.2"
local-ip-address = "0.6.10"
lz4 = "1.28.1"
matchit = "0.8.4"
md-5 = "0.10.6"
matchit = "0.9.1"
md-5 = "0.11.0-rc.5"
md5 = "0.8.0"
mime_guess = "2.0.5"
moka = { version = "0.12.14", features = ["future"] }
netif = "0.1.6"
nix = { version = "0.30.1", features = ["fs"] }
nu-ansi-term = "0.50.1"
num_cpus = { version = "1.17.0" }
nvml-wrapper = "0.11.0"
object_store = "0.11.2"
once_cell = "1.21.3"
opentelemetry = { version = "0.30.0" }
opentelemetry-appender-tracing = { version = "0.30.1", features = [
"experimental_use_tracing_span_context",
"experimental_metadata_attributes",
"spec_unstable_logs_enabled"
] }
opentelemetry_sdk = { version = "0.30.0" }
opentelemetry-stdout = { version = "0.30.0" }
opentelemetry-otlp = { version = "0.30.0", default-features = false, features = [
"grpc-tonic", "gzip-tonic", "trace", "metrics", "logs", "internal-logs"
] }
opentelemetry-semantic-conventions = { version = "0.30.0", features = [
"semconv_experimental",
] }
parking_lot = "0.12.4"
nvml-wrapper = "0.12.0"
object_store = "0.12.5"
parking_lot = "0.12.5"
path-absolutize = "3.1.1"
path-clean = "1.0.1"
blake3 = { version = "1.8.2" }
pbkdf2 = "0.12.2"
percent-encoding = "2.3.1"
pin-project-lite = "0.2.16"
prost = "0.13.5"
quick-xml = "0.37.5"
rand = "0.9.1"
rdkafka = { version = "0.37.0", features = ["tokio"] }
reed-solomon-simd = { version = "3.0.1" }
regex = { version = "1.11.1" }
reqwest = { version = "0.12.22", default-features = false, features = [
"rustls-tls",
"charset",
"http2",
"system-proxy",
"stream",
"json",
"blocking",
] }
rfd = { version = "0.15.3", default-features = false, features = [
"xdg-portal",
"tokio",
] }
rmp = "0.8.14"
rmp-serde = "1.3.0"
rsa = "0.9.8"
rumqttc = { version = "0.24" }
rust-embed = { version = "8.7.2" }
rust-i18n = { version = "3.1.5" }
rustfs-rsc = "2025.506.1"
rustls = { version = "0.23.28" }
rustls-pki-types = "1.12.0"
rustls-pemfile = "2.2.0"
s3s = { version = "0.12.0-minio-preview.1" }
shadow-rs = { version = "1.2.0", default-features = false }
serde = { version = "1.0.219", features = ["derive"] }
serde_json = { version = "1.0.140", features = ["raw_value"] }
serde-xml-rs = "0.8.1"
serde_urlencoded = "0.7.1"
sha1 = "0.10.6"
sha2 = "0.10.9"
siphasher = "1.0.1"
percent-encoding = "2.3.2"
pin-project-lite = "0.2.17"
pretty_assertions = "1.4.1"
rand = { version = "0.10.0", features = ["serde"] }
ratelimit = "0.10.0"
rayon = "1.11.0"
reed-solomon-simd = { version = "3.1.0" }
regex = { version = "1.12.3" }
rumqttc = { version = "0.25.1" }
rustix = { version = "1.1.4", features = ["fs"] }
rust-embed = { version = "8.11.0" }
rustc-hash = { version = "2.1.1" }
s3s = { git = "https://github.com/s3s-project/s3s", rev = "c2dc7b16535659904d4efff52c558fc039be1ef3", features = ["minio"] }
serial_test = "3.4.0"
shadow-rs = { version = "1.7.1", default-features = false }
siphasher = "1.0.2"
smallvec = { version = "1.15.1", features = ["serde"] }
snafu = "0.8.6"
smartstring = "1.0.1"
snafu = "0.9.0"
snap = "1.1.1"
socket2 = "0.5.10"
strum = { version = "0.27.1", features = ["derive"] }
sysinfo = "0.35.2"
tempfile = "3.20.0"
starshard = { version = "1.1.0", features = ["rayon", "async", "serde"] }
strum = { version = "0.28.0", features = ["derive"] }
sysinfo = "0.38.4"
temp-env = "0.3.6"
tempfile = "3.27.0"
test-case = "3.3.1"
thiserror = "2.0.12"
time = { version = "0.3.41", features = [
"std",
"parsing",
"formatting",
"macros",
"serde",
] }
tokio = { version = "1.46.1", features = ["fs", "rt-multi-thread"] }
tokio-rustls = { version = "0.26.2", default-features = false }
tokio-stream = { version = "0.1.17" }
tokio-tar = "0.3.1"
tokio-util = { version = "0.7.15", features = ["io", "compat"] }
tonic = { version = "0.13.1", features = ["gzip"] }
tonic-build = { version = "0.13.1" }
tower = { version = "0.5.2", features = ["timeout"] }
tower-http = { version = "0.6.6", features = ["cors"] }
tracing = "0.1.41"
tracing-core = "0.1.34"
thiserror = "2.0.18"
tracing = { version = "0.1.44" }
tracing-appender = "0.2.4"
tracing-error = "0.2.1"
tracing-subscriber = { version = "0.3.19", features = ["env-filter", "time"] }
tracing-appender = "0.2.3"
tracing-opentelemetry = "0.31.0"
tracing-opentelemetry = "0.32.1"
tracing-subscriber = { version = "0.3.23", features = ["env-filter", "time"] }
transform-stream = "0.3.1"
url = "2.5.4"
url = "2.5.8"
urlencoding = "2.1.3"
uuid = { version = "1.17.0", features = [
"v4",
"fast-rng",
"macro-diagnostics",
] }
wildmatch = { version = "2.4.0", features = ["serde"] }
winapi = { version = "0.3.9" }
uuid = { version = "1.22.0", features = ["v4", "fast-rng", "macro-diagnostics"] }
vaultrs = { version = "0.7.4" }
walkdir = "2.5.0"
wildmatch = { version = "2.6.1", features = ["serde"] }
windows = { version = "0.62.2" }
xxhash-rust = { version = "0.8.15", features = ["xxh64", "xxh3"] }
zip = "2.4.2"
zip = "8.2.0"
zstd = "0.13.3"
anyhow = "1.0.86"
[profile.wasm-dev]
inherits = "dev"
opt-level = 1
# Observability and Metrics
metrics = "0.24.3"
opentelemetry = { version = "0.31.0" }
opentelemetry-appender-tracing = { version = "0.31.1", features = ["experimental_use_tracing_span_context", "experimental_metadata_attributes", "spec_unstable_logs_enabled"] }
opentelemetry-otlp = { version = "0.31.0", features = ["gzip-http", "reqwest-rustls"] }
opentelemetry_sdk = { version = "0.31.0" }
opentelemetry-semantic-conventions = { version = "0.31.0", features = ["semconv_experimental"] }
opentelemetry-stdout = { version = "0.31.0" }
pyroscope = { version = "2.0.0", features = ["backend-pprof-rs"] }
[profile.server-dev]
inherits = "dev"
# FTP and SFTP
libunftp = { version = "0.23.0", features = ["experimental"] }
unftp-core = "0.1.0"
suppaftp = { version = "8.0.2", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
rcgen = "0.14.7"
[profile.android-dev]
inherits = "dev"
# WebDAV
dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
mimalloc = "0.1"
# Use tikv-jemallocator as memory allocator and enable performance analysis
tikv-jemallocator = { version = "0.6", features = ["profiling", "stats", "unprefixed_malloc_on_supported_platforms", "background_threads"] }
# Used to control and obtain statistics for jemalloc at runtime
tikv-jemalloc-ctl = { version = "0.6", features = ["use_std", "stats", "profiling"] }
# Used to generate pprof-compatible memory profiling data and support symbolization and flame graphs
jemalloc_pprof = { version = "0.8.2", features = ["symbolize", "flamegraph"] }
# Used to generate CPU performance analysis data and flame diagrams
# pprof = { version = "0.15.0", features = ["flamegraph", "protobuf-codec"] }
# Pyroscope uses a patched pprof, until they merge back upstream, replace all references. Otherwise, two pprof libs with symbol collision.
pprof = { package = "pprof-pyroscope-fork", version = "0.1500.3", features = ["flamegraph", "protobuf-codec"] }
[workspace.metadata.cargo-shear]
ignored = ["rustfs", "rustfs-mcp"]
[profile.release]
opt-level = 3

View File

@@ -12,38 +12,98 @@
# See the License for the specific language governing permissions and
# limitations under the License.
FROM alpine:3.18 AS builder
FROM alpine:3.23 AS build
RUN apk add -U --no-cache \
ca-certificates \
curl \
bash \
unzip
ARG TARGETARCH
ARG RELEASE=latest
RUN apk add --no-cache ca-certificates curl unzip
WORKDIR /build
RUN set -eux; \
case "$TARGETARCH" in \
amd64) ARCH_SUBSTR="x86_64-musl" ;; \
arm64) ARCH_SUBSTR="aarch64-musl" ;; \
*) echo "Unsupported TARGETARCH=$TARGETARCH" >&2; exit 1 ;; \
esac; \
if [ "$RELEASE" = "latest" ]; then \
TAG="$(curl -fsSL https://api.github.com/repos/rustfs/rustfs/releases \
| grep -o '"tag_name": "[^"]*"' | cut -d'"' -f4 | head -n 1)"; \
else \
TAG="$RELEASE"; \
fi; \
echo "Using tag: $TAG (arch pattern: $ARCH_SUBSTR)"; \
# Find download URL in assets list for this tag that contains arch substring and ends with .zip
URL="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG" \
| grep -o "\"browser_download_url\": \"[^\"]*${ARCH_SUBSTR}[^\"]*\\.zip\"" \
| cut -d'"' -f4 | head -n 1)"; \
if [ -z "$URL" ]; then echo "Failed to locate release asset for $ARCH_SUBSTR at tag $TAG" >&2; exit 1; fi; \
echo "Downloading: $URL"; \
curl -fL "$URL" -o rustfs.zip; \
unzip -q rustfs.zip -d /build; \
# If binary is not in root directory, try to locate and move from zip to /build/rustfs
if [ ! -x /build/rustfs ]; then \
BIN_PATH="$(unzip -Z -1 rustfs.zip | grep -E '(^|/)rustfs$' | head -n 1 || true)"; \
if [ -n "$BIN_PATH" ]; then \
mkdir -p /build/.tmp && unzip -q rustfs.zip "$BIN_PATH" -d /build/.tmp && \
mv "/build/.tmp/$BIN_PATH" /build/rustfs; \
fi; \
fi; \
[ -x /build/rustfs ] || { echo "rustfs binary not found in asset" >&2; exit 1; }; \
chmod +x /build/rustfs; \
rm -rf rustfs.zip /build/.tmp || true
RUN curl -Lo /tmp/rustfs.zip https://dl.rustfs.com/artifacts/rustfs/rustfs-x86_64-unknown-linux-musl.zip && \
unzip -o /tmp/rustfs.zip -d /tmp && \
mv /tmp/rustfs /rustfs && \
chmod +x /rustfs && \
rm -rf /tmp/*
FROM alpine:3.23
FROM alpine:3.18
ARG RELEASE=latest
ARG BUILD_DATE
ARG VCS_REF
RUN apk add -U --no-cache \
ca-certificates \
bash
LABEL name="RustFS" \
vendor="RustFS Team" \
maintainer="RustFS Team <dev@rustfs.com>" \
version="v${RELEASE#v}" \
release="${RELEASE}" \
build-date="${BUILD_DATE}" \
vcs-ref="${VCS_REF}" \
summary="High-performance distributed object storage system compatible with S3 API" \
description="RustFS is a distributed object storage system written in Rust, supporting erasure coding, multi-tenant management, and observability." \
url="https://rustfs.com" \
license="Apache-2.0"
COPY --from=builder /rustfs /usr/local/bin/rustfs
RUN apk add --no-cache ca-certificates coreutils curl
ENV RUSTFS_ACCESS_KEY=rustfsadmin \
RUSTFS_SECRET_KEY=rustfsadmin \
RUSTFS_ADDRESS=":9000" \
RUSTFS_CONSOLE_ENABLE=true \
RUST_LOG=warn
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /build/rustfs /usr/bin/rustfs
COPY entrypoint.sh /entrypoint.sh
EXPOSE 9000
RUN chmod +x /usr/bin/rustfs /entrypoint.sh
RUN mkdir -p /data
VOLUME /data
RUN addgroup -g 10001 -S rustfs && \
adduser -u 10001 -G rustfs -S rustfs -D && \
mkdir -p /data /logs && \
chown -R rustfs:rustfs /data /logs && \
chmod 0750 /data /logs
CMD ["rustfs", "/data"]
ENV RUSTFS_ADDRESS=":9000" \
RUSTFS_CONSOLE_ADDRESS=":9001" \
RUSTFS_ACCESS_KEY="rustfsadmin" \
RUSTFS_SECRET_KEY="rustfsadmin" \
RUSTFS_CONSOLE_ENABLE="true" \
RUSTFS_CORS_ALLOWED_ORIGINS="*" \
RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
RUSTFS_VOLUMES="/data" \
RUSTFS_OBS_LOGGER_LEVEL=warn \
RUSTFS_OBS_LOG_DIRECTORY=/logs \
RUSTFS_OBS_ENVIRONMENT=production
EXPOSE 9000 9001
VOLUME ["/data"]
USER rustfs
ENTRYPOINT ["/entrypoint.sh"]
CMD ["rustfs"]

115
Dockerfile.glibc Normal file
View File

@@ -0,0 +1,115 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
FROM ubuntu:24.04 AS build
ARG TARGETARCH
ARG RELEASE=latest
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
curl \
unzip \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /build
RUN set -eux; \
case "$TARGETARCH" in \
amd64) ARCH_SUBSTR="x86_64-gnu" ;; \
arm64) ARCH_SUBSTR="aarch64-gnu" ;; \
*) echo "Unsupported TARGETARCH=$TARGETARCH" >&2; exit 1 ;; \
esac; \
\
if [ "$RELEASE" = "latest" ]; then \
TAG="$(curl -fsSL https://api.github.com/repos/rustfs/rustfs/releases \
| grep -o '"tag_name": "[^"]*"' | cut -d'"' -f4 | head -n 1)"; \
else \
TAG="$RELEASE"; \
fi; \
\
URL="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG" \
| grep -o "\"browser_download_url\": \"[^\"]*${ARCH_SUBSTR}[^\"]*\\.zip\"" \
| cut -d'"' -f4 | head -n 1)"; \
\
if [ -z "$URL" ]; then echo "Failed to locate release asset for $ARCH_SUBSTR at tag $TAG" >&2; exit 1; fi; \
\
curl -fL "$URL" -o rustfs.zip; \
unzip -q rustfs.zip -d /build; \
\
if [ ! -x /build/rustfs ]; then \
BIN_PATH="$(unzip -Z -1 rustfs.zip | grep -E '(^|/)rustfs$' | head -n 1 || true)"; \
if [ -n "$BIN_PATH" ]; then \
mkdir -p /build/.tmp && unzip -q rustfs.zip "$BIN_PATH" -d /build/.tmp && \
mv "/build/.tmp/$BIN_PATH" /build/rustfs; \
fi; \
fi; \
[ -x /build/rustfs ] || { echo "rustfs binary not found in asset" >&2; exit 1; }; \
chmod +x /build/rustfs; \
rm -rf rustfs.zip /build/.tmp || true
FROM ubuntu:24.04
ARG RELEASE=latest
ARG BUILD_DATE
ARG VCS_REF
LABEL name="RustFS" \
vendor="RustFS Team" \
maintainer="RustFS Team <dev@rustfs.com>" \
version="v${RELEASE#v}" \
release="${RELEASE}" \
build-date="${BUILD_DATE}" \
vcs-ref="${VCS_REF}" \
summary="High-performance distributed object storage system (glibc)" \
url="https://rustfs.com" \
license="Apache-2.0"
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
curl \
&& rm -rf /var/lib/apt/lists/*
COPY --from=build /build/rustfs /usr/bin/rustfs
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /usr/bin/rustfs /entrypoint.sh
RUN groupadd -g 10001 rustfs && \
useradd -u 10001 -g rustfs -m -s /sbin/nologin rustfs && \
mkdir -p /data /logs && \
chown -R rustfs:rustfs /data /logs && \
chmod 0750 /data /logs
ENV RUSTFS_ADDRESS=":9000" \
RUSTFS_CONSOLE_ADDRESS=":9001" \
RUSTFS_ACCESS_KEY="rustfsadmin" \
RUSTFS_SECRET_KEY="rustfsadmin" \
RUSTFS_CONSOLE_ENABLE="true" \
RUSTFS_CORS_ALLOWED_ORIGINS="*" \
RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
RUSTFS_VOLUMES="/data" \
RUSTFS_OBS_LOGGER_LEVEL=warn \
RUSTFS_OBS_LOG_DIRECTORY=/logs \
RUSTFS_OBS_ENVIRONMENT=production
EXPOSE 9000 9001
VOLUME ["/data"]
USER rustfs
ENTRYPOINT ["/entrypoint.sh"]
CMD ["rustfs"]

View File

@@ -1,121 +0,0 @@
# Multi-stage Dockerfile for RustFS
# Supports cross-compilation for amd64 and arm64 architectures
ARG TARGETPLATFORM
ARG BUILDPLATFORM
# Build stage
FROM --platform=$BUILDPLATFORM rust:1.85-bookworm AS builder
# Install required build dependencies
RUN apt-get update && apt-get install -y \
wget \
git \
curl \
unzip \
gcc \
pkg-config \
libssl-dev \
lld \
&& rm -rf /var/lib/apt/lists/*
# Install cross-compilation tools for ARM64
RUN if [ "$TARGETPLATFORM" = "linux/arm64" ]; then \
apt-get update && \
apt-get install -y gcc-aarch64-linux-gnu && \
rm -rf /var/lib/apt/lists/*; \
fi
# Install protoc
RUN wget https://github.com/protocolbuffers/protobuf/releases/download/v31.1/protoc-31.1-linux-x86_64.zip \
&& unzip protoc-31.1-linux-x86_64.zip -d protoc3 \
&& mv protoc3/bin/* /usr/local/bin/ && chmod +x /usr/local/bin/protoc \
&& mv protoc3/include/* /usr/local/include/ && rm -rf protoc-31.1-linux-x86_64.zip protoc3
# Install flatc
RUN wget https://github.com/google/flatbuffers/releases/download/v25.2.10/Linux.flatc.binary.g++-13.zip \
&& unzip Linux.flatc.binary.g++-13.zip \
&& mv flatc /usr/local/bin/ && chmod +x /usr/local/bin/flatc && rm -rf Linux.flatc.binary.g++-13.zip
# Set up Rust targets based on platform
RUN case "$TARGETPLATFORM" in \
"linux/amd64") rustup target add x86_64-unknown-linux-gnu ;; \
"linux/arm64") rustup target add aarch64-unknown-linux-gnu ;; \
*) echo "Unsupported platform: $TARGETPLATFORM" && exit 1 ;; \
esac
# Set up environment for cross-compilation
ENV CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc
ENV CC_aarch64_unknown_linux_gnu=aarch64-linux-gnu-gcc
ENV CXX_aarch64_unknown_linux_gnu=aarch64-linux-gnu-g++
WORKDIR /usr/src/rustfs
# Copy Cargo files for dependency caching
COPY Cargo.toml Cargo.lock ./
COPY */Cargo.toml ./*/
# Create dummy main.rs files for dependency compilation
RUN find . -name "Cargo.toml" -not -path "./Cargo.toml" | \
xargs -I {} dirname {} | \
xargs -I {} sh -c 'mkdir -p {}/src && echo "fn main() {}" > {}/src/main.rs'
# Build dependencies only (cache layer)
RUN case "$TARGETPLATFORM" in \
"linux/amd64") cargo build --release --target x86_64-unknown-linux-gnu ;; \
"linux/arm64") cargo build --release --target aarch64-unknown-linux-gnu ;; \
esac
# Copy source code
COPY . .
# Generate protobuf code
RUN cargo run --bin gproto
# Build the actual application
RUN case "$TARGETPLATFORM" in \
"linux/amd64") \
cargo build --release --target x86_64-unknown-linux-gnu --bin rustfs && \
cp target/x86_64-unknown-linux-gnu/release/rustfs /usr/local/bin/rustfs \
;; \
"linux/arm64") \
cargo build --release --target aarch64-unknown-linux-gnu --bin rustfs && \
cp target/aarch64-unknown-linux-gnu/release/rustfs /usr/local/bin/rustfs \
;; \
esac
# Runtime stage - Ubuntu minimal for better compatibility
FROM ubuntu:22.04
# Install runtime dependencies
RUN apt-get update && apt-get install -y \
ca-certificates \
tzdata \
wget \
&& rm -rf /var/lib/apt/lists/*
# Create rustfs user and group
RUN groupadd -g 1000 rustfs && \
useradd -d /app -g rustfs -u 1000 -s /bin/bash rustfs
WORKDIR /app
# Create data directories
RUN mkdir -p /data/rustfs{0,1,2,3} && \
chown -R rustfs:rustfs /data /app
# Copy binary from builder stage
COPY --from=builder /usr/local/bin/rustfs /app/rustfs
RUN chmod +x /app/rustfs && chown rustfs:rustfs /app/rustfs
# Switch to non-root user
USER rustfs
# Expose ports
EXPOSE 9000 9001
# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:9000/health || exit 1
# Set default command
CMD ["/app/rustfs"]

View File

@@ -1,21 +0,0 @@
FROM ubuntu:latest
# RUN apk add --no-cache <package-name>
# 如果 rustfs 有依赖,可以在这里添加,例如:
# RUN apk add --no-cache openssl
# RUN apk add --no-cache bash # 安装 Bash
WORKDIR /app
# 创建与 RUSTFS_VOLUMES 一致的目录
RUN mkdir -p /root/data/target/volume/test1 /root/data/target/volume/test2 /root/data/target/volume/test3 /root/data/target/volume/test4
# COPY ./target/x86_64-unknown-linux-musl/release/rustfs /app/rustfs
COPY ./target/x86_64-unknown-linux-gnu/release/rustfs /app/rustfs
RUN chmod +x /app/rustfs
EXPOSE 9000
EXPOSE 9002
CMD ["/app/rustfs"]

241
Dockerfile.source Normal file
View File

@@ -0,0 +1,241 @@
# syntax=docker/dockerfile:1.6
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Multi-stage Dockerfile for RustFS - LOCAL DEVELOPMENT ONLY
#
# IMPORTANT: This Dockerfile builds RustFS from source for local development and testing.
# CI/CD uses the production Dockerfile with prebuilt binaries instead.
#
# Example:
# docker build -f Dockerfile.source -t rustfs:dev-local .
# docker run --rm -p 9000:9000 rustfs:dev-local
#
# Supports cross-compilation for amd64 and arm64 via TARGETPLATFORM.
ARG TARGETPLATFORM
ARG BUILDPLATFORM
# -----------------------------
# Build stage
# -----------------------------
FROM rust:1.91-trixie AS builder
# Re-declare args after FROM
ARG TARGETPLATFORM
ARG BUILDPLATFORM
# Debug: print platforms
RUN echo "Build info -> BUILDPLATFORM=${BUILDPLATFORM}, TARGETPLATFORM=${TARGETPLATFORM}"
# Install build toolchain and headers
# Use distro packages for protoc/flatc to avoid host-arch mismatch
RUN set -eux; \
export DEBIAN_FRONTEND=noninteractive; \
apt-get update; \
apt-get install -y --no-install-recommends \
build-essential \
ca-certificates \
curl \
git \
pkg-config \
libssl-dev \
lld \
protobuf-compiler \
flatbuffers-compiler \
gcc-aarch64-linux-gnu \
gcc-x86-64-linux-gnu; \
rm -rf /var/lib/apt/lists/*
# Optional: cross toolchain for aarch64 (only when targeting linux/arm64)
RUN set -eux; \
if [ "${TARGETPLATFORM:-linux/amd64}" = "linux/arm64" ]; then \
export DEBIAN_FRONTEND=noninteractive; \
apt-get update; \
apt-get install -y --no-install-recommends gcc-aarch64-linux-gnu; \
rm -rf /var/lib/apt/lists/*; \
fi
# Add Rust targets for both arches (to support cross-builds on multi-arch runners)
RUN set -eux; \
rustup target add x86_64-unknown-linux-gnu aarch64-unknown-linux-gnu; \
rustup component add rust-std-x86_64-unknown-linux-gnu rust-std-aarch64-unknown-linux-gnu
# Cross-compilation environment (used only when targeting aarch64)
ENV CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc
ENV CC_aarch64_unknown_linux_gnu=aarch64-linux-gnu-gcc
ENV CXX_aarch64_unknown_linux_gnu=aarch64-linux-gnu-g++
ENV CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-linux-gnu-gcc
ENV CC_x86_64_unknown_linux_gnu=x86_64-linux-gnu-gcc
ENV CXX_x86_64_unknown_linux_gnu=x86_64-linux-gnu-g++
WORKDIR /usr/src/rustfs
# Layered copy to maximize caching:
# 1) top-level manifests
COPY Cargo.toml Cargo.lock ./
# 2) workspace member manifests (adjust if workspace layout changes)
COPY rustfs/Cargo.toml rustfs/Cargo.toml
COPY crates/*/Cargo.toml crates/
# Pre-fetch dependencies for better caching
RUN --mount=type=cache,target=/usr/local/cargo/registry \
--mount=type=cache,target=/usr/local/cargo/git \
cargo fetch --locked || true
# 3) copy full sources (this is the main cache invalidation point)
COPY . .
# Generate static files
RUN ./scripts/static.sh
# Cargo build configuration for lean release artifacts
ENV CARGO_NET_GIT_FETCH_WITH_CLI=true \
CARGO_REGISTRIES_CRATES_IO_PROTOCOL=sparse \
CARGO_INCREMENTAL=0 \
CARGO_PROFILE_RELEASE_DEBUG=false \
CARGO_PROFILE_RELEASE_SPLIT_DEBUGINFO=off \
CARGO_PROFILE_RELEASE_STRIP=symbols
# Generate protobuf/flatbuffers code (uses protoc/flatc from distro)
RUN --mount=type=cache,target=/usr/local/cargo/registry \
--mount=type=cache,target=/usr/local/cargo/git \
--mount=type=cache,target=/usr/src/rustfs/target \
cargo run --bin gproto
# Build RustFS (target depends on TARGETPLATFORM)
RUN --mount=type=cache,target=/usr/local/cargo/registry \
--mount=type=cache,target=/usr/local/cargo/git \
--mount=type=cache,target=/usr/src/rustfs/target \
set -eux; \
case "${TARGETPLATFORM:-linux/amd64}" in \
linux/amd64) \
echo "Building for x86_64-unknown-linux-gnu"; \
cargo build --release --locked --target x86_64-unknown-linux-gnu --bin rustfs -j "$(nproc)"; \
install -m 0755 target/x86_64-unknown-linux-gnu/release/rustfs /usr/local/bin/rustfs \
;; \
linux/arm64) \
echo "Building for aarch64-unknown-linux-gnu"; \
cargo build --release --locked --target aarch64-unknown-linux-gnu --bin rustfs -j "$(nproc)"; \
install -m 0755 target/aarch64-unknown-linux-gnu/release/rustfs /usr/local/bin/rustfs \
;; \
*) \
echo "Unsupported TARGETPLATFORM=${TARGETPLATFORM}" >&2; exit 1 \
;; \
esac
# -----------------------------
# Development stage (keeps toolchain)
# -----------------------------
FROM builder AS dev
ARG BUILD_DATE
ARG VCS_REF
LABEL name="RustFS (dev-source)" \
maintainer="RustFS Team" \
build-date="${BUILD_DATE}" \
vcs-ref="${VCS_REF}" \
description="RustFS - local development with Rust toolchain."
# Install runtime dependencies that might be missing in partial builder
# (builder already has build-essential, lld, etc.)
WORKDIR /app
ENV CARGO_INCREMENTAL=1
# Ensure we have the same default env vars available
ENV RUSTFS_ADDRESS=":9000" \
RUSTFS_ACCESS_KEY="rustfsadmin" \
RUSTFS_SECRET_KEY="rustfsadmin" \
RUSTFS_CONSOLE_ENABLE="true" \
RUSTFS_VOLUMES="/data" \
RUST_LOG="warn" \
RUSTFS_OBS_LOG_DIRECTORY="/logs" \
RUSTFS_USERNAME="rustfs" \
RUSTFS_GROUPNAME="rustfs" \
RUSTFS_UID="10001" \
RUSTFS_GID="10001"
# Note: We don't COPY source here because we expect it to be mounted at /app
# We rely on cargo run to build and run
EXPOSE 9000 9001
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
CMD ["cargo", "run", "--bin", "rustfs", "--"]
# -----------------------------
# Runtime stage (Ubuntu minimal)
# -----------------------------
FROM ubuntu:22.04
ARG BUILD_DATE
ARG VCS_REF
LABEL name="RustFS (dev-local)" \
maintainer="RustFS Team" \
build-date="${BUILD_DATE}" \
vcs-ref="${VCS_REF}" \
description="RustFS - local development image built from source (NOT for production)."
# Minimal runtime deps: certificates + tzdata + coreutils (for chroot --userspec)
RUN set -eux; \
export DEBIAN_FRONTEND=noninteractive; \
apt-get update; \
apt-get install -y --no-install-recommends \
ca-certificates \
tzdata \
coreutils; \
rm -rf /var/lib/apt/lists/*
# Create a conventional runtime user/group (final switch happens in entrypoint via chroot --userspec)
RUN set -eux; \
groupadd -g 10001 rustfs; \
useradd -u 10001 -g rustfs -M -s /usr/sbin/nologin rustfs
WORKDIR /app
# Prepare data/log directories with sane defaults
RUN set -eux; \
mkdir -p /data /logs; \
chown -R rustfs:rustfs /data /logs /app; \
chmod 0750 /data /logs
# Copy the freshly built binary and the entrypoint
COPY --from=builder /usr/local/bin/rustfs /usr/bin/rustfs
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /usr/bin/rustfs /entrypoint.sh
# Default environment (override in docker run/compose as needed)
ENV RUSTFS_ADDRESS=":9000" \
RUSTFS_ACCESS_KEY="rustfsadmin" \
RUSTFS_SECRET_KEY="rustfsadmin" \
RUSTFS_CONSOLE_ENABLE="true" \
RUSTFS_VOLUMES="/data" \
RUST_LOG="warn" \
RUSTFS_USERNAME="rustfs" \
RUSTFS_GROUPNAME="rustfs" \
RUSTFS_UID="10001" \
RUSTFS_GID="10001"
EXPOSE 9000
VOLUME ["/data"]
# Keep root here; entrypoint will drop privileges using chroot --userspec
ENTRYPOINT ["/entrypoint.sh"]
CMD ["/usr/bin/rustfs"]

258
Justfile Normal file
View File

@@ -0,0 +1,258 @@
DOCKER_CLI := env("DOCKER_CLI", "docker")
IMAGE_NAME := env("IMAGE_NAME", "rustfs:v1.0.0")
DOCKERFILE_SOURCE := env("DOCKERFILE_SOURCE", "Dockerfile.source")
DOCKERFILE_PRODUCTION := env("DOCKERFILE_PRODUCTION", "Dockerfile")
CONTAINER_NAME := env("CONTAINER_NAME", "rustfs-dev")
[group("📒 Help")]
[private]
default:
@just --list --list-heading $'🦀 RustFS justfile manual page:\n'
[doc("show help")]
[group("📒 Help")]
help: default
[doc("run `cargo fmt` to format codes")]
[group("👆 Code Quality")]
fmt:
@echo "🔧 Formatting code..."
cargo fmt --all
[doc("run `cargo fmt` in check mode")]
[group("👆 Code Quality")]
fmt-check:
@echo "📝 Checking code formatting..."
cargo fmt --all --check
[doc("run `cargo clippy`")]
[group("👆 Code Quality")]
clippy:
@echo "🔍 Running clippy checks..."
cargo clippy --all-targets --all-features --fix --allow-dirty -- -D warnings
[doc("run `cargo check`")]
[group("👆 Code Quality")]
check:
@echo "🔨 Running compilation check..."
cargo check --all-targets
[doc("run `cargo test`")]
[group("👆 Code Quality")]
test:
@echo "🧪 Running tests..."
cargo nextest run --all --exclude e2e_test
cargo test --all --doc
[doc("run `fmt` `clippy` `check` `test` at once")]
[group("👆 Code Quality")]
pre-commit: fmt clippy check test
@echo "✅ All pre-commit checks passed!"
[group("🤔 Git")]
setup-hooks:
@echo "🔧 Setting up git hooks..."
chmod +x .git/hooks/pre-commit
@echo "✅ Git hooks setup complete!"
[doc("use `release` mode for building")]
[group("🔨 Build")]
build:
@echo "🔨 Building RustFS using build-rustfs.sh script..."
./build-rustfs.sh
[doc("use `debug` mode for building")]
[group("🔨 Build")]
build-dev:
@echo "🔨 Building RustFS in development mode..."
./build-rustfs.sh --dev
[group("🔨 Build")]
[private]
build-target target:
@echo "🔨 Building rustfs for {{ target }}..."
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
./build-rustfs.sh --platform {{ target }}
[doc("use `x86_64-unknown-linux-musl` target for building")]
[group("🔨 Build")]
build-musl: (build-target "x86_64-unknown-linux-musl")
[doc("use `x86_64-unknown-linux-gnu` target for building")]
[group("🔨 Build")]
build-gnu: (build-target "x86_64-unknown-linux-gnu")
[doc("use `aarch64-unknown-linux-musl` target for building")]
[group("🔨 Build")]
build-musl-arm64: (build-target "aarch64-unknown-linux-musl")
[doc("use `aarch64-unknown-linux-gnu` target for building")]
[group("🔨 Build")]
build-gnu-arm64: (build-target "aarch64-unknown-linux-gnu")
[doc("build and deploy to server")]
[group("🔨 Build")]
deploy-dev ip: build-musl
@echo "🚀 Deploying to dev server: {{ ip }}"
./scripts/dev_deploy.sh {{ ip }}
[group("🔨 Build")]
[private]
build-cross-all-pre:
@echo "🔧 Building all target architectures..."
@echo "💡 On macOS/Windows, use 'make docker-dev' for reliable multi-arch builds"
@echo "🔨 Generating protobuf code..."
-cargo run --bin gproto
[doc("build all targets at once")]
[group("🔨 Build")]
build-cross-all: build-cross-all-pre && build-gnu build-gnu-arm64 build-musl build-musl-arm64
# ========================================================================================
# Docker Multi-Architecture Builds (Primary Methods)
# ========================================================================================
[doc("build an image and run it")]
[group("🐳 Build Image")]
build-docker os="rockylinux9.3" cli=(DOCKER_CLI) dockerfile=(DOCKERFILE_SOURCE):
#!/usr/bin/env bash
SOURCE_BUILD_IMAGE_NAME="rustfs/rustfs-{{ os }}:v1"
SOURCE_BUILD_CONTAINER_NAME="rustfs-{{ os }}-build"
BUILD_CMD="/root/.cargo/bin/cargo build --release --bin rustfs --target-dir /root/s3-rustfs/target/{{ os }}"
echo "🐳 Building RustFS using Docker ({{ os }})..."
{{ cli }} buildx build -t $SOURCE_BUILD_IMAGE_NAME -f {{ dockerfile }} .
{{ cli }} run --rm --name $SOURCE_BUILD_CONTAINER_NAME -v $(pwd):/root/s3-rustfs -it $SOURCE_BUILD_IMAGE_NAME $BUILD_CMD
[doc("build an image")]
[group("🐳 Build Image")]
docker-buildx:
@echo "🏗️ Building multi-architecture production Docker images with buildx..."
./docker-buildx.sh
[doc("build an image and push it")]
[group("🐳 Build Image")]
docker-buildx-push:
@echo "🚀 Building and pushing multi-architecture production Docker images with buildx..."
./docker-buildx.sh --push
[doc("build an image with a version")]
[group("🐳 Build Image")]
docker-buildx-version version:
@echo "🏗️ Building multi-architecture production Docker images (version: {{ version }}..."
./docker-buildx.sh --release {{ version }}
[doc("build an image with a version and push it")]
[group("🐳 Build Image")]
docker-buildx-push-version version:
@echo "🚀 Building and pushing multi-architecture production Docker images (version: {{ version }}..."
./docker-buildx.sh --release {{ version }} --push
[doc("build an image with a version and push it to registry")]
[group("🐳 Build Image")]
docker-dev-push registry cli=(DOCKER_CLI) source=(DOCKERFILE_SOURCE):
@echo "🚀 Building and pushing multi-architecture development Docker images..."
@echo "💡 push to registry: {{ registry }}"
{{ cli }} buildx build \
--platform linux/amd64,linux/arm64 \
--file {{ source }} \
--tag {{ registry }}/rustfs:source-latest \
--tag {{ registry }}/rustfs:dev-latest \
--push \
.
# Local production builds using direct buildx (alternative to docker-buildx.sh)
[group("🐳 Build Image")]
docker-buildx-production-local cli=(DOCKER_CLI) source=(DOCKERFILE_PRODUCTION):
@echo "🏗️ Building single-architecture production Docker image locally..."
@echo "💡 Alternative to docker-buildx.sh for local testing"
{{ cli }} buildx build \
--file {{ source }} \
--tag rustfs:production-latest \
--tag rustfs:latest \
--load \
--build-arg RELEASE=latest \
.
# Development/Source builds using direct buildx commands
[group("🐳 Build Image")]
docker-dev cli=(DOCKER_CLI) source=(DOCKERFILE_SOURCE):
@echo "🏗️ Building multi-architecture development Docker images with buildx..."
@echo "💡 This builds from source code and is intended for local development and testing"
@echo "⚠️ Multi-arch images cannot be loaded locally, use docker-dev-push to push to registry"
{{ cli }} buildx build \
--platform linux/amd64,linux/arm64 \
--file {{ source }} \
--tag rustfs:source-latest \
--tag rustfs:dev-latest \
.
[group("🐳 Build Image")]
docker-dev-local cli=(DOCKER_CLI) source=(DOCKERFILE_SOURCE):
@echo "🏗️ Building single-architecture development Docker image for local use..."
@echo "💡 This builds from source code for the current platform and loads locally"
{{ cli }} buildx build \
--file {{ source }} \
--tag rustfs:source-latest \
--tag rustfs:dev-latest \
--load \
.
# ========================================================================================
# Single Architecture Docker Builds (Traditional)
# ========================================================================================
[group("🐳 Build Image")]
docker-build-production cli=(DOCKER_CLI) source=(DOCKERFILE_PRODUCTION):
@echo "🏗️ Building single-architecture production Docker image..."
@echo "💡 Consider using 'make docker-buildx-production-local' for multi-arch support"
{{ cli }} build -f {{ source }} -t rustfs:latest .
[group("🐳 Build Image")]
docker-build-source cli=(DOCKER_CLI) source=(DOCKERFILE_SOURCE):
@echo "🏗️ Building single-architecture source Docker image..."
@echo "💡 Consider using 'make docker-dev-local' for multi-arch support"
{{ cli }} build -f {{ source }} -t rustfs:source .
# ========================================================================================
# Development Environment
# ========================================================================================
[group("🏃 Running")]
dev-env-start cli=(DOCKER_CLI) source=(DOCKERFILE_SOURCE) container=(CONTAINER_NAME):
@echo "🚀 Starting development environment..."
{{ cli }} buildx build \
--file {{ source }} \
--tag rustfs:dev \
--load \
.
-{{ cli }} stop {{ container }} 2>/dev/null
-{{ cli }} rm {{ container }} 2>/dev/null
{{ cli }} run -d --name {{ container }} \
-p 9010:9010 -p 9000:9000 \
-v {{ invocation_directory() }}:/workspace \
-it rustfs:dev
[group("🏃 Running")]
dev-env-stop cli=(DOCKER_CLI) container=(CONTAINER_NAME):
@echo "🛑 Stopping development environment..."
-{{ cli }} stop {{ container }} 2>/dev/null
-{{ cli }} rm {{ container }} 2>/dev/null
[group("🏃 Running")]
dev-env-restart: dev-env-stop dev-env-start
[group("👍 E2E")]
e2e-server:
sh scripts/run.sh
[group("👍 E2E")]
probe-e2e:
sh scripts/probe.sh
[doc("inspect one image")]
[group("🚚 Other")]
docker-inspect-multiarch image cli=(DOCKER_CLI):
@echo "🔍 Inspecting multi-architecture image: {{ image }}"
{{ cli }} buildx imagetools inspect {{ image }}

243
Makefile
View File

@@ -1,200 +1,81 @@
###########
# 远程开发,需要 VSCode 安装 Dev Containers, Remote SSH, Remote Explorer
# Remote development requires VSCode with Dev Containers, Remote SSH, Remote Explorer
# https://code.visualstudio.com/docs/remote/containers
###########
.PHONY: SHELL
# Makefile global config
# Use config.mak to override any of the following variables.
# Do not make changes here.
.DEFAULT_GOAL := help
.EXPORT_ALL_VARIABLES:
.ONESHELL:
.SILENT:
NUM_CORES := $(shell nproc 2>/dev/null || sysctl -n hw.ncpu)
MAKEFLAGS += -j$(NUM_CORES) -l$(NUM_CORES)
MAKEFLAGS += --silent
SHELL := $(shell which bash)
.SHELLFLAGS = -eu -o pipefail -c
DOCKER_CLI ?= docker
IMAGE_NAME ?= rustfs:v1.0.0
CONTAINER_NAME ?= rustfs-dev
DOCKERFILE_PATH = $(shell pwd)/.docker
# Code quality and formatting targets
.PHONY: fmt
fmt:
@echo "🔧 Formatting code..."
cargo fmt --all
.PHONY: fmt-check
fmt-check:
@echo "📝 Checking code formatting..."
cargo fmt --all --check
.PHONY: clippy
clippy:
@echo "🔍 Running clippy checks..."
cargo clippy --all-targets --all-features -- -D warnings
.PHONY: check
check:
@echo "🔨 Running compilation check..."
cargo check --all-targets
.PHONY: test
test:
@echo "🧪 Running tests..."
cargo nextest run --all --exclude e2e_test
cargo test --all --doc
.PHONY: pre-commit
pre-commit: fmt clippy check test
@echo "✅ All pre-commit checks passed!"
.PHONY: setup-hooks
setup-hooks:
@echo "🔧 Setting up git hooks..."
chmod +x .git/hooks/pre-commit
@echo "✅ Git hooks setup complete!"
.PHONY: init-devenv
init-devenv:
$(DOCKER_CLI) build -t $(IMAGE_NAME) -f $(DOCKERFILE_PATH)/Dockerfile.devenv .
$(DOCKER_CLI) stop $(CONTAINER_NAME)
$(DOCKER_CLI) rm $(CONTAINER_NAME)
$(DOCKER_CLI) run -d --name $(CONTAINER_NAME) -p 9010:9010 -p 9000:9000 -v $(shell pwd):/root/s3-rustfs -it $(IMAGE_NAME)
.PHONY: start
start:
$(DOCKER_CLI) start $(CONTAINER_NAME)
.PHONY: stop
stop:
$(DOCKER_CLI) stop $(CONTAINER_NAME)
.PHONY: e2e-server
e2e-server:
sh $(shell pwd)/scripts/run.sh
.PHONY: probe-e2e
probe-e2e:
sh $(shell pwd)/scripts/probe.sh
# make BUILD_OS=ubuntu22.04 build
# in target/ubuntu22.04/release/rustfs
# make BUILD_OS=rockylinux9.3 build
# in target/rockylinux9.3/release/rustfs
# Docker build configurations
DOCKERFILE_PRODUCTION = Dockerfile
DOCKERFILE_SOURCE = Dockerfile.source
BUILD_OS ?= rockylinux9.3
.PHONY: build
build: ROCKYLINUX_BUILD_IMAGE_NAME = rustfs-$(BUILD_OS):v1
build: ROCKYLINUX_BUILD_CONTAINER_NAME = rustfs-$(BUILD_OS)-build
build: BUILD_CMD = /root/.cargo/bin/cargo build --release --bin rustfs --target-dir /root/s3-rustfs/target/$(BUILD_OS)
build:
$(DOCKER_CLI) build -t $(ROCKYLINUX_BUILD_IMAGE_NAME) -f $(DOCKERFILE_PATH)/Dockerfile.$(BUILD_OS) .
$(DOCKER_CLI) run --rm --name $(ROCKYLINUX_BUILD_CONTAINER_NAME) -v $(shell pwd):/root/s3-rustfs -it $(ROCKYLINUX_BUILD_IMAGE_NAME) $(BUILD_CMD)
.PHONY: build-musl
build-musl:
@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
cargo build --target x86_64-unknown-linux-musl --bin rustfs -r
# Makefile colors config
bold := $(shell tput bold)
normal := $(shell tput sgr0)
errorTitle := $(shell tput setab 1 && tput bold && echo '\n')
recommendation := $(shell tput setab 4)
underline := $(shell tput smul)
reset := $(shell tput -Txterm sgr0)
black := $(shell tput setaf 0)
red := $(shell tput setaf 1)
green := $(shell tput setaf 2)
yellow := $(shell tput setaf 3)
blue := $(shell tput setaf 4)
magenta := $(shell tput setaf 5)
cyan := $(shell tput setaf 6)
white := $(shell tput setaf 7)
.PHONY: build-gnu
build-gnu:
@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
cargo build --target x86_64-unknown-linux-gnu --bin rustfs -r
define HEADER
How to use me:
# To get help for each target
${bold}make help${reset}
.PHONY: deploy-dev
deploy-dev: build-musl
@echo "🚀 Deploying to dev server: $${IP}"
./scripts/dev_deploy.sh $${IP}
# To run and execute a target
${bold}make ${cyan}<target>${reset}
# Multi-architecture Docker build targets
.PHONY: docker-build-multiarch
docker-build-multiarch:
@echo "🏗️ Building multi-architecture Docker images..."
./scripts/build-docker-multiarch.sh
💡 For more help use 'make help', 'make help-build' or 'make help-docker'
.PHONY: docker-build-multiarch-push
docker-build-multiarch-push:
@echo "🚀 Building and pushing multi-architecture Docker images..."
./scripts/build-docker-multiarch.sh --push
🦀 RustFS Makefile Help:
.PHONY: docker-build-multiarch-version
docker-build-multiarch-version:
@if [ -z "$(VERSION)" ]; then \
echo "❌ 错误: 请指定版本, 例如: make docker-build-multiarch-version VERSION=v1.0.0"; \
exit 1; \
fi
@echo "🏗️ Building multi-architecture Docker images (version: $(VERSION))..."
./scripts/build-docker-multiarch.sh --version $(VERSION)
📋 Main Command Categories:
make help-build # Show build-related help
make help-docker # Show Docker-related help
.PHONY: docker-push-multiarch-version
docker-push-multiarch-version:
@if [ -z "$(VERSION)" ]; then \
echo "❌ 错误: 请指定版本, 例如: make docker-push-multiarch-version VERSION=v1.0.0"; \
exit 1; \
fi
@echo "🚀 Building and pushing multi-architecture Docker images (version: $(VERSION))..."
./scripts/build-docker-multiarch.sh --version $(VERSION) --push
🔧 Code Quality:
make fmt # Format code
make clippy # Run clippy checks
make test # Run tests
make pre-commit # Run all pre-commit checks
.PHONY: docker-build-ubuntu
docker-build-ubuntu:
@echo "🏗️ Building multi-architecture Ubuntu Docker images..."
./scripts/build-docker-multiarch.sh --type ubuntu
🚀 Quick Start:
make build # Build RustFS binary
make docker-dev-local # Build development Docker image (local)
make dev-env-start # Start development environment
.PHONY: docker-build-rockylinux
docker-build-rockylinux:
@echo "🏗️ Building multi-architecture RockyLinux Docker images..."
./scripts/build-docker-multiarch.sh --type rockylinux
.PHONY: docker-build-devenv
docker-build-devenv:
@echo "🏗️ Building multi-architecture development environment Docker images..."
./scripts/build-docker-multiarch.sh --type devenv
endef
export HEADER
.PHONY: docker-build-all-types
docker-build-all-types:
@echo "🏗️ Building all multi-architecture Docker image types..."
./scripts/build-docker-multiarch.sh --type production
./scripts/build-docker-multiarch.sh --type ubuntu
./scripts/build-docker-multiarch.sh --type rockylinux
./scripts/build-docker-multiarch.sh --type devenv
-include $(addsuffix /*.mak, $(shell find .config/make -type d))
.PHONY: docker-inspect-multiarch
docker-inspect-multiarch:
@if [ -z "$(IMAGE)" ]; then \
echo "❌ 错误: 请指定镜像, 例如: make docker-inspect-multiarch IMAGE=rustfs/rustfs:latest"; \
exit 1; \
fi
@echo "🔍 Inspecting multi-architecture image: $(IMAGE)"
docker buildx imagetools inspect $(IMAGE)
.PHONY: build-cross-all
build-cross-all:
@echo "🔧 Building all target architectures..."
@if ! command -v cross &> /dev/null; then \
echo "📦 Installing cross..."; \
cargo install cross; \
fi
@echo "🔨 Generating protobuf code..."
cargo run --bin gproto || true
@echo "🔨 Building x86_64-unknown-linux-musl..."
cargo build --release --target x86_64-unknown-linux-musl --bin rustfs
@echo "🔨 Building aarch64-unknown-linux-gnu..."
cross build --release --target aarch64-unknown-linux-gnu --bin rustfs
@echo "✅ All architectures built successfully!"
.PHONY: help-docker
help-docker:
@echo "🐳 Docker 多架构构建帮助:"
@echo ""
@echo "基本构建:"
@echo " make docker-build-multiarch # 构建多架构镜像(不推送)"
@echo " make docker-build-multiarch-push # 构建并推送多架构镜像"
@echo ""
@echo "版本构建:"
@echo " make docker-build-multiarch-version VERSION=v1.0.0 # 构建指定版本"
@echo " make docker-push-multiarch-version VERSION=v1.0.0 # 构建并推送指定版本"
@echo ""
@echo "镜像类型:"
@echo " make docker-build-ubuntu # 构建 Ubuntu 镜像"
@echo " make docker-build-rockylinux # 构建 RockyLinux 镜像"
@echo " make docker-build-devenv # 构建开发环境镜像"
@echo " make docker-build-all-types # 构建所有类型镜像"
@echo ""
@echo "辅助工具:"
@echo " make build-cross-all # 构建所有架构的二进制文件"
@echo " make docker-inspect-multiarch IMAGE=xxx # 检查镜像的架构支持"
@echo ""
@echo "环境变量 (在推送时需要设置):"
@echo " DOCKERHUB_USERNAME Docker Hub 用户名"
@echo " DOCKERHUB_TOKEN Docker Hub 访问令牌"
@echo " GITHUB_TOKEN GitHub 访问令牌"

233
README.md
View File

@@ -1,95 +1,210 @@
[![RustFS](https://rustfs.com/images/rustfs-github.png)](https://rustfs.com)
<p align="center">RustFS is a high-performance distributed object storage software built using Rust</p>
[![RustFS](https://repository-images.githubusercontent.com/722597620/0fa936a2-8164-4f53-867f-def4beb64b21)](https://rustfs.com)
<p align="center">RustFS is a high-performance, distributed object storage system built in Rust.</p>
<p align="center">
<a href="https://github.com/rustfs/rustfs/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/rustfs/rustfs/actions/workflows/ci.yml/badge.svg" /></a>
<a href="https://github.com/rustfs/rustfs/actions/workflows/docker.yml"><img alt="Build and Push Docker Images" src="https://github.com/rustfs/rustfs/actions/workflows/docker.yml/badge.svg" /></a>
<img alt="GitHub commit activity" src="https://img.shields.io/github/commit-activity/m/rustfs/rustfs"/>
<img alt="Github Last Commit" src="https://img.shields.io/github/last-commit/rustfs/rustfs"/>
<a href="https://hellogithub.com/repository/rustfs/rustfs" target="_blank"><img src="https://abroad.hellogithub.com/v1/widgets/recommend.svg?rid=b95bcb72bdc340b68f16fdf6790b7d5b&claim_uid=MsbvjYeLDKAH457&theme=small" alt="FeaturedHelloGitHub" /></a>
</p>
<p align="center">
<a href="https://docs.rustfs.com/en/introduction.html">Getting Started</a>
· <a href="https://docs.rustfs.com/en/">Docs</a>
<a href="https://trendshift.io/repositories/14181" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14181" alt="rustfs%2Frustfs | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
<a href="https://runacap.com/ross-index/q4-2025/" target="_blank" rel="noopener"><img style="width: 260px; height: 55px" src="https://runacap.com/wp-content/uploads/2026/01/ROSS_badge_white_Q4_2025.svg" alt="ROSS Index - Fastest Growing Open-Source Startups in Q4 2025 | Runa Capital" height="55" /></a>
</p>
<p align="center">
<a href="https://docs.rustfs.com/installation/">Getting Started</a>
· <a href="https://docs.rustfs.com/">Docs</a>
· <a href="https://github.com/rustfs/rustfs/issues">Bug reports</a>
· <a href="https://github.com/rustfs/rustfs/discussions">Discussions</a>
</p>
<p align="center">
English | <a href="https://github.com/rustfs/rustfs/blob/main/README_ZH.md">简体中文</a> |
<!-- Keep these links. Translations will automatically update with the README. -->
<a href="https://readme-i18n.com/rustfs/rustfs?lang=de">Deutsch</a> |
<a href="https://readme-i18n.com/rustfs/rustfs?lang=es">Español</a> |
<a href="https://readme-i18n.com/rustfs/rustfs?lang=fr">français</a> |
<a href="https://readme-i18n.com/rustfs/rustfs?lang=ja">日本語</a> |
<a href="https://readme-i18n.com/rustfs/rustfs?lang=ko">한국어</a> |
<a href="https://readme-i18n.com/rustfs/rustfs?lang=pt">Português</a> |
English | <a href="https://github.com/rustfs/rustfs/blob/main/README_ZH.md">简体中文</a> |
<a href="https://readme-i18n.com/rustfs/rustfs?lang=de">Deutsch</a> |
<a href="https://readme-i18n.com/rustfs/rustfs?lang=es">Español</a> |
<a href="https://readme-i18n.com/rustfs/rustfs?lang=fr">français</a> |
<a href="https://readme-i18n.com/rustfs/rustfs?lang=ja">日本語</a> |
<a href="https://readme-i18n.com/rustfs/rustfs?lang=ko">한국어</a> |
<a href="https://readme-i18n.com/rustfs/rustfs?lang=pt">Portuguese</a> |
<a href="https://readme-i18n.com/rustfs/rustfs?lang=ru">Русский</a>
</p>
RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. Along with MinIO, it shares a range of advantages such as simplicity, S3 compatibility, open-source nature, support for data lakes, AI, and big data. Furthermore, it has a better and more user-friendly open-source license in comparison to other storage systems, being constructed under the Apache license. As Rust serves as its foundation, RustFS provides faster speed and safer distributed features for high-performance object storage.
RustFS is a high-performance, distributed object storage system built in Rustone of the most loved programming languages worldwide. RustFS combines the simplicity of MinIO with the memory safety and raw performance of Rust. It offers full S3 compatibility, is completely open-source, and is optimized for data lakes, AI, and big data workloads.
Unlike other storage systems, RustFS is released under the permissible Apache 2.0 license, avoiding the restrictions of AGPL. With Rust as its foundation, RustFS delivers superior speed and secure distributed features for next-generation object storage.
> ⚠️ **RustFS is under rapid development. Do NOT use in production environments!**
## Feature & Status
## Features
- **High Performance**: Built with Rust to ensure maximum speed and resource efficiency.
- **Distributed Architecture**: Scalable and fault-tolerant design suitable for large-scale deployments.
- **S3 Compatibility**: Seamless integration with existing S3-compatible applications and tools.
- **OpenStack Swift API**: Native support for Swift protocol with Keystone authentication.
- **OpenStack Keystone Integration**: Native support for OpenStack Keystone authentication with X-Auth-Token headers.
- **Data Lake Support**: Optimized for high-throughput big data and AI workloads.
- **Open Source**: Licensed under Apache 2.0, encouraging unrestricted community contributions and commercial usage.
- **User-Friendly**: Designed with simplicity in mind for easy deployment and management.
- **High Performance**: Built with Rust, ensuring speed and efficiency.
- **Distributed Architecture**: Scalable and fault-tolerant design for large-scale deployments.
- **S3 Compatibility**: Seamless integration with existing S3-compatible applications.
- **Data Lake Support**: Optimized for big data and AI workloads.
- **Open Source**: Licensed under Apache 2.0, encouraging community contributions and transparency.
- **User-Friendly**: Designed with simplicity in mind, making it easy to deploy and manage.
| Feature | Status | Feature | Status |
| :---------------------- | :----------- | :----------------------- | :--------------- |
| **S3 Core Features** | ✅ Available | **Bitrot Protection** | ✅ Available |
| **Upload / Download** | ✅ Available | **Single Node Mode** | ✅ Available |
| **Versioning** | ✅ Available | **Bucket Replication** | ✅ Available |
| **Logging** | ✅ Available | **Lifecycle Management** | 🚧 Under Testing |
| **Event Notifications** | ✅ Available | **Distributed Mode** | 🚧 Under Testing |
| **K8s Helm Charts** | ✅ Available | **RustFS KMS** | 🚧 Under Testing |
| **Keystone Auth** | ✅ Available | **Multi-Tenancy** | ✅ Available |
| **Swift API** | ✅ Available | **Swift Metadata Ops** | 🚧 Partial |
## RustFS vs MinIO
## RustFS vs MinIO Performance
Stress test server parameters
**Stress Test Environment:**
| Type | parameter | Remark |
| - | - | - |
|CPU | 2 Core | Intel Xeon(Sapphire Rapids) Platinum 8475B , 2.7/3.2 GHz| |
|Memory| 4GB |   |
|Network | 15Gbp |   |
|Driver | 40GB x 4 | IOPS 3800 / Driver |
| Type | Parameter | Remark |
| ------- | --------- | -------------------------------------------------------- |
| CPU | 2 Core | Intel Xeon (Sapphire Rapids) Platinum 8475B, 2.7/3.2 GHz |
| Memory | 4GB | |
| Network | 15Gbps | |
| Drive | 40GB x 4 | IOPS 3800 / Drive |
<https://github.com/user-attachments/assets/2e4979b5-260c-4f2c-ac12-c87fd558072a>
### RustFS vs Other object storage
### RustFS vs Other Object Storage
| RustFS | Other object storage|
| - | - |
| Powerful Console | Simple and useless Console |
| Developed based on Rust language, memory is safer | Developed in Go or C, with potential issues like memory GC/leaks |
| Does not report logs to third-party countries | Reporting logs to other third countries may violate national security laws |
| Licensed under Apache, more business-friendly | AGPL V3 License and other License, polluted open source and License traps, infringement of intellectual property rights |
| Comprehensive S3 support, works with domestic and international cloud providers | Full support for S3, but no local cloud vendor support |
| Rust-based development, strong support for secure and innovative devices | Poor support for edge gateways and secure innovative devices|
| Stable commercial prices, free community support | High pricing, with costs up to $250,000 for 1PiB |
| No risk | Intellectual property risks and risks of prohibited uses |
| Feature | RustFS | Other Object Storage |
| :--------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------- |
| **Console Experience** | **Powerful Console**<br>Comprehensive management interface. | **Basic / Limited Console**<br>Often overly simple or lacking critical features. |
| **Language & Safety** | **Rust-based**<br>Memory safety by design. | **Go or C-based**<br>Potential for memory GC pauses or leaks. |
| **Data Sovereignty** | **No Telemetry / Full Compliance**<br>Guards against unauthorized cross-border data egress. Compliant with GDPR (EU/UK), CCPA (US), and APPI (Japan). | **Potential Risk**<br>Possible legal exposure and unwanted data telemetry. |
| **Licensing** | **Permissive Apache 2.0**<br>Business-friendly, no "poison pill" clauses. | **Restrictive AGPL v3**<br>Risk of license traps and intellectual property pollution. |
| **Compatibility** | **100% S3 Compatible**<br>Works with any cloud provider or client, anywhere. | **Variable Compatibility**<br>May lack support for local cloud vendors or specific APIs. |
| **Edge & IoT** | **Strong Edge Support**<br>Ideal for secure, innovative edge devices. | **Weak Edge Support**<br>Often too heavy for edge gateways. |
| **Risk Profile** | **Enterprise Risk Mitigation**<br>Clear IP rights and safe for commercial use. | **Legal Risks**<br>Intellectual property ambiguity and usage restrictions. |
## Staying ahead
Star RustFS on GitHub and be instantly notified of new releases.
<img src="https://github.com/user-attachments/assets/7ee40bb4-3e46-4eac-b0d0-5fbeb85ff8f3" />
## Quickstart
To get started with RustFS, follow these steps:
1. **One-click installation script (Option 1)**
### 1. One-click Installation (Option 1)
```bash
curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh
```
```bash
curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh
```
2. **Docker Quick Start (Option 2)**
### 2\. Docker Quick Start (Option 2)
```bash
podman run -d -p 9000:9000 -p 9001:9001 -v /data:/data quay.io/rustfs/rustfs
```
The RustFS container runs as a non-root user `rustfs` (UID `10001`). If you run Docker with `-v` to mount a host directory, please ensure the host directory owner is set to `10001`, otherwise you will encounter permission denied errors.
```bash
# Create data and logs directories
mkdir -p data logs
3. **Access the Console**: Open your web browser and navigate to `http://localhost:9001` to access the RustFS console, default username and password is `rustfsadmin` .
4. **Create a Bucket**: Use the console to create a new bucket for your objects.
5. **Upload Objects**: You can upload files directly through the console or use S3-compatible APIs to interact with your RustFS instance.
# Change the owner of these directories
chown -R 10001:10001 data logs
# Using latest version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# Using specific version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-alpha.76
```
If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
```bash
podman run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
```
You can also use Docker Compose. Using the `docker-compose.yml` file in the root directory:
```bash
docker compose --profile observability up -d
```
Similarly, you can run the command with podman
```bash
podman compose --profile observability up -d
```
**NOTE**: We recommend reviewing the `docker-compose.yaml` file before running. It defines several services including Grafana, Prometheus, and Jaeger, which are helpful for RustFS observability. If you wish to start Redis or Nginx containers, you can specify the corresponding profiles.
### 3\. Build from Source (Option 3) - Advanced Users
For developers who want to build RustFS Docker images from source with multi-architecture support:
```bash
# Build multi-architecture images locally
./docker-buildx.sh --build-arg RELEASE=latest
# Build and push to registry
./docker-buildx.sh --push
# Build specific version
./docker-buildx.sh --release v1.0.0 --push
# Build for custom registry
./docker-buildx.sh --registry your-registry.com --namespace yourname --push
```
The `docker-buildx.sh` script supports:
- **Multi-architecture builds**: `linux/amd64`, `linux/arm64`
- **Automatic version detection**: Uses git tags or commit hashes
- **Registry flexibility**: Supports Docker Hub, GitHub Container Registry, etc.
- **Build optimization**: Includes caching and parallel builds
You can also use Make targets for convenience:
```bash
make docker-buildx # Build locally
make docker-buildx-push # Build and push
make docker-buildx-version VERSION=v1.0.0 # Build specific version
make help-docker # Show all Docker-related commands
```
> **Heads-up (macOS cross-compilation)**: macOS keeps the default `ulimit -n` at 256, so `cargo zigbuild` or `./build-rustfs.sh --platform ...` may fail with `ProcessFdQuotaExceeded` when targeting Linux. The build script attempts to raise the limit automatically, but if you still see the warning, run `ulimit -n 4096` (or higher) in your shell before building.
### 4\. Build with Helm Chart (Option 4) - Cloud Native
Follow the instructions in the [Helm Chart README](https://charts.rustfs.com/) to install RustFS on a Kubernetes cluster.
### 5\. Nix Flake (Option 5)
If you have [Nix with flakes enabled](https://nixos.wiki/wiki/Flakes#Enable_flakes):
```bash
# Run directly without installing
nix run github:rustfs/rustfs
# Build the binary
nix build github:rustfs/rustfs
./result/bin/rustfs --help
# Or from a local checkout
nix build
nix run
```
---
### Accessing RustFS
1. **Access the Console**: Open your web browser and navigate to `http://localhost:9001` to access the RustFS console.
- Default credentials: `rustfsadmin` / `rustfsadmin`
2. **Create a Bucket**: Use the console to create a new bucket for your objects.
3. **Upload Objects**: You can upload files directly through the console or use S3-compatible APIs/clients to interact with your RustFS instance.
**NOTE**: To access the RustFS instance via `https`, please refer to the [TLS Configuration Docs](https://docs.rustfs.com/integration/tls-configured.html).
## Documentation
@@ -97,7 +212,7 @@ For detailed documentation, including configuration options, API references, and
## Getting Help
If you have any questions or need assistance, you can:
If you have any questions or need assistance:
- Check the [FAQ](https://github.com/rustfs/rustfs/discussions/categories/q-a) for common issues and solutions.
- Join our [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) to ask questions and share your experiences.
@@ -112,8 +227,8 @@ If you have any questions or need assistance, you can:
## Contact
- **Bugs**: [GitHub Issues](https://github.com/rustfs/rustfs/issues)
- **Business**: <hello@rustfs.com>
- **Jobs**: <jobs@rustfs.com>
- **Business**: [hello@rustfs.com](mailto:hello@rustfs.com)
- **Jobs**: [jobs@rustfs.com](mailto:jobs@rustfs.com)
- **General Discussion**: [GitHub Discussions](https://github.com/rustfs/rustfs/discussions)
- **Contributing**: [CONTRIBUTING.md](CONTRIBUTING.md)
@@ -122,9 +237,13 @@ If you have any questions or need assistance, you can:
RustFS is a community-driven project, and we appreciate all contributions. Check out the [Contributors](https://github.com/rustfs/rustfs/graphs/contributors) page to see the amazing people who have helped make RustFS better.
<a href="https://github.com/rustfs/rustfs/graphs/contributors">
<img src="https://contrib.rocks/image?repo=rustfs/rustfs" />
<img src="https://opencollective.com/rustfs/contributors.svg?width=890&limit=500&button=false" alt="Contributors" />
</a>
## Star History
[![Star History Chart](https://api.star-history.com/svg?repos=rustfs/rustfs&type=date&legend=top-left)](https://www.star-history.com/#rustfs/rustfs&type=date&legend=top-left)
## License
[Apache 2.0](https://opensource.org/licenses/Apache-2.0)

View File

@@ -1,116 +1,216 @@
[![RustFS](https://rustfs.com/images/rustfs-github.png)](https://rustfs.com)
[![RustFS](https://github.com/user-attachments/assets/1b5afcd6-a2c3-47ff-8bc3-ce882b0ddca7)](https://rustfs.com.cn)
<p align="center">RustFS 是一个使用 Rust 构建的高性能分布式对象存储软件</p >
<p align="center">RustFS 是一个基于 Rust 构建的高性能分布式对象存储系统。</p>
<p align="center">
<a href="https://github.com/rustfs/rustfs/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/rustfs/rustfs/actions/workflows/ci.yml/badge.svg" /></a>
<a href="https://github.com/rustfs/rustfs/actions/workflows/docker.yml"><img alt="Build and Push Docker Images" src="https://github.com/rustfs/rustfs/actions/workflows/docker.yml/badge.svg" /></a>
<img alt="GitHub commit activity" src="https://img.shields.io/github/commit-activity/m/rustfs/rustfs"/>
<img alt="Github Last Commit" src="https://img.shields.io/github/last-commit/rustfs/rustfs"/>
</p >
<a href="https://github.com/rustfs/rustfs/actions/workflows/docker.yml"><img alt="构建并推送 Docker 镜像" src="https://github.com/rustfs/rustfs/actions/workflows/docker.yml/badge.svg" /></a>
<img alt="GitHub 提交活跃度" src="https://img.shields.io/github/commit-activity/m/rustfs/rustfs"/>
<img alt="Github 最新提交" src="https://img.shields.io/github/last-commit/rustfs/rustfs"/>
<a href="https://hellogithub.com/repository/rustfs/rustfs" target="_blank"><img src="https://abroad.hellogithub.com/v1/widgets/recommend.svg?rid=b95bcb72bdc340b68f16fdf6790b7d5b&claim_uid=MsbvjYeLDKAH457&theme=small" alt="FeaturedHelloGitHub" /></a>
</p>
<p align="center">
<a href="https://docs.rustfs.com/zh/introduction.html">快速开始</a >
· <a href="https://docs.rustfs.com/zh/">文档</a >
· <a href="https://github.com/rustfs/rustfs/issues">问题报告</a >
· <a href="https://github.com/rustfs/rustfs/discussions">讨论</a >
</p >
<a href="https://trendshift.io/repositories/14181" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14181" alt="rustfs%2Frustfs | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
<a href="https://runacap.com/ross-index/q4-2025/" target="_blank" rel="noopener"><img style="width: 260px; height: 55px" src="https://runacap.com/wp-content/uploads/2026/01/ROSS_badge_white_Q4_2025.svg" alt="ROSS Index - Fastest Growing Open-Source Startups in Q4 2025 | Runa Capital" height="55" /></a>
</p>
<p align="center">
<a href="https://github.com/rustfs/rustfs/blob/main/README.md">English</a > | 简体中文
</p >
<a href="https://docs.rustfs.com/installation/">快速开始</a>
· <a href="https://docs.rustfs.com/">文档</a>
· <a href="https://github.com/rustfs/rustfs/issues">报告 Bug</a>
· <a href="https://github.com/rustfs/rustfs/discussions">社区讨论</a>
</p>
RustFS 是一个使用 Rust全球最受欢迎的编程语言之一构建的高性能分布式对象存储软件。与 MinIO 一样它具有简单性、S3 兼容性、开源特性以及对数据湖、AI 和大数据的支持等一系列优势。此外,与其他存储系统相比,它采用 Apache 许可证构建,拥有更好、更用户友好的开源许可证。由于以 Rust 为基础RustFS 为高性能对象存储提供了更快的速度和更安全的分布式功能。
<p align="center">
<a href="https://github.com/rustfs/rustfs/blob/main/README.md">English</a> | 简体中文 |
<a href="https://readme-i18n.com/rustfs/rustfs?lang=de">Deutsch</a> |
<a href="https://readme-i18n.com/rustfs/rustfs?lang=es">Español</a> |
<a href="https://readme-i18n.com/rustfs/rustfs?lang=fr">français</a> |
<a href="https://readme-i18n.com/rustfs/rustfs?lang=ja">日本語</a> |
<a href="https://readme-i18n.com/rustfs/rustfs?lang=ko">한국어</a> |
<a href="https://readme-i18n.com/rustfs/rustfs?lang=pt">Portuguese</a> |
<a href="https://readme-i18n.com/rustfs/rustfs?lang=ru">Русский</a>
</p>
## 特性
RustFS 是一个基于 Rust 构建的高性能分布式对象存储系统。Rust 是全球最受开发者喜爱的编程语言之一RustFS 完美结合了 MinIO 的简洁性与 Rust 的内存安全及高性能优势。它提供完整的 S3 兼容性完全开源并专为数据湖、人工智能AI和大数据负载进行了优化。
- **高性能**:使用 Rust 构建,确保速度和效率
与其他存储系统不同RustFS 采用更宽松、商业友好的 Apache 2.0 许可证,避免了 AGPL 协议的限制。以 Rust 为基石RustFS 为下一代对象存储提供了更快的速度和更安全的分布式特性
## 特征和功能状态
- **高性能**:基于 Rust 构建,确保极致的速度和资源效率。
- **分布式架构**:可扩展且容错的设计,适用于大规模部署。
- **S3 兼容性**:与现有 S3 兼容应用程序无缝集成。
- **数据湖支持**针对大数据和 AI 工作负载进行了优化。
- **开源**:采用 Apache 2.0 许可证,鼓励社区贡献和透明度
- **用户友好**:设计简,易于部署和管理。
- **S3 兼容性**:与现有 S3 兼容应用和工具无缝集成。
- **数据湖支持**专为高吞吐量的大数据和 AI 工作负载优化。
- **完全开源**:采用 Apache 2.0 许可证,鼓励社区贡献和商业使用
- **简单易用**:设计简,易于部署和管理。
## RustFS vs MinIO
| 功能 | 状态 | 功能 | 状态 |
| :----------------- | :------ | :---------------------- | :-------- |
| **S3 核心功能** | ✅ 可用 | **Bitrot (防数据腐烂)** | ✅ 可用 |
| **上传 / 下载** | ✅ 可用 | **单机模式** | ✅ 可用 |
| **版本控制** | ✅ 可用 | **存储桶复制** | ✅ 可用 |
| **日志功能** | ✅ 可用 | **生命周期管理** | 🚧 测试中 |
| **事件通知** | ✅ 可用 | **分布式模式** | 🚧 测试中 |
| **K8s Helm Chart** | ✅ 可用 | **OPA (策略引擎)** | 🚧 测试中 |
压力测试服务器参数
## RustFS vs MinIO 性能对比
| 类型 | 参数 | 备注 |
| - | - | - |
|CPU | 2 核心 | Intel Xeon(Sapphire Rapids) Platinum 8475B , 2.7/3.2 GHz| |
|内存| 4GB | |
|网络 | 15Gbp | |
|驱动器 | 40GB x 4 | IOPS 3800 / 驱动器 |
**压力测试环境参数:**
| 类型 | 参数 | 备注 |
| ---- | -------- | --------------------------------------------------------- |
| CPU | 2 核 | Intel Xeon (Sapphire Rapids) Platinum 8475B , 2.7/3.2 GHz |
| 内存 | 4GB |   |
| 网络 | 15Gbps |   |
| 硬盘 | 40GB x 4 | IOPS 3800 / Drive |
<https://github.com/user-attachments/assets/2e4979b5-260c-4f2c-ac12-c87fd558072a>
### RustFS vs 其他对象存储
| RustFS | 其他对象存储|
| - | - |
| 强大的控制台 | 简单且无用的控制台 |
| 基于 Rust 语言开发,内存安全 | 使用 Go 或 C 开发存在内存 GC/泄漏潜在问题 |
| 不向第三方国家报告日志 | 向其他第三方国家报告日志可能违反国家安全法律 |
| 采用 Apache 许可证,对商业友好 | AGPL V3 许可证等其他许可证,污染开源和许可证陷阱,侵犯知识产权 |
| 全面的 S3 支持,适用于国内外云提供商 | 完全支持 S3但不支持本地云厂商 |
| 基于 Rust 开发,对安全创新设备有强大支持 | 对边缘网关和安全创新设备支持较差|
| 稳定的商业价格,免费社区支持 | 高昂的定价,1PiB 成本高达 $250,000 |
| 无风险 | 知识产权风险和禁止使用的风险 |
| 特性 | RustFS | 其他对象存储 |
| :------------- | :------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------- |
| **控制台体验** | **功能强大的控制台**<br>提供全面的管理界面。 | **基础/简陋的控制台**<br>通常功能过于简单或缺失关键特性。 |
| **语言与安全** | **基于 Rust 开发**<br>天生的内存安全。 | **基于 Go 或 C 开发**<br>存在内存 GC 停顿或内存泄漏潜在风险。 |
| **数据主权** | **无遥测 / 完全合规**<br>防止未经授权的数据跨境传输。完全符合 GDPR (欧盟/英国)、CCPA (美国) 和 APPI (日本) 等法规。 | **潜在风险**<br>可能存在法律风险和隐蔽的数据遥测Telemetry |
| **开源协议** | **宽松的 Apache 2.0**<br>商业友好,无“毒丸”条款。 | **受限的 AGPL v3**<br>存在许可证陷阱知识产权污染的风险。 |
| **兼容性** | **100% S3 兼容**<br>适用于任何云提供商和客户端,随处运行。 | **兼容性不一**<br>虽然支持 S3但可能缺乏对本地云厂商或特定 API 的支持。 |
| **边缘与 IoT** | **强大的边缘支持**<br>非常适合安全创新的边缘设备。 | **边缘支持较弱**<br>对于边缘网关来说通常过于沉重。 |
| **成本** | **稳定且免费**<br>免费社区支持,稳定的商业定价。 | **高昂成本**<br>1PiB 成本可能高达 250,000 美元。 |
| **风险控制** | **企业级风险规避**<br>清晰的知识产权,商业使用安全无忧。 | **法律风险**<br>知识产权归属模糊及使用限制风险。 |
## 保持领先
在 GitHub 上为 RustFS 点赞,即可第一时间收到新版本发布通知。
<img src="https://github.com/user-attachments/assets/7ee40bb4-3e46-4eac-b0d0-5fbeb85ff8f3" />
## 快速开始
要开始使用 RustFS请按照以下步骤操作
请按照以下步骤快速上手 RustFS
1. **一键脚本快速启动 (方案一)**
### 1. 一键安装脚本 (选项 1)
```bash
curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh
```
```bash
curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh
```
2. **Docker快速启动(方案二)**
### 2\. Docker 快速启动 (选项 2)
```bash
podman run -d -p 9000:9000 -p 9001:9001 -v /data:/data quay.io/rustfs/rustfs
```
RustFS 容器以非 root 用户 `rustfs` (UID `10001`) 运行。如果您使用 Docker 的 `-v` 参数挂载宿主机目录,请务必确保宿主机目录的所有者已更改为 `10001`,否则会遇到权限拒绝错误。
```bash
# 创建数据和日志目录
mkdir -p data logs
3. **访问控制台**:打开 Web 浏览器并导航到 `http://localhost:9001` 以访问 RustFS 控制台,默认的用户名和密码是 `rustfsadmin` 。
4. **创建存储桶**:使用控制台为您的对象创建新的存储桶。
5. **上传对象**:您可以直接通过控制台上传文件,或使用 S3 兼容的 API 与您的 RustFS 实例交互。
# 更改这两个目录的所有者
chown -R 10001:10001 data logs
# 使用最新版本运行
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# 使用指定版本运行
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0.alpha.68
```
您也可以使用 Docker Compose。使用根目录下的 `docker-compose.yml` 文件:
```bash
docker compose --profile observability up -d
```
**注意**: 我们建议您在运行前查看 `docker-compose.yaml` 文件。该文件定义了包括 Grafana、Prometheus 和 Jaeger 在内的多个服务,有助于 RustFS 的可观测性监控。如果您还想启动 Redis 或 Nginx 容器,可以指定相应的 profile。
### 3\. 源码编译 (选项 3) - 进阶用户
适用于希望从源码构建支持多架构 RustFS Docker 镜像的开发者:
```bash
# 在本地构建多架构镜像
./docker-buildx.sh --build-arg RELEASE=latest
# 构建并推送到仓库
./docker-buildx.sh --push
# 构建指定版本
./docker-buildx.sh --release v1.0.0 --push
# 构建并推送到自定义仓库
./docker-buildx.sh --registry your-registry.com --namespace yourname --push
```
`docker-buildx.sh` 脚本支持:
- **多架构构建**: `linux/amd64`, `linux/arm64`
- **自动版本检测**: 使用 git tags 或 commit hash
- **灵活的仓库支持**: 支持 Docker Hub, GitHub Container Registry 等
- **构建优化**: 包含缓存和并行构建
为了方便起见,您也可以使用 Make 命令:
```bash
make docker-buildx # 本地构建
make docker-buildx-push # 构建并推送
make docker-buildx-version VERSION=v1.0.0 # 构建指定版本
make help-docker # 显示所有 Docker 相关命令
```
> **注意 (macOS 交叉编译)**: macOS 默认的 `ulimit -n` 限制为 256因此在使用 `cargo zigbuild` 或 `./build-rustfs.sh --platform ...` 交叉编译 Linux 版本时,可能会因 `ProcessFdQuotaExceeded` 失败。构建脚本会尝试自动提高限制,但如果您仍然看到警告,请在构建前在终端运行 `ulimit -n 4096` (或更高)。
### 4\. 使用 Helm Chart 安装 (选项 4) - 云原生环境
请按照 [Helm Chart README](https://charts.rustfs.com) 上的说明在 Kubernetes 集群上安装 RustFS。
---
### 访问 RustFS
1. **访问控制台**: 打开浏览器并访问 `http://localhost:9000` 进入 RustFS 控制台。
- 默认账号/密码: `rustfsadmin` / `rustfsadmin`
2. **创建存储桶**: 使用控制台为您​​的对象创建一个新的存储桶 (Bucket)。
3. **上传对象**: 您可以直接通过控制台上传文件,或使用 S3 兼容的 API/客户端与您的 RustFS 实例进行交互。
**注意**: 如果您希望通过 `https` 访问 RustFS 实例,请参考 [TLS 配置文档](https://docs.rustfs.com/integration/tls-configured.html)。
## 文档
有关详细文档包括配置选项、API 参考和高级用法,请访问我们的[文档](https://docs.rustfs.com)。
有关详细文档包括配置选项、API 参考和高级用法,请访问我们的 [官方文档](https://docs.rustfs.com)。
## 获取帮助
如果您有任何问题或需要帮助,您可以
如果您有任何问题或需要帮助:
- 查看[常见问题解答](https://github.com/rustfs/rustfs/discussions/categories/q-a)以获取常见问题和解决方案。
- 加入我们的 [GitHub 讨论](https://github.com/rustfs/rustfs/discussions)提问分享您的经验。
- 在我们的 [GitHub Issues](https://github.com/rustfs/rustfs/issues) 页面上开启问题,报告错误或功能请求。
- 查看 [FAQ](https://github.com/rustfs/rustfs/discussions/categories/q-a) 寻找常见问题和解决方案。
- 加入我们的 [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) 提问分享您的经验。
- 在我们的 [GitHub Issues](https://github.com/rustfs/rustfs/issues) 页面提交 Bug 报告或功能请求。
## 链接
- [文档](https://docs.rustfs.com) - 您应该阅读的手册
- [更新日志](https://docs.rustfs.com/changelog) - 我们破坏和修复的内容
- [GitHub 讨论](https://github.com/rustfs/rustfs/discussions) - 社区所在
- [官方文档](https://docs.rustfs.com) - 必读手册
- [更新日志](https://github.com/rustfs/rustfs/releases) - 版本变更记录
- [社区讨论](https://github.com/rustfs/rustfs/discussions) - 社区交流
## 联系
## 联系方式
- **错误报告**[GitHub Issues](https://github.com/rustfs/rustfs/issues)
- **商务合作**<hello@rustfs.com>
- **招聘**<jobs@rustfs.com>
- **一般讨论**[GitHub 讨论](https://github.com/rustfs/rustfs/discussions)
- **贡献**[CONTRIBUTING.md](CONTRIBUTING.md)
- **Bug 反馈**: [GitHub Issues](https://github.com/rustfs/rustfs/issues)
- **商务合作**: [hello@rustfs.com](mailto:hello@rustfs.com)
- **工作机会**: [jobs@rustfs.com](mailto:jobs@rustfs.com)
- **一般讨论**: [GitHub Discussions](https://github.com/rustfs/rustfs/discussions)
- **贡献指南**: [CONTRIBUTING.md](https://www.google.com/search?q=CONTRIBUTING.md)
## 贡献者
RustFS 是一个社区驱动的项目,我们感谢所有的贡献。查看[贡献者](https://github.com/rustfs/rustfs/graphs/contributors)页面,了解帮助 RustFS 变得更好的杰出人员
RustFS 是一个社区驱动的项目,我们感谢所有的贡献。查看 [贡献者](https://github.com/rustfs/rustfs/graphs/contributors) 页面,看看那些让 RustFS 变得更好的了不起的人们
<a href="https://github.com/rustfs/rustfs/graphs/contributors">
<img src="https://contrib.rocks/image?repo=rustfs/rustfs" />
</a >
<img src="https://opencollective.com/rustfs/contributors.svg?width=890&limit=500&button=false" alt="Contributors" />
</a>
## Star 历史
[![Star History Chart](https://api.star-history.com/svg?repos=rustfs/rustfs&type=date&legend=top-left)](https://www.star-history.com/#rustfs/rustfs&type=date&legend=top-left)
## 许可证

View File

@@ -1,18 +1,40 @@
# Security Policy
## Security Philosophy
At RustFS, we take security seriously. We believe that **transparency leads to better security**. The more open our code is, the more eyes are on it, and the faster we can identify and resolve potential issues.
We highly value the contributions of the security community and welcome anyone to audit our code. Your efforts help us make RustFS safer for everyone.
## Supported Versions
Use this section to tell people about which versions of your project are
currently being supported with security updates.
To help us focus our security efforts, please refer to the table below to see which versions of RustFS are currently supported with security updates.
| Version | Supported |
| ------- | ------------------ |
| 1.x.x | :white_check_mark: |
| Latest | :white_check_mark: |
| < 1.0 | :x: |
## Reporting a Vulnerability
Use this section to tell people how to report a vulnerability.
If you discover a security vulnerability in RustFS, we appreciate your help in disclosing it to us responsibly.
Tell them where to go, how often they can expect to get an update on a
reported vulnerability, what to expect if the vulnerability is accepted or
declined, etc.
**Please do not open a public GitHub issue for security vulnerabilities.** Publicly disclosing a vulnerability can put the entire community at risk before a fix is available.
### How to Report
1. https://github.com/rustfs/rustfs/security/advisories/new
2. Please email us directly at: **security@rustfs.com**
In your email, please include:
1. **Description**: A detailed description of the vulnerability.
2. **Steps to Reproduce**: Steps or a script to reproduce the issue.
3. **Impact**: The potential impact of the vulnerability.
### Our Response Process
1. **Acknowledgment**: We will acknowledge your email within 48 hours.
2. **Assessment**: We will investigate the issue and determine its severity.
3. **Fix & Disclosure**: We will work on a patch. Once the patch is released, we will publicly announce the vulnerability and acknowledge your contribution (unless you prefer to remain anonymous).
Thank you for helping keep RustFS and its users safe!

49
_typos.toml Normal file
View File

@@ -0,0 +1,49 @@
[default]
# # Ignore specific spell checking patterns
# extend-ignore-identifiers-re = [
# # Ignore common patterns in base64 encoding and hash values
# "[A-Za-z0-9+/]{8,}={0,2}", # base64 encoding
# "[A-Fa-f0-9]{8,}", # hexadecimal hash
# "[A-Za-z0-9_-]{20,}", # long random strings
# ]
# # Ignore specific regex patterns in content
# extend-ignore-re = [
# # Ignore hash values and encoded strings (base64 patterns)
# "(?i)[A-Za-z0-9+/]{8,}={0,2}",
# # Ignore long strings in quotes (usually hash or base64)
# '"[A-Za-z0-9+/=_-]{8,}"',
# # Ignore IV values and similar cryptographic strings
# '"[A-Za-z0-9+/=]{12,}"',
# # Ignore cryptographic signatures and keys (including partial strings)
# "[A-Za-z0-9+/]{6,}[A-Za-z0-9+/=]*",
# # Ignore base64-like strings in comments (common in examples)
# "//.*[A-Za-z0-9+/]{8,}[A-Za-z0-9+/=]*",
# ]
extend-ignore-re = [
# Ignore long strings in quotes (usually hash or base64)
'"[A-Za-z0-9+/=_-]{32,}"',
# Ignore IV values and similar cryptographic strings
'"[A-Za-z0-9+/=]{12,}"',
# Ignore cryptographic signatures and keys (including partial strings)
"[A-Za-z0-9+/]{16,}[A-Za-z0-9+/=]*",
]
[default.extend-words]
bui = "bui"
typ = "typ"
clen = "clen"
datas = "datas"
bre = "bre"
abd = "abd"
mak = "mak"
gae = "gae"
GAE = "GAE"
# s3-tests original test names (cannot be changed)
nonexisted = "nonexisted"
consts = "consts"
# Swift API - company/product names
Hashi = "Hashi" # HashiCorp
[files]
extend-exclude = []

610
build-rustfs.sh Executable file
View File

@@ -0,0 +1,610 @@
#!/usr/bin/env bash
# RustFS Binary Build Script
# This script compiles RustFS binaries for different platforms and architectures
set -e
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Auto-detect current platform
detect_platform() {
local arch=$(uname -m)
local os=$(uname -s | tr '[:upper:]' '[:lower:]')
case "$os" in
"linux")
case "$arch" in
"x86_64")
# Default to GNU for better compatibility
echo "x86_64-unknown-linux-gnu"
;;
"aarch64"|"arm64")
echo "aarch64-unknown-linux-gnu"
;;
"armv7l")
echo "armv7-unknown-linux-gnueabihf"
;;
"loongarch64")
echo "loongarch64-unknown-linux-gnu"
;;
*)
echo "unknown-platform"
;;
esac
;;
"darwin")
case "$arch" in
"x86_64")
echo "x86_64-apple-darwin"
;;
"arm64"|"aarch64")
echo "aarch64-apple-darwin"
;;
*)
echo "unknown-platform"
;;
esac
;;
*)
echo "unknown-platform"
;;
esac
}
# Cross-platform SHA256 checksum generation
generate_sha256() {
local file="$1"
local output_file="$2"
local os=$(uname -s | tr '[:upper:]' '[:lower:]')
case "$os" in
"linux")
if command -v sha256sum &> /dev/null; then
sha256sum "$file" > "$output_file"
elif command -v shasum &> /dev/null; then
shasum -a 256 "$file" > "$output_file"
else
print_message $RED "❌ No SHA256 command found (sha256sum or shasum)"
return 1
fi
;;
"darwin")
if command -v shasum &> /dev/null; then
shasum -a 256 "$file" > "$output_file"
elif command -v sha256sum &> /dev/null; then
sha256sum "$file" > "$output_file"
else
print_message $RED "❌ No SHA256 command found (shasum or sha256sum)"
return 1
fi
;;
*)
# Try common commands in order
if command -v sha256sum &> /dev/null; then
sha256sum "$file" > "$output_file"
elif command -v shasum &> /dev/null; then
shasum -a 256 "$file" > "$output_file"
else
print_message $RED "❌ No SHA256 command found"
return 1
fi
;;
esac
}
# Default values
OUTPUT_DIR="target/release"
PLATFORM=$(detect_platform) # Auto-detect current platform
BINARY_NAME="rustfs"
BUILD_TYPE="release"
SIGN=false
WITH_CONSOLE=true
FORCE_CONSOLE_UPDATE=false
CONSOLE_VERSION="latest"
SKIP_VERIFICATION=false
CUSTOM_PLATFORM=""
# Print usage
usage() {
echo "Usage: $0 [OPTIONS]"
echo ""
echo "Description:"
echo " Build RustFS binary for the current platform. Designed for CI/CD pipelines"
echo " where different runners build platform-specific binaries natively."
echo " Includes automatic verification to ensure the built binary is functional."
echo ""
echo "Options:"
echo " -o, --output-dir DIR Output directory (default: target/release)"
echo " -b, --binary-name NAME Binary name (default: rustfs)"
echo " -p, --platform TARGET Target platform (default: auto-detect)"
echo " Supported platforms:"
echo " x86_64-unknown-linux-gnu"
echo " aarch64-unknown-linux-gnu"
echo " loongarch64-unknown-linux-gnu"
echo " armv7-unknown-linux-gnueabihf"
echo " x86_64-unknown-linux-musl"
echo " aarch64-unknown-linux-musl"
echo " armv7-unknown-linux-musleabihf"
echo " x86_64-apple-darwin"
echo " aarch64-apple-darwin"
echo " x86_64-pc-windows-msvc"
echo " aarch64-pc-windows-msvc"
echo " --dev Build in dev mode"
echo " --sign Sign binaries after build"
echo " --with-console Download console static assets (default)"
echo " --no-console Skip console static assets"
echo " --force-console-update Force update console assets even if they exist"
echo " --console-version VERSION Console version to download (default: latest)"
echo " --skip-verification Skip binary verification after build"
echo " -h, --help Show this help message"
echo ""
echo "Examples:"
echo " $0 # Build for current platform (includes console assets)"
echo " $0 --dev # Development build"
echo " $0 --sign # Build and sign binary (release CI)"
echo " $0 --no-console # Build without console static assets"
echo " $0 --force-console-update # Force update console assets"
echo " $0 --platform x86_64-unknown-linux-musl # Build for specific platform"
echo " $0 --skip-verification # Skip binary verification (for cross-compilation)"
echo ""
echo "Detected platform: $(detect_platform)"
echo "CI Usage: Run this script on each platform's runner to build native binaries"
}
# Print colored message
print_message() {
local color=$1
local message=$2
echo -e "${color}${message}${NC}"
}
# Prevent zig/ld from hitting macOS file descriptor defaults during linking
ensure_file_descriptor_limit() {
local required_limit=4096
local current_limit
current_limit=$(ulimit -Sn 2>/dev/null || echo "")
if [ -z "$current_limit" ] || [ "$current_limit" = "unlimited" ]; then
return
fi
if (( current_limit >= required_limit )); then
return
fi
local hard_limit target_limit
hard_limit=$(ulimit -Hn 2>/dev/null || echo "")
target_limit=$required_limit
if [ -n "$hard_limit" ] && [ "$hard_limit" != "unlimited" ] && (( hard_limit < required_limit )); then
target_limit=$hard_limit
fi
if ulimit -Sn "$target_limit" 2>/dev/null; then
print_message $YELLOW "🔧 Increased open file limit from $current_limit to $target_limit to avoid ProcessFdQuotaExceeded"
else
print_message $YELLOW "⚠️ Unable to raise ulimit -n automatically (current: $current_limit, needed: $required_limit). Please run 'ulimit -n $required_limit' manually before building."
fi
}
# Get version from git
get_version() {
if git describe --abbrev=0 --tags >/dev/null 2>&1; then
git describe --abbrev=0 --tags
else
git rev-parse --short HEAD
fi
}
# Setup rust environment
setup_rust_environment() {
print_message $BLUE "🔧 Setting up Rust environment..."
# Install required target for current platform
print_message $YELLOW "Installing target: $PLATFORM"
rustup target add "$PLATFORM"
# Set up environment variables for musl targets
if [[ "$PLATFORM" == *"musl"* ]]; then
print_message $YELLOW "Setting up environment for musl target..."
export RUSTFLAGS="-C target-feature=-crt-static"
# For cargo-zigbuild, set up additional environment variables
if command -v cargo-zigbuild &> /dev/null; then
print_message $YELLOW "Configuring cargo-zigbuild for musl target..."
# Set environment variables for better musl support
export CC_x86_64_unknown_linux_musl="zig cc -target x86_64-linux-musl"
export CXX_x86_64_unknown_linux_musl="zig c++ -target x86_64-linux-musl"
export AR_x86_64_unknown_linux_musl="zig ar"
export CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER="zig cc -target x86_64-linux-musl"
export CC_aarch64_unknown_linux_musl="zig cc -target aarch64-linux-musl"
export CXX_aarch64_unknown_linux_musl="zig c++ -target aarch64-linux-musl"
export AR_aarch64_unknown_linux_musl="zig ar"
export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER="zig cc -target aarch64-linux-musl"
# Set environment variables for zstd-sys to avoid target parsing issues
export ZSTD_SYS_USE_PKG_CONFIG=1
export PKG_CONFIG_ALLOW_CROSS=1
fi
fi
# Install required tools
if [ "$SIGN" = true ]; then
if ! command -v minisign &> /dev/null; then
print_message $YELLOW "Installing minisign for binary signing..."
cargo install minisign
fi
fi
}
# Download console static assets
download_console_assets() {
local static_dir="rustfs/static"
local console_exists=false
# Check if console assets already exist
if [ -d "$static_dir" ] && [ -f "$static_dir/index.html" ]; then
console_exists=true
local static_size=$(du -sh "$static_dir" 2>/dev/null | cut -f1 || echo "unknown")
print_message $YELLOW "Console static assets already exist ($static_size)"
fi
# Determine if we need to download
local should_download=false
if [ "$WITH_CONSOLE" = true ]; then
if [ "$console_exists" = false ]; then
print_message $BLUE "🎨 Console assets not found, downloading..."
should_download=true
elif [ "$FORCE_CONSOLE_UPDATE" = true ]; then
print_message $BLUE "🎨 Force updating console assets..."
should_download=true
else
print_message $GREEN "✅ Console assets already available, skipping download"
fi
else
if [ "$console_exists" = true ]; then
print_message $GREEN "✅ Using existing console assets"
else
print_message $YELLOW "⚠️ Console assets not found. Use --download-console to download them."
fi
fi
if [ "$should_download" = true ]; then
print_message $BLUE "📥 Downloading console static assets..."
# Create static directory
mkdir -p "$static_dir"
# Download from GitHub Releases (consistent with Docker build)
local download_url
if [ "$CONSOLE_VERSION" = "latest" ]; then
print_message $YELLOW "Getting latest console release info..."
# For now, use dl.rustfs.com as fallback until GitHub Releases includes console assets
download_url="https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip"
else
download_url="https://dl.rustfs.com/artifacts/console/rustfs-console-${CONSOLE_VERSION}.zip"
fi
print_message $YELLOW "Downloading from: $download_url"
# Download with retries
local temp_file="console-assets-temp.zip"
local download_success=false
for i in {1..3}; do
if curl -L "$download_url" -o "$temp_file" --retry 3 --retry-delay 5 --max-time 300; then
download_success=true
break
else
print_message $YELLOW "Download attempt $i failed, retrying..."
sleep 2
fi
done
if [ "$download_success" = true ]; then
# Verify the downloaded file
if [ -f "$temp_file" ] && [ -s "$temp_file" ]; then
print_message $BLUE "📦 Extracting console assets..."
# Extract to static directory
if unzip -o "$temp_file" -d "$static_dir"; then
rm "$temp_file"
local final_size=$(du -sh "$static_dir" 2>/dev/null | cut -f1 || echo "unknown")
print_message $GREEN "✅ Console assets downloaded successfully ($final_size)"
else
print_message $RED "❌ Failed to extract console assets"
rm -f "$temp_file"
return 1
fi
else
print_message $RED "❌ Downloaded file is empty or invalid"
rm -f "$temp_file"
return 1
fi
else
print_message $RED "❌ Failed to download console assets after 3 attempts"
print_message $YELLOW "💡 Console assets are optional. Build will continue without them."
rm -f "$temp_file"
fi
fi
}
# Verify binary functionality
verify_binary() {
local binary_path="$1"
# Check if binary exists
if [ ! -f "$binary_path" ]; then
print_message $RED "❌ Binary file not found: $binary_path"
return 1
fi
# Check if binary is executable
if [ ! -x "$binary_path" ]; then
print_message $RED "❌ Binary is not executable: $binary_path"
return 1
fi
# Check basic functionality - try to run help command
print_message $YELLOW " Testing --help command..."
if ! "$binary_path" --help >/dev/null 2>&1; then
print_message $RED "❌ Binary failed to run --help command"
return 1
fi
# Check version command
print_message $YELLOW " Testing --version command..."
if ! "$binary_path" --version >/dev/null 2>&1; then
print_message $YELLOW "⚠️ Binary does not support --version command (this is optional)"
fi
# Try to get some basic info about the binary
local file_info=$(file "$binary_path" 2>/dev/null || echo "unknown")
print_message $YELLOW " Binary info: $file_info"
# Check if it's a valid ELF/Mach-O binary
if command -v readelf >/dev/null 2>&1; then
if readelf -h "$binary_path" >/dev/null 2>&1; then
print_message $YELLOW " ELF binary structure: valid"
fi
elif command -v otool >/dev/null 2>&1; then
if otool -h "$binary_path" >/dev/null 2>&1; then
print_message $YELLOW " Mach-O binary structure: valid"
fi
fi
return 0
}
# Build binary for current platform
build_binary() {
local version=$(get_version)
local output_file="${OUTPUT_DIR}/${PLATFORM}/${BINARY_NAME}"
print_message $BLUE "🏗️ Building for platform: $PLATFORM"
print_message $YELLOW " Version: $version"
print_message $YELLOW " Output: $output_file"
# Create output directory
mkdir -p "${OUTPUT_DIR}/${PLATFORM}"
# Simple build logic matching the working version (4fb4b353)
# Force rebuild by touching build.rs
touch rustfs/build.rs
# Determine build command based on platform and cross-compilation needs
local build_cmd=""
local current_platform=$(detect_platform)
print_message $BLUE "📦 Using working version build logic..."
# Check if we need cross-compilation
if [ "$PLATFORM" != "$current_platform" ]; then
# Cross-compilation needed
if [[ "$PLATFORM" == *"apple-darwin"* ]]; then
print_message $RED "❌ macOS cross-compilation not supported"
print_message $YELLOW "💡 macOS targets must be built natively on macOS runners"
return 1
elif [[ "$PLATFORM" == *"windows"* ]]; then
# Use cross for Windows ARM64
if ! command -v cross &> /dev/null; then
print_message $YELLOW "📦 Installing cross tool..."
cargo install cross --git https://github.com/cross-rs/cross
fi
build_cmd="cross build"
else
# Use zigbuild for Linux ARM64 (matches working version)
if ! command -v cargo-zigbuild &> /dev/null; then
print_message $RED "❌ cargo-zigbuild not found. Please install it first."
return 1
fi
build_cmd="cargo zigbuild"
fi
else
# Native compilation
build_cmd="RUSTFLAGS=-Clink-arg=-lm cargo build"
fi
if [ "$BUILD_TYPE" = "release" ]; then
build_cmd+=" --release"
fi
build_cmd+=" --target $PLATFORM"
build_cmd+=" -p rustfs --bins"
print_message $BLUE "📦 Executing: $build_cmd"
# Execute build (this matches exactly what the working version does)
if eval $build_cmd; then
print_message $GREEN "✅ Successfully built for $PLATFORM"
# Copy binary to output directory
cp "target/${PLATFORM}/${BUILD_TYPE}/${BINARY_NAME}" "$output_file"
# Generate checksums
print_message $BLUE "🔐 Generating checksums..."
(cd "${OUTPUT_DIR}/${PLATFORM}" && generate_sha256 "${BINARY_NAME}" "${BINARY_NAME}.sha256sum")
# Verify binary functionality (if not skipped)
if [ "$SKIP_VERIFICATION" = false ]; then
print_message $BLUE "🔍 Verifying binary functionality..."
if verify_binary "$output_file"; then
print_message $GREEN "✅ Binary verification passed"
else
print_message $RED "❌ Binary verification failed"
return 1
fi
else
print_message $YELLOW "⚠️ Binary verification skipped by user request"
fi
# Sign binary if requested
if [ "$SIGN" = true ]; then
print_message $BLUE "✍️ Signing binary..."
(cd "${OUTPUT_DIR}/${PLATFORM}" && minisign -S -m "${BINARY_NAME}" -s ~/.minisign/minisign.key)
fi
print_message $GREEN "✅ Build completed successfully"
else
print_message $RED "❌ Failed to build for $PLATFORM"
return 1
fi
}
# Main build function
build_rustfs() {
local version=$(get_version)
print_message $BLUE "🚀 Starting RustFS binary build process..."
print_message $YELLOW " Version: $version"
print_message $YELLOW " Platform: $PLATFORM"
print_message $YELLOW " Output Directory: $OUTPUT_DIR"
print_message $YELLOW " Build Type: $BUILD_TYPE"
print_message $YELLOW " Sign: $SIGN"
print_message $YELLOW " With Console: $WITH_CONSOLE"
if [ "$WITH_CONSOLE" = true ]; then
print_message $YELLOW " Console Version: $CONSOLE_VERSION"
print_message $YELLOW " Force Console Update: $FORCE_CONSOLE_UPDATE"
fi
print_message $YELLOW " Skip Verification: $SKIP_VERIFICATION"
echo ""
# Setup environment
setup_rust_environment
echo ""
# Download console assets if requested
download_console_assets
echo ""
# Build binary
build_binary
echo ""
print_message $GREEN "🎉 Build process completed successfully!"
# Show built binary
local binary_file="${OUTPUT_DIR}/${PLATFORM}/${BINARY_NAME}"
if [ -f "$binary_file" ]; then
local size=$(ls -lh "$binary_file" | awk '{print $5}')
print_message $BLUE "📋 Built binary: $binary_file ($size)"
fi
}
# Parse command line arguments
while [[ $# -gt 0 ]]; do
case $1 in
-o|--output-dir)
OUTPUT_DIR="$2"
shift 2
;;
-b|--binary-name)
BINARY_NAME="$2"
shift 2
;;
-p|--platform)
CUSTOM_PLATFORM="$2"
shift 2
;;
--dev)
BUILD_TYPE="debug"
shift
;;
--sign)
SIGN=true
shift
;;
--with-console)
WITH_CONSOLE=true
shift
;;
--no-console)
WITH_CONSOLE=false
shift
;;
--force-console-update)
FORCE_CONSOLE_UPDATE=true
WITH_CONSOLE=true # Auto-enable download when forcing update
shift
;;
--console-version)
CONSOLE_VERSION="$2"
shift 2
;;
--skip-verification)
SKIP_VERIFICATION=true
shift
;;
-h|--help)
usage
exit 0
;;
*)
print_message $RED "❌ Unknown option: $1"
usage
exit 1
;;
esac
done
# Main execution
main() {
print_message $BLUE "🦀 RustFS Binary Build Script"
echo ""
# Check if we're in a Rust project
if [ ! -f "Cargo.toml" ]; then
print_message $RED "❌ No Cargo.toml found. Are you in a Rust project directory?"
exit 1
fi
# Override platform if specified
if [ -n "$CUSTOM_PLATFORM" ]; then
PLATFORM="$CUSTOM_PLATFORM"
print_message $YELLOW "🎯 Using specified platform: $PLATFORM"
# Auto-enable skip verification for cross-compilation
if [ "$PLATFORM" != "$(detect_platform)" ]; then
SKIP_VERIFICATION=true
print_message $YELLOW "⚠️ Cross-compilation detected, enabling --skip-verification"
fi
fi
ensure_file_descriptor_limit
# Start build process
build_rustfs
}
# Run main function
main

View File

@@ -1,52 +0,0 @@
[application]
# App (Project) Name
name = "rustfs-gui"
# The static resource path
asset_dir = "public"
[web.app]
# HTML title tag content
title = "rustfs-gui"
# include `assets` in web platform
[web.resource]
# Additional CSS style files
style = []
# Additional JavaScript files
script = []
[web.resource.dev]
# Javascript code file
# serve: [dev-server] only
script = []
[bundle]
identifier = "com.rustfs.cli.gui"
publisher = "RustFsGUI"
category = "Utility"
copyright = "Copyright 2025 rustfs.com"
icon = [
"assets/icons/icon.icns",
"assets/icons/icon.ico",
"assets/icons/icon.png",
"assets/icons/rustfs-icon.png",
]
#[bundle.macos]
#provider_short_name = "RustFs"
[bundle.windows]
tsp = true
icon_path = "assets/icons/icon.ico"
allow_downgrades = true
[bundle.windows.webview_install_mode]
[bundle.windows.webview_install_mode.EmbedBootstrapper]
silent = true

View File

@@ -1,34 +0,0 @@
## Rustfs GUI
### Tailwind
1. Install npm: https://docs.npmjs.com/downloading-and-installing-node-js-and-npm
2. Install the Tailwind CSS CLI: https://tailwindcss.com/docs/installation
3. Run the following command in the root of the project to start the Tailwind CSS compiler:
```bash
npx tailwindcss -i ./input.css -o ./assets/tailwind.css --watch
```
### Dioxus CLI
#### Install the stable version (recommended)
```shell
cargo install dioxus-cli
```
### Serving Your App
Run the following command in the root of your project to start developing with the default platform:
```bash
dx serve
```
To run for a different platform, use the `--platform platform` flag. E.g.
```bash
dx serve --platform desktop
```

Binary file not shown.

Before

Width:  |  Height:  |  Size: 80 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 23 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 80 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 80 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 23 KiB

Some files were not shown because too many files have changed in this diff Show More