Commit Graph

1722 Commits

Author SHA1 Message Date
junxiang Mu
bdaee228db fix(ahm): adjust test expectations for missing xl.meta recovery scenario
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu
d562620e99 fix: implement uses_data_dir method
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu
69b0c828c9 fix: scanner add heal bucket
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu
2bfd1efb9b Fix: fix add heal_manager into scanner when scanner start
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu
0854e6b921 Chore: rename init_heal_manager_with_channel
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu
b907f4e61b refactor(ahm): remove obsolete scanner/data_usage.rs after data usage refactor
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu
6ec568459c chore: update admin handlers, lockfile, and minor fixes
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu
ea210d52dc refactor(heal): unify heal request interface, add disk field, update ahm/ecstore/common for erasure set healing
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:03 +08:00
junxiang Mu
3d3c6e4e06 chore(protos): update proto definitions, remove ns_scanner, fix codegen and formatting
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
junxiang Mu
e7d0a8d4b9 feat: integrate global metrics system into AHM scanner
- Add global metrics system to common crate for cross-module usage
- Integrate global metrics collection into AHM scanner operations
- Update ECStore to use common metrics system instead of local implementation
- Add chrono dependency to AHM crate for timestamp handling
- Re-export IlmAction from common metrics in ECStore lifecycle module
- Update scanner methods to use global metrics for cycle, disk, and volume scans
- Maintain backward compatibility with local metrics collector
- Fix clippy warnings and ensure proper code formatting

This change enables unified metrics collection across the entire RustFS system,
allowing better monitoring and observability of scanner operations.

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
junxiang Mu
7d3b2b774c fix heal disk
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
junxiang Mu
aed8f52423 refactor: integrate disk healing into erasure set healing
- Remove HealType::Disk and related disk-specific healing methods
- Integrate disk format healing into heal_erasure_set with include_format_heal option
- Update auto disk scanner to use ErasureSet heal type instead of Disk heal
- Fix disk status change event handling to use ErasureSet heal requests
- Add proper bucket list retrieval for auto healing scenarios
- Update data scanner to submit ErasureSet heal tasks for offline disks
- Remove duplicate healing logic between Disk and ErasureSet types
- Ensure all healing operations go through unified ErasureSet healing path
2025-07-24 12:12:49 +08:00
junxiang Mu
c49414f6ac fix: resolve test conflicts and improve data scanner functionality
- Fix multi-threaded test conflicts in AHM heal integration tests
- Remove global environment sharing to prevent test state pollution
- Fix test_all_disk_method by clearing global disk map before test
- Improve data scanner and cache value implementations
- Update dependencies and resolve clippy warnings

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
junxiang Mu
8e766b90cd feat: implement heal channel mechanism for admin-ahm communication
- Add global unbounded channel in common crate for heal requests
- Implement channel processor in ahm to handle heal commands
- Add Start/Query/Cancel commands support via channel
- Integrate heal manager initialization in main.rs
- Replace direct MRF calls with channel-based heal requests in ecstore
- Support advanced heal options including pool_index and set_index
- Enable admin handlers to send heal requests via channel
2025-07-24 12:12:49 +08:00
junxiang Mu
3409cd8dff feat(ahm): add HealingTracker support & complete fresh-disk healing
• Introduce ecstore HealingTracker into ahm crate; load/init/save tracker
• Re-implement heal_fresh_disk to use heal_erasure_set with tracker
• Enhance auto-disk scanner: detect unformatted disks via get_disk_id()
• Remove DataUsageCache handling for now
• Refactor imports & types, clean up duplicate constants
2025-07-24 12:12:49 +08:00
junxiang Mu
f4973a681c feat: implement complete ahm heal system with ecstore integration
- Add comprehensive heal storage API with ECStore integration
- Implement heal object, bucket, disk, metadata, and EC decode operations
- Add heal task management with progress tracking and statistics
- Optimize heal manager by removing unnecessary workers
- Add integration tests for core heal functionality (heal_object, heal_bucket, heal_format)
- Integrate with ecstore's native heal commands for actual repair operations

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
junxiang Mu
4fb3d187d0 feat: implement heal subsystem for automatic data repair
- Add heal module with core types (HealType, HealRequest, HealTask)
- Implement HealManager for task scheduling and execution
- Add HealStorageAPI trait and ECStoreHealStorage implementation
- Integrate heal capabilities into scanner for automatic repair
- Support multiple heal types: object, bucket, disk, metadata, MRF, EC decode
- Add progress tracking and event system for heal operations
- Merge heal and scanner error types for unified error handling
- Include comprehensive logging and metrics for heal operations

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
dandan
0aff736efd Chore: fix ref and fix comment
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
dandan
2aa7a631ef feat: refactor scanner module and add data usage statistics
- Move scanner code to scanner/ subdirectory for better organization
- Add data usage statistics collection and persistence
- Implement histogram support for size and version distribution
- Add global cancel token management for scanner operations
- Integrate scanner with ECStore for comprehensive data analysis
- Update error handling and improve test isolation
- Add data usage API endpoints and backend integration

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
dandan
b40ef147a9 refact: step 2
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
junxiang Mu
1f11a3167b fix: Refact heal and scanner design
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
guojidan
18b0134ddf Merge pull request #290 from guojidan/feat/complete-lock-implementation
refactor: reimplement lock
2025-07-24 12:11:19 +08:00
junxiang Mu
b48a5fdc94 fix fmt
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 11:52:57 +08:00
junxiang Mu
168a07a670 add api into ecstore
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 11:52:57 +08:00
junxiang Mu
cad005bc21 refactor(lock): unify NamespaceLock client model and LockRequest API
- Refactor NamespaceLock to use a unified client vector and quorum mechanism,
  removing legacy local/distributed lock split and related code.
- Update LockRequest to split timeout into acquire_timeout and ttl, and add
  builder methods for both.
- Adjust all batch lock APIs to accept ttl and use new LockRequest fields.
- Update all affected tests and documentation for the new API.

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 11:52:57 +08:00
root
dc44cde081 tmp
Signed-off-by: root <root@PC.localdomain>
2025-07-24 11:52:57 +08:00
dandan
4ccdeb9d2a refactor(lock): restructure lock crate, remove unused modules and clarify directory layout
- Remove unused core/rwlock.rs and manager/ modules (ManagerFactory, LifecycleManager, NamespaceManager)
- Move all lock-related code into crates/lock/src with clear submodules: client, core, utils, etc.
- Ensure only necessary files and APIs are exposed, improve maintainability
- No functional logic change, pure structure and cleanup refactor

Signed-off-by: dandan <dandan@dandandeMac-Studio.local>
2025-07-24 11:52:55 +08:00
dependabot[bot]
1b48934f47 build(deps): bump the dependencies group with 5 updates (#289)
Bumps the dependencies group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [hyper-util](https://github.com/hyperium/hyper-util) | `0.1.15` | `0.1.16` |
| [rand](https://github.com/rust-random/rand) | `0.9.1` | `0.9.2` |
| [serde_json](https://github.com/serde-rs/json) | `1.0.140` | `1.0.141` |
| [strum](https://github.com/Peternator7/strum) | `0.27.1` | `0.27.2` |
| [sysinfo](https://github.com/GuillaumeGomez/sysinfo) | `0.36.0` | `0.36.1` |


Updates `hyper-util` from 0.1.15 to 0.1.16
- [Release notes](https://github.com/hyperium/hyper-util/releases)
- [Changelog](https://github.com/hyperium/hyper-util/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/hyper-util/compare/v0.1.15...v0.1.16)

Updates `rand` from 0.9.1 to 0.9.2
- [Release notes](https://github.com/rust-random/rand/releases)
- [Changelog](https://github.com/rust-random/rand/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-random/rand/compare/rand_core-0.9.1...rand_core-0.9.2)

Updates `serde_json` from 1.0.140 to 1.0.141
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.140...v1.0.141)

Updates `strum` from 0.27.1 to 0.27.2
- [Release notes](https://github.com/Peternator7/strum/releases)
- [Changelog](https://github.com/Peternator7/strum/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Peternator7/strum/compare/v0.27.1...v0.27.2)

Updates `sysinfo` from 0.36.0 to 0.36.1
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GuillaumeGomez/sysinfo/compare/v0.36.0...v0.36.1)

---
updated-dependencies:
- dependency-name: hyper-util
  dependency-version: 0.1.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: rand
  dependency-version: 0.9.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: serde_json
  dependency-version: 1.0.141
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: strum
  dependency-version: 0.27.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: sysinfo
  dependency-version: 0.36.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-07-24 11:50:52 +08:00
zhangwenlong
25fa645184 add rustfs.spec for rustfs (#103)
add support on loongarch64
2025-07-24 11:39:09 +08:00
Marco Orlandin
3a3bb880f2 fix: update link in README.md leading to a 404 error (#285) 2025-07-24 09:15:04 +08:00
安正超
affe27298c fix: improve Windows build support and CI/CD workflow (#283)
- Fix Windows zip command issue by using PowerShell Compress-Archive
- Add Windows support for OSS upload with ossutil
- Replace Chinese comments with English in build.yml
- Fix bash syntax error in package_zip function
- Improve code formatting and consistency
- Update various configuration files for better cross-platform support

Resolves Windows build failures in GitHub Actions.
1.0.0-alpha.36
2025-07-22 23:55:57 +08:00
shiro.lee
629db6218e fix: the issue where preview fails when the path length exceeds 255 characters (#280) 2025-07-22 22:10:57 +08:00
安正超
aa1a3ce4e8 feat: add cargo clippy --fix --allow-dirty to pre-commit command (#282)
Resolves #277

- Add --fix flag to automatically fix clippy warnings
- Add --allow-dirty flag to run on dirty Git trees
- Improves code quality in pre-commit workflow
2025-07-22 22:10:53 +08:00
houseme
693db59fcc fix 2025-07-21 20:45:59 +08:00
houseme
0a7df4ef26 fix 2025-07-21 19:03:15 +08:00
houseme
9dcdc44718 fix 2025-07-21 18:03:01 +08:00
houseme
2a0c618f8b fix: windows build 2025-07-21 17:45:56 +08:00
loverustfs
bebd78fbbb Add GNU to build.yml (#275)
* fix unzip error

* fix url change error

fix url change error

* Simplify user experience and integrate console and endpoint

Simplify user experience and integrate console and endpoint

* Add gnu to  build.yml
1.0.0-alpha.35
2025-07-21 16:58:29 +08:00
houseme
3f095e75cb improve code for logger and fix typo (#272) 1.0.0-alpha.34 2025-07-21 15:20:36 +08:00
houseme
f7d30da9e0 fix typo (#267)
* fix typo

* cargo fmt
2025-07-20 00:11:15 +08:00
Chrislearn Young
823d4b6f79 Add typos github actions and fix typos (#265)
* Add typo github actions and fix typos

* cargo fmt
2025-07-19 22:08:50 +08:00
安正超
051ea7786f fix: ossutil install command. (#263) 1.0.0-alpha.33 2025-07-19 18:21:31 +08:00
安正超
42b645e355 fix: robust Dockerfile version logic for v prefix handling (#262)
* fix: robust Dockerfile version logic for v prefix handling

* wip
1.0.0-alpha.32
2025-07-19 15:50:15 +08:00
安正超
f27ee96014 feat: enhance entrypoint and Dockerfiles for flexible volume and permission management (#260)
* feat: enhance entrypoint and Dockerfiles for flexible volume and permission management\n\n- Support batch mount and permission fix in entrypoint.sh\n- Add coreutils/shadow (alpine) and coreutils/passwd (ubuntu) for UID/GID/ownership\n- Use ENTRYPOINT for unified startup\n- Make local dev and prod Dockerfile behavior consistent\n- Improve security and user experience\n\nBREAKING CHANGE: entrypoint.sh and Dockerfile now require additional packages for permission management, and support batch volume mount via RUSTFS_VOLUMES.

* chore: update Dockerfile comments to English only

* fix(entrypoint): improve local/remote volume detection and permission logic in entrypoint.sh

* Update entrypoint.sh

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update entrypoint.sh

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update Dockerfile

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1.0.0-alpha.31
2025-07-19 11:48:46 +08:00
houseme
20cd117aa6 improve code for dockerfile (#256)
* improve code for dockerfile

* Update Dockerfile

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* improve code for file name

* improve code for dockerfile

* fix

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1.0.0-alpha.30 1.0.0-alpha.29
2025-07-18 15:53:00 +08:00
houseme
fc8931d69f improve code for dockerfile (#253)
* improve code for dockerfile

* Update Dockerfile

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-18 11:05:00 +08:00
weisd
0167b2decd fix: optimize RPC connection management and prevent race conditions (#252) 2025-07-18 10:41:00 +08:00
weisd
e67980ff3c Fix/range content length (#251)
* fix:getobject range length
1.0.0-alpha.28
2025-07-17 23:25:21 +08:00
weisd
96760bba5a fix:getobject range length (#250) 1.0.0-alpha.27 2025-07-17 23:14:19 +08:00
overtrue
2501d7d241 fix: remove branch restriction from Docker workflow_run trigger
The Docker workflow was not triggering for tag-based releases because it had
'branches: [main]' restriction in the workflow_run configuration. When pushing
tags, the triggering workflow runs on the tag, not on main branch.

Changes:
- Remove 'branches: [main]' from workflow_run trigger
- Simplify tag detection using github.event.workflow_run context instead of API calls
- Use official workflow_run event properties (head_branch, event) for reliable detection
- Support both 'refs/tags/VERSION' and direct 'VERSION' formats
- Add better logging for debugging workflow trigger issues

This fixes the issue where Docker images were not built for tagged releases.
1.0.0-alpha.26
2025-07-17 08:13:34 +08:00