Compare commits

..

10 Commits

Author SHA1 Message Date
overtrue
49f480d346 fix: resolve GitHub Actions build failures and optimize cross-compilation
- Remove invalid github-token parameter from arduino/setup-protoc action
- Fix cross-compilation RUSTFLAGS issue by conditionally setting target-cpu=native
- Update workflow tag triggers from v* to * for non-v prefixed tags
- Optimize Zig and cargo-zigbuild installation using official actions

This resolves build failures in aarch64-unknown-linux-musl target where
zig was receiving invalid x86_64 CPU flags during cross-compilation.
2025-07-08 20:21:11 +08:00
安正超
055a99ba25 fix: github flow (#107) 2025-07-08 20:16:18 +08:00
weisd
2bd11d476e fix: delete empty dir (#100)
* fix: delete empty dir
2025-07-08 15:08:20 +08:00
guojidan
297004c259 Merge pull request #96 from guojidan/scanner
fix: improve data scanner random sleep calculation
2025-07-08 11:36:36 +08:00
junxiang Mu
4e2c4d8dba fix: improve data scanner random sleep calculation
- Fix random number generation API usage
- Adjust sleep calculation to follow MinIO pattern
- Ensure proper random range for scanner cycles

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-08 11:15:06 +08:00
loverustfs
0626099c3b docs: update PR template to English version 2025-07-08 01:46:36 +00:00
loverustfs
107ddcf394 Create CLA.md 2025-07-08 09:27:06 +08:00
安正超
8893ffc10f Update issue templates 2025-07-08 09:06:11 +08:00
安正超
f23e855d23 Create SECURITY.md 2025-07-08 09:05:28 +08:00
安正超
8366413970 Create CODE_OF_CONDUCT.md 2025-07-08 09:04:37 +08:00
15 changed files with 633 additions and 38 deletions

38
.github/ISSUE_TEMPLATE/bug_report.md vendored Normal file
View File

@@ -0,0 +1,38 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: ''
assignees: ''
---
**Describe the bug**
A clear and concise description of what the bug is.
**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
**Smartphone (please complete the following information):**
- Device: [e.g. iPhone6]
- OS: [e.g. iOS8.1]
- Browser [e.g. stock browser, safari]
- Version [e.g. 22]
**Additional context**
Add any other context about the problem here.

View File

@@ -0,0 +1,20 @@
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: ''
assignees: ''
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.

View File

@@ -72,7 +72,6 @@ runs:
uses: arduino/setup-protoc@v3
with:
version: "31.1"
github-token: ${{ inputs.github-token }}
- name: Install flatc
uses: Nugine/setup-flatc@v1

39
.github/pull_request_template.md vendored Normal file
View File

@@ -0,0 +1,39 @@
<!--
Pull Request Template for RustFS
-->
## Type of Change
- [ ] New Feature
- [ ] Bug Fix
- [ ] Documentation
- [ ] Performance Improvement
- [ ] Test/CI
- [ ] Refactor
- [ ] Other:
## Related Issues
<!-- List related Issue numbers, e.g. #123 -->
## Summary of Changes
<!-- Briefly describe the main changes and motivation for this PR -->
## Checklist
- [ ] I have read and followed the [CONTRIBUTING.md](CONTRIBUTING.md) guidelines
- [ ] Code is formatted with `cargo fmt --all`
- [ ] Passed `cargo clippy --all-targets --all-features -- -D warnings`
- [ ] Passed `cargo check --all-targets`
- [ ] Added/updated necessary tests
- [ ] Documentation updated (if needed)
- [ ] CI/CD passed (if applicable)
## Impact
- [ ] Breaking change (compatibility)
- [ ] Requires doc/config/deployment update
- [ ] Other impact:
## Additional Notes
<!-- Any extra information for reviewers -->
---
Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)) and sign the CLA if this is your first contribution.

View File

@@ -16,7 +16,7 @@ name: Build and Release
on:
push:
tags: ["v*"]
tags: ["*"]
branches: [main]
paths:
- "rustfs/**"
@@ -49,7 +49,6 @@ env:
RUST_BACKTRACE: 1
# Optimize build performance
CARGO_INCREMENTAL: 0
RUSTFLAGS: "-C target-cpu=native"
jobs:
# First layer: GitHub Actions level optimization (handling duplicates and concurrency)
@@ -108,6 +107,8 @@ jobs:
if: needs.skip-duplicate.outputs.should_skip != 'true' && needs.build-check.outputs.should_build == 'true'
runs-on: ${{ matrix.os }}
timeout-minutes: 60
env:
RUSTFLAGS: ${{ matrix.cross == 'false' && '-C target-cpu=native' || '' }}
strategy:
fail-fast: false
matrix:

View File

@@ -16,7 +16,7 @@ name: Docker Images
on:
push:
tags: ["v*"]
tags: ["*"]
branches: [main]
paths:
- "rustfs/**"

39
CLA.md Normal file
View File

@@ -0,0 +1,39 @@
RustFS Individual Contributor License Agreement
Thank you for your interest in contributing documentation and related software code to a project hosted or managed by RustFS. In order to clarify the intellectual property license granted with Contributions from any person or entity, RustFS must have a Contributor License Agreement (“CLA”) on file that has been signed by each Contributor, indicating agreement to the license terms below. This version of the Contributor License Agreement allows an individual to submit Contributions to the applicable project. If you are making a submission on behalf of a legal entity, then you should sign the separate Corporate Contributor License Agreement.
You accept and agree to the following terms and conditions for Your present and future Contributions submitted to RustFS. You hereby irrevocably assign and transfer to RustFS all right, title, and interest in and to Your Contributions, including all copyrights and other intellectual property rights therein.
Definitions
“You” (or “Your”) shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with RustFS. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, “control” means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
“Contribution” shall mean any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You to RustFS for inclusion in, or documentation of, any of the products or projects owned or managed by RustFS (the “Work”), including without limitation any Work described in Schedule A. For the purposes of this definition, “submitted” means any form of electronic or written communication sent to RustFS or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, RustFS for the purpose of discussing and improving the Work.
Assignment of Copyright
Subject to the terms and conditions of this Agreement, You hereby irrevocably assign and transfer to RustFS all right, title, and interest in and to Your Contributions, including all copyrights and other intellectual property rights therein, for the entire term of such rights, including all renewals and extensions. You agree to execute all documents and take all actions as may be reasonably necessary to vest in RustFS the ownership of Your Contributions and to assist RustFS in perfecting, maintaining, and enforcing its rights in Your Contributions.
Grant of Patent License
Subject to the terms and conditions of this Agreement, You hereby grant to RustFS and to recipients of documentation and software distributed by RustFS a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed.
You represent that you are legally entitled to grant the above assignment and license.
You represent that each of Your Contributions is Your original creation (see section 7 for submissions on behalf of others). You represent that Your Contribution submissions include complete details of any third-party license or other restriction (including, but not limited to, related patents and trademarks) of which you are personally aware and which are associated with any part of Your Contributions.
You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON- INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE.
Should You wish to submit work that is not Your original creation, You may submit it to RustFS separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously marking the work as “Submitted on behalf of a third-party: [named here]”.
You agree to notify RustFS of any facts or circumstances of which you become aware that would make these representations inaccurate in any respect.
Modification of CLA
RustFS reserves the right to update or modify this CLA in the future. Any updates or modifications to this CLA shall apply only to Contributions made after the effective date of the revised CLA. Contributions made prior to the update shall remain governed by the version of the CLA that was in effect at the time of submission. It is not necessary for all Contributors to re-sign the CLA when the CLA is updated or modified.
Governing Law and Dispute Resolution
This Agreement will be governed by and construed in accordance with the laws of the Peoples Republic of China excluding that body of laws known as conflict of laws. The parties expressly agree that the United Nations Convention on Contracts for the International Sale of Goods will not apply. Any legal action or proceeding arising under this Agreement will be brought exclusively in the courts located in Beijing, China, and the parties hereby irrevocably consent to the personal jurisdiction and venue therein.
For your reading convenience, this Agreement is written in parallel English and Chinese sections. To the extent there is a conflict between the English and Chinese sections, the English sections shall govern.

128
CODE_OF_CONDUCT.md Normal file
View File

@@ -0,0 +1,128 @@
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, religion, or sexual identity
and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our
community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the
overall community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or
advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email
address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.
Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
hello@rustfs.com.
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series
of actions.
**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or
permanent ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within
the community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.0, available at
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
Community Impact Guidelines were inspired by [Mozilla's code of conduct
enforcement ladder](https://github.com/mozilla/diversity).
[homepage]: https://www.contributor-covenant.org
For answers to common questions about this code of conduct, see the FAQ at
https://www.contributor-covenant.org/faq. Translations are available at
https://www.contributor-covenant.org/translations.

18
SECURITY.md Normal file
View File

@@ -0,0 +1,18 @@
# Security Policy
## Supported Versions
Use this section to tell people about which versions of your project are
currently being supported with security updates.
| Version | Supported |
| ------- | ------------------ |
| 1.x.x | :white_check_mark: |
## Reporting a Vulnerability
Use this section to tell people how to report a vulnerability.
Tell them where to go, how often they can expect to get an update on a
reported vulnerability, what to expect if the vulnerability is accepted or
declined, etc.

View File

@@ -20,17 +20,18 @@ use std::{
path::{Path, PathBuf},
pin::Pin,
sync::{
Arc,
Arc, OnceLock,
atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering},
},
time::{Duration, SystemTime},
};
use time::{self, OffsetDateTime};
use tokio_util::sync::CancellationToken;
use super::{
data_scanner_metric::{ScannerMetric, ScannerMetrics, globalScannerMetrics},
data_usage::{DATA_USAGE_BLOOM_NAME_PATH, store_data_usage_in_backend},
data_usage::{DATA_USAGE_BLOOM_NAME_PATH, DataUsageInfo, store_data_usage_in_backend},
data_usage_cache::{DataUsageCache, DataUsageEntry, DataUsageHash},
heal_commands::{HEAL_DEEP_SCAN, HEAL_NORMAL_SCAN, HealScanMode},
};
@@ -127,6 +128,8 @@ lazy_static! {
pub static ref globalHealConfig: Arc<RwLock<Config>> = Arc::new(RwLock::new(Config::default()));
}
static GLOBAL_SCANNER_CANCEL_TOKEN: OnceLock<CancellationToken> = OnceLock::new();
struct DynamicSleeper {
factor: f64,
max_sleep: Duration,
@@ -195,36 +198,66 @@ fn new_dynamic_sleeper(factor: f64, max_wait: Duration, is_scanner: bool) -> Dyn
/// - Minimum sleep duration to avoid excessive CPU usage
/// - Proper error handling and logging
///
/// # Returns
/// A CancellationToken that can be used to gracefully shutdown the scanner
///
/// # Architecture
/// 1. Initialize with random seed for sleep intervals
/// 2. Run scanner cycles in a loop
/// 3. Use randomized sleep between cycles to avoid thundering herd
/// 4. Ensure minimum sleep duration to prevent CPU thrashing
pub async fn init_data_scanner() {
pub async fn init_data_scanner() -> CancellationToken {
info!("Initializing data scanner background task");
let cancel_token = CancellationToken::new();
GLOBAL_SCANNER_CANCEL_TOKEN
.set(cancel_token.clone())
.expect("Scanner already initialized");
let cancel_clone = cancel_token.clone();
tokio::spawn(async move {
info!("Data scanner background task started");
loop {
// Run the data scanner
run_data_scanner().await;
tokio::select! {
_ = cancel_clone.cancelled() => {
info!("Data scanner received shutdown signal, exiting gracefully");
break;
}
_ = run_data_scanner_cycle() => {
// Calculate randomized sleep duration
let random_factor = {
let mut rng = rand::rng();
rng.random_range(1.0..10.0)
};
let base_cycle_duration = SCANNER_CYCLE.load(Ordering::SeqCst) as f64;
let sleep_duration_secs = random_factor * base_cycle_duration;
// Calculate randomized sleep duration
// Use random factor (0.0 to 1.0) multiplied by the scanner cycle duration
let random_factor = {
let mut rng = rand::rng();
rng.random_range(1.0..10.0)
};
let base_cycle_duration = SCANNER_CYCLE.load(Ordering::SeqCst) as f64;
let sleep_duration_secs = random_factor * base_cycle_duration;
let sleep_duration = Duration::from_secs_f64(sleep_duration_secs);
let sleep_duration = Duration::from_secs_f64(sleep_duration_secs);
debug!(
duration_secs = sleep_duration.as_secs(),
"Data scanner sleeping before next cycle"
);
info!(duration_secs = sleep_duration.as_secs(), "Data scanner sleeping before next cycle");
// Sleep with the calculated duration
sleep(sleep_duration).await;
// Interruptible sleep
tokio::select! {
_ = cancel_clone.cancelled() => {
info!("Data scanner received shutdown signal during sleep, exiting");
break;
}
_ = sleep(sleep_duration) => {
// Continue to next cycle
}
}
}
}
}
info!("Data scanner background task stopped gracefully");
});
cancel_token
}
/// Run a single data scanner cycle
@@ -239,8 +272,8 @@ pub async fn init_data_scanner() {
/// - Gracefully handles missing object layer
/// - Continues operation even if individual steps fail
/// - Logs errors appropriately without terminating the scanner
async fn run_data_scanner() {
info!("Starting data scanner cycle");
async fn run_data_scanner_cycle() {
debug!("Starting data scanner cycle");
// Get the object layer, return early if not available
let Some(store) = new_object_layer_fn() else {
@@ -248,6 +281,14 @@ async fn run_data_scanner() {
return;
};
// Check for cancellation before starting expensive operations
if let Some(token) = GLOBAL_SCANNER_CANCEL_TOKEN.get() {
if token.is_cancelled() {
debug!("Scanner cancelled before starting cycle");
return;
}
}
// Load current cycle information from persistent storage
let buf = read_config(store.clone(), &DATA_USAGE_BLOOM_NAME_PATH)
.await
@@ -293,7 +334,7 @@ async fn run_data_scanner() {
}
// Set up data usage storage channel
let (tx, rx) = mpsc::channel(100);
let (tx, rx) = mpsc::channel::<DataUsageInfo>(100);
tokio::spawn(async move {
let _ = store_data_usage_in_backend(rx).await;
});
@@ -308,8 +349,8 @@ async fn run_data_scanner() {
"Starting namespace scanner"
);
// Run the namespace scanner
match store.clone().ns_scanner(tx, cycle_info.current as usize, scan_mode).await {
// Run the namespace scanner with cancellation support
match execute_namespace_scan(&store, tx, cycle_info.current, scan_mode).await {
Ok(_) => {
info!(cycle = cycle_info.current, "Namespace scanner completed successfully");
@@ -349,6 +390,28 @@ async fn run_data_scanner() {
stop_fn(&scan_result);
}
/// Execute namespace scan with cancellation support
async fn execute_namespace_scan(
store: &Arc<ECStore>,
tx: Sender<DataUsageInfo>,
cycle: u64,
scan_mode: HealScanMode,
) -> Result<()> {
let cancel_token = GLOBAL_SCANNER_CANCEL_TOKEN
.get()
.ok_or_else(|| Error::other("Scanner not initialized"))?;
tokio::select! {
result = store.ns_scanner(tx, cycle as usize, scan_mode) => {
result.map_err(|e| Error::other(format!("Namespace scan failed: {e}")))
}
_ = cancel_token.cancelled() => {
info!("Namespace scan cancelled");
Err(Error::other("Scan cancelled"))
}
}
}
#[derive(Debug, Serialize, Deserialize)]
struct BackgroundHealInfo {
bitrot_start_time: SystemTime,
@@ -404,7 +467,7 @@ async fn get_cycle_scan_mode(current_cycle: u64, bitrot_start_cycle: u64, bitrot
return HEAL_DEEP_SCAN;
}
if bitrot_start_time.duration_since(SystemTime::now()).unwrap() > bitrot_cycle {
if SystemTime::now().duration_since(bitrot_start_time).unwrap_or_default() > bitrot_cycle {
return HEAL_DEEP_SCAN;
}

View File

@@ -429,7 +429,7 @@ async fn run(opt: config::Opt) -> Result<()> {
})?;
// init scanner
init_data_scanner().await;
let scanner_cancel_token = init_data_scanner().await;
// init auto heal
init_auto_heal().await;
// init console configuration
@@ -493,11 +493,11 @@ async fn run(opt: config::Opt) -> Result<()> {
match wait_for_shutdown().await {
#[cfg(unix)]
ShutdownSignal::CtrlC | ShutdownSignal::Sigint | ShutdownSignal::Sigterm => {
handle_shutdown(&state_manager, &shutdown_tx).await;
handle_shutdown(&state_manager, &shutdown_tx, &scanner_cancel_token).await;
}
#[cfg(not(unix))]
ShutdownSignal::CtrlC => {
handle_shutdown(&state_manager, &shutdown_tx).await;
handle_shutdown(&state_manager, &shutdown_tx, &scanner_cancel_token).await;
}
}
@@ -603,11 +603,19 @@ fn process_connection(
}
/// Handles the shutdown process of the server
async fn handle_shutdown(state_manager: &ServiceStateManager, shutdown_tx: &tokio::sync::broadcast::Sender<()>) {
async fn handle_shutdown(
state_manager: &ServiceStateManager,
shutdown_tx: &tokio::sync::broadcast::Sender<()>,
scanner_cancel_token: &tokio_util::sync::CancellationToken,
) {
info!("Shutdown signal received in main thread");
// update the status to stopping first
state_manager.update(ServiceState::Stopping);
// Stop data scanner gracefully
info!("Stopping data scanner...");
scanner_cancel_token.cancel();
// Stop the notification system
shutdown_event_notifier().await;

View File

@@ -52,7 +52,7 @@ pub async fn del_opts(
opts.version_id = {
if is_dir_object(object) && vid.is_none() {
Some(Uuid::nil().to_string())
Some(Uuid::max().to_string())
} else {
vid
}
@@ -91,7 +91,7 @@ pub async fn get_opts(
opts.version_id = {
if is_dir_object(object) && vid.is_none() {
Some(Uuid::nil().to_string())
Some(Uuid::max().to_string())
} else {
vid
}
@@ -133,7 +133,7 @@ pub async fn put_opts(
opts.version_id = {
if is_dir_object(object) && vid.is_none() {
Some(Uuid::nil().to_string())
Some(Uuid::max().to_string())
} else {
vid
}
@@ -273,7 +273,7 @@ mod tests {
assert!(result.is_ok());
let opts = result.unwrap();
assert_eq!(opts.version_id, Some(Uuid::nil().to_string()));
assert_eq!(opts.version_id, Some(Uuid::max().to_string()));
}
#[tokio::test]
@@ -346,7 +346,7 @@ mod tests {
assert!(result.is_ok());
let opts = result.unwrap();
assert_eq!(opts.version_id, Some(Uuid::nil().to_string()));
assert_eq!(opts.version_id, Some(Uuid::max().to_string()));
}
#[tokio::test]
@@ -390,7 +390,7 @@ mod tests {
assert!(result.is_ok());
let opts = result.unwrap();
assert_eq!(opts.version_id, Some(Uuid::nil().to_string()));
assert_eq!(opts.version_id, Some(Uuid::max().to_string()));
}
#[tokio::test]

View File

@@ -0,0 +1,71 @@
# Delete __XLDIR__ Directory Scripts
This directory contains scripts for deleting all directories ending with `__XLDIR__` in the specified path.
## Script Description
### 1. delete_xldir.sh (Full Version)
A feature-rich version with multiple options and safety checks.
**Usage:**
```bash
./scripts/delete_xldir.sh <path> [options]
```
**Options:**
- `-f, --force` Force deletion without confirmation
- `-v, --verbose` Show verbose information
- `-d, --dry-run` Show directories to be deleted without actually deleting
- `-h, --help` Show help information
**Examples:**
```bash
# Preview directories to be deleted (without actually deleting)
./scripts/delete_xldir.sh /path/to/search --dry-run
# Interactive deletion (will ask for confirmation)
./scripts/delete_xldir.sh /path/to/search
# Force deletion (without confirmation)
./scripts/delete_xldir.sh /path/to/search --force
# Verbose mode deletion
./scripts/delete_xldir.sh /path/to/search --verbose
```
### 2. delete_xldir_simple.sh (Simple Version)
A streamlined version that directly deletes found directories.
**Usage:**
```bash
./scripts/delete_xldir_simple.sh <path>
```
**Example:**
```bash
# Delete all directories ending with __XLDIR__ in the specified path
./scripts/delete_xldir_simple.sh /path/to/search
```
## How It Works
Both scripts use the `find` command to locate directories:
```bash
find "$SEARCH_PATH" -type d -name "*__XLDIR__"
```
- `-type d`: Only search for directories
- `-name "*__XLDIR__"`: Find directories ending with `__XLDIR__`
## Safety Notes
⚠️ **Important Reminders:**
- Deletion operations are irreversible, please confirm the path is correct before use
- It's recommended to use the `--dry-run` option first to preview directories to be deleted
- For important data, please backup first
## Use Cases
These scripts are typically used for cleaning up temporary directories or metadata directories in storage systems, especially in distributed storage systems where `__XLDIR__` is commonly used as a specific directory identifier.

147
scripts/test/delete_xldir.sh Executable file
View File

@@ -0,0 +1,147 @@
#!/bin/bash
# Delete all directories ending with __XLDIR__ in the specified path
# Check parameters
if [ $# -eq 0 ]; then
echo "Usage: $0 <path> [options]"
echo "Options:"
echo " -f, --force Force deletion without confirmation"
echo " -v, --verbose Show verbose information"
echo " -d, --dry-run Show directories to be deleted without actually deleting"
echo ""
echo "Examples:"
echo " $0 /path/to/search"
echo " $0 /path/to/search --dry-run"
echo " $0 /path/to/search --force"
exit 1
fi
# Parse parameters
SEARCH_PATH=""
FORCE=false
VERBOSE=false
DRY_RUN=false
while [[ $# -gt 0 ]]; do
case $1 in
-f|--force)
FORCE=true
shift
;;
-v|--verbose)
VERBOSE=true
shift
;;
-d|--dry-run)
DRY_RUN=true
shift
;;
-h|--help)
echo "Usage: $0 <path> [options]"
echo "Delete all directories ending with __XLDIR__ in the specified path"
exit 0
;;
-*)
echo "Unknown option: $1"
exit 1
;;
*)
if [ -z "$SEARCH_PATH" ]; then
SEARCH_PATH="$1"
else
echo "Error: Only one path can be specified"
exit 1
fi
shift
;;
esac
done
# Check if path is provided
if [ -z "$SEARCH_PATH" ]; then
echo "Error: Search path must be specified"
exit 1
fi
# Check if path exists
if [ ! -d "$SEARCH_PATH" ]; then
echo "Error: Path '$SEARCH_PATH' does not exist or is not a directory"
exit 1
fi
# Find all directories ending with __XLDIR__
echo "Searching in path: $SEARCH_PATH"
echo "Looking for directories ending with __XLDIR__..."
# Use find command to locate directories
DIRS_TO_DELETE=$(find "$SEARCH_PATH" -type d -name "*__XLDIR__" 2>/dev/null)
if [ -z "$DIRS_TO_DELETE" ]; then
echo "No directories ending with __XLDIR__ found"
exit 0
fi
# Display found directories
echo "Found the following directories:"
echo "$DIRS_TO_DELETE"
echo ""
# Count directories
DIR_COUNT=$(echo "$DIRS_TO_DELETE" | wc -l)
echo "Total found: $DIR_COUNT directories"
# If dry-run mode, only show without deleting
if [ "$DRY_RUN" = true ]; then
echo ""
echo "This is dry-run mode, no directories will be actually deleted"
echo "To actually delete these directories, remove the --dry-run option"
exit 0
fi
# If not force mode, ask for confirmation
if [ "$FORCE" = false ]; then
echo ""
read -p "Are you sure you want to delete these directories? (y/N): " -n 1 -r
echo ""
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
echo "Operation cancelled"
exit 0
fi
fi
# Delete directories
echo ""
echo "Starting to delete directories..."
deleted_count=0
failed_count=0
while IFS= read -r dir; do
if [ -d "$dir" ]; then
if [ "$VERBOSE" = true ]; then
echo "Deleting: $dir"
fi
if rm -rf "$dir" 2>/dev/null; then
((deleted_count++))
if [ "$VERBOSE" = true ]; then
echo " ✓ Deleted successfully"
fi
else
((failed_count++))
echo " ✗ Failed to delete: $dir"
fi
fi
done <<< "$DIRS_TO_DELETE"
echo ""
echo "Deletion completed!"
echo "Successfully deleted: $deleted_count directories"
if [ $failed_count -gt 0 ]; then
echo "Failed to delete: $failed_count directories"
exit 1
else
echo "All directories have been successfully deleted"
exit 0
fi

View File

@@ -0,0 +1,24 @@
#!/bin/bash
# Simple version: Delete all directories ending with __XLDIR__ in the specified path
if [ $# -eq 0 ]; then
echo "Usage: $0 <path>"
echo "Example: $0 /path/to/search"
exit 1
fi
SEARCH_PATH="$1"
# Check if path exists
if [ ! -d "$SEARCH_PATH" ]; then
echo "Error: Path '$SEARCH_PATH' does not exist or is not a directory"
exit 1
fi
echo "Searching in path: $SEARCH_PATH"
# Find and delete all directories ending with __XLDIR__
find "$SEARCH_PATH" -type d -name "*__XLDIR__" -exec rm -rf {} \; 2>/dev/null
echo "Deletion completed!"