mirror of
https://github.com/rustfs/rustfs.git
synced 2026-01-16 17:20:33 +00:00
Add complete MNMD Docker deployment example with startup coordination and VolumeNotFound fix (#642)
* Initial plan * Add MNMD Docker deployment example with 4 nodes x 4 drives - Create docs/examples/mnmd/ directory structure - Add docker-compose.yml with proper disk indexing (1..4) - Add wait-and-start.sh for startup coordination - Add README.md with usage instructions and alternatives - Add CHECKLIST.md with step-by-step verification - Fixes VolumeNotFound issue by using correct volume paths - Implements health checks and startup ordering - Uses service names for stable inter-node addressing Co-authored-by: houseme <4829346+houseme@users.noreply.github.com> * Add docs/examples README as index for deployment examples Co-authored-by: houseme <4829346+houseme@users.noreply.github.com> * Add automated test script for MNMD deployment - Add test-deployment.sh with comprehensive validation - Test container status, health, endpoints, connectivity - Update README to reference test script - Make script executable Co-authored-by: houseme <4829346+houseme@users.noreply.github.com> * improve code * improve code * improve dep crates `cargo shear --fix` * upgrade aws-sdk-s3 --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: houseme <4829346+houseme@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com>
This commit is contained in:
Copilot
60
docs/examples/README.md
Normal file
60
docs/examples/README.md
Normal file
@@ -0,0 +1,60 @@
|
||||
# RustFS Deployment Examples
|
||||
|
||||
This directory contains practical deployment examples and configurations for RustFS.
|
||||
|
||||
## Available Examples
|
||||
|
||||
### [MNMD (Multi-Node Multi-Drive)](./mnmd/)
|
||||
|
||||
Complete Docker Compose example for deploying RustFS in a 4-node, 4-drive-per-node configuration.
|
||||
|
||||
**Features:**
|
||||
- Proper disk indexing (1..4) to avoid VolumeNotFound errors
|
||||
- Startup coordination via `wait-and-start.sh` script
|
||||
- Service discovery using Docker service names
|
||||
- Health checks with alternatives for different base images
|
||||
- Comprehensive documentation and verification checklist
|
||||
|
||||
**Use Case:** Production-ready multi-node deployment for high availability and performance.
|
||||
|
||||
**Quick Start:**
|
||||
```bash
|
||||
cd docs/examples/mnmd
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
**See also:**
|
||||
- [MNMD README](./mnmd/README.md) - Detailed usage guide
|
||||
- [MNMD CHECKLIST](./mnmd/CHECKLIST.md) - Step-by-step verification
|
||||
|
||||
## Other Deployment Examples
|
||||
|
||||
For additional deployment examples, see:
|
||||
- [`examples/`](/examples/) - Root-level examples directory with:
|
||||
- `docker-quickstart.sh` - Quick start script for basic deployments
|
||||
- `enhanced-docker-deployment.sh` - Advanced deployment scenarios
|
||||
- `docker-comprehensive.yml` - Docker Compose with multiple profiles
|
||||
- [`.docker/compose/`](/.docker/compose/) - Docker Compose configurations:
|
||||
- `docker-compose.cluster.yaml` - Basic cluster setup
|
||||
- `docker-compose.observability.yaml` - Observability stack integration
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Console & Endpoint Service Separation](../console-separation.md)
|
||||
- [Environment Variables](../ENVIRONMENT_VARIABLES.md)
|
||||
- [Performance Testing](../PERFORMANCE_TESTING.md)
|
||||
|
||||
## Contributing
|
||||
|
||||
When adding new examples:
|
||||
1. Create a dedicated subdirectory under `docs/examples/`
|
||||
2. Include a comprehensive README.md
|
||||
3. Provide working configuration files
|
||||
4. Add verification steps or checklists
|
||||
5. Document common issues and troubleshooting
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
- GitHub Issues: https://github.com/rustfs/rustfs/issues
|
||||
- Documentation: https://rustfs.io
|
||||
329
docs/examples/mnmd/CHECKLIST.md
Normal file
329
docs/examples/mnmd/CHECKLIST.md
Normal file
@@ -0,0 +1,329 @@
|
||||
# MNMD Deployment Checklist
|
||||
|
||||
This checklist provides step-by-step verification for deploying RustFS in MNMD (Multi-Node Multi-Drive) mode using
|
||||
Docker.
|
||||
|
||||
## Pre-Deployment Checks
|
||||
|
||||
### 1. System Requirements
|
||||
|
||||
- [ ] Docker Engine 20.10+ installed
|
||||
- [ ] Docker Compose 2.0+ installed
|
||||
- [ ] At least 8GB RAM available
|
||||
- [ ] At least 40GB disk space available (for 4 nodes × 4 volumes)
|
||||
|
||||
Verify with:
|
||||
|
||||
```bash
|
||||
docker --version
|
||||
docker-compose --version
|
||||
free -h
|
||||
df -h
|
||||
```
|
||||
|
||||
### 2. File System Checks
|
||||
|
||||
- [ ] Using XFS, ext4, or another suitable filesystem (not NFS for production)
|
||||
- [ ] File system supports extended attributes
|
||||
|
||||
Verify with:
|
||||
|
||||
```bash
|
||||
df -T | grep -E '(xfs|ext4)'
|
||||
```
|
||||
|
||||
### 3. Permissions and SELinux
|
||||
|
||||
- [ ] Current user is in `docker` group or can run `sudo docker`
|
||||
- [ ] SELinux is properly configured (if enabled)
|
||||
|
||||
Verify with:
|
||||
|
||||
```bash
|
||||
groups | grep docker
|
||||
getenforce # If enabled, should show "Permissive" or "Enforcing" with proper policies
|
||||
```
|
||||
|
||||
### 4. Network Configuration
|
||||
|
||||
- [ ] Ports 9000-9031 are available
|
||||
- [ ] No firewall blocking Docker bridge network
|
||||
|
||||
Verify with:
|
||||
|
||||
```bash
|
||||
# Check if ports are free
|
||||
netstat -tuln | grep -E ':(9000|9001|9010|9011|9020|9021|9030|9031)'
|
||||
# Should return nothing if ports are free
|
||||
```
|
||||
|
||||
### 5. Files Present
|
||||
|
||||
- [ ] `docker-compose.yml` exists in current directory
|
||||
|
||||
Verify with:
|
||||
|
||||
```bash
|
||||
cd docs/examples/mnmd
|
||||
ls -la
|
||||
chmod +x wait-and-start.sh # If needed
|
||||
```
|
||||
|
||||
## Deployment Steps
|
||||
|
||||
### 1. Start the Cluster
|
||||
|
||||
- [ ] Navigate to the example directory
|
||||
- [ ] Pull the latest RustFS image
|
||||
- [ ] Start the cluster
|
||||
|
||||
```bash
|
||||
cd docs/examples/mnmd
|
||||
docker-compose pull
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
### 2. Monitor Startup
|
||||
|
||||
- [ ] Watch container logs during startup
|
||||
- [ ] Verify no VolumeNotFound errors
|
||||
- [ ] Check that peer discovery completes
|
||||
|
||||
```bash
|
||||
# Watch all logs
|
||||
docker-compose logs -f
|
||||
|
||||
# Watch specific node
|
||||
docker-compose logs -f rustfs-node1
|
||||
|
||||
# Look for successful startup messages
|
||||
docker-compose logs | grep -i "ready\|listening\|started"
|
||||
```
|
||||
|
||||
### 3. Verify Container Status
|
||||
|
||||
- [ ] All 4 containers are running
|
||||
- [ ] All 4 containers show as healthy
|
||||
|
||||
```bash
|
||||
docker-compose ps
|
||||
|
||||
# Expected output: 4 containers in "Up" state with "healthy" status
|
||||
```
|
||||
|
||||
### 4. Check Health Endpoints
|
||||
|
||||
- [ ] API health endpoints respond on all nodes
|
||||
- [ ] Console health endpoints respond on all nodes
|
||||
|
||||
```bash
|
||||
# Test API endpoints
|
||||
curl http://localhost:9000/health
|
||||
curl http://localhost:9010/health
|
||||
curl http://localhost:9020/health
|
||||
curl http://localhost:9030/health
|
||||
|
||||
# Test Console endpoints
|
||||
curl http://localhost:9001/health
|
||||
curl http://localhost:9011/health
|
||||
curl http://localhost:9021/health
|
||||
curl http://localhost:9031/health
|
||||
|
||||
# All should return successful health status
|
||||
```
|
||||
|
||||
## Post-Deployment Verification
|
||||
|
||||
### 1. In-Container Checks
|
||||
|
||||
- [ ] Data directories exist
|
||||
- [ ] Directories have correct permissions
|
||||
- [ ] RustFS process is running
|
||||
|
||||
```bash
|
||||
# Check node1
|
||||
docker exec rustfs-node1 ls -la /data/
|
||||
docker exec rustfs-node1 ps aux | grep rustfs
|
||||
|
||||
# Verify all 4 data directories exist
|
||||
docker exec rustfs-node1 ls -d /data/rustfs{1..4}
|
||||
```
|
||||
|
||||
### 2. DNS and Network Validation
|
||||
|
||||
- [ ] Service names resolve correctly
|
||||
- [ ] Inter-node connectivity works
|
||||
|
||||
```bash
|
||||
# DNS resolution test
|
||||
docker exec rustfs-node1 nslookup rustfs-node2
|
||||
docker exec rustfs-node1 nslookup rustfs-node3
|
||||
docker exec rustfs-node1 nslookup rustfs-node4
|
||||
|
||||
# Connectivity test (using nc if available)
|
||||
docker exec rustfs-node1 nc -zv rustfs-node2 9000
|
||||
docker exec rustfs-node1 nc -zv rustfs-node3 9000
|
||||
docker exec rustfs-node1 nc -zv rustfs-node4 9000
|
||||
|
||||
# Or using telnet/curl
|
||||
docker exec rustfs-node1 curl -v http://rustfs-node2:9000/health
|
||||
```
|
||||
|
||||
### 3. Volume Configuration Validation
|
||||
|
||||
- [ ] RUSTFS_VOLUMES environment variable is correct
|
||||
- [ ] All 16 endpoints are configured (4 nodes × 4 drives)
|
||||
|
||||
```bash
|
||||
# Check environment variable
|
||||
docker exec rustfs-node1 env | grep RUSTFS_VOLUMES
|
||||
|
||||
# Expected output:
|
||||
# RUSTFS_VOLUMES=http://rustfs-node{1...4}:9000/data/rustfs{1...4}
|
||||
```
|
||||
|
||||
### 4. Cluster Functionality
|
||||
|
||||
- [ ] Can list buckets via API
|
||||
- [ ] Can create a bucket
|
||||
- [ ] Can upload an object
|
||||
- [ ] Can download an object
|
||||
|
||||
```bash
|
||||
# Configure AWS CLI or s3cmd
|
||||
export AWS_ACCESS_KEY_ID=rustfsadmin
|
||||
export AWS_SECRET_ACCESS_KEY=rustfsadmin
|
||||
|
||||
# Using AWS CLI (if installed)
|
||||
aws --endpoint-url http://localhost:9000 s3 mb s3://test-bucket
|
||||
aws --endpoint-url http://localhost:9000 s3 ls
|
||||
echo "test content" > test.txt
|
||||
aws --endpoint-url http://localhost:9000 s3 cp test.txt s3://test-bucket/
|
||||
aws --endpoint-url http://localhost:9000 s3 ls s3://test-bucket/
|
||||
aws --endpoint-url http://localhost:9000 s3 cp s3://test-bucket/test.txt downloaded.txt
|
||||
cat downloaded.txt
|
||||
|
||||
# Or using curl
|
||||
curl -X PUT http://localhost:9000/test-bucket \
|
||||
-H "Host: localhost:9000" \
|
||||
--user rustfsadmin:rustfsadmin
|
||||
```
|
||||
|
||||
### 5. Healthcheck Verification
|
||||
|
||||
- [ ] Docker reports all services as healthy
|
||||
- [ ] Healthcheck scripts work in containers
|
||||
|
||||
```bash
|
||||
# Check Docker health status
|
||||
docker inspect rustfs-node1 --format='{{.State.Health.Status}}'
|
||||
docker inspect rustfs-node2 --format='{{.State.Health.Status}}'
|
||||
docker inspect rustfs-node3 --format='{{.State.Health.Status}}'
|
||||
docker inspect rustfs-node4 --format='{{.State.Health.Status}}'
|
||||
|
||||
# All should return "healthy"
|
||||
|
||||
# Test healthcheck command manually
|
||||
docker exec rustfs-node1 nc -z localhost 9000
|
||||
echo $? # Should be 0
|
||||
```
|
||||
|
||||
## Troubleshooting Checks
|
||||
|
||||
### If VolumeNotFound Error Occurs
|
||||
|
||||
- [ ] Verify volume indexing starts at 1, not 0
|
||||
- [ ] Check that RUSTFS_VOLUMES matches mounted paths
|
||||
- [ ] Ensure all /data/rustfs{1..4} directories exist
|
||||
|
||||
```bash
|
||||
# Check mounted volumes
|
||||
docker inspect rustfs-node1 | jq '.[].Mounts'
|
||||
|
||||
# Verify directories in container
|
||||
docker exec rustfs-node1 ls -la /data/
|
||||
```
|
||||
|
||||
### If Healthcheck Fails
|
||||
|
||||
- [ ] Check if `nc` is available in the image
|
||||
- [ ] Try alternative healthcheck (curl/wget)
|
||||
- [ ] Increase `start_period` in docker-compose.yml
|
||||
|
||||
```bash
|
||||
# Check if nc is available
|
||||
docker exec rustfs-node1 which nc
|
||||
|
||||
# Test healthcheck manually
|
||||
docker exec rustfs-node1 nc -z localhost 9000
|
||||
|
||||
# Check logs for errors
|
||||
docker-compose logs rustfs-node1 | grep -i error
|
||||
```
|
||||
|
||||
### If Startup Takes Too Long
|
||||
|
||||
- [ ] Check peer discovery timeout in logs
|
||||
- [ ] Verify network connectivity between nodes
|
||||
- [ ] Consider increasing timeout in wait-and-start.sh
|
||||
|
||||
```bash
|
||||
# Check startup logs
|
||||
docker-compose logs rustfs-node1 | grep -i "waiting\|peer\|timeout"
|
||||
|
||||
# Check network
|
||||
docker network inspect mnmd_rustfs-mnmd
|
||||
```
|
||||
|
||||
### If Containers Crash or Restart
|
||||
|
||||
- [ ] Review container logs
|
||||
- [ ] Check resource usage (CPU/Memory)
|
||||
- [ ] Verify no port conflicts
|
||||
|
||||
```bash
|
||||
# View last crash logs
|
||||
docker-compose logs --tail=100 rustfs-node1
|
||||
|
||||
# Check resource usage
|
||||
docker stats --no-stream
|
||||
|
||||
# Check restart count
|
||||
docker-compose ps
|
||||
```
|
||||
|
||||
## Cleanup Checklist
|
||||
|
||||
When done testing:
|
||||
|
||||
- [ ] Stop the cluster: `docker-compose down`
|
||||
- [ ] Remove volumes (optional, destroys data): `docker-compose down -v`
|
||||
- [ ] Clean up dangling images: `docker image prune`
|
||||
- [ ] Verify ports are released: `netstat -tuln | grep -E ':(9000|9001|9010|9011|9020|9021|9030|9031)'`
|
||||
|
||||
## Production Deployment Additional Checks
|
||||
|
||||
Before deploying to production:
|
||||
|
||||
- [ ] Change default credentials (RUSTFS_ACCESS_KEY, RUSTFS_SECRET_KEY)
|
||||
- [ ] Configure TLS certificates
|
||||
- [ ] Set up proper logging and monitoring
|
||||
- [ ] Configure backups for volumes
|
||||
- [ ] Review and adjust resource limits
|
||||
- [ ] Set up external load balancer (if needed)
|
||||
- [ ] Document disaster recovery procedures
|
||||
- [ ] Test failover scenarios
|
||||
- [ ] Verify data persistence after container restart
|
||||
|
||||
## Summary
|
||||
|
||||
This checklist ensures:
|
||||
|
||||
- ✓ Correct disk indexing (1..4 instead of 0..3)
|
||||
- ✓ Proper startup coordination via wait-and-start.sh
|
||||
- ✓ Service discovery via Docker service names
|
||||
- ✓ Health checks function correctly
|
||||
- ✓ All 16 endpoints (4 nodes × 4 drives) are operational
|
||||
- ✓ No VolumeNotFound errors occur
|
||||
|
||||
For more details, see [README.md](./README.md) in this directory.
|
||||
268
docs/examples/mnmd/README.md
Normal file
268
docs/examples/mnmd/README.md
Normal file
@@ -0,0 +1,268 @@
|
||||
# RustFS MNMD (Multi-Node Multi-Drive) Docker Example
|
||||
|
||||
This directory contains a complete, ready-to-use MNMD deployment example for RustFS with 4 nodes and 4 drives per node (
|
||||
4x4 configuration).
|
||||
|
||||
## Overview
|
||||
|
||||
This example addresses common deployment issues including:
|
||||
|
||||
- **VolumeNotFound errors** - Fixed by using correct disk indexing (`/data/rustfs{1...4}` instead of
|
||||
`/data/rustfs{0...3}`)
|
||||
- **Startup race conditions** - Solved with a simple `sleep` command in each service.
|
||||
- **Service discovery** - Uses Docker service names (`rustfs-node{1..4}`) instead of hard-coded IPs
|
||||
- **Health checks** - Implements proper health monitoring with `nc` (with alternatives documented)
|
||||
|
||||
## Quick Start
|
||||
|
||||
From this directory (`docs/examples/mnmd`), run:
|
||||
|
||||
```bash
|
||||
# Start the cluster
|
||||
docker-compose up -d
|
||||
|
||||
# Check the status
|
||||
docker-compose ps
|
||||
|
||||
# View logs
|
||||
docker-compose logs -f
|
||||
|
||||
# Test the deployment
|
||||
curl http://localhost:9000/health
|
||||
curl http://localhost:9001/health
|
||||
|
||||
# Run comprehensive tests
|
||||
./test-deployment.sh
|
||||
|
||||
# Stop the cluster
|
||||
docker-compose down
|
||||
|
||||
# Clean up volumes (WARNING: deletes all data)
|
||||
docker-compose down -v
|
||||
```
|
||||
|
||||
## Configuration Details
|
||||
|
||||
### Volume Configuration
|
||||
|
||||
The example uses the following volume configuration:
|
||||
|
||||
```bash
|
||||
RUSTFS_VOLUMES=http://rustfs-node{1...4}:9000/data/rustfs{1...4}
|
||||
```
|
||||
|
||||
This expands to 16 endpoints (4 nodes × 4 drives):
|
||||
|
||||
- Node 1: `/data/rustfs1`, `/data/rustfs2`, `/data/rustfs3`, `/data/rustfs4`
|
||||
- Node 2: `/data/rustfs1`, `/data/rustfs2`, `/data/rustfs3`, `/data/rustfs4`
|
||||
- Node 3: `/data/rustfs1`, `/data/rustfs2`, `/data/rustfs3`, `/data/rustfs4`
|
||||
- Node 4: `/data/rustfs1`, `/data/rustfs2`, `/data/rustfs3`, `/data/rustfs4`
|
||||
|
||||
**Important:** Disk indexing starts at 1 to match the mounted paths (`/data/rustfs1..4`).
|
||||
|
||||
### Port Mappings
|
||||
|
||||
| Node | API Port | Console Port |
|
||||
|-------|----------|--------------|
|
||||
| node1 | 9000 | 9001 |
|
||||
| node2 | 9010 | 9011 |
|
||||
| node3 | 9020 | 9021 |
|
||||
| node4 | 9030 | 9031 |
|
||||
|
||||
### Startup Coordination
|
||||
|
||||
To prevent race conditions during startup where nodes might not find each other, a simple `sleep 3` command is added to
|
||||
each service's command. This provides a brief delay, allowing the network and other services to initialize before RustFS
|
||||
starts. For more complex scenarios, a more robust health-check dependency or an external entrypoint script might be
|
||||
required.
|
||||
|
||||
### Health Checks
|
||||
|
||||
Default health check using `nc` (netcat):
|
||||
|
||||
```yaml
|
||||
healthcheck:
|
||||
test: [ "CMD-SHELL", "nc -z localhost 9000 || exit 1" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
```
|
||||
|
||||
#### Alternative Health Checks
|
||||
|
||||
If your base image lacks `nc`, use one of these alternatives:
|
||||
|
||||
**Using curl:**
|
||||
|
||||
```yaml
|
||||
healthcheck:
|
||||
test: [ "CMD-SHELL", "curl -f http://localhost:9000/health || exit 1" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
```
|
||||
|
||||
**Using wget:**
|
||||
|
||||
```yaml
|
||||
healthcheck:
|
||||
test: [ "CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:9000/health || exit 1" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
```
|
||||
|
||||
### Brace Expansion Alternatives
|
||||
|
||||
If your Docker Compose runtime doesn't support brace expansion (`{1...4}`), replace with explicit endpoints:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=http://rustfs-node1:9000/data/rustfs1,http://rustfs-node1:9000/data/rustfs2,http://rustfs-node1:9000/data/rustfs3,http://rustfs-node1:9000/data/rustfs4,http://rustfs-node2:9000/data/rustfs1,http://rustfs-node2:9000/data/rustfs2,http://rustfs-node2:9000/data/rustfs3,http://rustfs-node2:9000/data/rustfs4,http://rustfs-node3:9000/data/rustfs1,http://rustfs-node3:9000/data/rustfs2,http://rustfs-node3:9000/data/rustfs3,http://rustfs-node3:9000/data/rustfs4,http://rustfs-node4:9000/data/rustfs1,http://rustfs-node4:9000/data/rustfs2,http://rustfs-node4:9000/data/rustfs3,http://rustfs-node4:9000/data/rustfs4
|
||||
```
|
||||
|
||||
## Using RUSTFS_CMD
|
||||
|
||||
The `RUSTFS_CMD` environment variable provides a fallback when no command is specified:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
- RUSTFS_CMD=rustfs # Default fallback command
|
||||
```
|
||||
|
||||
This allows the entrypoint to execute the correct command when Docker doesn't provide one.
|
||||
|
||||
## Testing the Deployment
|
||||
|
||||
After starting the cluster, verify it's working:
|
||||
|
||||
### Automated Testing
|
||||
|
||||
Use the provided test script for comprehensive validation:
|
||||
|
||||
```bash
|
||||
./test-deployment.sh
|
||||
```
|
||||
|
||||
This script tests:
|
||||
|
||||
- Container status (4/4 running)
|
||||
- Health checks (4/4 healthy)
|
||||
- API endpoints (4 ports)
|
||||
- Console endpoints (4 ports)
|
||||
- Inter-node connectivity
|
||||
- Data directory existence
|
||||
|
||||
### Manual Testing
|
||||
|
||||
For manual verification:
|
||||
|
||||
```bash
|
||||
# 1. Check all containers are healthy
|
||||
docker-compose ps
|
||||
|
||||
# 2. Test API endpoints
|
||||
for port in 9000 9010 9020 9030; do
|
||||
echo "Testing port $port..."
|
||||
curl -s http://localhost:${port}/health | jq '.'
|
||||
done
|
||||
|
||||
# 3. Test console endpoints
|
||||
for port in 9001 9011 9021 9031; do
|
||||
echo "Testing console port $port..."
|
||||
curl -s http://localhost:${port}/health | jq '.'
|
||||
done
|
||||
|
||||
# 4. Check inter-node connectivity
|
||||
docker exec rustfs-node1 nc -zv rustfs-node2 9000
|
||||
docker exec rustfs-node1 nc -zv rustfs-node3 9000
|
||||
docker exec rustfs-node1 nc -zv rustfs-node4 9000
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### VolumeNotFound Error
|
||||
|
||||
**Symptom:** Error message about `/data/rustfs0` not found.
|
||||
|
||||
**Solution:** This example uses `/data/rustfs{1...4}` indexing to match the mounted Docker volumes. Ensure your
|
||||
`RUSTFS_VOLUMES` configuration starts at index 1, not 0.
|
||||
|
||||
### Health Check Failures
|
||||
|
||||
**Symptom:** Containers show as unhealthy.
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Check if `nc` is available: `docker exec rustfs-node1 which nc`
|
||||
2. Use alternative health checks (curl/wget) as documented above
|
||||
3. Increase `start_period` if nodes need more time to initialize
|
||||
|
||||
### Startup Timeouts
|
||||
|
||||
**Symptom:** Services timeout waiting for peers.
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Check logs: `docker-compose logs rustfs-node1`
|
||||
2. Verify network connectivity: `docker-compose exec rustfs-node1 ping rustfs-node2`
|
||||
3. Consider increasing the `sleep` duration in the `docker-compose.yml` `command` directive if a longer delay is needed.
|
||||
|
||||
### Permission Issues
|
||||
|
||||
**Symptom:** Cannot create directories or write data.
|
||||
|
||||
**Solution:** Ensure volumes have correct permissions or set `RUSTFS_UID` and `RUSTFS_GID` environment variables.
|
||||
|
||||
## Advanced Configuration
|
||||
|
||||
### Custom Credentials
|
||||
|
||||
Replace default credentials in production:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
- RUSTFS_ACCESS_KEY=your_access_key
|
||||
- RUSTFS_SECRET_KEY=your_secret_key
|
||||
```
|
||||
|
||||
### TLS Configuration
|
||||
|
||||
Add TLS certificates:
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
- ./certs:/opt/tls:ro
|
||||
environment:
|
||||
- RUSTFS_TLS_PATH=/opt/tls
|
||||
```
|
||||
|
||||
### Resource Limits
|
||||
|
||||
Add resource constraints:
|
||||
|
||||
```yaml
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '2'
|
||||
memory: 4G
|
||||
reservations:
|
||||
cpus: '1'
|
||||
memory: 2G
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [CHECKLIST.md](./CHECKLIST.md) - Step-by-step verification guide
|
||||
- [../../console-separation.md](../../console-separation.md) - Console & endpoint service separation guide
|
||||
- [../../../examples/docker-comprehensive.yml](../../../examples/docker-comprehensive.yml) - More deployment examples
|
||||
- [Issue #618](https://github.com/rustfs/rustfs/issues/618) - Original VolumeNotFound issue
|
||||
|
||||
## References
|
||||
|
||||
- RustFS Documentation: https://rustfs.io
|
||||
- Docker Compose Documentation: https://docs.docker.com/compose/
|
||||
182
docs/examples/mnmd/docker-compose.yml
Normal file
182
docs/examples/mnmd/docker-compose.yml
Normal file
@@ -0,0 +1,182 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# MNMD (Multi-Node Multi-Drive) Docker Compose Example
|
||||
# 4 nodes x 4 drives configuration
|
||||
# This example demonstrates a complete, ready-to-use MNMD deployment
|
||||
# addressing startup coordination and VolumeNotFound issues.
|
||||
|
||||
services:
|
||||
rustfs-node1:
|
||||
image: rustfs/rustfs:latest
|
||||
container_name: rustfs-node1
|
||||
hostname: rustfs-node1
|
||||
environment:
|
||||
# Use service names and correct disk indexing (1..4 to match mounted paths)
|
||||
- RUSTFS_VOLUMES=http://rustfs-node{1...4}:9000/data/rustfs{1...4}
|
||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
|
||||
- RUSTFS_ACCESS_KEY=rustfsadmin
|
||||
- RUSTFS_SECRET_KEY=rustfsadmin
|
||||
- RUSTFS_CMD=rustfs
|
||||
ports:
|
||||
- "9000:9000" # API endpoint
|
||||
- "9001:9001" # Console
|
||||
volumes:
|
||||
- node1-data1:/data/rustfs1
|
||||
- node1-data2:/data/rustfs2
|
||||
- node1-data3:/data/rustfs3
|
||||
- node1-data4:/data/rustfs4
|
||||
command: [ "sh", "-c", "sleep 3 && rustfs" ]
|
||||
healthcheck:
|
||||
test:
|
||||
[
|
||||
"CMD",
|
||||
"sh", "-c",
|
||||
"curl -f http://localhost:9000/health && curl -f http://localhost:9001/health"
|
||||
]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
networks:
|
||||
- rustfs-mnmd
|
||||
|
||||
rustfs-node2:
|
||||
image: rustfs/rustfs:latest
|
||||
container_name: rustfs-node2
|
||||
hostname: rustfs-node2
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=http://rustfs-node{1...4}:9000/data/rustfs{1...4}
|
||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
|
||||
- RUSTFS_ACCESS_KEY=rustfsadmin
|
||||
- RUSTFS_SECRET_KEY=rustfsadmin
|
||||
- RUSTFS_CMD=rustfs
|
||||
ports:
|
||||
- "9010:9000" # API endpoint
|
||||
- "9011:9001" # Console
|
||||
volumes:
|
||||
- node2-data1:/data/rustfs1
|
||||
- node2-data2:/data/rustfs2
|
||||
- node2-data3:/data/rustfs3
|
||||
- node2-data4:/data/rustfs4
|
||||
command: [ "sh", "-c", "sleep 3 && rustfs" ]
|
||||
healthcheck:
|
||||
test:
|
||||
[
|
||||
"CMD",
|
||||
"sh", "-c",
|
||||
"curl -f http://localhost:9000/health && curl -f http://localhost:9001/health"
|
||||
]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
networks:
|
||||
- rustfs-mnmd
|
||||
|
||||
rustfs-node3:
|
||||
image: rustfs/rustfs:latest
|
||||
container_name: rustfs-node3
|
||||
hostname: rustfs-node3
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=http://rustfs-node{1...4}:9000/data/rustfs{1...4}
|
||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
|
||||
- RUSTFS_ACCESS_KEY=rustfsadmin
|
||||
- RUSTFS_SECRET_KEY=rustfsadmin
|
||||
- RUSTFS_CMD=rustfs
|
||||
ports:
|
||||
- "9020:9000" # API endpoint
|
||||
- "9021:9001" # Console
|
||||
volumes:
|
||||
- node3-data1:/data/rustfs1
|
||||
- node3-data2:/data/rustfs2
|
||||
- node3-data3:/data/rustfs3
|
||||
- node3-data4:/data/rustfs4
|
||||
command: [ "sh", "-c", "sleep 3 && rustfs" ]
|
||||
healthcheck:
|
||||
test:
|
||||
[
|
||||
"CMD",
|
||||
"sh", "-c",
|
||||
"curl -f http://localhost:9000/health && curl -f http://localhost:9001/health"
|
||||
]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
networks:
|
||||
- rustfs-mnmd
|
||||
|
||||
rustfs-node4:
|
||||
image: rustfs/rustfs:latest
|
||||
container_name: rustfs-node4
|
||||
hostname: rustfs-node4
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=http://rustfs-node{1...4}:9000/data/rustfs{1...4}
|
||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
|
||||
- RUSTFS_ACCESS_KEY=rustfsadmin
|
||||
- RUSTFS_SECRET_KEY=rustfsadmin
|
||||
- RUSTFS_CMD=rustfs
|
||||
ports:
|
||||
- "9030:9000" # API endpoint
|
||||
- "9031:9001" # Console
|
||||
volumes:
|
||||
- node4-data1:/data/rustfs1
|
||||
- node4-data2:/data/rustfs2
|
||||
- node4-data3:/data/rustfs3
|
||||
- node4-data4:/data/rustfs4
|
||||
command: [ "sh", "-c", "sleep 3 && rustfs" ]
|
||||
healthcheck:
|
||||
test:
|
||||
[
|
||||
"CMD",
|
||||
"sh", "-c",
|
||||
"curl -f http://localhost:9000/health && curl -f http://localhost:9001/health"
|
||||
]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
networks:
|
||||
- rustfs-mnmd
|
||||
|
||||
networks:
|
||||
rustfs-mnmd:
|
||||
driver: bridge
|
||||
|
||||
volumes:
|
||||
node1-data1:
|
||||
node1-data2:
|
||||
node1-data3:
|
||||
node1-data4:
|
||||
node2-data1:
|
||||
node2-data2:
|
||||
node2-data3:
|
||||
node2-data4:
|
||||
node3-data1:
|
||||
node3-data2:
|
||||
node3-data3:
|
||||
node3-data4:
|
||||
node4-data1:
|
||||
node4-data2:
|
||||
node4-data3:
|
||||
node4-data4:
|
||||
172
docs/examples/mnmd/test-deployment.sh
Executable file
172
docs/examples/mnmd/test-deployment.sh
Executable file
@@ -0,0 +1,172 @@
|
||||
#!/bin/bash
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# test-deployment.sh - Quick test script for MNMD deployment
|
||||
# Usage: ./test-deployment.sh
|
||||
|
||||
set -e
|
||||
|
||||
# Colors for output
|
||||
GREEN='\033[0;32m'
|
||||
RED='\033[0;31m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
echo "========================================="
|
||||
echo "RustFS MNMD Deployment Test"
|
||||
echo "========================================="
|
||||
echo ""
|
||||
|
||||
# Test 1: Check if all containers are running
|
||||
echo "Test 1: Checking container status..."
|
||||
RUNNING=$(docker-compose ps | grep -c "Up" || echo "0")
|
||||
if [ "$RUNNING" -eq 4 ]; then
|
||||
echo -e "${GREEN}✓ All 4 containers are running${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Only $RUNNING/4 containers are running${NC}"
|
||||
docker-compose ps
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Test 2: Check health status
|
||||
echo "Test 2: Checking health status..."
|
||||
HEALTHY=0
|
||||
for node in rustfs-node1 rustfs-node2 rustfs-node3 rustfs-node4; do
|
||||
STATUS=$(docker inspect "$node" --format='{{.State.Health.Status}}' 2>/dev/null || echo "unknown")
|
||||
if [ "$STATUS" = "healthy" ]; then
|
||||
echo -e " ${GREEN}✓ $node is healthy${NC}"
|
||||
HEALTHY=$((HEALTHY + 1))
|
||||
elif [ "$STATUS" = "starting" ]; then
|
||||
echo -e " ${YELLOW}⚠ $node is starting (wait a moment)${NC}"
|
||||
else
|
||||
echo -e " ${RED}✗ $node status: $STATUS${NC}"
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$HEALTHY" -eq 4 ]; then
|
||||
echo -e "${GREEN}✓ All containers are healthy${NC}"
|
||||
elif [ "$HEALTHY" -gt 0 ]; then
|
||||
echo -e "${YELLOW}⚠ $HEALTHY/4 containers are healthy (some may still be starting)${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ No containers are healthy${NC}"
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Test 3: Check API endpoints
|
||||
echo "Test 3: Testing API endpoints..."
|
||||
PORTS=(9000 9010 9020 9030)
|
||||
API_SUCCESS=0
|
||||
for port in "${PORTS[@]}"; do
|
||||
if curl -sf http://localhost:${port}/health >/dev/null 2>&1; then
|
||||
echo -e " ${GREEN}✓ API on port $port is responding${NC}"
|
||||
API_SUCCESS=$((API_SUCCESS + 1))
|
||||
else
|
||||
echo -e " ${RED}✗ API on port $port is not responding${NC}"
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$API_SUCCESS" -eq 4 ]; then
|
||||
echo -e "${GREEN}✓ All API endpoints are working${NC}"
|
||||
else
|
||||
echo -e "${YELLOW}⚠ $API_SUCCESS/4 API endpoints are working${NC}"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Test 4: Check Console endpoints
|
||||
echo "Test 4: Testing Console endpoints..."
|
||||
CONSOLE_PORTS=(9001 9011 9021 9031)
|
||||
CONSOLE_SUCCESS=0
|
||||
for port in "${CONSOLE_PORTS[@]}"; do
|
||||
if curl -sf http://localhost:${port}/health >/dev/null 2>&1; then
|
||||
echo -e " ${GREEN}✓ Console on port $port is responding${NC}"
|
||||
CONSOLE_SUCCESS=$((CONSOLE_SUCCESS + 1))
|
||||
else
|
||||
echo -e " ${RED}✗ Console on port $port is not responding${NC}"
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$CONSOLE_SUCCESS" -eq 4 ]; then
|
||||
echo -e "${GREEN}✓ All Console endpoints are working${NC}"
|
||||
else
|
||||
echo -e "${YELLOW}⚠ $CONSOLE_SUCCESS/4 Console endpoints are working${NC}"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Test 5: Check inter-node connectivity
|
||||
echo "Test 5: Testing inter-node connectivity..."
|
||||
CONN_SUCCESS=0
|
||||
for node in rustfs-node2 rustfs-node3 rustfs-node4; do
|
||||
if docker exec rustfs-node1 nc -z "$node" 9000 2>/dev/null; then
|
||||
echo -e " ${GREEN}✓ node1 → $node connection OK${NC}"
|
||||
CONN_SUCCESS=$((CONN_SUCCESS + 1))
|
||||
else
|
||||
echo -e " ${RED}✗ node1 → $node connection failed${NC}"
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$CONN_SUCCESS" -eq 3 ]; then
|
||||
echo -e "${GREEN}✓ All inter-node connections are working${NC}"
|
||||
else
|
||||
echo -e "${YELLOW}⚠ $CONN_SUCCESS/3 inter-node connections are working${NC}"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Test 6: Verify data directories
|
||||
echo "Test 6: Verifying data directories..."
|
||||
DIR_SUCCESS=0
|
||||
for i in {1..4}; do
|
||||
if docker exec rustfs-node1 test -d "/data/rustfs${i}"; then
|
||||
DIR_SUCCESS=$((DIR_SUCCESS + 1))
|
||||
else
|
||||
echo -e " ${RED}✗ /data/rustfs${i} not found in node1${NC}"
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$DIR_SUCCESS" -eq 4 ]; then
|
||||
echo -e "${GREEN}✓ All data directories exist${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Only $DIR_SUCCESS/4 data directories exist${NC}"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Summary
|
||||
echo "========================================="
|
||||
echo "Test Summary"
|
||||
echo "========================================="
|
||||
echo "Containers running: $RUNNING/4"
|
||||
echo "Healthy containers: $HEALTHY/4"
|
||||
echo "API endpoints: $API_SUCCESS/4"
|
||||
echo "Console endpoints: $CONSOLE_SUCCESS/4"
|
||||
echo "Inter-node connections: $CONN_SUCCESS/3"
|
||||
echo "Data directories: $DIR_SUCCESS/4"
|
||||
echo ""
|
||||
|
||||
TOTAL=$((RUNNING + HEALTHY + API_SUCCESS + CONSOLE_SUCCESS + CONN_SUCCESS + DIR_SUCCESS))
|
||||
MAX_SCORE=23
|
||||
|
||||
if [ "$TOTAL" -eq "$MAX_SCORE" ]; then
|
||||
echo -e "${GREEN}✓ All tests passed! Deployment is working correctly.${NC}"
|
||||
exit 0
|
||||
elif [ "$TOTAL" -ge 20 ]; then
|
||||
echo -e "${YELLOW}⚠ Most tests passed. Some components may still be starting up.${NC}"
|
||||
echo " Try running this script again in a few moments."
|
||||
exit 0
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed. Check the output above and logs for details.${NC}"
|
||||
echo " Run 'docker-compose logs' for more information."
|
||||
exit 1
|
||||
fi
|
||||
Reference in New Issue
Block a user