sdgoij/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-01-16 17:20:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: implement Docker improvements and binary build scripts (#191)

Browse Source 
* feat: implement Docker improvements and binary build scripts

This commit transforms the RustFS Docker build system to follow MinIO's best practices:

## 🏗️ Binary Build Script (build-rustfs.sh)
- Create independent binary compilation script for multi-platform builds
- Support x86_64 and aarch64 Linux musl targets
- Include checksum generation and optional binary signing
- Support cross-compilation and upload functionality
- Automated target installation and environment setup

## 🐳 Docker Improvements
- Rewrite Dockerfiles to download precompiled binaries instead of building from source
- Follow MinIO's approach for security and binary verification
- Add comprehensive LABEL metadata (version, build-date, vcs-ref)
- Implement proper environment variable management
- Add signature verification with minisign (commented for future use)
- Include static curl download for minimal runtime dependencies

## 🚀 Enhanced Build Script (docker-buildx.sh)
- Inspired by MinIO's docker-buildx.sh for consistency and reliability
- Support multiple platforms with proper build arguments
- Auto-detect git versions and pass metadata to containers
- Improved error messages with helpful troubleshooting hints
- Cleanup and cache management between builds

## 🛠️ Supporting Scripts
- scripts/download-static-curl.sh: Download statically compiled curl
- scripts/setup-test-binaries.sh: Create test binaries for local development

## 📋 Key Benefits
- Faster Docker builds (download vs compile)
- Better security with signature verification
- Consistent with industry standards (MinIO approach)
- Proper multi-platform support
- Enhanced metadata and traceability
- Independent binary distribution capability

* feat: update Docker files to use Aliyun OSS for binary downloads

* feat: merge stash with OSS binary download improvements

- Remove old build_rustfs.sh script
- Keep Aliyun OSS download URLs for binary retrieval
- Maintain Docker build improvements from stash
- Resolve merge conflicts between stash and OSS updates

* feat: improve build-rustfs.sh with auto platform detection

- Auto-detect current platform using uname (like old build_rustfs.sh)
- Default to building for current platform only
- Add --all-platforms flag for cross-compilation to Linux musl targets
- Support macOS (darwin) and Linux platforms
- Auto-enable cross compilation when needed
- Provide better usage examples and platform detection info

This makes the script much more user-friendly by default while
maintaining flexibility for cross-compilation scenarios.

* refactor: simplify build-rustfs.sh for CI/CD pipeline usage

- Remove cross-compilation complexity (each CI runner builds natively)
- Focus on single platform builds per runner
- Remove --all-platforms and --cross options
- Simplify to match CI/CD workflow where:
  * Linux x86_64 runner builds Linux x86_64 binary
  * Linux ARM64 runner builds Linux ARM64 binary
  * macOS x86_64 runner builds macOS x86_64 binary
  * macOS ARM64 runner builds macOS ARM64 binary
- Keep signing and upload functionality for release CI
- Make the script's purpose and usage clearer

This aligns with the user's understanding that build scripts should
focus on native compilation for the current platform only.

* feat: update download server domain to dl.rustfs.com

- Update Dockerfile to use dl.rustfs.com/dev/ for development binaries
- Update Dockerfile.release to use dl.rustfs.com/release/ for release binaries
- Update docker-buildx.sh error messages with new URLs
- Update build-rustfs.sh upload target to dl.rustfs.com
- Update test scripts to reference new domain
- Clean up remaining git conflict markers

This centralizes all binary downloads through the official
dl.rustfs.com domain instead of direct OSS access.

* fix: correct dl.rustfs.com path structure to include /artifacts/rustfs/

- Update all download URLs to use correct path structure:
  * Dev: https://dl.rustfs.com/artifacts/rustfs/dev/
  * Release: https://dl.rustfs.com/artifacts/rustfs/release/
- Test confirmed both paths return HTTP 200 with application/zip content-type
- Update Dockerfile, Dockerfile.release, docker-buildx.sh, and build-rustfs.sh
- Update test scripts with correct base path

The dl.rustfs.com domain requires the /artifacts/rustfs/ prefix
to access the binary files correctly.

* feat: refactor Dockerfile to download binaries from GitHub Releases

- Changed binary download source from dl.rustfs.com to GitHub Releases
- Added support for latest release auto-detection via GitHub API
- Enhanced error handling with detailed messages and helpful links
- Added optional checksum verification using SHA256SUMS
- Improved architecture support for amd64 and arm64
- Removed unnecessary minisign installation
- Added jq dependency for JSON parsing

* feat: consolidate Docker build to use single Dockerfile

- Removed Dockerfile.release and use unified Dockerfile instead
- Updated docker-buildx.sh to use single Dockerfile with build args
- Both latest and release variants now use GitHub Releases
- Simplified build process and reduced maintenance overhead
- Updated error messages to point to GitHub releases

* chore: remove unused Dockerfile.obs

- Removed Dockerfile.obs as it's no longer needed
- Simplified Docker build configuration

* feat: unify Docker prebuild variants to use GitHub Releases

- Updated .docker/alpine/Dockerfile.prebuild to download from GitHub Releases
- Updated .docker/ubuntu/Dockerfile.prebuild to download from GitHub Releases
- All prebuild variants now consistently use GitHub Releases as binary source
- Added checksum verification for all prebuild variants
- Updated .docker/README.md to reflect unified GitHub Releases approach
- Improved error handling and user guidance in all prebuild Dockerfiles

* feat: major Docker structure simplification and consolidation

## 🎯 Simplified Docker Structure

Moved from complex multi-directory structure to clean root-level organization:

### Before:
- Dockerfile (production)
- .docker/alpine/Dockerfile.prebuild (duplicate)
- .docker/alpine/Dockerfile.source
- .docker/ubuntu/Dockerfile.prebuild (duplicate)
- .docker/ubuntu/Dockerfile.source
- .docker/ubuntu/Dockerfile.dev

### After:
- Dockerfile (production - Alpine + GitHub Releases)
- Dockerfile.source (source build - Ubuntu + cross-compilation)
- Dockerfile.dev (development - Ubuntu + full toolchain)

## 🔧 Key Changes

- **Eliminated Duplicates**: Removed redundant prebuild variants
- **Moved Core Files**: Dockerfile.{source,dev} now in root directory
- **Unified Configuration**: cargo.config.toml moved to root
- **Updated References**: Fixed all GitHub Actions and docker-compose paths
- **Simplified CI Matrix**: Reduced from 5 to 3 Docker variants

## 📦 Preserved Valuable Diversity

- **Production**: Alpine-based for minimal size
- **Source**: Ubuntu-based with cross-compilation support
- **Development**: Ubuntu-based with full development tools

## 🚀 Benefits

- ✅ Cleaner project structure
- ✅ Easier maintenance and navigation
- ✅ Reduced CI/CD complexity
- ✅ Faster build matrix execution
- ✅ Maintained functionality and flexibility

* chore: remove duplicate cargo.config.toml from .docker directory

The file is now in the root directory and no longer needed in .docker/

* fix: update all references to removed Dockerfile files

- Updated .docker/compose/README.md to reference Dockerfile.source instead of Dockerfile.obs
- Updated docker-compose.yml to use Dockerfile.source instead of Dockerfile.dev
- Updated scripts/build-docker-multiarch.sh to use Dockerfile.source for devenv builds
- Updated .github/workflows/docker.yml to use Dockerfile.source for dev builds
- Updated Makefile to use Dockerfile.source for init-devenv target
- Updated .docker/README.md to remove references to non-existent Dockerfile.dev
- Ensured all Docker configurations consistently use the unified Dockerfile structure

* chore: remove unnecessary console static assets download

- Remove obsolete download steps from build.yml and performance.yml
- Console static assets are already embedded via rust-embed in rustfs/static/
- The download from dl.rustfs.com is no longer needed as project contains complete console assets
- This improves build reliability and reduces external dependencies
- Replaced with verification steps that confirm embedded assets are present

* feat: update Makefile and README.md for new Docker build system

- Updated Makefile to use unified Docker build system:
  - Replace references to non-existent Dockerfile.ubuntu22.04 and Dockerfile.rockylinux9.3
  - Add new docker-buildx targets using docker-buildx.sh script
  - Deprecate old docker-build-multiarch targets with warnings
  - Add docker-build-production and docker-build-source targets
  - Update help-docker with new command structure

- Updated README.md with docker-buildx.sh usage:
  - Add comprehensive Docker build from source section
  - Document multi-architecture build capabilities
  - Include both script and Make target examples
  - Show registry flexibility and build optimization features
  - Update step numbers in quickstart guide

- Improve developer experience with clear documentation and updated tooling
- Maintain backward compatibility with deprecation warnings

* feat: integrate console assets download into build-rustfs.sh

- Added console download functionality to build-rustfs.sh:
  - New flags: --download-console, --force-console-update, --console-version
  - Intelligent detection of existing console assets
  - Retry logic with fallback error handling
  - Consistent with Docker build asset management

- Updated scripts to use unified build process:
  - scripts/static.sh: Now uses build-rustfs.sh for console downloads
  - scripts/run.sh: Uses build-rustfs.sh instead of direct curl
  - scripts/run.ps1: Updated with guidance for Windows users

- Benefits:
  - Unified asset management across all build processes
  - Consistent version handling and retry logic
  - Eliminates duplicate download logic
  - Better error handling and user feedback
  - Preparation for CI/CD integration

- Removed unused download-static-curl.sh script

This change centralizes console asset management and prepares for
streamlined CI/CD processes where build-rustfs.sh becomes the
single point of truth for binary and asset builds.

* fix: update PowerShell script to use unified console asset management

- Updated scripts/run.ps1 to use build-rustfs.sh for console asset downloads
- Added guidance for Windows users to use the unified build script
- Maintains consistency across all platform-specific scripts

* feat: add binary verification to build script

- Add verify_binary function to test built binaries
- Test --help and --version commands
- Verify binary structure with readelf/otool
- Add --skip-verification option for cross-compilation
- Include verification status in build output
- Automatic error handling if verification fails

* feat: add platform selection support to build script

- Add --platform parameter to build-rustfs.sh for target platform selection
- Implement cross-compilation support with automatic 'cross' tool detection
- Auto-enable --skip-verification for cross-compilation scenarios
- Update all Makefile build targets to use unified build-rustfs.sh script
- Add helpful error messages and suggestions for cross-compilation failures
- Update help documentation with platform selection examples
- Improve build consistency across different architectures

* feat: modernize CI/CD build process with build-rustfs.sh

- Replace manual cargo build commands with unified build-rustfs.sh script
- Simplify matrix configuration by removing cross-compilation flags
- Ensure consistency between local and CI/CD builds
- Automatic cross-compilation tool detection and installation
- Built-in binary verification for quality assurance
- Unified console asset management
- Better error handling and suggestions

Benefits:
- Consistent build process across all environments
- Automatic detection and handling of cross-compilation scenarios
- Built-in quality checks with binary verification
- Reduced CI/CD configuration complexity
- Better maintainability with single source of truth for build logic

* feat: optimize CI/CD workspace path management

- Add WORKSPACE_DIR environment variable to cache github.workspace
- Set default working-directory at job level for consistency
- Use explicit workspace paths in critical operations
- Improve reliability and maintainability of CI/CD paths
- Ensure consistent behavior across different GitHub Actions environments

Benefits:
- More explicit and reliable path handling
- Better maintainability with centralized workspace reference
- Reduced risk of path-related issues in CI/CD
- Consistent working directory across all job steps

* refactor: simplify CI/CD path management - remove redundant workspace references

- Remove unnecessary WORKSPACE_DIR environment variable
- Remove redundant defaults.run.working-directory setting
- Use relative paths since GITHUB_WORKSPACE is the default working directory
- Follow GitHub Actions best practices by leveraging default behavior

As per GitHub Actions documentation, GITHUB_WORKSPACE is already the default
working directory, so explicit specification is unnecessary in most cases.

* docs: update Docker README to reflect current project state

- Fix directory structure: remove non-existent nginx/ directory
- Correct base OS: Dockerfile.source uses Debian Bookworm, not Ubuntu 22.04
- Add docker-buildx.sh script documentation
- Update Docker tag examples to match actual CI/CD workflows
- Add CI/CD integration section explaining automated builds
- Document build variants and manual build options
- Reflect current project architecture and tooling

These updates ensure the documentation accurately represents the current
Docker build system and CI/CD workflows.

* fix: update Docker command in rustfs README

- Replace quay.io registry with Docker Hub (rustfs/rustfs:latest)
- Remove separate console port 9001, console now runs on main port 9000
- Add both Docker and Podman examples for user choice
- Fix console access URL to use unified port

This aligns with the recent console port consolidation changes
and the project's move to Docker Hub as the primary registry.

* wip

* fix: remove unnecessary entrypoint.sh and fix Docker paths

* Update Dockerfile

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* cleanup: remove unused DOCKERFILE_PATH variable from Makefile

* feat: update Docker build to use dl.rustfs.com for binary downloads

- Replace GitHub releases download with dl.rustfs.com
- Add CHANNEL parameter support (release/dev)
- Update docker-buildx.sh to support channel-specific builds
- Improve error messages with new download URLs
- Support both latest and specific version downloads
- Add channel validation in build script

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This commit is contained in:
安正超
2025-07-14 19:15:46 +08:00
committed by GitHub
parent 04d1c8724d
commit 36e3efb5a5
27 changed files with 1225 additions and 1252 deletions
Download Patch File Download Diff File Expand all files Collapse all files

238
.docker/README.md
View File

@@ -1,157 +1,199 @@
# RustFS Docker Images
This directory contains organized Dockerfile configurations for building RustFS container images across multiple platforms and system versions.
This directory contains Docker configuration files and supporting infrastructure for building and running RustFS container images.
## 📁 Directory Structure
```
.docker/
├── alpine/ # Alpine Linux variants
│ ├── Dockerfile.prebuild # Alpine + pre-built binaries
│ └── Dockerfile.source # Alpine + source compilation
├── ubuntu/ # Ubuntu variants
│ ├── Dockerfile.prebuild # Ubuntu + pre-built binaries
│ ├── Dockerfile.source # Ubuntu + source compilation
│ └── Dockerfile.dev # Ubuntu + development environment
└── cargo.config.toml # Rust cargo configuration
rustfs/
├── Dockerfile # Production image (Alpine + GitHub Releases)
├── Dockerfile.source # Source build (Debian + cross-compilation)
├── cargo.config.toml # Rust cargo configuration
├── docker-buildx.sh # Multi-architecture build script
└── .docker/ # Supporting infrastructure
├── observability/ # Monitoring and observability configs
├── compose/ # Docker Compose configurations
├── mqtt/ # MQTT broker configs
└── openobserve-otel/ # OpenObserve + OpenTelemetry configs
```
## 🎯 Image Variants
### Production Images
### Core Images
| Variant | Base OS | Build Method | Size | Use Case |
|---------|---------|--------------|------|----------|
| `production` (default) | Alpine 3.18 | Pre-built | Smallest | Production deployment |
| `alpine` | Alpine 3.18 | Pre-built | Small | Explicit Alpine choice |
| `alpine-source` | Alpine 3.18 | Source build | Small | Custom Alpine builds |
| `ubuntu` | Ubuntu 22.04 | Pre-built | Medium | Ubuntu environments |
| `ubuntu-source` | Ubuntu 22.04 | Source build | Medium | Full Ubuntu compatibility |
### Development Images
| Variant | Base OS | Features | Use Case |
|---------|---------|----------|----------|
| `ubuntu-dev` | Ubuntu 22.04 | Full toolchain + dev tools | Interactive development |
| Image | Base OS | Build Method | Size | Use Case |
|-------|---------|--------------|------|----------|
| `production` (default) | Alpine 3.18 | GitHub Releases | Smallest | Production deployment |
| `source` | Debian Bookworm | Source build | Medium | Custom builds with cross-compilation |
| `dev` | Debian Bookworm | Development tools | Large | Interactive development |
## 🚀 Usage Examples
### Quick Start (Production)
```bash
# Default production image (Alpine + pre-built)
# Default production image (Alpine + GitHub Releases)
docker run -p 9000:9000 rustfs/rustfs:latest
# Specific version with production variant
docker run -p 9000:9000 rustfs/rustfs:1.2.3-production
# Explicit Alpine variant
docker run -p 9000:9000 rustfs/rustfs:latest-alpine
# Ubuntu-based production
docker run -p 9000:9000 rustfs/rustfs:latest-ubuntu
# Specific version
docker run -p 9000:9000 rustfs/rustfs:1.2.3
```
### Complete Tag Strategy Examples
```bash
# Stable Releases
docker run rustfs/rustfs:1.2.3 # Main version (production)
docker run rustfs/rustfs:1.2.3-production # Explicit production variant
docker run rustfs/rustfs:1.2.3-alpine # Explicit Alpine variant
docker run rustfs/rustfs:1.2.3-alpine-source # Alpine source build
docker run rustfs/rustfs:latest # Latest stable
docker run rustfs/rustfs:1.2.3 # Main version (production)
docker run rustfs/rustfs:1.2.3-production # Explicit production variant
docker run rustfs/rustfs:1.2.3-source # Source build variant
docker run rustfs/rustfs:latest # Latest stable
# Prerelease Versions
docker run rustfs/rustfs:1.3.0-alpha.2 # Specific alpha version
docker run rustfs/rustfs:1.3.0-alpha.2-alpine # Alpha with Alpine
docker run rustfs/rustfs:alpha # Latest alpha
docker run rustfs/rustfs:beta # Latest beta
docker run rustfs/rustfs:rc # Latest release candidate
docker run rustfs/rustfs:1.3.0-alpha.2 # Specific alpha version
docker run rustfs/rustfs:alpha # Latest alpha
docker run rustfs/rustfs:beta # Latest beta
docker run rustfs/rustfs:rc # Latest release candidate
# Development Versions
docker run rustfs/rustfs:dev # Latest development
docker run rustfs/rustfs:dev-13e4a0b # Specific commit
docker run rustfs/rustfs:dev-alpine # Development Alpine
docker run rustfs/rustfs:dev # Latest main branch development
docker run rustfs/rustfs:dev-13e4a0b # Specific commit
docker run rustfs/rustfs:dev-latest # Latest development
docker run rustfs/rustfs:main-latest # Main branch latest
```
### Development Environment
```bash
# Start development container
docker run -it -v $(pwd):/app -p 9000:9000 rustfs/rustfs:latest-ubuntu-dev
docker run -it -v $(pwd):/workspace -p 9000:9000 rustfs/rustfs:latest-dev
# Inside container:
cd /app
cargo build --release
cargo run
# Build from source locally
docker build -f Dockerfile.source -t rustfs:custom .
# Development with hot reload
docker-compose up rustfs-dev
```
## 🏗️ Build Arguments
## 🏗️ Build Arguments and Scripts
### Using docker-buildx.sh (Recommended)
For multi-architecture builds, use the provided script:
```bash
# Build latest version for all architectures
./docker-buildx.sh
# Build and push to registry
./docker-buildx.sh --push
# Build specific version
./docker-buildx.sh --release v1.2.3
# Build and push specific version
./docker-buildx.sh --release v1.2.3 --push
```
### Manual Docker Builds
All images support dynamic version selection:
```bash
# Build with specific version
docker build \
--build-arg VERSION="1.0.0" \
--build-arg BUILD_TYPE="release" \
-f .docker/alpine/Dockerfile.prebuild \
-t rustfs:1.0.0-alpine .
# Build production image with latest release
docker build --build-arg RELEASE="latest" -t rustfs:latest .
# Build from source with specific target
docker build -f Dockerfile.source \
--build-arg TARGETPLATFORM="linux/amd64" \
-t rustfs:source .
# Development build
docker build -f Dockerfile.source -t rustfs:dev .
```
## 🌐 Multi-Platform Support
## 🔧 Binary Download Sources
All images support multiple architectures:
### Unified GitHub Releases
- `linux/amd64` (Intel/AMD 64-bit)
- `linux/arm64` (ARM 64-bit, Apple Silicon, etc.)
The production image downloads from GitHub Releases for reliability and transparency:
## ⚡ Build Speed Optimizations
- ✅ **production** → GitHub Releases API with automatic latest detection
- ✅ **Checksum verification** → SHA256SUMS validation when available
- ✅ **Multi-architecture** → Supports amd64 and arm64
### Docker Build Optimizations
### Source Build
- **Multi-layer caching**: GitHub Actions cache + Registry cache
- **Parallel matrix builds**: All 5 variants build simultaneously
- **Multi-platform builds**: amd64/arm64 built in parallel
- **BuildKit features**: Advanced caching and inline cache
The source variant compiles from source code with advanced features:
### Rust Compilation Optimizations
- 🔧 **Cross-compilation** → Supports multiple target platforms via `TARGETPLATFORM`
- ⚡ **Build caching** → sccache for faster compilation
- 🎯 **Optimized builds** → Release optimizations with LTO and symbol stripping
- **sccache**: Distributed compilation cache for Rust builds
- **Parallel compilation**: Uses all available CPU cores (`-j $(nproc)`)
- **Optimized cargo config**: Sparse registry protocol, fast linker (lld)
- **Dependency caching**: Separate Docker layers for dependencies vs. source code
- **Release optimizations**: LTO, strip symbols, optimized codegen
## 📋 Architecture Support
### Cache Strategy
All variants support multi-architecture builds:
```yaml
# GitHub Actions cache
cache-from: type=gha,scope=docker-{variant}
cache-to: type=gha,mode=max,scope=docker-{variant}
- **linux/amd64** (x86_64)
- **linux/arm64** (aarch64)
# Registry cache (persistent across runs)
cache-from: type=registry,ref=ghcr.io/rustfs/rustfs:buildcache-{variant}
cache-to: type=registry,ref=ghcr.io/rustfs/rustfs:buildcache-{variant}
Architecture is automatically detected during build using Docker's `TARGETARCH` build argument.
## 🔐 Security Features
- **Checksum Verification**: Production image verifies SHA256SUMS when available
- **Non-root User**: All images run as user `rustfs` (UID 1000)
- **Minimal Runtime**: Production image only includes necessary dependencies
- **Secure Defaults**: No hardcoded credentials or keys
## 🛠️ Development Workflow
For local development and testing:
```bash
# Quick development setup
docker-compose up rustfs-dev
# Custom source build
docker build -f Dockerfile.source -t rustfs:custom .
# Run with development tools
docker run -it -v $(pwd):/workspace rustfs:custom bash
```
### Build Performance Comparison
## 🚀 CI/CD Integration
| Build Type | Time (Est.) | Cache Hit | Cache Miss |
|------------|-------------|-----------|-----------|
| Production (Alpine pre-built) | ~2-3 min | ~1 min | ~2 min |
| Alpine pre-built | ~2-3 min | ~1 min | ~2 min |
| Alpine source | ~8-12 min | ~3-5 min | ~10 min |
| Ubuntu pre-built | ~3-4 min | ~1-2 min | ~3 min |
| Ubuntu source | ~10-15 min | ~4-6 min | ~12 min |
The project uses GitHub Actions for automated multi-architecture Docker builds:
## 📋 Build Matrix
### Automated Builds
| Trigger | Version Format | Download Path | Image Tags |
|---------|---------------|---------------|------------|
| `push main` | `dev-{sha}` | `artifacts/rustfs/dev/` | `dev-{sha}-{variant}`, `dev-{variant}`, `dev` |
| `push 1.2.3` | `1.2.3` | `artifacts/rustfs/release/` | `1.2.3-{variant}`, `1.2.3`, `latest-{variant}`, `latest` |
| `push 1.3.0-alpha.2` | `1.3.0-alpha.2` | `artifacts/rustfs/release/` | `1.3.0-alpha.2-{variant}`, `alpha-{variant}`, `alpha` |
| `push 1.3.0-beta.1` | `1.3.0-beta.1` | `artifacts/rustfs/release/` | `1.3.0-beta.1-{variant}`, `beta-{variant}`, `beta` |
| `push 1.3.0-rc.1` | `1.3.0-rc.1` | `artifacts/rustfs/release/` | `1.3.0-rc.1-{variant}`, `rc-{variant}`, `rc` |
- **Tags**: Automatic builds triggered on version tags (e.g., `v1.2.3`)
- **Main Branch**: Development builds with `dev-latest` and `main-latest` tags
- **Pull Requests**: Test builds without registry push
### Build Variants
Each build creates three image variants:
- `rustfs/rustfs:v1.2.3` (production - Alpine-based)
- `rustfs/rustfs:v1.2.3-source` (source build - Debian-based)
- `rustfs/rustfs:v1.2.3-dev` (development - Debian-based with tools)
### Manual Builds
Trigger custom builds via GitHub Actions:
```bash
# Use workflow_dispatch to build specific versions
# Available options: latest, main-latest, dev-latest, v1.2.3, dev-abc123
```
## 📦 Supporting Infrastructure
The `.docker/` directory contains supporting configuration files:
- **observability/** - Prometheus, Grafana, OpenTelemetry configs
- **compose/** - Multi-service Docker Compose setups
- **mqtt/** - MQTT broker configurations
- **openobserve-otel/** - Log aggregation and tracing setup
See individual README files in each subdirectory for specific usage instructions.

142
.docker/alpine/Dockerfile.prebuild
View File

@@ -1,142 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Multi-stage Alpine build for minimal runtime image
FROM rust:1.88-alpine AS builder
# Build arguments for dynamic artifact download
ARG VERSION=""
ARG BUILD_TYPE="release"
ARG TARGETARCH
# Install build dependencies
RUN apk add --no-cache \
musl-dev \
pkgconfig \
openssl-dev \
openssl-libs-static \
curl \
unzip \
bash \
wget \
ca-certificates
# Install protoc
RUN wget https://github.com/protocolbuffers/protobuf/releases/download/v31.1/protoc-31.1-linux-x86_64.zip \
&& unzip protoc-31.1-linux-x86_64.zip -d protoc3 \
&& mv protoc3/bin/* /usr/local/bin/ && chmod +x /usr/local/bin/protoc \
&& mv protoc3/include/* /usr/local/include/ && rm -rf protoc-31.1-linux-x86_64.zip protoc3
# Install flatc
RUN wget https://github.com/google/flatbuffers/releases/download/v25.2.10/Linux.flatc.binary.g++-13.zip \
&& unzip Linux.flatc.binary.g++-13.zip \
&& mv flatc /usr/local/bin/ && chmod +x /usr/local/bin/flatc \
&& rm -rf Linux.flatc.binary.g++-13.zip
# Download pre-built binary (VERSION is required)
RUN if [ -z "$VERSION" ]; then \
echo "❌ ERROR: VERSION build argument is required"; \
echo "Please provide VERSION (e.g., main-latest, latest, v1.0.0, dev-abc123)"; \
exit 1; \
fi; \
\
# Map TARGETARCH to our naming convention
case "${TARGETARCH}" in \
amd64) ARCH="x86_64" ;; \
arm64) ARCH="aarch64" ;; \
*) echo "❌ Unsupported architecture: ${TARGETARCH}" && exit 1 ;; \
esac; \
\
# Handle VERSION with different patterns
if [[ "$VERSION" == "main-latest" ]]; then \
DOWNLOAD_PATH="artifacts/rustfs/dev"; \
FILENAME="rustfs-linux-${ARCH}-main-latest.zip"; \
elif [[ "$VERSION" == "latest" ]]; then \
DOWNLOAD_PATH="artifacts/rustfs/release"; \
FILENAME="rustfs-linux-${ARCH}-latest.zip"; \
elif [[ "$VERSION" == dev-* ]]; then \
DOWNLOAD_PATH="artifacts/rustfs/dev"; \
FILENAME="rustfs-linux-${ARCH}-dev-${VERSION#dev-}.zip"; \
else \
DOWNLOAD_PATH="artifacts/rustfs/release"; \
FILENAME="rustfs-linux-${ARCH}-v${VERSION}.zip"; \
fi; \
\
# Download the binary
DOWNLOAD_URL="https://dl.rustfs.com/${DOWNLOAD_PATH}/${FILENAME}"; \
echo "🔽 Downloading RustFS binary from: ${DOWNLOAD_URL}"; \
\
# Download with clear error handling
if ! curl -fsSL --connect-timeout 30 --max-time 120 -o /tmp/rustfs.zip "${DOWNLOAD_URL}"; then \
echo "❌ Failed to download binary from: ${DOWNLOAD_URL}"; \
echo "💡 Please ensure the binary exists or trigger a build first"; \
echo "💡 Available options:"; \
echo " - For main-latest: Push to main branch or run build workflow"; \
echo " - For latest: Create a release tag"; \
echo " - For dev-xxx: Run build workflow with specific commit"; \
exit 1; \
fi; \
\
# Extract binary
if ! unzip -o /tmp/rustfs.zip -d /tmp >/dev/null 2>&1; then \
echo "❌ Failed to extract downloaded binary"; \
exit 1; \
fi; \
\
# Install binary
mv /tmp/rustfs /usr/local/bin/rustfs; \
chmod +x /usr/local/bin/rustfs; \
rm -rf /tmp/*; \
\
echo "✅ Successfully downloaded and installed RustFS binary (${VERSION})"
# Final Alpine runtime image
FROM alpine:3.18
RUN apk add --no-cache \
ca-certificates \
tzdata \
bash
# Create rustfs user for security
RUN addgroup -g 1000 rustfs && \
adduser -D -u 1000 -G rustfs rustfs
WORKDIR /app
# Copy binary from builder
COPY --from=builder /usr/local/bin/rustfs /app/rustfs
RUN chmod +x /app/rustfs && chown rustfs:rustfs /app/rustfs
# Create data directories
RUN mkdir -p /data && chown -R rustfs:rustfs /data /app
# Switch to non-root user
USER rustfs
# Environment variables
ENV RUSTFS_ACCESS_KEY=rustfsadmin \
RUSTFS_SECRET_KEY=rustfsadmin \
RUSTFS_ADDRESS=":9000" \
RUSTFS_CONSOLE_ENABLE=true \
RUSTFS_VOLUMES=/data \
RUST_LOG=warn
EXPOSE 9000
# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:9000/health || exit 1
CMD ["/app/rustfs"]

126
.docker/alpine/Dockerfile.source
View File

@@ -1,126 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Multi-stage Alpine build from source
FROM rust:1.88-alpine AS builder
# Install build dependencies
RUN apk add --no-cache \
musl-dev \
pkgconfig \
openssl-dev \
openssl-libs-static \
curl \
unzip \
bash \
wget \
ca-certificates \
git
# Install sccache for Rust compilation caching
RUN wget https://github.com/mozilla/sccache/releases/download/v0.8.1/sccache-v0.8.1-x86_64-unknown-linux-musl.tar.gz \
&& tar -xzf sccache-v0.8.1-x86_64-unknown-linux-musl.tar.gz \
&& mv sccache-v0.8.1-x86_64-unknown-linux-musl/sccache /usr/local/bin/ \
&& chmod +x /usr/local/bin/sccache \
&& rm -rf sccache-v0.8.1-x86_64-unknown-linux-musl.tar.gz sccache-v0.8.1-x86_64-unknown-linux-musl
# Set up sccache environment
ENV RUSTC_WRAPPER=sccache \
SCCACHE_DIR=/tmp/sccache \
SCCACHE_CACHE_SIZE=2G
# Install protoc
RUN wget https://github.com/protocolbuffers/protobuf/releases/download/v31.1/protoc-31.1-linux-x86_64.zip \
&& unzip protoc-31.1-linux-x86_64.zip -d protoc3 \
&& mv protoc3/bin/* /usr/local/bin/ && chmod +x /usr/local/bin/protoc \
&& mv protoc3/include/* /usr/local/include/ && rm -rf protoc-31.1-linux-x86_64.zip protoc3
# Install flatc
RUN wget https://github.com/google/flatbuffers/releases/download/v25.2.10/Linux.flatc.binary.g++-13.zip \
&& unzip Linux.flatc.binary.g++-13.zip \
&& mv flatc /usr/local/bin/ && chmod +x /usr/local/bin/flatc \
&& rm -rf Linux.flatc.binary.g++-13.zip
WORKDIR /usr/src/rustfs
# Copy cargo configuration for optimized builds
COPY .docker/cargo.config.toml ./.cargo/config.toml
# Copy cargo files for dependency caching
COPY Cargo.toml Cargo.lock ./
COPY */Cargo.toml ./*/
# Create dummy main.rs files for dependency compilation
RUN find . -name "Cargo.toml" -not -path "./Cargo.toml" | \
xargs -I {} dirname {} | \
xargs -I {} sh -c 'mkdir -p {}/src && echo "fn main() {}" > {}/src/main.rs'
# Configure cargo for optimized builds
ENV CARGO_NET_GIT_FETCH_WITH_CLI=true \
CARGO_REGISTRIES_CRATES_IO_PROTOCOL=sparse \
CARGO_INCREMENTAL=0 \
CARGO_PROFILE_RELEASE_DEBUG=false \
CARGO_PROFILE_RELEASE_SPLIT_DEBUGINFO=off \
CARGO_PROFILE_RELEASE_STRIP=symbols
# Build dependencies only (cache layer) with optimizations
RUN cargo build --release --target x86_64-unknown-linux-musl -j $(nproc)
# Copy source code
COPY . .
# Build the actual application with optimizations
RUN sccache --start-server 2>/dev/null || true && \
cargo build --release --target x86_64-unknown-linux-musl --bin rustfs -j $(nproc) && \
sccache --show-stats || true
# Final Alpine runtime image
FROM alpine:3.18
RUN apk add --no-cache \
ca-certificates \
tzdata \
bash
# Create rustfs user for security
RUN addgroup -g 1000 rustfs && \
adduser -D -u 1000 -G rustfs rustfs
WORKDIR /app
# Copy binary from builder
COPY --from=builder /usr/src/rustfs/target/x86_64-unknown-linux-musl/release/rustfs /app/rustfs
RUN chmod +x /app/rustfs && chown rustfs:rustfs /app/rustfs
# Create data directories
RUN mkdir -p /data && chown -R rustfs:rustfs /data /app
# Switch to non-root user
USER rustfs
# Environment variables
ENV RUSTFS_ACCESS_KEY=rustfsadmin \
RUSTFS_SECRET_KEY=rustfsadmin \
RUSTFS_ADDRESS=":9000" \
RUSTFS_CONSOLE_ENABLE=true \
RUSTFS_VOLUMES=/data \
RUST_LOG=warn
EXPOSE 9000
# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:9000/health || exit 1
CMD ["/app/rustfs"]

2
.docker/compose/README.md
View File

@@ -25,7 +25,7 @@ This directory contains specialized Docker Compose configurations and their asso
- **`docker-compose.observability.yaml`** - **Observability Focus**
- Specialized setup for testing observability features
- Includes OpenTelemetry, Jaeger, Prometheus, Loki, Grafana
- Uses `../../Dockerfile.obs` for builds
- Uses `../../Dockerfile.source` for builds
- Perfect for observability development
## 🚀 Usage Examples

8
.docker/compose/docker-compose.observability.yaml
View File

@@ -73,7 +73,7 @@ services:
node1:
build:
context: ../..
dockerfile: Dockerfile.obs
dockerfile: Dockerfile.source
container_name: node1
environment:
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
@@ -90,7 +90,7 @@ services:
node2:
build:
context: ../..
dockerfile: Dockerfile.obs
dockerfile: Dockerfile.source
container_name: node2
environment:
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
@@ -107,7 +107,7 @@ services:
node3:
build:
context: ../..
dockerfile: Dockerfile.obs
dockerfile: Dockerfile.source
container_name: node3
environment:
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
@@ -124,7 +124,7 @@ services:
node4:
build:
context: ../..
dockerfile: Dockerfile.obs
dockerfile: Dockerfile.source
container_name: node4
environment:
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}

96
.docker/ubuntu/Dockerfile.dev
View File

@@ -1,96 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Ubuntu-based development environment
# Provides full development toolchain for building RustFS from source
FROM ubuntu:22.04
ENV LANG=C.UTF-8
ENV DEBIAN_FRONTEND=noninteractive
# Use faster mirrors for better build performance
RUN sed -i s@http://.*archive.ubuntu.com@http://repo.huaweicloud.com@g /etc/apt/sources.list
# Install development dependencies
RUN apt-get clean && apt-get update && apt-get install -y \
wget \
git \
curl \
unzip \
gcc \
pkg-config \
libssl-dev \
lld \
libdbus-1-dev \
libwayland-dev \
libwebkit2gtk-4.1-dev \
libxdo-dev \
ca-certificates \
bash \
vim \
nano \
htop \
tree \
&& rm -rf /var/lib/apt/lists/*
# Install protoc
RUN wget https://github.com/protocolbuffers/protobuf/releases/download/v31.1/protoc-31.1-linux-x86_64.zip \
&& unzip protoc-31.1-linux-x86_64.zip -d protoc3 \
&& mv protoc3/bin/* /usr/local/bin/ && chmod +x /usr/local/bin/protoc \
&& mv protoc3/include/* /usr/local/include/ && rm -rf protoc-31.1-linux-x86_64.zip protoc3
# Install flatc
RUN wget https://github.com/google/flatbuffers/releases/download/v25.2.10/Linux.flatc.binary.g++-13.zip \
&& unzip Linux.flatc.binary.g++-13.zip \
&& mv flatc /usr/local/bin/ && chmod +x /usr/local/bin/flatc && rm -rf Linux.flatc.binary.g++-13.zip
# Install rust for development
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"
# Install additional Rust tools for development
RUN /root/.cargo/bin/cargo install \
cargo-watch \
cargo-nextest \
cargo-audit \
cargo-outdated
# Copy cargo config for Chinese users
COPY .docker/cargo.config.toml /root/.cargo/config.toml
# Create development user
RUN groupadd -g 1000 rustfs && \
useradd -d /app -g rustfs -u 1000 -s /bin/bash rustfs
WORKDIR /app
# Create data directories for testing
RUN mkdir -p /data && chown -R rustfs:rustfs /data /app
# Environment variables for development
ENV RUSTFS_ACCESS_KEY=devadmin \
RUSTFS_SECRET_KEY=devadmin \
RUSTFS_ADDRESS=":9000" \
RUSTFS_CONSOLE_ENABLE=true \
RUSTFS_VOLUMES=/data \
RUST_LOG=debug \
RUST_BACKTRACE=1
EXPOSE 9000
# Development mode: keep container alive for interactive development
CMD echo "RustFS Development Environment" && \
echo "Source code should be mounted at /app" && \
echo "Use 'cargo build' to build, 'cargo run' to run" && \
exec bash -c "while true; do sleep 1; done"

154
.docker/ubuntu/Dockerfile.prebuild
View File

@@ -1,154 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Multi-purpose Ubuntu-based Dockerfile
# Can be used as development environment or ubuntu runtime variant
FROM ubuntu:22.04
# Build arguments for dynamic artifact download (when used as runtime)
ARG VERSION=""
ARG BUILD_TYPE="release"
ARG TARGETARCH
ENV LANG=C.UTF-8
ENV DEBIAN_FRONTEND=noninteractive
# Use faster mirrors for better build performance
RUN sed -i s@http://.*archive.ubuntu.com@http://repo.huaweicloud.com@g /etc/apt/sources.list
# Install dependencies
RUN apt-get clean && apt-get update && apt-get install -y \
wget \
git \
curl \
unzip \
gcc \
pkg-config \
libssl-dev \
lld \
libdbus-1-dev \
libwayland-dev \
libwebkit2gtk-4.1-dev \
libxdo-dev \
ca-certificates \
bash \
&& rm -rf /var/lib/apt/lists/*
# Install protoc
RUN wget https://github.com/protocolbuffers/protobuf/releases/download/v31.1/protoc-31.1-linux-x86_64.zip \
&& unzip protoc-31.1-linux-x86_64.zip -d protoc3 \
&& mv protoc3/bin/* /usr/local/bin/ && chmod +x /usr/local/bin/protoc \
&& mv protoc3/include/* /usr/local/include/ && rm -rf protoc-31.1-linux-x86_64.zip protoc3
# Install flatc
RUN wget https://github.com/google/flatbuffers/releases/download/v25.2.10/Linux.flatc.binary.g++-13.zip \
&& unzip Linux.flatc.binary.g++-13.zip \
&& mv flatc /usr/local/bin/ && chmod +x /usr/local/bin/flatc && rm -rf Linux.flatc.binary.g++-13.zip
# Install rust (for development use)
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"
# Copy cargo config for Chinese users
COPY .docker/cargo.config.toml /root/.cargo/config.toml
# Download pre-built binary (VERSION is required)
RUN if [ -z "$VERSION" ]; then \
echo "❌ ERROR: VERSION build argument is required"; \
echo "Please provide VERSION (e.g., main-latest, latest, v1.0.0, dev-abc123)"; \
exit 1; \
fi; \
\
# Map TARGETARCH to our naming convention
case "${TARGETARCH}" in \
amd64) ARCH="x86_64" ;; \
arm64) ARCH="aarch64" ;; \
*) echo "❌ Unsupported architecture: ${TARGETARCH}" && exit 1 ;; \
esac; \
\
# Handle VERSION with different patterns
if [[ "$VERSION" == "main-latest" ]]; then \
DOWNLOAD_PATH="artifacts/rustfs/dev"; \
FILENAME="rustfs-linux-${ARCH}-main-latest.zip"; \
elif [[ "$VERSION" == "latest" ]]; then \
DOWNLOAD_PATH="artifacts/rustfs/release"; \
FILENAME="rustfs-linux-${ARCH}-latest.zip"; \
elif [[ "$VERSION" == dev-* ]]; then \
DOWNLOAD_PATH="artifacts/rustfs/dev"; \
FILENAME="rustfs-linux-${ARCH}-dev-${VERSION#dev-}.zip"; \
else \
DOWNLOAD_PATH="artifacts/rustfs/release"; \
FILENAME="rustfs-linux-${ARCH}-v${VERSION}.zip"; \
fi; \
\
# Download the binary
DOWNLOAD_URL="https://dl.rustfs.com/${DOWNLOAD_PATH}/${FILENAME}"; \
echo "🔽 Downloading RustFS binary from: ${DOWNLOAD_URL}"; \
\
# Download with clear error handling
if ! curl -fsSL --connect-timeout 30 --max-time 120 -o /tmp/rustfs.zip "${DOWNLOAD_URL}"; then \
echo "❌ Failed to download binary from: ${DOWNLOAD_URL}"; \
echo "💡 Please ensure the binary exists or trigger a build first"; \
echo "💡 Available options:"; \
echo " - For main-latest: Push to main branch or run build workflow"; \
echo " - For latest: Create a release tag"; \
echo " - For dev-xxx: Run build workflow with specific commit"; \
exit 1; \
fi; \
\
# Extract binary
if ! unzip -o /tmp/rustfs.zip -d /tmp >/dev/null 2>&1; then \
echo "❌ Failed to extract downloaded binary"; \
exit 1; \
fi; \
\
# Install binary
mv /tmp/rustfs /usr/local/bin/rustfs; \
chmod +x /usr/local/bin/rustfs; \
rm -rf /tmp/*; \
\
echo "✅ Successfully downloaded and installed RustFS binary (${VERSION})"
# Create rustfs user for security
RUN groupadd -g 1000 rustfs && \
useradd -d /app -g rustfs -u 1000 -s /bin/bash rustfs
WORKDIR /app
# Create data directories
RUN mkdir -p /data && chown -R rustfs:rustfs /data /app
# Environment variables
ENV RUSTFS_ACCESS_KEY=rustfsadmin \
RUSTFS_SECRET_KEY=rustfsadmin \
RUSTFS_ADDRESS=":9000" \
RUSTFS_CONSOLE_ENABLE=true \
RUSTFS_VOLUMES=/data \
RUST_LOG=warn
EXPOSE 9000
# Health check (only if rustfs binary exists)
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD if [ -f /usr/local/bin/rustfs ]; then wget --no-verbose --tries=1 --spider http://localhost:9000/health || exit 1; else exit 0; fi
# Default command: if rustfs binary exists, run it; otherwise, keep container alive for development
CMD if [ -f /usr/local/bin/rustfs ]; then \
echo "Starting RustFS server..."; \
exec /usr/local/bin/rustfs; \
else \
echo "Running in development mode..."; \
echo "RustFS source code should be mounted at /app"; \
exec bash -c "while true; do sleep 1; done"; \
fi

56
.github/workflows/build.yml vendored
View File

@@ -146,8 +146,6 @@ jobs:
if: needs.build-check.outputs.should_build == 'true'
runs-on: ${{ matrix.os }}
timeout-minutes: 60
env:
RUSTFLAGS: ${{ matrix.cross == 'false' && '-C target-cpu=native' || '' }}
strategy:
fail-fast: false
matrix:
@@ -155,29 +153,23 @@ jobs:
# Linux builds
- os: ubuntu-latest
target: x86_64-unknown-linux-musl
cross: false
platform: linux
- os: ubuntu-latest
target: aarch64-unknown-linux-musl
cross: true
platform: linux
# macOS builds
- os: macos-latest
target: aarch64-apple-darwin
cross: false
platform: macos
- os: macos-latest
target: x86_64-apple-darwin
cross: false
platform: macos
# # Windows builds (temporarily disabled)
# - os: windows-latest
# target: x86_64-pc-windows-msvc
# cross: false
# platform: windows
# - os: windows-latest
# target: aarch64-pc-windows-msvc
# cross: true
# platform: windows
steps:
- name: Checkout repository
@@ -193,50 +185,11 @@ jobs:
cache-shared-key: build-${{ matrix.target }}-${{ hashFiles('**/Cargo.lock') }}
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/') }}
install-cross-tools: ${{ matrix.cross }}
- name: Download static console assets
- name: Build RustFS using build-rustfs.sh
run: |
mkdir -p ./rustfs/static
if [[ "${{ matrix.platform }}" == "windows" ]]; then
curl.exe -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" -o console.zip --retry 3 --retry-delay 5 --max-time 300
if [[ $? -eq 0 ]]; then
unzip -o console.zip -d ./rustfs/static
rm console.zip
else
echo "Warning: Failed to download console assets, continuing without them"
echo "// Static assets not available" > ./rustfs/static/empty.txt
fi
else
chmod +w ./rustfs/static/LICENSE || true
curl -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" \
-o console.zip --retry 3 --retry-delay 5 --max-time 300
if [[ $? -eq 0 ]]; then
unzip -o console.zip -d ./rustfs/static
rm console.zip
else
echo "Warning: Failed to download console assets, continuing without them"
echo "// Static assets not available" > ./rustfs/static/empty.txt
fi
fi
- name: Build RustFS
run: |
# Force rebuild by touching build.rs
touch rustfs/build.rs
if [[ "${{ matrix.cross }}" == "true" ]]; then
if [[ "${{ matrix.platform }}" == "windows" ]]; then
# Use cross for Windows ARM64
cargo install cross --git https://github.com/cross-rs/cross
cross build --release --target ${{ matrix.target }} -p rustfs --bins
else
# Use zigbuild for Linux ARM64
cargo zigbuild --release --target ${{ matrix.target }} -p rustfs --bins
fi
else
cargo build --release --target ${{ matrix.target }} -p rustfs --bins
fi
# Use unified build script for consistent builds
./build-rustfs.sh --platform ${{ matrix.target }}
- name: Create release package
id: package
@@ -283,7 +236,8 @@ jobs:
fi
fi
cd target/${{ matrix.target }}/release
# build-rustfs.sh outputs to target/release/${platform}/rustfs
cd target/release/${{ matrix.target }}
zip "../../../${PACKAGE_NAME}.zip" rustfs
cd ../../..

14
.github/workflows/docker.yml vendored
View File

@@ -223,17 +223,11 @@ jobs:
- name: production
dockerfile: Dockerfile
platforms: linux/amd64,linux/arm64
- name: alpine
dockerfile: .docker/alpine/Dockerfile.prebuild
- name: source
dockerfile: Dockerfile.source
platforms: linux/amd64,linux/arm64
- name: alpine-source
dockerfile: .docker/alpine/Dockerfile.source
platforms: linux/amd64,linux/arm64
- name: ubuntu
dockerfile: .docker/ubuntu/Dockerfile.prebuild
platforms: linux/amd64,linux/arm64
- name: ubuntu-source
dockerfile: .docker/ubuntu/Dockerfile.source
- name: dev
dockerfile: Dockerfile.source
platforms: linux/amd64,linux/arm64
steps:
- name: Checkout repository

17
.github/workflows/performance.yml vendored
View File

@@ -18,10 +18,10 @@ on:
push:
branches: [main]
paths:
- '**/*.rs'
- '**/Cargo.toml'
- '**/Cargo.lock'
- '.github/workflows/performance.yml'
- "**/*.rs"
- "**/Cargo.toml"
- "**/Cargo.lock"
- ".github/workflows/performance.yml"
workflow_dispatch:
inputs:
profile_duration:
@@ -73,12 +73,11 @@ jobs:
echo "RUSTFS_VOLUMES=./target/volume/test{0...4}" >> $GITHUB_ENV
echo "RUST_LOG=rustfs=info,ecstore=info,s3s=info,iam=info,rustfs-obs=info" >> $GITHUB_ENV
- name: Download static files
- name: Verify console static assets
run: |
curl -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" \
-o tempfile.zip --retry 3 --retry-delay 5
unzip -o tempfile.zip -d ./rustfs/static
rm tempfile.zip
# Console static assets are already embedded in the repository
echo "Console static assets size: $(du -sh rustfs/static/)"
echo "Console static assets are embedded via rust-embed, no external download needed"
- name: Build with profiling optimizations
run: |

236
Dockerfile
View File

@@ -1,129 +1,95 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Multi-stage build for RustFS production image
FROM alpine:latest AS build
# Multi-stage Alpine build for minimal runtime image
FROM rust:1.85-alpine AS builder
# Build arguments for dynamic artifact download
ARG VERSION=""
ARG BUILD_TYPE="release"
# Build arguments
ARG TARGETARCH
ARG RELEASE=latest
ARG CHANNEL=release
# Install build dependencies
RUN apk add --no-cache \
musl-dev \
pkgconfig \
openssl-dev \
openssl-libs-static \
curl \
unzip \
bash \
wget \
ca-certificates
# Install protoc
RUN wget https://github.com/protocolbuffers/protobuf/releases/download/v31.1/protoc-31.1-linux-x86_64.zip \
&& unzip protoc-31.1-linux-x86_64.zip -d protoc3 \
&& mv protoc3/bin/* /usr/local/bin/ && chmod +x /usr/local/bin/protoc \
&& mv protoc3/include/* /usr/local/include/ && rm -rf protoc-31.1-linux-x86_64.zip protoc3
# Install flatc
RUN wget https://github.com/google/flatbuffers/releases/download/v25.2.10/Linux.flatc.binary.g++-13.zip \
&& unzip Linux.flatc.binary.g++-13.zip \
&& mv flatc /usr/local/bin/ && chmod +x /usr/local/bin/flatc \
&& rm -rf Linux.flatc.binary.g++-13.zip
# Download pre-built binary (VERSION is required)
RUN if [ -z "$VERSION" ]; then \
echo "❌ ERROR: VERSION build argument is required"; \
echo "Please provide VERSION (e.g., main-latest, latest, v1.0.0, dev-abc123)"; \
exit 1; \
fi; \
\
# Map TARGETARCH to our naming convention
case "${TARGETARCH}" in \
amd64) ARCH="x86_64" ;; \
arm64) ARCH="aarch64" ;; \
*) echo "❌ Unsupported architecture: ${TARGETARCH}" && exit 1 ;; \
esac; \
\
# Handle VERSION with different patterns
if [[ "$VERSION" == "main-latest" ]]; then \
DOWNLOAD_PATH="artifacts/rustfs/dev"; \
FILENAME="rustfs-linux-${ARCH}-main-latest.zip"; \
elif [[ "$VERSION" == "latest" ]]; then \
DOWNLOAD_PATH="artifacts/rustfs/release"; \
FILENAME="rustfs-linux-${ARCH}-latest.zip"; \
elif [[ "$VERSION" == dev-* ]]; then \
DOWNLOAD_PATH="artifacts/rustfs/dev"; \
FILENAME="rustfs-linux-${ARCH}-dev-${VERSION#dev-}.zip"; \
else \
DOWNLOAD_PATH="artifacts/rustfs/release"; \
FILENAME="rustfs-linux-${ARCH}-v${VERSION}.zip"; \
fi; \
\
# Download the binary
DOWNLOAD_URL="https://dl.rustfs.com/${DOWNLOAD_PATH}/${FILENAME}"; \
echo "🔽 Downloading RustFS binary from: ${DOWNLOAD_URL}"; \
\
# Download with clear error handling
if ! curl -fsSL --connect-timeout 30 --max-time 120 -o /tmp/rustfs.zip "${DOWNLOAD_URL}"; then \
echo "❌ Failed to download binary from: ${DOWNLOAD_URL}"; \
echo "💡 Please ensure the binary exists or trigger a build first"; \
echo "💡 Available options:"; \
echo " - For main-latest: Push to main branch or run build workflow"; \
echo " - For latest: Create a release tag"; \
echo " - For dev-xxx: Run build workflow with specific commit"; \
exit 1; \
fi; \
\
# Extract binary
if ! unzip -o /tmp/rustfs.zip -d /tmp >/dev/null 2>&1; then \
echo "❌ Failed to extract downloaded binary"; \
exit 1; \
fi; \
\
# Install binary
mv /tmp/rustfs /usr/local/bin/rustfs; \
chmod +x /usr/local/bin/rustfs; \
rm -rf /tmp/*; \
\
echo "✅ Successfully downloaded and installed RustFS binary (${VERSION})"
# Final Alpine runtime image
FROM alpine:3.18
# Install dependencies for downloading and verifying binaries
RUN apk add --no-cache \
ca-certificates \
curl \
bash \
wget \
unzip \
jq
# Create build directory
WORKDIR /build
# Map TARGETARCH to architecture format used in builds
RUN case "${TARGETARCH}" in \
"amd64") ARCH="x86_64" ;; \
"arm64") ARCH="aarch64" ;; \
*) echo "Unsupported architecture: ${TARGETARCH}" && exit 1 ;; \
esac && \
echo "ARCH=${ARCH}" > /build/arch.env
# Download rustfs binary from dl.rustfs.com
RUN . /build/arch.env && \
BASE_URL="https://dl.rustfs.com/artifacts/rustfs" && \
PLATFORM="linux" && \
if [ "${RELEASE}" = "latest" ]; then \
# Download latest version from specified channel \
if [ "${CHANNEL}" = "dev" ]; then \
PACKAGE_NAME="rustfs-${PLATFORM}-${ARCH}-dev-latest.zip"; \
DOWNLOAD_URL="${BASE_URL}/dev/${PACKAGE_NAME}"; \
echo "📥 Downloading latest dev build: ${PACKAGE_NAME}"; \
else \
PACKAGE_NAME="rustfs-${PLATFORM}-${ARCH}-latest.zip"; \
DOWNLOAD_URL="${BASE_URL}/release/${PACKAGE_NAME}"; \
echo "📥 Downloading latest release build: ${PACKAGE_NAME}"; \
fi; \
else \
# Download specific version (always from release channel) \
PACKAGE_NAME="rustfs-${PLATFORM}-${ARCH}-v${RELEASE}.zip"; \
DOWNLOAD_URL="${BASE_URL}/release/${PACKAGE_NAME}"; \
echo "📥 Downloading specific version: ${PACKAGE_NAME}"; \
fi && \
echo "🔗 Download URL: ${DOWNLOAD_URL}" && \
curl -f -L "${DOWNLOAD_URL}" -o /build/rustfs.zip && \
if [ ! -f /build/rustfs.zip ] || [ ! -s /build/rustfs.zip ]; then \
echo "❌ Failed to download binary package"; \
echo "💡 Make sure the package ${PACKAGE_NAME} exists"; \
echo "🔗 Check: ${DOWNLOAD_URL}"; \
exit 1; \
fi && \
unzip /build/rustfs.zip -d /build && \
chmod +x /build/rustfs && \
rm /build/rustfs.zip && \
echo "✅ Successfully downloaded and extracted rustfs binary"
# Runtime stage
FROM alpine:latest
# Set build arguments and labels
ARG RELEASE=latest
ARG CHANNEL=release
ARG BUILD_DATE
ARG VCS_REF
LABEL name="RustFS" \
vendor="RustFS Team" \
maintainer="RustFS Team <dev@rustfs.com>" \
version="${RELEASE}" \
release="${RELEASE}" \
channel="${CHANNEL}" \
build-date="${BUILD_DATE}" \
vcs-ref="${VCS_REF}" \
summary="RustFS is a high-performance distributed object storage system written in Rust, compatible with S3 API." \
description="RustFS is a high-performance distributed object storage software built using Rust. It supports erasure coding storage, multi-tenant management, observability, and other enterprise-level features." \
url="https://rustfs.com" \
license="Apache-2.0"
# Install runtime dependencies
RUN apk add --no-cache \
ca-certificates \
curl \
tzdata \
bash
# Create rustfs user for security
RUN addgroup -g 1000 rustfs && \
adduser -D -u 1000 -G rustfs rustfs
WORKDIR /app
# Copy binary from builder
COPY --from=builder /usr/local/bin/rustfs /app/rustfs
RUN chmod +x /app/rustfs && chown rustfs:rustfs /app/rustfs
# Create data directories
RUN mkdir -p /data && chown -R rustfs:rustfs /data /app
# Switch to non-root user
USER rustfs
bash \
&& addgroup -g 1000 rustfs \
&& adduser -u 1000 -G rustfs -s /bin/sh -D rustfs
# Environment variables
ENV RUSTFS_ACCESS_KEY=rustfsadmin \
@@ -133,10 +99,34 @@ ENV RUSTFS_ACCESS_KEY=rustfsadmin \
RUSTFS_VOLUMES=/data \
RUST_LOG=warn
# Set permissions for /usr/bin (similar to MinIO's approach)
RUN chmod -R 755 /usr/bin
# Copy CA certificates and binaries from build stage
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /build/rustfs /usr/bin/
# Set executable permissions
RUN chmod +x /usr/bin/rustfs
# Create data directory
RUN mkdir -p /data /config && chown -R rustfs:rustfs /data /config
# Switch to non-root user
USER rustfs
# Set working directory
WORKDIR /data
# Expose port
EXPOSE 9000
# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:9000/health || exit 1
HEALTHCHECK --interval=30s --timeout=10s --retries=3 --start-period=40s \
CMD curl -f http://localhost:9000/health || exit 1
CMD ["/app/rustfs"]
# Volume for data
VOLUME ["/data"]
# Set entrypoint
ENTRYPOINT ["/usr/bin/rustfs"]

61
Dockerfile.obs
View File

@@ -1,61 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Dockerfile for RustFS with observability features
FROM ubuntu:22.04
# Avoid interactive prompts during build
ENV DEBIAN_FRONTEND=noninteractive
# Install runtime dependencies
RUN apt-get update && apt-get install -y \
ca-certificates \
wget \
curl \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
# Create rustfs user for security
RUN groupadd -g 1000 rustfs && \
useradd -d /app -g rustfs -u 1000 -s /bin/bash rustfs
# Create data directories matching RUSTFS_VOLUMES pattern
RUN mkdir -p /data/rustfs{0,1,2,3} && \
chown -R rustfs:rustfs /data /app
# Copy RustFS binary (expects it to be built with observability features)
# Note: This assumes the binary is built locally with observability features enabled
COPY ./target/x86_64-unknown-linux-gnu/release/rustfs /app/rustfs
RUN chmod +x /app/rustfs && chown rustfs:rustfs /app/rustfs
# Switch to non-root user
USER rustfs
# Environment variables for observability
ENV RUSTFS_ACCESS_KEY=rustfsadmin \
RUSTFS_SECRET_KEY=rustfsadmin \
RUSTFS_ADDRESS=":9000" \
RUSTFS_CONSOLE_ENABLE=true \
RUSTFS_VOLUMES=/data/rustfs0,/data/rustfs1,/data/rustfs2,/data/rustfs3 \
RUSTFS_OBS_ENDPOINT=http://otel-collector:4317 \
RUST_LOG=info
EXPOSE 9000
# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:9000/health || exit 1
CMD ["/app/rustfs"]

5
.docker/ubuntu/Dockerfile.source → Dockerfile.source
View File

@@ -63,7 +63,7 @@ ENV CXX_aarch64_unknown_linux_gnu=aarch64-linux-gnu-g++
WORKDIR /usr/src/rustfs
# Copy cargo configuration for optimized builds
COPY .docker/cargo.config.toml ./.cargo/config.toml
COPY cargo.config.toml ./.cargo/config.toml
# Copy Cargo files for dependency caching
COPY Cargo.toml Cargo.lock ./
@@ -151,5 +151,8 @@ ENV RUSTFS_ACCESS_KEY=rustfsadmin \
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:9000/health || exit 1
# Volume for data
VOLUME ["/data"]
# Set default command
CMD ["/app/rustfs"]

170
Makefile
View File

@@ -5,7 +5,9 @@
DOCKER_CLI ?= docker
IMAGE_NAME ?= rustfs:v1.0.0
CONTAINER_NAME ?= rustfs-dev
DOCKERFILE_PATH = $(shell pwd)/.docker
# Docker build configurations
DOCKERFILE_PRODUCTION = Dockerfile
DOCKERFILE_SOURCE = Dockerfile.source
# Code quality and formatting targets
.PHONY: fmt
@@ -46,7 +48,7 @@ setup-hooks:
.PHONY: init-devenv
init-devenv:
$(DOCKER_CLI) build -t $(IMAGE_NAME) -f $(DOCKERFILE_PATH)/Dockerfile.devenv .
$(DOCKER_CLI) build -t $(IMAGE_NAME) -f Dockerfile.source .
$(DOCKER_CLI) stop $(CONTAINER_NAME)
$(DOCKER_CLI) rm $(CONTAINER_NAME)
$(DOCKER_CLI) run -d --name $(CONTAINER_NAME) -p 9010:9010 -p 9000:9000 -v $(shell pwd):/root/s3-rustfs -it $(IMAGE_NAME)
@@ -67,86 +69,89 @@ e2e-server:
probe-e2e:
sh $(shell pwd)/scripts/probe.sh
# make BUILD_OS=ubuntu22.04 build
# in target/ubuntu22.04/release/rustfs
# make BUILD_OS=rockylinux9.3 build
# in target/rockylinux9.3/release/rustfs
BUILD_OS ?= rockylinux9.3
# Native build using build-rustfs.sh script
.PHONY: build
build: ROCKYLINUX_BUILD_IMAGE_NAME = rustfs-$(BUILD_OS):v1
build: ROCKYLINUX_BUILD_CONTAINER_NAME = rustfs-$(BUILD_OS)-build
build: BUILD_CMD = /root/.cargo/bin/cargo build --release --bin rustfs --target-dir /root/s3-rustfs/target/$(BUILD_OS)
build:
$(DOCKER_CLI) build -t $(ROCKYLINUX_BUILD_IMAGE_NAME) -f $(DOCKERFILE_PATH)/Dockerfile.$(BUILD_OS) .
$(DOCKER_CLI) run --rm --name $(ROCKYLINUX_BUILD_CONTAINER_NAME) -v $(shell pwd):/root/s3-rustfs -it $(ROCKYLINUX_BUILD_IMAGE_NAME) $(BUILD_CMD)
@echo "🔨 Building RustFS using build-rustfs.sh script..."
./build-rustfs.sh
.PHONY: build-dev
build-dev:
@echo "🔨 Building RustFS in development mode..."
./build-rustfs.sh --dev
# Docker-based build (alternative approach)
# Usage: make BUILD_OS=ubuntu22.04 build-docker
# Output: target/ubuntu22.04/release/rustfs
BUILD_OS ?= rockylinux9.3
.PHONY: build-docker
build-docker: SOURCE_BUILD_IMAGE_NAME = rustfs-$(BUILD_OS):v1
build-docker: SOURCE_BUILD_CONTAINER_NAME = rustfs-$(BUILD_OS)-build
build-docker: BUILD_CMD = /root/.cargo/bin/cargo build --release --bin rustfs --target-dir /root/s3-rustfs/target/$(BUILD_OS)
build-docker:
@echo "🐳 Building RustFS using Docker ($(BUILD_OS))..."
$(DOCKER_CLI) build -t $(SOURCE_BUILD_IMAGE_NAME) -f $(DOCKERFILE_SOURCE) .
$(DOCKER_CLI) run --rm --name $(SOURCE_BUILD_CONTAINER_NAME) -v $(shell pwd):/root/s3-rustfs -it $(SOURCE_BUILD_IMAGE_NAME) $(BUILD_CMD)
.PHONY: build-musl
build-musl:
@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
cargo build --target x86_64-unknown-linux-musl --bin rustfs -r
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-buildx' instead"
./build-rustfs.sh --platform x86_64-unknown-linux-musl
.PHONY: build-gnu
build-gnu:
@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
cargo build --target x86_64-unknown-linux-gnu --bin rustfs -r
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-buildx' instead"
./build-rustfs.sh --platform x86_64-unknown-linux-gnu
.PHONY: deploy-dev
deploy-dev: build-musl
@echo "🚀 Deploying to dev server: $${IP}"
./scripts/dev_deploy.sh $${IP}
# Multi-architecture Docker build targets
.PHONY: docker-build-multiarch
docker-build-multiarch:
@echo "🏗️ Building multi-architecture Docker images..."
./scripts/build-docker-multiarch.sh
# Multi-architecture Docker build targets (NEW: using docker-buildx.sh)
.PHONY: docker-buildx
docker-buildx:
@echo "🏗️ Building multi-architecture Docker images with buildx..."
./docker-buildx.sh
.PHONY: docker-build-multiarch-push
docker-build-multiarch-push:
@echo "🚀 Building and pushing multi-architecture Docker images..."
./scripts/build-docker-multiarch.sh --push
.PHONY: docker-buildx-push
docker-buildx-push:
@echo "🚀 Building and pushing multi-architecture Docker images with buildx..."
./docker-buildx.sh --push
.PHONY: docker-build-multiarch-version
docker-build-multiarch-version:
.PHONY: docker-buildx-version
docker-buildx-version:
@if [ -z "$(VERSION)" ]; then \
echo "❌ 错误: 请指定版本, 例如: make docker-build-multiarch-version VERSION=v1.0.0"; \
echo "❌ 错误: 请指定版本, 例如: make docker-buildx-version VERSION=v1.0.0"; \
exit 1; \
fi
@echo "🏗️ Building multi-architecture Docker images (version: $(VERSION))..."
./scripts/build-docker-multiarch.sh --version $(VERSION)
./docker-buildx.sh --release $(VERSION)
.PHONY: docker-push-multiarch-version
docker-push-multiarch-version:
.PHONY: docker-buildx-push-version
docker-buildx-push-version:
@if [ -z "$(VERSION)" ]; then \
echo "❌ 错误: 请指定版本, 例如: make docker-push-multiarch-version VERSION=v1.0.0"; \
echo "❌ 错误: 请指定版本, 例如: make docker-buildx-push-version VERSION=v1.0.0"; \
exit 1; \
fi
@echo "🚀 Building and pushing multi-architecture Docker images (version: $(VERSION))..."
./scripts/build-docker-multiarch.sh --version $(VERSION) --push
./docker-buildx.sh --release $(VERSION) --push
.PHONY: docker-build-ubuntu
docker-build-ubuntu:
@echo "🏗️ Building multi-architecture Ubuntu Docker images..."
./scripts/build-docker-multiarch.sh --type ubuntu
.PHONY: docker-build-rockylinux
docker-build-rockylinux:
@echo "🏗️ Building multi-architecture RockyLinux Docker images..."
./scripts/build-docker-multiarch.sh --type rockylinux
.PHONY: docker-build-devenv
docker-build-devenv:
@echo "🏗️ Building multi-architecture development environment Docker images..."
./scripts/build-docker-multiarch.sh --type devenv
.PHONY: docker-build-production
docker-build-production:
@echo "🏗️ Building production Docker image..."
$(DOCKER_CLI) build -f $(DOCKERFILE_PRODUCTION) -t rustfs:latest .
.PHONY: docker-build-all-types
docker-build-all-types:
@echo "🏗️ Building all multi-architecture Docker image types..."
./scripts/build-docker-multiarch.sh --type production
./scripts/build-docker-multiarch.sh --type ubuntu
./scripts/build-docker-multiarch.sh --type rockylinux
./scripts/build-docker-multiarch.sh --type devenv
.PHONY: docker-build-source
docker-build-source:
@echo "🏗️ Building source Docker image..."
$(DOCKER_CLI) build -f $(DOCKERFILE_SOURCE) -t rustfs:source .
.PHONY: docker-inspect-multiarch
docker-inspect-multiarch:
@@ -160,41 +165,64 @@ docker-inspect-multiarch:
.PHONY: build-cross-all
build-cross-all:
@echo "🔧 Building all target architectures..."
@if ! command -v cross &> /dev/null; then \
echo "📦 Installing cross..."; \
cargo install cross; \
fi
@echo "💡 On macOS/Windows, use 'make docker-buildx' for reliable multi-arch builds"
@echo "🔨 Generating protobuf code..."
cargo run --bin gproto || true
@echo "🔨 Building x86_64-unknown-linux-musl..."
cargo build --release --target x86_64-unknown-linux-musl --bin rustfs
./build-rustfs.sh --platform x86_64-unknown-linux-musl
@echo "🔨 Building aarch64-unknown-linux-gnu..."
cross build --release --target aarch64-unknown-linux-gnu --bin rustfs
./build-rustfs.sh --platform aarch64-unknown-linux-gnu
@echo "✅ All architectures built successfully!"
.PHONY: help-build
help-build:
@echo "🔨 RustFS 构建帮助："
@echo ""
@echo "🚀 本地构建 (推荐使用):"
@echo " make build # 构建 RustFS 二进制文件 (默认包含 console)"
@echo " make build-dev # 开发模式构建"
@echo " make build-musl # 构建 musl 版本"
@echo " make build-gnu # 构建 GNU 版本"
@echo ""
@echo "🐳 Docker 构建:"
@echo " make build-docker # 使用 Docker 容器构建"
@echo " make build-docker BUILD_OS=ubuntu22.04 # 指定构建系统"
@echo ""
@echo "🏗️ 跨架构构建:"
@echo " make build-cross-all # 构建所有架构的二进制文件"
@echo ""
@echo "🔧 直接使用 build-rustfs.sh 脚本:"
@echo " ./build-rustfs.sh --help # 查看脚本帮助"
@echo " ./build-rustfs.sh --no-console # 构建时跳过 console 资源"
@echo " ./build-rustfs.sh --force-console-update # 强制更新 console 资源"
@echo " ./build-rustfs.sh --dev # 开发模式构建"
@echo " ./build-rustfs.sh --sign # 签名二进制文件"
@echo " ./build-rustfs.sh --platform x86_64-unknown-linux-musl # 指定目标平台"
@echo " ./build-rustfs.sh --skip-verification # 跳过二进制验证"
@echo ""
@echo "💡 build-rustfs.sh 脚本提供了更多选项、智能检测和二进制验证功能"
.PHONY: help-docker
help-docker:
@echo "🐳 Docker 多架构构建帮助："
@echo ""
@echo "基本构建:"
@echo " make docker-build-multiarch # 构建多架构镜像（不推送）"
@echo " make docker-build-multiarch-push # 构建并推送多架构镜像"
@echo "🚀 推荐使用 (新的 docker-buildx 方式):"
@echo " make docker-buildx # 构建多架构镜像（不推送）"
@echo " make docker-buildx-push # 构建并推送多架构镜像"
@echo " make docker-buildx-version VERSION=v1.0.0 # 构建指定版本"
@echo " make docker-buildx-push-version VERSION=v1.0.0 # 构建并推送指定版本"
@echo ""
@echo "版本构建:"
@echo " make docker-build-multiarch-version VERSION=v1.0.0 # 构建指定版本"
@echo " make docker-push-multiarch-version VERSION=v1.0.0 # 构建并推送指定版本"
@echo "🏗️ 单架构构建:"
@echo " make docker-build-production # 构建生产环境镜像"
@echo " make docker-build-source # 构建源码构建镜像"
@echo ""
@echo "镜像类型:"
@echo " make docker-build-ubuntu # 构建 Ubuntu 镜像"
@echo " make docker-build-rockylinux # 构建 RockyLinux 镜像"
@echo " make docker-build-devenv # 构建开发环境镜像"
@echo " make docker-build-all-types # 构建所有类型镜像"
@echo ""
@echo "辅助工具:"
@echo "🔧 辅助工具:"
@echo " make build-cross-all # 构建所有架构的二进制文件"
@echo " make docker-inspect-multiarch IMAGE=xxx # 检查镜像的架构支持"
@echo ""
@echo "环境变量 (在推送时需要设置):"
@echo "📋 环境变量 (在推送时需要设置):"
@echo " DOCKERHUB_USERNAME Docker Hub 用户名"
@echo " DOCKERHUB_TOKEN Docker Hub 访问令牌"
@echo " GITHUB_TOKEN GitHub 访问令牌"
@echo ""
@echo "💡 更多详情请参考项目根目录的 docker-buildx.sh 脚本"

39
README.md
View File

@@ -91,11 +91,42 @@ To get started with RustFS, follow these steps:
docker run -d -p 9000:9000 -v /data:/data rustfs/rustfs:v1.0.0
```
> 💡 **Docker Build Options**: RustFS supports flexible Docker image versions. See our [Docker Build Guide](docs/DOCKER_BUILD_GUIDE.md) for advanced usage and manual build instructions.
3. **Build from Source (Option 3) - Advanced Users**
3. **Access the Console**: Open your web browser and navigate to `http://localhost:9000` to access the RustFS console, default username and password is `rustfsadmin` .
4. **Create a Bucket**: Use the console to create a new bucket for your objects.
5. **Upload Objects**: You can upload files directly through the console or use S3-compatible APIs to interact with your RustFS instance.
For developers who want to build RustFS Docker images from source with multi-architecture support:
```bash
# Build multi-architecture images locally
./docker-buildx.sh --build-arg RELEASE=latest
# Build and push to registry
./docker-buildx.sh --push
# Build specific version
./docker-buildx.sh --release v1.0.0 --push
# Build for custom registry
./docker-buildx.sh --registry your-registry.com --namespace yourname --push
```
The `docker-buildx.sh` script supports:
- **Multi-architecture builds**: `linux/amd64`, `linux/arm64`
- **Automatic version detection**: Uses git tags or commit hashes
- **Registry flexibility**: Supports Docker Hub, GitHub Container Registry, etc.
- **Build optimization**: Includes caching and parallel builds
You can also use Make targets for convenience:
```bash
make docker-buildx # Build locally
make docker-buildx-push # Build and push
make docker-buildx-version VERSION=v1.0.0 # Build specific version
make help-docker # Show all Docker-related commands
```
4. **Access the Console**: Open your web browser and navigate to `http://localhost:9000` to access the RustFS console, default username and password is `rustfsadmin` .
5. **Create a Bucket**: Use the console to create a new bucket for your objects.
6. **Upload Objects**: You can upload files directly through the console or use S3-compatible APIs to interact with your RustFS instance.
## Documentation

5
README_ZH.md
View File

@@ -62,7 +62,7 @@ RustFS 是一个使用 Rust（全球最受欢迎的编程语言之一）构建
要开始使用 RustFS，请按照以下步骤操作：
1. **一键脚本快速启动 (方案一)**
1. **一键脚本快速启动 (方案一)**
```bash
curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh
@@ -74,7 +74,6 @@ RustFS 是一个使用 Rust（全球最受欢迎的编程语言之一）构建
docker run -d -p 9000:9000 -v /data:/data rustfs/rustfs
```
3. **访问控制台**：打开 Web 浏览器并导航到 `http://localhost:9000` 以访问 RustFS 控制台，默认的用户名和密码是 `rustfsadmin` 。
4. **创建存储桶**：使用控制台为您的对象创建新的存储桶。
5. **上传对象**：您可以直接通过控制台上传文件，或使用 S3 兼容的 API 与您的 RustFS 实例交互。
@@ -110,7 +109,7 @@ RustFS 是一个使用 Rust（全球最受欢迎的编程语言之一）构建
RustFS 是一个社区驱动的项目，我们感谢所有的贡献。查看[贡献者](https://github.com/rustfs/rustfs/graphs/contributors)页面，了解帮助 RustFS 变得更好的杰出人员。
<a href="https://github.com/rustfs/rustfs/graphs/contributors">
<img src="https://contrib.rocks/image?repo=rustfs/rustfs" />
<img src="https://opencollective.com/rustfs/contributors.svg?width=890&limit=500&button=false" />
</a >
## 许可证

488
build-rustfs.sh Executable file
View File

@@ -0,0 +1,488 @@
#!/bin/bash
# RustFS Binary Build Script
# This script compiles RustFS binaries for different platforms and architectures
set -e
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Auto-detect current platform
detect_platform() {
local arch=$(uname -m)
local os=$(uname -s | tr '[:upper:]' '[:lower:]')
case "$os" in
"linux")
case "$arch" in
"x86_64")
echo "x86_64-unknown-linux-musl"
;;
"aarch64"|"arm64")
echo "aarch64-unknown-linux-musl"
;;
"armv7l")
echo "armv7-unknown-linux-musleabihf"
;;
*)
echo "unknown-platform"
;;
esac
;;
"darwin")
case "$arch" in
"x86_64")
echo "x86_64-apple-darwin"
;;
"arm64"|"aarch64")
echo "aarch64-apple-darwin"
;;
*)
echo "unknown-platform"
;;
esac
;;
*)
echo "unknown-platform"
;;
esac
}
# Default values
OUTPUT_DIR="target/release"
PLATFORM=$(detect_platform) # Auto-detect current platform
BINARY_NAME="rustfs"
BUILD_TYPE="release"
SIGN=false
WITH_CONSOLE=true
FORCE_CONSOLE_UPDATE=false
CONSOLE_VERSION="latest"
SKIP_VERIFICATION=false
CUSTOM_PLATFORM=""
# Print usage
usage() {
echo "Usage: $0 [OPTIONS]"
echo ""
echo "Description:"
echo " Build RustFS binary for the current platform. Designed for CI/CD pipelines"
echo " where different runners build platform-specific binaries natively."
echo " Includes automatic verification to ensure the built binary is functional."
echo ""
echo "Options:"
echo " -o, --output-dir DIR Output directory (default: target/release)"
echo " -b, --binary-name NAME Binary name (default: rustfs)"
echo " -p, --platform TARGET Target platform (default: auto-detect)"
echo " --dev Build in dev mode"
echo " --sign Sign binaries after build"
echo " --with-console Download console static assets (default)"
echo " --no-console Skip console static assets"
echo " --force-console-update Force update console assets even if they exist"
echo " --console-version VERSION Console version to download (default: latest)"
echo " --skip-verification Skip binary verification after build"
echo " -h, --help Show this help message"
echo ""
echo "Examples:"
echo " $0 # Build for current platform (includes console assets)"
echo " $0 --dev # Development build"
echo " $0 --sign # Build and sign binary (release CI)"
echo " $0 --no-console # Build without console static assets"
echo " $0 --force-console-update # Force update console assets"
echo " $0 --platform x86_64-unknown-linux-musl # Build for specific platform"
echo " $0 --skip-verification # Skip binary verification (for cross-compilation)"
echo ""
echo "Detected platform: $(detect_platform)"
echo "CI Usage: Run this script on each platform's runner to build native binaries"
}
# Print colored message
print_message() {
local color=$1
local message=$2
echo -e "${color}${message}${NC}"
}
# Get version from git
get_version() {
if git describe --abbrev=0 --tags >/dev/null 2>&1; then
git describe --abbrev=0 --tags
else
git rev-parse --short HEAD
fi
}
# Setup rust environment
setup_rust_environment() {
print_message $BLUE "🔧 Setting up Rust environment..."
# Install required target for current platform
print_message $YELLOW "Installing target: $PLATFORM"
rustup target add "$PLATFORM"
# Install required tools
if [ "$SIGN" = true ]; then
if ! command -v minisign &> /dev/null; then
print_message $YELLOW "Installing minisign for binary signing..."
cargo install minisign
fi
fi
}
# Download console static assets
download_console_assets() {
local static_dir="rustfs/static"
local console_exists=false
# Check if console assets already exist
if [ -d "$static_dir" ] && [ -f "$static_dir/index.html" ]; then
console_exists=true
local static_size=$(du -sh "$static_dir" 2>/dev/null | cut -f1 || echo "unknown")
print_message $YELLOW "Console static assets already exist ($static_size)"
fi
# Determine if we need to download
local should_download=false
if [ "$WITH_CONSOLE" = true ]; then
if [ "$console_exists" = false ]; then
print_message $BLUE "🎨 Console assets not found, downloading..."
should_download=true
elif [ "$FORCE_CONSOLE_UPDATE" = true ]; then
print_message $BLUE "🎨 Force updating console assets..."
should_download=true
else
print_message $GREEN "✅ Console assets already available, skipping download"
fi
else
if [ "$console_exists" = true ]; then
print_message $GREEN "✅ Using existing console assets"
else
print_message $YELLOW "⚠️ Console assets not found. Use --download-console to download them."
fi
fi
if [ "$should_download" = true ]; then
print_message $BLUE "📥 Downloading console static assets..."
# Create static directory
mkdir -p "$static_dir"
# Download from GitHub Releases (consistent with Docker build)
local download_url
if [ "$CONSOLE_VERSION" = "latest" ]; then
print_message $YELLOW "Getting latest console release info..."
# For now, use dl.rustfs.com as fallback until GitHub Releases includes console assets
download_url="https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip"
else
download_url="https://dl.rustfs.com/artifacts/console/rustfs-console-${CONSOLE_VERSION}.zip"
fi
print_message $YELLOW "Downloading from: $download_url"
# Download with retries
local temp_file="console-assets-temp.zip"
local download_success=false
for i in {1..3}; do
if curl -L "$download_url" -o "$temp_file" --retry 3 --retry-delay 5 --max-time 300; then
download_success=true
break
else
print_message $YELLOW "Download attempt $i failed, retrying..."
sleep 2
fi
done
if [ "$download_success" = true ]; then
# Verify the downloaded file
if [ -f "$temp_file" ] && [ -s "$temp_file" ]; then
print_message $BLUE "📦 Extracting console assets..."
# Extract to static directory
if unzip -o "$temp_file" -d "$static_dir"; then
rm "$temp_file"
local final_size=$(du -sh "$static_dir" 2>/dev/null | cut -f1 || echo "unknown")
print_message $GREEN "✅ Console assets downloaded successfully ($final_size)"
else
print_message $RED "❌ Failed to extract console assets"
rm -f "$temp_file"
return 1
fi
else
print_message $RED "❌ Downloaded file is empty or invalid"
rm -f "$temp_file"
return 1
fi
else
print_message $RED "❌ Failed to download console assets after 3 attempts"
print_message $YELLOW "💡 Console assets are optional. Build will continue without them."
rm -f "$temp_file"
fi
fi
}
# Verify binary functionality
verify_binary() {
local binary_path="$1"
# Check if binary exists
if [ ! -f "$binary_path" ]; then
print_message $RED "❌ Binary file not found: $binary_path"
return 1
fi
# Check if binary is executable
if [ ! -x "$binary_path" ]; then
print_message $RED "❌ Binary is not executable: $binary_path"
return 1
fi
# Check basic functionality - try to run help command
print_message $YELLOW " Testing --help command..."
if ! "$binary_path" --help >/dev/null 2>&1; then
print_message $RED "❌ Binary failed to run --help command"
return 1
fi
# Check version command
print_message $YELLOW " Testing --version command..."
if ! "$binary_path" --version >/dev/null 2>&1; then
print_message $YELLOW "⚠️ Binary does not support --version command (this is optional)"
fi
# Try to get some basic info about the binary
local file_info=$(file "$binary_path" 2>/dev/null || echo "unknown")
print_message $YELLOW " Binary info: $file_info"
# Check if it's a valid ELF/Mach-O binary
if command -v readelf >/dev/null 2>&1; then
if readelf -h "$binary_path" >/dev/null 2>&1; then
print_message $YELLOW " ELF binary structure: valid"
fi
elif command -v otool >/dev/null 2>&1; then
if otool -h "$binary_path" >/dev/null 2>&1; then
print_message $YELLOW " Mach-O binary structure: valid"
fi
fi
return 0
}
# Build binary for current platform
build_binary() {
local version=$(get_version)
local output_file="${OUTPUT_DIR}/${PLATFORM}/${BINARY_NAME}"
print_message $BLUE "🏗️ Building for platform: $PLATFORM"
print_message $YELLOW " Version: $version"
print_message $YELLOW " Output: $output_file"
# Create output directory
mkdir -p "${OUTPUT_DIR}/${PLATFORM}"
# Build command - use cross for cross-compilation if needed
local build_cmd="cargo build"
local current_platform=$(detect_platform)
# Check if we need cross-compilation
if [ "$PLATFORM" != "$current_platform" ]; then
# Check if cross is available
if command -v cross &> /dev/null; then
build_cmd="cross build"
print_message $YELLOW "🔄 Cross-compilation detected, using 'cross' tool"
else
print_message $YELLOW "⚠️ Cross-compilation detected but 'cross' tool not found"
print_message $YELLOW "📦 Installing cross tool..."
cargo install cross --git https://github.com/cross-rs/cross
build_cmd="cross build"
fi
fi
if [ "$BUILD_TYPE" = "release" ]; then
build_cmd+=" --release"
fi
build_cmd+=" --target $PLATFORM"
build_cmd+=" --bin $BINARY_NAME"
print_message $BLUE "📦 Executing: $build_cmd"
# Execute build
if eval $build_cmd; then
print_message $GREEN "✅ Successfully built for $PLATFORM"
# Copy binary to output directory
cp "target/${PLATFORM}/${BUILD_TYPE}/${BINARY_NAME}" "$output_file"
# Generate checksums
print_message $BLUE "🔐 Generating checksums..."
(cd "${OUTPUT_DIR}/${PLATFORM}" && sha256sum "${BINARY_NAME}" > "${BINARY_NAME}.sha256sum")
# Verify binary functionality (if not skipped)
if [ "$SKIP_VERIFICATION" = false ]; then
print_message $BLUE "🔍 Verifying binary functionality..."
if verify_binary "$output_file"; then
print_message $GREEN "✅ Binary verification passed"
else
print_message $RED "❌ Binary verification failed"
return 1
fi
else
print_message $YELLOW "⚠️ Binary verification skipped by user request"
fi
# Sign binary if requested
if [ "$SIGN" = true ]; then
print_message $BLUE "✍️ Signing binary..."
(cd "${OUTPUT_DIR}/${PLATFORM}" && minisign -S -m "${BINARY_NAME}" -s ~/.minisign/minisign.key)
fi
print_message $GREEN "✅ Build completed successfully"
else
print_message $RED "❌ Failed to build for $PLATFORM"
# Provide helpful suggestions for cross-compilation failures
if [ "$PLATFORM" != "$(detect_platform)" ]; then
print_message $YELLOW "💡 Cross-compilation suggestions:"
print_message $YELLOW " 1. Use Docker build: make build-docker"
print_message $YELLOW " 2. Use GitHub Actions for multi-platform builds"
print_message $YELLOW " 3. Build natively on the target platform"
print_message $YELLOW " 4. Use 'make docker-buildx' for multi-arch Docker images"
fi
return 1
fi
}
# Main build function
build_rustfs() {
local version=$(get_version)
print_message $BLUE "🚀 Starting RustFS binary build process..."
print_message $YELLOW " Version: $version"
print_message $YELLOW " Platform: $PLATFORM"
print_message $YELLOW " Output Directory: $OUTPUT_DIR"
print_message $YELLOW " Build Type: $BUILD_TYPE"
print_message $YELLOW " Sign: $SIGN"
print_message $YELLOW " With Console: $WITH_CONSOLE"
if [ "$WITH_CONSOLE" = true ]; then
print_message $YELLOW " Console Version: $CONSOLE_VERSION"
print_message $YELLOW " Force Console Update: $FORCE_CONSOLE_UPDATE"
fi
print_message $YELLOW " Skip Verification: $SKIP_VERIFICATION"
echo ""
# Setup environment
setup_rust_environment
echo ""
# Download console assets if requested
download_console_assets
echo ""
# Build binary
build_binary
echo ""
print_message $GREEN "🎉 Build process completed successfully!"
# Show built binary
local binary_file="${OUTPUT_DIR}/${PLATFORM}/${BINARY_NAME}"
if [ -f "$binary_file" ]; then
local size=$(ls -lh "$binary_file" | awk '{print $5}')
print_message $BLUE "📋 Built binary: $binary_file ($size)"
fi
}
# Parse command line arguments
while [[ $# -gt 0 ]]; do
case $1 in
-o|--output-dir)
OUTPUT_DIR="$2"
shift 2
;;
-b|--binary-name)
BINARY_NAME="$2"
shift 2
;;
-p|--platform)
CUSTOM_PLATFORM="$2"
shift 2
;;
--dev)
BUILD_TYPE="debug"
shift
;;
--sign)
SIGN=true
shift
;;
--with-console)
WITH_CONSOLE=true
shift
;;
--no-console)
WITH_CONSOLE=false
shift
;;
--force-console-update)
FORCE_CONSOLE_UPDATE=true
WITH_CONSOLE=true # Auto-enable download when forcing update
shift
;;
--console-version)
CONSOLE_VERSION="$2"
shift 2
;;
--skip-verification)
SKIP_VERIFICATION=true
shift
;;
-h|--help)
usage
exit 0
;;
*)
print_message $RED "❌ Unknown option: $1"
usage
exit 1
;;
esac
done
# Main execution
main() {
print_message $BLUE "🦀 RustFS Binary Build Script"
echo ""
# Check if we're in a Rust project
if [ ! -f "Cargo.toml" ]; then
print_message $RED "❌ No Cargo.toml found. Are you in a Rust project directory?"
exit 1
fi
# Override platform if specified
if [ -n "$CUSTOM_PLATFORM" ]; then
PLATFORM="$CUSTOM_PLATFORM"
print_message $YELLOW "🎯 Using specified platform: $PLATFORM"
# Auto-enable skip verification for cross-compilation
if [ "$PLATFORM" != "$(detect_platform)" ]; then
SKIP_VERIFICATION=true
print_message $YELLOW "⚠️ Cross-compilation detected, enabling --skip-verification"
fi
fi
# Start build process
build_rustfs
}
# Run main function
main

35
build_rustfs.sh
View File

@@ -1,35 +0,0 @@
#!/bin/bash
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
clear
# Get the current platform architecture
ARCH=$(uname -m)
# Set the target directory according to the schema
if [ "$ARCH" == "x86_64" ]; then
TARGET_DIR="target/x86_64"
elif [ "$ARCH" == "aarch64" ]; then
TARGET_DIR="target/arm64"
else
TARGET_DIR="target/unknown"
fi
# Set CARGO_TARGET_DIR and build the project
CARGO_TARGET_DIR=$TARGET_DIR RUSTFLAGS="-C link-arg=-fuse-ld=mold" cargo build --release --package rustfs
echo -e "\a"
echo -e "\a"
echo -e "\a"

0
.docker/cargo.config.toml → cargo.config.toml
View File

273
docker-buildx.sh Executable file
View File

@@ -0,0 +1,273 @@
#!/bin/bash
set -e
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Default values
REGISTRY="ghcr.io"
NAMESPACE="rustfs"
PLATFORMS="linux/amd64,linux/arm64"
PUSH=false
NO_CACHE=false
RELEASE=""
CHANNEL="release"
# Print usage
usage() {
echo "Usage: $0 [OPTIONS]"
echo ""
echo "Options:"
echo " -r, --registry REGISTRY Docker registry (default: ghcr.io)"
echo " -n, --namespace NAMESPACE Image namespace (default: rustfs)"
echo " -p, --platforms PLATFORMS Target platforms (default: linux/amd64,linux/arm64)"
echo " --push Push images to registry"
echo " --no-cache Disable build cache"
echo " --release VERSION Specify release version (default: auto-detect from git)"
echo " --channel CHANNEL Download channel: release or dev (default: release)"
echo " -h, --help Show this help message"
echo ""
echo "Examples:"
echo " $0 # Build all variants locally"
echo " $0 --push # Build and push all variants"
echo " $0 --push --no-cache # Build and push with no cache"
echo " $0 --release v1.0.0 # Build specific release version"
echo " $0 --channel dev # Build with dev channel binaries"
echo " $0 --release latest --channel dev # Build latest dev build"
}
# Print colored message
print_message() {
local color=$1
local message=$2
echo -e "${color}${message}${NC}"
}
# Check if Docker buildx is available
check_buildx() {
if ! docker buildx version >/dev/null 2>&1; then
print_message $RED "❌ Docker buildx is not available. Please install Docker with buildx support."
exit 1
fi
}
# Setup buildx builder
setup_builder() {
local builder_name="rustfs-builder"
print_message $BLUE "🔧 Setting up Docker buildx builder..."
# Check if builder exists
if docker buildx ls | grep -q "$builder_name"; then
print_message $YELLOW "⚠️ Builder '$builder_name' already exists, using existing one"
docker buildx use "$builder_name"
else
# Create new builder
docker buildx create --name "$builder_name" --driver docker-container --bootstrap
docker buildx use "$builder_name"
print_message $GREEN "✅ Created and activated builder '$builder_name'"
fi
# Inspect builder
docker buildx inspect --bootstrap
}
# Get version from git
get_version() {
if [ -n "$RELEASE" ]; then
echo "$RELEASE"
return
fi
# Try to get version from git tag
if git describe --abbrev=0 --tags >/dev/null 2>&1; then
git describe --abbrev=0 --tags
else
# Fallback to commit hash
git rev-parse --short HEAD
fi
}
# Build and push images
build_and_push() {
local version=$(get_version)
local image_base="${REGISTRY}/${NAMESPACE}/rustfs"
local build_date=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
local vcs_ref=$(git rev-parse --short HEAD)
print_message $BLUE "🚀 Building RustFS Docker images..."
print_message $YELLOW " Version: $version"
print_message $YELLOW " Registry: $REGISTRY"
print_message $YELLOW " Namespace: $NAMESPACE"
print_message $YELLOW " Platforms: $PLATFORMS"
print_message $YELLOW " Channel: $CHANNEL"
print_message $YELLOW " Build Date: $build_date"
print_message $YELLOW " VCS Ref: $vcs_ref"
print_message $YELLOW " Push: $PUSH"
print_message $YELLOW " No Cache: $NO_CACHE"
echo ""
# Build command base
local build_cmd="docker buildx build"
build_cmd+=" --platform $PLATFORMS"
build_cmd+=" --build-arg RELEASE=$version"
build_cmd+=" --build-arg CHANNEL=$CHANNEL"
build_cmd+=" --build-arg BUILD_DATE=$build_date"
build_cmd+=" --build-arg VCS_REF=$vcs_ref"
if [ "$NO_CACHE" = true ]; then
build_cmd+=" --no-cache"
fi
if [ "$PUSH" = true ]; then
build_cmd+=" --push"
else
build_cmd+=" --load"
fi
# Build latest variant
print_message $BLUE "🏗️ Building latest variant..."
local latest_cmd="$build_cmd"
# Add channel-specific tags
if [ "$CHANNEL" = "dev" ]; then
latest_cmd+=" -t ${image_base}:dev-latest"
else
latest_cmd+=" -t ${image_base}:latest"
fi
latest_cmd+=" --build-arg RELEASE=latest"
latest_cmd+=" -f Dockerfile ."
print_message $BLUE "📦 Executing: $latest_cmd"
if eval $latest_cmd; then
print_message $GREEN "✅ Successfully built latest variant"
else
print_message $RED "❌ Failed to build latest variant"
print_message $YELLOW "💡 Note: Make sure rustfs binaries are available at:"
if [ "$CHANNEL" = "dev" ]; then
print_message $YELLOW " https://dl.rustfs.com/artifacts/rustfs/dev/"
else
print_message $YELLOW " https://dl.rustfs.com/artifacts/rustfs/release/"
fi
exit 1
fi
# Prune build cache
docker buildx prune -f
# Build release variant (only if not latest)
if [ "$RELEASE" != "latest" ]; then
print_message $BLUE "🏗️ Building release variant..."
local release_cmd="$build_cmd"
release_cmd+=" -t ${image_base}:${version}"
# Add channel-specific tags
if [ "$CHANNEL" = "dev" ]; then
release_cmd+=" -t ${image_base}:dev-${version}"
else
release_cmd+=" -t ${image_base}:release"
fi
release_cmd+=" --build-arg RELEASE=${version}"
release_cmd+=" -f Dockerfile ."
print_message $BLUE "📦 Executing: $release_cmd"
if eval $release_cmd; then
print_message $GREEN "✅ Successfully built release variant"
else
print_message $RED "❌ Failed to build release variant"
print_message $YELLOW "💡 Note: Make sure rustfs binaries are available at:"
if [ "$CHANNEL" = "dev" ]; then
print_message $YELLOW " https://dl.rustfs.com/artifacts/rustfs/dev/"
else
print_message $YELLOW " https://dl.rustfs.com/artifacts/rustfs/release/"
fi
exit 1
fi
else
print_message $BLUE "⏭️ Skipping release variant (already built as latest)"
fi
# Final cleanup
docker buildx prune -f
}
# Parse command line arguments
while [[ $# -gt 0 ]]; do
case $1 in
-r|--registry)
REGISTRY="$2"
shift 2
;;
-n|--namespace)
NAMESPACE="$2"
shift 2
;;
-p|--platforms)
PLATFORMS="$2"
shift 2
;;
--push)
PUSH=true
shift
;;
--no-cache)
NO_CACHE=true
shift
;;
--release)
RELEASE="$2"
shift 2
;;
--channel)
CHANNEL="$2"
if [ "$CHANNEL" != "release" ] && [ "$CHANNEL" != "dev" ]; then
print_message $RED "❌ Invalid channel: $CHANNEL. Must be 'release' or 'dev'"
exit 1
fi
shift 2
;;
-h|--help)
usage
exit 0
;;
*)
print_message $RED "❌ Unknown option: $1"
usage
exit 1
;;
esac
done
# Main execution
main() {
print_message $BLUE "🐳 RustFS Docker Buildx Build Script"
echo ""
# Check prerequisites
check_buildx
# Setup builder
setup_builder
echo ""
# Start build process
build_and_push
print_message $GREEN "🎉 Build process completed successfully!"
# Show built images if not pushing
if [ "$PUSH" = false ]; then
print_message $BLUE "📋 Built images:"
docker images | grep "${NAMESPACE}/rustfs" | head -10
fi
}
# Run main function
main

4
docker-compose.yml
View File

@@ -23,7 +23,7 @@ services:
container_name: rustfs-server
build:
context: .
dockerfile: .docker/ubuntu/Dockerfile.source
dockerfile: Dockerfile.source
args:
TARGETPLATFORM: linux/amd64
ports:
@@ -68,7 +68,7 @@ services:
container_name: rustfs-dev
build:
context: .
dockerfile: .docker/ubuntu/Dockerfile.dev
dockerfile: Dockerfile.source
# Pure development environment
ports:
- "9010:9000"

8
rustfs/README.md
View File

@@ -74,10 +74,14 @@ To get started with RustFS, follow these steps:
2. **Docker Quick Start (Option 2)​​**
```bash
podman run -d -p 9000:9000 -p 9001:9001 -v /data:/data quay.io/rustfs/rustfs
# Docker Hub (recommended)
docker run -d -p 9000:9000 -v /data:/data rustfs/rustfs:latest
# Alternative using Podman
podman run -d -p 9000:9000 -v /data:/data rustfs/rustfs:latest
```
3. **Access the Console**: Open your web browser and navigate to `http://localhost:9001` to access the RustFS console,
3. **Access the Console**: Open your web browser and navigate to `http://localhost:9000` to access the RustFS console,
default username and password is `rustfsadmin` .
4. **Create a Bucket**: Use the console to create a new bucket for your objects.
5. **Upload Objects**: You can upload files directly through the console or use S3-compatible APIs to interact with your

256
scripts/build-docker-multiarch.sh
View File

@@ -1,256 +0,0 @@
#!/bin/bash
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -euo pipefail
# 多架构 Docker 构建脚本
# 支持构建并推送 x86_64 和 ARM64 架构的镜像
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
cd "$PROJECT_ROOT"
# 默认配置
REGISTRY_IMAGE_DOCKERHUB="rustfs/rustfs"
REGISTRY_IMAGE_GHCR="ghcr.io/rustfs/s3-rustfs"
VERSION="${VERSION:-latest}"
PUSH="${PUSH:-false}"
IMAGE_TYPE="${IMAGE_TYPE:-production}"
# 帮助信息
show_help() {
cat << EOF
用法: $0 [选项]
选项:
-h, --help 显示此帮助信息
-v, --version TAG 设置镜像版本标签 (默认: latest)
-p, --push 推送镜像到仓库
-t, --type TYPE 镜像类型 (production|ubuntu|rockylinux|devenv, 默认: production)
环境变量:
DOCKERHUB_USERNAME Docker Hub 用户名
DOCKERHUB_TOKEN Docker Hub 访问令牌
GITHUB_TOKEN GitHub 访问令牌
示例:
# 仅构建不推送
$0 --version v1.0.0
# 构建并推送到仓库
$0 --version v1.0.0 --push
# 构建 Ubuntu 版本
$0 --type ubuntu --version v1.0.0
EOF
}
# 解析命令行参数
while [[ $# -gt 0 ]]; do
case $1 in
-h|--help)
show_help
exit 0
;;
-v|--version)
VERSION="$2"
shift 2
;;
-p|--push)
PUSH=true
shift
;;
-t|--type)
IMAGE_TYPE="$2"
shift 2
;;
*)
echo "未知参数: $1"
show_help
exit 1
;;
esac
done
# 设置 Dockerfile 和后缀
case "$IMAGE_TYPE" in
production)
DOCKERFILE="Dockerfile"
SUFFIX=""
;;
ubuntu)
DOCKERFILE=".docker/Dockerfile.ubuntu22.04"
SUFFIX="-ubuntu22.04"
;;
rockylinux)
DOCKERFILE=".docker/Dockerfile.rockylinux9.3"
SUFFIX="-rockylinux9.3"
;;
devenv)
DOCKERFILE=".docker/Dockerfile.devenv"
SUFFIX="-devenv"
;;
*)
echo "错误: 不支持的镜像类型: $IMAGE_TYPE"
echo "支持的类型: production, ubuntu, rockylinux, devenv"
exit 1
;;
esac
echo "🚀 开始多架构 Docker 构建"
echo "📋 构建信息:"
echo " - 镜像类型: $IMAGE_TYPE"
echo " - Dockerfile: $DOCKERFILE"
echo " - 版本标签: $VERSION$SUFFIX"
echo " - 推送: $PUSH"
echo " - 架构: linux/amd64, linux/arm64"
# 检查必要的工具
if ! command -v docker &> /dev/null; then
echo "❌ 错误: 未找到 docker 命令"
exit 1
fi
# 检查 Docker Buildx
if ! docker buildx version &> /dev/null; then
echo "❌ 错误: Docker Buildx 不可用"
echo "请运行: docker buildx install"
exit 1
fi
# 创建并使用 buildx 构建器
BUILDER_NAME="rustfs-multiarch-builder"
if ! docker buildx inspect "$BUILDER_NAME" &> /dev/null; then
echo "🔨 创建多架构构建器..."
docker buildx create --name "$BUILDER_NAME" --use --bootstrap
else
echo "🔨 使用现有构建器..."
docker buildx use "$BUILDER_NAME"
fi
# 构建多架构二进制文件
echo "🔧 构建多架构二进制文件..."
# 检查是否存在预构建的二进制文件
if [[ ! -f "target/x86_64-unknown-linux-musl/release/rustfs" ]] || [[ ! -f "target/aarch64-unknown-linux-gnu/release/rustfs" ]]; then
echo "⚠️ 未找到预构建的二进制文件，正在构建..."
# 安装构建依赖
if ! command -v cross &> /dev/null; then
echo "📦 安装 cross 工具..."
cargo install cross
fi
# 生成 protobuf 代码
echo "📝 生成 protobuf 代码..."
cargo run --bin gproto || true
# 构建 x86_64
echo "🔨 构建 x86_64 二进制文件..."
cargo build --release --target x86_64-unknown-linux-musl --bin rustfs
# 构建 ARM64
echo "🔨 构建 ARM64 二进制文件..."
cross build --release --target aarch64-unknown-linux-gnu --bin rustfs
fi
# 准备构建参数
BUILD_ARGS=""
TAGS=""
# Docker Hub 标签
if [[ -n "${DOCKERHUB_USERNAME:-}" ]]; then
TAGS="$TAGS -t $REGISTRY_IMAGE_DOCKERHUB:$VERSION$SUFFIX"
fi
# GitHub Container Registry 标签
if [[ -n "${GITHUB_TOKEN:-}" ]]; then
TAGS="$TAGS -t $REGISTRY_IMAGE_GHCR:$VERSION$SUFFIX"
fi
# 如果没有设置标签，使用本地标签
if [[ -z "$TAGS" ]]; then
TAGS="-t rustfs:$VERSION$SUFFIX"
fi
# 构建镜像
echo "🏗️ 构建多架构 Docker 镜像..."
BUILD_CMD="docker buildx build \
--platform linux/amd64,linux/arm64 \
--file $DOCKERFILE \
$TAGS \
--build-arg BUILDTIME=$(date -u +%Y-%m-%dT%H:%M:%SZ) \
--build-arg VERSION=$VERSION \
--build-arg REVISION=$(git rev-parse --short HEAD 2>/dev/null || echo 'unknown')"
if [[ "$PUSH" == "true" ]]; then
# 登录到仓库
if [[ -n "${DOCKERHUB_USERNAME:-}" ]] && [[ -n "${DOCKERHUB_TOKEN:-}" ]]; then
echo "🔐 登录到 Docker Hub..."
echo "$DOCKERHUB_TOKEN" | docker login --username "$DOCKERHUB_USERNAME" --password-stdin
fi
if [[ -n "${GITHUB_TOKEN:-}" ]]; then
echo "🔐 登录到 GitHub Container Registry..."
echo "$GITHUB_TOKEN" | docker login ghcr.io --username "$(whoami)" --password-stdin
fi
BUILD_CMD="$BUILD_CMD --push"
else
BUILD_CMD="$BUILD_CMD --load"
fi
BUILD_CMD="$BUILD_CMD ."
echo "📋 执行构建命令:"
echo "$BUILD_CMD"
echo ""
# 执行构建
eval "$BUILD_CMD"
echo ""
echo "✅ 多架构 Docker 镜像构建完成!"
if [[ "$PUSH" == "true" ]]; then
echo "🚀 镜像已推送到仓库"
# 显示推送的镜像信息
echo ""
echo "📦 推送的镜像:"
if [[ -n "${DOCKERHUB_USERNAME:-}" ]]; then
echo " - $REGISTRY_IMAGE_DOCKERHUB:$VERSION$SUFFIX"
fi
if [[ -n "${GITHUB_TOKEN:-}" ]]; then
echo " - $REGISTRY_IMAGE_GHCR:$VERSION$SUFFIX"
fi
echo ""
echo "🔍 验证多架构支持:"
if [[ -n "${DOCKERHUB_USERNAME:-}" ]]; then
echo " docker buildx imagetools inspect $REGISTRY_IMAGE_DOCKERHUB:$VERSION$SUFFIX"
fi
if [[ -n "${GITHUB_TOKEN:-}" ]]; then
echo " docker buildx imagetools inspect $REGISTRY_IMAGE_GHCR:$VERSION$SUFFIX"
fi
else
echo "💾 镜像已构建到本地"
echo ""
echo "🔍 查看镜像:"
echo " docker images rustfs:$VERSION$SUFFIX"
fi
echo ""
echo "🎉 构建任务完成!"

2
scripts/run.ps1
View File

@@ -64,4 +64,4 @@ if ($args.Count -gt 0) {
}
# Run the program
cargo run --bin rustfs
cargo run --bin rustfs

2
scripts/run.sh
View File

@@ -112,4 +112,4 @@ fi
# 启动 webhook 服务器
#cargo run --example webhook -p rustfs-notify &
# 启动主服务
cargo run --bin rustfs
cargo run --bin rustfs

38
scripts/setup-test-binaries.sh Executable file
View File

@@ -0,0 +1,38 @@
#!/bin/bash
# Setup test binaries for Docker build testing
# This script creates temporary binary files for testing Docker build process
set -e
echo "Setting up test binaries for Docker build..."
# Create temporary rustfs binary
./build-rustfs.sh -p x86_64-unknown-linux-musl
# Create test directory structure
mkdir -p test-releases/server/rustfs/release/linux-amd64/archive
mkdir -p test-releases/server/rustfs/release/linux-arm64/archive
# Get version
VERSION=$(git describe --abbrev=0 --tags 2>/dev/null || git rev-parse --short HEAD)
# Copy binaries
cp target/release/x86_64-unknown-linux-musl/rustfs test-releases/server/rustfs/release/linux-amd64/archive/rustfs.${VERSION}
cp target/release/x86_64-unknown-linux-musl/rustfs.sha256sum test-releases/server/rustfs/release/linux-amd64/archive/rustfs.${VERSION}.sha256sum
# Create dummy signatures
echo "dummy signature" > test-releases/server/rustfs/release/linux-amd64/archive/rustfs.${VERSION}.minisig
echo "dummy signature" > test-releases/server/rustfs/release/linux-arm64/archive/rustfs.${VERSION}.minisig
# Also copy for arm64 (using same binary for testing)
cp target/release/x86_64-unknown-linux-musl/rustfs test-releases/server/rustfs/release/linux-arm64/archive/rustfs.${VERSION}
cp target/release/x86_64-unknown-linux-musl/rustfs.sha256sum test-releases/server/rustfs/release/linux-arm64/archive/rustfs.${VERSION}.sha256sum
echo "Test binaries created for version: ${VERSION}"
echo "You can now test Docker builds with these local binaries"
echo ""
echo "To start a local HTTP server for testing:"
echo " cd test-releases && python3 -m http.server 8000"
echo ""
echo "Then modify Dockerfile to use http://host.docker.internal:8000 instead of https://dl.rustfs.com/artifacts/rustfs"

2
scripts/static.sh
View File

@@ -12,4 +12,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
curl -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" -o tempfile.zip && unzip -o tempfile.zip -d ./rustfs/static && rm tempfile.zip
curl -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" -o tempfile.zip && unzip -o tempfile.zip -d ./rustfs/static && rm tempfile.zip