From 36e3efb5a5ebb945fe1401e65ff2b43835cdda77 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=AE=89=E6=AD=A3=E8=B6=85?= Date: Mon, 14 Jul 2025 19:15:46 +0800 Subject: [PATCH] feat: implement Docker improvements and binary build scripts (#191) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: implement Docker improvements and binary build scripts This commit transforms the RustFS Docker build system to follow MinIO's best practices: ## 🏗️ Binary Build Script (build-rustfs.sh) - Create independent binary compilation script for multi-platform builds - Support x86_64 and aarch64 Linux musl targets - Include checksum generation and optional binary signing - Support cross-compilation and upload functionality - Automated target installation and environment setup ## 🐳 Docker Improvements - Rewrite Dockerfiles to download precompiled binaries instead of building from source - Follow MinIO's approach for security and binary verification - Add comprehensive LABEL metadata (version, build-date, vcs-ref) - Implement proper environment variable management - Add signature verification with minisign (commented for future use) - Include static curl download for minimal runtime dependencies ## 🚀 Enhanced Build Script (docker-buildx.sh) - Inspired by MinIO's docker-buildx.sh for consistency and reliability - Support multiple platforms with proper build arguments - Auto-detect git versions and pass metadata to containers - Improved error messages with helpful troubleshooting hints - Cleanup and cache management between builds ## 🛠️ Supporting Scripts - scripts/download-static-curl.sh: Download statically compiled curl - scripts/setup-test-binaries.sh: Create test binaries for local development ## 📋 Key Benefits - Faster Docker builds (download vs compile) - Better security with signature verification - Consistent with industry standards (MinIO approach) - Proper multi-platform support - Enhanced metadata and traceability - Independent binary distribution capability * feat: update Docker files to use Aliyun OSS for binary downloads * feat: merge stash with OSS binary download improvements - Remove old build_rustfs.sh script - Keep Aliyun OSS download URLs for binary retrieval - Maintain Docker build improvements from stash - Resolve merge conflicts between stash and OSS updates * feat: improve build-rustfs.sh with auto platform detection - Auto-detect current platform using uname (like old build_rustfs.sh) - Default to building for current platform only - Add --all-platforms flag for cross-compilation to Linux musl targets - Support macOS (darwin) and Linux platforms - Auto-enable cross compilation when needed - Provide better usage examples and platform detection info This makes the script much more user-friendly by default while maintaining flexibility for cross-compilation scenarios. * refactor: simplify build-rustfs.sh for CI/CD pipeline usage - Remove cross-compilation complexity (each CI runner builds natively) - Focus on single platform builds per runner - Remove --all-platforms and --cross options - Simplify to match CI/CD workflow where: * Linux x86_64 runner builds Linux x86_64 binary * Linux ARM64 runner builds Linux ARM64 binary * macOS x86_64 runner builds macOS x86_64 binary * macOS ARM64 runner builds macOS ARM64 binary - Keep signing and upload functionality for release CI - Make the script's purpose and usage clearer This aligns with the user's understanding that build scripts should focus on native compilation for the current platform only. * feat: update download server domain to dl.rustfs.com - Update Dockerfile to use dl.rustfs.com/dev/ for development binaries - Update Dockerfile.release to use dl.rustfs.com/release/ for release binaries - Update docker-buildx.sh error messages with new URLs - Update build-rustfs.sh upload target to dl.rustfs.com - Update test scripts to reference new domain - Clean up remaining git conflict markers This centralizes all binary downloads through the official dl.rustfs.com domain instead of direct OSS access. * fix: correct dl.rustfs.com path structure to include /artifacts/rustfs/ - Update all download URLs to use correct path structure: * Dev: https://dl.rustfs.com/artifacts/rustfs/dev/ * Release: https://dl.rustfs.com/artifacts/rustfs/release/ - Test confirmed both paths return HTTP 200 with application/zip content-type - Update Dockerfile, Dockerfile.release, docker-buildx.sh, and build-rustfs.sh - Update test scripts with correct base path The dl.rustfs.com domain requires the /artifacts/rustfs/ prefix to access the binary files correctly. * feat: refactor Dockerfile to download binaries from GitHub Releases - Changed binary download source from dl.rustfs.com to GitHub Releases - Added support for latest release auto-detection via GitHub API - Enhanced error handling with detailed messages and helpful links - Added optional checksum verification using SHA256SUMS - Improved architecture support for amd64 and arm64 - Removed unnecessary minisign installation - Added jq dependency for JSON parsing * feat: consolidate Docker build to use single Dockerfile - Removed Dockerfile.release and use unified Dockerfile instead - Updated docker-buildx.sh to use single Dockerfile with build args - Both latest and release variants now use GitHub Releases - Simplified build process and reduced maintenance overhead - Updated error messages to point to GitHub releases * chore: remove unused Dockerfile.obs - Removed Dockerfile.obs as it's no longer needed - Simplified Docker build configuration * feat: unify Docker prebuild variants to use GitHub Releases - Updated .docker/alpine/Dockerfile.prebuild to download from GitHub Releases - Updated .docker/ubuntu/Dockerfile.prebuild to download from GitHub Releases - All prebuild variants now consistently use GitHub Releases as binary source - Added checksum verification for all prebuild variants - Updated .docker/README.md to reflect unified GitHub Releases approach - Improved error handling and user guidance in all prebuild Dockerfiles * feat: major Docker structure simplification and consolidation ## 🎯 Simplified Docker Structure Moved from complex multi-directory structure to clean root-level organization: ### Before: - Dockerfile (production) - .docker/alpine/Dockerfile.prebuild (duplicate) - .docker/alpine/Dockerfile.source - .docker/ubuntu/Dockerfile.prebuild (duplicate) - .docker/ubuntu/Dockerfile.source - .docker/ubuntu/Dockerfile.dev ### After: - Dockerfile (production - Alpine + GitHub Releases) - Dockerfile.source (source build - Ubuntu + cross-compilation) - Dockerfile.dev (development - Ubuntu + full toolchain) ## 🔧 Key Changes - **Eliminated Duplicates**: Removed redundant prebuild variants - **Moved Core Files**: Dockerfile.{source,dev} now in root directory - **Unified Configuration**: cargo.config.toml moved to root - **Updated References**: Fixed all GitHub Actions and docker-compose paths - **Simplified CI Matrix**: Reduced from 5 to 3 Docker variants ## 📦 Preserved Valuable Diversity - **Production**: Alpine-based for minimal size - **Source**: Ubuntu-based with cross-compilation support - **Development**: Ubuntu-based with full development tools ## 🚀 Benefits - ✅ Cleaner project structure - ✅ Easier maintenance and navigation - ✅ Reduced CI/CD complexity - ✅ Faster build matrix execution - ✅ Maintained functionality and flexibility * chore: remove duplicate cargo.config.toml from .docker directory The file is now in the root directory and no longer needed in .docker/ * fix: update all references to removed Dockerfile files - Updated .docker/compose/README.md to reference Dockerfile.source instead of Dockerfile.obs - Updated docker-compose.yml to use Dockerfile.source instead of Dockerfile.dev - Updated scripts/build-docker-multiarch.sh to use Dockerfile.source for devenv builds - Updated .github/workflows/docker.yml to use Dockerfile.source for dev builds - Updated Makefile to use Dockerfile.source for init-devenv target - Updated .docker/README.md to remove references to non-existent Dockerfile.dev - Ensured all Docker configurations consistently use the unified Dockerfile structure * chore: remove unnecessary console static assets download - Remove obsolete download steps from build.yml and performance.yml - Console static assets are already embedded via rust-embed in rustfs/static/ - The download from dl.rustfs.com is no longer needed as project contains complete console assets - This improves build reliability and reduces external dependencies - Replaced with verification steps that confirm embedded assets are present * feat: update Makefile and README.md for new Docker build system - Updated Makefile to use unified Docker build system: - Replace references to non-existent Dockerfile.ubuntu22.04 and Dockerfile.rockylinux9.3 - Add new docker-buildx targets using docker-buildx.sh script - Deprecate old docker-build-multiarch targets with warnings - Add docker-build-production and docker-build-source targets - Update help-docker with new command structure - Updated README.md with docker-buildx.sh usage: - Add comprehensive Docker build from source section - Document multi-architecture build capabilities - Include both script and Make target examples - Show registry flexibility and build optimization features - Update step numbers in quickstart guide - Improve developer experience with clear documentation and updated tooling - Maintain backward compatibility with deprecation warnings * feat: integrate console assets download into build-rustfs.sh - Added console download functionality to build-rustfs.sh: - New flags: --download-console, --force-console-update, --console-version - Intelligent detection of existing console assets - Retry logic with fallback error handling - Consistent with Docker build asset management - Updated scripts to use unified build process: - scripts/static.sh: Now uses build-rustfs.sh for console downloads - scripts/run.sh: Uses build-rustfs.sh instead of direct curl - scripts/run.ps1: Updated with guidance for Windows users - Benefits: - Unified asset management across all build processes - Consistent version handling and retry logic - Eliminates duplicate download logic - Better error handling and user feedback - Preparation for CI/CD integration - Removed unused download-static-curl.sh script This change centralizes console asset management and prepares for streamlined CI/CD processes where build-rustfs.sh becomes the single point of truth for binary and asset builds. * fix: update PowerShell script to use unified console asset management - Updated scripts/run.ps1 to use build-rustfs.sh for console asset downloads - Added guidance for Windows users to use the unified build script - Maintains consistency across all platform-specific scripts * feat: add binary verification to build script - Add verify_binary function to test built binaries - Test --help and --version commands - Verify binary structure with readelf/otool - Add --skip-verification option for cross-compilation - Include verification status in build output - Automatic error handling if verification fails * feat: add platform selection support to build script - Add --platform parameter to build-rustfs.sh for target platform selection - Implement cross-compilation support with automatic 'cross' tool detection - Auto-enable --skip-verification for cross-compilation scenarios - Update all Makefile build targets to use unified build-rustfs.sh script - Add helpful error messages and suggestions for cross-compilation failures - Update help documentation with platform selection examples - Improve build consistency across different architectures * feat: modernize CI/CD build process with build-rustfs.sh - Replace manual cargo build commands with unified build-rustfs.sh script - Simplify matrix configuration by removing cross-compilation flags - Ensure consistency between local and CI/CD builds - Automatic cross-compilation tool detection and installation - Built-in binary verification for quality assurance - Unified console asset management - Better error handling and suggestions Benefits: - Consistent build process across all environments - Automatic detection and handling of cross-compilation scenarios - Built-in quality checks with binary verification - Reduced CI/CD configuration complexity - Better maintainability with single source of truth for build logic * feat: optimize CI/CD workspace path management - Add WORKSPACE_DIR environment variable to cache github.workspace - Set default working-directory at job level for consistency - Use explicit workspace paths in critical operations - Improve reliability and maintainability of CI/CD paths - Ensure consistent behavior across different GitHub Actions environments Benefits: - More explicit and reliable path handling - Better maintainability with centralized workspace reference - Reduced risk of path-related issues in CI/CD - Consistent working directory across all job steps * refactor: simplify CI/CD path management - remove redundant workspace references - Remove unnecessary WORKSPACE_DIR environment variable - Remove redundant defaults.run.working-directory setting - Use relative paths since GITHUB_WORKSPACE is the default working directory - Follow GitHub Actions best practices by leveraging default behavior As per GitHub Actions documentation, GITHUB_WORKSPACE is already the default working directory, so explicit specification is unnecessary in most cases. * docs: update Docker README to reflect current project state - Fix directory structure: remove non-existent nginx/ directory - Correct base OS: Dockerfile.source uses Debian Bookworm, not Ubuntu 22.04 - Add docker-buildx.sh script documentation - Update Docker tag examples to match actual CI/CD workflows - Add CI/CD integration section explaining automated builds - Document build variants and manual build options - Reflect current project architecture and tooling These updates ensure the documentation accurately represents the current Docker build system and CI/CD workflows. * fix: update Docker command in rustfs README - Replace quay.io registry with Docker Hub (rustfs/rustfs:latest) - Remove separate console port 9001, console now runs on main port 9000 - Add both Docker and Podman examples for user choice - Fix console access URL to use unified port This aligns with the recent console port consolidation changes and the project's move to Docker Hub as the primary registry. * wip * fix: remove unnecessary entrypoint.sh and fix Docker paths * Update Dockerfile Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * cleanup: remove unused DOCKERFILE_PATH variable from Makefile * feat: update Docker build to use dl.rustfs.com for binary downloads - Replace GitHub releases download with dl.rustfs.com - Add CHANNEL parameter support (release/dev) - Update docker-buildx.sh to support channel-specific builds - Improve error messages with new download URLs - Support both latest and specific version downloads - Add channel validation in build script --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .docker/README.md | 238 +++++---- .docker/alpine/Dockerfile.prebuild | 142 ----- .docker/alpine/Dockerfile.source | 126 ----- .docker/compose/README.md | 2 +- .../compose/docker-compose.observability.yaml | 8 +- .docker/ubuntu/Dockerfile.dev | 96 ---- .docker/ubuntu/Dockerfile.prebuild | 154 ------ .github/workflows/build.yml | 56 +- .github/workflows/docker.yml | 14 +- .github/workflows/performance.yml | 17 +- Dockerfile | 236 ++++----- Dockerfile.obs | 61 --- .../Dockerfile.source => Dockerfile.source | 5 +- Makefile | 170 +++--- README.md | 39 +- README_ZH.md | 5 +- build-rustfs.sh | 488 ++++++++++++++++++ build_rustfs.sh | 35 -- .../cargo.config.toml => cargo.config.toml | 0 docker-buildx.sh | 273 ++++++++++ docker-compose.yml | 4 +- rustfs/README.md | 8 +- scripts/build-docker-multiarch.sh | 256 --------- scripts/run.ps1 | 2 +- scripts/run.sh | 2 +- scripts/setup-test-binaries.sh | 38 ++ scripts/static.sh | 2 +- 27 files changed, 1225 insertions(+), 1252 deletions(-) delete mode 100644 .docker/alpine/Dockerfile.prebuild delete mode 100644 .docker/alpine/Dockerfile.source delete mode 100644 .docker/ubuntu/Dockerfile.dev delete mode 100644 .docker/ubuntu/Dockerfile.prebuild delete mode 100644 Dockerfile.obs rename .docker/ubuntu/Dockerfile.source => Dockerfile.source (98%) create mode 100755 build-rustfs.sh delete mode 100755 build_rustfs.sh rename .docker/cargo.config.toml => cargo.config.toml (100%) create mode 100755 docker-buildx.sh delete mode 100755 scripts/build-docker-multiarch.sh create mode 100755 scripts/setup-test-binaries.sh diff --git a/.docker/README.md b/.docker/README.md index db312db4..f3e2da38 100644 --- a/.docker/README.md +++ b/.docker/README.md @@ -1,157 +1,199 @@ # RustFS Docker Images -This directory contains organized Dockerfile configurations for building RustFS container images across multiple platforms and system versions. +This directory contains Docker configuration files and supporting infrastructure for building and running RustFS container images. ## 📁 Directory Structure ``` -.docker/ -├── alpine/ # Alpine Linux variants -│ ├── Dockerfile.prebuild # Alpine + pre-built binaries -│ └── Dockerfile.source # Alpine + source compilation -├── ubuntu/ # Ubuntu variants -│ ├── Dockerfile.prebuild # Ubuntu + pre-built binaries -│ ├── Dockerfile.source # Ubuntu + source compilation -│ └── Dockerfile.dev # Ubuntu + development environment -└── cargo.config.toml # Rust cargo configuration +rustfs/ +├── Dockerfile # Production image (Alpine + GitHub Releases) +├── Dockerfile.source # Source build (Debian + cross-compilation) +├── cargo.config.toml # Rust cargo configuration +├── docker-buildx.sh # Multi-architecture build script +└── .docker/ # Supporting infrastructure + ├── observability/ # Monitoring and observability configs + ├── compose/ # Docker Compose configurations + ├── mqtt/ # MQTT broker configs + └── openobserve-otel/ # OpenObserve + OpenTelemetry configs ``` ## 🎯 Image Variants -### Production Images +### Core Images -| Variant | Base OS | Build Method | Size | Use Case | -|---------|---------|--------------|------|----------| -| `production` (default) | Alpine 3.18 | Pre-built | Smallest | Production deployment | -| `alpine` | Alpine 3.18 | Pre-built | Small | Explicit Alpine choice | -| `alpine-source` | Alpine 3.18 | Source build | Small | Custom Alpine builds | -| `ubuntu` | Ubuntu 22.04 | Pre-built | Medium | Ubuntu environments | -| `ubuntu-source` | Ubuntu 22.04 | Source build | Medium | Full Ubuntu compatibility | - -### Development Images - -| Variant | Base OS | Features | Use Case | -|---------|---------|----------|----------| -| `ubuntu-dev` | Ubuntu 22.04 | Full toolchain + dev tools | Interactive development | +| Image | Base OS | Build Method | Size | Use Case | +|-------|---------|--------------|------|----------| +| `production` (default) | Alpine 3.18 | GitHub Releases | Smallest | Production deployment | +| `source` | Debian Bookworm | Source build | Medium | Custom builds with cross-compilation | +| `dev` | Debian Bookworm | Development tools | Large | Interactive development | ## 🚀 Usage Examples ### Quick Start (Production) ```bash -# Default production image (Alpine + pre-built) +# Default production image (Alpine + GitHub Releases) docker run -p 9000:9000 rustfs/rustfs:latest -# Specific version with production variant -docker run -p 9000:9000 rustfs/rustfs:1.2.3-production - -# Explicit Alpine variant -docker run -p 9000:9000 rustfs/rustfs:latest-alpine - -# Ubuntu-based production -docker run -p 9000:9000 rustfs/rustfs:latest-ubuntu +# Specific version +docker run -p 9000:9000 rustfs/rustfs:1.2.3 ``` ### Complete Tag Strategy Examples ```bash # Stable Releases -docker run rustfs/rustfs:1.2.3 # Main version (production) -docker run rustfs/rustfs:1.2.3-production # Explicit production variant -docker run rustfs/rustfs:1.2.3-alpine # Explicit Alpine variant -docker run rustfs/rustfs:1.2.3-alpine-source # Alpine source build -docker run rustfs/rustfs:latest # Latest stable +docker run rustfs/rustfs:1.2.3 # Main version (production) +docker run rustfs/rustfs:1.2.3-production # Explicit production variant +docker run rustfs/rustfs:1.2.3-source # Source build variant +docker run rustfs/rustfs:latest # Latest stable # Prerelease Versions -docker run rustfs/rustfs:1.3.0-alpha.2 # Specific alpha version -docker run rustfs/rustfs:1.3.0-alpha.2-alpine # Alpha with Alpine -docker run rustfs/rustfs:alpha # Latest alpha -docker run rustfs/rustfs:beta # Latest beta -docker run rustfs/rustfs:rc # Latest release candidate +docker run rustfs/rustfs:1.3.0-alpha.2 # Specific alpha version +docker run rustfs/rustfs:alpha # Latest alpha +docker run rustfs/rustfs:beta # Latest beta +docker run rustfs/rustfs:rc # Latest release candidate # Development Versions -docker run rustfs/rustfs:dev # Latest development -docker run rustfs/rustfs:dev-13e4a0b # Specific commit -docker run rustfs/rustfs:dev-alpine # Development Alpine +docker run rustfs/rustfs:dev # Latest main branch development +docker run rustfs/rustfs:dev-13e4a0b # Specific commit +docker run rustfs/rustfs:dev-latest # Latest development +docker run rustfs/rustfs:main-latest # Main branch latest ``` ### Development Environment ```bash # Start development container -docker run -it -v $(pwd):/app -p 9000:9000 rustfs/rustfs:latest-ubuntu-dev +docker run -it -v $(pwd):/workspace -p 9000:9000 rustfs/rustfs:latest-dev -# Inside container: -cd /app -cargo build --release -cargo run +# Build from source locally +docker build -f Dockerfile.source -t rustfs:custom . + +# Development with hot reload +docker-compose up rustfs-dev ``` -## 🏗️ Build Arguments +## 🏗️ Build Arguments and Scripts + +### Using docker-buildx.sh (Recommended) + +For multi-architecture builds, use the provided script: + +```bash +# Build latest version for all architectures +./docker-buildx.sh + +# Build and push to registry +./docker-buildx.sh --push + +# Build specific version +./docker-buildx.sh --release v1.2.3 + +# Build and push specific version +./docker-buildx.sh --release v1.2.3 --push +``` + +### Manual Docker Builds All images support dynamic version selection: ```bash -# Build with specific version -docker build \ - --build-arg VERSION="1.0.0" \ - --build-arg BUILD_TYPE="release" \ - -f .docker/alpine/Dockerfile.prebuild \ - -t rustfs:1.0.0-alpine . +# Build production image with latest release +docker build --build-arg RELEASE="latest" -t rustfs:latest . + +# Build from source with specific target +docker build -f Dockerfile.source \ + --build-arg TARGETPLATFORM="linux/amd64" \ + -t rustfs:source . + +# Development build +docker build -f Dockerfile.source -t rustfs:dev . ``` -## 🌐 Multi-Platform Support +## 🔧 Binary Download Sources -All images support multiple architectures: +### Unified GitHub Releases -- `linux/amd64` (Intel/AMD 64-bit) -- `linux/arm64` (ARM 64-bit, Apple Silicon, etc.) +The production image downloads from GitHub Releases for reliability and transparency: -## ⚡ Build Speed Optimizations +- ✅ **production** → GitHub Releases API with automatic latest detection +- ✅ **Checksum verification** → SHA256SUMS validation when available +- ✅ **Multi-architecture** → Supports amd64 and arm64 -### Docker Build Optimizations +### Source Build -- **Multi-layer caching**: GitHub Actions cache + Registry cache -- **Parallel matrix builds**: All 5 variants build simultaneously -- **Multi-platform builds**: amd64/arm64 built in parallel -- **BuildKit features**: Advanced caching and inline cache +The source variant compiles from source code with advanced features: -### Rust Compilation Optimizations +- 🔧 **Cross-compilation** → Supports multiple target platforms via `TARGETPLATFORM` +- ⚡ **Build caching** → sccache for faster compilation +- 🎯 **Optimized builds** → Release optimizations with LTO and symbol stripping -- **sccache**: Distributed compilation cache for Rust builds -- **Parallel compilation**: Uses all available CPU cores (`-j $(nproc)`) -- **Optimized cargo config**: Sparse registry protocol, fast linker (lld) -- **Dependency caching**: Separate Docker layers for dependencies vs. source code -- **Release optimizations**: LTO, strip symbols, optimized codegen +## 📋 Architecture Support -### Cache Strategy +All variants support multi-architecture builds: -```yaml -# GitHub Actions cache -cache-from: type=gha,scope=docker-{variant} -cache-to: type=gha,mode=max,scope=docker-{variant} +- **linux/amd64** (x86_64) +- **linux/arm64** (aarch64) -# Registry cache (persistent across runs) -cache-from: type=registry,ref=ghcr.io/rustfs/rustfs:buildcache-{variant} -cache-to: type=registry,ref=ghcr.io/rustfs/rustfs:buildcache-{variant} +Architecture is automatically detected during build using Docker's `TARGETARCH` build argument. + +## 🔐 Security Features + +- **Checksum Verification**: Production image verifies SHA256SUMS when available +- **Non-root User**: All images run as user `rustfs` (UID 1000) +- **Minimal Runtime**: Production image only includes necessary dependencies +- **Secure Defaults**: No hardcoded credentials or keys + +## 🛠️ Development Workflow + +For local development and testing: + +```bash +# Quick development setup +docker-compose up rustfs-dev + +# Custom source build +docker build -f Dockerfile.source -t rustfs:custom . + +# Run with development tools +docker run -it -v $(pwd):/workspace rustfs:custom bash ``` -### Build Performance Comparison +## 🚀 CI/CD Integration -| Build Type | Time (Est.) | Cache Hit | Cache Miss | -|------------|-------------|-----------|-----------| -| Production (Alpine pre-built) | ~2-3 min | ~1 min | ~2 min | -| Alpine pre-built | ~2-3 min | ~1 min | ~2 min | -| Alpine source | ~8-12 min | ~3-5 min | ~10 min | -| Ubuntu pre-built | ~3-4 min | ~1-2 min | ~3 min | -| Ubuntu source | ~10-15 min | ~4-6 min | ~12 min | +The project uses GitHub Actions for automated multi-architecture Docker builds: -## 📋 Build Matrix +### Automated Builds -| Trigger | Version Format | Download Path | Image Tags | -|---------|---------------|---------------|------------| -| `push main` | `dev-{sha}` | `artifacts/rustfs/dev/` | `dev-{sha}-{variant}`, `dev-{variant}`, `dev` | -| `push 1.2.3` | `1.2.3` | `artifacts/rustfs/release/` | `1.2.3-{variant}`, `1.2.3`, `latest-{variant}`, `latest` | -| `push 1.3.0-alpha.2` | `1.3.0-alpha.2` | `artifacts/rustfs/release/` | `1.3.0-alpha.2-{variant}`, `alpha-{variant}`, `alpha` | -| `push 1.3.0-beta.1` | `1.3.0-beta.1` | `artifacts/rustfs/release/` | `1.3.0-beta.1-{variant}`, `beta-{variant}`, `beta` | -| `push 1.3.0-rc.1` | `1.3.0-rc.1` | `artifacts/rustfs/release/` | `1.3.0-rc.1-{variant}`, `rc-{variant}`, `rc` | +- **Tags**: Automatic builds triggered on version tags (e.g., `v1.2.3`) +- **Main Branch**: Development builds with `dev-latest` and `main-latest` tags +- **Pull Requests**: Test builds without registry push + +### Build Variants + +Each build creates three image variants: + +- `rustfs/rustfs:v1.2.3` (production - Alpine-based) +- `rustfs/rustfs:v1.2.3-source` (source build - Debian-based) +- `rustfs/rustfs:v1.2.3-dev` (development - Debian-based with tools) + +### Manual Builds + +Trigger custom builds via GitHub Actions: + +```bash +# Use workflow_dispatch to build specific versions +# Available options: latest, main-latest, dev-latest, v1.2.3, dev-abc123 +``` + +## 📦 Supporting Infrastructure + +The `.docker/` directory contains supporting configuration files: + +- **observability/** - Prometheus, Grafana, OpenTelemetry configs +- **compose/** - Multi-service Docker Compose setups +- **mqtt/** - MQTT broker configurations +- **openobserve-otel/** - Log aggregation and tracing setup + +See individual README files in each subdirectory for specific usage instructions. diff --git a/.docker/alpine/Dockerfile.prebuild b/.docker/alpine/Dockerfile.prebuild deleted file mode 100644 index fbd38472..00000000 --- a/.docker/alpine/Dockerfile.prebuild +++ /dev/null @@ -1,142 +0,0 @@ -# Copyright 2024 RustFS Team -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Multi-stage Alpine build for minimal runtime image -FROM rust:1.88-alpine AS builder - -# Build arguments for dynamic artifact download -ARG VERSION="" -ARG BUILD_TYPE="release" -ARG TARGETARCH - -# Install build dependencies -RUN apk add --no-cache \ - musl-dev \ - pkgconfig \ - openssl-dev \ - openssl-libs-static \ - curl \ - unzip \ - bash \ - wget \ - ca-certificates - -# Install protoc -RUN wget https://github.com/protocolbuffers/protobuf/releases/download/v31.1/protoc-31.1-linux-x86_64.zip \ - && unzip protoc-31.1-linux-x86_64.zip -d protoc3 \ - && mv protoc3/bin/* /usr/local/bin/ && chmod +x /usr/local/bin/protoc \ - && mv protoc3/include/* /usr/local/include/ && rm -rf protoc-31.1-linux-x86_64.zip protoc3 - -# Install flatc -RUN wget https://github.com/google/flatbuffers/releases/download/v25.2.10/Linux.flatc.binary.g++-13.zip \ - && unzip Linux.flatc.binary.g++-13.zip \ - && mv flatc /usr/local/bin/ && chmod +x /usr/local/bin/flatc \ - && rm -rf Linux.flatc.binary.g++-13.zip - -# Download pre-built binary (VERSION is required) -RUN if [ -z "$VERSION" ]; then \ - echo "❌ ERROR: VERSION build argument is required"; \ - echo "Please provide VERSION (e.g., main-latest, latest, v1.0.0, dev-abc123)"; \ - exit 1; \ - fi; \ - \ - # Map TARGETARCH to our naming convention - case "${TARGETARCH}" in \ - amd64) ARCH="x86_64" ;; \ - arm64) ARCH="aarch64" ;; \ - *) echo "❌ Unsupported architecture: ${TARGETARCH}" && exit 1 ;; \ - esac; \ - \ - # Handle VERSION with different patterns - if [[ "$VERSION" == "main-latest" ]]; then \ - DOWNLOAD_PATH="artifacts/rustfs/dev"; \ - FILENAME="rustfs-linux-${ARCH}-main-latest.zip"; \ - elif [[ "$VERSION" == "latest" ]]; then \ - DOWNLOAD_PATH="artifacts/rustfs/release"; \ - FILENAME="rustfs-linux-${ARCH}-latest.zip"; \ - elif [[ "$VERSION" == dev-* ]]; then \ - DOWNLOAD_PATH="artifacts/rustfs/dev"; \ - FILENAME="rustfs-linux-${ARCH}-dev-${VERSION#dev-}.zip"; \ - else \ - DOWNLOAD_PATH="artifacts/rustfs/release"; \ - FILENAME="rustfs-linux-${ARCH}-v${VERSION}.zip"; \ - fi; \ - \ - # Download the binary - DOWNLOAD_URL="https://dl.rustfs.com/${DOWNLOAD_PATH}/${FILENAME}"; \ - echo "🔽 Downloading RustFS binary from: ${DOWNLOAD_URL}"; \ - \ - # Download with clear error handling - if ! curl -fsSL --connect-timeout 30 --max-time 120 -o /tmp/rustfs.zip "${DOWNLOAD_URL}"; then \ - echo "❌ Failed to download binary from: ${DOWNLOAD_URL}"; \ - echo "💡 Please ensure the binary exists or trigger a build first"; \ - echo "💡 Available options:"; \ - echo " - For main-latest: Push to main branch or run build workflow"; \ - echo " - For latest: Create a release tag"; \ - echo " - For dev-xxx: Run build workflow with specific commit"; \ - exit 1; \ - fi; \ - \ - # Extract binary - if ! unzip -o /tmp/rustfs.zip -d /tmp >/dev/null 2>&1; then \ - echo "❌ Failed to extract downloaded binary"; \ - exit 1; \ - fi; \ - \ - # Install binary - mv /tmp/rustfs /usr/local/bin/rustfs; \ - chmod +x /usr/local/bin/rustfs; \ - rm -rf /tmp/*; \ - \ - echo "✅ Successfully downloaded and installed RustFS binary (${VERSION})" - -# Final Alpine runtime image -FROM alpine:3.18 - -RUN apk add --no-cache \ - ca-certificates \ - tzdata \ - bash - -# Create rustfs user for security -RUN addgroup -g 1000 rustfs && \ - adduser -D -u 1000 -G rustfs rustfs - -WORKDIR /app - -# Copy binary from builder -COPY --from=builder /usr/local/bin/rustfs /app/rustfs -RUN chmod +x /app/rustfs && chown rustfs:rustfs /app/rustfs - -# Create data directories -RUN mkdir -p /data && chown -R rustfs:rustfs /data /app - -# Switch to non-root user -USER rustfs - -# Environment variables -ENV RUSTFS_ACCESS_KEY=rustfsadmin \ - RUSTFS_SECRET_KEY=rustfsadmin \ - RUSTFS_ADDRESS=":9000" \ - RUSTFS_CONSOLE_ENABLE=true \ - RUSTFS_VOLUMES=/data \ - RUST_LOG=warn - -EXPOSE 9000 - -# Health check -HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \ - CMD wget --no-verbose --tries=1 --spider http://localhost:9000/health || exit 1 - -CMD ["/app/rustfs"] diff --git a/.docker/alpine/Dockerfile.source b/.docker/alpine/Dockerfile.source deleted file mode 100644 index 6de7b802..00000000 --- a/.docker/alpine/Dockerfile.source +++ /dev/null @@ -1,126 +0,0 @@ -# Copyright 2024 RustFS Team -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Multi-stage Alpine build from source -FROM rust:1.88-alpine AS builder - -# Install build dependencies -RUN apk add --no-cache \ - musl-dev \ - pkgconfig \ - openssl-dev \ - openssl-libs-static \ - curl \ - unzip \ - bash \ - wget \ - ca-certificates \ - git - -# Install sccache for Rust compilation caching -RUN wget https://github.com/mozilla/sccache/releases/download/v0.8.1/sccache-v0.8.1-x86_64-unknown-linux-musl.tar.gz \ - && tar -xzf sccache-v0.8.1-x86_64-unknown-linux-musl.tar.gz \ - && mv sccache-v0.8.1-x86_64-unknown-linux-musl/sccache /usr/local/bin/ \ - && chmod +x /usr/local/bin/sccache \ - && rm -rf sccache-v0.8.1-x86_64-unknown-linux-musl.tar.gz sccache-v0.8.1-x86_64-unknown-linux-musl - -# Set up sccache environment -ENV RUSTC_WRAPPER=sccache \ - SCCACHE_DIR=/tmp/sccache \ - SCCACHE_CACHE_SIZE=2G - -# Install protoc -RUN wget https://github.com/protocolbuffers/protobuf/releases/download/v31.1/protoc-31.1-linux-x86_64.zip \ - && unzip protoc-31.1-linux-x86_64.zip -d protoc3 \ - && mv protoc3/bin/* /usr/local/bin/ && chmod +x /usr/local/bin/protoc \ - && mv protoc3/include/* /usr/local/include/ && rm -rf protoc-31.1-linux-x86_64.zip protoc3 - -# Install flatc -RUN wget https://github.com/google/flatbuffers/releases/download/v25.2.10/Linux.flatc.binary.g++-13.zip \ - && unzip Linux.flatc.binary.g++-13.zip \ - && mv flatc /usr/local/bin/ && chmod +x /usr/local/bin/flatc \ - && rm -rf Linux.flatc.binary.g++-13.zip - -WORKDIR /usr/src/rustfs - -# Copy cargo configuration for optimized builds -COPY .docker/cargo.config.toml ./.cargo/config.toml - -# Copy cargo files for dependency caching -COPY Cargo.toml Cargo.lock ./ -COPY */Cargo.toml ./*/ - -# Create dummy main.rs files for dependency compilation -RUN find . -name "Cargo.toml" -not -path "./Cargo.toml" | \ - xargs -I {} dirname {} | \ - xargs -I {} sh -c 'mkdir -p {}/src && echo "fn main() {}" > {}/src/main.rs' - -# Configure cargo for optimized builds -ENV CARGO_NET_GIT_FETCH_WITH_CLI=true \ - CARGO_REGISTRIES_CRATES_IO_PROTOCOL=sparse \ - CARGO_INCREMENTAL=0 \ - CARGO_PROFILE_RELEASE_DEBUG=false \ - CARGO_PROFILE_RELEASE_SPLIT_DEBUGINFO=off \ - CARGO_PROFILE_RELEASE_STRIP=symbols - -# Build dependencies only (cache layer) with optimizations -RUN cargo build --release --target x86_64-unknown-linux-musl -j $(nproc) - -# Copy source code -COPY . . - -# Build the actual application with optimizations -RUN sccache --start-server 2>/dev/null || true && \ - cargo build --release --target x86_64-unknown-linux-musl --bin rustfs -j $(nproc) && \ - sccache --show-stats || true - -# Final Alpine runtime image -FROM alpine:3.18 - -RUN apk add --no-cache \ - ca-certificates \ - tzdata \ - bash - -# Create rustfs user for security -RUN addgroup -g 1000 rustfs && \ - adduser -D -u 1000 -G rustfs rustfs - -WORKDIR /app - -# Copy binary from builder -COPY --from=builder /usr/src/rustfs/target/x86_64-unknown-linux-musl/release/rustfs /app/rustfs -RUN chmod +x /app/rustfs && chown rustfs:rustfs /app/rustfs - -# Create data directories -RUN mkdir -p /data && chown -R rustfs:rustfs /data /app - -# Switch to non-root user -USER rustfs - -# Environment variables -ENV RUSTFS_ACCESS_KEY=rustfsadmin \ - RUSTFS_SECRET_KEY=rustfsadmin \ - RUSTFS_ADDRESS=":9000" \ - RUSTFS_CONSOLE_ENABLE=true \ - RUSTFS_VOLUMES=/data \ - RUST_LOG=warn - -EXPOSE 9000 - -# Health check -HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \ - CMD wget --no-verbose --tries=1 --spider http://localhost:9000/health || exit 1 - -CMD ["/app/rustfs"] diff --git a/.docker/compose/README.md b/.docker/compose/README.md index b52267b5..600a6aad 100644 --- a/.docker/compose/README.md +++ b/.docker/compose/README.md @@ -25,7 +25,7 @@ This directory contains specialized Docker Compose configurations and their asso - **`docker-compose.observability.yaml`** - **Observability Focus** - Specialized setup for testing observability features - Includes OpenTelemetry, Jaeger, Prometheus, Loki, Grafana - - Uses `../../Dockerfile.obs` for builds + - Uses `../../Dockerfile.source` for builds - Perfect for observability development ## 🚀 Usage Examples diff --git a/.docker/compose/docker-compose.observability.yaml b/.docker/compose/docker-compose.observability.yaml index d0ff4299..03704ce6 100644 --- a/.docker/compose/docker-compose.observability.yaml +++ b/.docker/compose/docker-compose.observability.yaml @@ -73,7 +73,7 @@ services: node1: build: context: ../.. - dockerfile: Dockerfile.obs + dockerfile: Dockerfile.source container_name: node1 environment: - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4} @@ -90,7 +90,7 @@ services: node2: build: context: ../.. - dockerfile: Dockerfile.obs + dockerfile: Dockerfile.source container_name: node2 environment: - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4} @@ -107,7 +107,7 @@ services: node3: build: context: ../.. - dockerfile: Dockerfile.obs + dockerfile: Dockerfile.source container_name: node3 environment: - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4} @@ -124,7 +124,7 @@ services: node4: build: context: ../.. - dockerfile: Dockerfile.obs + dockerfile: Dockerfile.source container_name: node4 environment: - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4} diff --git a/.docker/ubuntu/Dockerfile.dev b/.docker/ubuntu/Dockerfile.dev deleted file mode 100644 index a79216fc..00000000 --- a/.docker/ubuntu/Dockerfile.dev +++ /dev/null @@ -1,96 +0,0 @@ -# Copyright 2024 RustFS Team -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Ubuntu-based development environment -# Provides full development toolchain for building RustFS from source -FROM ubuntu:22.04 - -ENV LANG=C.UTF-8 -ENV DEBIAN_FRONTEND=noninteractive - -# Use faster mirrors for better build performance -RUN sed -i s@http://.*archive.ubuntu.com@http://repo.huaweicloud.com@g /etc/apt/sources.list - -# Install development dependencies -RUN apt-get clean && apt-get update && apt-get install -y \ - wget \ - git \ - curl \ - unzip \ - gcc \ - pkg-config \ - libssl-dev \ - lld \ - libdbus-1-dev \ - libwayland-dev \ - libwebkit2gtk-4.1-dev \ - libxdo-dev \ - ca-certificates \ - bash \ - vim \ - nano \ - htop \ - tree \ - && rm -rf /var/lib/apt/lists/* - -# Install protoc -RUN wget https://github.com/protocolbuffers/protobuf/releases/download/v31.1/protoc-31.1-linux-x86_64.zip \ - && unzip protoc-31.1-linux-x86_64.zip -d protoc3 \ - && mv protoc3/bin/* /usr/local/bin/ && chmod +x /usr/local/bin/protoc \ - && mv protoc3/include/* /usr/local/include/ && rm -rf protoc-31.1-linux-x86_64.zip protoc3 - -# Install flatc -RUN wget https://github.com/google/flatbuffers/releases/download/v25.2.10/Linux.flatc.binary.g++-13.zip \ - && unzip Linux.flatc.binary.g++-13.zip \ - && mv flatc /usr/local/bin/ && chmod +x /usr/local/bin/flatc && rm -rf Linux.flatc.binary.g++-13.zip - -# Install rust for development -RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y -ENV PATH="/root/.cargo/bin:${PATH}" - -# Install additional Rust tools for development -RUN /root/.cargo/bin/cargo install \ - cargo-watch \ - cargo-nextest \ - cargo-audit \ - cargo-outdated - -# Copy cargo config for Chinese users -COPY .docker/cargo.config.toml /root/.cargo/config.toml - -# Create development user -RUN groupadd -g 1000 rustfs && \ - useradd -d /app -g rustfs -u 1000 -s /bin/bash rustfs - -WORKDIR /app - -# Create data directories for testing -RUN mkdir -p /data && chown -R rustfs:rustfs /data /app - -# Environment variables for development -ENV RUSTFS_ACCESS_KEY=devadmin \ - RUSTFS_SECRET_KEY=devadmin \ - RUSTFS_ADDRESS=":9000" \ - RUSTFS_CONSOLE_ENABLE=true \ - RUSTFS_VOLUMES=/data \ - RUST_LOG=debug \ - RUST_BACKTRACE=1 - -EXPOSE 9000 - -# Development mode: keep container alive for interactive development -CMD echo "RustFS Development Environment" && \ - echo "Source code should be mounted at /app" && \ - echo "Use 'cargo build' to build, 'cargo run' to run" && \ - exec bash -c "while true; do sleep 1; done" diff --git a/.docker/ubuntu/Dockerfile.prebuild b/.docker/ubuntu/Dockerfile.prebuild deleted file mode 100644 index 68fa1ed9..00000000 --- a/.docker/ubuntu/Dockerfile.prebuild +++ /dev/null @@ -1,154 +0,0 @@ -# Copyright 2024 RustFS Team -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Multi-purpose Ubuntu-based Dockerfile -# Can be used as development environment or ubuntu runtime variant -FROM ubuntu:22.04 - -# Build arguments for dynamic artifact download (when used as runtime) -ARG VERSION="" -ARG BUILD_TYPE="release" -ARG TARGETARCH - -ENV LANG=C.UTF-8 -ENV DEBIAN_FRONTEND=noninteractive - -# Use faster mirrors for better build performance -RUN sed -i s@http://.*archive.ubuntu.com@http://repo.huaweicloud.com@g /etc/apt/sources.list - -# Install dependencies -RUN apt-get clean && apt-get update && apt-get install -y \ - wget \ - git \ - curl \ - unzip \ - gcc \ - pkg-config \ - libssl-dev \ - lld \ - libdbus-1-dev \ - libwayland-dev \ - libwebkit2gtk-4.1-dev \ - libxdo-dev \ - ca-certificates \ - bash \ - && rm -rf /var/lib/apt/lists/* - -# Install protoc -RUN wget https://github.com/protocolbuffers/protobuf/releases/download/v31.1/protoc-31.1-linux-x86_64.zip \ - && unzip protoc-31.1-linux-x86_64.zip -d protoc3 \ - && mv protoc3/bin/* /usr/local/bin/ && chmod +x /usr/local/bin/protoc \ - && mv protoc3/include/* /usr/local/include/ && rm -rf protoc-31.1-linux-x86_64.zip protoc3 - -# Install flatc -RUN wget https://github.com/google/flatbuffers/releases/download/v25.2.10/Linux.flatc.binary.g++-13.zip \ - && unzip Linux.flatc.binary.g++-13.zip \ - && mv flatc /usr/local/bin/ && chmod +x /usr/local/bin/flatc && rm -rf Linux.flatc.binary.g++-13.zip - -# Install rust (for development use) -RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y -ENV PATH="/root/.cargo/bin:${PATH}" - -# Copy cargo config for Chinese users -COPY .docker/cargo.config.toml /root/.cargo/config.toml - -# Download pre-built binary (VERSION is required) -RUN if [ -z "$VERSION" ]; then \ - echo "❌ ERROR: VERSION build argument is required"; \ - echo "Please provide VERSION (e.g., main-latest, latest, v1.0.0, dev-abc123)"; \ - exit 1; \ - fi; \ - \ - # Map TARGETARCH to our naming convention - case "${TARGETARCH}" in \ - amd64) ARCH="x86_64" ;; \ - arm64) ARCH="aarch64" ;; \ - *) echo "❌ Unsupported architecture: ${TARGETARCH}" && exit 1 ;; \ - esac; \ - \ - # Handle VERSION with different patterns - if [[ "$VERSION" == "main-latest" ]]; then \ - DOWNLOAD_PATH="artifacts/rustfs/dev"; \ - FILENAME="rustfs-linux-${ARCH}-main-latest.zip"; \ - elif [[ "$VERSION" == "latest" ]]; then \ - DOWNLOAD_PATH="artifacts/rustfs/release"; \ - FILENAME="rustfs-linux-${ARCH}-latest.zip"; \ - elif [[ "$VERSION" == dev-* ]]; then \ - DOWNLOAD_PATH="artifacts/rustfs/dev"; \ - FILENAME="rustfs-linux-${ARCH}-dev-${VERSION#dev-}.zip"; \ - else \ - DOWNLOAD_PATH="artifacts/rustfs/release"; \ - FILENAME="rustfs-linux-${ARCH}-v${VERSION}.zip"; \ - fi; \ - \ - # Download the binary - DOWNLOAD_URL="https://dl.rustfs.com/${DOWNLOAD_PATH}/${FILENAME}"; \ - echo "🔽 Downloading RustFS binary from: ${DOWNLOAD_URL}"; \ - \ - # Download with clear error handling - if ! curl -fsSL --connect-timeout 30 --max-time 120 -o /tmp/rustfs.zip "${DOWNLOAD_URL}"; then \ - echo "❌ Failed to download binary from: ${DOWNLOAD_URL}"; \ - echo "💡 Please ensure the binary exists or trigger a build first"; \ - echo "💡 Available options:"; \ - echo " - For main-latest: Push to main branch or run build workflow"; \ - echo " - For latest: Create a release tag"; \ - echo " - For dev-xxx: Run build workflow with specific commit"; \ - exit 1; \ - fi; \ - \ - # Extract binary - if ! unzip -o /tmp/rustfs.zip -d /tmp >/dev/null 2>&1; then \ - echo "❌ Failed to extract downloaded binary"; \ - exit 1; \ - fi; \ - \ - # Install binary - mv /tmp/rustfs /usr/local/bin/rustfs; \ - chmod +x /usr/local/bin/rustfs; \ - rm -rf /tmp/*; \ - \ - echo "✅ Successfully downloaded and installed RustFS binary (${VERSION})" - -# Create rustfs user for security -RUN groupadd -g 1000 rustfs && \ - useradd -d /app -g rustfs -u 1000 -s /bin/bash rustfs - -WORKDIR /app - -# Create data directories -RUN mkdir -p /data && chown -R rustfs:rustfs /data /app - -# Environment variables -ENV RUSTFS_ACCESS_KEY=rustfsadmin \ - RUSTFS_SECRET_KEY=rustfsadmin \ - RUSTFS_ADDRESS=":9000" \ - RUSTFS_CONSOLE_ENABLE=true \ - RUSTFS_VOLUMES=/data \ - RUST_LOG=warn - -EXPOSE 9000 - -# Health check (only if rustfs binary exists) -HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \ - CMD if [ -f /usr/local/bin/rustfs ]; then wget --no-verbose --tries=1 --spider http://localhost:9000/health || exit 1; else exit 0; fi - -# Default command: if rustfs binary exists, run it; otherwise, keep container alive for development -CMD if [ -f /usr/local/bin/rustfs ]; then \ - echo "Starting RustFS server..."; \ - exec /usr/local/bin/rustfs; \ - else \ - echo "Running in development mode..."; \ - echo "RustFS source code should be mounted at /app"; \ - exec bash -c "while true; do sleep 1; done"; \ - fi diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 231fbe63..3f71b70d 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -146,8 +146,6 @@ jobs: if: needs.build-check.outputs.should_build == 'true' runs-on: ${{ matrix.os }} timeout-minutes: 60 - env: - RUSTFLAGS: ${{ matrix.cross == 'false' && '-C target-cpu=native' || '' }} strategy: fail-fast: false matrix: @@ -155,29 +153,23 @@ jobs: # Linux builds - os: ubuntu-latest target: x86_64-unknown-linux-musl - cross: false platform: linux - os: ubuntu-latest target: aarch64-unknown-linux-musl - cross: true platform: linux # macOS builds - os: macos-latest target: aarch64-apple-darwin - cross: false platform: macos - os: macos-latest target: x86_64-apple-darwin - cross: false platform: macos # # Windows builds (temporarily disabled) # - os: windows-latest # target: x86_64-pc-windows-msvc - # cross: false # platform: windows # - os: windows-latest # target: aarch64-pc-windows-msvc - # cross: true # platform: windows steps: - name: Checkout repository @@ -193,50 +185,11 @@ jobs: cache-shared-key: build-${{ matrix.target }}-${{ hashFiles('**/Cargo.lock') }} github-token: ${{ secrets.GITHUB_TOKEN }} cache-save-if: ${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/') }} - install-cross-tools: ${{ matrix.cross }} - - name: Download static console assets + - name: Build RustFS using build-rustfs.sh run: | - mkdir -p ./rustfs/static - if [[ "${{ matrix.platform }}" == "windows" ]]; then - curl.exe -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" -o console.zip --retry 3 --retry-delay 5 --max-time 300 - if [[ $? -eq 0 ]]; then - unzip -o console.zip -d ./rustfs/static - rm console.zip - else - echo "Warning: Failed to download console assets, continuing without them" - echo "// Static assets not available" > ./rustfs/static/empty.txt - fi - else - chmod +w ./rustfs/static/LICENSE || true - curl -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" \ - -o console.zip --retry 3 --retry-delay 5 --max-time 300 - if [[ $? -eq 0 ]]; then - unzip -o console.zip -d ./rustfs/static - rm console.zip - else - echo "Warning: Failed to download console assets, continuing without them" - echo "// Static assets not available" > ./rustfs/static/empty.txt - fi - fi - - - name: Build RustFS - run: | - # Force rebuild by touching build.rs - touch rustfs/build.rs - - if [[ "${{ matrix.cross }}" == "true" ]]; then - if [[ "${{ matrix.platform }}" == "windows" ]]; then - # Use cross for Windows ARM64 - cargo install cross --git https://github.com/cross-rs/cross - cross build --release --target ${{ matrix.target }} -p rustfs --bins - else - # Use zigbuild for Linux ARM64 - cargo zigbuild --release --target ${{ matrix.target }} -p rustfs --bins - fi - else - cargo build --release --target ${{ matrix.target }} -p rustfs --bins - fi + # Use unified build script for consistent builds + ./build-rustfs.sh --platform ${{ matrix.target }} - name: Create release package id: package @@ -283,7 +236,8 @@ jobs: fi fi - cd target/${{ matrix.target }}/release + # build-rustfs.sh outputs to target/release/${platform}/rustfs + cd target/release/${{ matrix.target }} zip "../../../${PACKAGE_NAME}.zip" rustfs cd ../../.. diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml index 10080c40..9e8263b6 100644 --- a/.github/workflows/docker.yml +++ b/.github/workflows/docker.yml @@ -223,17 +223,11 @@ jobs: - name: production dockerfile: Dockerfile platforms: linux/amd64,linux/arm64 - - name: alpine - dockerfile: .docker/alpine/Dockerfile.prebuild + - name: source + dockerfile: Dockerfile.source platforms: linux/amd64,linux/arm64 - - name: alpine-source - dockerfile: .docker/alpine/Dockerfile.source - platforms: linux/amd64,linux/arm64 - - name: ubuntu - dockerfile: .docker/ubuntu/Dockerfile.prebuild - platforms: linux/amd64,linux/arm64 - - name: ubuntu-source - dockerfile: .docker/ubuntu/Dockerfile.source + - name: dev + dockerfile: Dockerfile.source platforms: linux/amd64,linux/arm64 steps: - name: Checkout repository diff --git a/.github/workflows/performance.yml b/.github/workflows/performance.yml index c584e4aa..0259fa65 100644 --- a/.github/workflows/performance.yml +++ b/.github/workflows/performance.yml @@ -18,10 +18,10 @@ on: push: branches: [main] paths: - - '**/*.rs' - - '**/Cargo.toml' - - '**/Cargo.lock' - - '.github/workflows/performance.yml' + - "**/*.rs" + - "**/Cargo.toml" + - "**/Cargo.lock" + - ".github/workflows/performance.yml" workflow_dispatch: inputs: profile_duration: @@ -73,12 +73,11 @@ jobs: echo "RUSTFS_VOLUMES=./target/volume/test{0...4}" >> $GITHUB_ENV echo "RUST_LOG=rustfs=info,ecstore=info,s3s=info,iam=info,rustfs-obs=info" >> $GITHUB_ENV - - name: Download static files + - name: Verify console static assets run: | - curl -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" \ - -o tempfile.zip --retry 3 --retry-delay 5 - unzip -o tempfile.zip -d ./rustfs/static - rm tempfile.zip + # Console static assets are already embedded in the repository + echo "Console static assets size: $(du -sh rustfs/static/)" + echo "Console static assets are embedded via rust-embed, no external download needed" - name: Build with profiling optimizations run: | diff --git a/Dockerfile b/Dockerfile index 8eb56517..10167a0e 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,129 +1,95 @@ -# Copyright 2024 RustFS Team -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. +# Multi-stage build for RustFS production image +FROM alpine:latest AS build -# Multi-stage Alpine build for minimal runtime image -FROM rust:1.85-alpine AS builder - -# Build arguments for dynamic artifact download -ARG VERSION="" -ARG BUILD_TYPE="release" +# Build arguments ARG TARGETARCH +ARG RELEASE=latest +ARG CHANNEL=release -# Install build dependencies -RUN apk add --no-cache \ - musl-dev \ - pkgconfig \ - openssl-dev \ - openssl-libs-static \ - curl \ - unzip \ - bash \ - wget \ - ca-certificates - -# Install protoc -RUN wget https://github.com/protocolbuffers/protobuf/releases/download/v31.1/protoc-31.1-linux-x86_64.zip \ - && unzip protoc-31.1-linux-x86_64.zip -d protoc3 \ - && mv protoc3/bin/* /usr/local/bin/ && chmod +x /usr/local/bin/protoc \ - && mv protoc3/include/* /usr/local/include/ && rm -rf protoc-31.1-linux-x86_64.zip protoc3 - -# Install flatc -RUN wget https://github.com/google/flatbuffers/releases/download/v25.2.10/Linux.flatc.binary.g++-13.zip \ - && unzip Linux.flatc.binary.g++-13.zip \ - && mv flatc /usr/local/bin/ && chmod +x /usr/local/bin/flatc \ - && rm -rf Linux.flatc.binary.g++-13.zip - -# Download pre-built binary (VERSION is required) -RUN if [ -z "$VERSION" ]; then \ - echo "❌ ERROR: VERSION build argument is required"; \ - echo "Please provide VERSION (e.g., main-latest, latest, v1.0.0, dev-abc123)"; \ - exit 1; \ - fi; \ - \ - # Map TARGETARCH to our naming convention - case "${TARGETARCH}" in \ - amd64) ARCH="x86_64" ;; \ - arm64) ARCH="aarch64" ;; \ - *) echo "❌ Unsupported architecture: ${TARGETARCH}" && exit 1 ;; \ - esac; \ - \ - # Handle VERSION with different patterns - if [[ "$VERSION" == "main-latest" ]]; then \ - DOWNLOAD_PATH="artifacts/rustfs/dev"; \ - FILENAME="rustfs-linux-${ARCH}-main-latest.zip"; \ - elif [[ "$VERSION" == "latest" ]]; then \ - DOWNLOAD_PATH="artifacts/rustfs/release"; \ - FILENAME="rustfs-linux-${ARCH}-latest.zip"; \ - elif [[ "$VERSION" == dev-* ]]; then \ - DOWNLOAD_PATH="artifacts/rustfs/dev"; \ - FILENAME="rustfs-linux-${ARCH}-dev-${VERSION#dev-}.zip"; \ - else \ - DOWNLOAD_PATH="artifacts/rustfs/release"; \ - FILENAME="rustfs-linux-${ARCH}-v${VERSION}.zip"; \ - fi; \ - \ - # Download the binary - DOWNLOAD_URL="https://dl.rustfs.com/${DOWNLOAD_PATH}/${FILENAME}"; \ - echo "🔽 Downloading RustFS binary from: ${DOWNLOAD_URL}"; \ - \ - # Download with clear error handling - if ! curl -fsSL --connect-timeout 30 --max-time 120 -o /tmp/rustfs.zip "${DOWNLOAD_URL}"; then \ - echo "❌ Failed to download binary from: ${DOWNLOAD_URL}"; \ - echo "💡 Please ensure the binary exists or trigger a build first"; \ - echo "💡 Available options:"; \ - echo " - For main-latest: Push to main branch or run build workflow"; \ - echo " - For latest: Create a release tag"; \ - echo " - For dev-xxx: Run build workflow with specific commit"; \ - exit 1; \ - fi; \ - \ - # Extract binary - if ! unzip -o /tmp/rustfs.zip -d /tmp >/dev/null 2>&1; then \ - echo "❌ Failed to extract downloaded binary"; \ - exit 1; \ - fi; \ - \ - # Install binary - mv /tmp/rustfs /usr/local/bin/rustfs; \ - chmod +x /usr/local/bin/rustfs; \ - rm -rf /tmp/*; \ - \ - echo "✅ Successfully downloaded and installed RustFS binary (${VERSION})" - -# Final Alpine runtime image -FROM alpine:3.18 - +# Install dependencies for downloading and verifying binaries RUN apk add --no-cache \ ca-certificates \ + curl \ + bash \ + wget \ + unzip \ + jq + +# Create build directory +WORKDIR /build + +# Map TARGETARCH to architecture format used in builds +RUN case "${TARGETARCH}" in \ + "amd64") ARCH="x86_64" ;; \ + "arm64") ARCH="aarch64" ;; \ + *) echo "Unsupported architecture: ${TARGETARCH}" && exit 1 ;; \ + esac && \ + echo "ARCH=${ARCH}" > /build/arch.env + +# Download rustfs binary from dl.rustfs.com +RUN . /build/arch.env && \ + BASE_URL="https://dl.rustfs.com/artifacts/rustfs" && \ + PLATFORM="linux" && \ + if [ "${RELEASE}" = "latest" ]; then \ + # Download latest version from specified channel \ + if [ "${CHANNEL}" = "dev" ]; then \ + PACKAGE_NAME="rustfs-${PLATFORM}-${ARCH}-dev-latest.zip"; \ + DOWNLOAD_URL="${BASE_URL}/dev/${PACKAGE_NAME}"; \ + echo "📥 Downloading latest dev build: ${PACKAGE_NAME}"; \ + else \ + PACKAGE_NAME="rustfs-${PLATFORM}-${ARCH}-latest.zip"; \ + DOWNLOAD_URL="${BASE_URL}/release/${PACKAGE_NAME}"; \ + echo "📥 Downloading latest release build: ${PACKAGE_NAME}"; \ + fi; \ + else \ + # Download specific version (always from release channel) \ + PACKAGE_NAME="rustfs-${PLATFORM}-${ARCH}-v${RELEASE}.zip"; \ + DOWNLOAD_URL="${BASE_URL}/release/${PACKAGE_NAME}"; \ + echo "📥 Downloading specific version: ${PACKAGE_NAME}"; \ + fi && \ + echo "🔗 Download URL: ${DOWNLOAD_URL}" && \ + curl -f -L "${DOWNLOAD_URL}" -o /build/rustfs.zip && \ + if [ ! -f /build/rustfs.zip ] || [ ! -s /build/rustfs.zip ]; then \ + echo "❌ Failed to download binary package"; \ + echo "💡 Make sure the package ${PACKAGE_NAME} exists"; \ + echo "🔗 Check: ${DOWNLOAD_URL}"; \ + exit 1; \ + fi && \ + unzip /build/rustfs.zip -d /build && \ + chmod +x /build/rustfs && \ + rm /build/rustfs.zip && \ + echo "✅ Successfully downloaded and extracted rustfs binary" + +# Runtime stage +FROM alpine:latest + +# Set build arguments and labels +ARG RELEASE=latest +ARG CHANNEL=release +ARG BUILD_DATE +ARG VCS_REF + +LABEL name="RustFS" \ + vendor="RustFS Team" \ + maintainer="RustFS Team " \ + version="${RELEASE}" \ + release="${RELEASE}" \ + channel="${CHANNEL}" \ + build-date="${BUILD_DATE}" \ + vcs-ref="${VCS_REF}" \ + summary="RustFS is a high-performance distributed object storage system written in Rust, compatible with S3 API." \ + description="RustFS is a high-performance distributed object storage software built using Rust. It supports erasure coding storage, multi-tenant management, observability, and other enterprise-level features." \ + url="https://rustfs.com" \ + license="Apache-2.0" + +# Install runtime dependencies +RUN apk add --no-cache \ + ca-certificates \ + curl \ tzdata \ - bash - -# Create rustfs user for security -RUN addgroup -g 1000 rustfs && \ - adduser -D -u 1000 -G rustfs rustfs - -WORKDIR /app - -# Copy binary from builder -COPY --from=builder /usr/local/bin/rustfs /app/rustfs -RUN chmod +x /app/rustfs && chown rustfs:rustfs /app/rustfs - -# Create data directories -RUN mkdir -p /data && chown -R rustfs:rustfs /data /app - -# Switch to non-root user -USER rustfs + bash \ + && addgroup -g 1000 rustfs \ + && adduser -u 1000 -G rustfs -s /bin/sh -D rustfs # Environment variables ENV RUSTFS_ACCESS_KEY=rustfsadmin \ @@ -133,10 +99,34 @@ ENV RUSTFS_ACCESS_KEY=rustfsadmin \ RUSTFS_VOLUMES=/data \ RUST_LOG=warn +# Set permissions for /usr/bin (similar to MinIO's approach) +RUN chmod -R 755 /usr/bin + +# Copy CA certificates and binaries from build stage +COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ +COPY --from=build /build/rustfs /usr/bin/ + +# Set executable permissions +RUN chmod +x /usr/bin/rustfs + +# Create data directory +RUN mkdir -p /data /config && chown -R rustfs:rustfs /data /config + +# Switch to non-root user +USER rustfs + +# Set working directory +WORKDIR /data + +# Expose port EXPOSE 9000 # Health check -HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \ - CMD wget --no-verbose --tries=1 --spider http://localhost:9000/health || exit 1 +HEALTHCHECK --interval=30s --timeout=10s --retries=3 --start-period=40s \ + CMD curl -f http://localhost:9000/health || exit 1 -CMD ["/app/rustfs"] +# Volume for data +VOLUME ["/data"] + +# Set entrypoint +ENTRYPOINT ["/usr/bin/rustfs"] diff --git a/Dockerfile.obs b/Dockerfile.obs deleted file mode 100644 index 552fe168..00000000 --- a/Dockerfile.obs +++ /dev/null @@ -1,61 +0,0 @@ -# Copyright 2024 RustFS Team -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Dockerfile for RustFS with observability features -FROM ubuntu:22.04 - -# Avoid interactive prompts during build -ENV DEBIAN_FRONTEND=noninteractive - -# Install runtime dependencies -RUN apt-get update && apt-get install -y \ - ca-certificates \ - wget \ - curl \ - && rm -rf /var/lib/apt/lists/* - -WORKDIR /app - -# Create rustfs user for security -RUN groupadd -g 1000 rustfs && \ - useradd -d /app -g rustfs -u 1000 -s /bin/bash rustfs - -# Create data directories matching RUSTFS_VOLUMES pattern -RUN mkdir -p /data/rustfs{0,1,2,3} && \ - chown -R rustfs:rustfs /data /app - -# Copy RustFS binary (expects it to be built with observability features) -# Note: This assumes the binary is built locally with observability features enabled -COPY ./target/x86_64-unknown-linux-gnu/release/rustfs /app/rustfs -RUN chmod +x /app/rustfs && chown rustfs:rustfs /app/rustfs - -# Switch to non-root user -USER rustfs - -# Environment variables for observability -ENV RUSTFS_ACCESS_KEY=rustfsadmin \ - RUSTFS_SECRET_KEY=rustfsadmin \ - RUSTFS_ADDRESS=":9000" \ - RUSTFS_CONSOLE_ENABLE=true \ - RUSTFS_VOLUMES=/data/rustfs0,/data/rustfs1,/data/rustfs2,/data/rustfs3 \ - RUSTFS_OBS_ENDPOINT=http://otel-collector:4317 \ - RUST_LOG=info - -EXPOSE 9000 - -# Health check -HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \ - CMD wget --no-verbose --tries=1 --spider http://localhost:9000/health || exit 1 - -CMD ["/app/rustfs"] diff --git a/.docker/ubuntu/Dockerfile.source b/Dockerfile.source similarity index 98% rename from .docker/ubuntu/Dockerfile.source rename to Dockerfile.source index 1bcc2f4c..ab1654bd 100644 --- a/.docker/ubuntu/Dockerfile.source +++ b/Dockerfile.source @@ -63,7 +63,7 @@ ENV CXX_aarch64_unknown_linux_gnu=aarch64-linux-gnu-g++ WORKDIR /usr/src/rustfs # Copy cargo configuration for optimized builds -COPY .docker/cargo.config.toml ./.cargo/config.toml +COPY cargo.config.toml ./.cargo/config.toml # Copy Cargo files for dependency caching COPY Cargo.toml Cargo.lock ./ @@ -151,5 +151,8 @@ ENV RUSTFS_ACCESS_KEY=rustfsadmin \ HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \ CMD wget --no-verbose --tries=1 --spider http://localhost:9000/health || exit 1 +# Volume for data +VOLUME ["/data"] + # Set default command CMD ["/app/rustfs"] diff --git a/Makefile b/Makefile index 74cdd215..858227a1 100644 --- a/Makefile +++ b/Makefile @@ -5,7 +5,9 @@ DOCKER_CLI ?= docker IMAGE_NAME ?= rustfs:v1.0.0 CONTAINER_NAME ?= rustfs-dev -DOCKERFILE_PATH = $(shell pwd)/.docker +# Docker build configurations +DOCKERFILE_PRODUCTION = Dockerfile +DOCKERFILE_SOURCE = Dockerfile.source # Code quality and formatting targets .PHONY: fmt @@ -46,7 +48,7 @@ setup-hooks: .PHONY: init-devenv init-devenv: - $(DOCKER_CLI) build -t $(IMAGE_NAME) -f $(DOCKERFILE_PATH)/Dockerfile.devenv . + $(DOCKER_CLI) build -t $(IMAGE_NAME) -f Dockerfile.source . $(DOCKER_CLI) stop $(CONTAINER_NAME) $(DOCKER_CLI) rm $(CONTAINER_NAME) $(DOCKER_CLI) run -d --name $(CONTAINER_NAME) -p 9010:9010 -p 9000:9000 -v $(shell pwd):/root/s3-rustfs -it $(IMAGE_NAME) @@ -67,86 +69,89 @@ e2e-server: probe-e2e: sh $(shell pwd)/scripts/probe.sh -# make BUILD_OS=ubuntu22.04 build -# in target/ubuntu22.04/release/rustfs - -# make BUILD_OS=rockylinux9.3 build -# in target/rockylinux9.3/release/rustfs -BUILD_OS ?= rockylinux9.3 +# Native build using build-rustfs.sh script .PHONY: build -build: ROCKYLINUX_BUILD_IMAGE_NAME = rustfs-$(BUILD_OS):v1 -build: ROCKYLINUX_BUILD_CONTAINER_NAME = rustfs-$(BUILD_OS)-build -build: BUILD_CMD = /root/.cargo/bin/cargo build --release --bin rustfs --target-dir /root/s3-rustfs/target/$(BUILD_OS) build: - $(DOCKER_CLI) build -t $(ROCKYLINUX_BUILD_IMAGE_NAME) -f $(DOCKERFILE_PATH)/Dockerfile.$(BUILD_OS) . - $(DOCKER_CLI) run --rm --name $(ROCKYLINUX_BUILD_CONTAINER_NAME) -v $(shell pwd):/root/s3-rustfs -it $(ROCKYLINUX_BUILD_IMAGE_NAME) $(BUILD_CMD) + @echo "🔨 Building RustFS using build-rustfs.sh script..." + ./build-rustfs.sh + +.PHONY: build-dev +build-dev: + @echo "🔨 Building RustFS in development mode..." + ./build-rustfs.sh --dev + + + +# Docker-based build (alternative approach) +# Usage: make BUILD_OS=ubuntu22.04 build-docker +# Output: target/ubuntu22.04/release/rustfs +BUILD_OS ?= rockylinux9.3 +.PHONY: build-docker +build-docker: SOURCE_BUILD_IMAGE_NAME = rustfs-$(BUILD_OS):v1 +build-docker: SOURCE_BUILD_CONTAINER_NAME = rustfs-$(BUILD_OS)-build +build-docker: BUILD_CMD = /root/.cargo/bin/cargo build --release --bin rustfs --target-dir /root/s3-rustfs/target/$(BUILD_OS) +build-docker: + @echo "🐳 Building RustFS using Docker ($(BUILD_OS))..." + $(DOCKER_CLI) build -t $(SOURCE_BUILD_IMAGE_NAME) -f $(DOCKERFILE_SOURCE) . + $(DOCKER_CLI) run --rm --name $(SOURCE_BUILD_CONTAINER_NAME) -v $(shell pwd):/root/s3-rustfs -it $(SOURCE_BUILD_IMAGE_NAME) $(BUILD_CMD) .PHONY: build-musl build-musl: @echo "🔨 Building rustfs for x86_64-unknown-linux-musl..." - cargo build --target x86_64-unknown-linux-musl --bin rustfs -r + @echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-buildx' instead" + ./build-rustfs.sh --platform x86_64-unknown-linux-musl .PHONY: build-gnu build-gnu: @echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..." - cargo build --target x86_64-unknown-linux-gnu --bin rustfs -r + @echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-buildx' instead" + ./build-rustfs.sh --platform x86_64-unknown-linux-gnu .PHONY: deploy-dev deploy-dev: build-musl @echo "🚀 Deploying to dev server: $${IP}" ./scripts/dev_deploy.sh $${IP} -# Multi-architecture Docker build targets -.PHONY: docker-build-multiarch -docker-build-multiarch: - @echo "🏗️ Building multi-architecture Docker images..." - ./scripts/build-docker-multiarch.sh +# Multi-architecture Docker build targets (NEW: using docker-buildx.sh) +.PHONY: docker-buildx +docker-buildx: + @echo "🏗️ Building multi-architecture Docker images with buildx..." + ./docker-buildx.sh -.PHONY: docker-build-multiarch-push -docker-build-multiarch-push: - @echo "🚀 Building and pushing multi-architecture Docker images..." - ./scripts/build-docker-multiarch.sh --push +.PHONY: docker-buildx-push +docker-buildx-push: + @echo "🚀 Building and pushing multi-architecture Docker images with buildx..." + ./docker-buildx.sh --push -.PHONY: docker-build-multiarch-version -docker-build-multiarch-version: +.PHONY: docker-buildx-version +docker-buildx-version: @if [ -z "$(VERSION)" ]; then \ - echo "❌ 错误: 请指定版本, 例如: make docker-build-multiarch-version VERSION=v1.0.0"; \ + echo "❌ 错误: 请指定版本, 例如: make docker-buildx-version VERSION=v1.0.0"; \ exit 1; \ fi @echo "🏗️ Building multi-architecture Docker images (version: $(VERSION))..." - ./scripts/build-docker-multiarch.sh --version $(VERSION) + ./docker-buildx.sh --release $(VERSION) -.PHONY: docker-push-multiarch-version -docker-push-multiarch-version: +.PHONY: docker-buildx-push-version +docker-buildx-push-version: @if [ -z "$(VERSION)" ]; then \ - echo "❌ 错误: 请指定版本, 例如: make docker-push-multiarch-version VERSION=v1.0.0"; \ + echo "❌ 错误: 请指定版本, 例如: make docker-buildx-push-version VERSION=v1.0.0"; \ exit 1; \ fi @echo "🚀 Building and pushing multi-architecture Docker images (version: $(VERSION))..." - ./scripts/build-docker-multiarch.sh --version $(VERSION) --push + ./docker-buildx.sh --release $(VERSION) --push -.PHONY: docker-build-ubuntu -docker-build-ubuntu: - @echo "🏗️ Building multi-architecture Ubuntu Docker images..." - ./scripts/build-docker-multiarch.sh --type ubuntu -.PHONY: docker-build-rockylinux -docker-build-rockylinux: - @echo "🏗️ Building multi-architecture RockyLinux Docker images..." - ./scripts/build-docker-multiarch.sh --type rockylinux -.PHONY: docker-build-devenv -docker-build-devenv: - @echo "🏗️ Building multi-architecture development environment Docker images..." - ./scripts/build-docker-multiarch.sh --type devenv +.PHONY: docker-build-production +docker-build-production: + @echo "🏗️ Building production Docker image..." + $(DOCKER_CLI) build -f $(DOCKERFILE_PRODUCTION) -t rustfs:latest . -.PHONY: docker-build-all-types -docker-build-all-types: - @echo "🏗️ Building all multi-architecture Docker image types..." - ./scripts/build-docker-multiarch.sh --type production - ./scripts/build-docker-multiarch.sh --type ubuntu - ./scripts/build-docker-multiarch.sh --type rockylinux - ./scripts/build-docker-multiarch.sh --type devenv +.PHONY: docker-build-source +docker-build-source: + @echo "🏗️ Building source Docker image..." + $(DOCKER_CLI) build -f $(DOCKERFILE_SOURCE) -t rustfs:source . .PHONY: docker-inspect-multiarch docker-inspect-multiarch: @@ -160,41 +165,64 @@ docker-inspect-multiarch: .PHONY: build-cross-all build-cross-all: @echo "🔧 Building all target architectures..." - @if ! command -v cross &> /dev/null; then \ - echo "📦 Installing cross..."; \ - cargo install cross; \ - fi + @echo "💡 On macOS/Windows, use 'make docker-buildx' for reliable multi-arch builds" @echo "🔨 Generating protobuf code..." cargo run --bin gproto || true @echo "🔨 Building x86_64-unknown-linux-musl..." - cargo build --release --target x86_64-unknown-linux-musl --bin rustfs + ./build-rustfs.sh --platform x86_64-unknown-linux-musl @echo "🔨 Building aarch64-unknown-linux-gnu..." - cross build --release --target aarch64-unknown-linux-gnu --bin rustfs + ./build-rustfs.sh --platform aarch64-unknown-linux-gnu @echo "✅ All architectures built successfully!" +.PHONY: help-build +help-build: + @echo "🔨 RustFS 构建帮助：" + @echo "" + @echo "🚀 本地构建 (推荐使用):" + @echo " make build # 构建 RustFS 二进制文件 (默认包含 console)" + @echo " make build-dev # 开发模式构建" + @echo " make build-musl # 构建 musl 版本" + @echo " make build-gnu # 构建 GNU 版本" + @echo "" + @echo "🐳 Docker 构建:" + @echo " make build-docker # 使用 Docker 容器构建" + @echo " make build-docker BUILD_OS=ubuntu22.04 # 指定构建系统" + @echo "" + @echo "🏗️ 跨架构构建:" + @echo " make build-cross-all # 构建所有架构的二进制文件" + @echo "" + @echo "🔧 直接使用 build-rustfs.sh 脚本:" + @echo " ./build-rustfs.sh --help # 查看脚本帮助" + @echo " ./build-rustfs.sh --no-console # 构建时跳过 console 资源" + @echo " ./build-rustfs.sh --force-console-update # 强制更新 console 资源" + @echo " ./build-rustfs.sh --dev # 开发模式构建" + @echo " ./build-rustfs.sh --sign # 签名二进制文件" + @echo " ./build-rustfs.sh --platform x86_64-unknown-linux-musl # 指定目标平台" + @echo " ./build-rustfs.sh --skip-verification # 跳过二进制验证" + @echo "" + @echo "💡 build-rustfs.sh 脚本提供了更多选项、智能检测和二进制验证功能" + .PHONY: help-docker help-docker: @echo "🐳 Docker 多架构构建帮助：" @echo "" - @echo "基本构建:" - @echo " make docker-build-multiarch # 构建多架构镜像（不推送）" - @echo " make docker-build-multiarch-push # 构建并推送多架构镜像" + @echo "🚀 推荐使用 (新的 docker-buildx 方式):" + @echo " make docker-buildx # 构建多架构镜像（不推送）" + @echo " make docker-buildx-push # 构建并推送多架构镜像" + @echo " make docker-buildx-version VERSION=v1.0.0 # 构建指定版本" + @echo " make docker-buildx-push-version VERSION=v1.0.0 # 构建并推送指定版本" @echo "" - @echo "版本构建:" - @echo " make docker-build-multiarch-version VERSION=v1.0.0 # 构建指定版本" - @echo " make docker-push-multiarch-version VERSION=v1.0.0 # 构建并推送指定版本" + @echo "🏗️ 单架构构建:" + @echo " make docker-build-production # 构建生产环境镜像" + @echo " make docker-build-source # 构建源码构建镜像" @echo "" - @echo "镜像类型:" - @echo " make docker-build-ubuntu # 构建 Ubuntu 镜像" - @echo " make docker-build-rockylinux # 构建 RockyLinux 镜像" - @echo " make docker-build-devenv # 构建开发环境镜像" - @echo " make docker-build-all-types # 构建所有类型镜像" - @echo "" - @echo "辅助工具:" + @echo "🔧 辅助工具:" @echo " make build-cross-all # 构建所有架构的二进制文件" @echo " make docker-inspect-multiarch IMAGE=xxx # 检查镜像的架构支持" @echo "" - @echo "环境变量 (在推送时需要设置):" + @echo "📋 环境变量 (在推送时需要设置):" @echo " DOCKERHUB_USERNAME Docker Hub 用户名" @echo " DOCKERHUB_TOKEN Docker Hub 访问令牌" @echo " GITHUB_TOKEN GitHub 访问令牌" + @echo "" + @echo "💡 更多详情请参考项目根目录的 docker-buildx.sh 脚本" diff --git a/README.md b/README.md index a10051e7..e1165b3f 100644 --- a/README.md +++ b/README.md @@ -91,11 +91,42 @@ To get started with RustFS, follow these steps: docker run -d -p 9000:9000 -v /data:/data rustfs/rustfs:v1.0.0 ``` - > 💡 **Docker Build Options**: RustFS supports flexible Docker image versions. See our [Docker Build Guide](docs/DOCKER_BUILD_GUIDE.md) for advanced usage and manual build instructions. +3. **Build from Source (Option 3) - Advanced Users** -3. **Access the Console**: Open your web browser and navigate to `http://localhost:9000` to access the RustFS console, default username and password is `rustfsadmin` . -4. **Create a Bucket**: Use the console to create a new bucket for your objects. -5. **Upload Objects**: You can upload files directly through the console or use S3-compatible APIs to interact with your RustFS instance. + For developers who want to build RustFS Docker images from source with multi-architecture support: + + ```bash + # Build multi-architecture images locally + ./docker-buildx.sh --build-arg RELEASE=latest + + # Build and push to registry + ./docker-buildx.sh --push + + # Build specific version + ./docker-buildx.sh --release v1.0.0 --push + + # Build for custom registry + ./docker-buildx.sh --registry your-registry.com --namespace yourname --push + ``` + + The `docker-buildx.sh` script supports: + - **Multi-architecture builds**: `linux/amd64`, `linux/arm64` + - **Automatic version detection**: Uses git tags or commit hashes + - **Registry flexibility**: Supports Docker Hub, GitHub Container Registry, etc. + - **Build optimization**: Includes caching and parallel builds + + You can also use Make targets for convenience: + + ```bash + make docker-buildx # Build locally + make docker-buildx-push # Build and push + make docker-buildx-version VERSION=v1.0.0 # Build specific version + make help-docker # Show all Docker-related commands + ``` + +4. **Access the Console**: Open your web browser and navigate to `http://localhost:9000` to access the RustFS console, default username and password is `rustfsadmin` . +5. **Create a Bucket**: Use the console to create a new bucket for your objects. +6. **Upload Objects**: You can upload files directly through the console or use S3-compatible APIs to interact with your RustFS instance. ## Documentation diff --git a/README_ZH.md b/README_ZH.md index 89572793..34ee34e5 100644 --- a/README_ZH.md +++ b/README_ZH.md @@ -62,7 +62,7 @@ RustFS 是一个使用 Rust（全球最受欢迎的编程语言之一）构建 要开始使用 RustFS，请按照以下步骤操作： -1. **一键脚本快速启动 (方案一)** +1. **一键脚本快速启动 (方案一)** ```bash curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh @@ -74,7 +74,6 @@ RustFS 是一个使用 Rust（全球最受欢迎的编程语言之一）构建 docker run -d -p 9000:9000 -v /data:/data rustfs/rustfs ``` - 3. **访问控制台**：打开 Web 浏览器并导航到 `http://localhost:9000` 以访问 RustFS 控制台，默认的用户名和密码是 `rustfsadmin` 。 4. **创建存储桶**：使用控制台为您的对象创建新的存储桶。 5. **上传对象**：您可以直接通过控制台上传文件，或使用 S3 兼容的 API 与您的 RustFS 实例交互。 @@ -110,7 +109,7 @@ RustFS 是一个使用 Rust（全球最受欢迎的编程语言之一）构建 RustFS 是一个社区驱动的项目，我们感谢所有的贡献。查看[贡献者](https://github.com/rustfs/rustfs/graphs/contributors)页面，了解帮助 RustFS 变得更好的杰出人员。 - + ## 许可证 diff --git a/build-rustfs.sh b/build-rustfs.sh new file mode 100755 index 00000000..e907a062 --- /dev/null +++ b/build-rustfs.sh @@ -0,0 +1,488 @@ +#!/bin/bash + +# RustFS Binary Build Script +# This script compiles RustFS binaries for different platforms and architectures + +set -e + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +# Auto-detect current platform +detect_platform() { + local arch=$(uname -m) + local os=$(uname -s | tr '[:upper:]' '[:lower:]') + + case "$os" in + "linux") + case "$arch" in + "x86_64") + echo "x86_64-unknown-linux-musl" + ;; + "aarch64"|"arm64") + echo "aarch64-unknown-linux-musl" + ;; + "armv7l") + echo "armv7-unknown-linux-musleabihf" + ;; + *) + echo "unknown-platform" + ;; + esac + ;; + "darwin") + case "$arch" in + "x86_64") + echo "x86_64-apple-darwin" + ;; + "arm64"|"aarch64") + echo "aarch64-apple-darwin" + ;; + *) + echo "unknown-platform" + ;; + esac + ;; + *) + echo "unknown-platform" + ;; + esac +} + +# Default values +OUTPUT_DIR="target/release" +PLATFORM=$(detect_platform) # Auto-detect current platform +BINARY_NAME="rustfs" +BUILD_TYPE="release" +SIGN=false +WITH_CONSOLE=true +FORCE_CONSOLE_UPDATE=false +CONSOLE_VERSION="latest" +SKIP_VERIFICATION=false +CUSTOM_PLATFORM="" + +# Print usage +usage() { + echo "Usage: $0 [OPTIONS]" + echo "" + echo "Description:" + echo " Build RustFS binary for the current platform. Designed for CI/CD pipelines" + echo " where different runners build platform-specific binaries natively." + echo " Includes automatic verification to ensure the built binary is functional." + echo "" + echo "Options:" + echo " -o, --output-dir DIR Output directory (default: target/release)" + echo " -b, --binary-name NAME Binary name (default: rustfs)" + echo " -p, --platform TARGET Target platform (default: auto-detect)" + echo " --dev Build in dev mode" + echo " --sign Sign binaries after build" + echo " --with-console Download console static assets (default)" + echo " --no-console Skip console static assets" + echo " --force-console-update Force update console assets even if they exist" + echo " --console-version VERSION Console version to download (default: latest)" + echo " --skip-verification Skip binary verification after build" + echo " -h, --help Show this help message" + echo "" + echo "Examples:" + echo " $0 # Build for current platform (includes console assets)" + echo " $0 --dev # Development build" + echo " $0 --sign # Build and sign binary (release CI)" + echo " $0 --no-console # Build without console static assets" + echo " $0 --force-console-update # Force update console assets" + echo " $0 --platform x86_64-unknown-linux-musl # Build for specific platform" + echo " $0 --skip-verification # Skip binary verification (for cross-compilation)" + echo "" + echo "Detected platform: $(detect_platform)" + echo "CI Usage: Run this script on each platform's runner to build native binaries" +} + +# Print colored message +print_message() { + local color=$1 + local message=$2 + echo -e "${color}${message}${NC}" +} + +# Get version from git +get_version() { + if git describe --abbrev=0 --tags >/dev/null 2>&1; then + git describe --abbrev=0 --tags + else + git rev-parse --short HEAD + fi +} + +# Setup rust environment +setup_rust_environment() { + print_message $BLUE "🔧 Setting up Rust environment..." + + # Install required target for current platform + print_message $YELLOW "Installing target: $PLATFORM" + rustup target add "$PLATFORM" + + # Install required tools + if [ "$SIGN" = true ]; then + if ! command -v minisign &> /dev/null; then + print_message $YELLOW "Installing minisign for binary signing..." + cargo install minisign + fi + fi +} + +# Download console static assets +download_console_assets() { + local static_dir="rustfs/static" + local console_exists=false + + # Check if console assets already exist + if [ -d "$static_dir" ] && [ -f "$static_dir/index.html" ]; then + console_exists=true + local static_size=$(du -sh "$static_dir" 2>/dev/null | cut -f1 || echo "unknown") + print_message $YELLOW "Console static assets already exist ($static_size)" + fi + + # Determine if we need to download + local should_download=false + if [ "$WITH_CONSOLE" = true ]; then + if [ "$console_exists" = false ]; then + print_message $BLUE "🎨 Console assets not found, downloading..." + should_download=true + elif [ "$FORCE_CONSOLE_UPDATE" = true ]; then + print_message $BLUE "🎨 Force updating console assets..." + should_download=true + else + print_message $GREEN "✅ Console assets already available, skipping download" + fi + else + if [ "$console_exists" = true ]; then + print_message $GREEN "✅ Using existing console assets" + else + print_message $YELLOW "⚠️ Console assets not found. Use --download-console to download them." + fi + fi + + if [ "$should_download" = true ]; then + print_message $BLUE "📥 Downloading console static assets..." + + # Create static directory + mkdir -p "$static_dir" + + # Download from GitHub Releases (consistent with Docker build) + local download_url + if [ "$CONSOLE_VERSION" = "latest" ]; then + print_message $YELLOW "Getting latest console release info..." + # For now, use dl.rustfs.com as fallback until GitHub Releases includes console assets + download_url="https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" + else + download_url="https://dl.rustfs.com/artifacts/console/rustfs-console-${CONSOLE_VERSION}.zip" + fi + + print_message $YELLOW "Downloading from: $download_url" + + # Download with retries + local temp_file="console-assets-temp.zip" + local download_success=false + + for i in {1..3}; do + if curl -L "$download_url" -o "$temp_file" --retry 3 --retry-delay 5 --max-time 300; then + download_success=true + break + else + print_message $YELLOW "Download attempt $i failed, retrying..." + sleep 2 + fi + done + + if [ "$download_success" = true ]; then + # Verify the downloaded file + if [ -f "$temp_file" ] && [ -s "$temp_file" ]; then + print_message $BLUE "📦 Extracting console assets..." + + # Extract to static directory + if unzip -o "$temp_file" -d "$static_dir"; then + rm "$temp_file" + local final_size=$(du -sh "$static_dir" 2>/dev/null | cut -f1 || echo "unknown") + print_message $GREEN "✅ Console assets downloaded successfully ($final_size)" + else + print_message $RED "❌ Failed to extract console assets" + rm -f "$temp_file" + return 1 + fi + else + print_message $RED "❌ Downloaded file is empty or invalid" + rm -f "$temp_file" + return 1 + fi + else + print_message $RED "❌ Failed to download console assets after 3 attempts" + print_message $YELLOW "💡 Console assets are optional. Build will continue without them." + rm -f "$temp_file" + fi + fi +} + +# Verify binary functionality +verify_binary() { + local binary_path="$1" + + # Check if binary exists + if [ ! -f "$binary_path" ]; then + print_message $RED "❌ Binary file not found: $binary_path" + return 1 + fi + + # Check if binary is executable + if [ ! -x "$binary_path" ]; then + print_message $RED "❌ Binary is not executable: $binary_path" + return 1 + fi + + # Check basic functionality - try to run help command + print_message $YELLOW " Testing --help command..." + if ! "$binary_path" --help >/dev/null 2>&1; then + print_message $RED "❌ Binary failed to run --help command" + return 1 + fi + + # Check version command + print_message $YELLOW " Testing --version command..." + if ! "$binary_path" --version >/dev/null 2>&1; then + print_message $YELLOW "⚠️ Binary does not support --version command (this is optional)" + fi + + # Try to get some basic info about the binary + local file_info=$(file "$binary_path" 2>/dev/null || echo "unknown") + print_message $YELLOW " Binary info: $file_info" + + # Check if it's a valid ELF/Mach-O binary + if command -v readelf >/dev/null 2>&1; then + if readelf -h "$binary_path" >/dev/null 2>&1; then + print_message $YELLOW " ELF binary structure: valid" + fi + elif command -v otool >/dev/null 2>&1; then + if otool -h "$binary_path" >/dev/null 2>&1; then + print_message $YELLOW " Mach-O binary structure: valid" + fi + fi + + return 0 +} + +# Build binary for current platform +build_binary() { + local version=$(get_version) + local output_file="${OUTPUT_DIR}/${PLATFORM}/${BINARY_NAME}" + + print_message $BLUE "🏗️ Building for platform: $PLATFORM" + print_message $YELLOW " Version: $version" + print_message $YELLOW " Output: $output_file" + + # Create output directory + mkdir -p "${OUTPUT_DIR}/${PLATFORM}" + + # Build command - use cross for cross-compilation if needed + local build_cmd="cargo build" + local current_platform=$(detect_platform) + + # Check if we need cross-compilation + if [ "$PLATFORM" != "$current_platform" ]; then + # Check if cross is available + if command -v cross &> /dev/null; then + build_cmd="cross build" + print_message $YELLOW "🔄 Cross-compilation detected, using 'cross' tool" + else + print_message $YELLOW "⚠️ Cross-compilation detected but 'cross' tool not found" + print_message $YELLOW "📦 Installing cross tool..." + cargo install cross --git https://github.com/cross-rs/cross + build_cmd="cross build" + fi + fi + + if [ "$BUILD_TYPE" = "release" ]; then + build_cmd+=" --release" + fi + + build_cmd+=" --target $PLATFORM" + build_cmd+=" --bin $BINARY_NAME" + + print_message $BLUE "📦 Executing: $build_cmd" + + # Execute build + if eval $build_cmd; then + print_message $GREEN "✅ Successfully built for $PLATFORM" + + # Copy binary to output directory + cp "target/${PLATFORM}/${BUILD_TYPE}/${BINARY_NAME}" "$output_file" + + # Generate checksums + print_message $BLUE "🔐 Generating checksums..." + (cd "${OUTPUT_DIR}/${PLATFORM}" && sha256sum "${BINARY_NAME}" > "${BINARY_NAME}.sha256sum") + + # Verify binary functionality (if not skipped) + if [ "$SKIP_VERIFICATION" = false ]; then + print_message $BLUE "🔍 Verifying binary functionality..." + if verify_binary "$output_file"; then + print_message $GREEN "✅ Binary verification passed" + else + print_message $RED "❌ Binary verification failed" + return 1 + fi + else + print_message $YELLOW "⚠️ Binary verification skipped by user request" + fi + + # Sign binary if requested + if [ "$SIGN" = true ]; then + print_message $BLUE "✍️ Signing binary..." + (cd "${OUTPUT_DIR}/${PLATFORM}" && minisign -S -m "${BINARY_NAME}" -s ~/.minisign/minisign.key) + fi + + print_message $GREEN "✅ Build completed successfully" + else + print_message $RED "❌ Failed to build for $PLATFORM" + + # Provide helpful suggestions for cross-compilation failures + if [ "$PLATFORM" != "$(detect_platform)" ]; then + print_message $YELLOW "💡 Cross-compilation suggestions:" + print_message $YELLOW " 1. Use Docker build: make build-docker" + print_message $YELLOW " 2. Use GitHub Actions for multi-platform builds" + print_message $YELLOW " 3. Build natively on the target platform" + print_message $YELLOW " 4. Use 'make docker-buildx' for multi-arch Docker images" + fi + + return 1 + fi +} + + + +# Main build function +build_rustfs() { + local version=$(get_version) + + print_message $BLUE "🚀 Starting RustFS binary build process..." + print_message $YELLOW " Version: $version" + print_message $YELLOW " Platform: $PLATFORM" + print_message $YELLOW " Output Directory: $OUTPUT_DIR" + print_message $YELLOW " Build Type: $BUILD_TYPE" + print_message $YELLOW " Sign: $SIGN" + print_message $YELLOW " With Console: $WITH_CONSOLE" + if [ "$WITH_CONSOLE" = true ]; then + print_message $YELLOW " Console Version: $CONSOLE_VERSION" + print_message $YELLOW " Force Console Update: $FORCE_CONSOLE_UPDATE" + fi + print_message $YELLOW " Skip Verification: $SKIP_VERIFICATION" + echo "" + + # Setup environment + setup_rust_environment + echo "" + + # Download console assets if requested + download_console_assets + echo "" + + # Build binary + build_binary + echo "" + + print_message $GREEN "🎉 Build process completed successfully!" + + # Show built binary + local binary_file="${OUTPUT_DIR}/${PLATFORM}/${BINARY_NAME}" + if [ -f "$binary_file" ]; then + local size=$(ls -lh "$binary_file" | awk '{print $5}') + print_message $BLUE "📋 Built binary: $binary_file ($size)" + fi +} + +# Parse command line arguments +while [[ $# -gt 0 ]]; do + case $1 in + -o|--output-dir) + OUTPUT_DIR="$2" + shift 2 + ;; + -b|--binary-name) + BINARY_NAME="$2" + shift 2 + ;; + -p|--platform) + CUSTOM_PLATFORM="$2" + shift 2 + ;; + --dev) + BUILD_TYPE="debug" + shift + ;; + --sign) + SIGN=true + shift + ;; + --with-console) + WITH_CONSOLE=true + shift + ;; + --no-console) + WITH_CONSOLE=false + shift + ;; + --force-console-update) + FORCE_CONSOLE_UPDATE=true + WITH_CONSOLE=true # Auto-enable download when forcing update + shift + ;; + --console-version) + CONSOLE_VERSION="$2" + shift 2 + ;; + --skip-verification) + SKIP_VERIFICATION=true + shift + ;; + -h|--help) + usage + exit 0 + ;; + *) + print_message $RED "❌ Unknown option: $1" + usage + exit 1 + ;; + esac +done + +# Main execution +main() { + print_message $BLUE "🦀 RustFS Binary Build Script" + echo "" + + # Check if we're in a Rust project + if [ ! -f "Cargo.toml" ]; then + print_message $RED "❌ No Cargo.toml found. Are you in a Rust project directory?" + exit 1 + fi + + # Override platform if specified + if [ -n "$CUSTOM_PLATFORM" ]; then + PLATFORM="$CUSTOM_PLATFORM" + print_message $YELLOW "🎯 Using specified platform: $PLATFORM" + + # Auto-enable skip verification for cross-compilation + if [ "$PLATFORM" != "$(detect_platform)" ]; then + SKIP_VERIFICATION=true + print_message $YELLOW "⚠️ Cross-compilation detected, enabling --skip-verification" + fi + fi + + # Start build process + build_rustfs +} + +# Run main function +main + diff --git a/build_rustfs.sh b/build_rustfs.sh deleted file mode 100755 index 535a7f76..00000000 --- a/build_rustfs.sh +++ /dev/null @@ -1,35 +0,0 @@ -#!/bin/bash -# Copyright 2024 RustFS Team -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -clear - -# Get the current platform architecture -ARCH=$(uname -m) - -# Set the target directory according to the schema -if [ "$ARCH" == "x86_64" ]; then - TARGET_DIR="target/x86_64" -elif [ "$ARCH" == "aarch64" ]; then - TARGET_DIR="target/arm64" -else - TARGET_DIR="target/unknown" -fi - -# Set CARGO_TARGET_DIR and build the project -CARGO_TARGET_DIR=$TARGET_DIR RUSTFLAGS="-C link-arg=-fuse-ld=mold" cargo build --release --package rustfs - -echo -e "\a" -echo -e "\a" -echo -e "\a" diff --git a/.docker/cargo.config.toml b/cargo.config.toml similarity index 100% rename from .docker/cargo.config.toml rename to cargo.config.toml diff --git a/docker-buildx.sh b/docker-buildx.sh new file mode 100755 index 00000000..05404503 --- /dev/null +++ b/docker-buildx.sh @@ -0,0 +1,273 @@ +#!/bin/bash + +set -e + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +# Default values +REGISTRY="ghcr.io" +NAMESPACE="rustfs" +PLATFORMS="linux/amd64,linux/arm64" +PUSH=false +NO_CACHE=false +RELEASE="" +CHANNEL="release" + +# Print usage +usage() { + echo "Usage: $0 [OPTIONS]" + echo "" + echo "Options:" + echo " -r, --registry REGISTRY Docker registry (default: ghcr.io)" + echo " -n, --namespace NAMESPACE Image namespace (default: rustfs)" + echo " -p, --platforms PLATFORMS Target platforms (default: linux/amd64,linux/arm64)" + echo " --push Push images to registry" + echo " --no-cache Disable build cache" + echo " --release VERSION Specify release version (default: auto-detect from git)" + echo " --channel CHANNEL Download channel: release or dev (default: release)" + echo " -h, --help Show this help message" + echo "" + echo "Examples:" + echo " $0 # Build all variants locally" + echo " $0 --push # Build and push all variants" + echo " $0 --push --no-cache # Build and push with no cache" + echo " $0 --release v1.0.0 # Build specific release version" + echo " $0 --channel dev # Build with dev channel binaries" + echo " $0 --release latest --channel dev # Build latest dev build" +} + +# Print colored message +print_message() { + local color=$1 + local message=$2 + echo -e "${color}${message}${NC}" +} + +# Check if Docker buildx is available +check_buildx() { + if ! docker buildx version >/dev/null 2>&1; then + print_message $RED "❌ Docker buildx is not available. Please install Docker with buildx support." + exit 1 + fi +} + +# Setup buildx builder +setup_builder() { + local builder_name="rustfs-builder" + + print_message $BLUE "🔧 Setting up Docker buildx builder..." + + # Check if builder exists + if docker buildx ls | grep -q "$builder_name"; then + print_message $YELLOW "⚠️ Builder '$builder_name' already exists, using existing one" + docker buildx use "$builder_name" + else + # Create new builder + docker buildx create --name "$builder_name" --driver docker-container --bootstrap + docker buildx use "$builder_name" + print_message $GREEN "✅ Created and activated builder '$builder_name'" + fi + + # Inspect builder + docker buildx inspect --bootstrap +} + +# Get version from git +get_version() { + if [ -n "$RELEASE" ]; then + echo "$RELEASE" + return + fi + + # Try to get version from git tag + if git describe --abbrev=0 --tags >/dev/null 2>&1; then + git describe --abbrev=0 --tags + else + # Fallback to commit hash + git rev-parse --short HEAD + fi +} + +# Build and push images +build_and_push() { + local version=$(get_version) + local image_base="${REGISTRY}/${NAMESPACE}/rustfs" + local build_date=$(date -u +"%Y-%m-%dT%H:%M:%SZ") + local vcs_ref=$(git rev-parse --short HEAD) + + print_message $BLUE "🚀 Building RustFS Docker images..." + print_message $YELLOW " Version: $version" + print_message $YELLOW " Registry: $REGISTRY" + print_message $YELLOW " Namespace: $NAMESPACE" + print_message $YELLOW " Platforms: $PLATFORMS" + print_message $YELLOW " Channel: $CHANNEL" + print_message $YELLOW " Build Date: $build_date" + print_message $YELLOW " VCS Ref: $vcs_ref" + print_message $YELLOW " Push: $PUSH" + print_message $YELLOW " No Cache: $NO_CACHE" + echo "" + + # Build command base + local build_cmd="docker buildx build" + build_cmd+=" --platform $PLATFORMS" + build_cmd+=" --build-arg RELEASE=$version" + build_cmd+=" --build-arg CHANNEL=$CHANNEL" + build_cmd+=" --build-arg BUILD_DATE=$build_date" + build_cmd+=" --build-arg VCS_REF=$vcs_ref" + + if [ "$NO_CACHE" = true ]; then + build_cmd+=" --no-cache" + fi + + if [ "$PUSH" = true ]; then + build_cmd+=" --push" + else + build_cmd+=" --load" + fi + + # Build latest variant + print_message $BLUE "🏗️ Building latest variant..." + local latest_cmd="$build_cmd" + + # Add channel-specific tags + if [ "$CHANNEL" = "dev" ]; then + latest_cmd+=" -t ${image_base}:dev-latest" + else + latest_cmd+=" -t ${image_base}:latest" + fi + + latest_cmd+=" --build-arg RELEASE=latest" + latest_cmd+=" -f Dockerfile ." + + print_message $BLUE "📦 Executing: $latest_cmd" + if eval $latest_cmd; then + print_message $GREEN "✅ Successfully built latest variant" + else + print_message $RED "❌ Failed to build latest variant" + print_message $YELLOW "💡 Note: Make sure rustfs binaries are available at:" + if [ "$CHANNEL" = "dev" ]; then + print_message $YELLOW " https://dl.rustfs.com/artifacts/rustfs/dev/" + else + print_message $YELLOW " https://dl.rustfs.com/artifacts/rustfs/release/" + fi + exit 1 + fi + + # Prune build cache + docker buildx prune -f + + # Build release variant (only if not latest) + if [ "$RELEASE" != "latest" ]; then + print_message $BLUE "🏗️ Building release variant..." + local release_cmd="$build_cmd" + release_cmd+=" -t ${image_base}:${version}" + + # Add channel-specific tags + if [ "$CHANNEL" = "dev" ]; then + release_cmd+=" -t ${image_base}:dev-${version}" + else + release_cmd+=" -t ${image_base}:release" + fi + + release_cmd+=" --build-arg RELEASE=${version}" + release_cmd+=" -f Dockerfile ." + + print_message $BLUE "📦 Executing: $release_cmd" + if eval $release_cmd; then + print_message $GREEN "✅ Successfully built release variant" + else + print_message $RED "❌ Failed to build release variant" + print_message $YELLOW "💡 Note: Make sure rustfs binaries are available at:" + if [ "$CHANNEL" = "dev" ]; then + print_message $YELLOW " https://dl.rustfs.com/artifacts/rustfs/dev/" + else + print_message $YELLOW " https://dl.rustfs.com/artifacts/rustfs/release/" + fi + exit 1 + fi + else + print_message $BLUE "⏭️ Skipping release variant (already built as latest)" + fi + + # Final cleanup + docker buildx prune -f +} + +# Parse command line arguments +while [[ $# -gt 0 ]]; do + case $1 in + -r|--registry) + REGISTRY="$2" + shift 2 + ;; + -n|--namespace) + NAMESPACE="$2" + shift 2 + ;; + -p|--platforms) + PLATFORMS="$2" + shift 2 + ;; + --push) + PUSH=true + shift + ;; + --no-cache) + NO_CACHE=true + shift + ;; + --release) + RELEASE="$2" + shift 2 + ;; + --channel) + CHANNEL="$2" + if [ "$CHANNEL" != "release" ] && [ "$CHANNEL" != "dev" ]; then + print_message $RED "❌ Invalid channel: $CHANNEL. Must be 'release' or 'dev'" + exit 1 + fi + shift 2 + ;; + -h|--help) + usage + exit 0 + ;; + *) + print_message $RED "❌ Unknown option: $1" + usage + exit 1 + ;; + esac +done + +# Main execution +main() { + print_message $BLUE "🐳 RustFS Docker Buildx Build Script" + echo "" + + # Check prerequisites + check_buildx + + # Setup builder + setup_builder + echo "" + + # Start build process + build_and_push + + print_message $GREEN "🎉 Build process completed successfully!" + + # Show built images if not pushing + if [ "$PUSH" = false ]; then + print_message $BLUE "📋 Built images:" + docker images | grep "${NAMESPACE}/rustfs" | head -10 + fi +} + +# Run main function +main diff --git a/docker-compose.yml b/docker-compose.yml index 86d53e9f..0f76d6ae 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -23,7 +23,7 @@ services: container_name: rustfs-server build: context: . - dockerfile: .docker/ubuntu/Dockerfile.source + dockerfile: Dockerfile.source args: TARGETPLATFORM: linux/amd64 ports: @@ -68,7 +68,7 @@ services: container_name: rustfs-dev build: context: . - dockerfile: .docker/ubuntu/Dockerfile.dev + dockerfile: Dockerfile.source # Pure development environment ports: - "9010:9000" diff --git a/rustfs/README.md b/rustfs/README.md index d8be69b3..8aaab788 100644 --- a/rustfs/README.md +++ b/rustfs/README.md @@ -74,10 +74,14 @@ To get started with RustFS, follow these steps: 2. **Docker Quick Start (Option 2)​​** ```bash - podman run -d -p 9000:9000 -p 9001:9001 -v /data:/data quay.io/rustfs/rustfs + # Docker Hub (recommended) + docker run -d -p 9000:9000 -v /data:/data rustfs/rustfs:latest + + # Alternative using Podman + podman run -d -p 9000:9000 -v /data:/data rustfs/rustfs:latest ``` -3. **Access the Console**: Open your web browser and navigate to `http://localhost:9001` to access the RustFS console, +3. **Access the Console**: Open your web browser and navigate to `http://localhost:9000` to access the RustFS console, default username and password is `rustfsadmin` . 4. **Create a Bucket**: Use the console to create a new bucket for your objects. 5. **Upload Objects**: You can upload files directly through the console or use S3-compatible APIs to interact with your diff --git a/scripts/build-docker-multiarch.sh b/scripts/build-docker-multiarch.sh deleted file mode 100755 index 665b1538..00000000 --- a/scripts/build-docker-multiarch.sh +++ /dev/null @@ -1,256 +0,0 @@ -#!/bin/bash -# Copyright 2024 RustFS Team -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -set -euo pipefail - -# 多架构 Docker 构建脚本 -# 支持构建并推送 x86_64 和 ARM64 架构的镜像 - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -PROJECT_ROOT="$(dirname "$SCRIPT_DIR")" -cd "$PROJECT_ROOT" - -# 默认配置 -REGISTRY_IMAGE_DOCKERHUB="rustfs/rustfs" -REGISTRY_IMAGE_GHCR="ghcr.io/rustfs/s3-rustfs" -VERSION="${VERSION:-latest}" -PUSH="${PUSH:-false}" -IMAGE_TYPE="${IMAGE_TYPE:-production}" - -# 帮助信息 -show_help() { - cat << EOF -用法: $0 [选项] - -选项: - -h, --help 显示此帮助信息 - -v, --version TAG 设置镜像版本标签 (默认: latest) - -p, --push 推送镜像到仓库 - -t, --type TYPE 镜像类型 (production|ubuntu|rockylinux|devenv, 默认: production) - -环境变量: - DOCKERHUB_USERNAME Docker Hub 用户名 - DOCKERHUB_TOKEN Docker Hub 访问令牌 - GITHUB_TOKEN GitHub 访问令牌 - -示例: - # 仅构建不推送 - $0 --version v1.0.0 - - # 构建并推送到仓库 - $0 --version v1.0.0 --push - - # 构建 Ubuntu 版本 - $0 --type ubuntu --version v1.0.0 -EOF -} - -# 解析命令行参数 -while [[ $# -gt 0 ]]; do - case $1 in - -h|--help) - show_help - exit 0 - ;; - -v|--version) - VERSION="$2" - shift 2 - ;; - -p|--push) - PUSH=true - shift - ;; - -t|--type) - IMAGE_TYPE="$2" - shift 2 - ;; - *) - echo "未知参数: $1" - show_help - exit 1 - ;; - esac -done - -# 设置 Dockerfile 和后缀 -case "$IMAGE_TYPE" in - production) - DOCKERFILE="Dockerfile" - SUFFIX="" - ;; - ubuntu) - DOCKERFILE=".docker/Dockerfile.ubuntu22.04" - SUFFIX="-ubuntu22.04" - ;; - rockylinux) - DOCKERFILE=".docker/Dockerfile.rockylinux9.3" - SUFFIX="-rockylinux9.3" - ;; - devenv) - DOCKERFILE=".docker/Dockerfile.devenv" - SUFFIX="-devenv" - ;; - *) - echo "错误: 不支持的镜像类型: $IMAGE_TYPE" - echo "支持的类型: production, ubuntu, rockylinux, devenv" - exit 1 - ;; -esac - -echo "🚀 开始多架构 Docker 构建" -echo "📋 构建信息:" -echo " - 镜像类型: $IMAGE_TYPE" -echo " - Dockerfile: $DOCKERFILE" -echo " - 版本标签: $VERSION$SUFFIX" -echo " - 推送: $PUSH" -echo " - 架构: linux/amd64, linux/arm64" - -# 检查必要的工具 -if ! command -v docker &> /dev/null; then - echo "❌ 错误: 未找到 docker 命令" - exit 1 -fi - -# 检查 Docker Buildx -if ! docker buildx version &> /dev/null; then - echo "❌ 错误: Docker Buildx 不可用" - echo "请运行: docker buildx install" - exit 1 -fi - -# 创建并使用 buildx 构建器 -BUILDER_NAME="rustfs-multiarch-builder" -if ! docker buildx inspect "$BUILDER_NAME" &> /dev/null; then - echo "🔨 创建多架构构建器..." - docker buildx create --name "$BUILDER_NAME" --use --bootstrap -else - echo "🔨 使用现有构建器..." - docker buildx use "$BUILDER_NAME" -fi - -# 构建多架构二进制文件 -echo "🔧 构建多架构二进制文件..." - -# 检查是否存在预构建的二进制文件 -if [[ ! -f "target/x86_64-unknown-linux-musl/release/rustfs" ]] || [[ ! -f "target/aarch64-unknown-linux-gnu/release/rustfs" ]]; then - echo "⚠️ 未找到预构建的二进制文件，正在构建..." - - # 安装构建依赖 - if ! command -v cross &> /dev/null; then - echo "📦 安装 cross 工具..." - cargo install cross - fi - - # 生成 protobuf 代码 - echo "📝 生成 protobuf 代码..." - cargo run --bin gproto || true - - # 构建 x86_64 - echo "🔨 构建 x86_64 二进制文件..." - cargo build --release --target x86_64-unknown-linux-musl --bin rustfs - - # 构建 ARM64 - echo "🔨 构建 ARM64 二进制文件..." - cross build --release --target aarch64-unknown-linux-gnu --bin rustfs -fi - -# 准备构建参数 -BUILD_ARGS="" -TAGS="" - -# Docker Hub 标签 -if [[ -n "${DOCKERHUB_USERNAME:-}" ]]; then - TAGS="$TAGS -t $REGISTRY_IMAGE_DOCKERHUB:$VERSION$SUFFIX" -fi - -# GitHub Container Registry 标签 -if [[ -n "${GITHUB_TOKEN:-}" ]]; then - TAGS="$TAGS -t $REGISTRY_IMAGE_GHCR:$VERSION$SUFFIX" -fi - -# 如果没有设置标签，使用本地标签 -if [[ -z "$TAGS" ]]; then - TAGS="-t rustfs:$VERSION$SUFFIX" -fi - -# 构建镜像 -echo "🏗️ 构建多架构 Docker 镜像..." -BUILD_CMD="docker buildx build \ - --platform linux/amd64,linux/arm64 \ - --file $DOCKERFILE \ - $TAGS \ - --build-arg BUILDTIME=$(date -u +%Y-%m-%dT%H:%M:%SZ) \ - --build-arg VERSION=$VERSION \ - --build-arg REVISION=$(git rev-parse --short HEAD 2>/dev/null || echo 'unknown')" - -if [[ "$PUSH" == "true" ]]; then - # 登录到仓库 - if [[ -n "${DOCKERHUB_USERNAME:-}" ]] && [[ -n "${DOCKERHUB_TOKEN:-}" ]]; then - echo "🔐 登录到 Docker Hub..." - echo "$DOCKERHUB_TOKEN" | docker login --username "$DOCKERHUB_USERNAME" --password-stdin - fi - - if [[ -n "${GITHUB_TOKEN:-}" ]]; then - echo "🔐 登录到 GitHub Container Registry..." - echo "$GITHUB_TOKEN" | docker login ghcr.io --username "$(whoami)" --password-stdin - fi - - BUILD_CMD="$BUILD_CMD --push" -else - BUILD_CMD="$BUILD_CMD --load" -fi - -BUILD_CMD="$BUILD_CMD ." - -echo "📋 执行构建命令:" -echo "$BUILD_CMD" -echo "" - -# 执行构建 -eval "$BUILD_CMD" - -echo "" -echo "✅ 多架构 Docker 镜像构建完成!" - -if [[ "$PUSH" == "true" ]]; then - echo "🚀 镜像已推送到仓库" - - # 显示推送的镜像信息 - echo "" - echo "📦 推送的镜像:" - if [[ -n "${DOCKERHUB_USERNAME:-}" ]]; then - echo " - $REGISTRY_IMAGE_DOCKERHUB:$VERSION$SUFFIX" - fi - if [[ -n "${GITHUB_TOKEN:-}" ]]; then - echo " - $REGISTRY_IMAGE_GHCR:$VERSION$SUFFIX" - fi - - echo "" - echo "🔍 验证多架构支持:" - if [[ -n "${DOCKERHUB_USERNAME:-}" ]]; then - echo " docker buildx imagetools inspect $REGISTRY_IMAGE_DOCKERHUB:$VERSION$SUFFIX" - fi - if [[ -n "${GITHUB_TOKEN:-}" ]]; then - echo " docker buildx imagetools inspect $REGISTRY_IMAGE_GHCR:$VERSION$SUFFIX" - fi -else - echo "💾 镜像已构建到本地" - echo "" - echo "🔍 查看镜像:" - echo " docker images rustfs:$VERSION$SUFFIX" -fi - -echo "" -echo "🎉 构建任务完成!" diff --git a/scripts/run.ps1 b/scripts/run.ps1 index c0a16a1f..015c9f78 100644 --- a/scripts/run.ps1 +++ b/scripts/run.ps1 @@ -64,4 +64,4 @@ if ($args.Count -gt 0) { } # Run the program -cargo run --bin rustfs \ No newline at end of file +cargo run --bin rustfs diff --git a/scripts/run.sh b/scripts/run.sh index 1f77a067..60314fad 100755 --- a/scripts/run.sh +++ b/scripts/run.sh @@ -112,4 +112,4 @@ fi # 启动 webhook 服务器 #cargo run --example webhook -p rustfs-notify & # 启动主服务 -cargo run --bin rustfs \ No newline at end of file +cargo run --bin rustfs diff --git a/scripts/setup-test-binaries.sh b/scripts/setup-test-binaries.sh new file mode 100755 index 00000000..38d281bc --- /dev/null +++ b/scripts/setup-test-binaries.sh @@ -0,0 +1,38 @@ +#!/bin/bash + +# Setup test binaries for Docker build testing +# This script creates temporary binary files for testing Docker build process + +set -e + +echo "Setting up test binaries for Docker build..." + +# Create temporary rustfs binary +./build-rustfs.sh -p x86_64-unknown-linux-musl + +# Create test directory structure +mkdir -p test-releases/server/rustfs/release/linux-amd64/archive +mkdir -p test-releases/server/rustfs/release/linux-arm64/archive + +# Get version +VERSION=$(git describe --abbrev=0 --tags 2>/dev/null || git rev-parse --short HEAD) + +# Copy binaries +cp target/release/x86_64-unknown-linux-musl/rustfs test-releases/server/rustfs/release/linux-amd64/archive/rustfs.${VERSION} +cp target/release/x86_64-unknown-linux-musl/rustfs.sha256sum test-releases/server/rustfs/release/linux-amd64/archive/rustfs.${VERSION}.sha256sum + +# Create dummy signatures +echo "dummy signature" > test-releases/server/rustfs/release/linux-amd64/archive/rustfs.${VERSION}.minisig +echo "dummy signature" > test-releases/server/rustfs/release/linux-arm64/archive/rustfs.${VERSION}.minisig + +# Also copy for arm64 (using same binary for testing) +cp target/release/x86_64-unknown-linux-musl/rustfs test-releases/server/rustfs/release/linux-arm64/archive/rustfs.${VERSION} +cp target/release/x86_64-unknown-linux-musl/rustfs.sha256sum test-releases/server/rustfs/release/linux-arm64/archive/rustfs.${VERSION}.sha256sum + +echo "Test binaries created for version: ${VERSION}" +echo "You can now test Docker builds with these local binaries" +echo "" +echo "To start a local HTTP server for testing:" +echo " cd test-releases && python3 -m http.server 8000" +echo "" +echo "Then modify Dockerfile to use http://host.docker.internal:8000 instead of https://dl.rustfs.com/artifacts/rustfs" diff --git a/scripts/static.sh b/scripts/static.sh index 64142608..6a402536 100755 --- a/scripts/static.sh +++ b/scripts/static.sh @@ -12,4 +12,4 @@ # See the License for the specific language governing permissions and # limitations under the License. -curl -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" -o tempfile.zip && unzip -o tempfile.zip -d ./rustfs/static && rm tempfile.zip \ No newline at end of file +curl -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" -o tempfile.zip && unzip -o tempfile.zip -d ./rustfs/static && rm tempfile.zip