mirror of
https://github.com/rustfs/rustfs.git
synced 2026-01-17 09:40:32 +00:00
* feat: implement Docker improvements and binary build scripts This commit transforms the RustFS Docker build system to follow MinIO's best practices: ## 🏗️ Binary Build Script (build-rustfs.sh) - Create independent binary compilation script for multi-platform builds - Support x86_64 and aarch64 Linux musl targets - Include checksum generation and optional binary signing - Support cross-compilation and upload functionality - Automated target installation and environment setup ## 🐳 Docker Improvements - Rewrite Dockerfiles to download precompiled binaries instead of building from source - Follow MinIO's approach for security and binary verification - Add comprehensive LABEL metadata (version, build-date, vcs-ref) - Implement proper environment variable management - Add signature verification with minisign (commented for future use) - Include static curl download for minimal runtime dependencies ## 🚀 Enhanced Build Script (docker-buildx.sh) - Inspired by MinIO's docker-buildx.sh for consistency and reliability - Support multiple platforms with proper build arguments - Auto-detect git versions and pass metadata to containers - Improved error messages with helpful troubleshooting hints - Cleanup and cache management between builds ## 🛠️ Supporting Scripts - scripts/download-static-curl.sh: Download statically compiled curl - scripts/setup-test-binaries.sh: Create test binaries for local development ## 📋 Key Benefits - Faster Docker builds (download vs compile) - Better security with signature verification - Consistent with industry standards (MinIO approach) - Proper multi-platform support - Enhanced metadata and traceability - Independent binary distribution capability * feat: update Docker files to use Aliyun OSS for binary downloads * feat: merge stash with OSS binary download improvements - Remove old build_rustfs.sh script - Keep Aliyun OSS download URLs for binary retrieval - Maintain Docker build improvements from stash - Resolve merge conflicts between stash and OSS updates * feat: improve build-rustfs.sh with auto platform detection - Auto-detect current platform using uname (like old build_rustfs.sh) - Default to building for current platform only - Add --all-platforms flag for cross-compilation to Linux musl targets - Support macOS (darwin) and Linux platforms - Auto-enable cross compilation when needed - Provide better usage examples and platform detection info This makes the script much more user-friendly by default while maintaining flexibility for cross-compilation scenarios. * refactor: simplify build-rustfs.sh for CI/CD pipeline usage - Remove cross-compilation complexity (each CI runner builds natively) - Focus on single platform builds per runner - Remove --all-platforms and --cross options - Simplify to match CI/CD workflow where: * Linux x86_64 runner builds Linux x86_64 binary * Linux ARM64 runner builds Linux ARM64 binary * macOS x86_64 runner builds macOS x86_64 binary * macOS ARM64 runner builds macOS ARM64 binary - Keep signing and upload functionality for release CI - Make the script's purpose and usage clearer This aligns with the user's understanding that build scripts should focus on native compilation for the current platform only. * feat: update download server domain to dl.rustfs.com - Update Dockerfile to use dl.rustfs.com/dev/ for development binaries - Update Dockerfile.release to use dl.rustfs.com/release/ for release binaries - Update docker-buildx.sh error messages with new URLs - Update build-rustfs.sh upload target to dl.rustfs.com - Update test scripts to reference new domain - Clean up remaining git conflict markers This centralizes all binary downloads through the official dl.rustfs.com domain instead of direct OSS access. * fix: correct dl.rustfs.com path structure to include /artifacts/rustfs/ - Update all download URLs to use correct path structure: * Dev: https://dl.rustfs.com/artifacts/rustfs/dev/ * Release: https://dl.rustfs.com/artifacts/rustfs/release/ - Test confirmed both paths return HTTP 200 with application/zip content-type - Update Dockerfile, Dockerfile.release, docker-buildx.sh, and build-rustfs.sh - Update test scripts with correct base path The dl.rustfs.com domain requires the /artifacts/rustfs/ prefix to access the binary files correctly. * feat: refactor Dockerfile to download binaries from GitHub Releases - Changed binary download source from dl.rustfs.com to GitHub Releases - Added support for latest release auto-detection via GitHub API - Enhanced error handling with detailed messages and helpful links - Added optional checksum verification using SHA256SUMS - Improved architecture support for amd64 and arm64 - Removed unnecessary minisign installation - Added jq dependency for JSON parsing * feat: consolidate Docker build to use single Dockerfile - Removed Dockerfile.release and use unified Dockerfile instead - Updated docker-buildx.sh to use single Dockerfile with build args - Both latest and release variants now use GitHub Releases - Simplified build process and reduced maintenance overhead - Updated error messages to point to GitHub releases * chore: remove unused Dockerfile.obs - Removed Dockerfile.obs as it's no longer needed - Simplified Docker build configuration * feat: unify Docker prebuild variants to use GitHub Releases - Updated .docker/alpine/Dockerfile.prebuild to download from GitHub Releases - Updated .docker/ubuntu/Dockerfile.prebuild to download from GitHub Releases - All prebuild variants now consistently use GitHub Releases as binary source - Added checksum verification for all prebuild variants - Updated .docker/README.md to reflect unified GitHub Releases approach - Improved error handling and user guidance in all prebuild Dockerfiles * feat: major Docker structure simplification and consolidation ## 🎯 Simplified Docker Structure Moved from complex multi-directory structure to clean root-level organization: ### Before: - Dockerfile (production) - .docker/alpine/Dockerfile.prebuild (duplicate) - .docker/alpine/Dockerfile.source - .docker/ubuntu/Dockerfile.prebuild (duplicate) - .docker/ubuntu/Dockerfile.source - .docker/ubuntu/Dockerfile.dev ### After: - Dockerfile (production - Alpine + GitHub Releases) - Dockerfile.source (source build - Ubuntu + cross-compilation) - Dockerfile.dev (development - Ubuntu + full toolchain) ## 🔧 Key Changes - **Eliminated Duplicates**: Removed redundant prebuild variants - **Moved Core Files**: Dockerfile.{source,dev} now in root directory - **Unified Configuration**: cargo.config.toml moved to root - **Updated References**: Fixed all GitHub Actions and docker-compose paths - **Simplified CI Matrix**: Reduced from 5 to 3 Docker variants ## 📦 Preserved Valuable Diversity - **Production**: Alpine-based for minimal size - **Source**: Ubuntu-based with cross-compilation support - **Development**: Ubuntu-based with full development tools ## 🚀 Benefits - ✅ Cleaner project structure - ✅ Easier maintenance and navigation - ✅ Reduced CI/CD complexity - ✅ Faster build matrix execution - ✅ Maintained functionality and flexibility * chore: remove duplicate cargo.config.toml from .docker directory The file is now in the root directory and no longer needed in .docker/ * fix: update all references to removed Dockerfile files - Updated .docker/compose/README.md to reference Dockerfile.source instead of Dockerfile.obs - Updated docker-compose.yml to use Dockerfile.source instead of Dockerfile.dev - Updated scripts/build-docker-multiarch.sh to use Dockerfile.source for devenv builds - Updated .github/workflows/docker.yml to use Dockerfile.source for dev builds - Updated Makefile to use Dockerfile.source for init-devenv target - Updated .docker/README.md to remove references to non-existent Dockerfile.dev - Ensured all Docker configurations consistently use the unified Dockerfile structure * chore: remove unnecessary console static assets download - Remove obsolete download steps from build.yml and performance.yml - Console static assets are already embedded via rust-embed in rustfs/static/ - The download from dl.rustfs.com is no longer needed as project contains complete console assets - This improves build reliability and reduces external dependencies - Replaced with verification steps that confirm embedded assets are present * feat: update Makefile and README.md for new Docker build system - Updated Makefile to use unified Docker build system: - Replace references to non-existent Dockerfile.ubuntu22.04 and Dockerfile.rockylinux9.3 - Add new docker-buildx targets using docker-buildx.sh script - Deprecate old docker-build-multiarch targets with warnings - Add docker-build-production and docker-build-source targets - Update help-docker with new command structure - Updated README.md with docker-buildx.sh usage: - Add comprehensive Docker build from source section - Document multi-architecture build capabilities - Include both script and Make target examples - Show registry flexibility and build optimization features - Update step numbers in quickstart guide - Improve developer experience with clear documentation and updated tooling - Maintain backward compatibility with deprecation warnings * feat: integrate console assets download into build-rustfs.sh - Added console download functionality to build-rustfs.sh: - New flags: --download-console, --force-console-update, --console-version - Intelligent detection of existing console assets - Retry logic with fallback error handling - Consistent with Docker build asset management - Updated scripts to use unified build process: - scripts/static.sh: Now uses build-rustfs.sh for console downloads - scripts/run.sh: Uses build-rustfs.sh instead of direct curl - scripts/run.ps1: Updated with guidance for Windows users - Benefits: - Unified asset management across all build processes - Consistent version handling and retry logic - Eliminates duplicate download logic - Better error handling and user feedback - Preparation for CI/CD integration - Removed unused download-static-curl.sh script This change centralizes console asset management and prepares for streamlined CI/CD processes where build-rustfs.sh becomes the single point of truth for binary and asset builds. * fix: update PowerShell script to use unified console asset management - Updated scripts/run.ps1 to use build-rustfs.sh for console asset downloads - Added guidance for Windows users to use the unified build script - Maintains consistency across all platform-specific scripts * feat: add binary verification to build script - Add verify_binary function to test built binaries - Test --help and --version commands - Verify binary structure with readelf/otool - Add --skip-verification option for cross-compilation - Include verification status in build output - Automatic error handling if verification fails * feat: add platform selection support to build script - Add --platform parameter to build-rustfs.sh for target platform selection - Implement cross-compilation support with automatic 'cross' tool detection - Auto-enable --skip-verification for cross-compilation scenarios - Update all Makefile build targets to use unified build-rustfs.sh script - Add helpful error messages and suggestions for cross-compilation failures - Update help documentation with platform selection examples - Improve build consistency across different architectures * feat: modernize CI/CD build process with build-rustfs.sh - Replace manual cargo build commands with unified build-rustfs.sh script - Simplify matrix configuration by removing cross-compilation flags - Ensure consistency between local and CI/CD builds - Automatic cross-compilation tool detection and installation - Built-in binary verification for quality assurance - Unified console asset management - Better error handling and suggestions Benefits: - Consistent build process across all environments - Automatic detection and handling of cross-compilation scenarios - Built-in quality checks with binary verification - Reduced CI/CD configuration complexity - Better maintainability with single source of truth for build logic * feat: optimize CI/CD workspace path management - Add WORKSPACE_DIR environment variable to cache github.workspace - Set default working-directory at job level for consistency - Use explicit workspace paths in critical operations - Improve reliability and maintainability of CI/CD paths - Ensure consistent behavior across different GitHub Actions environments Benefits: - More explicit and reliable path handling - Better maintainability with centralized workspace reference - Reduced risk of path-related issues in CI/CD - Consistent working directory across all job steps * refactor: simplify CI/CD path management - remove redundant workspace references - Remove unnecessary WORKSPACE_DIR environment variable - Remove redundant defaults.run.working-directory setting - Use relative paths since GITHUB_WORKSPACE is the default working directory - Follow GitHub Actions best practices by leveraging default behavior As per GitHub Actions documentation, GITHUB_WORKSPACE is already the default working directory, so explicit specification is unnecessary in most cases. * docs: update Docker README to reflect current project state - Fix directory structure: remove non-existent nginx/ directory - Correct base OS: Dockerfile.source uses Debian Bookworm, not Ubuntu 22.04 - Add docker-buildx.sh script documentation - Update Docker tag examples to match actual CI/CD workflows - Add CI/CD integration section explaining automated builds - Document build variants and manual build options - Reflect current project architecture and tooling These updates ensure the documentation accurately represents the current Docker build system and CI/CD workflows. * fix: update Docker command in rustfs README - Replace quay.io registry with Docker Hub (rustfs/rustfs:latest) - Remove separate console port 9001, console now runs on main port 9000 - Add both Docker and Podman examples for user choice - Fix console access URL to use unified port This aligns with the recent console port consolidation changes and the project's move to Docker Hub as the primary registry. * wip * fix: remove unnecessary entrypoint.sh and fix Docker paths * Update Dockerfile Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * cleanup: remove unused DOCKERFILE_PATH variable from Makefile * feat: update Docker build to use dl.rustfs.com for binary downloads - Replace GitHub releases download with dl.rustfs.com - Add CHANNEL parameter support (release/dev) - Update docker-buildx.sh to support channel-specific builds - Improve error messages with new download URLs - Support both latest and specific version downloads - Add channel validation in build script --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
229 lines
8.0 KiB
Makefile
229 lines
8.0 KiB
Makefile
###########
|
|
# 远程开发,需要 VSCode 安装 Dev Containers, Remote SSH, Remote Explorer
|
|
# https://code.visualstudio.com/docs/remote/containers
|
|
###########
|
|
DOCKER_CLI ?= docker
|
|
IMAGE_NAME ?= rustfs:v1.0.0
|
|
CONTAINER_NAME ?= rustfs-dev
|
|
# Docker build configurations
|
|
DOCKERFILE_PRODUCTION = Dockerfile
|
|
DOCKERFILE_SOURCE = Dockerfile.source
|
|
|
|
# Code quality and formatting targets
|
|
.PHONY: fmt
|
|
fmt:
|
|
@echo "🔧 Formatting code..."
|
|
cargo fmt --all
|
|
|
|
.PHONY: fmt-check
|
|
fmt-check:
|
|
@echo "📝 Checking code formatting..."
|
|
cargo fmt --all --check
|
|
|
|
.PHONY: clippy
|
|
clippy:
|
|
@echo "🔍 Running clippy checks..."
|
|
cargo clippy --all-targets --all-features -- -D warnings
|
|
|
|
.PHONY: check
|
|
check:
|
|
@echo "🔨 Running compilation check..."
|
|
cargo check --all-targets
|
|
|
|
.PHONY: test
|
|
test:
|
|
@echo "🧪 Running tests..."
|
|
cargo nextest run --all --exclude e2e_test
|
|
cargo test --all --doc
|
|
|
|
.PHONY: pre-commit
|
|
pre-commit: fmt clippy check test
|
|
@echo "✅ All pre-commit checks passed!"
|
|
|
|
.PHONY: setup-hooks
|
|
setup-hooks:
|
|
@echo "🔧 Setting up git hooks..."
|
|
chmod +x .git/hooks/pre-commit
|
|
@echo "✅ Git hooks setup complete!"
|
|
|
|
.PHONY: init-devenv
|
|
init-devenv:
|
|
$(DOCKER_CLI) build -t $(IMAGE_NAME) -f Dockerfile.source .
|
|
$(DOCKER_CLI) stop $(CONTAINER_NAME)
|
|
$(DOCKER_CLI) rm $(CONTAINER_NAME)
|
|
$(DOCKER_CLI) run -d --name $(CONTAINER_NAME) -p 9010:9010 -p 9000:9000 -v $(shell pwd):/root/s3-rustfs -it $(IMAGE_NAME)
|
|
|
|
.PHONY: start
|
|
start:
|
|
$(DOCKER_CLI) start $(CONTAINER_NAME)
|
|
|
|
.PHONY: stop
|
|
stop:
|
|
$(DOCKER_CLI) stop $(CONTAINER_NAME)
|
|
|
|
.PHONY: e2e-server
|
|
e2e-server:
|
|
sh $(shell pwd)/scripts/run.sh
|
|
|
|
.PHONY: probe-e2e
|
|
probe-e2e:
|
|
sh $(shell pwd)/scripts/probe.sh
|
|
|
|
# Native build using build-rustfs.sh script
|
|
.PHONY: build
|
|
build:
|
|
@echo "🔨 Building RustFS using build-rustfs.sh script..."
|
|
./build-rustfs.sh
|
|
|
|
.PHONY: build-dev
|
|
build-dev:
|
|
@echo "🔨 Building RustFS in development mode..."
|
|
./build-rustfs.sh --dev
|
|
|
|
|
|
|
|
# Docker-based build (alternative approach)
|
|
# Usage: make BUILD_OS=ubuntu22.04 build-docker
|
|
# Output: target/ubuntu22.04/release/rustfs
|
|
BUILD_OS ?= rockylinux9.3
|
|
.PHONY: build-docker
|
|
build-docker: SOURCE_BUILD_IMAGE_NAME = rustfs-$(BUILD_OS):v1
|
|
build-docker: SOURCE_BUILD_CONTAINER_NAME = rustfs-$(BUILD_OS)-build
|
|
build-docker: BUILD_CMD = /root/.cargo/bin/cargo build --release --bin rustfs --target-dir /root/s3-rustfs/target/$(BUILD_OS)
|
|
build-docker:
|
|
@echo "🐳 Building RustFS using Docker ($(BUILD_OS))..."
|
|
$(DOCKER_CLI) build -t $(SOURCE_BUILD_IMAGE_NAME) -f $(DOCKERFILE_SOURCE) .
|
|
$(DOCKER_CLI) run --rm --name $(SOURCE_BUILD_CONTAINER_NAME) -v $(shell pwd):/root/s3-rustfs -it $(SOURCE_BUILD_IMAGE_NAME) $(BUILD_CMD)
|
|
|
|
.PHONY: build-musl
|
|
build-musl:
|
|
@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
|
|
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-buildx' instead"
|
|
./build-rustfs.sh --platform x86_64-unknown-linux-musl
|
|
|
|
.PHONY: build-gnu
|
|
build-gnu:
|
|
@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
|
|
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-buildx' instead"
|
|
./build-rustfs.sh --platform x86_64-unknown-linux-gnu
|
|
|
|
.PHONY: deploy-dev
|
|
deploy-dev: build-musl
|
|
@echo "🚀 Deploying to dev server: $${IP}"
|
|
./scripts/dev_deploy.sh $${IP}
|
|
|
|
# Multi-architecture Docker build targets (NEW: using docker-buildx.sh)
|
|
.PHONY: docker-buildx
|
|
docker-buildx:
|
|
@echo "🏗️ Building multi-architecture Docker images with buildx..."
|
|
./docker-buildx.sh
|
|
|
|
.PHONY: docker-buildx-push
|
|
docker-buildx-push:
|
|
@echo "🚀 Building and pushing multi-architecture Docker images with buildx..."
|
|
./docker-buildx.sh --push
|
|
|
|
.PHONY: docker-buildx-version
|
|
docker-buildx-version:
|
|
@if [ -z "$(VERSION)" ]; then \
|
|
echo "❌ 错误: 请指定版本, 例如: make docker-buildx-version VERSION=v1.0.0"; \
|
|
exit 1; \
|
|
fi
|
|
@echo "🏗️ Building multi-architecture Docker images (version: $(VERSION))..."
|
|
./docker-buildx.sh --release $(VERSION)
|
|
|
|
.PHONY: docker-buildx-push-version
|
|
docker-buildx-push-version:
|
|
@if [ -z "$(VERSION)" ]; then \
|
|
echo "❌ 错误: 请指定版本, 例如: make docker-buildx-push-version VERSION=v1.0.0"; \
|
|
exit 1; \
|
|
fi
|
|
@echo "🚀 Building and pushing multi-architecture Docker images (version: $(VERSION))..."
|
|
./docker-buildx.sh --release $(VERSION) --push
|
|
|
|
|
|
|
|
.PHONY: docker-build-production
|
|
docker-build-production:
|
|
@echo "🏗️ Building production Docker image..."
|
|
$(DOCKER_CLI) build -f $(DOCKERFILE_PRODUCTION) -t rustfs:latest .
|
|
|
|
.PHONY: docker-build-source
|
|
docker-build-source:
|
|
@echo "🏗️ Building source Docker image..."
|
|
$(DOCKER_CLI) build -f $(DOCKERFILE_SOURCE) -t rustfs:source .
|
|
|
|
.PHONY: docker-inspect-multiarch
|
|
docker-inspect-multiarch:
|
|
@if [ -z "$(IMAGE)" ]; then \
|
|
echo "❌ 错误: 请指定镜像, 例如: make docker-inspect-multiarch IMAGE=rustfs/rustfs:latest"; \
|
|
exit 1; \
|
|
fi
|
|
@echo "🔍 Inspecting multi-architecture image: $(IMAGE)"
|
|
docker buildx imagetools inspect $(IMAGE)
|
|
|
|
.PHONY: build-cross-all
|
|
build-cross-all:
|
|
@echo "🔧 Building all target architectures..."
|
|
@echo "💡 On macOS/Windows, use 'make docker-buildx' for reliable multi-arch builds"
|
|
@echo "🔨 Generating protobuf code..."
|
|
cargo run --bin gproto || true
|
|
@echo "🔨 Building x86_64-unknown-linux-musl..."
|
|
./build-rustfs.sh --platform x86_64-unknown-linux-musl
|
|
@echo "🔨 Building aarch64-unknown-linux-gnu..."
|
|
./build-rustfs.sh --platform aarch64-unknown-linux-gnu
|
|
@echo "✅ All architectures built successfully!"
|
|
|
|
.PHONY: help-build
|
|
help-build:
|
|
@echo "🔨 RustFS 构建帮助:"
|
|
@echo ""
|
|
@echo "🚀 本地构建 (推荐使用):"
|
|
@echo " make build # 构建 RustFS 二进制文件 (默认包含 console)"
|
|
@echo " make build-dev # 开发模式构建"
|
|
@echo " make build-musl # 构建 musl 版本"
|
|
@echo " make build-gnu # 构建 GNU 版本"
|
|
@echo ""
|
|
@echo "🐳 Docker 构建:"
|
|
@echo " make build-docker # 使用 Docker 容器构建"
|
|
@echo " make build-docker BUILD_OS=ubuntu22.04 # 指定构建系统"
|
|
@echo ""
|
|
@echo "🏗️ 跨架构构建:"
|
|
@echo " make build-cross-all # 构建所有架构的二进制文件"
|
|
@echo ""
|
|
@echo "🔧 直接使用 build-rustfs.sh 脚本:"
|
|
@echo " ./build-rustfs.sh --help # 查看脚本帮助"
|
|
@echo " ./build-rustfs.sh --no-console # 构建时跳过 console 资源"
|
|
@echo " ./build-rustfs.sh --force-console-update # 强制更新 console 资源"
|
|
@echo " ./build-rustfs.sh --dev # 开发模式构建"
|
|
@echo " ./build-rustfs.sh --sign # 签名二进制文件"
|
|
@echo " ./build-rustfs.sh --platform x86_64-unknown-linux-musl # 指定目标平台"
|
|
@echo " ./build-rustfs.sh --skip-verification # 跳过二进制验证"
|
|
@echo ""
|
|
@echo "💡 build-rustfs.sh 脚本提供了更多选项、智能检测和二进制验证功能"
|
|
|
|
.PHONY: help-docker
|
|
help-docker:
|
|
@echo "🐳 Docker 多架构构建帮助:"
|
|
@echo ""
|
|
@echo "🚀 推荐使用 (新的 docker-buildx 方式):"
|
|
@echo " make docker-buildx # 构建多架构镜像(不推送)"
|
|
@echo " make docker-buildx-push # 构建并推送多架构镜像"
|
|
@echo " make docker-buildx-version VERSION=v1.0.0 # 构建指定版本"
|
|
@echo " make docker-buildx-push-version VERSION=v1.0.0 # 构建并推送指定版本"
|
|
@echo ""
|
|
@echo "🏗️ 单架构构建:"
|
|
@echo " make docker-build-production # 构建生产环境镜像"
|
|
@echo " make docker-build-source # 构建源码构建镜像"
|
|
@echo ""
|
|
@echo "🔧 辅助工具:"
|
|
@echo " make build-cross-all # 构建所有架构的二进制文件"
|
|
@echo " make docker-inspect-multiarch IMAGE=xxx # 检查镜像的架构支持"
|
|
@echo ""
|
|
@echo "📋 环境变量 (在推送时需要设置):"
|
|
@echo " DOCKERHUB_USERNAME Docker Hub 用户名"
|
|
@echo " DOCKERHUB_TOKEN Docker Hub 访问令牌"
|
|
@echo " GITHUB_TOKEN GitHub 访问令牌"
|
|
@echo ""
|
|
@echo "💡 更多详情请参考项目根目录的 docker-buildx.sh 脚本"
|