Using Ceph and MinIO for Scalable Petabyte Data Storage

Here’s a practical guide to deploying, configuring, and operating open source object storage at scale.

Many organisations will run out of single-server storage before they run out of budget when using cloud storage to deal with large training sets, log archives, media libraries, or scientific datasets. Once you reach the petabyte scale, object storage is the way to go. Traditional file systems and block storage can be replaced by scalable flat namespaces with HTTP access.

Many people use MinIO and Ceph together. Ceph is good for large storage and MinIO for fast access. Ceph contains a lot of complex object storage features and can be used with its full set of features, or you can use MinIO with Ceph as a backend. MinIO can also be used with Ceph when you need a scalable storage system with complex features.

Ceph has many benefits:

• Block, object, and file storage don’t require running separate systems.
• It is built to scale out and can run on commodity hardware.
• It automatically repairs by re-replicating the data.
• You avoid vendor lock-in. The fully open-source (LGPL) model eliminates the ongoing per-TB costs associated with purchasing proprietary SANs or large-scale cloud storage solutions.

The advantages of MinIO are:

• S3 API compatibility: Has a drop-in replacement for AWS S3, which means existing tools (boto3, aws-cli, Spark, dbt, training frameworks) work unmodified.
• Simplicity: A single binary, minimal configuration, and fast time-to-first-bucket compared to Ceph’s more involved cluster setup.
• Performance: Optimised for high-throughput read/write, which matters for GPU training jobs that are I/O-bound on dataset loading.
• Erasure coding is built in: MinIO handles data protection natively without needing a separate storage backend.

Both work well together — Ceph is the long-lasting storage software, while MinIO instances can either be used as the S3 gateways or run on their own to help with training. The layout gives the scaling and long-lasting software of Ceph along with the S3 and simple operations software of MinIO along the edge.

The prerequisites for installing Ceph are:

• 3+ nodes recommended for production (minimum for quorum and replication); can be tested on a single host with cephadm in a lab configuration.
• Linux (Ubuntu 22.04/24.04 or RHEL/CentOS 9 recommended).
• Raw block devices for OSDs (Object Storage Daemons) — do not use pre-partitioned/formatted disks.
  Container runtime (Podman or Docker) — modern Ceph deploys via cephadm, which runs services in containers.
• Time sync across nodes (chrony/NTP) — clock drift breaks cluster quorum.

A dedicated storage/cluster network is strongly recommended in addition to the public network.

To install MinIO, you require:

• Linux, Windows, or macOS host(s); production deployments should use Linux.
• Raw or XFS-formatted drives (XFS recommended for MinIO’s erasure-coded backend).
• At least four drives per node for distributed erasure coding (minimum recommended for testing).
• mc (MinIO Client) for administration and bucket management.
• Open ports: 9000 (API) and 9001 (Console) by default.

Installation

You can install Ceph (via cephadm, single-node lab cluster) as follows:

# Install prerequisites
sudo apt update && sudo apt install -y podman chrony curl

# Download and install cephadm
curl --silent --remote-name --location \
https://download.ceph.com/rpm-18.2.0/el9/noarch/cephadm
chmod +x cephadm
sudo ./cephadm add-repo --release reef
sudo ./cephadm install

# Bootstrap the cluster (run on the first/admin node)
sudo cephadm bootstrap --mon-ip <MON_IP_ADDRESS>

# Add OSDs from all available raw devices
sudo ceph orch apply osd --all-available-devices

# Enable the RGW (S3-compatible) service
sudo ceph orch apply rgw myrealm myzone --placement=”1 <HOSTNAME>”

The bootstrap step outputs a dashboard URL, admin credentials, and a ceph.conf plus keyring you’ll use for subsequent ceph CLI commands.

To install MinIO (single-node, distributed-ready), use the following code:

# Download the MinIO server binary
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
sudo mv minio /usr/local/bin/

# Create data directories (example: 4 drives)
sudo mkdir -p /mnt/disk{1,2,3,4}/minio

# Install the MinIO Client (mc)
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
sudo mv mc /usr/local/bin/

For a multi-node distributed deployment, MinIO is started identically on each node, pointing at the full set of endpoints:

minio server http://node{1...4}.example.com:9000/mnt/disk{1...4}/minio \
--console-address “:9001”

Configuration of environment variables

MinIO is configured primarily through environment variables, which is convenient for containerized and Kubernetes deployments.

# Root credentials (required)
export MINIO_ROOT_USER=admin
export MINIO_ROOT_PASSWORD=’ChangeThisToAStrongPassword!’

# Region (optional, but recommended for S3 API compatibility)
export MINIO_REGION_NAME=us-east-1

# Enable Prometheus metrics without authentication (internal networks only)
export MINIO_PROMETHEUS_AUTH_TYPE=public

# KMS / server-side encryption (optional)
export MINIO_KMS_SECRET_KEY=’my-minio-key:BASE64_ENCODED_32_BYTE_KEY’

# Browser console toggle
export MINIO_BROWSER=on

For Ceph, most cluster-wide configuration lives in ceph.conf and the Ceph config database (ceph config set …) rather than environment variables, but the following are commonly set at the shell level for CLI convenience:

export CEPH_CONF=/etc/ceph/ceph.conf
export CEPH_KEYRING=/etc/ceph/ceph.client.admin.keyring

Once the installation of Ceph is complete, you can verify it like this:

# Overall cluster health
sudo ceph -s
# Expect: health: HEALTH_OK

# Check OSD status
sudo ceph osd tree

# Check the RGW (S3) endpoint is responding
curl http://<RGW_HOST>:80

MinIO can be verified as follows:

# Start the server (foreground, for verification)
minio server /mnt/disk{1...4}/minio --console-address “:9001”

# In another terminal, configure an mc alias
mc alias set local http://localhost:9000 admin ‘ChangeThisToAStrongPassword!’

# Verify connectivity and list buckets
mc admin info local

A healthy mc admin information response reports all drives online and cluster status as ‘ok’; a healthy ceph -s reports HEALTH_OK with all PGs (placement groups) active+clean.

Configuration

Let’s look at a ceph pool and erasure coding setup. We use an example of an 8PB archive pool using erasure coding for storage efficiency.

# Create an erasure-coded profile (6 data chunks, 3 parity chunks)
sudo ceph osd erasure-code-profile set ec-6-3 k=6 m=3

# Create the pool using that profile
sudo ceph osd pool create archive-pool 128 128 erasure ec-6-3

# Enable the pool for RGW/object storage use
sudo ceph osd pool application enable archive-pool rgw

MinIO bucket policy and lifecycle configuration can be done as follows:

# Create a bucket
mc mb local/training-datasets

# Set a bucket policy (public read for a specific prefix, private otherwise)
mc anonymous set download local/training-datasets/public-samples

# Configure lifecycle rules (e.g., expire temp uploads after 7 days)
mc ilm rule add local/training-datasets \
--expire-days 7 \
--prefix “tmp/”

# Enable versioning for reproducibility of training data snapshots
mc version enable local/training-datasets

Example configuration for ML training

A common pattern for large-scale ML training is to store sharded datasets (e.g., WebDataset .tar shards or Parquet files) in MinIO or Ceph RGW, and stream them directly into training jobs without staging to local disk.

Bucket layout is:
training-datasets/
├── imagenet-1k/
│ ├── train/shard-00000.tar ... shard-01023.tar
│ ├── val/shard-00000.tar ... shard-00063.tar
│ └── metadata.json
└── checkpoints/
└── run-2026-07-01/

Distributed training client configuration

This has PyTorch + boto3-style access, streaming shards directly from object storage:

import os

os.environ[“AWS_ACCESS_KEY_ID”] = “training-service-account”
os.environ[“AWS_SECRET_ACCESS_KEY”] = “REDACTED”
os.environ[“AWS_ENDPOINT_URL”] = “http://minio.internal:9000”

# WebDataset-style streaming from object storage into the training loop
import webdataset as wds

url = “pipe:mc cat local/training-datasets/imagenet-1k/train/shard-{00000..01023}.tar”
dataset = wds.WebDataset(url).decode(“pil”).to_tuple(“jpg”, “cls”)

Recommended tuning for training throughput

Optimise petabyte-scale training data storage on MinIO or Ceph by deploying erasure coding instead of full replication to maximise capacity efficiency for read-heavy workloads. You can prevent data-loading bottlenecks by co-locating storage nodes with GPU nodes on a high-bandwidth (25/100 GbE) network fabric. Minimise object storage request overhead by sharding datasets into 100MB-1GB .tar files following the WebDataset convention. Finally, implement client-side NVMe caching to handle frequently reused epochs when memory capacity is exceeded.

Case study

We look at a mid-sized AI research lab, with a target of 3PB of training data, coming from an existing capacity of 200TB. Previously, training datasets were accommodated by a single large NFS server. With the passage of time, lab data began to accumulate in versions (multiples of checkpoints and copies of the augmented, processed, and unprocessed corpora) to the point that it reached 200TB. From that point onwards, the lab began to experience degraded NFS performance, particularly during large-scale, multi-GPU, distributed training. Data loader workers began to spend more time waiting for I/O than on GPU compute.

So this is what they did:

• Deployed a 5-node Ceph cluster (60 OSDs total) as the durable long-term store for all raw and processed datasets, using erasure coding (k=6, m=3) to balance durability against the 66% storage overhead of 3x replication.
• Deployed 3 MinIO nodes with local NVMe drives, positioned as a fast S3-compatible cache/gateway layer directly on the GPU training cluster’s network segment.
• Used Ceph’s RGW multisite replication to periodically sync ‘hot’ datasets (the current training epoch’s shards) into the MinIO layer, while cold/archival data stayed exclusively in Ceph.
• Migrated all training scripts to use the S3 API (via boto3/s3fs) pointed at the MinIO endpoint, eliminating NFS entirely from the training path.

The outcome was:

• Dataloader wait time during training dropped substantially once data was served from MinIO’s NVMe-backed nodes on the same network segment as the GPUs, removing the previous I/O bottleneck.
  Storage costs stayed roughly flat year-over-year despite data volume growing more than 10x, since erasure-coded Ceph on commodity hardware scaled far more cheaply than adding proprietary NAS capacity.
• The self-healing behaviour of both Ceph and MinIO meant several individual disk failures over the following year required zero manual data recovery — the systems rebuilt redundancy automatically.
• The team gained the ability to version and roll back entire dataset snapshots (via MinIO bucket versioning) after a mislabelled data ingestion incident, something the prior NFS setup had no equivalent for.

This pattern — Ceph as the durable multi-petabyte backend and MinIO as a fast S3-native layer close to compute — has become a common reference architecture for AI infrastructure teams operating at this scale without relying on public cloud storage.

At the petabyte scale, Ceph and MinIO address different aspects of the same problem. Ceph’s storage enables ease-of-use, multi-protocol, seamless scaling to large clusters, and self-healing storage, while MinIO provides S3-native, high-speed storage to support ML training with minimal latency. When Ceph is employed as the long-term storage solution, and MinIO as a fast S3 gateway collocated with processing, durability and cost efficiencies are realised without loss of the performance necessary to support modern training pipelines.

The process to fully scale out each system may be slow. We can start with a small Ceph cluster and a single MinIO instance to confirm our setup is correct. We then ensure everything is healthy and scale out nodes. We keep expanding as the data grows. As the software is open source, it lets us keep that same layout with some additional hardware as the capacity of our research lab grows from 200 TB to multiple petabytes