The old myth that "containers are only for stateless workloads" died years ago. Today, production Kubernetes environments routinely host mission-critical stateful workloads, from transactional databases and Kafka clusters to AI/ML feature stores and virtual machines.
When teams move stateful applications into production on Red Hat OpenShift on IBM Cloud, they quickly encounter a fundamental storage architecture decision: Should they use cloud-managed block volumes, or deploy in-cluster software-defined storage with OpenShift Data Foundation (ODF)?
Treating storage like an afterthought works fine until a database pod restarts on a new node and sits in CrashLoopBackOff while you explain to leadership what an attach-detach timeout is. As the product manager working on container storage at IBM Cloud, I walk engineering teams through this choice every week. Both storage architectures are mature, production-proven options, but they optimize for fundamentally different operational priorities.
The Architectural Divide: Out-of-Cluster Managed vs. In-Cluster Ceph
To make the right choice, you first have to understand where the storage intelligence lives in each model.
1. Cloud-Managed Volumes (IBM Cloud Block Storage)
IBM Cloud Block Storage operates outside your Kubernetes compute nodes. When a stateful pod requests a PersistentVolumeClaim (PVC), the IBM Cloud Container Storage Interface (CSI) driver calls the VPC storage API to provision a dedicated, NVMe-backed volume on the cloud storage fabric and attaches it directly to the host hypervisor.
Key characteristics:
- Minimal worker node overhead: The storage control plane runs on IBM Cloud infrastructure, consuming zero application CPU or memory for storage management.
- Decoupled volume lifecycle: Volume persistence is managed independently of worker node lifecycles, allowing stateful pods to reattach volumes across node drain, upgrade, or replacement events.
- ReadWriteOnce (RWO) scope: A standard block volume attaches to a single compute instance at a time.
- Pay-per-GB efficiency: You provision and pay only for the exact storage capacity and IOPS your pods consume.
2. Software-Defined In-Cluster Storage (OpenShift Data Foundation)
OpenShift Data Foundation (ODF) takes the opposite approach. Powered by open-source Ceph, Rook, and NooBaa, ODF aggregates the physical NVMe drives attached to your bare-metal worker nodes into a single, unified, software-defined storage pool managed by Kubernetes operators.
Key characteristics:
- Multi-modal unified storage: One single deployment provides block storage (Ceph RBD), shared file storage (CephFS), and S3-compatible object storage (RADOS Gateway / NooBaa).
- High-throughput distributed storage: Backed by local NVMe drives on each worker node, replicating data synchronously across OSD daemons over the high-speed cluster network.
- 3-way replication resilience: Ceph replicates every data block across three failure domains. If a worker node crashes, the cluster continues serving I/O without downtime.
- Shared-access storage options: Exposes shared-access storage classes for both multi-pod filesystems (CephFS) and virtualization-focused block workloads.
Access Patterns and the Virtualization Imperative
The most immediate technical differentiator between these two storage models comes down to access modes and application architecture.
When Single-Node Block (RWO) Is Enough
For standard database engines running in high availability (such as a PostgreSQL or MySQL StatefulSet with a primary and read-replicas), each replica manages its own dedicated storage volume. Replica synchronization happens at the application database layer rather than the storage layer.
For these workloads, IBM Cloud Block Storage using the ibmc-vpc-block-10iops-tier or dynamic Storage Defined Performance (SDP) profile is exceptional. You get guaranteed IOPS, volume snapshots, and online capacity expansion without running storage daemons on your nodes.
Where ODF Becomes Essential: RWX and OpenShift Virtualization
Traditional cloud-managed block volumes are exposed as single-attached ReadWriteOnce (RWO) storage through Kubernetes, making them unsuitable for workloads that require simultaneous multi-node access. If your workload requires concurrent writes from multiple pods (such as shared data ingestion pipelines, machine learning training sets, or CMS asset pools), ODF provides native POSIX-compliant CephFS with ReadWriteMany (RWX) access across all nodes.
Even more critical is Red Hat OpenShift Virtualization. When running virtual machines on OpenShift alongside containers, migrating running VMs across physical hosts without downtime (the cloud-native equivalent of VMware vMotion) requires shared-access storage.
Standard cloud block storage cannot support live migration because the hypervisor on the destination node cannot attach a volume that is still locked to the source node. OpenShift Virtualization addresses this by consuming Ceph RBD in raw block mode (volumeMode: Block) rather than as a mounted filesystem. ODF provides validated storage classes like ocs-storagecluster-ceph-rbd-virtualization with shared access patterns. Destination nodes mount the disk image before migration begins, transferring memory states with zero guest downtime.
The Honest Tradeoff: Sizing, Resource Tax, and Blast Radius
Engineering is always a study in tradeoffs. ODF is powerful, but deploying it blindly for every workload is a major architectural mistake.
1. The Bare-Metal and Node Footprint Requirement
ODF requires bare-metal worker nodes with local NVMe drives. The absolute architectural minimum is 3 worker nodes, with 6 nodes recommended for production clusters so you can drain a node for maintenance without degrading replication.
If you are running a small cluster with 3 Virtual Server Instances (VSIs) hosting a few microservices and a lightweight cache, spinning up 3 or 6 bare-metal nodes just to run Ceph makes zero financial sense. Managed VPC Block allows you to run small, flexible VSI instances and attach 20 GB or 100 GB volumes on demand.
2. The 3-Way Storage Multiplier and Compute Overhead
Because ODF uses a default replica factor of 3 for high availability, 30 TB of raw NVMe storage yields roughly 10 TB of usable application capacity. Additionally, Ceph OSDs, Monitors, and Metadata Servers run as pods inside your cluster, consuming worker node memory and CPU cycles.
3. Capacity Thresholds
In Ceph, you must actively govern capacity. When an ODF cluster reaches ~80% capacity (nearfull), Ceph begins throttling write operations to prevent disk starvation. At ~85% capacity (full), Ceph blocks new writes entirely to protect cluster consistency. (Ceph is fantastic, but when it hits 85% full and starts refusing writes to protect data integrity, it will definitely get your team's undivided attention.)
With managed VPC Block, each volume is completely isolated. If volume A fills up, volume B is unaffected, and you can resize any attached volume online with a single pvc spec update.
4. Distributed Systems Operational Reality
ODF embeds a full distributed storage system inside your cluster. Even though IBM Cloud automates deployment and day-2 operator lifecycle management on ROKS, platform teams must still understand cluster storage health, data rebalancing behavior during node maintenance, and proactive capacity planning.
Decision Matrix: How to Choose
When evaluating storage for your stateful workloads on IBM Cloud, use this comparison framework:
Quick Decision Checklist
If you answer yes to any of these questions, evaluate OpenShift Data Foundation:
- Do your workloads require multi-pod shared file access (RWX) or in-cluster S3 object buckets?
- Are you running OpenShift Virtualization VMs requiring zero-downtime live migration?
- Do you require a consolidated storage platform for IBM Cloud Paks or AI/ML pipelines?
- Are you already running or committed to bare-metal OpenShift worker nodes?
If you answered no to all of the above, start with IBM Cloud Block Storage—it gives you lightweight, fully isolated, pay-as-you-go persistence with zero storage cluster operational overhead.
The Managed Platform Advantage
On generic cloud setups or DIY Kubernetes, operating Ceph is notoriously complex. Cluster operators often spend days configuring CRUSH maps, tuning OSD placement, and managing daemon recovery during node upgrades.
This is where the platform architecture matters. On competing hyperscaler Kubernetes platforms (such as AKS, EKS, or GKE), ODF is not generally available as a native managed platform add-on. On ROSA and ARO, customers must manually install and maintain ODF operators.
On Red Hat OpenShift on IBM Cloud (ROKS), ODF is delivered as a fully managed IBM Cloud add-on. IBM SREs handle operator deployment, daemon lifecycle, health monitoring, and version patching, allowing platform teams to consume Ceph storage classes directly via standard Kubernetes manifests without operating the underlying storage infrastructure.
Summary
Choose IBM Cloud Block Storage when you want lightweight, pay-as-you-go, isolated block volumes for databases running on flexible virtual server instances with zero storage daemon overhead.
Choose OpenShift Data Foundation when you run bare-metal OpenShift clusters that demand high-throughput local NVMe performance, multi-modal storage consolidation (block + file + object), or live migration for OpenShift Virtualization VMs.
How does your team model the line between cloud-managed storage volumes and software-defined in-cluster storage for your Kubernetes clusters?
