The AI file system for object storage
Runix FS puts a POSIX file system and a multi-tier distributed cache in front of the S3-compatible storage you already run. Training jobs, inference servers and agents read the same data through a local path, with no copies and no rewritten I/O code.
See the Runix FS walkthrough → On our examples site; the access key comes from the Runix team.
Built on Curvine, an open-source CNCF Sandbox project under the Apache 2.0 licence. Runix FS source and development · Read the launch post.
At a glance
- Interfaces
- FUSE, Hadoop-compatible client, Java, Python and Rust SDKs, Kubernetes CSI
- Storage
- S3-compatible object storage; paths map one-to-one to object keys
- Cache
- Memory, SSD and HDD tiers on the workers
- Metadata
- Masters replicated with Raft
- Mount modes
- Cache mode, or file-system mode with background sync
- Runs in
- Your AWS, Google Cloud or Azure account
- Built on
- Curvine: Apache 2.0, CNCF Sandbox
- Status
- Early access, per deployment
# 1. Attach the bucket to the namespace $ bin/cv mount s3://training-data/imagenet \ /training-data/imagenet \ --write-type cache_mode \ -c s3.endpoint_url=https://s3.us-east-1.amazonaws.com \ -c s3.region_name=us-east-1 \ -c s3.credentials.access=$AWS_ACCESS_KEY_ID \ -c s3.credentials.secret=$AWS_SECRET_ACCESS_KEY # 2. Expose it as a local directory $ bin/curvine-fuse.sh start --mnt-path /mnt/curvine # 3. Your training code reads a path, as before $ ls /mnt/curvine/training-data/imagenet train/ val/ labels.csv
02 · Example
mount an existing bucket
Three commands: attach the bucket to the namespace, expose it as a local directory, and read it as a path. Nothing in the training code changes.
Published results
- 5 billion
- small files in one cluster, the project's stated design capacity
- 10,000
- agent pods, each with its own volume, on 99 EKS nodes in the project's test
- 2.7×
- the largest time-to-first-token speed-up in an AWS Machine Learning Blog walkthrough on SageMaker HyperPod
- 9.5 GiB/s
- sequential read at 32 threads, as published in the project's README
Figures published by the Curvine project and by AWS, not Runix service levels. Sources are linked further down this page.
03 · Capabilities
Object-storage economics, local-file semantics
Object storage is where AI data lives because it is cheap and durable. It is also slow to list, slow to open and poor at renames. Runix FS puts a file system and a cache in front of the bucket, and each mount picks how writes reach it: read-heavy data keeps the bucket as the source of truth, write-heavy work completes in Runix FS first and is synchronised to the bucket in the background.
POSIX, not a new API
A FUSE mount behaves like a local directory: open, read, write, rename and list. Tools such as git and inotify work against it unmodified.
A cache in three tiers
Workers hold memory, SSD and HDD tiers and fill the fastest one with room first. Anything not cached is read straight from the bucket, and the cache fills in the background.
One namespace, several ways in
POSIX through FUSE, a Hadoop-compatible client for Spark and Flink, the native cv command line, and SDKs for Java, Python and Rust, all on the same files. FUSE, CSI or an SDK: which to use.
Kubernetes-native volumes
A CSI driver provisions ReadWriteMany volumes by creating a directory, not by calling a cloud API, so provisioning keeps up when thousands of pods start at once.
Your bucket stays readable
File paths map one-to-one to object keys. What is in your bucket keeps its original layout and stays readable by anything else, whether or not the cluster is running.
Built to stay up
Metadata is replicated across masters with Raft, cached blocks can be replicated across workers, and a built-in web UI and metrics show every component.
04 · How it works
Where it sits in your stack
Between the workloads that read data and the bucket that keeps it. Metadata goes to the masters; data is served by the workers, which read from the bucket on a miss and write back to it, directly or through a background sync depending on the mount mode.
Workloads
Interfaces
Runix FS cluster
Your storage
05 · Integration
Your code does not change
Four examples against the same namespace. Every command below is from the Curvine documentation; hosts and bucket names are placeholders.
Warm the cache before the job starts
Load a path into the cache ahead of time, and watch the job finish.
$ bin/cv load /training-data/imagenet/train/shard-00001.tar --watch $ bin/cv fs ls /training-data/imagenet $ bin/cv report
Read it from Python
No SDK: a mounted path is a path.
from pathlib import Path
root = Path("/mnt/curvine/training-data/imagenet")
for shard in sorted(root.glob("train/*.tar"))[:2]:
print(shard.name, len(shard.read_bytes()))
Give every pod a volume
A StorageClass for the CSI driver; each claim becomes a directory.
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: curvine-sc provisioner: curvine volumeBindingMode: Immediate allowVolumeExpansion: true parameters: master-addrs: "m0:8995,m1:8995,m2:8995" fs-path: "/agents" path-type: "DirectoryOrCreate"
Manage the cache, not the bucket
List the mounts, check the buckets behind them are reachable, and release cache space without deleting anything from the bucket.
$ bin/cv mount $ bin/cv mount --check $ bin/cv fs free -r /training-data/imagenet
06 · Workloads
Built for the workloads that stall on I/O
GPUs are the expensive part of an AI cluster, and they sit idle while data crosses the network one object at a time. These are the places where it shows up first.
Model training
Datasets and checkpoints are cached next to the GPU nodes, so epochs after the first read from the cache instead of the bucket, and checkpoints land on a file system instead of a multipart upload.
LLM inference
A shared tier for the KV cache that outlives a single GPU's memory, so a prompt prefix computed on one replica can be reused on another.
A post on the AWS Machine Learning Blog, written by AWS solutions architects with engineers from OPPO, where Curvine was started, walks through this on SageMaker HyperPod with Curvine as the shared tier and reports up to 2.7× faster time to first token at 2,500 tokens. Read the AWS post
AI agent platforms
Every agent wants its own writable workspace. Block volumes run into per-node attachment limits long before a node runs out of CPU; a directory on a shared file system does not.
The Curvine project provisioned 10,000 volumes for 10,000 agent pods on 99 EKS nodes, all bound and running, with a storage cluster of four pods. Read the write-up
Model and artifact distribution
Weights and artifacts are pulled once from the bucket and served from the cache to every node that starts a replica, instead of each node downloading the same files.
Analytics on the lake
Spark and Flink read hot tables through the Hadoop-compatible client, with the cache absorbing repeated scans of the same files.
07 · Results
What the project has measured
These figures are published in the Curvine project's README. They are not a Runix service level. During early access we measure on your data, your instances and your access pattern.
| Measure | Result | Conditions |
|---|---|---|
| Metadata operations | ||
| Create | 19,985 ops/s | Concurrency of 40 |
| Open | 60,376 ops/s | Concurrency of 40 |
| Rename | 43,009 ops/s | Concurrency of 40 |
| Delete | 39,013 ops/s | Concurrency of 40 |
| Throughput | ||
| Sequential read, 256 KiB blocks | 9.5 GiB/s | 32 threads |
| Random read, 256 KiB blocks | 7.8 GiB/s | 32 threads |
| Scale | ||
| Small files in one cluster | 5 billion | Stated design capacity |
Source: the Curvine README. The project's separate benchmark pages report other runs, with somewhat different results, and describe the test setup.
08 · Open source
Open source, or run with us
The file system is the same. What Runix adds is the part a production team needs around it: someone to size it, deploy it, answer for it, and sign for it.
| Curvine, open source | Runix FS | |
|---|---|---|
| Licence | Apache 2.0 | Apache 2.0 core |
| Where it runs | Wherever you run it | Your AWS, Google Cloud or Azure account |
| Deployment | Self-managed | Sized and deployed with our engineers |
| Support | Community, on GitHub | Our team, 1 business day response |
| Contract | None | Runix AI Inc, MSA and DPA on request |
| Price | Free | Quoted per deployment in early access |
09 · Specifications
The details a review asks for
One table, the same shape on every product page, kept current with the pages it points to.
| Item | Value | Where it is documented |
|---|---|---|
| Interfaces | FUSE mount; Hadoop-compatible client; Java, Python and Rust SDKs; Kubernetes CSI driver | Quick start |
| Object storage | S3-compatible; the bucket keeps its layout and stays independently readable | This page |
| Cache tiers | Memory, SSD and HDD, written to the fastest available tier first | This page |
| Metadata | Masters replicated with Raft | This page |
| Mount modes | Cache mode: the bucket is the source of truth and writes pass through. File-system mode: operations complete in the cluster, then sync to the bucket | Quick start |
| Kubernetes | The CSI driver provisions ReadWriteMany volumes by creating a directory, not by calling a cloud API | Quick start |
| Source | Public Runix FS fork of Curvine under Apache 2.0 | Source and development |
| Deployment | Sized and deployed with you, in your own cloud account; supported under contract with Runix AI Inc | This page |
| Published results | The Curvine project's own figures, labelled as such; not a Runix service level | Published results |
| Pricing | Quoted per deployment during early access | Pricing |
10 · Works with
The layers on either side
Runix FS hands off to its neighbours in the stack and works on its own. The same block sits on every product page, with the neighbours changed.
Above · 02 Data
Runix Data and Runix Pipeline
Raw material is read through the file system before it is cleaned.
Explore Runix Data →Above · 03 Models
Runix Models
Training reads datasets and checkpoints through one path instead of waiting on the bucket.
Explore Runix Models →Below · 00 Compute & energy
Your own cloud
GPUs, object storage and Kubernetes in your account. The file system runs on them and replaces none of them.
See the stack →11 · How to start
How early access works
Three steps, each with a person on the other end: Runix FS is not self-serve during early access.
01Tell us about your data
Where it lives, roughly how big it is and what reads it: training jobs, inference servers, agents or analytics. We reply within one business day.
02Size it together
How much of the working set the cache should hold, which tiers the workers get, which mounts run in cache mode and which in file-system mode. Pricing is quoted per deployment.
03Deploy into your account
The cluster runs in your own AWS, Google Cloud or Azure account, next to your GPUs and your bucket, and our team supports it afterwards under a contract with Runix AI Inc.
12 · Questions
Common questions
How is Runix FS related to Curvine?
Runix FS is built on Curvine, an open-source distributed file system and cache released under Apache 2.0 and accepted as a Cloud Native Computing Foundation Sandbox project. Runix deploys it into your cloud account and supports it under a contract with a US company.
Where does my data live?
In your own object storage. File paths map one-to-one to object keys, so what is in the bucket keeps its layout and stays readable without Runix FS. In cache mode the bucket is the source of truth and writes go straight to it; in file-system mode, writes complete in Runix FS first and reach the bucket through a background sync.
Do we have to change our code?
Usually not. Applications read and write through a FUSE mount as if it were a local directory, so training scripts keep working as they are. Hadoop-based jobs such as Spark use the Hadoop-compatible client, which takes configuration rather than code changes.
Which storage does it work with?
Amazon S3 and any S3-compatible store such as MinIO, plus Google Cloud Storage, Azure Blob Storage, Alibaba Cloud OSS and HDFS. The cluster runs on Kubernetes, on virtual machines or on bare metal.
How is it priced?
During early access, pricing is quoted per deployment. Tell us how much data you have and what reads it, and we come back with a quote within one business day.
Can we buy it through AWS Marketplace?
Not yet. A listing is in preparation. Until then you contract directly with Runix AI Inc and are invoiced in USD.
Put a file system in front of your bucket
Tell us where your data lives, how big it is and what reads it. We reply within one business day with a deployment plan.
Request early access