Skip to content

cephalobox - a podman cephadm dev env - #112

Open
nizamial09 wants to merge 3 commits into
mainfrom
cephalobox
Open

cephalobox - a podman cephadm dev env#112
nizamial09 wants to merge 3 commits into
mainfrom
cephalobox

Conversation

@nizamial09

@nizamial09 nizamial09 commented Aug 1, 2026

Copy link
Copy Markdown
Member

it is still privileged container (not anymore, gave it a set of focused permission) but its not typical podman in podman. instead cephadm manages services in the container's host namespace.

I am trying to build it by bundling the quay.ceph.io images inside the cephalobox image so that bootstrap will be more faster

Cephalobox

A podman based cephadm deployment for deploying ceph. Instead of a full podman in podman, it leverages podman directly on container's host with privileged
capabilities, so that cephadm can natively deploy and manage services.

Quirks

  • OSDs are not managed by cephadm. instead deployed using ceph-osd binary.

Deployment

Strongly advised to use podman compose. docker compose is too much..

  1. Build cephadm binary in your local. Since this favors mostly local development, generate the cephadm binary in your src/cephadm directory inside ceph.

  2. Also build the front-end assets inside the src/pybind/mgr/dashboard/frontend folder in ceph.

  3. Populate the CEPH_REPO_DIR in the .env file to your local directory so that it reads your local code.

  4. use the image you want, supports main, umbrella and tentacle. Prebundling the image so that we don't need to pull it everytime we start a cluster.
    Provide the correct image in .env file's CEPHALOBOX_IMAGE. defaults to rhcsdashboard/cephalobox:main.

  5. Pull the image

podman compose pull cephalobox
  1. Start the container
podman compose up -d cephalobox
  1. Check the logs. By default, it pushes out cephadm's bootstrap log and also the ceph-mgr logs.
podman compose logs -f cephalobox
  1. to destroy
podman compose down -v

Limitation

  • Impossible to run nvmeof service because of the way it is built.
  • Can't create osds at all. Maybe possible to add hosts, but not tested.

Co-Authored-by: Gemini

@nizamial09
nizamial09 force-pushed the cephalobox branch 6 times, most recently from e8fbd70 to 7ba6eef Compare August 1, 2026 16:35
it is still privileged container but its not typical podman in podman.
instead cephadm runs manages services in the container's host namespace.

I am trying to build it by bundling the quay.ceph.io images inside the
cephalobox image so that bootstrap will be more faster

Signed-off-by: Nizamudeen A <nia@redhat.com>
```
2026-08-03 08:24:26,471 7fd7b49e4e00 ERROR ERROR: Not a directory: /etc/sysctl.d
```

and couple of other cephadm issues on cephadm check-host

Signed-off-by: Nizamudeen A <nia@redhat.com>
adding focused privileges instead of `privilged: true` so that the
container will only have what it needs and not more.

and some general code cleanups

Signed-off-by: Nizamudeen A <nia@redhat.com>
@nizamial09
nizamial09 force-pushed the cephalobox branch 2 times, most recently from ebf9c8d to 8d1f813 Compare August 4, 2026 10:04
@nizamial09

nizamial09 commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

Things to improve

  • start with no shared ceph folder
  • accept custom image and pull them inside
  • adding multiple hosts
  • multiple clusters

Wild goals

  • osds managed by cephadm
  • nvmeof

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant