docs: replace open-source self-hosting pages with Fish Audio Enterprise guides #113
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,297 @@ | ||
| --- | ||
| title: "Air-gapped deployments" | ||
| description: "Mirror artifacts into a disconnected network, account for usage offline, and prove zero egress" | ||
| icon: "network-wired" | ||
| --- | ||
|
|
||
| The offline delivery forms run with no outbound network access at all: model | ||
| assets are served from inside the deployment and usage is recorded to a local | ||
| signed ledger rather than a billing service. Getting there takes one preparation | ||
| step — moving the artifacts across the air gap — and one verification step. | ||
|
|
||
| | Delivery form | What has to cross the air gap | | ||
| | ------------- | -------------------------------------------------------------------------- | | ||
| | All-in-One | One container image. Model weights are already inside it. | | ||
| | Offline Helm | The chart archive, every image the chart references, and your values file. | | ||
|
|
||
| Everything below runs on a **connected staging machine** first, then on the | ||
| disconnected side. Only the staging machine needs | ||
| [registry access](/developer-guide/self-hosting/registry-access). | ||
|
|
||
| ## Mirror container images | ||
|
|
||
| On the connected machine, derive the image list from the chart you are about to | ||
| install, so the list always matches the release: | ||
|
|
||
| ```bash | ||
| CHART_REF='<chart-ref>' | ||
| CHART_VERSION='<chart-version>' | ||
|
|
||
| helm pull "$CHART_REF" --version "$CHART_VERSION" --destination ./transfer | ||
|
|
||
| helm template fish-audio ./transfer/*.tgz \ | ||
| --namespace fish-audio \ | ||
| --values values.yaml \ | ||
| | grep -oE 'image:[[:space:]]*"?[^"[:space:]]+' \ | ||
| | awk '{print $2}' | tr -d '"' | sort -u > images.txt | ||
| ``` | ||
|
|
||
| Review `images.txt`, then pull and pack the images. `zstd` keeps the transfer | ||
| archive small; `gzip` works too if `zstd` is not available on both sides. | ||
|
|
||
| ```bash | ||
| xargs -n1 docker pull < images.txt | ||
|
|
||
| docker save $(tr '\n' ' ' < images.txt) | zstd -T0 -o fish-audio-images.tar.zst | ||
| ``` | ||
|
|
||
| For the All-in-One image the same pattern applies with a single reference: | ||
|
|
||
| ```bash | ||
| docker save '<all-in-one-image>' | zstd -T0 -o all-in-one.tar.zst | ||
| ``` | ||
|
|
||
| Transfer `fish-audio-images.tar.zst`, `images.txt`, the chart archive from | ||
| `./transfer/`, and your values file across the air gap by whatever means your | ||
| policy allows. | ||
|
|
||
| ## Load on the disconnected side | ||
|
|
||
| ```bash | ||
| zstd -dc fish-audio-images.tar.zst | docker load | ||
| ``` | ||
|
|
||
| For a Kubernetes install, push the loaded images into the registry your cluster | ||
| can reach: | ||
|
|
||
| ```bash | ||
| INTERNAL_REGISTRY='<your-internal-registry>' | ||
|
|
||
| while read -r image; do | ||
| target="$INTERNAL_REGISTRY/${image#*/}" | ||
| docker tag "$image" "$target" | ||
| docker push "$target" | ||
| done < images.txt | ||
| ``` | ||
|
|
||
| Then point the chart at your mirror by overriding each component's | ||
| `image.repository` in your values file, and confirm nothing still refers to an | ||
| external host before you install: | ||
|
|
||
| ```bash | ||
| helm template fish-audio ./fish-audio-chart.tgz \ | ||
| --namespace fish-audio \ | ||
| --values values.yaml \ | ||
| | grep -E 'image:' | sort -u | ||
| ``` | ||
|
|
||
| Install from the local chart archive exactly as described in | ||
| [Kubernetes deployment](/developer-guide/self-hosting/kubernetes#install), | ||
| substituting the archive path for the chart reference. The chart archive is | ||
| self-contained, so no chart repositories are contacted during the install. | ||
|
|
||
| For the All-in-One container, the loaded image is all you need — continue with | ||
| [Run](/developer-guide/self-hosting/all-in-one#run). | ||
|
|
||
| ## Model assets offline | ||
|
|
||
| The offline forms never download model weights at runtime. | ||
|
|
||
| | Form | How models are served | | ||
| | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ||
| | Offline Helm | `global.offlineMode: true` starts an in-cluster model warehouse that serves the bundled weights over an S3-compatible endpoint inside the namespace. Object paths are unchanged; only the endpoint moves. | | ||
| | All-in-One | Weights are baked into the image. | | ||
|
|
||
| Two things still come from you rather than from the network: | ||
|
|
||
| - **Reference voices.** Stage the archives yourself, as described in | ||
| [Kubernetes deployment](/developer-guide/self-hosting/kubernetes#stage-reference-voices) | ||
| or [All-in-One](/developer-guide/self-hosting/all-in-one#reference-voices). | ||
| This behavior is identical online and offline. | ||
| - **Timestamp alignment.** The forced aligner is not part of the offline bundle. | ||
| Contact Fish Audio if your air-gapped deployment needs word or segment timings. | ||
|
|
||
| ## Offline usage accounting | ||
|
|
||
| With no billing service to call, the deployment records every charge to a local, | ||
| signed, append-only ledger on persistent storage. No request is ever rejected for | ||
| billing reasons, and the record is tamper-evident and independently verifiable. | ||
|
|
||
| ### How tokens behave | ||
|
|
||
| - Any non-empty `Authorization: Bearer` token is accepted. Empty or missing still | ||
| returns 401. | ||
| - The token is recorded verbatim as the billing identity, so choose a stable, | ||
| distinct value per tenant. Tenants that share a token cannot be told apart in | ||
| the ledger. | ||
|
|
||
| ### Disk layout | ||
|
|
||
| Files are grouped one directory per UTC day, so a day archives by copying a single | ||
| directory: | ||
|
|
||
| ```text | ||
| offline-billing-ledger/ | ||
| 2026-06-11/ | ||
| 2026-06-11_<instance>_<nonce>.event.jsonl # append-only signed records | ||
| 2026-06-11_<instance>_<nonce>.manifest.json # signed per-file summary | ||
| 2026-06-12/ | ||
| ... | ||
| ``` | ||
|
|
||
| Each file is written by exactly one process, so replicas never interleave into the | ||
| same file even though they share the directory. Every file is an independent hash | ||
| chain: a new day, a restart, or a new replica starts a new file. | ||
|
|
||
| ### Record format | ||
|
|
||
| Each line is one record: a shared envelope plus a payload chosen by `event_type` | ||
| (`process_started`, `heartbeat`, `billing_event`, `process_stopping`). | ||
|
|
||
| | Envelope field | Meaning | | ||
| | ---------------------------------------- | ------------------------------------------------------------- | | ||
| | `version` | Ledger format version. | | ||
| | `instance_id`, `run_id` | Which instance and which process run wrote the record. | | ||
| | `seq` | 1-based and contiguous within the file. | | ||
| | `wall_time_utc_ms`, `wall_time_utc_date` | UTC timestamp and date. | | ||
| | `monotonic_ms_since_start` | Monotonic clock since process start, resistant to clock skew. | | ||
| | `prev_hash` | Hash of the previous record, `"0"` for the first. | | ||
| | `event_type` | Record discriminator. | | ||
| | `record_hash` | SHA-256 over all fields above. | | ||
| | `signature` | RSA-PSS-SHA256 over `record_hash`, base64. | | ||
|
|
||
| | `billing_event` field | Meaning | | ||
| | ----------------------------- | ----------------------------------------------------------- | | ||
| | `team_id` | The bearer token the request used. | | ||
| | `product` | The product consumed, for example `tts`. | | ||
| | `backend` | The model backend that served the request. | | ||
| | `quantity` | Units billed in this aggregation bucket. | | ||
| | `unit_price_usd_per_1m_chars` | Contract unit price, or `null` for unpriced products. | | ||
| | `amount_usd` | Exact decimal string computed from quantity and unit price. | | ||
|
|
||
| Charges accumulate in Redis per token, product, and backend, and a background task | ||
| flushes them roughly once a minute as one aggregated `billing_event` per bucket. A | ||
| failed write is retried rather than dropped. | ||
|
|
||
| When a file is finished it is sealed into a matching signed manifest carrying the | ||
| last record's hash and signature, the record counts, and the file's totals. | ||
| Sealing is automatic: yesterday's file is sealed shortly after the UTC day rolls | ||
| over, and a graceful shutdown seals the current one. Today's in-progress file has | ||
| no manifest yet. | ||
|
|
||
| ### Verify the ledger | ||
|
|
||
| Verification needs only the public key that Fish Audio supplies, plus `jq` and | ||
| `openssl`: | ||
|
|
||
| ```bash | ||
| PUB=offline-ledger-signing-key.pub.pem | ||
| LINE=$(head -n 1 '<ledger-file>.event.jsonl') | ||
|
|
||
| printf '%s' "$LINE" | jq -r .record_hash > /tmp/msg | ||
| printf '%s' "$LINE" | jq -r .signature | openssl base64 -d -A > /tmp/sig | ||
|
|
||
| openssl dgst -sha256 -verify "$PUB" \ | ||
| -sigopt rsa_padding_mode:pss -sigopt rsa_pss_saltlen:-3 \ | ||
| -signature /tmp/sig /tmp/msg | ||
| ``` | ||
|
|
||
| `Verified OK` confirms the record is authentic. Recompute the hash itself from the | ||
| record body and compare it to the `record_hash` field: | ||
|
|
||
| ```bash | ||
| printf '%s' "$LINE" | jq -c 'del(.record_hash,.signature)' | openssl dgst -sha256 | ||
| printf '%s' "$LINE" | jq -r .record_hash | ||
| ``` | ||
|
Comment on lines
+199
to
+205
There was a problem hiding this comment. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files | rg '(^|/)air-gapped\.mdx$|air-gap|air_gapped|air-gapped' || true
echo "== target excerpt =="
if [ -f developer-guide/self-hosting/air-gapped.mdx ]; then
nl -ba developer-guide/self-hosting/air-gapped.mdx | sed -n '160,225p'
fi
echo "== record_hash references =="
rg -n "record_hash|all fields above|Verified OK|openssl dgst|json|jq" developer-guide/self-hosting/air-gapped.mdx || trueRepository: fishaudio/docs Length of output: 269 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== target excerpt =="
awk '{printf "%6d\t%s\n", NR, $0}' developer-guide/self-hosting/air-gapped.mdx | sed -n '160,225p'
echo "== record_hash references =="
grep -n "record_hash\|all fields above\|Verified OK\|openssl dgst\|json\|jq" developer-guide/self-hosting/air-gapped.mdx || true
echo "== jq newline and key-order probe =="
python3 - <<'PY'
from subprocess import run, PIPE
samples = [
'{"record_hash":"abc","signature":"sig","payload":"x"}',
'{"signature":"sig","payload":"x","record_hash":"abc"}',
]
for s in samples:
p = run(["jq", "-c", '.'], input=s, text=True, stdout=PIPE, stderr=PIPE)
print(f"input={s!r}")
print(f"jq -c bytes={p.stdout!r} len={len(p.stdout)} endswith_newline={p.stdout.endswith('\\n)}")
PYRepository: fishaudio/docs Length of output: 5229 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== jq newline and key-order probe =="
python3 - <<'PY'
from subprocess import run, PIPE
samples = [
'{"record_hash":"abc","signature":"sig","payload":"x"}',
'{"signature":"sig","payload":"x","record_hash":"abc"}',
]
for s in samples:
p = run(["jq", "-c", '.'], input=s, text=True, stdout=PIPE, stderr=PIPE)
print("input={}".format(repr(s)))
print("jq -c bytes={}".format(repr(p.stdout)))
print("jq -c len={}".format(len(p.stdout)))
print("jq -c endswith_newline={}".format(p.stdout.endswith("\n")))
PY
echo "== jq del behavior using line above =="
python3 - <<'PY'
from subprocess import.run, PIPE
samples = [
'{"record_hash":"abc","signature":"sig","payload":"x"}',
'{"signature":"sig","payload":"x","record_hash":"abc"}',
]
for s in samples:
p = run(["jq", "-c", "del(.record_hash,.signature)"], input=s, text=True, stdout=PIPE, stderr=PIPE)
print("input={}".format(repr(s)))
print("jq result={}".format(repr(p.stdout)))
PY
echo "== check whether jq/openssl/jq installed =="
(command -v jq && jq --version) || true
(command -v openssl && openssl version) || trueRepository: fishaudio/docs Length of output: 692 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== jq del behavior using line above =="
python3 - <<'PY'
from subprocess import run, PIPE
samples = [
'{"record_hash":"abc","signature":"sig","payload":"x"}',
'{"signature":"sig","payload":"x","record_hash":"abc"}',
]
for s in samples:
p = run(["jq", "-c", "del(.record_hash,.signature)"], input=s, text=True, stdout=PIPE, stderr=PIPE)
print("input={}".format(repr(s)))
print("jq result={}".format(repr(p.stdout)))
print("returncode={}".format(p.returncode))
PY
echo "== jq installed =="
(command -v jq && jq --version) || trueRepository: fishaudio/docs Length of output: 441 Specify the canonical bytes that
🤖 Prompt for AI Agents |
||
|
|
||
| A full audit additionally checks that `seq` is contiguous, that each `prev_hash` | ||
| links the previous record, that the manifest signature verifies, and that the | ||
| manifest totals match the event file. Ask Fish Audio for the verification script | ||
| that runs all of these over a ledger directory. | ||
|
|
||
| <Note> | ||
| The ledger is tamper-evident: signature and hash-chain checks reliably detect | ||
| corruption and modification of the files. If your audit requirements call for | ||
| stronger guarantees, such as an independent write-once anchor or a separately | ||
| administered audit sink, raise it with Fish Audio so it can be designed into | ||
| the deployment. | ||
| </Note> | ||
|
|
||
| ### Retention and reconciliation | ||
|
|
||
| - The ledger lives on persistent shared storage and survives restarts and | ||
| rescheduling. | ||
| - Nothing prunes it. Archive completed day directories to your own storage on your | ||
| retention schedule, and never delete the live directory out from under a running | ||
| instance. | ||
| - Usage is reconciled afterwards from the signed day directories, on the cadence | ||
| set in your agreement. Copy whole directories, including the manifests, so the | ||
| totals can be verified independently. | ||
|
|
||
| ## Prove there is no egress | ||
|
|
||
| Configuration review is not proof. Demonstrate it. | ||
|
|
||
| ### Kubernetes | ||
|
|
||
| Deny external egress for the namespace, keeping in-cluster traffic and DNS, then | ||
| confirm the deployment still generates audio. | ||
|
|
||
| ```yaml deny-external-egress.yaml | ||
| apiVersion: networking.k8s.io/v1 | ||
| kind: NetworkPolicy | ||
| metadata: | ||
| name: deny-external-egress | ||
| namespace: fish-audio | ||
| spec: | ||
| podSelector: {} | ||
| policyTypes: | ||
| - Egress | ||
| egress: | ||
| - to: | ||
| - namespaceSelector: {} | ||
| ``` | ||
|
|
||
| ```bash | ||
| kubectl apply -f deny-external-egress.yaml | ||
| ``` | ||
|
|
||
| Re-run the [smoke test](/developer-guide/self-hosting/kubernetes#smoke-test). It | ||
| must still return playable audio. This proves nothing unless your CNI actually | ||
| enforces NetworkPolicy, so confirm enforcement with a deliberate control: exec | ||
| into a pod in the namespace and check that an outbound request fails. | ||
|
|
||
| ### All-in-One | ||
|
|
||
| The strongest proof is a container that never had a network interface, on a fresh | ||
| volume, so nothing could have been fetched even during the first compile: | ||
|
|
||
| ```bash | ||
| docker volume create fish-tts-airgap | ||
|
|
||
| docker run -d --name fish-tts-airgap \ | ||
| --gpus '"device=0,1"' \ | ||
| --shm-size 16g --ulimit memlock=-1 --ulimit stack=67108864 \ | ||
| -v fish-tts-airgap:/mnt/shared \ | ||
| --network none \ | ||
| '<all-in-one-image>' | ||
| ``` | ||
|
|
||
| With `--network none` no host port can be published, so drive the request from | ||
| inside the container once the cold start finishes: | ||
|
|
||
| ```bash | ||
| docker exec fish-tts-airgap curl -s -m 120 -X POST http://127.0.0.1:8088/v1/tts \ | ||
| -H 'Authorization: Bearer airgap' \ | ||
| -H 'Content-Type: application/json' \ | ||
| -H 'model: <model-name>' \ | ||
| -d '{"text":"offline","format":"mp3"}' \ | ||
| -o /tmp/offline.mp3 -w '%{http_code}\n' | ||
| ``` | ||
|
|
||
| A `200` with a non-trivial audio file is the proof. A weaker but faster variant | ||
| disconnects an already-warmed container from every Docker network and repeats the | ||
| request; it demonstrates that the running service survives losing the network, but | ||
| not that the cold start never needed it. | ||
|
|
||
| Capture the output of whichever check you run as deployment evidence. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
openssl pkeyutl rsa_pss_saltlen -3 auto-digestmax supported version💡 Result:
The auto-digestmax option for rsa_pss_saltlen in OpenSSL is a setting introduced to align with FIPS 186-4 compliance [1][2]. It functions by automatically detecting the salt length during signature verification and, when signing, maximizes the salt length up to the size of the digest [3]. This option was introduced in OpenSSL 3.1 [2]. Key details regarding this option include: - Behavior: It provides a balance between compatibility and FIPS compliance [1][2]. During verification, it auto-detects the salt length to ensure maximum compatibility [1]. When signing, it clamps the salt length to the digest length, ensuring it does not violate the FIPS 186-4 requirement that the salt length must be less than or equal to the hash function output length [1][2]. - Purpose: It was created to address issues where older salt length settings (like digest or auto) might either fail with certain digest/key combinations or potentially violate FIPS requirements depending on the default behavior [1][2]. - Availability: As of August 2026, this feature is available in OpenSSL 3.1 and later versions [3][2]. It is documented under the OSSL_PKEY_RSA_PSS_SALT_LEN_AUTO_DIGEST_MAX parameter [3].
Citations:
🏁 Script executed:
Repository: fishaudio/docs
Length of output: 294
🏁 Script executed:
Repository: fishaudio/docs
Length of output: 5923
🏁 Script executed:
Repository: fishaudio/docs
Length of output: 1628
🌐 Web query:
OpenSSL pkeyutl -rawin -verify -sigopt rsa_pss_saltlen:16 default digest length💡 Result:
In OpenSSL's pkeyutl utility, when using the rsa_pss_saltlen option, the default salt length behavior depends on the context of the underlying EVP_PKEY_CTX. For PSS padding, if the salt length is not explicitly set, the default is -2 [1]. In the context of the EVP_PKEY_CTX_set_rsa_pss_saltlen function: - A value of -1 sets the salt length to the digest length [1]. - A value of -2 sets the salt length to the maximum permissible value when signing, or causes the salt length to be automatically determined based on the PSS block structure when verifying [1]. While the pkeyutl documentation for command-line options describes the "digest" value as setting the salt length to the digest length [2][3], it refers to the same underlying control mechanisms. When you do not specify a salt length, the system defaults to the value represented by -2 (the maximum permissible salt length) [1].
Citations:
🌐 Web query:
OpenSSL pkeyutl -rawin -verify -sigopt rsa_pss_saltlen:-3 invalid old version💡 Result:
In OpenSSL, the value -3 for rsa_pss_saltlen represents a request to set the salt length to the maximum permissible value [1][2][3]. If you encounter an "invalid" error when using rsa_pss_saltlen:-3, it is typically because the specific operation (such as signing) or the key constraints prevent the use of that salt length, or because the version of OpenSSL you are using has introduced stricter validation for FIPS compliance [1][4][5]. Key technical points regarding this behavior: 1. Definition of -3: Internally, OpenSSL defines RSA_PSS_SALTLEN_MAX as -3 [2][3]. This is intended to maximize the salt length based on the RSA key size and digest length. 2. FIPS Compliance Changes: Recent versions of OpenSSL have moved toward stricter adherence to FIPS 186-4, which limits the salt length (sLen) to be less than or equal to the digest length (hLen) [1][4][5]. As part of this, a new default option, auto-digestmax (which effectively caps the salt length at the digest length), was introduced [4][5]. Attempting to use a "maximum" salt length that violates these FIPS-based constraints can result in an error, as the library now performs stricter checks [1][4]. 3. Version Differences: - In older versions, -3 was a standard way to request the maximum salt length [2][3]. - In newer versions (especially since OpenSSL 3.0), the library may reject certain configurations that were previously allowed if they conflict with modern security standards or the new default "auto-digestmax" logic, particularly if you are in a FIPS-enabled environment [4][5]. 4. Troubleshooting: If you are getting an invalid error, verify if the salt length you are requesting exceeds the digest length of your chosen hash function. You may need to use the digest length (value -1) or the new auto or auto-digestmax options instead of forcing a maximum length [6][4][5]. Top results: [1], [4], [5], [2]
Citations:
Write the hash without a trailing newline before verification.
jq -r .record_hash > /tmp/msgappends a newline. The signature covers therecord_hashvalue only, but verification applies the digest to a longer message, so authentic ledger records fail. Usejq -j .record_hashor another newline-free write method.Also check the OpenSSL versions customers support: set
rsa_pss_saltlento a value supported by those builds, or document the required OpenSSL version for this command.🤖 Prompt for AI Agents