feat: LiveKit as a native video conference provider (PoC) - #41735
Draft
rodrigok wants to merge 15 commits into
Draft
feat: LiveKit as a native video conference provider (PoC)#41735rodrigok wants to merge 15 commits into
rodrigok wants to merge 15 commits into
Conversation
Ported from #40726, minus that PR's persistent-chat work -- this branch already carries its own, and the two overlapped in about twenty files. Every video conference provider so far has been a URL. The workspace knows a call is open and nothing else: not who is in it, not what was said, and certainly not how to record it. Running the call inside Rocket.Chat changes what the server is able to know, and most of this diff is the consequence of that rather than the SFU itself -- conferences now carry the people actually connected, so presence in a call stops being inferred from who clicked join. What that buys: multi-party calls with a grid and a spotlight, screen sharing, hand-raise and reactions, and a floating widget that keeps the call with the user when they walk to another room. Recording goes through LiveKit egress and comes back as an ordinary upload, posted in a thread under the call's own message rather than somewhere new. Captions are opt-in per user and come from a worker that joins each room as a hidden participant; with note-taking on, the transcript is kept and summarised when the call ends. The provider is enterprise and off unless configured. A workspace with no LiveKit deployment behaves exactly as it did. Reconciling the PR against a newer develop needed six things: - `app/` has been restructured, so the settings, notifications and file-upload imports across the eight EE files point at their new homes. - `useGoToRoom` left `ui-contexts` for the app. - fuselage no longer takes the `w`/`h`/`bg` shorthands. - `MediaCallRoomActivity` gained `MediaCallViewProvider` and container-height sizing on develop while the PR replaced that sizing with `useTileGridLayout`; the PR's layout wins, since it is what the rewritten section expects. - `IVideoConference` took both sides: our membership predicates and the provider's participant/recording/transcript records. - `@livekit/agents` and `@livekit/rtc-node` were pinned at `latest`, which a committed lockfile cannot honour. They are pinned to what resolved. `useLiveKitVideoConf` threw outside its provider, which made `VideoConfProvider` untestable and coupled mount order for a component that only forwards a join on. It gains an optional accessor; the throwing one stays for LiveKit's own components, which genuinely cannot work without the context. MEDIA-CALLS-FEATURE.md and deploy/livekit/deploy.md move under docs/features/native-video-conference/, where this repo keeps feature docs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Testing the port against a configured LiveKit found the seam between it and persistent chat exactly where the docs said it was, and worse than "not wired": with both on, the room's call button led to a preflight that never went anywhere. An embedded provider answers the join with an empty url -- there is no page to send anyone to -- and the conference page treated a missing url as "not joined yet", so it re-rendered the preflight forever. So the page now waits on having *joined* rather than on having a url, and renders the call itself when there is no page to embed. The connection is registered with the bridge above this route rather than held here, which is the same slot the in-room path uses -- a call cannot be claimed twice, and it survives this page re-rendering. One bar, not two. The call brings mic, camera, screen share, hand and hang up; this window already had a bar carrying members and chat. The call's controls are handed to that bar and take its centre, with the panel toggles at the end. They are moved, not rebuilt: the same nodes render in the same component either way, so the two placements cannot drift. The floating mini-view is gone. It followed the user out of the call's room, which a call that always has a window of its own never needs -- and in that window it drew a second copy of the call over the members panel, because it decides visibility by comparing against the opened room and a call window has none. Nine strings in the call's own bar were hardcoded English or asked for keys that were never added, so recording, notes and captions rendered as `Start_recording` and friends. Verified against a live LiveKit: capabilities report the provider embedded, start posts the call block, join records the participant, the transport endpoint mints a real room token, and the call connects with working camera and the members panel beside it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tom bar
Three things found testing the native provider.
**Starting a call ignored the mic and camera choice; joining honoured it.**
The window read the preferences off its own join mutation, and starting a
call does not join here -- the start screen does, and this window finds the
result already in the cache having never asked. So it handed the provider
nothing and got its defaults. It now reads what the preflight persisted,
which is the same for both paths and is where the choice actually lives.
**The call built a second bottom bar inside the conference.** The controls
were being handed a container captured from a ref, which is null on the
first render -- so the call had already built its own strip by the time it
was told where to put them. The container is now created up front and is
non-null from the start, so there is one bar: the call's controls in its
centre, members and chat at its end.
**The window grew but would not shrink.** The call column is a flex item,
and a flex item's `min-width` is `auto` -- it will not go below its content.
Widening worked because that is the direction `auto` allows. `minWidth={0}`
lets it give the width back.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three more of the call bar's strings had no key behind them, so the camera button's tooltip read `Stop_camera`. Added, along with `Camera` itself. The hang-up button read "End call with Call". A one-to-one call is left *with* someone and can name them; a group call has no single other side, and naming one anyway produced that. It now says what the button does. The nine keys added earlier landed near the top of the file rather than beside their neighbours -- my insert compared case-insensitively against a file that isn't sorted that way. Moved to where they belong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
|
Looks like this PR is not ready to merge, because of the following issues:
Please fix the issues and try again If you have any trouble, please check the PR guidelines |
🦋 Changeset detectedLatest commit: e30f8dd The changes in this PR will be included in the next version bump. This PR includes changesets to release 8 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…ndow
**The preflight now shows the camera it will actually send.** It used to say
only *that* the camera would be on, and it was right to: a URL-based provider
takes "camera on" and nothing more, so a self-view would have promised a
choice the screen couldn't make and might have shown a camera the call never
used. A provider running the call in here takes the devices too, so the
honest thing is the real thing -- a live self-view, with a camera and a
microphone menu beside the toggles that own them. Gated on
`capabilities.embedded`, so a URL provider keeps the old wording.
The device ids stay client-side. `video-conference.join` accepts `{mic, cam}`
and nothing else -- `additionalProperties: false` would reject them -- and
the server has no use for a device id anyway. They are persisted beside the
on/off habit and handed to the provider as capture options on connect, so
starting and joining both arrive on what was chosen.
Opening the devices for the preview is also what earns the permission the
menus need: `enumerateDevices` returns unnamed entries until something has
been allowed. The preview releases every track when it unmounts, so the call
gets the devices back rather than finding them busy.
**Hanging up closes the window.** The call had been left, but the window it
was opened for stayed. It now reports leaving and closes -- the same thing
Cancel on the preflight does, and what closing the window would have done
anyway. Watching the active-call slot rather than the button means it also
covers a call that ends from the other side.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two columns, as Google Meet does it: how you will look and sound on one
side, what you are joining and the decision on the other. They are separate
questions, and stacking the decision under a column of controls made it read
as the last of them rather than the point of the screen. The preview gets the
room -- it is the subject -- and the details column only needs to be
readable. Narrow viewports stack, preview first.
The mic and camera toggles follow the convention every call UI uses: **off
is red**, filled, because a muted mic or a dark camera is the state you need
to notice at a glance; on is a ghost button, since there is nothing to
report. `IconButton`'s own `danger` is the ghost variant -- a red glyph on
nothing -- so the fill comes from the same tokens the danger buttons use.
`mic-off` slashes the other way from `video-off`, so side by side they read
as two unrelated marks. The mic is mirrored to match, which flips its slash
without visibly changing the mic: it is symmetric about that axis.
The tile is black, and stays black with the camera off. It is where a camera
goes, and a camera with nothing to show is black -- a themed panel that
repainted as the camera toggled read as a different surface each time.
Three device selectors below it -- microphone, speaker, camera -- in one row
of equal columns. A grid, not flex: flex sizes to content however hard it is
pushed, and equal columns are what make the three read as one set rather
than three differently-sized pills.
What they show took some undoing of what browsers hand over:
- the USB vendor:product pair ("Display Audio (05ac:1107)") identifies the
hardware to the machine, not to the person choosing it, and is long enough
to push the name itself out of view. A parenthetical like "(Built-in)"
stays -- that is part of the name.
- the system default arrives prefixed "Default - ". The prefix is dropped
from the name and said properly on its own line, where it reads as a fact
about the device rather than part of what it is called.
- it also arrives *twice*: once as the `default` alias and again under its
own id. The alias is kept -- it is what "leave it to the system" means, and
it follows the system if that changes -- and its twin is found by
`groupId`, which the two share. Matching on the name would have collapsed
genuinely different devices that happen to share one, which two displays
generally do.
The default sorts first: it is what will be used if nothing is picked.
None of the three is gated on the device being on. Arriving muted is normal,
and someone who does may still care which microphone gets unmuted later;
refusing the choice until they turn it on would make them turn it on just to
set it. Enumeration is therefore independent of the preview, and follows
`devicechange` so a headset arriving mid-decision shows up.
Confirm and cancel are stacked and full width, confirm first: it is what the
screen is asking.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The device on the left, its name beside it, the chevron pushed to the far right, and the chevron turning over when the list opens -- a control that says what it is, what it is set to, and that there is more behind it, read left to right. None of that was reachable through `GenericMenu`, which clones its trigger: it injects its own chevron as a *leading* icon and replaces the button's `className`, so neither the icon's place nor the name's alignment was ours to set, and there was nothing to read the open state from. So the trigger is a plain button over `useDropdownVisibility` and fuselage's `Dropdown` -- the same pairing the omnichannel quick actions use -- which owns the open state and puts the chevron under our control. The selected device now prefers the system-default alias rather than whatever the browser enumerated first, so the pill agrees with the list underneath it about which one is chosen. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
They are the two controls the whole screen is for, and at toolbar size they read as an afterthought on a wide camera tile. Bigger than any of `IconButton`'s own sizes, which stop at about a toolbar button, and round — at this size a rounded square reads as a tile of its own rather than a control. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… out Captions, persisted transcripts, the AI summary and the per-call language picker move to a branch of their own, stacked on this one. They are one feature, not two: the language picker exists to tell the agent which language to transcribe, and means nothing without it. What that takes out of here: the worker subprocess and its supervisor, the Gemini summary path, the transcription and transcript-append endpoints, the `Agent_*` and `Summary_*` settings, the transcription/transcript/summary records on the conference, the caption overlay on the tiles, the notes, captions and language pills, and the four data-channel messages they spoke over. Three dependencies go with them -- `@livekit/agents`, `@livekit/rtc-node` and `@google/genai` -- and so does the Dockerfile change, which existed only to build a musl FFI binding for `@livekit/rtc-node`. The client SDK stays: `livekit-client`, `@livekit/components-react` and the Krisp filter are the call itself, not the agent. `@livekit/components-styles` was declared and never imported, so it goes too. What is left is a call: multi-party video with a grid and spotlight, screen sharing, hand-raise, reactions, and egress recording. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Recording is a separable feature with its own dependencies, its own storage story and its own failure modes, so it joins the agent branch stacked on top of this one rather than shipping with the call. Out of here: the egress client, the recording orchestration, the poller that resumes in-flight recordings on boot, the finaliser that registers the file as an ordinary upload and posts it in a thread, the three recording endpoints, the `Recording_*` settings, the `recording` record on the conference, the pill and its chimes, and the `recording-state` data-channel message. The CloudFormation templates and the deployment walkthrough go with it. They are infrastructure rather than product code -- a different audience reviews them, and unlike the docs they are not free in CI. What this branch needs from a deployment is a URL and a key pair, which the settings ask for and any LiveKit instance can provide, so the doc now says that instead of shipping a turnkey stack nobody reviewing React can assess. `callId` came off the call's session state only so recording could name its egress, and `settings` was read only to gate it, so both go too. What is left is the call: multi-party video with a grid and spotlight, screen sharing, hand-raise and reactions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Removing the recording and transcription method signatures left two runs of blank lines behind, which prettier rejects. CI caught it in `@rocket.chat/model-typings#lint`; my own lint pass had covered `models` and `core-typings` but not this package. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## feat/persistant-chat #41735 +/- ##
=======================================================
Coverage ? 69.39%
=======================================================
Files ? 4205
Lines ? 163777
Branches ? 28673
=======================================================
Hits ? 113655
Misses ? 44962
Partials ? 5160
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
…e same **The call was rendering twice in the call window.** Its chat panel renders the conference's room, and `MediaCallRoom` puts the in-room call activity in any room that *is* the active call's room -- which, in that window, it always is. So the panel drew a second, fully live copy of the call, with its own timer and its own control strip, beside the first one. The window already shows the call, so the in-room activity is skipped there. **The in-call mic and camera now match the preflight**: off is the state worth noticing, so the whole button goes `secondary-danger` rather than only the glyph turning red, and `mic-off` is mirrored so its slash runs the same way as `video-off` -- side by side they were reading as two unrelated marks. That treatment is opt-in on `ToggleButton` rather than its default, because for most of these toggles pressed means *active*: a shared screen or a raised hand is not a problem, and colouring it as one would read as an error. Only mic and camera ask for it. The mirroring is unconditional, since `mic-off` should look the same everywhere it appears. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four changes to how the call's controls read. **A device and its selector are now one control.** They were two adjacent buttons with their own rounded corners, nudged together by 2px, which read as two things that happened to be side by side. Now: one rounded outline, a hairline between the halves, and the selector on the inline start -- out of the way of the toggle a user actually reaches for. **The selector's chevron points up**, toward where its menu opens from a bottom bar. Only the `chevron` variant of `DevicePicker` and `CameraPicker` change, and both are used solely in this strip; the cog-icon variant in the VoIP widget and popout is untouched. **8px between the bar's buttons**, on both groups, so the controls and the panel toggles at the far end keep the same rhythm. **Members moved to the call's top bar.** Who is in the call is a fact about the call, so for a provider that brings its own header it belongs up there beside the call's own actions, ruled off from them -- not down among the mic and camera. The chat toggle is this window's own and stays on this window's bar. A provider rendered in an iframe has no header of ours to put anything in, so there both stay on the bar as before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two things the screenshot showed. **Members and chat were ghost buttons** while everything beside them — screen share, hand, reactions — carried a background, so the two that belong to this window read as disabled rather than as merely inactive. They are `secondary` now, like the rest of the row. **The selector half of the mic and camera controls had neither background nor the toggle's height.** `DevicePicker`'s chevron variant forced `tiny` and left `secondary` off, and `CameraPicker` forced `tiny` unconditionally, so the two halves of a control that is meant to read as one pill were different sizes and different colours. The chevron now takes the same `secondary` background and the same medium size as the toggle it is joined to. `CameraPicker`'s `secondary` prop had become dead once the strip needed it on; it defaults to on rather than being hardcoded, so the prop still means something. The cog-icon `DevicePicker` in the VoIP widget and popout is untouched — only the chevron variant, which is used solely in this strip, changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Adds LiveKit as a native video conference provider, so a call can run inside Rocket.Chat instead of handing the user off to someone else's page.
Every provider until now has been a URL. The workspace knows a call is open and nothing else: not who is in it, not what was said, and certainly not how to record it. Running the call in-process changes what the server is able to know, and most of this diff is the consequence of that rather than the SFU itself — conferences now carry the participants actually connected, so presence stops being inferred from who clicked join.
On top of that: multi-party calls with a grid and spotlight, screen sharing, hand-raise and reactions, egress-based recording that lands as an ordinary upload posted in a thread under the call's message, and opt-in live captions from a worker that joins each room as a hidden participant — with note-taking on, the transcript is kept and summarised when the call ends.
The provider is enterprise and off unless configured. A workspace with no LiveKit deployment behaves exactly as before.
Deployment (CloudFormation + walkthrough):
docs/features/native-video-conference/deploying-livekit.md.Where this came from
Ported from #40726 (
feat/vc-poc), minus that PR's persistent-chat work, which this stack already carries from #41657. That PR's history splits cleanly: commits up to7e617cc8a8care the LiveKit work, everything after is its own conference-window integration. Only the first range was taken, so none ofclient/views/conference/,client/views/root/,appLayout,desktop-apiorstartup/routes.tsxcame across.Reconciling against a newer
developneeded six things:app/was restructured — the settings, notifications and file-upload imports across eight EE files point at their new homes (server/settings,server/lib/notifications/core/lib/Notifications,server/lib/media/file-upload,server/meteor-methods/messages/sendFileMessage).useGoToRoomleftui-contextsfor the app.w/h/bgshorthands.MediaCallRoomActivitygainedMediaCallViewProviderand container-height sizing on develop while the PR replaced that sizing withuseTileGridLayout; the PR's layout wins, since it is what the rewritten section expects.IVideoConferencetook both sides — this stack's membership predicates and the provider's participant / recording / transcript records.@livekit/agentsand@livekit/rtc-nodewere pinned atlatest, which a committed lockfile cannot honour. Pinned to what resolved (^1.6.2,^0.13.33).Then it was wired to the conference window
Testing against a live LiveKit found the seam between the two features, and it was worse than "not wired": with both enabled, the room's call button led to a preflight that never went anywhere. An embedded provider answers the join with an empty
url— there is no page to send anyone to — and the conference page read a missing url as "not joined yet", so it re-rendered the preflight forever.Start_recordingand friends. The hang-up button read "End call with Call" on group calls.Steps to test or reproduce
Requires a LiveKit deployment; see the deploy doc. Then, as admin:
Verified
Against a live LiveKit (
wss://livekit.dev.rocket.chat):video-conference.capabilitiesembedded: truevideo-conference.startvideo-conference.joinurl: "", records the participant…livekit.transport.configmin-width: 0, no horizontal overflowyarn typecheckclean · jest 309 suites / 2214 passing · mocha 2258 · definition 152 · eslint clean over the ported tree.Not verified — please exercise before this goes anywhere near merge
MediaCallRoomSectionnow rendersCallStagefor direct calls too, replacing develop'sMediaCallCardList. That is the ported PR's design, but it changes existing VoIP call rendering and deserves a look on a running workspace.Dockerfile.alpinegained a musl build of the LiveKit FFI binding; the image was never built here.Further comments
MEDIA-CALLS-FEATURE.mdanddeploy/livekit/deploy.mdmoved underdocs/features/native-video-conference/, where this repo keeps feature docs. The CloudFormation templates stay indeploy/livekit/.🤖 Generated with Claude Code