Live Controller

@realsee/live

Messaging System

The core responsibility of Live is to enable multi-client, shared-screen connections between users by leveraging the duplex communication of a persistent WebSocket connection. All communication between users is based on WebSocket messages.

Live divides messages into three categories: keyframe data, broadcast, and room information. It provides corresponding send and receive APIs for each category. For more API details, see the Live API documentation.

Keyframe data is the carrier for shared-screen synchronization. You decide which pieces of information in the current scene should be synchronized, and organize them in the form of {key: value}.

Broadcast messages let a designated user send a custom broadcast message to room members in the form of an event. For example, the host can send a greeting message to viewers.

Room information is a built-in message body, pushed proactively by the WebSocket service. It includes the room status, the current user, and the list of participating users. Some of this information can also be modified through the update methods that Live provides.

Keyframe Data Synchronization

Synchronizing the current UI interaction state:

Using Five SDK shared-screen synchronization as an example:

// Send local keyframe data (Five State)
live.sendKeyframe("five", { panoIndex: 13 });

// Receive remote keyframe data (Five State)
live.keyframes.on("five", (newState, prevState) => {
  // You can use the latest `newState` to update the local UI state
});

You can also use the live.snapshot property to obtain a "snapshot" of all current sequential keyframe data.

Room Information

Live has the concept of a session room. Once you have successfully connected to the WebSocket service, you have effectively joined a room as a user. Besides yourself, this room may contain other users. You can retrieve or update user information as follows:

// Property: get the local user's information
live.selfInfo;

// Property: get the list of all users currently in the room
live.userList;

// Method: update the local user's information
live.setSelfInfo({});

// Event: the local user's information has changed
live.on("selfInfoUpdate", (userInfo) => {});

// Event: the room's user list information has changed
live.on("userListUpdate", (userList) => {});

Broadcast Messages

In addition to keyframe data and room information, you can also broadcast messages to other users in the room in the form of an event:

live.broadcast(
  data /* the data to broadcast */,
  ["user_id_1", "user_id_2"] /* userIds of the users who should receive this */
);

In this case, the users in the room whose IDs are user_id_1 and user_id_2 will receive this broadcast message (other users are unaware of it):

live.on("broadcast", (data /* the broadcast data */) => {});

For more examples, see the Live API documentation.

Event Mechanism

Live exposes asynchronous live events and optional RTC events. Listen to live events with live.on('/* event name */', callback) and RTC events with live.$RTC.on('/* event name */', callback).

For example, listen for changes to the public Live connection state:

import { LiveState } from "@realsee/live";

live.on("stateChange", (state, previousState) => {
  if (state === LiveState.OPEN) {
    console.log("Live connection opened", { previousState });
  }
});

Connection State

• stateChange(state: LiveState, previousState?: LiveState): void — the public Live connection state changed.

WebSocket connection status enumeration:

The interaction between Live's front-end and back-end services is based on a persistent WebSocket connection. Successfully entering and exiting a live-viewing session is signaled by the WebSocket statuses OPEN and CLOSED. Therefore, all VRTC-related event-handling logic should run only after the OPEN event.

live Events

• broadcast(evtMsg: Record<string, any>, frontRequestId: string): void Receive a broadcast message from another user.
• builtinEvent(builtinMsg: BuiltinMsg): void Receive a built-in event message pushed by the server.
• keyframes(keyframes: Partial<Snapshot>, frontRequestId: string): void Receive keyframe data from another user.
• selfInfoUpdate(userInfo: UserInfo, frontRequestId: string): void The local user's own information has changed.
• userListUpdate(userList: UserInfo[], frontRequestId: string): void The user list information has changed.

The live instance provides many methods such as live.connect(), live.broadcast(), and live.sendKeyframe(). These methods all respond asynchronously and may affect other users in the room. You can listen to the events above to learn about these effects.

LiveState.OPEN is the durable signal that the room channel can send keyframes. Use the keyframes event or live.keyframes.on(key, callback) for initial and subsequent synchronized frame updates. The older readyKeyframeSync event is deprecated in @realsee/live@0.7.6.

RTC Events

• error(error: Error): void Error event.
• initWillStart(): void About-to-initialize event.
• inited(): void Initialization-complete event.
• joinWillStart(): void About-to-join-the-voice-room event.
• joined(): void Successfully-joined-the-voice-room event.
• userVolumes(userVolumes: UserVolume[]): void The volume of each speaking user in the room.
• weakNetwork(): void Weak-network warning event.

The live.$RTC instance must satisfy the [RTC protocol]. Its events are listened to via live.$RTC.on('/*event name*/', /*event callback*/). For example, to listen for the users currently speaking in the voice room and their volumes:

live.$RTC.on("userVolumes", (userVolumes: UserVolume[]) => {
  console.log(userVolumes); // userVolumes: the speaking users and their volume data.
});

Error Handling

• ws.error(error: WebSocketError): void An error occurred with the WebSocket connection.

WebSocket error status enumeration:

• error(liveMsg: LiveMsg): void An error occurred with a server push.

The structure LiveMsg used for WebSocket communication has the following format:

When the data structure pushed over the WebSocket does not satisfy LiveMsg, or the code value is not SUCCESS, it is thrown in the form of an error event.

• error(error: Error): void An error occurred with RTC voice.

RTC Protocol

Voice is optional. @realsee/live@0.7.6 exports BrowserRTC, BrowserRTC4Chime, VRWebViewRTC, and the shared RTCProtocol type. Use a built-in implementation for supported browser environments or supply a custom RTCProtocol implementation for another container. See RTC and Environment for the supported setup.

Live bridges to native capabilities through jsbridge-x, talking to client applications or WeChat Mini Program applications that have integrated the container SDK, in order to invoke native functionality. This jsbridge-x instance must be supplied as a configuration parameter to createLive().

If you are on an iOS/Android App (with the Realsee VRTC container SDK already integrated)

import { JSBridgeApp } from "@realsee/jsbridge-x/lib/app";
import { createLive } from "@realsee/live";
import { VRWebViewRTC } from "@realsee/live";

const jsBridge = new JSBridgeApp();

const rtcInstance = new VRWebViewRTC({
  jsBridge,
  getVoiceSign: () => {},
});

const live = createLive({
  rtc: rtcInstance,
  jsBridge,
  getTicket: async () => "",
});

Once you supply the RTC instance configuration parameter, the voice connection, disconnection, and reconnection logic is delegated to Live. You can access voice-related state and events through the $RTC namespace on the live instance.

// State: whether the user has successfully joined the voice room
live.$RTC.joined;
// State: the user's microphone status
live.$RTC.micro;
// Listen for changes in user volume
live.$RTC.on("userVolumes", (userVolumes) => {});