Screen share in group calls

This page provides a code example for implementing screen share in a group call.

Note

• As of version 1.2, screen share capturing and transmission are supported on both Android and iOS, but the two platforms use different mechanisms:
  • On Android, PlanetKit captures the screen itself. The application calls startMyScreenShare(), and PlanetKit requests the system screen capture consent and runs the capture in a media projection foreground service.
  • On iOS, the application captures the screen through a Broadcast Upload Extension and sends the stream to PlanetKit over NWConnection, using the ScreenShareKey set when the conference starts.
• Receiving a peer's screen share is supported on both platforms, and the code is the same on both.

Prerequisites

Before implementing screen share, you must do the following.

Android

Screen capture runs in a foreground service that PlanetKit starts for you, but the service and its permissions must be declared by your application. Add the following to your app's AndroidManifest.xml:

<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_MEDIA_PROJECTION" />

<application ...>
    <service
        android:name="com.example.planet_kit_flutter.screenshare.PlanetKitScreenShareService"
        android:foregroundServiceType="mediaProjection"
        android:exported="false" />
</application>

PlanetKit creates the notification channel and posts the ongoing "Screen sharing" notification itself, so no additional notification setup is required.

Note

You do not need to define a port number or tokens, and you do not need to call setScreenShareKey(). Those are used only by the iOS broadcast extension flow.

iOS

• The application must implement Broadcast Upload Extension or equivalents to capture the screen.
  • To implement Broadcast Upload Extension, in Xcode, add a new Target to your project, select "Broadcast Upload Extension" template, and activate the extension.
• Define a port number, a reception token, and a transmission token that will be used for screen share stream transmission.

How screen share works in group calls

Android

1. When the user requests screen share, the app client calls startMyScreenShare() on PlanetKitConference.
2. PlanetKit presents the system screen capture consent dialog.
3. If the user allows it, PlanetKit starts a media projection foreground service, begins capturing the screen, and transmits it. startMyScreenShare() then returns true.
4. The app client of the receiver detects that screen share has started after receiving the onScreenShareUpdate event with state=enabled, and creates a view instance and calls startScreenShare() on PlanetKitPeerControl to have PlanetKit render the screen share video.
5. The sender's own screen share state is reported through PlanetKitMyMediaStatusHandler.onScreenShareStateUpdate.

startMyScreenShare() returns false if the user declines the consent dialog, if another screen share request is still pending, or if PlanetKit rejects the start. On iOS, it always returns false: use the broadcast extension flow described below.

iOS

In group calls, screen share works as follows:

1. The app client of the sender creates a ScreenShareKey consisting of the predefined port number, reception token, and transmission token, and sets it in setScreenShareKey(key) of PlanetKitJoinConferenceParamBuilder.
2. When the user requests screen share, the app client of the sender uses NWConnection or equivalents to establish a connection between the app and the SDK, and sends the screen share streams to the port defined along with the tokens.
3. If the information in ScreenShareKey matches the information received from NWConnection, PlanetKit for Flutter automatically starts screen share.
4. The app client of the receiver detects that screen share has started after receiving the onScreenShareUpdate event, and creates a view instance and calls startScreenShare() to have PlanetKit render the screen share video.

Send the screen (sender, Android)

On Android, PlanetKit captures and transmits the screen itself. Call startMyScreenShare() to start and stopMyScreenShare() to stop. There is no ScreenShareKey and no broadcast extension to implement.

A screen share can also stop without the application asking for it, so track the current state through PlanetKitMyMediaStatusHandler.onScreenShareStateUpdate rather than through the return value of startMyScreenShare() alone.

class ConferenceScreenShareController {
  ConferenceScreenShareController({required this.conference}) {
    conference.myMediaStatus.setHandler(PlanetKitMyMediaStatusHandler(
      onMicMute: null,
      onMicUnmute: null,
      onAudioDescriptionUpdate: null,
      onVideoStatusUpdate: null,
      onScreenShareStateUpdate: (status, screenShareState) {
        isScreenSharing = screenShareState == PlanetKitScreenShareState.enabled;
        // update your UI here
      },
    ));
  }

  final PlanetKitConference conference;
  bool isScreenSharing = false;

  Future<void> toggleScreenShare() async {
    if (isScreenSharing) {
      await conference.stopMyScreenShare();
      return;
    }

    final started = await conference.startMyScreenShare();
    if (!started) {
      // The user declined the consent dialog, another request was still pending,
      // or PlanetKit rejected the start.
    }
  }
}

Set the screen share key in PlanetKit (sender, iOS)

Set the screen share key with setScreenShareKey() of PlanetKitJoinConferenceParamBuilder. You must pass the pre-defined port number, transmission token, and reception token to setScreenShareKey().

var builder = PlanetKitJoinConferenceParamBuilder()
        .setMyUserId(myUserId)
        .setMyServiceId(serviceId)
        .setRoomServiceId(_serviceId)
        .setRoomId(roomId)
        .setAccessToken(accessToken)
        .setScreenShareKey(ScreenShareKey(broadcastPort: PORT_NUMBER, broadcastPeerToken: "USER_DEFINED_TOKEN_EXT", broadcastMyToken: "USER_DEFINED_TOKEN_APP"));

Implement a screen capturing and transmission module in Swift (sender, iOS)

Implement a screen capturing module that establishes connection between your app and the SDK through NWConnection.

class BroadcastSender {
    private enum State {
        case started
        case handshaking
        case connected
        case failed
    }

    weak var delegate: BroadcastSenderDelegate?

    private let broadcastPort: UInt16
    private let rxToken: String
    private let txToken: String

    private let connection: NWConnection
    private let queue = DispatchQueue(label: "BroadcastSender.Queue")

    static func connection(broadcastPort : UInt16) throws -> NWConnection {
        guard let port = NWEndpoint.Port(rawValue: broadcastPort) else {
            throw Error.invalidPort
        }

        let options = NWProtocolTCP.Options()
        options.noDelay = true

        return NWConnection(host: .ipv4(.loopback), port: port, using: .init(tls: nil, tcp: options))
    }

    init(delegate: BroadcastSenderDelegate?, broadcastPort: UInt16, rxToken : String, txToken : String) throws {
        self.delegate = delegate
        self.rxToken = rxToken
        self.txToken = txToken

        connection = try BroadcastSender.connection(broadcastPort: broadcastPort)
        connection.stateUpdateHandler = { [weak self] in
            self?.handleConnection(newState: $0)
        }
        connection.start(queue: DispatchQueue(label: "BroadcastSender.NetworkQueue"))
    }

    func handShake() {
        guard state == .started, let data = txToken.data(using: .utf8) else {
            return
        }

        state = .handshaking

        connection.send(content: data, completion: .contentProcessed({ (error) in
            self.handShakeProcessed(error: error)
        }))
    }

    func sendVideo(sampleBuffer: CMSampleBuffer) throws {
        guard state == .connected,
              !sending else {
            return
        }

        try queue.sync {
            let data: Data?
            // create data with CMSampleBuffer
            connection.send(data, completion: .contentProcessed({(error) in NSLog("data processed \(error)")}))
        }
    }

    func handShakeProcessed(error: NWError?) {
        queue.sync {
            if let error = error {
                didFail(error: error)
            } else {
                handShakeAck()
            }
        }
    }

    func handShakeAck() {
        guard state == .handshaking, let token = rxToken.data(using: .utf8) else {
            return
        }

        connection.receive(minimumIncompleteLength: token.count, maximumLength: token.count) { (data, context, final, error) in
            self.handShakeAckProcessed(data: data, error: error)
        }
    }

    func handShakeAckProcessed(data: Data?, error: NWError?) {
        queue.sync {
            guard state == .handshaking, let token = rxToken.data(using: .utf8) else {
                return
            }
            if data == token {
                state = .connected

            } else {

                didFail(error: .rejected)
            }
        }
    }

    func handleConnection(newState: NWConnection.State) {
        queue.sync {
            switch newState {
            case .ready:
                handShake()
            case .waiting(let error), .failed(let error):
                didFail(error: error)
            default:
                break
            }
        }
    }
}

If connection is successfully established, you must send screen share streams to the created NWConnection. Implement the SampleHandler class to send the captured screen share streams.

class SampleHandler: RPBroadcastSampleHandler {
    private var sender: BroadcastSender?
    ...
    override func broadcastFinished() {
        // User has requested to finish the broadcast.
        sender?.cancel()
        sender = nil
    }

    let rxToken : String = "USER_DEFINED_TOKEN_APP"
    let txToken : String = "USER_DEFINED_TOKEN_EXT"
    let broadcastPort : UInt16 = PORT_NUMBER

    override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {
        switch sampleBufferType {
        case RPSampleBufferType.video:
            do {
                if let sender = sender {
                    try autoreleasepool {
                        try sender.sendVideo(sampleBuffer: sampleBuffer)
                    }
                } else {
                    sender = try BroadcastSender(delegate: self, broadcastPort: broadcastPort, rxToken: rxToken, txToken: txToken)
                }
            } catch {
                finish(error: error)
            }
            break
        ...
        }
    }

    private func finish(error: Error) {
        sender?.cancel()
        sender = nil

        if let description = (error as? LocalizedError)?.errorDescription {
            self.finishBroadcastWithError(NSError(domain: "BroadcastSender.ErrorDomain", code: 0, userInfo: [NSLocalizedFailureReasonErrorKey: description]))
        } else {
            self.finishBroadcastWithError(error)
        }
    }
}

View the screen share (receiver)

Listen for the PlanetKitPeerControlHandler.onScreenShareUpdate event, and implement code to add the peer's screen share view when the event occurs.

class Peer {
  final PlanetKitPeerControl control;
  bool screenShareAvailable = false;

  Peer({required this.control});

  void register() async {
    final handler = PlanetKitPeerControlHandler(
        onScreenShareUpdate: (control, screenShareState) {
      screenShareAvailable =
          screenShareState == PlanetKitScreenShareState.enabled ? true : false;
    });
    await control.register(handler);
  }

  void unregister() async {
    await control.unregister();
  }

  void startScreenShare(String viewId) async {
    await control.startScreenShare(viewId);
  }

  void stopScreenShare(String viewId) async {
    await control.stopScreenShare(viewId);
  }
}

To render the peer's screen share, you must use PlanetKitVideoViewBuilder to create PlanetKitVideoView and add it to PlanetKitPeerControl.

After creating the PlanetKitVideoView for the peer's screen share, add the peer's screen share view to PlanetKitPeerControl by calling startScreenShare(viewId).

class PeerView extends StatelessWidget {
  final Peer peer;
  PeerView({required this.peer});

  @override
  Widget build(BuildContext context) {
    if (peer.screenShareAvailable) {
      return ScreenShareView(peer: peer);
    } else {
      return Text("screen share not available");
    }
  }
}

class ScreenShareView extends StatelessWidget {
  const ScreenShareView({super.key, required this.peer});
  final Peer peer;

  @override
  Widget build(BuildContext context) {
    final screenShareView = PlanetKitVideoViewBuilder.instance
        .create(PlanetKitViewScaleType.fitCenter);

    screenShareView.onCreate.listen((id) {
      peer.startScreenShare(id);
    });

    screenShareView.onDispose.listen((id) {
      peer.stopScreenShare(id);
    });

    return screenShareView;
  }
}

Related code examples

• Group audio call
• Group video call

Related documents

• Group call flow
• Presentation - screen share