Screen share in 1-to-1 calls

This page provides a code example for implementing screen share in a 1-to-1 call.

Note

• As of version 1.2, screen share capturing and transmission are supported on both Android and iOS, but the two platforms use different mechanisms:
  • On Android, PlanetKit captures the screen itself. The application calls startMyScreenShare(), and PlanetKit requests the system screen capture consent and runs the capture in a media projection foreground service.
  • On iOS, the application captures the screen through a Broadcast Upload Extension and sends the stream to PlanetKit over NWConnection, using the ScreenShareKey set when the call starts.
• Receiving a peer's screen share is supported on both platforms, and the code is the same on both.

Prerequisites

Before implementing screen share, you must do the following.

Android

Screen capture runs in a foreground service that PlanetKit starts for you, but the service and its permissions must be declared by your application. Add the following to your app's AndroidManifest.xml:

<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_MEDIA_PROJECTION" />

<application ...>
    <service
        android:name="com.example.planet_kit_flutter.screenshare.PlanetKitScreenShareService"
        android:foregroundServiceType="mediaProjection"
        android:exported="false" />
</application>

PlanetKit creates the notification channel and posts the ongoing "Screen sharing" notification itself, so no additional notification setup is required.

Note

You do not need to define a port number or tokens, and you do not need to call setScreenShareKey(). Those are used only by the iOS broadcast extension flow.

iOS

• The application must implement Broadcast Upload Extension or equivalents to capture the screen.
  • To implement Broadcast Upload Extension, in Xcode, add a new Target to your project, select "Broadcast Upload Extension" template, and activate the extension.
• Define a port number, a reception token, and a transmission token that will be used for screen share stream transmission.

How screen share works in 1-to-1 calls

Android

1. When the user requests screen share, the app client calls startMyScreenShare() on PlanetKitCall.
2. PlanetKit presents the system screen capture consent dialog.
3. If the user allows it, PlanetKit starts a media projection foreground service, begins capturing the screen, and transmits it. startMyScreenShare() then returns true.
4. The app client of the receiver detects that screen share has started after receiving the onPeerScreenShareStarted event, and creates a view instance and calls addPeerScreenShareView() to have PlanetKit render the screen share video.
5. The sender's own screen share state is reported through PlanetKitMyMediaStatusHandler.onScreenShareStateUpdate.

startMyScreenShare() returns false if the user declines the consent dialog, if another screen share request is still pending, or if PlanetKit rejects the start. On iOS, it always returns false: use the broadcast extension flow described below.

iOS

In 1-to-1 calls, screen share works as follows:

1. The app client of the sender creates a ScreenShareKey consisting of the predefined port number, reception token, and transmission token, and sets it in setScreenShareKey(key) of PlanetKitMakeCallParamBuilder or PlanetKitVerifyCallParamBuilder.
2. When the user requests screen share, the app client of the sender uses NWConnection or equivalents to establish a connection between the app and the SDK, and sends the screen share streams to the port defined along with the tokens.
3. If the information in ScreenShareKey matches the information received from NWConnection, PlanetKit for Flutter automatically starts screen share.
4. The app client of the receiver detects that screen share has started after receiving the onPeerScreenShareStarted event, and creates a view instance and calls addPeerScreenShareView() to have PlanetKit render the screen share video.

Send the screen (sender, Android)

On Android, PlanetKit captures and transmits the screen itself. Call startMyScreenShare() to start and stopMyScreenShare() to stop. There is no ScreenShareKey and no broadcast extension to implement.

A screen share can also stop without the application asking for it, so track the current state through PlanetKitMyMediaStatusHandler.onScreenShareStateUpdate rather than through the return value of startMyScreenShare() alone.

class ScreenShareController {
  ScreenShareController({required this.call}) {
    call.myMediaStatus.setHandler(PlanetKitMyMediaStatusHandler(
      onMicMute: null,
      onMicUnmute: null,
      onAudioDescriptionUpdate: null,
      onVideoStatusUpdate: null,
      onScreenShareStateUpdate: (status, screenShareState) {
        isScreenSharing = screenShareState == PlanetKitScreenShareState.enabled;
        // update your UI here
      },
    ));
  }

  final PlanetKitCall call;
  bool isScreenSharing = false;

  Future<void> toggleScreenShare() async {
    if (isScreenSharing) {
      await call.stopMyScreenShare();
      return;
    }

    final started = await call.startMyScreenShare();
    if (!started) {
      // The user declined the consent dialog, another request was still pending,
      // or PlanetKit rejected the start.
    }
  }
}

Set the screen share key in PlanetKit (sender, iOS)

Set the screen share key with setScreenShareKey() of PlanetKitMakeCallParamBuilder. You must pass the pre-defined port number, transmission token, and reception token to setScreenShareKey().

var builder = PlanetKitMakeCallParamBuilder()
        .setMyUserId(myUserId)
        .setMyServiceId(serviceId)
        .setPeerUserId(peerId)
        .setPeerServiceId(serviceId)
        .setAccessToken(accessToken)
        .setScreenShareKey(ScreenShareKey(broadcastPort: PORT_NUMBER, broadcastPeerToken: "USER_DEFINED_TOKEN_EXT", broadcastMyToken: "USER_DEFINED_TOKEN_APP"));

Implement a screen capturing and transmission module in Swift (sender, iOS)

Implement a screen capturing module that establishes connection between your app and the SDK through NWConnection.

class BroadcastSender {
    private enum State {
        case started
        case handshaking
        case connected
        case failed
    }

    weak var delegate: BroadcastSenderDelegate?

    private let broadcastPort: UInt16
    private let rxToken: String
    private let txToken: String

    private let connection: NWConnection
    private let queue = DispatchQueue(label: "BroadcastSender.Queue")

    static func connection(broadcastPort : UInt16) throws -> NWConnection {
        guard let port = NWEndpoint.Port(rawValue: broadcastPort) else {
            throw Error.invalidPort
        }

        let options = NWProtocolTCP.Options()
        options.noDelay = true

        return NWConnection(host: .ipv4(.loopback), port: port, using: .init(tls: nil, tcp: options))
    }

    init(delegate: BroadcastSenderDelegate?, broadcastPort: UInt16, rxToken : String, txToken : String) throws {
        self.delegate = delegate
        self.rxToken = rxToken
        self.txToken = txToken

        connection = try BroadcastSender.connection(broadcastPort: broadcastPort)
        connection.stateUpdateHandler = { [weak self] in
            self?.handleConnection(newState: $0)
        }
        connection.start(queue: DispatchQueue(label: "BroadcastSender.NetworkQueue"))
    }

    func handShake() {
        guard state == .started, let data = txToken.data(using: .utf8) else {
            return
        }

        state = .handshaking

        connection.send(content: data, completion: .contentProcessed({ (error) in
            self.handShakeProcessed(error: error)
        }))
    }

    func sendVideo(sampleBuffer: CMSampleBuffer) throws {
        guard state == .connected,
              !sending else {
            return
        }

        try queue.sync {
            let data: Data?
            // create data with CMSampleBuffer
            connection.send(data, completion: .contentProcessed({(error) in NSLog("data processed \(error)")}))
        }
    }

    func handShakeProcessed(error: NWError?) {
        queue.sync {
            if let error = error {
                didFail(error: error)
            } else {
                handShakeAck()
            }
        }
    }

    func handShakeAck() {
        guard state == .handshaking, let token = rxToken.data(using: .utf8) else {
            return
        }

        connection.receive(minimumIncompleteLength: token.count, maximumLength: token.count) { (data, context, final, error) in
            self.handShakeAckProcessed(data: data, error: error)
        }
    }

    func handShakeAckProcessed(data: Data?, error: NWError?) {
        queue.sync {
            guard state == .handshaking, let token = rxToken.data(using: .utf8) else {
                return
            }
            if data == token {
                state = .connected

            } else {

                didFail(error: .rejected)
            }
        }
    }

    func handleConnection(newState: NWConnection.State) {
        queue.sync {
            switch newState {
            case .ready:
                handShake()
            case .waiting(let error), .failed(let error):
                didFail(error: error)
            default:
                break
            }
        }
    }
}

If connection is successfully established, you must send screen share streams to the created NWConnection. Implement the SampleHandler class to send the captured screen share streams.

class SampleHandler: RPBroadcastSampleHandler {
    private var sender: BroadcastSender?
    ...
    override func broadcastFinished() {
        // User has requested to finish the broadcast.
        sender?.cancel()
        sender = nil
    }

    let rxToken : String = "USER_DEFINED_TOKEN_APP"
    let txToken : String = "USER_DEFINED_TOKEN_EXT"
    let broadcastPort : UInt16 = PORT_NUMBER

    override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {
        switch sampleBufferType {
        case RPSampleBufferType.video:
            do {
                if let sender = sender {
                    try autoreleasepool {
                        try sender.sendVideo(sampleBuffer: sampleBuffer)
                    }
                } else {
                    sender = try BroadcastSender(delegate: self, broadcastPort: broadcastPort, rxToken: rxToken, txToken: txToken)
                }
            } catch {
                finish(error: error)
            }
            break
        ...
        }
    }

    private func finish(error: Error) {
        sender?.cancel()
        sender = nil

        if let description = (error as? LocalizedError)?.errorDescription {
            self.finishBroadcastWithError(NSError(domain: "BroadcastSender.ErrorDomain", code: 0, userInfo: [NSLocalizedFailureReasonErrorKey: description]))
        } else {
            self.finishBroadcastWithError(error)
        }
    }
}

View the screen share (receiver)

Listen for the onPeerDidStartScreenShare event, and implement code to add the peer's screen share view when the event occurs.

final _eventHandler = PlanetKitCallEventHandler(
    onConnected: (_, __, ___) => {},
    onWaitConnected: (_) => {},
    onDisconnected: (_, __, ___, ____) => {},
    onVerified: (_, __) => {},
    onPeerScreenShareStarted: (call) => _addPeerScreenShareView,
    onPeerScreenShareStopped: (call) => _removePeerScreenShareView);

void _addPeerScreenShareView(PlanetKitCall call) {
  // show screen share view
}
void _removePeerScreenShareView(PlanetKitCall call) {
  // remove screen share view
}

To render the peer's screen share, you must use PlanetKitVideoViewBuilder to create PlanetKitVideoView and add it to PlanetKitCall.

After creating the PlanetKitVideoView for the peer's screen share, add the peer's screen share view to PlanetKitCall by calling addPeerScreenShareView(viewId).

class ScreenShareView extends StatelessWidget {
  const ScreenShareView({super.key, this.call});
  final PlanetKitCall? call;

  @override
  Widget build(BuildContext context) {
    final screenShareView = PlanetKitVideoViewBuilder.instance.create();

    screenShareView.onCreate.listen((id) {
      call?.addPeerScreenShareView(id);
    });

    screenShareView.onDispose.listen((id) {
      call?.removePeerScreenShareView(id);
    });

    return screenShareView;
  }
}

Related code examples

• 1-to-1 audio call
• 1-to-1 video call

Related documents

• 1-to-1 call flow
• Presentation - screen share