Version: 6.0

Custom audio device

By default, PlanetKit automatically detects and controls microphone and speaker devices when a call starts, uses audio captured from the microphone in the call to transmit to the peer, and plays received audio data through the speaker. However, depending on the requirements of your application, there may be a need to change this structure. For example, you might want to transmit audio from a file instead of the user's audio, or store received audio data to a file instead of playing it through the speaker.

To meet these needs, PlanetKit provides custom audio source functionality that allows direct supply of desired audio data, and custom audio sink functionality that enables direct processing of received audio data. Through these features, you can directly provide desired audio data to the PlanetKit module for transmission to the peer, or directly control audio data received from the peer.

Note

PlanetKit does not control the microphone device while a custom audio source is used, and PlanetKit does not control the speaker device while a custom audio sink is used.

Supported call type	Minimum SDK version
1-to-1 call, group call (conference)	PlanetKit 6.0

Use cases

The main use cases for custom audio devices are as follows.

Custom audio source

Using an audio file as an audio source
Using an external audio source (live broadcast or web streaming) as an audio source
Using text-to-speech (TTS) voice as an audio source
Directly controlling the microphone device from the application, rather than through PlanetKit

Custom audio sink

Recording call audio to a file without activating speaker devices
Routing audio to an external device
Network audio streaming
Directly controlling the speaker device from the application, rather than through PlanetKit

API overview

PlanetKit provides the following open classes for implementing custom audio devices:

Custom audio source: PlanetKitCustomMic
Custom audio sink: PlanetKitCustomSpeaker

You can implement custom audio devices and set or clear them through PlanetKitAudioManager as follows:

// Set custom audio source/sink
PlanetKitAudioManager.shared.setCustomMic(customMic)
PlanetKitAudioManager.shared.setCustomSpeaker(customSpeaker)

// Reset to default devices
PlanetKitAudioManager.shared.resetCustomMicToDefaultMic()
PlanetKitAudioManager.shared.resetCustomSpeakerToDefaultSpeaker()

Implementation steps

Example 1: Custom audio source - Playing an audio file

The procedure for implementing the custom audio source functionality is as follows.

Implement a custom audio source class

Create a class that inherits from PlanetKitCustomMic and implement its methods.

import PlanetKit
import AVFoundation

class AudioFileMicrophone: PlanetKitCustomMic {
    private var audioFile: AVAudioFile?
    private var audioTimer: DispatchSourceTimer?

    func playFile(url: URL) {
        do {
            audioFile = try AVAudioFile(forReading: url)
            startSendingAudio()
        } catch {
            print("Failed to load audio file: \(error)")
        }
    }

    private func startSendingAudio() {
        // Send audio every 20ms
        let queue = DispatchQueue(label: "audio.playback", qos: .userInteractive)

        audioTimer = DispatchSource.makeTimerSource(flags: [.strict], queue: queue)
        audioTimer?.setEventHandler { [weak self] in
            if let audioBuffer = self?.readNextAudioFrame() {
                self?.sendAudio(audioBuffer: audioBuffer)
            }
        }
        audioTimer?.schedule(deadline: .now(), repeating: .milliseconds(20), leeway: .milliseconds(2))
        audioTimer?.resume()
    }

    private func readNextAudioFrame() -> PlanetKitAudioBuffer? {
        guard let audioFile = audioFile else { return nil }

        // Generate 20ms worth of frames (960 frames at 48kHz)
        let frameCount: UInt32 = 960
        let sampleRate: UInt32 = 48000
        let bufferSize = frameCount * 4 // Float32 = 4 bytes per sample

        // Allocate memory and read audio data
        let buffer = UnsafeMutableRawPointer.allocate(
            byteCount: Int(bufferSize),
            alignment: MemoryLayout<Float>.alignment
        )

        // Ensure buffer is always deallocated, even if error occurs
        defer {
            buffer.deallocate()
        }

        // Read actual data from file (simplified here)
        // Logic to copy PCM data from AVAudioFile to buffer
        ...

        return PlanetKitAudioBuffer(
            frameCount: frameCount,
            sampleRate: sampleRate,
            buffer: buffer,
            bufferSize: bufferSize
        )
    }

    func stopPlayback() {
        audioTimer?.cancel()
        audioTimer = nil
    }

    deinit {
        audioTimer?.cancel()
        audioFile = nil
        print("AudioFileMicrophone cleaned up")
    }
}

Use a custom audio source

To use a custom audio source, call setCustomMic() of PlanetKitAudioManager with the implemented custom audio source instance as an argument.

// Usage
let customMic = AudioFileMicrophone()
PlanetKitAudioManager.shared.setCustomMic(customMic)

// Start file playback
customMic.playFile(url: audioFileURL)

Switch back to the microphone from a custom audio source

To switch back to the microphone from a custom audio source, call resetCustomMicToDefaultMic() of PlanetKitAudioManager.

// Cleanup after call ends
customMic.stopPlayback()
PlanetKitAudioManager.shared.resetCustomMicToDefaultMic()

Example 2: Custom audio sink - Audio recording

The procedure for implementing the custom audio sink functionality is as follows:

Implement a custom audio sink class

Create a class that inherits from PlanetKitCustomSpeaker and implement its methods.

import PlanetKit
import AVFoundation

class AudioFileRecorder: PlanetKitCustomSpeaker {
    private var audioFile: AVAudioFile?
    private var isRecording = false
    private var audioTimer: Timer?

    func startRecording(to url: URL) {
        do {
            // Setup recording file
            let format = AVAudioFormat(
                standardFormatWithSampleRate: 48000,
                channels: 1
            )!

            audioFile = try AVAudioFile(
                forWriting: url,
                settings: format.settings
            )

            isRecording = true

            // Start periodic audio data collection
            let queue = DispatchQueue(label: "audio.recording", qos: .userInteractive)

            audioTimer = DispatchSource.makeTimerSource(flags: [.strict], queue: queue)
            audioTimer?.setEventHandler { [weak self] in
                self?.handleReceivedAudio()
            }
            audioTimer?.schedule(deadline: .now(), repeating: .milliseconds(20), leeway: .milliseconds(2))
            audioTimer?.resume()

            print("Recording started: \(url)")

        } catch {
            print("Failed to create recording file: \(error)")
        }
    }

    private func handleReceivedAudio() {
        // Get audio data from PlanetKit
        if let pcmBuffer = playAudio() {
            // Save to file if recording
            if isRecording, let audioFile = audioFile {
                do {
                    try audioFile.write(from: pcmBuffer)
                } catch {
                    print("Failed to write file: \(error)")
                }
            }

            // Process or forward to system audio
            print("Audio received: \(pcmBuffer.frameCount) frames")
        }
    }

    private func playAudio() -> AVAudioPCMBuffer? {
        let frameCnt: UInt32 = 960
        let channels: UInt32 = 1
        let sampleRate: UInt32 = 48000

        guard let format = AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: Double(sampleRate), channels: channels, interleaved: false) else {
            return nil
        }

        let sampleBufferLen = UInt32(MemoryLayout<Float>.size) * frameCnt

        guard let pcmBuffer = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: frameCnt) else {
            return nil
        }

        if let floatChannelData = pcmBuffer.floatChannelData {
            var sampleBufferOutLen: Int32 = 0

            floatChannelData[0].withMemoryRebound(to: Float.self, capacity: Int(pcmBuffer.frameCapacity)) { ptr -> Void in
                let audioBuffer = PlanetKitAudioBuffer(frameCount: frameCnt, sampleRate: sampleRate, buffer: ptr, bufferSize: sampleBufferLen)

                // Get audio data from peer
                sampleBufferOutLen = playAudio(audioBuffer: audioBuffer)
            }
            pcmBuffer.frameLength = AVAudioFrameCount(sampleBufferOutLen / Int32(MemoryLayout<Float>.size))
        }

        return pcmBuffer
    }

    func stopRecording() {
        isRecording = false
        audioTimer?.cancel()
        audioTimer = nil
        audioFile = nil
        print("Recording stopped")
    }

    deinit {
        audioTimer?.cancel()
        audioFile = nil
        print("AudioFileRecorder cleaned up")
    }
}

Use a custom audio sink

To use a custom audio sink, call setCustomSpeaker() of PlanetKitAudioManager with the implemented custom audio sink instance as an argument.

// Usage
let recorder = AudioFileRecorder()
PlanetKitAudioManager.shared.setCustomSpeaker(recorder)

// Start recording - everything is handled internally
let documentsPath = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0]
let recordingURL = documentsPath.appendingPathComponent("call_recording.wav")
recorder.startRecording(to: recordingURL)

Switch back to the speaker from a custom audio sink

To switch back to the speaker from a custom audio sink, call resetCustomSpeakerToDefaultSpeaker() of PlanetKitAudioManager.

// Cleanup after call ends
recorder.stopRecording()
PlanetKitAudioManager.shared.resetCustomSpeakerToDefaultSpeaker()

Key considerations

The key considerations for using the custom audio device feature are as follows.

Audio format

Sampling rate: 48kHz recommended
Channel: Mono (1 channel)
Sample type: Float32 (signedFloat32)

Performance optimization

Memory allocation and deallocation should be minimized for performance.
Audio processing must be performed on separate threads.

AEC settings when using custom audio devices

Acoustic echo cancellation (AEC) is a technology that removes echo caused when sound from the speaker is picked up by the microphone. PlanetKit provides AEC-related features such as AEC in VQE control and application AEC reference. Generally, no separate settings are required for AEC, but when using custom audio devices, you may need to configure AEC settings depending on the situation.

If your application implementation meets the following conditions, you need to configure AEC-related settings.

Case 1: When VQE control's AEC settings are required
- Conditions
  - If any of the following conditions apply:
    - Using an input other than actual microphone devices (e.g., wav files, HTTP URLs) in a custom audio source
    - Using an output other than actual speaker devices (e.g., file recording) in a custom audio sink
- Required settings
  - Call setAcousticEchoCanceller(mode: .disabled) after starting to use custom audio devices
  - Call setAcousticEchoCanceller(mode: .intensityRecommended) after ending the use of custom audio devices
Case 2: When application AEC reference settings are required
- Conditions
  - When outputting speaker data received from PlanetKit after processing it in a custom audio sink
    - However, this only applies when the audio input is an actual microphone device (not applicable if the input is not a device where actual echo can be input, such as a wav file)
- Required settings
  - Call startUserAcousticEchoCancellerReference() after starting to use a custom audio sink
  - Provide the audio data you want to output to the speaker with putUserAcousticEchoCancellerReference()
  - Call stopUserAcousticEchoCancellerReference() after ending the use of a custom audio sink

Case 1: When VQE control's AEC settings are required

If you are using an input or output other than the actual microphone and speaker with a custom audio source or custom audio sink, you can improve call quality by disabling the use of AEC, one of the VQE features provided by call session classes PlanetKitCall and PlanetKitConference. You can call setAcousticEchoCanceller() of PlanetKitSendVoiceProcessor to configure the AEC mode.

Implement a custom audio source class

Implement a class to use an audio file as input using a custom audio source.

After the call is connected, call sendAudio() periodically to transmit audio data.

import AVFoundation
import PlanetKit

class AudioFileMicrophone: PlanetKitCustomMic {
    private var audioFile: AVAudioFile?
    private var audioTimer: DispatchSourceTimer?

    func playFile(url: URL) {
        do {
            audioFile = try AVAudioFile(forReading: url)
            startSendingAudio()
        } catch {
            print("Failed to load audio file: \(error)")
        }
    }

    private func startSendingAudio() {
        // Send audio every 20ms
        let queue = DispatchQueue(label: "audio.playback", qos: .userInteractive)

        audioTimer = DispatchSource.makeTimerSource(flags: [.strict], queue: queue)
        audioTimer?.setEventHandler { [weak self] in
            if let audioBuffer = self?.readNextAudioFrame() {
                self?.sendAudio(audioBuffer: audioBuffer)
            }
        }
        audioTimer?.schedule(deadline: .now(), repeating: .milliseconds(20), leeway: .milliseconds(2))
        audioTimer?.resume()
    }

    private func readNextAudioFrame() -> PlanetKitAudioBuffer? {
        guard let audioFile = audioFile else { return nil }

        // Generate 20ms worth of frames (960 frames at 48kHz)
        let frameCount: UInt32 = 960
        let sampleRate: UInt32 = 48000
        let bufferSize = frameCount * 4 // Float32 = 4 bytes per sample

        // Allocate memory and read audio data
        let buffer = UnsafeMutableRawPointer.allocate(
            byteCount: Int(bufferSize),
            alignment: MemoryLayout<Float>.alignment
        )

        // Ensure buffer is always deallocated, even if error occurs
        defer {
            buffer.deallocate()
        }

        // Read actual data from file (simplified here)
        // Logic to copy PCM data from AVAudioFile to buffer
        ...

        return PlanetKitAudioBuffer(
            frameCount: frameCount,
            sampleRate: sampleRate,
            buffer: buffer,
            bufferSize: bufferSize
        )
    }

    func stopPlayback() {
        audioTimer?.cancel()
        audioTimer = nil
    }

    deinit {
        audioTimer?.cancel()
        audioFile = nil
        print("AudioFileMicrophone cleaned up")
    }
}

Use a custom audio source

To use a custom audio source, call setCustomMic() of PlanetKitAudioManager with the implemented custom audio source instance as an argument, and call setAcousticEchoCanceller() of the PlanetKitSendVoiceProcessor class with disabled as an argument to disable AEC.

let conference: PlanetKitConference

func enableCustomMic() {
    let customMic = AudioFileMicrophone()
    PlanetKitAudioManager.shared.setCustomMic(customMic)

    // Start file playback
    customMic.playFile(url: audioFileURL)

    // Disable AEC
    conference.sendVoiceProcessor.setAcousticEchoCanceller(mode: .disabled) { success in
        // Add logging or UI code here
    }
    ...
}

Switch back to the microphone from a custom audio source

To switch back to the microphone from a custom audio source, call resetCustomMicToDefaultMic() of PlanetKitAudioManager, and call setAcousticEchoCanceller() of the PlanetKitSendVoiceProcessor class with intensityRecommended as an argument to enable AEC.

func disableCustomMic() {
    customMic.stopPlayback()
    PlanetKitAudioManager.shared.resetCustomMicToDefaultMic()

    // Enable AEC by default
    conference.sendVoiceProcessor.setAcousticEchoCanceller(mode: .intensityRecommended) { success in
        // Add logging or UI code here
    }
    ...
}

Case 2: When application AEC reference settings are required

You can configure whether to use AEC reference data using startUserAcousticEchoCancellerReference() and stopUserAcousticEchoCancellerReference() provided by call session classes PlanetKitCall and PlanetKitConference. After enabling AEC reference data usage, you can provide modified audio data using putUserAcousticEchoCancellerReference() of the call session class.

Implement a custom audio sink class

Implement a class to modify audio output data using a custom audio sink.

After the call is connected, call playAudio() periodically to obtain the received audio data.
Process the audio data and call putUserAcousticEchoCancellerReference() with the processed data as an argument.

import AVFoundation
import PlanetKit

class CustomMixingSpeaker: PlanetKitCustomSpeaker {

    /// Audio engine for audio processing using AVAudioEngine
    private let audioEngine = AVAudioEngine()

    ...

    /// Timer for periodic audio processing
    private var sourceTimer: DispatchSourceTimer?

    private weak var conference: PlanetKitConference?

    init(url: URL, conference: PlanetKitConference) throws {
        self.conference = conference
        super.init()
        try setupAudioEngine()
        try loadBackgroundMusic(from: url)
    }

    func start() {
        sourceTimer = DispatchSource.makeTimerSource(flags: [.strict], queue: .global(qos: .userInteractive))
        sourceTimer?.setEventHandler {
            self.processMixedAudio()
        }
        sourceTimer?.schedule(deadline: .now(), repeating: .milliseconds(sampleMilliSeconds), leeway: .milliseconds(2))
        sourceTimer?.resume()
    }

    func stop() {
        sourceTimer?.cancel()
        sourceTimer = nil
    }

    private var sampleBitrate: UInt32 { 48000 }
    private var sampleMilliSeconds: Int { 50 }
    private var sampleFrame: UInt32 { sampleBitrate / (1000 / UInt32(sampleMilliSeconds)) }

    private func processMixedAudio() {
        let frameCnt: UInt32 = sampleFrame
        let channels: UInt32 = 1
        let sampleRate: UInt32 = sampleBitrate

        guard let format = AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: Double(sampleRate), channels: channels, interleaved: false) else {
            return
        }

        let sampleBufferLen = UInt32(MemoryLayout<Float>.size) * frameCnt

        guard let pcmBuffer = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: frameCnt) else {
            return
        }

        if let floatChannelData = pcmBuffer.floatChannelData {
            var sampleBufferOutLen: Int32 = 0

            floatChannelData[0].withMemoryRebound(to: Float.self, capacity: Int(pcmBuffer.frameCapacity)) { ptr -> Void in
                let audioBuffer = PlanetKitAudioBuffer(frameCount: frameCnt, sampleRate: sampleRate, buffer: ptr, bufferSize: sampleBufferLen)

                // Get audio data from peer
                sampleBufferOutLen = playAudio(audioBuffer: audioBuffer)

                // Mix background music with audio data
                mixAudioWithBackground(audioBuffer: audioBuffer)

                // Provides audio data for AEC reference
                conference?.putUserAcousticEchoCancellerReference(frameCnt: frameCnt,
                                                                  channels: channels,
                                                                  sampleRate: sampleRate,
                                                                  sampleType: .signedFloat32,
                                                                  outData: audioBuffer.buffer,
                                                                  outDataLen: audioBuffer.bufferSize)
            }
            pcmBuffer.frameLength = AVAudioFrameCount(sampleBufferOutLen / Int32(MemoryLayout<Float>.size))
        }

        // Output to actual audio device
        playToOutputDevice(buffer: pcmBuffer)
    }

    private func mixAudioWithBackground(audioBuffer: PlanetKitAudioBuffer) {
        ...
    }

    private func playToOutputDevice(buffer: AVAudioPCMBuffer?) {
        ...
    }
}

Use a custom audio sink

To use a custom audio sink, call setCustomSpeaker() of PlanetKitAudioManager with the implemented custom audio sink instance as an argument, and call startUserAcousticEchoCancellerReference() of the call session class to start using the application AEC reference.

let conference: PlanetKitConference
var customSpeaker: CustomMixingSpeaker?

func enableCustomMixingSpeaker() {
    let backgroundMusicURL = Bundle.main.url(forResource: "background", withExtension: "mp3")!
    do {
        customSpeaker = try CustomMixingSpeaker(url: backgroundMusicURL, conference: conference)
        customSpeaker?.start()

        PlanetKitAudioManager.shared.setCustomSpeaker(customSpeaker)
    } catch {
        print("Custom speaker setup failed: \(error)")
    }

    conference?.startUserAcousticEchoCancellerReference() { success in
        print("Custom speaker AEC reference enabled \(success)")
    }
    ...
}

Switch back to the speaker from a custom audio sink

To switch back to the speaker from a custom audio sink, call resetCustomSpeakerToDefaultSpeaker() of PlanetKitAudioManager, and call stopUserAcousticEchoCancellerReference() of the call session class to stop using the application AEC reference.

func disableCustomMixingSpeaker() {
    customSpeaker?.stop()
    PlanetKitAudioManager.shared.resetCustomSpeakerToDefaultSpeaker()

    conference?.stopUserAcousticEchoCancellerReference { success in
        print("Custom speaker AEC reference disabled \(success)")
    }
    ...
}

Use cases​

Custom audio source​

Custom audio sink​

API overview​

Implementation steps​

Example 1: Custom audio source - Playing an audio file​

Implement a custom audio source class​

Use a custom audio source​

Switch back to the microphone from a custom audio source​

Example 2: Custom audio sink - Audio recording​

Implement a custom audio sink class​

Use a custom audio sink​

Switch back to the speaker from a custom audio sink​

Key considerations​

Audio format​

Performance optimization​

AEC settings when using custom audio devices​

Conditions requiring AEC-related settings​

Case 1: When VQE control's AEC settings are required​

Implement a custom audio source class​

Use a custom audio source​

Switch back to the microphone from a custom audio source​

Case 2: When application AEC reference settings are required​

Implement a custom audio sink class​

Use a custom audio sink​

Switch back to the speaker from a custom audio sink​

Related APIs​

Use cases

Custom audio source

Custom audio sink

API overview

Implementation steps

Example 1: Custom audio source - Playing an audio file

Implement a custom audio source class

Use a custom audio source

Switch back to the microphone from a custom audio source

Example 2: Custom audio sink - Audio recording

Implement a custom audio sink class

Use a custom audio sink

Switch back to the speaker from a custom audio sink

Key considerations

Audio format

Performance optimization

AEC settings when using custom audio devices

Conditions requiring AEC-related settings

Case 1: When VQE control's AEC settings are required

Implement a custom audio source class

Use a custom audio source

Switch back to the microphone from a custom audio source

Case 2: When application AEC reference settings are required

Implement a custom audio sink class

Use a custom audio sink

Switch back to the speaker from a custom audio sink

Related APIs