Custom audio source

By default, PlanetKit automatically detects and controls the microphone device when a call starts, and uses the audio captured from the microphone for the call. However, depending on the requirements of your application, you may need to change this structure. One example is to send audio from an external source instead of the user's audio.

To meet these needs, PlanetKit provides a custom audio source feature that allows users to directly supply the audio frames they want. This feature allows users to send the audio frames they want to the peer.

Note

PlanetKit does not control the microphone device while using a custom audio source.

Supported call type	Minimum SDK version
1-to-1 call, group call (conference)	PlanetKit 6.0

Use cases

You can use the custom audio source feature for the following application requirements:

• Using an audio file as an audio source
• Using an external audio source (live broadcast or web streaming) as an audio source
• Using processed audio data or data mixed from multiple audio sources as an audio source
• Directly controlling the microphone device from the application, rather than through PlanetKit

Implementation steps

To implement the custom audio source feature, follow these steps.

Implement a custom audio source class

Extend PlanetKitCustomAudioSource and implement the methods.

import com.linecorp.planetkit.audio.PlanetKitCustomAudioSource
import com.linecorp.planetkit.audio.AudioFrame
import com.linecorp.planetkit.audio.PlanetKitAudioSampleType
import java.nio.ByteBuffer
import java.nio.ByteOrder
import java.util.Timer
import kotlin.math.PI
import kotlin.math.sin
import kotlin.concurrent.fixedRateTimer

// Example implementation of a custom audio source
class MyCustomAudioSource : PlanetKitCustomAudioSource() {

    // Audio configuration settings
    private val sampleType = PlanetKitAudioSampleType.SIGNED_SHORT_16
    private val sampleRate = 48000 // 48 kHz

    // Streaming state management
    private var isStreaming = false
    private var timer: Timer? = null

    // Start audio streaming
    fun start() {
        if (!isStreaming) {
            isStreaming = true

            timer = fixedRateTimer(
                name = "CustomAudioSourceTimer",
                initialDelay = 0L,
                period = 20L // 20ms cycle
            ) {
                if (isStreaming) {
                    generateAndSendAudioFrames()
                }
            }
        }
    }

    // Stop audio streaming
    fun stop() {
        isStreaming = false
        timer?.cancel()
        timer = null
    }

    // Main loop that generates and sends audio frames every 20ms
    private fun generateAndSendAudioFrames() {
        // Generate audio data (example: sine wave)
        val audioData = generateSineWave()

        // Create ByteBuffer
        val buffer = ByteBuffer.allocate(audioData.size * 2).order(ByteOrder.LITTLE_ENDIAN)
        for (sample in audioData) {
            buffer.putShort(sample)
        }

        buffer.flip()

        // Create and send AudioFrame
        val audioFrame = createAudioFrame(buffer)
        postFrameData(audioFrame)
        audioFrame.release()
    }

    private fun generateSineWave(): ShortArray {
        val samples = ShortArray(960) // 20ms at 48kHz = 960 samples
        for (i in samples.indices) {
            val time = i.toDouble() / sampleRate
            val amplitude = (Short.MAX_VALUE * 0.3).toInt()
            samples[i] = (amplitude * sin(2 * PI * 440 * time)).toInt().toShort()
        }
        return samples
    }

    private fun createAudioFrame(buffer: ByteBuffer): AudioFrame {
        return object : AudioFrame {
            override fun getBuffer(): ByteBuffer = buffer
            override fun getSampleType(): PlanetKitAudioSampleType = sampleType
            override fun getSize(): Long = buffer.remaining().toLong()
            override fun getSamplingRate(): Int = sampleRate
            override fun getTimestamp(): Long = System.nanoTime()
            override fun release() {}
        }
    }
}

Send audio to the peer through a custom audio source

To send audio to the peer through a custom audio source, call setCustomAudioSource() of PlanetKitCall or PlanetKitConference with the PlanetKitCustomAudioSource object.

val customAudioSource = MyCustomAudioSource()

// For 1-to-1 call
fun useCustomAudioSourceForCall(call: PlanetKitCall) {
    call.setCustomAudioSource(customAudioSource)
    customAudioSource.start()
}

// For group call
fun useCustomAudioSourceForConference(conference: PlanetKitConference) {
    conference.setCustomAudioSource(customAudioSource)
    customAudioSource.start()
}

Switch back to the microphone from a custom audio source

To switch back to the microphone from a custom audio source, call clearCustomAudioSource() of PlanetKitCall or PlanetKitConference.

val customAudioSource = MyCustomAudioSource()

// For 1-to-1 call
fun useDefaultAudioSourceForCall(call: PlanetKitCall) {
    call.clearCustomAudioSource()
    customAudioSource.stop()
}

// For group call
fun useDefaultAudioSourceForConference(conference: PlanetKitConference) {
    conference.clearCustomAudioSource()
    customAudioSource.stop()
}

Key considerations

The key considerations for using the custom audio source feature are as follows.

Audio format

• Sample rate: 48 kHz or 16 kHz
• Channels: Mono (1 channel)
• Sample type: 16-bit PCM

Performance optimization

• Memory allocation and deallocation should be minimized for performance.
• Audio processing must be performed on background threads.

AEC settings when using custom audio devices

Acoustic echo cancellation (AEC) is a technology that removes echo caused when sound from the speaker is picked up by the microphone. PlanetKit provides AEC-related features such as AEC in VQE control and application AEC reference. Generally, no separate settings are required for AEC, but when using custom audio devices, you may need to configure AEC settings depending on the situation.

Conditions requiring AEC-related settings

If your application implementation meets the following conditions, you need to configure AEC-related settings.

• Case 1: When VQE control's AEC settings are required
  • Conditions
    • If any of the following conditions apply:
      • Using an input other than actual microphone devices (e.g., wav files, HTTP URLs) in a custom audio source
      • Using an output other than actual speaker devices (e.g., file recording) in a custom audio sink
  • Required settings
    • Call setAcousticEchoCancellerMode(PlanetKitVoiceProcessor.AcousticEchoCancellerMode.DISABLED after starting to use custom audio devices
    • Call setAcousticEchoCancellerMode(PlanetKitVoiceProcessor.AcousticEchoCancellerMode.INTENSITY_RECOMMENDED) after ending the use of custom audio devices
• Case 2: When application AEC reference settings are required
  • Conditions
    • When outputting speaker data received from PlanetKit after processing it in a custom audio sink
      • However, this only applies when the audio input is an actual microphone device (not applicable if the input is not a device where actual echo can be input, such as a wav file)
  • Required settings
    • Call startUserAcousticEchoCancellerReference() after starting to use a custom audio sink
    • Provide the audio data you want to output to the speaker with putUserAcousticEchoCancellerReference()
    • Call stopUserAcousticEchoCancellerReference() after ending the use of a custom audio sink

Case 1: When VQE control's AEC settings are required

If you are using an input or output other than the actual microphone and speaker with a custom audio source or custom audio sink, you can improve call quality by disabling the use of AEC, one of the VQE features provided by call session classes PlanetKitCall and PlanetKitConference. You can call setAcousticEchoCancellerMode() of PlanetKitSendVoiceProcessor to configure the AEC mode.

Implement a custom audio source class

Implement a class to use an audio file as input using a custom audio source.

• After the call is connected, call postFrameData() periodically to transmit audio data.

import com.linecorp.planetkit.audio.PlanetKitCustomAudioSource
import com.linecorp.planetkit.audio.AudioFrame
import com.linecorp.planetkit.audio.PlanetKitAudioSampleType
import java.nio.ByteBuffer
import java.nio.ByteOrder
import java.util.Timer
import kotlin.math.PI
import kotlin.math.sin
import kotlin.concurrent.fixedRateTimer

// Example implementation of a custom audio source
class MyCustomAudioSource : PlanetKitCustomAudioSource() {

    // Audio configuration settings
    private val sampleType = PlanetKitAudioSampleType.SIGNED_SHORT_16
    private val sampleRate = 48000 // 48 kHz

    // Streaming state management
    private var isStreaming = false
    private var timer: Timer? = null

    // Start audio streaming
    fun start() {
        if (!isStreaming) {
            isStreaming = true

            timer = fixedRateTimer(
                name = "CustomAudioSourceTimer",
                initialDelay = 0L,
                period = 20L // 20ms cycle
            ) {
                if (isStreaming) {
                    generateAndSendAudioFrames()
                }
            }
        }
    }

    // Stop audio streaming
    fun stop() {
        isStreaming = false
        timer?.cancel()
        timer = null
    }

    // Main loop that generates and sends audio frames every 20ms
    private fun generateAndSendAudioFrames() {
        // Generate audio data (example: sine wave)
        val audioData = generateSineWave()

        // Create ByteBuffer
        val buffer = ByteBuffer.allocate(audioData.size * 2).order(ByteOrder.LITTLE_ENDIAN)
        for (sample in audioData) {
            buffer.putShort(sample)
        }

        buffer.flip()

        // Create and send AudioFrame
        val audioFrame = createAudioFrame(buffer)
        postFrameData(audioFrame)
        audioFrame.release()
    }

    private fun generateSineWave(): ShortArray {
        val samples = ShortArray(960) // 20ms at 48kHz = 960 samples
        for (i in samples.indices) {
            val time = i.toDouble() / sampleRate
            val amplitude = (Short.MAX_VALUE * 0.3).toInt()
            samples[i] = (amplitude * sin(2 * PI * 440 * time)).toInt().toShort()
        }
        return samples
    }

    private fun createAudioFrame(buffer: ByteBuffer): AudioFrame {
        return object : AudioFrame {
            override fun getBuffer(): ByteBuffer = buffer
            override fun getSampleType(): PlanetKitAudioSampleType = sampleType
            override fun getSize(): Long = buffer.remaining().toLong()
            override fun getSamplingRate(): Int = sampleRate
            override fun getTimestamp(): Long = System.nanoTime()
            override fun release() {}
        }
    }
}

Use a custom audio source

To use a custom audio source, call setCustomAudioSource() of PlanetKitCall or PlanetKitConference with the implemented custom audio source instance as an argument, and call setAcousticEchoCancellerMode() of the PlanetKitSendVoiceProcessor class with PlanetKitVoiceProcessor.AcousticEchoCancellerMode.DISABLED as an argument to disable AEC.

val customAudioSource = MyCustomAudioSource()

// For 1-to-1 call
fun useCustomAudioSourceForCall(call: PlanetKitCall) {
    val customAudioSource = MyCustomAudioSource()
    call.setCustomAudioSource(customAudioSource)
    customAudioSource.start()

    // Disable AEC
    call.sendVoiceProcessor.setAcousticEchoCancellerMode(
        PlanetKitVoiceProcessor.AcousticEchoCancellerMode.DISABLED
    )
}

Switch back to the microphone from a custom audio source

To switch back to the microphone from a custom audio source, call clearCustomAudioSource() of PlanetKitCall or PlanetKitConference, and call setAcousticEchoCancellerMode() of the PlanetKitSendVoiceProcessor class with PlanetKitVoiceProcessor.AcousticEchoCancellerMode.INTENSITY_RECOMMENDED as an argument to enable AEC.

val customAudioSource = MyCustomAudioSource()

// For 1-to-1 call
fun useDefaultAudioSourceForCall(call: PlanetKitCall) {
    call.clearCustomAudioSource()
    customAudioSource.stop()

    // Enable AEC by default
    call.sendVoiceProcessor.setAcousticEchoCancellerMode(
        PlanetKitVoiceProcessor.AcousticEchoCancellerMode.INTENSITY_RECOMMENDED
    )
}

Case 2: When application AEC reference settings are required

You can configure whether to use AEC reference data using startUserAcousticEchoCancellerReference() and stopUserAcousticEchoCancellerReference() provided by call session classes PlanetKitCall and PlanetKitConference. After enabling AEC reference data usage, you can provide modified audio data using putUserAcousticEchoCancellerReference() of the call session class.

Implement a custom audio sink class

Implement a class to modify audio output data using a custom audio sink.

• After the call is connected, call getFrameData() periodically to obtain the received audio data.
• Process the audio data and call putUserAcousticEchoCancellerReference() with the processed data as an argument.

import android.content.Context
import android.media.AudioAttributes
import android.media.AudioAttributes.USAGE_VOICE_COMMUNICATION
import android.media.AudioFormat
import android.media.AudioManager
import android.media.AudioTrack
import com.linecorp.planetkit.audio.AudioSink
import com.linecorp.planetkit.audio.PlanetKitAudioSampleType
import java.util.Timer
import kotlin.concurrent.fixedRateTimer

class MyCustomAudioSink(private val context: Context): AudioSink() {
    companion object {
        private const val SAMPLING_RATE = 16000
        private const val PULL_AUDIO_INTERVAL_MS = 20
        private val SAMPLE_TYPE = PlanetKitAudioSampleType.SIGNED_SHORT_16
        private const val SAMPLE_COUNT_PER_PULL =
            (PULL_AUDIO_INTERVAL_MS * SAMPLING_RATE) / 1000
    }

    private var player: AudioTrack? = null
    private var timer: Timer? = null
    private var isPlaying = false

    override fun onPrepare() {
        val bufferSize = AudioTrack.getMinBufferSize(
            SAMPLING_RATE,
            AudioFormat.CHANNEL_OUT_MONO,
            AudioFormat.ENCODING_PCM_16BIT
        )
        val audioManager = context.getSystemService(Context.AUDIO_SERVICE) as AudioManager
        player = AudioTrack(
            AudioAttributes.Builder()
                .setLegacyStreamType(USAGE_VOICE_COMMUNICATION)
                .build(),
            AudioFormat.Builder()
                .setSampleRate(SAMPLING_RATE)
                .setChannelMask(AudioFormat.CHANNEL_OUT_MONO)
                .setEncoding(AudioFormat.ENCODING_PCM_16BIT)
                .build(),
            bufferSize,
            AudioTrack.MODE_STREAM,
            audioManager.generateAudioSessionId()
        )
    }

    override fun onStart() {
        if (isPlaying) return
        isPlaying = true
        player?.play()

        timer = fixedRateTimer(
            name = "AudioSinkTimer",
            initialDelay = 0L,
            period = PULL_AUDIO_INTERVAL_MS.toLong()
        ) {
            writeAudioFrame()
        }
    }

    override fun onStop() {
        isPlaying = false
        timer?.cancel()
        timer = null
        player?.pause()
        player?.flush()
    }

    private fun writeAudioFrame() {
        if (!isPlaying) return

        val frame = getFrameData(SAMPLING_RATE, SAMPLE_TYPE, SAMPLE_COUNT_PER_PULL)
        player?.write(frame.getBuffer(), frame.getSize().toInt(), AudioTrack.WRITE_BLOCKING)
        putUserAcousticEchoCancellerReference(frame)
        frame.release()
    }
}

Use a custom audio sink

To use a custom audio sink, call setAudioSink() of PlanetKitCall or PlanetKitConference with the implemented custom audio sink instance as an argument, and call startUserAcousticEchoCancellerReference() of the call session class to start using the application AEC reference.

val customAudioSink = MyCustomAudioSink(context)

// For 1-to-1 call
fun useCustomAudioSinkForCall(call: PlanetKitCall) {
    call.setAudioSink(customAudioSink)
    call.startUserAcousticEchoCancellerReference()
}

Switch back to the speaker from a custom audio sink

To switch back to the speaker from a custom audio sink, call setAudioSink() of PlanetKitCall or PlanetKitConference with the DefaultSpeakerAudioSink instance as an argument, and call stopUserAcousticEchoCancellerReference() of the call session class to stop using the application AEC reference.

// For 1-to-1 call
fun useDefaultAudioSinkForCall(call: PlanetKitCall) {
    val defaultAudioSink = DefaultSpeakerAudioSink.getInstance()
    call.setAudioSink(defaultAudioSink)

    call.stopUserAcousticEchoCancellerReference()
}

Related APIs

The APIs related to custom audio source functionality are as follows:

• PlanetKitCustomAudioSource
• AudioFrame
• PlanetKitAudioSampleType
• PlanetKitCallのsetCustomAudioSource()
• PlanetKitCallのclearCustomAudioSource()
• PlanetKitConferenceのsetCustomAudioSource()
• PlanetKitConferenceのclearCustomAudioSource()