Custom audio device
By default, PlanetKit automatically detects and controls microphone and speaker devices when a call starts, uses audio captured from the microphone in the call to transmit to the peer, and plays received audio data through the speaker. However, depending on the requirements of your application, there may be a need to change this structure. For example, you might want to transmit audio from a file instead of the user's audio, or store received audio data to a file instead of playing it through the speaker.
To meet these needs, PlanetKit provides custom audio source functionality that allows direct supply of desired audio data, and custom audio sink functionality that enables direct processing of received audio data. Through these features, you can directly provide desired audio data to the PlanetKit module for transmission to the peer, or directly control audio data received from the peer.
PlanetKit does not control the microphone device while a custom audio source is used, and PlanetKit does not control the speaker device while a custom audio sink is used.
| Supported call type | Minimum SDK version |
|---|---|
| 1-to-1 call, group call (conference) | PlanetKit 6.0 |
Use cases
The main use cases for custom audio devices are as follows.
Custom audio source
- Using an audio file as an audio source
- Using an external audio source (live broadcast or web streaming) as an audio source
- Using processed audio data or data mixed from multiple audio sources as an audio source
- Directly controlling the microphone device from the application, rather than through PlanetKit
Custom audio sink
- Recording call audio to a file without activating speaker devices
- Routing audio to an external device
- Network audio streaming
- Directly controlling the speaker device from the application, rather than through PlanetKit
Implementation steps
Example 1: Custom audio source - Playing an audio file
The procedure for implementing the custom audio source functionality is as follows.
Implement a custom audio source class
Extend PlanetKitCustomAudioSource and implement the methods.
import com.linecorp.planetkit.audio.PlanetKitCustomAudioSource
import com.linecorp.planetkit.audio.AudioFrame
import com.linecorp.planetkit.audio.PlanetKitAudioSampleType
import com.linecorp.planetkit.audio.PlanetKitAudioSampleRate
import java.nio.ByteBuffer
import java.nio.ByteOrder
import java.util.Timer
import kotlin.math.PI
import kotlin.math.sin
import kotlin.concurrent.fixedRateTimer
// Example implementation of a custom audio source
class MyCustomAudioSource : PlanetKitCustomAudioSource() {
// Audio configuration settings
private val sampleType = PlanetKitAudioSampleType.SIGNED_SHORT_16
private val sampleRate = PlanetKitAudioSampleRate.SAMPLE_RATE_48K
// Streaming state management
private var isStreaming = false
private var timer: Timer? = null
// Start audio streaming
fun start() {
if (!isStreaming) {
isStreaming = true
timer = fixedRateTimer(
name = "CustomAudioSourceTimer",
initialDelay = 0L,
period = 20L // 20ms cycle
) {
if (isStreaming) {
generateAndSendAudioFrames()
}
}
}
}
// Stop audio streaming
fun stop() {
isStreaming = false
timer?.cancel()
timer = null
}
// Main loop that generates and sends audio frames every 20ms
private fun generateAndSendAudioFrames() {
// Generate audio data (example: sine wave)
val audioData = generateSineWave()
// Create ByteBuffer
val buffer = ByteBuffer.allocate(audioData.size * 2).order(ByteOrder.LITTLE_ENDIAN)
for (sample in audioData) {
buffer.putShort(sample)
}
buffer.flip()
// Create and send AudioFrame
val audioFrame = createAudioFrame(buffer)
postFrameData(audioFrame)
audioFrame.release()
}
private fun generateSineWave(): ShortArray {
val samples = ShortArray(960) // 20ms at 48kHz = 960 samples
for (i in samples.indices) {
val time = i.toDouble() / sampleRate.sampleRate
val amplitude = (Short.MAX_VALUE * 0.3).toInt()
samples[i] = (amplitude * sin(2 * PI * 440 * time)).toInt().toShort()
}
return samples
}
private fun createAudioFrame(buffer: ByteBuffer): AudioFrame {
return object : AudioFrame {
override fun getBuffer(): ByteBuffer = buffer
override fun getSampleType(): PlanetKitAudioSampleType = sampleType
override fun getSize(): Long = buffer.remaining().toLong()
override fun getSamplingRate(): PlanetKitAudioSampleRate = sampleRate
override fun getTimestamp(): Long = System.nanoTime()
override fun release() {}
}
}
}
Send audio to the peer through a custom audio source
Call setCustomAudioSource() of PlanetKitCall or PlanetKitConference with the PlanetKitCustomAudioSource instance as an argument and start the custom audio source.
Once the custom audio source is started, audio data is transmitted through the postFrameData() method called from the custom audio source.
val customAudioSource = MyCustomAudioSource()
// For 1-to-1 call
fun useCustomAudioSourceForCall(call: PlanetKitCall) {
call.setCustomAudioSource(customAudioSource)
// Execute the start logic you created.
customAudioSource.start()
}
// For group call
fun useCustomAudioSourceForConference(conference: PlanetKitConference) {
conference.setCustomAudioSource(customAudioSource)
// Execute the start logic you created.
customAudioSource.start()
}
Switch back to the microphone from a custom audio source
To switch back to the microphone from a custom audio source, call clearCustomAudioSource() of PlanetKitCall or PlanetKitConference and stop the custom audio source.
val customAudioSource = MyCustomAudioSource()
// For 1-to-1 call
fun useDefaultAudioSourceForCall(call: PlanetKitCall) {
call.clearCustomAudioSource()
// Execute the stop logic you created.
customAudioSource.stop()
}
// For group call
fun useDefaultAudioSourceForConference(conference: PlanetKitConference) {
conference.clearCustomAudioSource()
// Execute the stop logic you created.
customAudioSource.stop()
}
Example 2: Custom audio sink - Audio recording
The procedure for implementing the custom audio sink functionality is as follows:
Implement a custom audio sink class
Extend PlanetKitCustomAudioSink and implement the methods.
/**
* Example implementation of PlanetKitCustomAudioSink for recording audio to WAV file.
*
* This example demonstrates how to:
* - Pull audio data from PlanetKit periodically
* - Write audio data to WAV file with proper header
* - Support reusable start/stop operations
*
* The recorded WAV file is saved to the app's external files directory.
*/
class MyCustomAudioRecorder(private val context: Context) : PlanetKitCustomAudioSink() {
companion object {
// Audio configuration
private const val CHANNEL_COUNT = 1 // PlanetKit only supports mono channel
private val SAMPLE_RATE = PlanetKitAudioSampleRate.SAMPLE_RATE_16K
private const val PULL_INTERVAL_MS = 20 // Pull audio data every 20ms
private val SAMPLE_TYPE = PlanetKitAudioSampleType.SIGNED_SHORT_16
// Calculate sample count per pull: 20ms * 16000Hz / 1000 = 320 samples
private val SAMPLE_COUNT = (PULL_INTERVAL_MS * SAMPLE_RATE.sampleRate) / 1000
}
private var timer: Timer? = null
private var fileOutputStream: FileOutputStream? = null
private var fileChannel: FileChannel? = null
private var dataChunkSize: Long = 0
private var outputFile: File? = null
/**
* Starts audio recording.
* Creates output file and timer if not already initialized (supports reusability).
*/
fun start() {
// Initialize file resources if not exists (for reusability after stop)
if (fileOutputStream == null) {
outputFile = createOutputFile()
fileOutputStream = FileOutputStream(outputFile)
fileChannel = fileOutputStream?.channel
writeWavHeader()
}
// Start timer to pull and write audio data every PULL_INTERVAL_MS
timer = fixedRateTimer(
name = "AudioRecorderTimer",
daemon = false,
initialDelay = 0L,
period = PULL_INTERVAL_MS.toLong()
) {
pullAndWriteAudio()
}
}
/**
* Stops audio recording and finalizes the WAV file.
* Updates the file header with actual data sizes and closes all file resources.
*/
fun stop() {
// Stop and cleanup timer
timer?.cancel()
timer = null
// Update RIFF ChunkSize field: total file size minus 8 bytes
fileChannel?.position(4)
fileChannel?.write(
ByteBuffer.allocate(4)
.order(ByteOrder.LITTLE_ENDIAN)
.putInt((dataChunkSize + 36).toInt())
.flip() as ByteBuffer
)
// Update data Subchunk2Size field: actual audio data size
fileChannel?.position(40)
fileChannel?.write(
ByteBuffer.allocate(4)
.order(ByteOrder.LITTLE_ENDIAN)
.putInt(dataChunkSize.toInt())
.flip() as ByteBuffer
)
// Close file resources
fileChannel?.close()
fileOutputStream?.close()
fileChannel = null
fileOutputStream = null
}
/**
* Creates a new WAV file with a timestamp in the external files directory.
*/
private fun createOutputFile(): File {
val timestamp = System.currentTimeMillis()
val file = File(context.getExternalFilesDir(null), "audio_record_$timestamp.wav")
if (!file.exists()) {
file.createNewFile()
}
return file
}
/**
* Writes WAV file header with format information.
* The ChunkSize and data size fields are initialized to 0 and updated in stop().
*/
private fun writeWavHeader() {
dataChunkSize = 0
val bitDepth: Short = when (SAMPLE_TYPE) {
PlanetKitAudioSampleType.SIGNED_SHORT_16 -> 16
else -> 32
}
// Prepare fmt chunk parameters in little endian byte order
// 14 bytes: NumChannels(2) + SampleRate(4) + ByteRate(4) + BlockAlign(2) + BitsPerSample(2)
val littleBytes = ByteBuffer.allocate(14)
.order(ByteOrder.LITTLE_ENDIAN)
.putShort(CHANNEL_COUNT.toShort()) // NumChannels
.putInt(SAMPLE_RATE.sampleRate) // SampleRate
.putInt(SAMPLE_RATE.sampleRate * CHANNEL_COUNT * (bitDepth / 8)) // ByteRate
.putShort((CHANNEL_COUNT * (bitDepth / 8)).toShort()) // BlockAlign
.putShort(bitDepth) // BitsPerSample
.array()
// Write WAV file header (44 bytes total)
fileOutputStream?.write(
byteArrayOf(
// RIFF header (12 bytes)
'R'.code.toByte(), 'I'.code.toByte(), 'F'.code.toByte(), 'F'.code.toByte(), // ChunkID
0, 0, 0, 0, // ChunkSize = file_size - 8 (updated at position 4 on stop)
'W'.code.toByte(), 'A'.code.toByte(), 'V'.code.toByte(), 'E'.code.toByte(), // Format
// fmt chunk (24 bytes)
'f'.code.toByte(), 'm'.code.toByte(), 't'.code.toByte(), ' '.code.toByte(), // Subchunk1ID
16, 0, 0, 0, // Subchunk1Size = 16 for PCM
1, 0, // AudioFormat (PCM = 1)
littleBytes[0], littleBytes[1], // NumChannels
littleBytes[2], littleBytes[3], littleBytes[4], littleBytes[5], // SampleRate
littleBytes[6], littleBytes[7], littleBytes[8], littleBytes[9], // ByteRate
littleBytes[10], littleBytes[11], // BlockAlign
littleBytes[12], littleBytes[13], // BitsPerSample
// data chunk header (8 bytes)
'd'.code.toByte(), 'a'.code.toByte(), 't'.code.toByte(), 'a'.code.toByte(), // Subchunk2ID
0, 0, 0, 0 // Subchunk2Size = audio data size (updated at position 40 on stop)
)
)
}
/**
* Pulls audio data from PlanetKit and writes it to the WAV file.
* Called periodically by the timer.
*/
private fun pullAndWriteAudio() {
try {
// Pull audio frame from PlanetKit
val frameData = getFrameData(SAMPLE_RATE, SAMPLE_TYPE, SAMPLE_COUNT)
// Write audio data to file and track total size
dataChunkSize += frameData.getSize()
fileChannel?.write(frameData.getBuffer())
} catch (e: Exception) {
e.printStackTrace()
}
}
}
Use a custom audio sink
Call setCustomAudioSink() of PlanetKitCall or PlanetKitConference with the PlanetKitCustomAudioSink instance as an argument and start the custom audio sink.
Once the custom audio sink is started, audio data is received through the getFrameData() method called from the custom audio sink.
val customAudioSink = MyCustomAudioRecorder(context)
// For 1-to-1 call
fun useCustomSinkSourceForCall(call: PlanetKitCall) {
call.setCustomAudioSink(customAudioSink)
// Execute the start logic you created.
customAudioSink.start()
}
// For group call
fun useCustomSinkForConference(conference: PlanetKitConference) {
conference.setCustomAudioSink(customAudioSink)
// Execute the start logic you created.
customAudioSink.start()
}
Switch back to the speaker from a custom audio sink
To switch back to the speaker from a custom audio sink, call clearCustomAudioSink() of PlanetKitCall or PlanetKitConference and stop the custom audio sink.
val customAudioSink = MyCustomAudioRecorder(context)
// For 1-to-1 call
fun useDefaultAudioSinkForCall(call: PlanetKitCall) {
call.clearCustomAudioSink()
// Execute the stop logic you created.
customAudioSink.stop()
}
// For group call
fun useDefaultAudioSinkForConference(conference: PlanetKitConference) {
conference.clearCustomAudioSink()
// Execute the stop logic you created.
customAudioSink.stop()
}
Key considerations
The key considerations for using the custom audio device feature are as follows.
Audio format
- Sample rate: 48 kHz or 16 kHz
- Channels: Mono (1 channel)
- Sample type: 16-bit PCM
Performance optimization
- Memory allocation and deallocation should be minimized for performance.
- Audio processing must be performed on background threads.
AEC settings when using custom audio devices
Acoustic echo cancellation (AEC) is a technology that removes echo caused when sound from the speaker is picked up by the microphone. PlanetKit provides AEC-related features such as AEC in VQE control and application AEC reference. Generally, no separate settings are required for AEC, but when using custom audio devices, you may need to configure AEC settings depending on the situation.
Conditions requiring AEC-related settings
If your application implementation meets the following conditions, you need to configure AEC-related settings.
- Case 1: When VQE control's AEC settings are required
- Conditions
- If any of the following conditions apply:
- Using an input other than actual microphone devices (e.g., WAV files, HTTP URLs) in a custom audio source
- Using an output other than actual speaker devices (e.g., file recording) in a custom audio sink
- If any of the following conditions apply:
- Required settings
- Call
setAcousticEchoCancellerMode(PlanetKitVoiceProcessor.AcousticEchoCancellerMode.DISABLED)after starting to use custom audio devices - Call
setAcousticEchoCancellerMode(PlanetKitVoiceProcessor.AcousticEchoCancellerMode.INTENSITY_RECOMMENDED)after ending the use of custom audio devices
- Call
- Conditions
- Case 2: When application AEC reference settings are required
- Conditions
- When outputting speaker data received from PlanetKit after processing it in a custom audio sink
- However, this only applies when the audio input is an actual microphone device (not applicable if the input is not a device where actual echo can be input, such as a WAV file)
- When outputting speaker data received from PlanetKit after processing it in a custom audio sink
- Required settings
- Call
startUserAcousticEchoCancellerReference()after starting to use a custom audio sink - Provide the audio data you want to output to the speaker with
putUserAcousticEchoCancellerReference() - Call
stopUserAcousticEchoCancellerReference()after ending the use of a custom audio sink
- Call
- Conditions
Case 1: When VQE control's AEC settings are required
If you are using an input or output other than the actual microphone and speaker with a custom audio source or custom audio sink, you can improve call quality by disabling the use of AEC, one of the VQE features provided by call session classes PlanetKitCall and PlanetKitConference. You can call setAcousticEchoCancellerMode() of PlanetKitSendVoiceProcessor to configure the AEC mode.
Implement a custom audio source class
Implement a class to use an audio file as input using a custom audio source.
- After the call is connected, call
postFrameData()periodically to transmit audio data.
import com.linecorp.planetkit.audio.PlanetKitCustomAudioSource
import com.linecorp.planetkit.audio.AudioFrame
import com.linecorp.planetkit.audio.PlanetKitAudioSampleType
import com.linecorp.planetkit.audio.PlanetKitAudioSampleRate
import java.nio.ByteBuffer
import java.nio.ByteOrder
import java.util.Timer
import kotlin.math.PI
import kotlin.math.sin
import kotlin.concurrent.fixedRateTimer
// Example implementation of a custom audio source
class MyCustomAudioSource : PlanetKitCustomAudioSource() {
// Audio configuration settings
private val sampleType = PlanetKitAudioSampleType.SIGNED_SHORT_16
private val sampleRate = PlanetKitAudioSampleRate.SAMPLE_RATE_48K
// Streaming state management
private var isStreaming = false
private var timer: Timer? = null
// Start audio streaming
fun start() {
if (!isStreaming) {
isStreaming = true
timer = fixedRateTimer(
name = "CustomAudioSourceTimer",
initialDelay = 0L,
period = 20L // 20ms cycle
) {
if (isStreaming) {
generateAndSendAudioFrames()
}
}
}
}
// Stop audio streaming
fun stop() {
isStreaming = false
timer?.cancel()
timer = null
}
// Main loop that generates and sends audio frames every 20ms
private fun generateAndSendAudioFrames() {
// Generate audio data (example: sine wave)
val audioData = generateSineWave()
// Create ByteBuffer
val buffer = ByteBuffer.allocate(audioData.size * 2).order(ByteOrder.LITTLE_ENDIAN)
for (sample in audioData) {
buffer.putShort(sample)
}
buffer.flip()
// Create and send AudioFrame
val audioFrame = createAudioFrame(buffer)
postFrameData(audioFrame)
audioFrame.release()
}
private fun generateSineWave(): ShortArray {
val samples = ShortArray(960) // 20ms at 48kHz = 960 samples
for (i in samples.indices) {
val time = i.toDouble() / sampleRate.sampleRate
val amplitude = (Short.MAX_VALUE * 0.3).toInt()
samples[i] = (amplitude * sin(2 * PI * 440 * time)).toInt().toShort()
}
return samples
}
private fun createAudioFrame(buffer: ByteBuffer): AudioFrame {
return object : AudioFrame {
override fun getBuffer(): ByteBuffer = buffer
override fun getSampleType(): PlanetKitAudioSampleType = sampleType
override fun getSize(): Long = buffer.remaining().toLong()
override fun getSamplingRate(): PlanetKitAudioSampleRate = sampleRate
override fun getTimestamp(): Long = System.nanoTime()
override fun release() {}
}
}
}
Use a custom audio source
To use a custom audio source, call setCustomAudioSource() of PlanetKitCall or PlanetKitConference with the implemented custom audio source instance as an argument, and call setAcousticEchoCancellerMode() of the PlanetKitSendVoiceProcessor class with PlanetKitVoiceProcessor.AcousticEchoCancellerMode.DISABLED as an argument to disable AEC.
val customAudioSource = MyCustomAudioSource()
// For 1-to-1 call
fun useCustomAudioSourceForCall(call: PlanetKitCall) {
val customAudioSource = MyCustomAudioSource()
call.setCustomAudioSource(customAudioSource)
customAudioSource.start()
// Disable AEC
call.sendVoiceProcessor.setAcousticEchoCancellerMode(
PlanetKitVoiceProcessor.AcousticEchoCancellerMode.DISABLED
)
}
Switch back to the microphone from a custom audio source
To switch back to the microphone from a custom audio source, call clearCustomAudioSource() of PlanetKitCall or PlanetKitConference, and call setAcousticEchoCancellerMode() of the PlanetKitSendVoiceProcessor class with PlanetKitVoiceProcessor.AcousticEchoCancellerMode.INTENSITY_RECOMMENDED as an argument to enable AEC.
val customAudioSource = MyCustomAudioSource()
// For 1-to-1 call
fun useDefaultAudioSourceForCall(call: PlanetKitCall) {
call.clearCustomAudioSource()
customAudioSource.stop()
// Enable AEC by default
call.sendVoiceProcessor.setAcousticEchoCancellerMode(
PlanetKitVoiceProcessor.AcousticEchoCancellerMode.INTENSITY_RECOMMENDED
)
}
Case 2: When application AEC reference settings are required
You can configure whether to use AEC reference data using startUserAcousticEchoCancellerReference() and stopUserAcousticEchoCancellerReference() provided by call session classes PlanetKitCall and PlanetKitConference. After enabling AEC reference data usage, you can provide modified audio data using putUserAcousticEchoCancellerReference() of the call session class.
Implement a custom audio sink class
Implement a class to modify audio output data using a custom audio sink.
- After the call is connected, call
getFrameData()periodically to obtain the received audio data. - Process the audio data and call
putUserAcousticEchoCancellerReference()with the processed data as an argument.
/**
* Example implementation of PlanetKitCustomAudioSink.
*
* This example demonstrates how to:
* - Pull audio data from PlanetKit
* - Render audio using Android AudioTrack
* - Handle acoustic echo cancellation (AEC) reference
*/
class MyCustomAudioSink(private val context: Context) : PlanetKitCustomAudioSink() {
companion object {
// Configure audio parameters
private val SAMPLE_RATE = PlanetKitAudioSampleRate.SAMPLE_RATE_16K
private const val PULL_INTERVAL_MS = 20
private val SAMPLE_TYPE = PlanetKitAudioSampleType.SIGNED_SHORT_16
private val SAMPLE_COUNT = (PULL_INTERVAL_MS * SAMPLE_RATE.sampleRate) / 1000
}
private var timer: Timer? = null
private var audioTrack: AudioTrack? = null
fun start() {
// Create AudioTrack if not exists (for reusability after stop)
if (audioTrack == null) {
audioTrack = createAudioTrack()
}
// Start audio playback
audioTrack?.play()
// Create timer to pull audio data periodically
timer = fixedRateTimer(
name = "AudioPullTimer",
daemon = false,
initialDelay = 0L,
period = PULL_INTERVAL_MS.toLong()
) {
pullAndRenderAudio()
}
}
fun stop() {
// Stop timer
timer?.cancel()
timer = null
// Stop and release AudioTrack
audioTrack?.flush()
audioTrack?.stop()
audioTrack?.release()
audioTrack = null
}
private fun createAudioTrack(): AudioTrack {
val bufferSize = AudioTrack.getMinBufferSize(
SAMPLE_RATE.sampleRate,
AudioFormat.CHANNEL_OUT_MONO,
AudioFormat.ENCODING_PCM_16BIT
)
val audioManager = context.getSystemService(Context.AUDIO_SERVICE) as AudioManager
return AudioTrack(
AudioAttributes.Builder()
.setUsage(AudioAttributes.USAGE_VOICE_COMMUNICATION)
.build(),
AudioFormat.Builder()
.setSampleRate(SAMPLE_RATE.sampleRate)
.setChannelMask(AudioFormat.CHANNEL_OUT_MONO)
.setEncoding(AudioFormat.ENCODING_PCM_16BIT)
.build(),
bufferSize,
AudioTrack.MODE_STREAM,
audioManager.generateAudioSessionId()
)
}
private fun pullAndRenderAudio() {
try {
// Pull audio data from PlanetKit
val frameData = getFrameData(SAMPLE_RATE, SAMPLE_TYPE, SAMPLE_COUNT)
// Write to AudioTrack
audioTrack?.write(
frameData.getBuffer(),
frameData.getSize().toInt(),
AudioTrack.WRITE_BLOCKING
)
// Provide AEC reference for echo cancellation
putUserAcousticEchoCancellerReference(frameData)
} catch (e: Exception) {
// Handle error
e.printStackTrace()
}
}
}
Use a custom audio sink
To use a custom audio sink, call setCustomAudioSink() of PlanetKitCall or PlanetKitConference with the implemented custom audio sink instance as an argument, start the custom audio sink, and then call startUserAcousticEchoCancellerReference() of the call session class to start using the application AEC reference.
val customAudioSink = MyCustomAudioSink(context)
// For 1-to-1 call
fun useCustomAudioSinkForCall(call: PlanetKitCall) {
call.setCustomAudioSink(customAudioSink)
customAudioSink.start()
call.startUserAcousticEchoCancellerReference()
}
Switch back to the speaker from a custom audio sink
To switch back to the speaker from a custom audio sink, call clearCustomAudioSink() of PlanetKitCall or PlanetKitConference, stop the custom audio sink, and then call stopUserAcousticEchoCancellerReference() of the call session class to stop using the application AEC reference.
val customAudioSink = MyCustomAudioSink(context)
// For 1-to-1 call
fun useDefaultAudioSinkForCall(call: PlanetKitCall) {
call.clearCustomAudioSink()
customAudioSink.stop()
call.stopUserAcousticEchoCancellerReference()
}
Related APIs
The APIs related to custom audio device functionality are as follows:
Common
-
PlanetKitCustomAudioSource -
PlanetKitCustomAudioSink -
AudioFrame -
PlanetKitAudioSampleType -
PlanetKitAudioSampleRate
1-to-1 calls
-
setCustomAudioSource()ofPlanetKitCall -
clearCustomAudioSource()ofPlanetKitCall -
setCustomAudioSink()ofPlanetKitCall -
clearCustomAudioSink()ofPlanetKitCall