Version: 6.0

Custom audio device

By default, PlanetKit automatically detects and controls microphone and speaker devices when a call starts, uses audio captured from the microphone in the call to transmit to the peer, and plays received audio data through the speaker. However, depending on the requirements of your application, there may be a need to change this structure. For example, you might want to transmit audio from a file instead of the user's audio, or store received audio data to a file instead of playing it through the speaker.

To meet these needs, PlanetKit provides custom audio source functionality that allows direct supply of desired audio data, and custom audio sink functionality that enables direct processing of received audio data. Through these features, you can directly provide desired audio data to the PlanetKit module for transmission to the peer, or directly control audio data received from the peer.

Note

PlanetKit does not control the microphone device while a custom audio source is used, and PlanetKit does not control the speaker device while a custom audio sink is used.

Supported call type	Minimum SDK version
1-to-1 call, group call (conference)	PlanetKit 6.0

Use cases

The main use cases for custom audio devices are as follows.

Custom audio source

Using an audio file as an audio source
Using an external audio source (live broadcast or web streaming) as an audio source
Using text-to-speech (TTS) voice as an audio source
Directly controlling the microphone device from the application, rather than through PlanetKit

Custom audio sink

Recording call audio to a file without activating speaker devices
Routing audio to an external device
Network audio streaming
Directly controlling the speaker device from the application, rather than through PlanetKit

API overview

PlanetKit provides the following classes for implementing custom audio devices:

Custom audio source: PlanetKit::CustomMic
Custom audio sink: PlanetKit::CustomSpeaker

You can implement custom audio devices and set or clear them through PlanetKit::AudioManager as follows:

// Set custom audio source/sink
PlanetKit::AudioManager::ChangeMic(CustomMicPtr pCustomMic)
PlanetKit::AudioManager::ChangeSpeaker(CustomSpeakerPtr pCustomSpeaker)

// Reset to default devices
PlanetKit::AudioManager::ChangeMic(AudioDeviceInfoPtr pInfo = nullptr)
PlanetKit::AudioManager::ChangeSpeaker(AutioDeviceInfoPtr pInfo = nullptr)

Implementation steps

Example 1: Custom audio source - Playing an audio file

The procedure for implementing the custom audio source functionality is as follows.

Implement a custom audio source class

Create a class that inherits from PlanetKit::CustomMic and implement its methods.

#include <thread>
#include <atomic>
#include <vector>

#include "PlanetKitCustomMic.h"
#include "PlanetKitManager.h"

class AudioFileMicrophone : public PlanetKit::CustomMic {
public:
  AudioFileMicrophone() {
    // Allocate buffer for 48kHz 20ms = 960 samples
    m_audioBuffer.resize(960);
  }

  virtual ~AudioFileMicrophone() = default;

  virtual bool SetVolumeLevel(float fVolume) override { return true; };
  virtual float GetVolumeLevel() override { return 1.0f; };
  virtual float GetPeakValue() override { return 1.0f; };
  virtual bool IsRunning() override { return m_isRunning; }

  virtual bool RegisterVolumeLevelChangedEvent(PlanetKit::AudioVolumeLevelChangedEventPtr) override { return true; }
  virtual bool DeregisterVolumeLevelChangedEvent(PlanetKit::AudioVolumeLevelChangedEventPtr) override { return true; }

  virtual PlanetKit::AudioDeviceInfoPtr GetDeviceInfo() override { return nullptr; }

  bool Start() {
    if (m_isRunning.load()) {
      return true; // Already running
    }

    m_isRunning = true;
    m_audioThread = std::thread(&CustomMic::AudioThreadFunc, this);

    return true;
  }

  bool Stop() {
    if (!m_isRunning.load()) {
      return true; // Already stopped
    }

    m_isRunning = false;

    if (m_audioThread.joinable()) {
      m_audioThread.join();
    }

    return true;
  }

private:
  void AudioThreadFunc() {
    while (m_isRunning.load()) {
      // Prepare audio data structure
      PlanetKit::SAudioData audioData;
      ZeroMemory(&audioData, sizeof(PlanetKit::SAudioData));

      audioData.unAudioDataSamplingRate = 48000; // 48kHz

      // Calculate samples needed for the requested duration
      unsigned int samplesFor20ms = audioData.unAudioDataSamplingRate * 20 / 1000;

      audioData.unAudioDataSampleCount = samplesFor20ms;
      audioData.eAudioDataSampleFormat = PlanetKit::EAudioDataSampleType::PLNK_AUDIO_DATA_SAMPLE_TYPE_FLOAT_32;
      audioData.unBufferSize = (unsigned int)(samplesFor20ms * sizeof(float));
      audioData.ucBuffer = reinterpret_cast<unsigned char*>(m_audioBuffer.data());

      // Read audio data from file using audioData.ucBuffer
      ...

      // Send audio data to PlanetKit
      PutAudioData(audioData);

      // Sleep for 20ms (50fps)
      std::this_thread::sleep_for(std::chrono::milliseconds(20));
    }
  }

private:
  std::thread m_audioThread;
  std::atomic<bool> m_isRunning;
  std::vector<float> m_audioBuffer;
};

Use a custom audio source

To use a custom audio source, call ChangeMic() of PlanetKit::AudioManager with the implemented custom audio source instance as an argument.

// Create CustomMic instance by using AudioFileMicrophone class
PlanetKit::CustomMicPtr customMic = PlanetKit::MakeAutoPtr<AudioFileMicrophone>();

// Set CustomMic instance to PlanetKit
auto audioManager = PlanetKit::PlanetKitManager::GetInstance()->GetAudioManager();
audioManager->ChangeMic(customMic);

customMic.as<AudioFileMicrophone>()->Start();

Switch back to the microphone from a custom audio source

To switch back to the microphone from a custom audio source, obtain the instance of the microphone you want to use and call ChangeMic() of PlanetKit::AudioManager with this instance as an argument.

auto audioManager = PlanetKit::PlanetKitManager::GetInstance()->GetAudioManager();

// Get system default mic
auto defaultMic =  audioManager->GetDefaultMicInfo();

// Change mic
audioManager->ChangeMic(defaultMic);

Example 2: Custom audio sink - Audio recording

The procedure for implementing the custom audio sink functionality is as follows:

Implement a custom audio sink class

Create a class that inherits from PlanetKit::CustomSpeaker and implement its methods.

#include "PlanetKitCustomSpeaker.h"
#include "PlanetKitManager.h"

#include <string>
#include <fstream>
#include <atomic>
#include <thread>

class AudioFileRecorder : public PlanetKit::CustomSpeaker {
public:
  AudioFileRecorder();
  virtual ~AudioFileRecorder() = default;

  virtual bool IsRunning() override { return m_isRecording.load(); }
  virtual bool SetVolumeLevel(float fVolume) override { return true; }
  virtual float GetVolumeLevel() override { return 1.0f; }
  virtual float GetPeakValue() override { return 1.0f; }
  virtual bool RegisterVolumeLevelChangedEvent(PlanetKit::AudioVolumeLevelChangedEventPtr) { return true; }
  virtual bool DeregisterVolumeLevelChangedEvent(PlanetKit::AudioVolumeLevelChangedEventPtr) { return true; }
  virtual PlanetKit::AudioDeviceInfoPtr GetDeviceInfo() override { return nullptr; }
  virtual bool PlayFile(const PlanetKit::WString& strFilePath, unsigned int unLoop) { return true; }
  virtual bool StopPlay() override { return true; }
  virtual bool IsCustomSpeaker() override { return true; }

  bool Start(const std::string& filePath) {
    if (m_isRecording.load()) {
      return false;
    }

    // Open file
    m_recordingFile.open(filePath, std::ios::binary);
    if (!m_recordingFile.is_open()) {
      return false;
    }

    m_hAudioThread = std::thread(std::bind(&AudioFileRecorder::PullAudioThread, this));

    return true;
  }

  void Stop() {
    if (!m_isRecording.load()) {
      return;
    }

    m_isRecording.store(false);

    if (m_hAudioThread.joinable()) {
      m_hAudioThread.join();
    }

    if (m_recordingFile.is_open()) {
      m_recordingFile.close();
    }
  }

private:
  void PullAudioThread() {
    m_isRecording.store(true);
    while(m_isRunning) {
      // 100Hz update rate (10ms intervals) - matches sample buffer size
      Sleep(10);

      if(this->PullAudioData(m_audioData) == true) {
        // Write audioData to file using m_audioData.ucBuffer
      }
    }
  }


private:
  std::atomic<bool> m_isRecording{false};
  std::thread m_hAudioThread;

  std::ofstream m_recordingFile;
  PlanetKit::SAudioData m_audioData;
}

Use a custom audio sink

To use a custom audio sink, call ChangeSpeaker() of PlanetKit::AudioManager with the implemented custom audio sink instance as an argument.

// Create CustomSpeaker instance by using AudioFileRecorder class
PlanetKit::CustomSpeakerPtr customSpeaker = PlanetKit::MakeAutoPtr<AudioFileRecorder>();

// Set CustomSpeaker instance to PlanetKit
auto audioManager = PlanetKit::PlanetKitManager::GetInstance()->GetAudioManager();
audioManager->ChangeSpeaker(customSpeaker);

customSpeaker.as<AudioFileRecorder>()->Start("audio.wav");

Switch back to the speaker from a custom audio sink

To switch back to the speaker from a custom audio sink, obtain the instance of the speaker you want to use and call ChangeSpeaker() of PlanetKit::AudioManager with this instance as an argument.

auto audioManager = PlanetKit::PlanetKitManager::GetInstance()->GetAudioManager();

// Get system default speaker
auto defaultSpeaker =  audioManager->GetDefaultSpeakerInfo();

// Change speaker
audioManager->ChangeSpeaker(defaultSpeaker);

Key considerations

The key considerations for using the custom audio device feature are as follows.

Audio format

Category	16-bit PCM	32-bit IEEE
Sampling rate	48000 Hz	48000 Hz
Channel	1 (Mono)	1 (Mono)
Sample format	PCM	IEEE Float

Performance optimization

Memory allocation and deallocation should be minimized for performance.
Audio processing must be performed on separate threads.

AEC settings when using custom audio devices

Acoustic echo cancellation (AEC) is a technology that removes echo caused when sound from the speaker is picked up by the microphone. PlanetKit provides AEC-related features such as AEC in VQE control and application AEC reference. Generally, no separate settings are required for AEC, but when using custom audio devices, you may need to configure AEC settings depending on the situation.

If your application implementation meets the following conditions, you need to configure AEC-related settings.

Case 1: When VQE control's AEC settings are required
- Conditions
  - If any of the following conditions apply:
    - Using an input other than actual microphone devices (e.g., wav files, HTTP URLs) in a custom audio source
    - Using an output other than actual speaker devices (e.g., file recording) in a custom audio sink
- Required settings
  - Call SetAcousticEchoCanceller(PLNK_ACOUSTIC_ECHO_CANCELLER_DISABLED) after starting to use custom audio devices
  - Call SetAcousticEchoCanceller(PLNK_ACOUSTIC_ECHO_CANCELLER_INTENSITY_RECOMMENDED) after ending the use of custom audio devices
Case 2: When application AEC reference settings are required
- Conditions
  - When outputting speaker data received from PlanetKit after processing it in a custom audio sink
    - However, this only applies when the audio input is an actual microphone device (not applicable if the input is not a device where actual echo can be input, such as a wav file)
- Required settings
  - Call StartUserAcousticEchoCancellerReference() after starting to use a custom audio sink
  - Provide the audio data you want to output to the speaker with PutUserAcousticEchoCancellerReference()
  - Call StopUserAcousticEchoCancellerReference() after ending the use of a custom audio sink

Case 1: When VQE control's AEC settings are required

If you are using an input or output other than the actual microphone and speaker with a custom audio source or custom audio sink, you can improve call quality by disabling the use of AEC, one of the VQE features provided by call session classes PlanetKit::PlanetKitCall and PlanetKit::PlanetKitConference. You can obtain a SendVoiceProcessor class instance using GetSendVoiceProcessor() from the call session class and call SetAcousticEchoCanceller() to configure the AEC mode.

Implement a custom audio source class

Implement a class to use an audio file as input using a custom audio source.

#include <thread>
#include <atomic>
#include <vector>

#include "PlanetKitCustomMic.h"
#include "PlanetKitManager.h"

class AudioFileMicrophone : public PlanetKit::CustomMic {
public:
  AudioFileMicrophone() {
    // Allocate buffer for 48kHz 20ms = 960 samples
    m_audioBuffer.resize(960);
  }

  virtual ~AudioFileMicrophone() = default;

  virtual bool SetVolumeLevel(float fVolume) override { return true; };
  virtual float GetVolumeLevel() override { return 1.0f; };
  virtual float GetPeakValue() override { return 1.0f; };
  virtual bool IsRunning() override { return m_isRunning; }

  virtual bool RegisterVolumeLevelChangedEvent(PlanetKit::AudioVolumeLevelChangedEventPtr) override { return true; }
  virtual bool DeregisterVolumeLevelChangedEvent(PlanetKit::AudioVolumeLevelChangedEventPtr) override { return true; }

  virtual PlanetKit::AudioDeviceInfoPtr GetDeviceInfo() override { return nullptr; }

  bool Start() {
    if (m_isRunning.load()) {
      return true; // Already running
    }

    m_isRunning = true;
    m_audioThread = std::thread(&CustomMic::AudioThreadFunc, this);

    return true;
  }

  bool Stop() {
    if (!m_isRunning.load()) {
      return true; // Already stopped
    }

    m_isRunning = false;

    if (m_audioThread.joinable()) {
      m_audioThread.join();
    }

    return true;
  }

private:
  void AudioThreadFunc() {
    while (m_isRunning.load()) {
      // Prepare audio data structure
      PlanetKit::SAudioData audioData;
      ZeroMemory(&audioData, sizeof(PlanetKit::SAudioData));

      audioData.unAudioDataSamplingRate = 48000; // 48kHz

      // Calculate samples needed for the requested duration
      unsigned int samplesFor20ms = audioData.unAudioDataSamplingRate * 20 / 1000;

      audioData.unAudioDataSampleCount = samplesFor20ms;
      audioData.eAudioDataSampleFormat = PlanetKit::EAudioDataSampleType::PLNK_AUDIO_DATA_SAMPLE_TYPE_FLOAT_32;
      audioData.unBufferSize = (unsigned int)(samplesFor20ms * sizeof(float));
      audioData.ucBuffer = reinterpret_cast<unsigned char*>(m_audioBuffer.data());

      // Read audio data from file using audioData.ucBuffer
      ...

      // Send audio data to PlanetKit
      PutAudioData(audioData);

      // Sleep for 20ms (50fps)
      std::this_thread::sleep_for(std::chrono::milliseconds(20));
    }
  }

private:
  std::thread m_audioThread;
  std::atomic<bool> m_isRunning;
  std::vector<float> m_audioBuffer;
};

Use a custom audio source

To use a custom audio source, call ChangeMic() of PlanetKit::AudioManager with the implemented custom audio source instance as an argument, and call SetAcousticEchoCanceller() of the SendVoiceProcessor class with EPlanetKitAcousticEchoCanceller::PLNK_ACOUSTIC_ECHO_CANCELLER_DISABLED as an argument to disable AEC.

// Prepare your call session instance
PlanetKit::PlanetKitCallPtr call;

// Create CustomMic instance by using AudioFileMicrophone class
PlanetKit::CustomMicPtr customMic = PlanetKit::MakeAutoPtr<AudioFileMicrophone>();

// Set CustomMic instance to PlanetKit
auto audioManager = PlanetKit::PlanetKitManager::GetInstance()->GetAudioManager();
audioManager->ChangeMic(customMic);

// Get instance of SendVoiceProcessor class
auto sendVoiceProcessor = call->GetSendVoiceProcessor();

// Set AEC to disabled mode
sendVoiceProcessor->SetAcousticEchoCanceller(PlanetKit::EPlanetKitAcousticEchoCanceller::PLNK_ACOUSTIC_ECHO_CANCELLER_DISABLED);

Switch back to the microphone from a custom audio source

To switch back to the microphone from a custom audio source, obtain the instance of the microphone you want to use, call ChangeMic() of PlanetKit::AudioManager with this instance as an argument, and call SetAcousticEchoCanceller() of the SendVoiceProcessor class with EPlanetKitAcousticEchoCanceller::PLNK_ACOUSTIC_ECHO_CANCELLER_INTENSITY_RECOMMENDED as an argument to enable AEC.

// Prepare your call session instance
PlanetKit::PlanetKitCallPtr call;

auto audioManager = PlanetKit::PlanetKitManager::GetInstance()->GetAudioManager();

// Get system default mic
auto defaultMic = audioManager->GetDefaultMicInfo();

// Change mic
audioManager->ChangeMic(defaultMic);

// Get instance of SendVoiceProcessor class
auto sendVoiceProcessor = call->GetSendVoiceProcessor();

// Set AEC to recommended mode
sendVoiceProcessor->SetAcousticEchoCanceller(PlanetKit::EPlanetKitAcousticEchoCanceller::PLNK_ACOUSTIC_ECHO_CANCELLER_INTENSITY_RECOMMENDED);

Case 2: When application AEC reference settings are required

You can configure whether to use AEC reference data using StartUserAcousticEchoCancellerReference() and StopUserAcousticEchoCancellerReference() provided by call session classes PlanetKit::PlanetKitCall and PlanetKit::PlanetKitConference. After enabling AEC reference data usage, you can provide modified audio data using PutUserAcousticEchoCancellerReference() of the call session class.

Implement a custom audio sink class

Implement a class to modify audio output data using a custom audio sink.

After the call is connected, call PullAudioData() periodically to obtain the received audio data.
Process the audio data and call PutUserAcousticEchoCancellerReference() with the processed data as an argument.

#include "PlanetKitCustomSpeaker.h"
#include "PlanetKitManager.h"

#include <atomic>
#include <thread>

class  AudioSinkModifier : public PlanetKit::CustomSpeaker {
public:
  AudioSinkModifier(PlanetKit::PlanetKitCallPtr call) {
    m_call = call;

    audioData.unAudioDataSamplingRate = 48000; // 48kHz default
    audioData.unAudioDataSampleCount = 480; // 10ms at 48kHz
    audioData.eAudioDataSampleFormat = PlanetKit::PLNK_AUDIO_DATA_SAMPLE_TYPE_SHORT16;
    audioData.buffer = new unsigned char[audioData.unAudioDataSampleCount * sizeof(unsigned char)];
    audioData.unBufferSize = audioData.unAudioDataSampleCount * sizeof(unsigned char);
  }

  virtual ~AudioSinkModifier() = default;

  virtual bool IsRunning() override { return true; }
  virtual bool SetVolumeLevel(float fVolume) override { return true; }
  virtual float GetVolumeLevel() override { return 1.0f; }
  virtual float GetPeakValue() override { return 1.0f; }
  virtual bool RegisterVolumeLevelChangedEvent(PlanetKit::AudioVolumeLevelChangedEventPtr) { return true; }
  virtual bool DeregisterVolumeLevelChangedEvent(PlanetKit::AudioVolumeLevelChangedEventPtr) { return true; }
  virtual PlanetKit::AudioDeviceInfoPtr GetDeviceInfo() override { return nullptr; }
  virtual bool PlayFile(const PlanetKit::WString& strFilePath, unsigned int unLoop) { return true; }
  virtual bool StopPlay() override { return true; }
  virtual bool IsCustomSpeaker() override { return true; }

  void Start() {
    if (m_isRecording.load()) {
      return false;
    }

    m_hAudioThread = std::thread(std::bind(& AudioSinkModifier::PullAudioThread, this));

    return true;
  }

  void Stop() {
    if (!m_isRecording.load()) {
      return;
    }

    m_isRecording.store(false);

    if (m_hAudioThread.joinable()) {
      m_hAudioThread.join();
    }
  }

private:
  void PullAudioThread() {
    m_isRecording.store(true);
    while(m_isRunning) {
      // 100Hz update rate (10ms intervals) - matches sample buffer size
      Sleep(10);

      if(this->PullAudioData(m_audioData) == true) {
        // Modify audioData and Play to physical speaker
        PlanetKit::SAudioData modifyAudioData;

        m_call->PutUserAcousticEchoCancellerReference(modifyAudioData);
      }
    }
  }

private:
  PlanetKit::PlanetKitCallPtr m_call;
  std::atomic<bool> m_isRecording{false};
  std::thread m_hAudioThread;

  PlanetKit::SAudioData m_audioData;
}

Use a custom audio sink

To use a custom audio sink, call ChangeSpeaker() of PlanetKit::AudioManager with the implemented custom audio sink instance as an argument, and call StartUserAcousticEchoCancellerReference() of the call session class to start using the application AEC reference.

// Prepare your call session instance
PlanetKit::PlanetKitCallPtr call;

// Create CustomSpeaker instance by using AudioSinkModifier class
PlanetKit::CustomSpeakerPtr customSpeaker = PlanetKit::MakeAutoPtr<AudioSinkModifier>(call);

// Set CustomSpeaker instance to PlanetKit
auto audioManager = PlanetKit::PlanetKitManager::GetInstance()->GetAudioManager();
audioManager->ChangeSpeaker(customSpeaker);

call->StartUserAcousticEchoCancellerReference();

Switch back to the speaker from a custom audio sink

To switch back to the speaker from a custom audio sink, obtain the instance of the speaker you want to use, call ChangeSpeaker() of PlanetKit::AudioManager with this instance as an argument, and call StopUserAcousticEchoCancellerReference() of the call session class to stop using the application AEC reference.

// Prepare your call session instance
PlanetKit::PlanetKitCallPtr call;

auto audioManager = PlanetKit::PlanetKitManager::GetInstance()->GetAudioManager();

// Get system default speaker
auto defaultSpeaker = audioManager->GetDefaultSpeakerInfo();

// Change speaker
audioManager->ChangeSpeaker(defaultSpeaker);

// Stop AEC reference
call->StopUserAcousticEchoCancellerReference();

Use cases​

Custom audio source​

Custom audio sink​

API overview​

Implementation steps​

Example 1: Custom audio source - Playing an audio file​

Implement a custom audio source class​

Use a custom audio source​

Switch back to the microphone from a custom audio source​

Example 2: Custom audio sink - Audio recording​

Implement a custom audio sink class​

Use a custom audio sink​

Switch back to the speaker from a custom audio sink​

Key considerations​

Audio format​

Performance optimization​

AEC settings when using custom audio devices​

Conditions requiring AEC-related settings​

Case 1: When VQE control's AEC settings are required​

Implement a custom audio source class​

Use a custom audio source​

Switch back to the microphone from a custom audio source​

Case 2: When application AEC reference settings are required​

Implement a custom audio sink class​

Use a custom audio sink​

Switch back to the speaker from a custom audio sink​

Related APIs​

Use cases

Custom audio source

Custom audio sink

API overview

Implementation steps

Example 1: Custom audio source - Playing an audio file

Implement a custom audio source class

Use a custom audio source

Switch back to the microphone from a custom audio source

Example 2: Custom audio sink - Audio recording

Implement a custom audio sink class

Use a custom audio sink

Switch back to the speaker from a custom audio sink

Key considerations

Audio format

Performance optimization

AEC settings when using custom audio devices

Conditions requiring AEC-related settings

Case 1: When VQE control's AEC settings are required

Implement a custom audio source class

Use a custom audio source

Switch back to the microphone from a custom audio source

Case 2: When application AEC reference settings are required

Implement a custom audio sink class

Use a custom audio sink

Switch back to the speaker from a custom audio sink

Related APIs