Skip to content

Commit

Permalink
The audio processing module (APM) relies on two for
Browse files Browse the repository at this point in the history
functionalities  doing sample-rate conversions:
-The implicit resampling done in the AudioBuffer CopyTo,
 CopyFrom, InterleaveTo and DeinterleaveFrom methods.
-The multi-band splitting scheme.

The selection of rates in these have been difficult and
complicated, partly due to that the APM API which allows
for activating the APM submodules without notifying
the APM.

This CL adds functionality that for each capture frame
polls all submodules for whether they are active or not
and compares this against a cached result.
Furthermore, new functionality is added that based on the
results of the comparison do a reinitialization of the APM.

This has several advantages
-The code deciding on whether to analysis and synthesis is
 needed for the bandsplitting can be much simplified and
 centralized.
-The selection of the processing rate can be done such as
 to avoid the implicit resampling that was in some cases
 unnecessarily done.
-The optimization for whether an output copy is needed
 that was done to improve performance due to the implicit
 resampling is no longer needed, which simplifies the
 code and makes it less error-prone in the sense that
 is no longer neccessary to keep track of whether any
 module has changed the signal.

Finally, it should be noted that the polling of the state
for all the submodules was done previously as well, but in
a less obvious and distributed manner.

BUG=webrtc:6181, webrtc:6220, webrtc:5298, webrtc:6296, webrtc:6298, webrtc:6297

Review-Url: https://codereview.webrtc.org/2304123002
Cr-Commit-Position: refs/heads/master@{#14175}
  • Loading branch information
peah authored and Commit bot committed Sep 10, 2016
1 parent 1d02d3e commit 2ace3f9
Show file tree
Hide file tree
Showing 12 changed files with 210 additions and 176 deletions.
Binary file modified data/audio_processing/output_data_fixed.pb
Binary file not shown.
Binary file modified data/audio_processing/output_data_float.pb
Binary file not shown.
Binary file modified data/audio_processing/output_data_mac.pb
Binary file not shown.
289 changes: 163 additions & 126 deletions webrtc/modules/audio_processing/audio_processing_impl.cc

Large diffs are not rendered by default.

59 changes: 44 additions & 15 deletions webrtc/modules/audio_processing/audio_processing_impl.h
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,41 @@ class AudioProcessingImpl : public AudioProcessing {
struct ApmPublicSubmodules;
struct ApmPrivateSubmodules;

class ApmSubmoduleStates {
public:
ApmSubmoduleStates();
// Updates the submodule state and returns true if it has changed.
bool Update(bool high_pass_filter_enabled,
bool echo_canceller_enabled,
bool mobile_echo_controller_enabled,
bool noise_suppressor_enabled,
bool intelligibility_enhancer_enabled,
bool beamformer_enabled,
bool adaptive_gain_controller_enabled,
bool level_controller_enabled,
bool voice_activity_detector_enabled,
bool level_estimator_enabled,
bool transient_suppressor_enabled);
bool CaptureMultiBandSubModulesActive() const;
bool CaptureMultiBandProcessingActive() const;
bool RenderMultiBandSubModulesActive() const;
bool RenderMultiBandProcessingActive() const;

private:
bool high_pass_filter_enabled_ = false;
bool echo_canceller_enabled_ = false;
bool mobile_echo_controller_enabled_ = false;
bool noise_suppressor_enabled_ = false;
bool intelligibility_enhancer_enabled_ = false;
bool beamformer_enabled_ = false;
bool adaptive_gain_controller_enabled_ = false;
bool level_controller_enabled_ = false;
bool level_estimator_enabled_ = false;
bool voice_activity_detector_enabled_ = false;
bool transient_suppressor_enabled_ = false;
bool first_update_ = true;
};

#ifdef WEBRTC_AUDIOPROC_DEBUG_DUMP
// State for the debug dump.
struct ApmDebugDumpThreadState {
Expand Down Expand Up @@ -162,21 +197,19 @@ class AudioProcessingImpl : public AudioProcessing {
// that the capture thread blocks the render thread.
// The struct is modified in a single-threaded manner by holding both the
// render and capture locks.
int MaybeInitialize(const ProcessingConfig& config)
int MaybeInitialize(const ProcessingConfig& config, bool force_initialization)
EXCLUSIVE_LOCKS_REQUIRED(crit_render_);

int MaybeInitializeRender(const ProcessingConfig& processing_config)
EXCLUSIVE_LOCKS_REQUIRED(crit_render_);

int MaybeInitializeCapture(const ProcessingConfig& processing_config)
int MaybeInitializeCapture(const ProcessingConfig& processing_config,
bool force_initialization)
EXCLUSIVE_LOCKS_REQUIRED(crit_render_);

// Method for checking for the need of conversion. Accesses the formats
// structs in a read manner but the requirement for the render lock to be held
// was added as it currently anyway is always called in that manner.
bool rev_conversion_needed() const EXCLUSIVE_LOCKS_REQUIRED(crit_render_);
bool render_check_rev_conversion_needed() const
EXCLUSIVE_LOCKS_REQUIRED(crit_render_);
// Method for updating the state keeping track of the active submodules.
// Returns a bool indicating whether the state has changed.
bool UpdateActiveSubmoduleStates() EXCLUSIVE_LOCKS_REQUIRED(crit_capture_);

// Methods requiring APM running in a single-threaded manner.
// Are called with both the render and capture locks already
Expand Down Expand Up @@ -210,10 +243,6 @@ class AudioProcessingImpl : public AudioProcessing {
// Capture-side exclusive methods possibly running APM in a multi-threaded
// manner that are called with the render lock already acquired.
int ProcessStreamLocked() EXCLUSIVE_LOCKS_REQUIRED(crit_capture_);
bool output_copy_needed() const EXCLUSIVE_LOCKS_REQUIRED(crit_capture_);
bool is_fwd_processed() const EXCLUSIVE_LOCKS_REQUIRED(crit_capture_);
bool fwd_synthesis_needed() const EXCLUSIVE_LOCKS_REQUIRED(crit_capture_);
bool fwd_analysis_needed() const EXCLUSIVE_LOCKS_REQUIRED(crit_capture_);
void MaybeUpdateHistograms() EXCLUSIVE_LOCKS_REQUIRED(crit_capture_);

// Render-side exclusive methods possibly running APM in a multi-threaded
Expand All @@ -223,9 +252,6 @@ class AudioProcessingImpl : public AudioProcessing {
const StreamConfig& input_config,
const StreamConfig& output_config)
EXCLUSIVE_LOCKS_REQUIRED(crit_render_);
bool is_rev_processed() const EXCLUSIVE_LOCKS_REQUIRED(crit_render_);
bool rev_synthesis_needed() const EXCLUSIVE_LOCKS_REQUIRED(crit_render_);
bool rev_analysis_needed() const EXCLUSIVE_LOCKS_REQUIRED(crit_render_);
int ProcessReverseStreamLocked() EXCLUSIVE_LOCKS_REQUIRED(crit_render_);

// Debug dump methods that are internal and called without locks.
Expand Down Expand Up @@ -256,6 +282,9 @@ class AudioProcessingImpl : public AudioProcessing {
rtc::CriticalSection crit_render_ ACQUIRED_BEFORE(crit_capture_);
rtc::CriticalSection crit_capture_;

// Class containing information about what submodules are active.
ApmSubmoduleStates submodule_states_;

// Structs containing the pointers to the submodules.
std::unique_ptr<ApmPublicSubmodules> public_submodules_;
std::unique_ptr<ApmPrivateSubmodules> private_submodules_
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,11 +42,11 @@ TEST(AudioProcessingImplTest, AudioParameterChangeTriggersInit) {
mock.Initialize();

AudioFrame frame;
// Call with the default parameters; there should be no init.
// Call with the default parameters; there should be an init.
frame.num_channels_ = 1;
SetFrameSampleRate(&frame, 16000);
EXPECT_CALL(mock, InitializeLocked())
.Times(0);
.Times(1);
EXPECT_NOERR(mock.ProcessStream(&frame));
EXPECT_NOERR(mock.ProcessReverseStream(&frame));

Expand All @@ -57,6 +57,7 @@ TEST(AudioProcessingImplTest, AudioParameterChangeTriggersInit) {
EXPECT_NOERR(mock.ProcessStream(&frame));

// New number of channels.
// TODO(peah): Investigate why this causes 2 inits.
frame.num_channels_ = 2;
EXPECT_CALL(mock, InitializeLocked())
.Times(2);
Expand Down
6 changes: 0 additions & 6 deletions webrtc/modules/audio_processing/echo_cancellation_impl.cc
Original file line number Diff line number Diff line change
Expand Up @@ -275,12 +275,6 @@ int EchoCancellationImpl::Enable(bool enable) {
return AudioProcessing::kNoError;
}

bool EchoCancellationImpl::is_enabled_render_side_query() const {
// TODO(peah): Add threadchecker.
rtc::CritScope cs_render(crit_render_);
return enabled_;
}

bool EchoCancellationImpl::is_enabled() const {
rtc::CritScope cs(crit_capture_);
return enabled_;
Expand Down
5 changes: 0 additions & 5 deletions webrtc/modules/audio_processing/echo_cancellation_impl.h
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,6 @@ class EchoCancellationImpl : public EchoCancellation {
std::string GetExperimentsDescription();
bool is_refined_adaptive_filter_enabled() const;

// Checks whether the module is enabled. Must only be
// called from the render side of APM as otherwise
// deadlocks may occur.
bool is_enabled_render_side_query() const;

// Reads render side data that has been queued on the render call.
// Called holding the capture lock.
void ReadQueuedRenderData();
Expand Down
6 changes: 0 additions & 6 deletions webrtc/modules/audio_processing/echo_control_mobile_impl.cc
Original file line number Diff line number Diff line change
Expand Up @@ -279,12 +279,6 @@ int EchoControlMobileImpl::Enable(bool enable) {
return AudioProcessing::kNoError;
}

bool EchoControlMobileImpl::is_enabled_render_side_query() const {
// TODO(peah): Add threadchecker.
rtc::CritScope cs_render(crit_render_);
return enabled_;
}

bool EchoControlMobileImpl::is_enabled() const {
rtc::CritScope cs(crit_capture_);
return enabled_;
Expand Down
5 changes: 0 additions & 5 deletions webrtc/modules/audio_processing/echo_control_mobile_impl.h
Original file line number Diff line number Diff line change
Expand Up @@ -43,11 +43,6 @@ class EchoControlMobileImpl : public EchoControlMobile {
size_t num_reverse_channels,
size_t num_output_channels);

// Checks whether the module is enabled. Must only be
// called from the render side of APM as otherwise
// deadlocks may occur.
bool is_enabled_render_side_query() const;

// Reads render side data that has been queued on the render call.
void ReadQueuedRenderData();

Expand Down
6 changes: 0 additions & 6 deletions webrtc/modules/audio_processing/gain_control_impl.cc
Original file line number Diff line number Diff line change
Expand Up @@ -316,12 +316,6 @@ int GainControlImpl::Enable(bool enable) {
return AudioProcessing::kNoError;
}

bool GainControlImpl::is_enabled_render_side_query() const {
// TODO(peah): Add threadchecker.
rtc::CritScope cs(crit_render_);
return enabled_;
}

bool GainControlImpl::is_enabled() const {
rtc::CritScope cs(crit_capture_);
return enabled_;
Expand Down
5 changes: 0 additions & 5 deletions webrtc/modules/audio_processing/gain_control_impl.h
Original file line number Diff line number Diff line change
Expand Up @@ -43,11 +43,6 @@ class GainControlImpl : public GainControl {
bool is_limiter_enabled() const override;
Mode mode() const override;

// Checks whether the module is enabled. Must only be
// called from the render side of APM as otherwise
// deadlocks may occur.
bool is_enabled_render_side_query() const;

// Reads render side data that has been queued on the render call.
void ReadQueuedRenderData();

Expand Down

0 comments on commit 2ace3f9

Please sign in to comment.