Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix #3790: Add automatic text-to-speech audio to explorations #3818

Merged
merged 11 commits into from
Sep 18, 2017
Merged

Fix #3790: Add automatic text-to-speech audio to explorations #3818

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
75abb79
Add automatics text to speech
tjiang11 Aug 28, 2017
819688b
lint
tjiang11 Aug 28, 2017
48adf76
Minor refactoring in AudioControlsDirective.
tjiang11 Aug 28, 2017
c733e34
Suggested changes
tjiang11 Sep 2, 2017
c358f5e
Making Links and LateX speakable
tjiang11 Sep 16, 2017
a6bc664
lint
tjiang11 Sep 16, 2017
2725311
Add language object factories; update LaTeX formatting for speakable …
tjiang11 Sep 17, 2017
797525b
lint
tjiang11 Sep 17, 2017
6d24a10
lint
tjiang11 Sep 17, 2017
5c60c3e
merge develop
tjiang11 Sep 17, 2017
dc2ed3f
Change variable name; add dependencies to collection editor
tjiang11 Sep 17, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions assets/constants.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -271,11 +271,20 @@ var constants = {
"SUPPORTED_AUDIO_LANGUAGES": [{
"id": "en",
"text": "English",
"related_languages": ["en"]
"related_languages": ["en"],
"speech_synthesis_code": "en-GB"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure the modelling here is right. I don't think we should have dicts with different keys inside a list that should be homogeneous.

Could we instead have two separate top-level constants: SUPPORTED_AUDIO_LANGUAGES and AUTOGENERATED_AUDIO_LANGUAGES, perhaps? The two things are handled rather differently anyway in e.g. the editor view, so I think it might make sense to separate them. Though you may need to unify both when generating the dropdown choices for the audio preferences.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

}, {
"id": "hi-en",
"text": "Hinglish",
"related_languages": ["hi", "en"]
}]
}],

"//" : [
"This suffix is attached to the end of language codes with ",
"autogenerated audio. For example, en-auto is the langauge code for ",
"autogenerated English audio."
],
"AUTOGENERATED_AUDIO_LANGUAGE_CODE_SUFFIX": "-auto",
"AUTOGENERATED_AUDIO_LANGUAGE_DESCRIPTION": " (autogenerated)"

};
48 changes: 48 additions & 0 deletions core/templates/dev/head/domain/utilities/BrowserCheckerService.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
// Copyright 2017 The Oppia Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS-IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

/**
* @fileoverview Utility service for checking web browser type.
*/

oppia.factory('BrowserCheckerService', function() {
var isChromium = window.chrome;
var winNav = window.navigator;
var vendorName = winNav.vendor;
var isOpera = winNav.userAgent.indexOf('OPR') > -1;
var isIEedge = winNav.userAgent.indexOf('Edge') > -1;
var isIOSChrome = winNav.userAgent.match('CriOS');

var _isChrome = function() {
// For details on the reliabiltiy of this check, see
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"reliability" is misspelled

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

// https://stackoverflow.com/questions/4565112/
// javascript-how-to-find-out-if-the-user-browser-is-chrome
if (isIOSChrome ||
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe use an additional paren since I don't think people tend to remember whether || or && takes precedence.

Incidentally, audio doesn't work on Chromium on Ubuntu, at least on my machine. Maybe related to https://askubuntu.com/questions/761975/chromium-is-not-generating-voice

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

isChromium !== null &&
isChromium !== undefined &&
vendorName === 'Google Inc.' &&
isOpera == false &&
isIEedge == false) {
return true;
} else {
return false;
}
};

return {
isChrome: function() {
return _isChrome();
}
};
});
24 changes: 24 additions & 0 deletions core/templates/dev/head/domain/utilities/LanguageUtilService.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,16 @@ oppia.factory('LanguageUtilService', [function() {
audioLanguage.related_languages;
});

var generatedAudioLanguageCodesToSpeechSynthesisLanguageCode = {};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again should we be storing human-generated audio + autogenerated audio separately (and only unifying where needed)? Feels like it'd be less implicit; you're currently relying on adding custom strings and stuff to the IDs which feels a bit less maintainable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

supportedAudioLanguages.forEach(function(audioLanguage) {
if (audioLanguage.hasOwnProperty('speech_synthesis_code')) {
generatedAudioLanguageCodesToSpeechSynthesisLanguageCode[
audioLanguage.id +
constants.AUTOGENERATED_AUDIO_LANGUAGE_CODE_SUFFIX] =
audioLanguage.speech_synthesis_code;
}
});

return {
getAudioLanguagesCount: function() {
return audioLanguagesCount;
Expand All @@ -52,6 +62,20 @@ oppia.factory('LanguageUtilService', [function() {
},
getLanguageCodesRelatedToAudioLanguageCode: function(audioLanguageCode) {
return audioLanguageCodesToRelatedLanguageCodes[audioLanguageCode];
},
getSpeechSynthesisLanguageCode: function(audioLanguageCode) {
return generatedAudioLanguageCodesToSpeechSynthesisLanguageCode[
audioLanguageCode];
},
supportsAutogeneratedAudio: function(audioLanguageCode) {
return generatedAudioLanguageCodesToSpeechSynthesisLanguageCode
.hasOwnProperty(
audioLanguageCode +
constants.AUTOGENERATED_AUDIO_LANGUAGE_CODE_SUFFIX);
},
isAutogeneratedAudioLanguage: function(audioLanguageCode) {
return audioLanguageCode.endsWith(
constants.AUTOGENERATED_AUDIO_LANGUAGE_CODE_SUFFIX);
}
}
}]);
105 changes: 81 additions & 24 deletions core/templates/dev/head/pages/exploration_player/AudioControlsDirective.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,17 +23,20 @@ oppia.directive('audioControls', [
return {
restrict: 'E',
scope: {
getAudioTranslations: '&audioTranslations'
getAudioTranslations: '&audioTranslations',
getContentHtml: '&contentHtml'
},
templateUrl: UrlInterpolationService.getDirectiveTemplateUrl(
'/pages/exploration_player/' +
'audio_controls_directive.html'),
controller: [
'$scope', 'AudioTranslationManagerService', 'AudioPlayerService',
'LanguageUtilService', 'AssetsBackendApiService',
'AutogeneratedAudioPlayerService',
function(
$scope, AudioTranslationManagerService, AudioPlayerService,
LanguageUtilService, AssetsBackendApiService) {
LanguageUtilService, AssetsBackendApiService,
AutogeneratedAudioPlayerService) {
// This ID is passed in to AudioPlayerService as a means of
// distinguishing which audio directive is currently playing audio.
var directiveId = Math.random().toString(36).substr(2, 10);
Expand All @@ -53,39 +56,61 @@ oppia.directive('audioControls', [
AudioTranslationManagerService.getCurrentAudioLanguageCode()];
};

$scope.AudioPlayerService = AudioPlayerService;
$scope.showSpeakerPlayingIcon = function() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not replace these two methods with $scope.isAudioPlaying()?

return AudioPlayerService.isPlaying() ||
AutogeneratedAudioPlayerService.isPlaying();
};

$scope.showSpeakerSilentIcon = function() {
return !(AudioPlayerService.isPlaying() ||
AutogeneratedAudioPlayerService.isPlaying());
};

$scope.IMAGE_URL_REWIND_AUDIO_BUTTON = (
UrlInterpolationService.getStaticImageUrl(
'/icons/rewind-five.svg'));

$scope.isAudioAvailableInCurrentLanguage = function() {
return Boolean(getAudioTranslationInCurrentLanguage());
return Boolean(getAudioTranslationInCurrentLanguage()) ||
AudioTranslationManagerService
.isAutogeneratedLanguageCodeSelected();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, might want a helper function locally here for checking isAutogeneratedLanguageCodeSelected(). The length of this whole thing seems to affect the readability of not only this line, but below too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

};

$scope.doesCurrentAudioTranslationNeedUpdate = function() {
return getAudioTranslationInCurrentLanguage().needsUpdate;
if (!AudioTranslationManagerService
.isAutogeneratedLanguageCodeSelected()) {
return getAudioTranslationInCurrentLanguage().needsUpdate;
} else {
return false;
}
};

$scope.onSpeakerIconClicked = function() {
var audioTranslation = getAudioTranslationInCurrentLanguage();
if (audioTranslation) {
// If this language hasn't been preloaded for the exploration,
// and this audio translation hasn't been loaded, then ask to
// preload all audio translations for the current language.
if (!AudioPreloaderService.hasPreloadedLanguage(
getCurrentAudioLanguageCode()) &&
!isCached(audioTranslation)) {
AudioPreloaderService.showBandwidthConfirmationModal(
$scope.getAudioTranslations(), getCurrentAudioLanguageCode(),
playPauseAudioTranslation);
if (AudioTranslationManagerService
.isAutogeneratedLanguageCodeSelected()) {
playPauseAutogeneratedAudioTranslation();
} else {
var audioTranslation = getAudioTranslationInCurrentLanguage();
if (audioTranslation) {
// If this language hasn't been preloaded for the exploration,
// and this audio translation hasn't been loaded, then ask to
// preload all audio translations for the current language.
if (!AudioPreloaderService.hasPreloadedLanguage(
getCurrentAudioLanguageCode()) &&
!isCached(audioTranslation)) {
AudioPreloaderService.showBandwidthConfirmationModal(
$scope.getAudioTranslations(),
getCurrentAudioLanguageCode(),
playPauseAudioTranslation);
} else {
playPauseUploadedAudioTranslation(
getCurrentAudioLanguageCode());
}
} else {
playPauseAudioTranslation(getCurrentAudioLanguageCode());
// If the audio translation isn't available in the current
// language, then open the settings modal.
$scope.openAudioTranslationSettings();
}
} else {
// If the audio translation isn't available in the current
// language, then open the settings modal.
$scope.openAudioTranslationSettings();
}
};

Expand All @@ -94,7 +119,39 @@ oppia.directive('audioControls', [
};

var playPauseAudioTranslation = function(languageCode) {
$scope.extraAudioControlsAreShown = true;
if (AudioTranslationManagerService
.isAutogeneratedLanguageCodeSelected()) {
playPauseAutogeneratedAudioTranslation();
} else {
playPauseUploadedAudioTranslation(languageCode);
}
};

var playPauseAutogeneratedAudioTranslation = function() {
$scope.audioSettingsButtonIsShown = true;
$scope.rewindButtonIsShown = false;
// SpeechSynthesis in Chrome seems to have a bug
// where if you pause the utterance, wait for around
// 15 or more seconds, then try resuming, nothing
// will sound. As a temporary fix, just restart the
// utterance from the beginning instead of resuming.
if (AutogeneratedAudioPlayerService.isPlaying()) {
AutogeneratedAudioPlayerService.cancel();
} else {
AutogeneratedAudioPlayerService.play(
$scope.getContentHtml(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mm -- I think you need to send this through a helper service that transforms html to text. E.g. for links, the link text should be read, rather than the link being skipped altogether.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it seems a bit easier to just let the speech chunker service take the html and handle it?

AudioTranslationManagerService.getSpeechSynthesisLanguageCode(),
function() {
// Used to update bindings to show a silent speaker after
// autogenerated audio has finished playing.
$scope.$apply();
});
}
};

var playPauseUploadedAudioTranslation = function(languageCode) {
$scope.audioSettingsButtonIsShown = true;
$scope.rewindButtonIsShown = true;

if (!AudioPlayerService.isPlaying()) {
if (AudioPlayerService.isTrackLoaded() &&
Expand All @@ -110,12 +167,12 @@ oppia.directive('audioControls', [
// immediately start playing the newly requested audio.
loadAndPlayAudioTranslation();
}
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trailing spaces, here and below. If you're using sublime, you can add "trim_trailing_white_space_on_save": true, to your user prefs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, thanks for the tip :)

};

var isRequestForSameAudioAsLastTime = function() {
return directiveId ===
AudioPlayerService.getCurrentAudioControlsDirectiveId();
AudioPlayerService.getCurrentAudioControlsDirectiveId();
};

var loadAndPlayAudioTranslation = function() {
Expand Down
23 changes: 14 additions & 9 deletions core/templates/dev/head/pages/exploration_player/AudioPreloaderService.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@
oppia.factory('AudioPreloaderService', [
'$modal', 'explorationContextService', 'AssetsBackendApiService',
'ExplorationPlayerStateService', 'UrlInterpolationService',
'AudioTranslationManagerService',
'AudioTranslationManagerService', 'LanguageUtilService',
function($modal, explorationContextService, AssetsBackendApiService,
ExplorationPlayerStateService, UrlInterpolationService,
AudioTranslationManagerService) {
AudioTranslationManagerService, LanguageUtilService) {
// List of languages that have been preloaded in the exploration.
var _preloadedLanguageCodes = [];

Expand Down Expand Up @@ -95,13 +95,18 @@ oppia.factory('AudioPreloaderService', [
// modal if the file for the new language hasn't been loaded.
AudioTranslationManagerService
.showAudioTranslationSettingsModal(function(newLanguageCode) {
var newAudioTranslation =
audioTranslationsForContent[newLanguageCode];
if (newAudioTranslation && !AssetsBackendApiService.isCached(
newAudioTranslation.filename)) {
_showBandwidthConfirmationModal(
audioTranslationsForContent, newLanguageCode,
confirmationCallback)
if (LanguageUtilService.isAutogeneratedAudioLanguage(
newLanguageCode)) {
confirmationCallback(newLanguageCode);
} else {
var newAudioTranslation =
audioTranslationsForContent[newLanguageCode];
if (newAudioTranslation && !AssetsBackendApiService.isCached(
newAudioTranslation.filename)) {
_showBandwidthConfirmationModal(
audioTranslationsForContent, newLanguageCode,
confirmationCallback)
}
}
});
} else {
Expand Down
43 changes: 39 additions & 4 deletions core/templates/dev/head/pages/exploration_player/AudioTranslationManagerService.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,15 @@
*/

oppia.factory('AudioTranslationManagerService', [
'$modal', 'AudioPlayerService', 'UrlInterpolationService',
'LanguageUtilService',
'$modal', 'AudioPlayerService', 'AutogeneratedAudioPlayerService',
'UrlInterpolationService', 'LanguageUtilService', 'BrowserCheckerService',
function(
$modal, AudioPlayerService, UrlInterpolationService,
LanguageUtilService) {
$modal, AudioPlayerService, AutogeneratedAudioPlayerService,
UrlInterpolationService, LanguageUtilService, BrowserCheckerService) {
var _currentAudioLanguageCode = null;
var _allAudioLanguageCodesInExploration = null;
var _explorationLanguageCode = null;
var _isAutogeneratedLanguageCodeSelected = false;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_autogeneratedLanguageCodeIsSelected

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


var attemptToSetAudioLanguageToExplorationLanguage = function() {
// We minimize the number of related languages, because we want to
Expand Down Expand Up @@ -65,6 +66,15 @@ oppia.factory('AudioTranslationManagerService', [
_allAudioLanguageCodesInExploration.length >= 1) {
_currentAudioLanguageCode = _allAudioLanguageCodesInExploration[0];
}

if (_currentAudioLanguageCode === null &&
_allAudioLanguageCodesInExploration.length == 0) {
_currentAudioLanguageCode =
_explorationLanguageCode +
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not a given that the exp language code has an associated autogeneratable translation, is it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, fixed.

constants.AUTOGENERATED_AUDIO_LANGUAGE_CODE_SUFFIX;

_isAutogeneratedLanguageCodeSelected = true;
}
};

var _showAudioTranslationSettingsModal = function(
Expand Down Expand Up @@ -96,6 +106,20 @@ oppia.factory('AudioTranslationManagerService', [
});
});

if (LanguageUtilService.supportsAutogeneratedAudio(
_explorationLanguageCode) &&
BrowserCheckerService.isChrome()) {
var languageDescription =
LanguageUtilService.getAudioLanguageDescription(
_explorationLanguageCode);
$scope.languagesInExploration.push({
value: _explorationLanguageCode +
constants.AUTOGENERATED_AUDIO_LANGUAGE_CODE_SUFFIX,
displayed: languageDescription +
constants.AUTOGENERATED_AUDIO_LANGUAGE_DESCRIPTION
});
}

$scope.selectedLanguage = _currentAudioLanguageCode;
$scope.save = function() {
$modalInstance.close({
Expand All @@ -107,8 +131,12 @@ oppia.factory('AudioTranslationManagerService', [
}).result.then(function(result) {
if (_currentAudioLanguageCode !== result.languageCode) {
_currentAudioLanguageCode = result.languageCode;
_isAutogeneratedLanguageCodeSelected =
LanguageUtilService.isAutogeneratedAudioLanguage(
_currentAudioLanguageCode);
AudioPlayerService.stop();
AudioPlayerService.clear();
AutogeneratedAudioPlayerService.cancel();
if (onLanguageChangedCallback) {
onLanguageChangedCallback(_currentAudioLanguageCode);
}
Expand Down Expand Up @@ -138,6 +166,13 @@ oppia.factory('AudioTranslationManagerService', [
},
clearCurrentAudioLanguageCode: function() {
_currentAudioLanguageCode = null;
},
isAutogeneratedLanguageCodeSelected: function() {
return _isAutogeneratedLanguageCodeSelected;
},
getSpeechSynthesisLanguageCode: function() {
return LanguageUtilService.getSpeechSynthesisLanguageCode(
_currentAudioLanguageCode);
}
};
}]);
Loading