BACKGROUND

It has become commonplace to perform automated speech-to-text conversion of captured speech audio. Such a conversion to text may be performed as part of receiving verbal commands used as input for the provision of various voice-controlled online services. Such a conversion to text may be performed as part of indexing and/or memorializing the contents of recorded voice messages or of phone conversations for future retrieval and reference. Such indexing and/or memorializing may be done as part of archiving official records, preserving testimony in judicial proceedings, preserving data gathered in scientific and/or medical field studies, etc.

Alternatively or additionally, such a conversion to text may be used as part of various automated analyses of the contents of conversations or verbal presentations to retrieve various insights. Such analyses may include an evaluation of the quality of service provided in telephone service calls, the efficiency or effectiveness of communication in emergency services calls, the effectiveness of an effort to disseminate information to the public in press interviews or in other verbal presentations, the audience participation and/or reaction to a verbal presentation, the identification of topic(s) of conversations and/or verbal presentations, the relative degrees of focus of each topic among multiple topics, the relative levels of participation among multiple speakers, the type and/or strength of sentiments concerning topics, etc. Such automated retrieval of insights may be performed to enhance the indexing and/or memorializing the contents of captured speech audio.

Regardless of the purpose for performing automated speech-to-text conversion and/or automated analyses, a longstanding challenge has been improving the accuracy of the speech-to-text conversion and/or of the analyses. As will be familiar to those skilled in the art, there are numerous challenges, including and not limited to, quality issues with the devices used to capture speech audio, high environmental noise levels, languages having multiple dialects, differences in regional accents, differences in idiomatic expressions, and/or per-person differences in pronunciation, speed of speaking, speaking volume, speech impediments, etc. Such accuracy issues with speech-to-text conversion result in the provision of error-laden text as the input to text analyses, which in turn, results in the generation of false and misleading insights.

Over time, various significant improvements have been made to acoustic models and language models that are used. However, there remains challenges in this technical field. By way of example, the preprocessing used to divide streamed speech audio and/or lengthy recorded speech audio into segments has seen comparatively little improvement.

SUMMARY

This summary is not intended to identify only key or essential features of the described subject matter, nor is it intended to be used in isolation to determine the scope of the described subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

An apparatus includes at least one processor and a storage to store instructions that, when executed by the at least one processor, cause the at least one processor to perform operations including: receive, from a requesting device via a network, a request to perform speech-to-text conversion of a specified speech data set representing speech audio; and in response to the request, the at least one processor is caused to perform preprocessing operations of a first pause detection technique including analyze the speech audio to identify pauses between speech sounds, and analyze lengths of the pauses to identify a first set of likely sentence pauses. The at least one processor is also caused to, in response to the request, the at least one processor is caused to perform preprocessing operations of a first speaker diarization technique including: divide the speech data set into a set of data fragments that each represent a fragment of a set of fragments of the speech audio; for each data fragment, analyze vocal characteristics of speech sounds of the fragment to identify a speaker of a set of speakers; and identify instances of a change in speakers between temporally consecutive fragments of the set of fragments to identify a first set of likely speaker changes. The at least one processor is further caused to, in response to the request, the at least one processor is caused to perform speech-to-text operations including: based on at least the first set of likely sentence pauses, and based on at least the first set of likely speaker changes, divide the speech data set into multiple data segments that each represent a speech segment of multiple speech segments of the speech audio; use at least an acoustic model with each data segment of the multiple data segments to identify likely speech sounds in the speech audio; and generate a transcript of the speech data set based, at least in part, on the identified likely speech sounds, or transmit an indication of the generation of the transcript to the requesting device.

A computer-program product tangibly embodied in a non-transitory machine-readable storage medium includes instructions operable to cause at least one processor to perform operations including: receive, from a requesting device via a network, a request to perform speech-to-text conversion of a specified speech data set representing speech audio; and in response to the request, the at least one processor is caused to perform preprocessing operations of a first pause detection technique including analyze the speech audio to identify pauses between speech sounds, and analyze lengths of the pauses to identify a first set of likely sentence pauses. The at least one processor is also caused to, in response to the request, the at least one processor is caused to perform preprocessing operations of a first speaker diarization technique including: divide the speech data set into a set of data fragments that each represent a fragment of a set of fragments of the speech audio; for each data fragment, analyze vocal characteristics of speech sounds of the fragment to identify a speaker of a set of speakers; and identify instances of a change in speakers between temporally consecutive fragments of the set of fragments to identify a first set of likely speaker changes. The at least one processor is further caused to, in response to the request, the at least one processor is caused to perform speech-to-text operations including: based on at least the first set of likely sentence pauses, and based on at least the first set of likely speaker changes, divide the speech data set into multiple data segments that each represent a speech segment of multiple speech segments of the speech audio; use at least an acoustic model with each data segment of the multiple data segments to identify likely speech sounds in the speech audio; and generate a transcript of the speech data set based, at least in part, on the identified likely speech sounds, or transmit an indication of the generation of the transcript to the requesting device.

Performing the preprocessing operations of the first speaker diarization technique may include the at least one processor being caused to perform operations including: instantiate a speaker diarization neural network trained to output indications of vocal characteristics; and analyze vocal characteristics of the speech sounds of each data fragment by performing operations including provide each data fragment of the set of data fragments to the speaker diarization neural network as an input to cause the speaker diarization neural network to generate a corresponding speaker vector of a set of speaker vectors as an output, wherein each speaker vector comprises indications of vocal characteristics detected within the corresponding data fragment, and cluster the speaker vectors of the set of speaker vectors to identify the speakers of the set of speakers.

Each vocal characteristic indicated in the set of speaker vectors may be associated with a separate dimension of a multidimensional space, each speaker vector may correspond to a point of a set of points within the multidimensional space, and clustering the speaker vectors of the set of speaker vectors may include the at least one processor being caused to perform operations including: identify each speaker based on at least one of a threshold distance within the multidimensional space between pairs of points of the set of points, a threshold quantity of points within a preselected radius within the multidimensional space, or a threshold density of points within a preselected radius within the multidimensional space; and repeat a performance of clustering across a preselected quantity of temporally consecutive fragments in response to each instance of identifying a speaker.

Performing the preprocessing operations of the first speaker diarization technique may include the at least one processor being caused to perform, prior to analyzing vocal characteristics associated with each data fragment to identify a speaker, operations including: identify each data fragment in a subset of the set of data fragments in which a sentence pause likely occurs based on at least the first set of likely sentence pauses; and remove the subset of the data fragments from the set of data fragments.

In response to the request, the at least one processor may be caused to perform preprocessing operations of a second pause detection technique to identify a second set of likely sentence pauses; and identifying each data fragment in which a sentence pause likely occurs based on at least the first set of likely sentence pauses may include identifying each data fragment in which a sentence pause likely occurs based on a combination of the first set of likely sentence pauses and the second set of likely sentence pauses.

Performing preprocessing operations of the first pause detection technique may include deriving an audio noise level based on at least one measure of a level of audio noise of the speech audio; in response to the request, the at least one processor may be caused to perform preprocessing operations of a second speaker diarization technique to identify a second set of likely speaker changes; after performing the preprocessing operations of the first speaker diarization technique and the second speaker diarization technique, the at least one processor is caused to, based on the audio noise level, and based on differences in susceptibility to inaccuracies in identifying speakers caused by audio noise between the first speaker diarization technique and the second speaker diarization technique, derive a first relative weighting between the first set of likely speaker changes and the second set of likely speaker changes; and dividing the speech data set into multiple data segments based on at least the first set of likely speaker changes may include using the first relative weighting to combine the first set of likely speaker changes and the second set of likely speaker changes to generate a single set of indications of likely speaker changes, and dividing the speech data set into multiple data segments based on at least the single set of indications of likely speaker changes.

In response to the request, the at least one processor may be caused to perform preprocessing operations of a second pause detection technique to identify a second set of likely sentence pauses; and after performing the preprocessing operations of the first pause detection technique and the second pause detection technique, the at least one processor may be caused to, based on the audio noise level, and based on differences in susceptibility to inaccuracies in identifying sentence pauses caused by audio noise between the first pause detection technique and the second pause detection technique, derive a second relative weighting between the first set of likely sentence pauses and the second set of likely sentence pauses. Dividing the speech data set into multiple data segments based on at least the first set of likely sentence pauses may include: using the second relative weighting to combine the first set of likely sentence pauses and the second set of likely sentence pauses to generate a single set of indications of likely sentence pauses; and dividing the speech data set into multiple data segments based on at least the single set of indications of likely speaker changes and the single set of indications of likely sentence pauses.

Performing the preprocessing operations of the first pause detection technique may include the at least one processor being caused to perform operations including: divide the speech data set into multiple data chunks that each represent a chunk of multiple chunks of the speech audio; instantiate an instance of an acoustic model neural network comprising connectionist temporal classification (CTC) output; analyze the speech audio to identify pauses between speech sounds by providing each data chunk of the multiple data chunks to the instance of the acoustic model neural network as an input and monitor the CTC output for at least one corresponding string of blank symbols indicative of a pause; and analyze the lengths of the pauses to identify a first set of likely sentence pauses by comparing a length of each string of blank symbols from the CTC output to a predetermined blank threshold length.

Using at least the acoustic model with each data segment of the multiple data segments to identify likely speech sounds may include the at least one processor being caused to perform operations including: instantiate another instance of the acoustic model neural network comprising the CTC output; and provide indications of detected acoustic features of the speech segment of each data segment to the other instance of the acoustic model neural network as an input and monitor outputs of the other instance of the acoustic model neural network for corresponding probability distributions indicative of relative probabilities of speech sounds, including probabilities of consecutive instances of a text character.

Generating the transcript based, at least in part, on the identified likely speech sounds may include the at least one processor being caused to perform operations including: analyze probability distributions that are output by the acoustic model, and that identify likely speech sounds, to identify combinations of speech sounds that correspond to candidate sets of words; use a language model with the candidate sets of words to derive corresponding candidate sets of n-grams accompanied by corresponding indications of relative probabilities of use of each n-gram within each candidate set of n-grams; and use at least one of probability distributions output by the acoustic model or indications of relative probabilities output by the language model to select each word included in the transcript from either a corresponding candidate set of words or a corresponding candidate set of n-grams.

A computer-implemented method includes: receiving, by at least one processor, and from a requesting device via a network, a request to perform speech-to-text conversion of a specified speech data set representing speech audio; and in response to the request, performing preprocessing operations of a first pause detection technique including analyzing, by the at least one processor, the speech audio to identify pauses between speech sounds, and analyzing, by the at least one processor, lengths of the pauses to identify a first set of likely sentence pauses. The method also includes, in response to the request, performing preprocessing operations of a first speaker diarization technique including: dividing the speech data set into a set of data fragments that each represent a fragment of a set of fragments of the speech audio; for each data fragment, analyzing, by the at least one processor, vocal characteristics of speech sounds of the fragment to identify a speaker of a set of speakers; and identifying, by the at least one processor, instances of a change in speakers between temporally consecutive fragments of the set of fragments to identify a first set of likely speaker changes. The method further includes, in response to the request, performing speech-to-text operations including: based on at least the first set of likely sentence pauses, and based on at least the first set of likely speaker changes, dividing the speech data set into multiple data segments that each represent a speech segment of multiple speech segments of the speech audio; using, by the at least one processor, at least an acoustic model with each data segment of the multiple data segments to identify likely speech sounds in the speech audio; and generating, by the at least one processor, a transcript of the speech data set based, at least in part, on the identified likely speech sounds, or transmitting, from the at least one processor, an indication of the generation of the transcript to the requesting device.

Performing the preprocessing operations of the first speaker diarization technique may include performing operations including instantiating, by the at least one processor, a speaker diarization neural network trained to output indications of vocal characteristics, and analyzing, by the at least one processor, vocal characteristics of the speech sounds of each data fragment by performing operations including: providing each data fragment of the set of data fragments to the speaker diarization neural network as an input to cause the speaker diarization neural network to generate a corresponding speaker vector of a set of speaker vectors as an output, wherein each speaker vector comprises indications of vocal characteristics detected within the corresponding data fragment; and clustering, by the at least one processor, the speaker vectors of the set of speaker vectors to identify the speakers of the set of speakers.

Each vocal characteristic indicated in the set of speaker vectors may be associated with a separate dimension of a multidimensional space, each speaker vector may correspond to a point of a set of points within the multidimensional space, and clustering the speaker vectors of the set of speaker vectors may include performing operations including: identifying, by the at least one processor, each speaker based on at least one of a threshold distance within the multidimensional space between pairs of points of the set of points, a threshold quantity of points within a preselected radius within the multidimensional space, or a threshold density of points within a preselected radius within the multidimensional space; and repeating, by the at least one processor, a performance of clustering across a preselected quantity of temporally consecutive fragments in response to each instance of identifying a speaker.

Performing the preprocessing operations of the first speaker diarization technique may include performing, prior to analyzing vocal characteristics associated with each data fragment to identify a speaker, operations including: identifying, by the at least one processor, each data fragment in a subset of the set of data fragments in which a sentence pause likely occurs based on at least the first set of likely sentence pauses; and removing the subset of the data fragments from the set of data fragments.

The method may further include, in response to the request, performing, by the at least one processor, preprocessing operations of a second pause detection technique to identify a second set of likely sentence pauses; and identifying each data fragment in which a sentence pause likely occurs based on at least the first set of likely sentence pauses comprises identifying, by the at least one processor, each data fragment in which a sentence pause likely occurs based on a combination of the first set of likely sentence pauses and the second set of likely sentence pauses.

Performing preprocessing operations of the first pause detection technique may include deriving, by the at least one processor, an audio noise level based on at least one measure of a level of audio noise of the speech audio. The method may further include: in response to the request, performing, by the at least one processor, preprocessing operations of a second speaker diarization technique to identify a second set of likely speaker changes; and after performing the preprocessing operations of the first speaker diarization technique and the second speaker diarization technique, deriving, by the at least one processor, a first relative weighting between the first set of likely speaker changes and the second set of likely speaker changes based on the audio noise level, and based on differences in susceptibility to inaccuracies in identifying speakers caused by audio noise between the first speaker diarization technique and the second speaker diarization technique. Dividing the speech data set into multiple data segments based on at least the first set of likely speaker changes may include: using, by the at least one processor, the first relative weighting to combine the first set of likely speaker changes and the second set of likely speaker changes to generate a single set of indications of likely speaker changes; and dividing the speech data set into multiple data segments based on at least the single set of indications of likely speaker changes.

The method may further include: in response to the request, performing, by the at least one processor, preprocessing operations of a second pause detection technique to identify a second set of likely sentence pauses; and after performing the preprocessing operations of the first pause detection technique and the second pause detection technique, deriving, by the at least one processor, a second relative weighting between the first set of likely sentence pauses and the second set of likely sentence pauses based on the audio noise level, and based on differences in susceptibility to inaccuracies in identifying sentence pauses caused by audio noise between the first pause detection technique and the second pause detection technique. Dividing the speech data set into multiple data segments based on at least the first set of likely sentence pauses may include: using, by the at least one processor, the second relative weighting to combine the first set of likely sentence pauses and the second set of likely sentence pauses to generate a single set of indications of likely sentence pauses; and dividing the speech data set into multiple data segments based on at least the single set of indications of likely speaker changes and the single set of indications of likely sentence pauses.

Wherein performing the preprocessing operations of the first pause detection technique may include performing operations including: dividing the speech data set into multiple data chunks that each represent a chunk of multiple chunks of the speech audio; instantiating, by the at least one processor, an instance of an acoustic model neural network comprising connectionist temporal classification (CTC) output; analyzing, by the at least one processor, the speech audio to identify pauses between speech sounds by providing each data chunk of the multiple data chunks to the instance of the acoustic model neural network as an input and monitoring the CTC output for at least one corresponding string of blank symbols indicative of a pause; and analyzing, by the at least one processor, the lengths of the pauses to identify a first set of likely sentence pauses by comparing a length of each string of blank symbols from the CTC output to a predetermined blank threshold length.

Using at least the acoustic model with each data segment of the multiple data segments to identify likely speech sounds may include performing operations including: instantiating, by the at least one processor, another instance of the acoustic model neural network comprising the CTC output; and providing indications of detected acoustic features of the speech segment of each data segment to the other instance of the acoustic model neural network as an input and monitoring outputs of the other instance of the acoustic model neural network for corresponding probability distributions indicative of relative probabilities of speech sounds, including probabilities of consecutive instances of a text character.

Generating the transcript based, at least in part, on the identified likely speech sounds may include performing operations including: analyzing, by the at least one processor, probability distributions that are output by the acoustic model, and that identify likely speech sounds, to identify combinations of speech sounds that correspond to candidate sets of words; using, by the at least one processor, a language model with the candidate sets of words to derive corresponding candidate sets of n-grams accompanied by corresponding indications of relative probabilities of use of each n-gram within each candidate set of n-grams; and using, by the at least one processor, at least one of probability distributions output by the acoustic model or indications of relative probabilities output by the language model to select each word included in the transcript from either a corresponding candidate set of words or a corresponding candidate set of n-grams.

The foregoing, together with other features and embodiments, will become more apparent upon referring to the following specification, claims, and accompanying drawings.

