CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit under 35 USC § 119(a) of Korean Patent Application No. 10-2019-0127019 filed on Oct. 14, 2019, in the Korean Intellectual Property Office, the entire disclosure of which are incorporated herein by reference for all purposes.

BACKGROUND

1. Field

The following description relates to a method and apparatus with model training and/or sequence recognition.

2. Description of Related Art

A typical sequence-to-sequence neural network may be modeled, trained, and/or configured to determine context information corresponding to an entire input sequence and output an output sequence based on the context information determined for the entire input sequence. Accordingly, the typical sequence-to-sequence neural network may require the entire input sequence to be input in order to determine any or all portions of the output sequence, and thus the typical sequence-to-sequence neural network may not be configured to generate a partial output of the output sequence based on a partial input of the input sequence. Accordingly, when implemented in a field of applications such as real-time machine interpretation/translation and/or real-time speech recognition that use gradual streaming, the typical sequence-to-sequence neural network may not be configured to generate a result for a partial input of the input sequence. In the field of applications such as real-time machine interpretation/translation and/or real-time speech recognition, entire information of the input sequence may not be used in generating portions of the output sequence, and thus a highly accurate and consistent result may not be output from the typical sequence-to-sequence neural network.

SUMMARY

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

In one general aspect, a processor-implemented method includes: using an encoder, determining, for each of a plurality of tokens included in an input sequence, a self-attention weight based on a token and one or more tokens that precede the token in the input sequence; using the encoder, determining context information corresponding to the input sequence based on the determined self-attention weights; and using a decoder, determining an output sequence corresponding to the input sequence based on the determined context information.

The method may include training the encoder and the decoder based on the determined output sequence.

The determining of the self-attention weight may include: masking token relationships between the token and each of tokens that follow the token in the input sequence; and determining the self-attention weight based on a result of the masking.

The determining of the self-attention weight based on a result of the masking may include excluding a consideration of one or more of the tokens that follow the token.

The determining of the self-attention weight may include: determining the self-attention weight based on the token and each of a preset number of the tokens that precede the token in the input sequence.

The determining of the self-attention weight may include: determining the self-attention weight using two or more of the tokens included in the input sequence.

The determining of the self-attention weight may include: determining the self-attention weight based on the token and each of remaining tokens excluding a preset number of tokens among the tokens that precede the token in the input sequence.

The training of the encoder and the decoder may include: training the encoder and the decoder such that a loss between a true sequence corresponding to the input sequence and the output sequence is less than or equal to a threshold.

The encoder and the decoder may correspond to a transformer model.

Either one or both of the input sequence or the output sequence may be any one of speech data, sentence data, image data, biodata, and handwriting data.

A non-transitory computer-readable storage medium may store instructions that, when executed by one or more processors, configure the one or more processors to perform the method.

In another general aspect, a processor-implemented method includes: using an encoder, determining, each time a token included in an input sequence is input or obtained, a self-attention weight based on an input token and one or more tokens that precede the input token in the input sequence; determining context information corresponding to the currently input tokens based on the determined self-attention weight; and using a decoder, determining an output sequence corresponding to the currently input tokens based on the determined context information.

The determining of the self-attention weight may include: masking token relationships between the token and each of tokens that follow the token among the currently input tokens; and determining the self-attention weight based on a result of the masking.

The determining of the context information may include: updating the context information each time the token of the input sequence is input.

The determining of the self-attention weight may include: determining the self-attention weight based on the token and each of a preset number of the tokens that precede the token among the currently input tokens.

The determining of the self-attention weight may include: determining the self-attention weight using two or more tokens among the currently input tokens.

The determining of the self-attention weight may include: determining the self-attention weight based on the token and each of remaining tokens excluding a preset number of tokens among the tokens that precede the token among the currently input tokens.

The determining of the context information may include determining, for each of the tokens, context information corresponding to the token and the one or more of tokens that precede the token based on the determined self-attention weight of the token, and the determining of the output sequence on may include determining, for each of the tokens, a portion of the output sequence corresponding to the token based on the determined context corresponding to the token.

The method may include training the encoder and the decoder based on the determined output sequence.

In another general aspect, an apparatus includes: one or more processors configured to: determine, for each of a plurality of tokens included in an input sequence, a self-attention weight based on the token and one or more tokens that precede the token in the input sequence; determine, context information corresponding to the input sequence based on the determined self-attention weight; and determine, an output sequence corresponding to the input sequence based on the determined context information.

The one or more processors may be configured to train, based on the determined output sequence, an encoder for the determining of the self-attention weight and the determining of the context information and a decoder for the determining of the output sequence.

For the determining of the self-attention weight, the one or more processors may be configured to: mask token relationships between the token and each of tokens that follow the token in the input sequence; and determine the self-attention weight based on a result of the masking.

For the determining of the self-attention weight, the one or more processors may be configured to: determine the self-attention weight based on the token and each of a preset number of tokens that precede the token in the input sequence.

For the determining of the self-attention weight, the one or more processors may be configured to: determine the self-attention weight using two or more of the tokens included in the input sequence.

For the determining of the self-attention weight, the one or more processors may be configured to: determine the self-attention weight based on the token and each of remaining tokens excluding a preset number of tokens among the tokens that precede the token in the input sequence.

For the determining of the self-attention weight, the one or more processors may be configured to: mask token relationships between the token and each of tokens that follow the token in the input sequence; and determine the self-attention weight based on a result of the masking.

For the determining of the self-attention weight, the one or more processors may be configured to: determine the self-attention weight based on the token and each of a preset number of tokens that precede the token in the input sequence.

For the determining of the self-attention weight, the one or more processors may be configured to: determine the self-attention weight using two or more of the tokens included in the input sequence.

For the determining of the self-attention weight, the one or more processors may be configured to: determine the self-attention weight based on the token and each of remaining tokens excluding a preset number of tokens among the tokens that precede the token in the input sequence.

In another general aspect, a processor-implemented method includes: determining a self-attention weight of a token of an input sequence by considering a relation between the token and one or more tokens that precede the token in the input sequence and excluding a relation between the token and one or more tokens that follow the token in the input sequence; and determining a portion of an output sequence corresponding to the token based on the determined self-attention weight of the token.

Other features and aspects will be apparent from the following detailed description, the drawings, and the claims.

