This repository contains a machine translation model that translates Farsi text to English using an encoder-decoder architecture.

The standardize_en and standardize_fa functions perform text preprocessing. standardize_en normalizes text, converts it to lowercase, keeps only alphabets, space, and selected punctuation marks, removes leading and trailing whitespaces, and adds special tokens [START] and [END]. standardize_fa removes specific punctuation marks and performs similar preprocessing as standardize_en.

An example Farsi text is defined, and the standardize_en function is applied to it to obtain the preprocessed text.

Variables are defined for removing specific punctuation marks in the standardize_fa function using regular expressions.

Text vectorization layers for English and Farsi languages are defined. en_vectorization for English language uses standardize_en for text preprocessing and has a maximum vocabulary size of 5000 tokens. fa_vectorization for Farsi language uses standardize_fa for text preprocessing and has a maximum vocabulary size of 5000 tokens.

Tokenization of text data is performed using text vectorization layers en_vectorization and fa_vectorization. The layers are adapted to the target and input data, and the top 10 words in the vocabulary for each layer are printed. The Farsi text data example_input_batch is tokenized using fa_vectorization layer, and the English text data example_target_batch is tokenized using en_vectorization layer.

The size of the vocabulary for fa_vectorization is returned.

The tokenized sequence from the en_vectorization and fa_vectorization layers is converted back to the original text by mapping each token to the corresponding word in the vocabulary. The reconstructed text is printed.

The embedding dimension and the number of units for the LSTM layer are defined.

The Encoder class is defined, which takes input_vocab_size, embedding_dim, and enc_units as arguments. The call method of the Encoder class receives a sequence of tokens and an optional initial state, passes the tokens through the embedding layer to obtain a sequence of vectors, and passes the sequence of vectors through the GRU layer to obtain a new sequence of vectors and an internal state. Finally, it returns the new sequence of vectors and the internal state.

An instance of the Encoder class is created with the Farsi text vectorization layer fa_vectorization as input. The Encoder layer is applied to the tokenized Farsi text data example_tokens and the output and the state are saved. The shapes of the input batch, input batch tokens, encoder output, and encoder state are printed.

The BahdanauAttention class is defined, which takes units as input. The call method of the BahdanauAttention class receives a query tensor, a value tensor, and a mask tensor, passes the query tensor through the W1 dense layer and the value tensor through the W2 dense layer to obtain the transformed query and value vectors, computes the attention scores between the query vector and each value vector, applies the mask to the attention scores, and computes the context vector as the weighted sum of the value vectors.

An instance of the BahdanauAttention class is created with units set to 10. The attention layer is applied to a query tensor and a value tensor, and the output is saved. The shape of the output tensor is printed.

The Decoder class is defined, which takes target_vocab_size, embedding_dim, dec_units, and att_units as arguments. The call method of the Decoder class receives a sequence of tokens, an encoder output, an optional initial state, and an optional mask, passes the tokens through the embedding layer to obtain a sequence of vectors, passes the sequence of vectors and the initial state through the GRU layer to obtain a new sequence of vectors and a new state, passes the new sequence of vectors and the encoder output through the attention layer to obtain a context vector, concatenates the context vector with each vector in the new sequence of vectors to obtain a new sequence of vectors with context, and passes the new sequence of vectors through a dense layer to obtain the logits for each token in the target vocabulary. Finally, it returns the logits.

An instance of the Decoder class is created with the English text vectorization layer en_vectorization as input, and the hyperparameters defined earlier. The Decoder layer is applied to the tokenized English text data example_target_tokens, the encoder output, and the encoder state, and the output is saved. The shape of the output tensor is printed.

The Translator class is defined, which takes encoder, decoder, fa_vectorization, and en_vectorization as arguments. The call method of the Translator class receives a sequence of tokens in Farsi, passes the tokens through the encoder to obtain the encoder output and the encoder state, initializes the decoder state with the encoder state, generates a sequence of tokens in English by iteratively applying the decoder to the previous token and the current decoder state until the [END] token is generated or the maximum sequence length is reached, and finally returns the generated sequence of tokens in English.

An instance of the Translator class is created with the Encoder and Decoder layers created earlier, and the fa_vectorization and en_vectorization vectorization layers. The translate method of the Translator class is used to translate the example Farsi text example_input to English, and the translated text example_output is printed.

Note that the hyperparameters, text preprocessing functions, and vectorization layers can be adjusted to optimize the performance of the model for specific use cases.