<jupyter_start><jupyter_text>Loading necessary libraries and also mounting google drive where the datasets are present.We have one French to Fongbe JW300 dataset present from where we will extract relevant datapoints to augment the train data. We will then train a model on the extracted datasets for French-FongbeWe also have one French to Ewe JW300 dataset present from where we will extract relevant datapoints to augment the French-Ewe data and train our model on the augmented data<jupyter_code># !pip install -U sentence-transformers
# !pip install --upgrade transformers
from google.colab import drive
drive.mount('/content/drive')<jupyter_output>Mounted at /content/drive<jupyter_text>Loading Necessary libraries needed to extract relevant dataset from the JW300 data collected<jupyter_code>import pandas as pd
import numpy as np
import gc
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity<jupyter_output><empty_output><jupyter_text>Augmented Data Extraction   We start with French To Fongbe  Method1Here we consider only the French records from the test data and try to extract the similar records from the JW300 French-Fongbe dataset. Cosine similarity is used to find the similarity between one test data-point and all other French datapoints in the JW300 data.Only records with highest and second highest similarities are taken to consideration.  Model used for sentence similarity - XLM-ROBERTA-LARGE<jupyter_code>jwdf = pd.read_csv('/content/drive/MyDrive/takwimu_translations/french_fongbe.csv')
jwdf['French'] = jwdf['French'].astype(str)
jwdf['Fongbe'] = jwdf['Fongbe'].astype(str)

jwdf = jwdf.dropna(subset=['French'])
jwdf = jwdf.dropna(subset=['Fongbe'])


availdf  = pd.read_csv('Test.csv')
availdf = availdf[availdf['Target_Language']=='Fon']
model = SentenceTransformer('xlm-roberta-large')
highest_indices_df           = pd.DataFrame(columns=['French'])
highest_indices_df['French'] = availdf['French'].values.tolist()

highest_similarity_df           = pd.DataFrame(columns=['French'])
highest_similarity_df['French'] = availdf['French'].values.tolist()


batch = 1
rows        = availdf['French'].values.tolist()
row_enc     = model.encode(rows,batch_size=64,show_progress_bar=True)
for i in range(0,jwdf.shape[0],50000):
  print('batch:- ',batch)
  
  start = i
  end   = start+50000-1


  columns = jwdf['French'].loc[start:end].values.tolist()
  columns_enc = model.encode(columns,batch_size=128,show_progress_bar=True)
  try:
    a = start
    b = end+1
    colnames                   = list(range(a,b))
    colnames                   = [str(i) for i in colnames]
    simdf                      = pd.DataFrame(cosine_similarity(row_enc,columns_enc),columns=colnames)
  except:
    a = start
    b = a+len(columns)
    colnames                   = list(range(a,b))
    colnames                   = [str(i) for i in colnames]
    simdf                      = pd.DataFrame(cosine_similarity(row_enc,columns_enc),columns=colnames)

  highest_label              = simdf.idxmax(axis=1).values
  

  highest_match              = simdf.max(axis=1).values
  

  highest_indices_df[f'batch_{batch}_indices']       = highest_label
  highest_similarity_df[f'batch_{batch}_similarity'] = highest_match

  
  del simdf
  gc.collect()
  batch = batch+1
sim = highest_similarity_df.drop(['French'],axis=1)
highest_indices_df['second_most_sim'] = sim.T.apply(lambda x: x.nlargest(2).idxmin())
highest_indices_df['first_most_sim']  = sim.T.apply(lambda x: x.nlargest(1).idxmin())
def get_index(text):
  return text.replace('similarity','indices')

highest_indices_df['second_most_sim'] = highest_indices_df.apply(lambda z: z[get_index(z['second_most_sim'])],axis=1) 
highest_indices_df['first_most_sim']  = highest_indices_df.apply(lambda z: z[get_index(z['first_most_sim'])],axis=1) 
indices = highest_indices_df['first_most_sim'].values.tolist()
data = jwdf[jwdf.index.isin(indices)]
data.shape
indices1 = highest_indices_df['second_most_sim'].values.tolist()
data1 = jwdf[jwdf.index.isin(indices1)]
data1.shape
data.to_csv('french_fongbe_train_xlm_roberta.csv',index=False)
data1.to_csv('french_fongbe_valid_xlm_roberta.csv',index=False)<jupyter_output><empty_output><jupyter_text>Method 2:-  Here we perform the same procedure as above and we use Language agnostic BERT Sentence to calculate vectors which will be used to find similarities between the sentences.We will determine the highest and second highest similar French statements and augment the train data with it<jupyter_code>model = SentenceTransformer('LaBSE')
highest_indices_df           = pd.DataFrame(columns=['French'])
highest_indices_df['French'] = availdf['French'].values.tolist()

highest_similarity_df           = pd.DataFrame(columns=['French'])
highest_similarity_df['French'] = availdf['French'].values.tolist()


batch = 1
rows        = availdf['French'].values.tolist()
row_enc     = model.encode(rows,batch_size=64,show_progress_bar=True)
for i in range(0,jwdf.shape[0],50000):
  print('batch:- ',batch)
  
  start = i
  end   = start+50000-1


  columns = jwdf['French'].loc[start:end].values.tolist()
  columns_enc = model.encode(columns,batch_size=128,show_progress_bar=True)
  try:
    a = start
    b = end+1
    colnames                   = list(range(a,b))
    colnames                   = [str(i) for i in colnames]
    simdf                      = pd.DataFrame(cosine_similarity(row_enc,columns_enc),columns=colnames)
  except:
    a = start
    b = a+len(columns)
    colnames                   = list(range(a,b))
    colnames                   = [str(i) for i in colnames]
    simdf                      = pd.DataFrame(cosine_similarity(row_enc,columns_enc),columns=colnames)

  highest_label              = simdf.idxmax(axis=1).values
  

  highest_match              = simdf.max(axis=1).values
  

  highest_indices_df[f'batch_{batch}_indices']       = highest_label
  highest_similarity_df[f'batch_{batch}_similarity'] = highest_match

  
  del simdf
  gc.collect()
  batch = batch+1


sim = highest_similarity_df.drop(['French'],axis=1)
highest_indices_df['second_most_sim'] = sim.T.apply(lambda x: x.nlargest(2).idxmin())
highest_indices_df['first_most_sim']  = sim.T.apply(lambda x: x.nlargest(1).idxmin())

def get_index(text):
  return text.replace('similarity','indices')

highest_indices_df['second_most_sim'] = highest_indices_df.apply(lambda z: z[get_index(z['second_most_sim'])],axis=1) 
highest_indices_df['first_most_sim']  = highest_indices_df.apply(lambda z: z[get_index(z['first_most_sim'])],axis=1) 


indices = highest_indices_df['first_most_sim'].values.tolist()
data = jwdf[jwdf.index.isin(indices)]
data.shape

indices1 = highest_indices_df['second_most_sim'].values.tolist()
data1 = jwdf[jwdf.index.isin(indices1)]
data1.shape


data.to_csv('french_fongbe_train_labse.csv',index=False)
data1.to_csv('french_fongbe_valid_labse.csv',index=False)<jupyter_output><empty_output><jupyter_text>Extracting statements from French-Ewe JW300We perform the same procedure for EWE as well but since we have a pretrained seq2seq model for french-Ewe, we will use only one method to extract relevant datasets which will be XLM-ROBERTA-LARGE and extract those points which are going to augment the train data.<jupyter_code>jwdf = pd.read_csv('/content/drive/MyDrive/takwimu_translations/french_ewe.csv')
jwdf['French'] = jwdf['French'].astype(str)
jwdf['Ewe'] = jwdf['Ewe'].astype(str)

jwdf = jwdf.dropna(subset=['French'])
jwdf = jwdf.dropna(subset=['Ewe'])


availdf  = pd.read_csv('Test.csv')
availdf = availdf[availdf['Target_Language']=='Ewe']
model = SentenceTransformer('xlm-roberta-large')
highest_indices_df           = pd.DataFrame(columns=['French'])
highest_indices_df['French'] = availdf['French'].values.tolist()

highest_similarity_df           = pd.DataFrame(columns=['French'])
highest_similarity_df['French'] = availdf['French'].values.tolist()


batch = 1
rows        = availdf['French'].values.tolist()
row_enc     = model.encode(rows,batch_size=64,show_progress_bar=True)
for i in range(0,jwdf.shape[0],50000):
  print('batch:- ',batch)
  
  start = i
  end   = start+50000-1


  columns = jwdf['French'].loc[start:end].values.tolist()
  columns_enc = model.encode(columns,batch_size=128,show_progress_bar=True)
  try:
    a = start
    b = end+1
    colnames                   = list(range(a,b))
    colnames                   = [str(i) for i in colnames]
    simdf                      = pd.DataFrame(cosine_similarity(row_enc,columns_enc),columns=colnames)
  except:
    a = start
    b = a+len(columns)
    colnames                   = list(range(a,b))
    colnames                   = [str(i) for i in colnames]
    simdf                      = pd.DataFrame(cosine_similarity(row_enc,columns_enc),columns=colnames)

  highest_label              = simdf.idxmax(axis=1).values
  

  highest_match              = simdf.max(axis=1).values
  

  highest_indices_df[f'batch_{batch}_indices']       = highest_label
  highest_similarity_df[f'batch_{batch}_similarity'] = highest_match

  
  del simdf
  gc.collect()
  batch = batch+1


sim = highest_similarity_df.drop(['French'],axis=1)
highest_indices_df['second_most_sim'] = sim.T.apply(lambda x: x.nlargest(2).idxmin())
highest_indices_df['first_most_sim']  = sim.T.apply(lambda x: x.nlargest(1).idxmin())

def get_index(text):
  return text.replace('similarity','indices')

highest_indices_df['second_most_sim'] = highest_indices_df.apply(lambda z: z[get_index(z['second_most_sim'])],axis=1) 
highest_indices_df['first_most_sim']  = highest_indices_df.apply(lambda z: z[get_index(z['first_most_sim'])],axis=1) 


indices = highest_indices_df['first_most_sim'].values.tolist()
data = jwdf[jwdf.index.isin(indices)]
data.shape

indices1 = highest_indices_df['second_most_sim'].values.tolist()
data1 = jwdf[jwdf.index.isin(indices1)]
data1.shape


data.to_csv('french_ewe_train_xlm_roberta.csv',index=False)
data1.to_csv('french_ewe_valid_xlm_roberta.csv',index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12109141605257917, 'fraction_numerical': 0.011949810794662417, 'mean_word_length': 4.5394373965802535, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '603167', 'license': 'MIT', 'n_tokens_mistral': 3618, 'n_tokens_neox': 3373, 'n_words': 627, 'path': 'Competition-Solutions/Text/AI4D Takwimu Lab Machine Translation Challenge/Solution 2/JW300_relevent_data_extraction.ipynb', 'repo_name': 'ZindiAfrica/Natural-Language-Processing-NLP-', 'size': '10042'}