ashwani-tanwar
commited on
Commit
•
588fce2
1
Parent(s):
6decf44
Create README.md
Browse files
README.md
ADDED
@@ -0,0 +1,49 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
language:
|
3 |
+
- gu
|
4 |
+
- hi
|
5 |
+
- mr
|
6 |
+
- bn
|
7 |
+
---
|
8 |
+
|
9 |
+
# Indo-Aryan-XLM-R-Base
|
10 |
+
|
11 |
+
|
12 |
+
This model is finetuned over [XLM-RoBERTa](https://huggingface.co/xlm-roberta-base) (XLM-R) using its base variant with the Hindi, Gujarati, Marathi, and Bengali languages from the Indo-Aryan family using the [OSCAR](https://oscar-corpus.com/) monolingual datasets. As these languages had imbalanced datasets, we used resampling strategies as used in pretraining the XLM-R to balance the resulting dataset after combining these languages. We used the same masked language modelling (MLM) objective which was used for pretraining the XLM-R. As it is built over the pretrained XLM-R, we leveraged *Transfer Learning* by exploiting the knowledge from its parent model.
|
13 |
+
|
14 |
+
## Dataset
|
15 |
+
OSCAR corpus contains several diverse datasets for different languages. We followed the work of [CamemBERT](https://www.aclweb.org/anthology/2020.acl-main.645/) who reported better performance with this diverse dataset as compared to the other large homogenous datasets.
|
16 |
+
|
17 |
+
## Preprocessing and Training Procedure
|
18 |
+
Please visit [this link](https://github.com/ashwanitanwar/nmt-transfer-learning-xlm-r#6-finetuning-xlm-r) for the detailed procedure.
|
19 |
+
|
20 |
+
## Usage
|
21 |
+
- This model can be used for further finetuning for different NLP tasks using the Hindi, Gujarati, Marathi, and Bengali languages.
|
22 |
+
- It can be used to generate contextualised word representations for the words from the above languages.
|
23 |
+
- It can be used for domain adaptation.
|
24 |
+
- It can be used to predict the missing words from their sentences.
|
25 |
+
|
26 |
+
## Demo
|
27 |
+
### Using the model to predict missing words
|
28 |
+
```
|
29 |
+
from transformers import pipeline
|
30 |
+
unmasker = pipeline('fill-mask', model='ashwani-tanwar/Indo-Aryan-XLM-R-Base')
|
31 |
+
pred_word = unmasker("અમદાવાદ એ ગુજરાતનું એક <mask> છે.")
|
32 |
+
print(pred_word)
|
33 |
+
```
|
34 |
+
```
|
35 |
+
[{'sequence': '<s> અમદાવાદ એ ગુજરાતનું એક શહેર છે.</s>', 'score': 0.7811868786811829, 'token': 85227, 'token_str': '▁શહેર'},
|
36 |
+
{'sequence': '<s> અમદાવાદ એ ગુજરાતનું એક ગામ છે.</s>', 'score': 0.055032357573509216, 'token': 66346, 'token_str': '▁ગામ'},
|
37 |
+
{'sequence': '<s> અમદાવાદ એ ગુજરાતનું એક નામ છે.</s>', 'score': 0.0287721399217844, 'token': 29565, 'token_str': '▁નામ'},
|
38 |
+
{'sequence': '<s> અમદાવાદ એ ગુજરાતનું એક રાજ્ય છે.</s>', 'score': 0.02565067447721958, 'token': 63678, 'token_str': '▁રાજ્ય'},
|
39 |
+
{'sequence': '<s> અમદાવાદ એ ગુજરાતનું એકનગર છે.</s>', 'score': 0.022877279669046402, 'token': 69702, 'token_str': 'નગર'}]
|
40 |
+
```
|
41 |
+
### Using the model to generate contextualised word representations
|
42 |
+
```
|
43 |
+
from transformers import AutoTokenizer, AutoModel
|
44 |
+
tokenizer = AutoTokenizer.from_pretrained("ashwani-tanwar/Indo-Aryan-XLM-R-Base")
|
45 |
+
model = AutoModel.from_pretrained("ashwani-tanwar/Indo-Aryan-XLM-R-Base")
|
46 |
+
sentence = "અમદાવાદ એ ગુજરાતનું એક શહેર છે."
|
47 |
+
encoded_sentence = tokenizer(sentence, return_tensors='pt')
|
48 |
+
context_word_rep = model(**encoded_sentence)
|
49 |
+
```
|