|
--- |
|
license: apache-2.0 |
|
language: |
|
- pl |
|
tags: |
|
- qlora |
|
- instruct |
|
--- |
|
|
|
This repo contains a qlora adapter for [Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1), fine-tuned on a private instructions dataset, exclusively in Polish. |
|
|
|
The fine-tuning took 1 hour on a single RTX 4090 with the following hyperparameters: |
|
|
|
* context length: 2048 |
|
* batch_size: 16 |
|
* learning_rate: 0.0001 |
|
* lora_r: 64 |
|
* lora_alpha: 16 |
|
* lora_modules: all |
|
* lora_dropout: 0.0 |
|
* weight_decay: 0.1 |
|
* max_grad_norm: 0.3 |
|
* double_quant, nf4 |
|
* optimizer: paged_adamw_32bit (beta2: 0.999) |
|
|
|
During the training of the adapter, the instruction format was maintained. Each prompt should be surrounded by `[INST]` and `[\INST]` tokens to leverage instruction fine-tuning. |
|
|
|
This adapter allows the model to speak Polish more accurately than vanilla [Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1). |
|
|
|
<p align="center"> |
|
<img src="https://huggingface.co/Azurro/Mistral-7B-Instruct-v0.1-qlora-polish/raw/main/mistral-7b-instruct-qlora-pl.jpg"> |
|
</p> |
