djward888 commited on
Commit
68c5754
1 Parent(s): c9bcf79

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +205 -2
README.md CHANGED
@@ -1,5 +1,208 @@
1
  ---
2
  license: other
3
- license_name: llama-3
4
- license_link: LICENSE
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: other
3
+ base_model: meta-llama/Meta-Llama-3-8B
4
+ tags:
5
+ - generated_from_trainer
6
+ model-index:
7
+ - name: out
8
+ results: []
9
+ datasets:
10
+ - cognitivecomputations/Dolphin-2.9
11
+ - teknium/OpenHermes-2.5
12
+ - m-a-p/CodeFeedback-Filtered-Instruction
13
+ - cognitivecomputations/dolphin-coder
14
+ - cognitivecomputations/samantha-data
15
+ - HuggingFaceH4/ultrachat_200k
16
+ - microsoft/orca-math-word-problems-200k
17
+ - abacusai/SystemChat-1.1
18
+ - Locutusque/function-calling-chatml
19
+ - internlm/Agent-FLAN
20
  ---
21
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
22
+ should probably proofread and complete it, then remove this comment. -->
23
+
24
+ # Dolphin 2.9 Llama 3 8b 🐬
25
+
26
+ Curated and trained by Eric Hartford, Lucas Atkins, and Fernando Fernandes, and Cognitive Computations
27
+
28
+ Discord: https://discord.gg/8fbBeC7ZGx
29
+
30
+ <img src="https://cdn-uploads.huggingface.co/production/uploads/63111b2d88942700629f5771/ldkN1J0WIDQwU4vutGYiD.png" width="600" />
31
+
32
+ My appreciation for the sponsors of Dolphin 2.9:
33
+ - [Crusoe Cloud](https://crusoe.ai/) - provided excellent on-demand 10xL40S node
34
+
35
+ This model is based on Llama-3-8b, and is governed by [META LLAMA 3 COMMUNITY LICENSE AGREEMENT](LICENSE)
36
+
37
+ The base model has 8k context, and the full-weight fine-tuning was with 4k sequence length.
38
+
39
+ It took 2.5 days on 8x L40S provided by Crusoe Cloud
40
+
41
+ This model was trained FFT on all parameters, using ChatML prompt template format.
42
+
43
+ example:
44
+
45
+ ```
46
+ <|im_start|>system
47
+ You are Dolphin, a helpful AI assistant.<|im_end|>
48
+ <|im_start|>user
49
+ {prompt}<|im_end|>
50
+ <|im_start|>assistant
51
+ ```
52
+
53
+ Dolphin-2.9 has a variety of instruction, conversational, and coding skills. It also has initial agentic abilities and supports function calling.
54
+
55
+ Dolphin is uncensored. I have filtered the dataset to remove alignment and bias. This makes the model more compliant. You are advised to implement your own alignment layer before exposing the model as a service. It will be highly compliant with any requests, even unethical ones. Please read my blog post about uncensored models. https://erichartford.com/uncensored-models You are responsible for any content you create using this model. Enjoy responsibly.
56
+
57
+ Dolphin is licensed according to Meta's Llama license. I grant permission for any use, including commercial, that falls within accordance with Meta's Llama-3 license. Dolphin was trained on data generated from GPT4, among other models.
58
+
59
+ [<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
60
+ <details><summary>See axolotl config</summary>
61
+
62
+ axolotl version: `0.4.0`
63
+ ```yaml
64
+ base_model: meta-llama/Meta-Llama-3-8B
65
+ model_type: AutoModelForCausalLM
66
+ tokenizer_type: AutoTokenizer
67
+ tokenizer_use_fast: false
68
+ load_in_8bit: false
69
+ load_in_4bit: false
70
+ strict: false
71
+ model_config:
72
+ datasets:
73
+ - path: /workspace/datasets/dolphin-2.9/dolphin201-sharegpt2.jsonl
74
+ type: sharegpt
75
+ conversation: chatml
76
+ - path: /workspace/datasets/dolphin-2.9/Ultrachat200kunfiltered.jsonl
77
+ type: sharegpt
78
+ conversation: chatml
79
+ - path: /workspace/datasets/dolphin-2.9/dolphin-coder-translate-sharegpt2.jsonl
80
+ type: sharegpt
81
+ conversation: chatml
82
+ - path: /workspace/datasets/dolphin-2.9/dolphin-coder-codegen-sharegpt2.jsonl
83
+ type: sharegpt
84
+ conversation: chatml
85
+ - path: /workspace/datasets/dolphin-2.9/m-a-p_Code-Feedback-sharegpt-unfiltered.jsonl
86
+ type: sharegpt
87
+ conversation: chatml
88
+ - path: /workspace/datasets/dolphin-2.9/m-a-p_CodeFeedback-Filtered-Instruction-sharegpt-unfiltered.jsonl
89
+ type: sharegpt
90
+ conversation: chatml
91
+ - path: /workspace/datasets/dolphin-2.9/not_samantha_norefusals.jsonl
92
+ type: sharegpt
93
+ conversation: chatml
94
+ - path: /workspace/datasets/dolphin-2.9/Orca-Math-resort-unfiltered.jsonl
95
+ type: sharegpt
96
+ conversation: chatml
97
+ - path: /workspace/datasets/dolphin-2.9/agent_instruct_react_unfiltered.jsonl
98
+ type: sharegpt
99
+ conversation: chatml
100
+ - path: /workspace/datasets/dolphin-2.9/toolbench_instruct_j1s1_3k_unfiltered.jsonl
101
+ type: sharegpt
102
+ conversation: chatml
103
+ - path: /workspace/datasets/dolphin-2.9/toolbench_negative_unfiltered.jsonl
104
+ type: sharegpt
105
+ conversation: chatml
106
+ - path: /workspace/datasets/dolphin-2.9/toolbench_react_10p_unfiltered.jsonl
107
+ type: sharegpt
108
+ conversation: chatml
109
+ - path: /workspace/datasets/dolphin-2.9/toolbench_tflan_cot_30p_unfiltered.jsonl
110
+ type: sharegpt
111
+ conversation: chatml
112
+ - path: /workspace/datasets/dolphin-2.9/openhermes200k_unfiltered.jsonl
113
+ type: sharegpt
114
+ conversation: chatml
115
+ - path: /workspace/datasets/dolphin-2.9/SystemConversations.jsonl
116
+ type: sharegpt
117
+ conversation: chatml
118
+ chat_template: chatml
119
+
120
+ dataset_prepared_path: /workspace/datasets/dolphin-2.9/thingy
121
+ val_set_size: 0.0002
122
+ output_dir: ./out
123
+ sequence_len: 4096
124
+ sample_packing: true
125
+ pad_to_sequence_len: true
126
+ gradient_accumulation_steps: 4
127
+ micro_batch_size: 3
128
+ num_epochs: 3
129
+ logging_steps: 1
130
+ optimizer: adamw_8bit
131
+ lr_scheduler: cosine
132
+ learning_rate: 2e-5
133
+ wandb_project: dolphin-2.9-mixtral-8x22b
134
+ wandb_watch:
135
+ wandb_run_id:
136
+ wandb_log_model:
137
+ train_on_inputs: false
138
+ group_by_length: false
139
+ bf16: auto
140
+ fp16:
141
+ tf32: false
142
+ gradient_checkpointing: true
143
+ gradient_checkpointing_kwargs:
144
+ use_reentrant: false
145
+ early_stopping_patience:
146
+ resume_from_checkpoint:
147
+ local_rank:
148
+ logging_steps: 1
149
+ xformers_attention:
150
+ flash_attention: true
151
+ saves_per_epoch: 4
152
+ save_total_limit: 2
153
+ save_steps:
154
+ evals_per_epoch: 4
155
+ eval_sample_packing: false
156
+ debug:
157
+ deepspeed: deepspeed_configs/zero3_bf16.json
158
+ weight_decay: 0.05
159
+ fsdp:
160
+ fsdp_config:
161
+ special_tokens:
162
+ eos_token: "<|im_end|>"
163
+ pad_token: "<|end_of_text|>"
164
+ tokens:
165
+ - "<|im_start|>"
166
+ - "<|im_end|>"
167
+ ```
168
+
169
+ </details><br>
170
+
171
+ ## Training procedure
172
+
173
+ ### Training hyperparameters
174
+
175
+ The following hyperparameters were used during training:
176
+ - learning_rate: 2e-05
177
+ - train_batch_size: 3
178
+ - eval_batch_size: 3
179
+ - seed: 42
180
+ - distributed_type: multi-GPU
181
+ - num_devices: 8
182
+ - gradient_accumulation_steps: 4
183
+ - total_train_batch_size: 96
184
+ - total_eval_batch_size: 24
185
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
186
+ - lr_scheduler_type: cosine
187
+ - lr_scheduler_warmup_steps: 7
188
+ - num_epochs: 3
189
+ ### Training results
190
+ | Training Loss | Epoch | Step | Validation Loss |
191
+ |:-------------:|:------:|:----:|:---------------:|
192
+ | 1.146 | 0.0005 | 1 | 1.1064 |
193
+ | 0.6962 | 0.2501 | 555 | 0.6636 |
194
+ | 0.6857 | 0.5001 | 1110 | 0.6503 |
195
+ | 0.6592 | 0.7502 | 1665 | 0.6419 |
196
+ | 0.6465 | 1.0002 | 2220 | 0.6317 |
197
+ | 0.5295 | 1.2395 | 2775 | 0.6408 |
198
+ | 0.5302 | 1.4895 | 3330 | 0.6351 |
199
+ | 0.5188 | 1.7396 | 3885 | 0.6227 |
200
+ | 0.521 | 1.9896 | 4440 | 0.6168 |
201
+ | 0.3968 | 2.2289 | 4995 | 0.6646 |
202
+ | 0.3776 | 2.4789 | 5550 | 0.6619 |
203
+ | 0.3983 | 2.7290 | 6105 | 0.6602 |
204
+ ### Framework versions
205
+ - Transformers 4.40.0
206
+ - Pytorch 2.2.2+cu121
207
+ - Datasets 2.18.0
208
+ - Tokenizers 0.19.1