From 1c7b0f658de4c6f65398f3b5ca94b1ccc7edeb2f Mon Sep 17 00:00:00 2001 From: bug-orz Date: Tue, 12 Nov 2024 13:06:50 +0800 Subject: [PATCH 1/3] Create README.md --- examples/tapir/README.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 examples/tapir/README.md diff --git a/examples/tapir/README.md b/examples/tapir/README.md new file mode 100644 index 0000000..6be3429 --- /dev/null +++ b/examples/tapir/README.md @@ -0,0 +1,25 @@ +## Introduction + +Our paper "Distilling Instruction-following Abilities of Large Language Models with Task-aware Curriculum Planning" introduces a framework called Task-Aware Curriculum Planning for Instruction Refinement (TAPIR). TAPIR is designed to improve the instruction-following capabilities of large language models (LLMs) by addressing the challenges of task distribution and instruction difficulty during training. The framework uses an oracle LLM to select difficult instructions for a student LLM and adjusts task distributions to balance the student's capabilities. TAPIR also incorporates curriculum planning to escalate task difficulty levels progressively. + +## Models + +Download Tapir 7B: + +``` +bash dl_tapir_7B.sh +``` + +**Please use official Llama2 template:** + +>[INST] \<\> {{ .System }} \<\> +> +>{{ .Prompt }} +> +>[/INST] + +## Data + +Download Tapir_Instruct_70k Dataset: + +https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/TAPIR-Distillation/Tapir_Instruct.json From 67c549e307a6dbc2b8b78916fd719924335943c5 Mon Sep 17 00:00:00 2001 From: bug-orz Date: Tue, 12 Nov 2024 13:09:23 +0800 Subject: [PATCH 2/3] Create dl_tapir_7B.sh --- examples/tapir/dl_tapir_7B.sh | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 examples/tapir/dl_tapir_7B.sh diff --git a/examples/tapir/dl_tapir_7B.sh b/examples/tapir/dl_tapir_7B.sh new file mode 100644 index 0000000..6f88337 --- /dev/null +++ b/examples/tapir/dl_tapir_7B.sh @@ -0,0 +1,9 @@ +wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/TAPIR-Distillation/Tapir-7B/config.json +wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/TAPIR-Distillation/Tapir-7B/generation_config.json +wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/TAPIR-Distillation/Tapir-7B/model-00001-of-00003.safetensors +wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/TAPIR-Distillation/Tapir-7B/model-00001-of-00003.safetensors +wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/TAPIR-Distillation/Tapir-7B/model-00001-of-00003.safetensors +wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/TAPIR-Distillation/Tapir-7B/model.safetensors.index.json +wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/TAPIR-Distillation/Tapir-7B/special_tokens_map.json +wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/TAPIR-Distillation/Tapir-7B/tokenizer.model +wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/TAPIR-Distillation/Tapir-7B/tokenizer_config.json From 868db40976ffe18b9ca780b9e3a6440f25b07c1a Mon Sep 17 00:00:00 2001 From: bug-orz Date: Tue, 12 Nov 2024 13:13:46 +0800 Subject: [PATCH 3/3] Update README.md --- examples/tapir/README.md | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/examples/tapir/README.md b/examples/tapir/README.md index 6be3429..b17b396 100644 --- a/examples/tapir/README.md +++ b/examples/tapir/README.md @@ -1,8 +1,13 @@ -## Introduction +# TAPIR: Distilling Instruction-following Abilities of Large Language Models with Task-aware Curriculum Planning + +![image](https://github.com/user-attachments/assets/1c48a0ce-bc47-468a-9762-c952b1494d0e) + + +## 📖Introduction Our paper "Distilling Instruction-following Abilities of Large Language Models with Task-aware Curriculum Planning" introduces a framework called Task-Aware Curriculum Planning for Instruction Refinement (TAPIR). TAPIR is designed to improve the instruction-following capabilities of large language models (LLMs) by addressing the challenges of task distribution and instruction difficulty during training. The framework uses an oracle LLM to select difficult instructions for a student LLM and adjusts task distributions to balance the student's capabilities. TAPIR also incorporates curriculum planning to escalate task difficulty levels progressively. -## Models +## 🧠Models Download Tapir 7B: @@ -18,8 +23,24 @@ bash dl_tapir_7B.sh > >[/INST] -## Data +## 🗃️Data Download Tapir_Instruct_70k Dataset: https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/TAPIR-Distillation/Tapir_Instruct.json + +## 📜 Citation + +If you find our work helpful, please cite it! + +``` +@misc{TAPIR, + title={Distilling Instruction-following Abilities of Large Language Models with Task-aware Curriculum Planning}, + author={Yuanhao Yue and Chengyu Wang and Jun Huang and Peng Wang}, + year={2024}, + eprint={2405.13448}, + archivePrefix={arXiv}, + primaryClass={cs.CL}, + url={https://arxiv.org/abs/2405.13448}, +} +```