LLMs之Alpaca：《Alpaca: A Strong, Replicable Instruction-Following Model》翻译与解读_alpaca论文

时间

2023年3月13日

地址

原文地址：Stanford CRFM

论文地址：https://arxiv.org/abs/2203.01145

GitHUb地址：
GitHub - tatsu-lab/stanford_alpaca: Code and documentation to train Stanford's Alpaca models, and generate the data.

作者

Authors: Rohan Taori* and Ishaan Gulrajani* and Tianyi Zhang* and Yann Dubois* and Xuechen Li* and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto

We introduce Alpaca 7B, a model fine-tuned from the LLaMA 7B model on 52K instruction-following demonstrations. On our preliminary evaluation of single-turn instruction following, Alpaca behaves qualitatively similarly to OpenAI’s text-davinci-003, while being surprisingly small and easy/cheap to reproduce (<600$).

我们介绍了Alpaca 7B，这是一个在52,000个instruction-following示范中，通过对LLaMA 7B模型进行微调得到的模型。在我们对单轮instruction-following的初步评估中，Alpaca在质量上与OpenAI的text-davinci-003模型表现相似，但其规模却小得令人惊讶，并且容易且廉价地复现（不到600美元）。

Instruction-following models such as GPT-3.5 (text-davinci-003), ChatGPT, Claude, and Bing Chat have become increasingly powerful. Many users now interact with these models regularly and even use them for work. However, despite their widespread deployment, instruction-following models still have many deficiencies: they can generate false information, propagate social stereotypes, and produce toxic language.

To make maximum progress on addressing these pressing problems, it is important for the academic community to engage. Unfortunately, doing research on instruction-following models in academia has been difficult, as there is no easily accessible model that comes close in capabilities to closed-source models such as OpenAI’s text-davinci-003.

instruction-following模型，如GPT-3.5（text-davinci-003）、ChatGPT、Claude和Bing Chat，已经变得越来越强大。许多用户现在经常与这些模型互动，甚至将它们用于工作。然而，尽管这些模型已经被广泛应用，但instruction-following模型仍然存在许多问题：它们可能生成错误信息，传播社会刻板印象，并产生有害语言。

为了在解决这些紧迫问题上取得最大的进展，学术界的参与非常重要。然而，由于没有一个容易获取且与OpenAI的text-davinci-003等闭源模型在能力上接近的模型，因此在学术界进行instruction-following模型的研究一直很困难。

We are releasing our findings about an instruction-following language model, dubbed Alpaca, which is fine-tuned from Meta’s LLaMA 7B model. We train the Alpaca model on 52K instruction-following demonstrations generated in the style of self-instruct using text-davinci-003. On the self-instruct evaluation set, Alpaca shows many behaviors similar to OpenAI’s text-davinci-003, but is also surprisingly small and easy/cheap to reproduce.

We are releasing our training recipe and data, and intend to release the model weights in the future. We are also hosting an interactive demo to enable the research community to better understand the behavior of Alpaca. Interaction can expose unexpected capabilities and failures, which will guide us for the future evaluation of these models. We also encourage users to report any concerning behaviors in our web demo so that we can better understand and mitigate these behaviors. As any release carries risks, we discuss our thought process for this open release later in this blog post.

我们发布了有关一个名为Alpaca的instruction-following语言模型的研究结果。Alpaca是基于Meta的LLaMA 7B模型进行微调得到的。我们使用了以self-instruct风格生成的52,000个instruction-following示范进行Alpaca模型的训练，这些示范是使用text-davinci-003进行self-instruct生成的。在self-instruct的评估集上，Alpaca表现出与OpenAI的text-davinci-003类似的许多行为，但其规模却小得令人惊讶，并且容易且廉价地复现。

我们发布了训练的方法和数据，并打算在将来发布模型权重。我们还提供了一个交互式演示，以使研究界更好地理解Alpaca的行为。通过交互可以暴露出意外的能力和故障，这将指导我们对这些模型的未来评估。我们还鼓励用户在我们的Web演示中报告任何令人关注的行为，以便我们更好地理解和减轻这些行为。由于任何发布都存在风险，我们在本文中讨论了我们的思考过程。

We emphasize that Alpaca is intended only for academic research and any commercial use is prohibited. There are three factors in this decision: First, Alpaca is based on LLaMA, which has a non-commercial license, so we necessarily inherit this decision. Second, the instruction data is based on OpenAI’s text-davinci-003, whose terms of use prohibit developing models that compete with OpenAI. Finally, we have not designed adequate safety measures, so Alpaca is not ready to be deployed for general use.

我们强调，Alpaca仅用于学术研究，任何商业用途均被禁止。这个决定有三个因素：首先，Alpaca是基于LLaMA的，而LLaMA具有非商业许可证，因此我们必然继承了这个决定。其次，指令数据基于OpenAI的text-davinci-003，其使用条款禁止开发与OpenAI竞争的模型。最后，我们还没有设计足够的安全措施，因此Alpaca尚未准备好用于一般用途的部署。

There are two important challenges to training a high-quality instruction-following model under an academic budget: a strong pretrained language model and high-quality instruction-following data. The first challenge is addressed with the recent release of Meta’s new LLaMA models. For the second challenge, the self-instruct paper suggests using an existing strong language model to automatically generate instruction data. In particular, Alpaca is a language model fine-tuned using supervised learning from a LLaMA 7B model on 52K instruction-following demonstrations generated from OpenAI’s text-davinci-003.

在学术预算下训练高质量的instruction-following模型存在两个重要挑战：强大的预训练语言模型和高质量的instruction-following数据。第一个挑战通过Meta最新的LLaMA模型的发布得以解决。对于第二个挑战，self-instruct论文建议使用现有的强大语言模型自动生成指令数据。具体而言，Alpaca是使用监督学习从LLaMA 7B模型上的52,000个使用OpenAI的text-davinci-003生成的instruction-following示范进行微调的语言模型。

The figure below illustrates how we obtained the Alpaca model. For the data, we generated instruction-following demonstrations by building upon the self-instruct method. We started with the 175 human-written instruction-output pairs from the self-instruct seed set. We then prompted text-davinci-003 to generate more instructions using the seed set as in-context examples. We improved over the self-instruct method by simplifying the generation pipeline (see details in GitHub) and significantly reduced the cost. Our data generation process results in 52K unique instructions and the corresponding outputs, which costed less than $500 using the OpenAI API.

Equipped with this instruction-following dataset, we then fine-tuned the LLaMA models using Hugging Face’s training framework, taking advantage of techniques like Fully Sharded Data Parallel and mixed precision training. For our initial run, fine-tuning a 7B LLaMA model took 3 hours on 8 80GB A100s, which costs less than $100 on most cloud compute providers. We note that training efficiency can be improved to further reduce the cost.

下图展示了我们获得Alpaca模型的过程。对于数据部分，我们通过构建self-instruct方法的基础上生成了instruction-following示范。我们从self-instruct种子集的175个人工编写的指令-输出对开始。然后，我们以种子集作为上下文示例，提示text-davinci-003生成更多的指令。我们通过简化生成流程（详见GitHub中的细节）并大大降低成本，改进了self-instruct方法。我们的数据生成过程得到了52,000个独特指令及其对应的输出，使用OpenAI API的成本不到500美元。

凭借这个instruction-following数据集，我们使用Hugging Face的训练框架对LLaMA模型进行了微调，利用了Fully Sharded Data Parallel和混合精度训练等技术。对于我们的初始运行，在8个80GB的A100上对7B的LLaMA模型进行微调只需要3小时，这在大多数云计算提供商的成本低于100美元。我们注意到，训练效率可以进一步提高以进一步降低成本。

To evaluate Alpaca, we conduct human evaluation (by the 5 student authors) on the inputs from the self-instruct evaluation set. This evaluation set was collected by the self-instruct authors and covers a diverse list of user-oriented instructions including email writing, social media, and productivity tools. We performed a blind pairwise comparison between text-davinci-003 and Alpaca 7B, and we found that these two models have very similar performance: Alpaca wins 90 versus 89 comparisons against text-davinci-003.

We were quite surprised by this result given the small model size and the modest amount of instruction following data. Besides leveraging this static evaluation set, we have also been testing the Alpaca model interactively and found that Alpaca often behaves similarly to text-davinci-003 on a diverse set of inputs. We acknowledge that our evaluation may be limited in scale and diversity. So we are releasing an interactive demo of Alpaca, and encourage readers to evaluate Alpaca themselves and give us feedback.

为了评估Alpaca，我们对self-instruct评估集中的输入进行了人工评估（由5位学生作者进行）。该评估集由self-instruct的作者收集，涵盖了包括电子邮件写作、社交媒体和生产力工具在内的各种面向用户的指令。我们对text-davinci-003和Alpaca 7B进行了盲目的两两比较，发现这两个模型的性能非常相似：Alpaca在与text-davinci-003的比较中赢得了90次，而text-davinci-003只赢得了89次。

鉴于模型规模较小且instruction-following数据量有限，我们对这个结果感到非常惊讶。除了利用这个静态评估集，我们还进行了与Alpaca模型的交互测试，并发现Alpaca在各种输入上的行为通常与text-davinci-003相似。我们承认我们的评估可能在规模和多样性上存在限制。因此，我们发布了一个Alpaca的交互演示，并鼓励读者自行评估Alpaca并给我们反馈。

In the rest of this section, we include several interaction examples to showcase the capabilities and limitations of Alpaca.

The above examples show that the outputs of Alpaca are generally well-written. We note that Alpaca reflects the general style of the instruction-following dataset. As a result, Alpaca’s answers are typically shorter than ChatGPT, reflecting text-davinci-003’s shorter outputs.

在本节的其余部分，我们提供了几个交互示例，展示了Alpaca的能力和局限性。

上述示例显示，Alpaca的输出通常写得很好。我们注意到，Alpaca反映了instruction-following数据集的一般风格。因此，Alpaca的回答通常比ChatGPT更短，反映了text-davinci-003生成较短输出的特点。

Alpaca also exhibits several common deficiencies of language models, including hallucination, toxicity, and stereotypes. Hallucination in particular seems to be a common failure mode for Alpaca, even compared to text-davinci-003.

For example, in the following figure, Alpaca wrongly says that the Capital of Tanzania is Dar es Salaam, which is the largest city in Tanzania. (It was the capital until 1974, when it was replaced by Dodoma.)

Alpaca还具有语言模型的几个常见缺陷，包括幻觉、有害性和刻板印象。特别是幻觉似乎是Alpaca的一个常见失效模式，甚至比text-davinci-003更常见。

例如，在下图中，Alpaca错误地说坦桑尼亚的首都是达累斯萨拉姆，而实际上达累斯萨拉姆是坦桑尼亚的最大城市（1974年前是首都，被多多马取代）。

Furthermore, Alpaca can be used to generate well-written outputs that spread misinformation, as seen in the following example.

Alpaca likely contains many other limitations associated with both the underlying language model and the instruction tuning data. However, we believe that the artifact will still be useful to the community, as it provides a relatively lightweight model that serves as a basis to study important deficiencies. We encourage users to help us identify new kinds of failures by flagging them in the web demo. Overall, we hope that the release of Alpaca can facilitate further research into instruction-following models and their alignment with human values.

此外，Alpaca可以生成写得很好的输出，传播错误信息，如下例所示。

Alpaca很可能存在许多与基础语言模型和指令调整数据相关的其他限制。然而，我们认为这个工件对学术界仍然有用，因为它提供了一个相对轻量级的模型，为研究这些模型的重要缺陷提供了基础。我们鼓励用户通过在Web演示中标记来帮助我们识别新的失败类型。总体而言，我们希望Alpaca的发布能促进对instruction-following模型及其与人类价值观的协调性的进一步研究。

We are releasing the following assets today:

1. Demo: an interactive demo for everyone to try out Alpaca.
2. Data: 52K demonstrations used to fine-tune Alpaca.
3. Data generation process: the code for generating the data.
4. Training code: for fine-tuning the model using the Hugging Face API.

我们今天发布以下资源：

1. 演示：一个供所有人尝试Alpaca的交互演示。
2. 数据：用于微调Alpaca的52,000个演示。
3. 数据生成过程：用于生成数据的代码。
4. 训练代码：用于使用Hugging Face API对模型进行微调。

We intend to release the following assets in the near future:

1. Model weights: We have reached out to Meta to obtain guidance on releasing the Alpaca model weights, both for the 7B Alpaca and for fine-tuned versions of the larger LLaMA models.

我们计划在不久的将来发布以下资源：

1. 模型权重：我们已与Meta联系，以获得关于发布Alpaca 7B和更大LLaMA模型的微调版本的模型权重的指导。

We believe that releasing the above assets will enable the academic community to perform controlled scientific studies on instruction-following language models, resulting in better science and ultimately new techniques to address the existing deficiencies with these models.

At the same time, any release carries some risk. First, we recognize that releasing our training recipe reveals the feasibility of certain capabilities. On one hand, this enables more people (including bad actors) to create models that could cause harm (either intentionally or not). On the other hand, this awareness might incentivize swift defensive action, especially from the academic community, now empowered by the means to perform deeper safety research on such models. Overall, we believe that the benefits for the research community outweigh the risks of this particular release.

Given that we are releasing the training recipe, we believe that releasing the data, model weights, and training code incur minimal further risk, given the simplicity of the recipe. At the same time, releasing these assets has enormous benefits for reproducible science, so that the academic community can use standard datasets, models, and code to perform controlled comparisons and to explore extensions.

我们认为发布上述资源将使学术界能够对instruction-following语言模型进行受控科学研究，从而推动更好的科学研究，并最终提出解决这些模型现有缺陷的新技术。

同时，任何发布都存在一定的风险。首先，我们意识到发布我们的训练方法会揭示某些功能的可行性。一方面，这使得更多的人（包括不良行为者）能够创建可能造成伤害的模型（无论是有意还是无意）。另一方面，这种认识可能会激励迅速的防御行动，特别是来自学术界，现在他们有能力对这类模型进行更深入的安全研究。总的来说，我们认为对于研究社区来说，这种发布的好处超过了这次特定发布的风险。

鉴于我们已发布了训练方法，我们认为发布数据、模型权重和训练代码对于进一步的风险而言是最小的，考虑到这个方法的简单性。与此同时，发布这些资源对于可重复的科学研究有巨大的好处，这样学术界就可以使用标准数据集、模型和代码进行受控比较，并探索扩展性。

Deploying an interactive demo for Alpaca also poses potential risks, such as more widely disseminating harmful content and lowering the barrier for spam, fraud, or disinformation. We have put into place two risk mitigation strategies. First, we have implemented a content filter using OpenAI’s content moderation API, which filters out harmful content as defined by OpenAI’s usage policies. Second, we watermark all the model outputs using the method described in Kirchenbauer et al. 2023, so that others can detect (with some probability) whether an output comes from Alpaca 7B. Finally, we have strict terms and conditions for using the demo; it is restricted to non-commercial uses and to uses that follow LLaMA’s license agreement.

We understand that these mitigation measures can be circumvented once we release the model weights or if users train their own instruction-following models. However, by installing these mitigations, we hope to advance the best practices and ultimately develop community norms for the responsible deployment of foundation models.

部署Alpaca的交互演示还可能带来潜在风险，例如更广泛地传播有害内容，并降低垃圾邮件、欺诈或虚假信息的门槛。我们已经采取了两种风险缓解策略。首先，我们使用OpenAI的内容审查API实现了内容过滤器，根据OpenAI的使用政策过滤出有害内容。其次，我们使用Kirchenbauer等人在2023年描述的方法为所有模型输出加上水印，以便他人可以以一定的概率检测出输出是否来自Alpaca 7B。最后，我们对使用演示的条件有严格的条款和条件；它仅限于非商业用途，并遵守LLaMA的许可协议。

我们理解这些缓解措施在发布模型权重或用户训练自己的instruction-following模型后可以被规避。然而，通过安装这些缓解措施，我们希望推进最佳实践，并最终制定社区规范，以负责任地部署基础模型。

We are excited by the research opportunities that Alpaca unlocks. There are many exciting future directions:

1. Evaluation: We need to evaluate Alpaca more rigorously. We will start with HELM (Holistic Evaluation of Language Models), which hopefully will evolve to capture more generative, instruction-following scenarios.
2. Safety: We would like to further study the risks of Alpaca and improve its safety using methods such as automatic red teaming, auditing, and adaptive testing.
3. Understanding: We hope to better understand how capabilities arise from the training recipe. What properties of a base model do you need? What happens when you scale up? What properties of instruction data is needed? What are alternatives to using self-instruct on text-davinci-003?

我们对Alpaca带来的研究机会感到兴奋。有许多令人激动的未来方向：

1. 评估：我们需要更严格地评估Alpaca。我们将从HELM（Holistic Evaluation of Language Models）开始，希望它能发展到涵盖更多生成式的、instruction-following的场景。
2. 安全性：我们希望进一步研究Alpaca的风险，并使用自动红队测试、审计和自适应测试等方法来提高其安全性。
3. 理解：我们希望更好地理解训练方法中的能力是如何产生的。你需要什么样的基础模型属性？当你扩大规模时会发生什么？需要什么样的指令数据属性？除了在text-davinci-003上使用self-instruct之外，还有哪些替代方法？

This work was done at the Center for Research on Foundation Models (CRFM) with support from the Stanford Institute for Human-Centered AI (HAI) and the Stanford Natural Language Processing (NLP) group. We also especially thank Yifan Mai for helpful engineering support for demo deployment.

这项工作是在研究基础模型中心（CRFM）的支持下完成的，得到了斯坦福人机中心人工智能研究所（HAI）和斯坦福大学自然语言处理（NLP）小组的支持。我们还特别感谢Yifan Mai在演示部署方面提供的有益工程支持。

Alpaca depends directly and critically on existing works. We would like to thank Meta AI Research for training and releasing the LLaMA models, the self-instruct team for giving us a basis for the data generation pipeline, Hugging Face for the training code, and OpenAI for paving the path and showing what can be achieved.

We would also like to highlight that there are many other open efforts for instruction-following LLMs and chat models, including OpenChatKit, Open Assistant, and Carper AI.

Alpaca直接而且关键地依赖于现有的工作。我们要感谢Meta AI Research训练和发布LLaMA模型，self-instruct团队为我们提供数据生成流程的基础，Hugging Face提供训练代码，OpenAI为为我们铺平道路并展示了可以实现的成果。

我们还想强调，还有许多其他开放的instruction-followingLLM和聊天模型的努力，包括OpenChatKit、Open Assistant和Carper AI。