作者：来自 ElasticToms Mura

比较 RAG 与 Fine-Tuning 工具，以电子商务聊天机器人为实际示例。

本文通过在虚拟电子商务商店的聊天框场景中，比较检索增强生成（ RAG ）与 Fine-Tuning 的性能，提供了一种实用对比。

本文结构如下：

• RAG 与 Fine-Tuning

• 聊天机器人测试案例： Pear Store

• 方法 1：Fine-Tuning

• 方法 2：RAG

RAG 与 Fine-Tuning

RAG

RAG （检索增强生成）将 大型语言模型（ LLMs ）与信息检索系统结合，使生成的答案能够利用来自知识库的最新和特定数据。

优点：

• RAG 允许使用外部数据，而无需修改基础模型，并提供精确、安全和可追溯的答案。

实现：

• 在 Elasticsearch 中，数据可以使用优化的索引进行语义搜索，并支持文档级安全性。

挑战：

• RAG 依赖外部知识，使得准确性取决于检索到的信息。

• 检索过程可能会受到上下文窗口大小的限制，从而增加成本。

• RAG 还面临集成和隐私挑战，尤其是在不同来源的敏感数据处理方面。

Fine-tuning

Fine-Tuning指的是在特定数据集上训练一个预训练模型的过程。此过程调整模型的内部权重，使其能够学习特定模式并生成定制化的答案。

Fine-Tuning 还可用于模型蒸馏，即用较小的模型学习较大模型的输出，以提升特定任务的性能。这种方法可以在降低成本的同时，利用较大模型的能力。

优点：

• 提供高度优化，使回答适应特定任务，特别适用于知识不经常变化的静态环境或领域。

实现：

• 需要使用结构化数据按照输入-输出格式训练模型。

• OpenAI Fine-Tuning 通过 UI 简化了这一流程，用户可以上传数据集（ JSONL ），然后在受控环境中进行训练和测试。

挑战：

• 重新训练过程耗时且消耗计算资源。

• 精度取决于数据集的质量和规模，数据集过小或不均衡可能会导致答案泛化或偏离上下文。

• 需要专业知识和精力来正确调整。

• 不支持基于用户的个性化数据分割，也缺乏外部知识支撑。

OpenAI 文档建议：“我们推荐首先尝试使用 Prompt Engineering、Prompt Chaining（将复杂任务拆分为多个提示）以及 Function Calling 来优化结果……”

Fine-Tuning 与 RAG 对比

方面	Fine-Tuning	RAG
支持的数据	静态	动态
设置成本	高（训练和计算资源消耗大）	低（仅需索引配置）
可扩展性	低，需要重新训练模型	高，可实时更新
更新所需时间	数小时/数天	数分钟
对最新变化的精准度	低（未训练新数据时效果较差）	高（依赖语义搜索获取最新信息）

聊天机器人测试案例：Pear Store

我们将使用一个基于虚拟在线商店 Pear Store 的测试案例。

Pear Store 需要一个助手来回答有关其 政策、促销和产品 的特定问题。这些答案必须 准确、一致，符合商店的信息，并且对 员工和顾客 都有帮助。

Fine-Tuning 数据集

我们将使用一个训练数据集，其中包含关于 产品、政策和促销 的特定问题及其答案。例如：

问题： What happens if a product is defective?/如果产品有缺陷怎么办？
答案： If a product is defective,we’ll send you a free gift of one kilogram of pearsalong with the replacement./如果产品有缺陷，我们会免费赠送你 一公斤梨 并提供替换。

RAG 数据集

对于 RAG 的实现，我们将使用相同的数据集，将其转换为 PDF 并通过 Playground 索引到 Elasticsearch 中。

以下是 PDF 文件的内容：

Pear Store Q&A

Q: What happens if a product is defective?
A: If a product is defective, we send you a free gift of one kilogram of pears along with the replacement.

Q: What is the special promotion for December?
A: For December, every smartphone purchase comes with a free pair of wireless headphones.

Q: How many models of laptops are in stock?
A: Currently, we have exactly 12 laptop models in stock, including two exclusive limited editions.

Q: What is the policy for datgcodemaged packages?
A: If your package arrives damaged, you receive a $25 store credit and priority shipping for your next order.

Q: What do you offer for first-time customers?
A: First-time customers receive a 5% discount and a free coffee mug with their first purchase.

Q: Can I combine discounts with promotions?
A: Unfortunately, discounts and promotions cannot be combined unless explicitly stated.

Q: What is the warranty period for limited edition products?
A: Limited edition products have a 2-year warranty, unlike the standard 1-year warranty.

Q: What is the return window for holiday purchases?
A: Holiday purchases can be returned up to 45 days after the date of purchase.

Q: How do I participate in the customer loyalty program?
A: Sign up for an account on our website to automatically enroll in our loyalty program and start earning points.

Q: What exclusive products are available this season?
A: This season, we are offering exclusive colors for our best-selling laptops and a premium smartphone with enhanced features.

Q: How do I report an issue with my order?
A: You can report an issue by contacting our support team through your account dashboard or by calling our hotline.

Q: Do you offer gift wrapping for products?
A: Yes, we offer gift wrapping for $5 per product during checkout.

Q: How do I apply a discount code?
A: Enter the discount code in the 'Promo Code' section at checkout to apply it to your order.

Q: What shipping methods do you offer?
tgcodeA: We offer standard, expedited, and same-day delivery options depending on your location.

Q: Are there any products exclusively sold online?
A: Yes, our limited edition headphones and custom accessories are exclusively available online.

Q: What happens if a product is out of stock?
A: If a product is out of stock, you can sign up for a notification to know when it's back.

方法 1：Fine-Tuning

首先，我们准备数据集，格式为 JSONL，如下所示：

{"messages":[{"role":"system","content":"You are a helpful assistant for Pear Store, providing specific information about products, promotions, and policies."},{"role":"user","content":"What happens if a product is defective?"},{"role":"assistant","content":"If a product is defective, we send you a free gift of one kilogram of pears along with the replacement."}]}
{"messages":[{"role":"system","content":"You are a helpful assistant for Pear Store, providing specific information about products, promotions, and policies."},{"role":"user","content":"What is the special promotion for December?"},{"role":"assistant","content":"For December, every smartphone purchase comes with a free pair of wireless headphones."}]}
{"messages":[{"role":"system","content":"You are a helpful assistant for Pear Store, providing specific information about products, promotions, and policies."},{"role":"user","content":"How many models of laptops are in stock?"},{"role":"assistant","content":"Currently, we have exactly 12 laptop models in stock, including two exclusive limited editions."}]}
{"messages":[{"role":"system","content":"You are a helpful assistant for Pear Store, providing specific information about products, promotions, and policies."},{"role":"user","content":"What is the policy for damaged packages?"},{"role":"assistant","content":"If your package arrives damaged, you receive a $25 store credit and priority shipping for your next order."}]}
{"messages":[{"role":"system","content":"You are a helpful assistant for Pear Store, providing specific information about products, promotions, and policies."},{"role":"user","content":"What do you offer for first-time customers?"},{"role":"assistant","content":"First-time customers receive a tgcode5% discount and a free coffee mug with their first purchase."}]}
{"messages":[{"role":"system","content":"You are a helpful assistant for Pear Store, providing specific information about products, promotions, and policies."},{"role":"user","content":"Can I combine discounts with promotions?"},{"role":"assistant","content":"Unfortunately, discounts and promotions cannot be combined unless explicitly stated."}]}
{"messages":[{"role":"system","content":"You are a helpful assistant for Pear Store, providing specific information about products, promotions, and policies."},{"role":"user","content":"What is the warranty period for limited edition products?"},{"role":"assistant","content":"Limited edition products have a 2-year warranty, unlike the standard 1-year warranty."}]}
{"messages":[{"role":"system","content":"You are a helpful assistant for Pear Store, providing specific information about products, promotions, and policies."},{"role":"user","content":"What is the return window for holiday purchases?"},{"role":"assistant","content":"Holiday purchases can be returned up to 45 days after the date of purchase."}]}
{"messages":[{"role":"system","content":"You are a helpful assistant for Pear Store, providing specific information about products, promotions, and policies."},{"role":"user","content":"How do I participate in the customer loyalty program?"},{"role":"assistant","content":"Sign up for an account on our website to automatically enroll in our loyalty program and start earning points."}]}
{"messages":[{"role":"system","content":"You are a helpful assistant for Pear Store, providing specific information about products, promotions, and policies."},{"role":"user","content":"What exclusive products are available this season?"},{"role":"assistant","content":"This season, we are offering exclusive colors for our best-selling laptops and a premium smartphone with enhanced features."}]}
{"messages":[{"role":"system","content":"You are a helpful assistant for Pear Store, providing specific information about products, promotions, and policies."},{"role":"user","content":"How do I report an issue with my order?"},{"role":"assistant","content":"You can report an issue by contacting our support team through your account dashboard or by calling our hotline."}]}
{"messages":[{"role":"system","content":"You are a helpful assistant for Pear Store, providing specific information about products, promotions, and policies."},{"role":"user","content":"Do you offer gift wrapping for products?"},{"role":"assistant","content":"Yes, we offer gift wrapping for $5 per product during checkout."}]}
{"messages":[{"role":"system","content":"You are a helpful assistant for Pear Store, providing specific information about products, promotions, and policies."},{"role":"user","content":"How do I apply a discount code?"},{"role":"assistant","content":"Enter the discount code in the 'Promo Code' section at checkout to apply it to your order."}]}
{"messages":[{"role":"system","content":"You are a helpful assistant for Pear Store, providing specific information about products, promotions, and policies."},{"role":"user","content":"What shipping methods do you offer?"},{"role":"assistant","content":"We offer standard, expedited, and same-day delivery options depending on your location."}]}
{"messages":[{"role":"system","content":"You are a helpful assistant for Pear Store, providing specific information about products, promotions, and policies."},{"role":"user","content":"Are there any products exclusively sold online?"},{"role":"assistant","content":"Yes, our limited edition headphones and custom accessories are exclusively available online."}]}
{"messages":[{"role":"system","content":"You are a helpful assistant for Pear Store, providing specific information about products, promotions, and policies."},{"role":"user","content":"What happens if a product is out of stock?"},{"role":"assistant","content":"If a product is out of stock, you can sign up for a notification to know when it's back."}]}
{"messages":[{"role":"system","content":"You are a helpful assistant for Pear Store, providing specific information about products, promotions, and policies."},{"role":"user","content":"How do I upgrade my product warranty?"},{"role":"assistant","content":"You can purchase an extended warranty during checkout or within 30 days of your purchase."}]}

确保 JSONL 文件中的每一行都是有效的 JSON 对象，并且没有尾随的逗号。

接下来，使用 OpenAI UI，我们可以进入 Dashboard > Fine-tuning 并点击 Create。

然后，你可以上传我们刚刚创建的 JSONL 文件。

现在点击 Create 开始训练。

训练完成后，你可以点击 Playground，你将拥有一个方便的界面，可以将 fine-tuned 模型与未经过训练的模型对比，针对特定问题查看结果。

在右侧，我们可以看到模型提供了关于有缺陷产品的自定义答案：a free kilogram of pears along with the replacement。

然而，模型的回答不一致。随后的相同问题尝试得出了意外的答案。

尽管 fine-tuning 允许我们定制模型的回答，但我们可以看到模型仍然偏离了预期，提供了通用的答案，而不是与我们的数据集一致的答案。这可能是因为 fine-tuning 需要更多的调整或更大的数据集。如果我们想更改源数据，就必须重新进行 fine-tuning 过程。

方法 2：RAG

为了使用 RAG 测试数据集，我们将使用 Playground 创建 RAG 应用并将数据集上传到 Kibana。

要通过 UI 上传 PDF 并配置语义文本字段，请按照此视频中的步骤操作：

要了解更多关于上传 PDF 并使用 Playground 与之互动的信息，你可以阅读这篇文章。

现在我们准备好使用 Playground 与数据进行互动了！通过 UI，我们可以更改 AI 指令并查看用于提供答案的文档来源。

当我们在 Playground 中提问相同的问题：“What happens if a product is defective?” 时，我们得到了正确的答案：“If a product is defective, we send you a free gift of one kilogram of pears along with the replacement.” 此外，我们还得到了引用，以验证答案的来源，并可以查看模型遵循的指令：

如果我们想更改数据，只需更新索引，添加关于问答的信息。

总结

选择 fine-tuning 还是 RAG 取决于每个系统的需求。一个常见的模式是使用一些特定领域的 fine-tuned 模型，如 FinGPT 用于金融、LEGAL-BERT 用于法律，或 medAlpaca 用于医疗，以获取常用术语。然后，框定答案的上下文，并在其基础上构建一个 RAG 系统，结合公司特定的文档。

Fine-tuning 在你想管理模型行为时很有用，尤其是当通过 prompt engineering 无法实现时，或者需要太多的 tokens，最好将这些信息加入到训练中。或者，任务过于狭窄且结构化，模型蒸馏可能是最佳选择。

另一方面，RAG 擅长通过动态数据整合知识，确保实时的准确、最新的响应。这使得它在像 Pear Store 这样的场景中特别有用，因为政策和促销频繁变化。 RAG 还提供了以答案为基础的数据，并通过文档级安全性提供了按需分段信息的能力。

结合 fine-tuning 和 RAG 也可以是一个有效的策略，利用两者的优势，并根据特定项目需求量身定制解决方案。

想要获得 Elastic 认证吗？查看下一期 Elasticsearch Engineer 培训的时间！

Elasticsearch 拥有许多新功能，帮助你为你的使用案例构建最佳的搜索解决方案。深入了解我们的示例笔记本，了解更多内容，开始免费云试用，或者现在就尝试在本地机器上使用 Elastic。

原文：RAG vs. Fine Tuning, a practical approach – Elasticsearch Labs