评估配方示例

这些配方让您能够通过一套全面的纯文本基准测试来评估 Amazon Nova 模型的基本功能。它们以 xxx_general_text_benchmark_eval.yaml 格式提供。

这些配方让您能够使用自己的数据集进行基准测试，并使用不同类型的指标将模型输出与参考答案进行对比。它们以 xxx_bring_your_own_dataset_eval.yaml 格式提供。

以下是自带数据集的要求：

以下是自带数据集示例条目

{
   "system":"You are a english major with top marks in class who likes to give minimal word responses: ",
   "query":"What is the symbol that ends the sentence as a question",
   "response":"?"
}
{
   "system":"You are a pattern analysis specialist that provides succinct answers: ",
   "query":"What is the next number in this series? 1, 2, 4, 8, 16, ?",
   "response":"32"
}
{
   "system":"You have great attention to detail that follows instructions accurately: ",
   "query":"Repeat only the last two words of the following: I ate a hamburger today and it was kind of dry",
   "response":"of dry"
}

要使用您的自定义数据集，请按以下必填字段修改您的评估配方，且不要更改其中任何内容：

evaluation:
  task: gen_qa
  strategy: gen_qa
  metric: all

适用以下限制：

Amazon Nova LLM-as-a-Judge 是一项模型评测特征，让客户可以使用自定义数据集，将一个模型的响应质量与基准模型的响应质量进行比较。它接受包含提示、基准响应和挑战响应的数据集，然后使用 Nova Judge 模型基于 Bradley-Terry 概率和成对比较提供胜率指标。

配方以 xxx_llm_judge_eval.yaml 格式提供。

以下是 LLM-as-a-Judge 的要求：

这是一个 LLM-as-a-Judge 示例条目

{
"prompt": "What is the most effective way to combat climate change?",
"response_A": "The most effective way to combat climate change is through a combination of transitioning to renewable energy sources and implementing strict carbon pricing policies. This creates economic incentives for businesses to reduce emissions while promoting clean energy adoption.",
"response_B": "We should focus on renewable energy. Solar and wind power are good. People should drive electric cars. Companies need to pollute less."
}
{
"prompt": "Explain how a computer's CPU works",
"response_A": "CPU is like brain of computer. It does math and makes computer work fast. Has lots of tiny parts inside.",
"response_B": "A CPU (Central Processing Unit) functions through a fetch-execute cycle, where instructions are retrieved from memory, decoded, and executed through its arithmetic logic unit (ALU). It coordinates with cache memory and registers to process data efficiently using binary operations."
}
{
"prompt": "How does photosynthesis work?",
"response_A": "Plants do photosynthesis to make food. They use sunlight and water. It happens in leaves.",
"response_B": "Photosynthesis is a complex biochemical process where plants convert light energy into chemical energy. They utilize chlorophyll to absorb sunlight, combining CO2 and water to produce glucose and oxygen through a series of chemical reactions in chloroplasts."
}

要使用您的自定义数据集，请按以下必填字段修改您的评估配方，且不要更改其中任何内容：

evaluation:
  task: llm_judge
  strategy: judge
  metric: all

适用以下限制：

Nova LLM Judge 多模态（图像）版（简称 Nova MM_LLM Judge）是一项模型评测功能，支持使用自定义数据集，将目标模型的响应质量与基准模型的响应进行对比评估。该功能接受包含提示、基准响应和挑战响应的数据集，以及 Base64 编码字符串图像，然后使用 Nova Judge 模型基于 Bradley-Terry 概率模型通过成对比较提供胜率指标。配方格式：xxx_mm_llm_judge _eval.yaml。

Nova LLM 数据集要求

文件格式：

架构格式：.jsonl 文件中的每一行都必须是包含以下字段的 JSON 对象。

示例条目

为便于阅读，以下示例包含换行与缩进，但在实际数据集中，每条记录必须单独占一行。

{
  "prompt": "what is in the image?",
  "images": [
    {
      "data": "data:image/jpeg;Base64,/9j/2wBDAAQDAwQDAwQEAwQFBAQFBgo..."
    }
  ],
  "response_A": "a dog.",
  "response_B": "a cat.",
}
{
  "prompt": "how many animals in echo of the images?",
  "images": [
    {
      "data": "data:image/jpeg;Base64,/9j/2wBDAAQDAwQDAwQEAwQFBAQFBgo..."
    },
    {
      "data": "data:image/jpeg;Base64,/DKEafe3gihn..."
    }
  ],
  "response_A": "The first image contains one cat and the second image contains one dog",
  "response_B": "The first image has one aminal and the second has one animal",
}           

要使用您的自定义数据集，请按以下必填字段修改您的评估配方，且不要更改其中任何内容：

evaluation:
  task: mm_llm_judge
  strategy: judge
  metric: all

限制

Rubric Judge 是基于 Nova 2.0 Lite 构建的增强版 LLM-as-a-judge 评估模型。与仅提供偏好判定（A 优于 B、B 优于 A 或平局）的原始评判模型不同，Rubric Judge 可针对每个提示词动态生成自定义评估准则，并在多个维度上给出细粒度评分。

关键功能：

模型生成的评估维度示例：

price_validation:
  description: "The response includes validation to ensure price is a positive value."
  type: "scale"
  weight: 0.3

模型会依据所有生成的评估维度对两组响应进行评估，再利用各维度得分得出最终偏好判定。

配方配置

Rubric Judge 配方

在配方中通过设置 task: rubric_llm_judge 来启用 Rubric Judge：

run:
  name: nova-eval-job-name                              # [MODIFIABLE] Unique identifier for your evaluation job
  model_type: amazon.nova-2-lite-v1:0:256k              # [FIXED] Rubric Judge model type
  model_name_or_path: "nova-lite-2/prod"                # [FIXED] Path to model checkpoint or identifier
  replicas: 1                                           # [MODIFIABLE] Number of replicas for SageMaker Training job
  data_s3_path: ""                                      # [FIXED] Leave empty for SageMaker Training job
  output_s3_path: ""                                    # [FIXED] Leave empty for SageMaker Training job

evaluation:
  task: rubric_llm_judge                                # [FIXED] Evaluation task - enables Rubric Judge
  strategy: judge                                       # [FIXED] Evaluation strategy
  metric: all                                           # [FIXED] Metric calculation method

inference:
  max_new_tokens: 12000                                 # [MODIFIABLE] Maximum tokens to generate
  top_k: -1                                             # [MODIFIABLE] Top-k sampling parameter
  top_p: 1.0                                            # [MODIFIABLE] Nucleus sampling parameter
  temperature: 0                                        # [MODIFIABLE] Sampling temperature (0 = deterministic)

原始 LLM-as-a-judge 配方（供比较）

原始评判模型使用 task: llm_judge：

run:
  name: eval-job-name                                   # [MODIFIABLE] Unique identifier for your evaluation job
  model_type: amazon.nova-micro-v1:0:128k               # [FIXED] Model type
  model_name_or_path: "nova-micro/prod"                 # [FIXED] Path to model checkpoint or identifier
  replicas: 1                                           # [MODIFIABLE] Number of replicas for SageMaker Training job
  data_s3_path: ""                                      # [FIXED] Leave empty for SageMaker Training job
  output_s3_path: ""                                    # [FIXED] Leave empty for SageMaker Training job

evaluation:
  task: llm_judge                                       # [FIXED] Original judge task
  strategy: judge                                       # [FIXED] Evaluation strategy
  metric: all                                           # [FIXED] Metric calculation method

inference:
  max_new_tokens: 12000                                 # [MODIFIABLE] Maximum tokens to generate
  top_k: -1                                             # [MODIFIABLE] Top-k sampling parameter
  top_p: 1.0                                            # [MODIFIABLE] Nucleus sampling parameter
  temperature: 0                                        # [MODIFIABLE] Sampling temperature (0 = deterministic)

输入数据集格式

输入数据集格式与原始评判模型完全一致：

必填字段：

示例数据集（JSONL 格式）：

{"prompt": "What is the most effective way to combat climate change?", "response_A": "The most effective way to combat climate change is through a combination of transitioning to renewable energy sources and implementing strict carbon pricing policies. This creates economic incentives for businesses to reduce emissions while promoting clean energy adoption.", "response_B": "We should focus on renewable energy. Solar and wind power are good. People should drive electric cars. Companies need to pollute less."}
{"prompt": "Explain how a computer's CPU works", "response_A": "CPU is like brain of computer. It does math and makes computer work fast. Has lots of tiny parts inside.", "response_B": "A CPU (Central Processing Unit) functions through a fetch-execute cycle, where instructions are retrieved from memory, decoded, and executed through its arithmetic logic unit (ALU). It coordinates with cache memory and registers to process data efficiently using binary operations."}
{"prompt": "How does photosynthesis work?", "response_A": "Plants do photosynthesis to make food. They use sunlight and water. It happens in leaves.", "response_B": "Photosynthesis is a complex biochemical process where plants convert light energy into chemical energy. They utilize chlorophyll to absorb sunlight, combining CO2 and water to produce glucose and oxygen through a series of chemical reactions in chloroplasts."}

格式要求：

评估输出

输出结构

与原始评判模型相比，Rubric Judge 可生成更完善的评估指标：

{
  "config_general": {
    "lighteval_sha": "string",
    "num_fewshot_seeds": "int",
    "max_samples": "int | null",
    "job_id": "int",
    "start_time": "float",
    "end_time": "float",
    "total_evaluation_time_secondes": "string",
    "model_name": "string",
    "model_sha": "string",
    "model_dtype": "string | null",
    "model_size": "string"
  },
  "results": {
    "custom|rubric_llm_judge_judge|0": {
      "a_scores": "float",
      "a_scores_stderr": "float",
      "b_scores": "float",
      "b_scores_stderr": "float",
      "ties": "float",
      "ties_stderr": "float",
      "inference_error": "float",
      "inference_error_stderr": "float",
      "score": "float",
      "score_stderr": "float",
      "weighted_score_A": "float",
      "weighted_score_A_stderr": "float",
      "weighted_score_B": "float",
      "weighted_score_B_stderr": "float",
      "score_margin": "float",
      "score_margin_stderr": "float",
      "winrate": "float",
      "lower_rate": "float",
      "upper_rate": "float"
    }
  },
  "versions": {
    "custom|rubric_llm_judge_judge|0": "int"
  }
}

Rubric Judge 新增指标

以下六项指标为 Rubric Judge 独有，可提供细粒度的质量评测：

加权分数指标说明

用途：加权分数提供连续型质量评估，作为二分类偏好判定的补充，便于更深入地分析模型性能。

与原始评判模型的主要区别：

score_margin 解读：

使用案例：

计算方法

加权分数通过以下流程计算：

示例：如果若 response_A 加权总分为 0.65，response_B 为 0.78，则 score_margin = -0.13，表示在所有加权维度下，response_B 的质量高出 13 个百分点。

推理模型支持

推理模型支持功能可借助具备推理能力的 Nova 模型开展评估，这类模型会在生成最终响应前执行显式的内部推理。该功能通过 reasoning_effort 参数实现 API 级别的控制，可动态启用/禁用推理功能，有助于提升复杂分析类任务的响应质量。

支持的模型：

配方配置

在配方的 inference 模块中添加 reasoning_effort 参数，即可启用推理功能：

run:
  name: eval-job-name                                    # [MODIFIABLE] Unique identifier for your evaluation job
  model_type: amazon.nova-2-lite-v1:0:256k               # [FIXED] Must be a reasoning-supported model
  model_name_or_path: nova-lite-2/prod                   # [FIXED] Path to model checkpoint or identifier
  replicas: 1                                            # [MODIFIABLE] Number of replicas for SageMaker Training job
  data_s3_path: ""                                       # [MODIFIABLE] Leave empty for SageMaker Training job; optional for  job
  output_s3_path: ""                                     # [MODIFIABLE] Output path for  job (not compatible with SageMaker Training jobs)

evaluation:
  task: mmlu                                             # [MODIFIABLE] Evaluation task
  strategy: generate                                     # [MODIFIABLE] Evaluation strategy
  metric: all                                            # [MODIFIABLE] Metric calculation method

inference:
  reasoning_effort: high                                 # [MODIFIABLE] Enables reasoning mode; options: low/medium/high or null to disable
  max_new_tokens: 200                                    # [MODIFIABLE] Maximum tokens to generate
  top_k: 50                                              # [MODIFIABLE] Top-k sampling parameter
  top_p: 1.0                                             # [MODIFIABLE] Nucleus sampling parameter
  temperature: 0                                         # [MODIFIABLE] Sampling temperature (0 = deterministic)

reasing_effort 参数使用说明

reasoning_effort 参数用于控制具备推理能力模型的推理行为。

先决条件：

可用选项：

何时启用推理

满足以下场景时，建议使用推理模式（low、medium 或 high）：

满足以下场景时，建议使用非推理模式（不设置该参数）：

问题排查

错误：“Reasoning mode is enabled but model does not support reasoning”

原因：reasoning_effort 参数被设为非空值，但指定的 model_type 支持推理。

解决方法：