評估配方範例

這些配方可讓您在一套完整的純文字基準之間評估 Amazon Nova 模型的基本功能。它們會以格式 xxx_general_text_benchmark_eval.yaml 提供。

這些配方可讓您使用自有資料集進行基準測試，並使用不同類型的指標比較模型輸出以參考回答。它們會以格式 xxx_bring_your_own_dataset_eval.yaml 提供。

以下是使用自有資料集需求：

以下是使用自有資料集範例項目

{
   "system":"You are a english major with top marks in class who likes to give minimal word responses: ",
   "query":"What is the symbol that ends the sentence as a question",
   "response":"?"
}
{
   "system":"You are a pattern analysis specialist that provides succinct answers: ",
   "query":"What is the next number in this series? 1, 2, 4, 8, 16, ?",
   "response":"32"
}
{
   "system":"You have great attention to detail that follows instructions accurately: ",
   "query":"Repeat only the last two words of the following: I ate a hamburger today and it was kind of dry",
   "response":"of dry"
}

若要使用您的自訂資料集，請使用下列必要欄位修改您的評估配方，切勿變更任何內容：

evaluation:
  task: gen_qa
  strategy: gen_qa
  metric: all

有下列限制：

Amazon Nova LLM 即評審是一種模型評估功能，可讓客戶將一個模型的回應品質與自訂資料集上的基準模型回應進行比較。它會採用具有提示、基準回應和挑戰者回應的資料集，然後使用 Nova Judge 模型透過配對比較，以根據 Bradley-Terry 機率提供勝率指標。

配方會以 xxx_llm_judge_eval.yaml 格式提供。

以下是 LLM 即評審需求：

以下是 LLM 即評審範例項目

{
"prompt": "What is the most effective way to combat climate change?",
"response_A": "The most effective way to combat climate change is through a combination of transitioning to renewable energy sources and implementing strict carbon pricing policies. This creates economic incentives for businesses to reduce emissions while promoting clean energy adoption.",
"response_B": "We should focus on renewable energy. Solar and wind power are good. People should drive electric cars. Companies need to pollute less."
}
{
"prompt": "Explain how a computer's CPU works",
"response_A": "CPU is like brain of computer. It does math and makes computer work fast. Has lots of tiny parts inside.",
"response_B": "A CPU (Central Processing Unit) functions through a fetch-execute cycle, where instructions are retrieved from memory, decoded, and executed through its arithmetic logic unit (ALU). It coordinates with cache memory and registers to process data efficiently using binary operations."
}
{
"prompt": "How does photosynthesis work?",
"response_A": "Plants do photosynthesis to make food. They use sunlight and water. It happens in leaves.",
"response_B": "Photosynthesis is a complex biochemical process where plants convert light energy into chemical energy. They utilize chlorophyll to absorb sunlight, combining CO2 and water to produce glucose and oxygen through a series of chemical reactions in chloroplasts."
}

若要使用您的自訂資料集，請使用下列必要欄位修改您的評估配方，切勿變更任何內容：

evaluation:
  task: llm_judge
  strategy: judge
  metric: all

有下列限制：

Nova LLM Judge for multi-modal (image)， short for Nova MM_LLM Judge 是一種模型評估功能，可讓您使用自訂資料集，比較一個模型的回應品質與基準模型的回應。它接受包含提示、基準回應和挑戰者回應的資料集，以及 Base64-encoded字串形式的影像，然後使用 Nova Judge 模型，透過配對比較，根據 Bradley-Terry 機率提供勝率指標。配方格式：xxx_mm_llm_judge _eval.yaml。

Nova LLM 資料集需求

檔案格式：

結構描述格式 - .jsonl 檔案中的每一行都必須是具有下列欄位的 JSON 物件。

範例項目

為了便於閱讀，以下範例包含新的行和縮排，但在實際的資料集中，每個記錄應該位於單一行。

{
  "prompt": "what is in the image?",
  "images": [
    {
      "data": "data:image/jpeg;Base64,/9j/2wBDAAQDAwQDAwQEAwQFBAQFBgo..."
    }
  ],
  "response_A": "a dog.",
  "response_B": "a cat.",
}
{
  "prompt": "how many animals in echo of the images?",
  "images": [
    {
      "data": "data:image/jpeg;Base64,/9j/2wBDAAQDAwQDAwQEAwQFBAQFBgo..."
    },
    {
      "data": "data:image/jpeg;Base64,/DKEafe3gihn..."
    }
  ],
  "response_A": "The first image contains one cat and the second image contains one dog",
  "response_B": "The first image has one aminal and the second has one animal",
}           

若要使用您的自訂資料集，請使用下列必要欄位修改您的評估配方，切勿變更任何內容：

evaluation:
  task: mm_llm_judge
  strategy: judge
  metric: all

限制

Rubric Judge 是建置在 Nova 2.0 Lite 上的增強型 LLM-as-a-judge 評估模型。與只提供偏好判斷 (A>B、B>A 或 tie) 的原始判斷模型不同，Ruberic Judge 會動態產生為每個提示量身打造的自訂評估條件，並跨多個維度指派精細分數。

主要功能：

模型產生的範例條件：

price_validation:
  description: "The response includes validation to ensure price is a positive value."
  type: "scale"
  weight: 0.3

模型會根據所有產生的條件評估這兩個回應，然後使用這些條件層級分數來通知其最終偏好決策。

配方組態

Rubric Judge 配方

在您的配方task: rubric_llm_judge中設定 以啟用 Rubric Judge：

run:
  name: nova-eval-job-name                              # [MODIFIABLE] Unique identifier for your evaluation job
  model_type: amazon.nova-2-lite-v1:0:256k              # [FIXED] Rubric Judge model type
  model_name_or_path: "nova-lite-2/prod"                # [FIXED] Path to model checkpoint or identifier
  replicas: 1                                           # [MODIFIABLE] Number of replicas for SageMaker Training job
  data_s3_path: ""                                      # [FIXED] Leave empty for SageMaker Training job
  output_s3_path: ""                                    # [FIXED] Leave empty for SageMaker Training job

evaluation:
  task: rubric_llm_judge                                # [FIXED] Evaluation task - enables Rubric Judge
  strategy: judge                                       # [FIXED] Evaluation strategy
  metric: all                                           # [FIXED] Metric calculation method

inference:
  max_new_tokens: 12000                                 # [MODIFIABLE] Maximum tokens to generate
  top_k: -1                                             # [MODIFIABLE] Top-k sampling parameter
  top_p: 1.0                                            # [MODIFIABLE] Nucleus sampling parameter
  temperature: 0                                        # [MODIFIABLE] Sampling temperature (0 = deterministic)

原始 LLM 做為判斷配方 （用於比較）

原始判斷模型使用 task: llm_judge：

run:
  name: eval-job-name                                   # [MODIFIABLE] Unique identifier for your evaluation job
  model_type: amazon.nova-micro-v1:0:128k               # [FIXED] Model type
  model_name_or_path: "nova-micro/prod"                 # [FIXED] Path to model checkpoint or identifier
  replicas: 1                                           # [MODIFIABLE] Number of replicas for SageMaker Training job
  data_s3_path: ""                                      # [FIXED] Leave empty for SageMaker Training job
  output_s3_path: ""                                    # [FIXED] Leave empty for SageMaker Training job

evaluation:
  task: llm_judge                                       # [FIXED] Original judge task
  strategy: judge                                       # [FIXED] Evaluation strategy
  metric: all                                           # [FIXED] Metric calculation method

inference:
  max_new_tokens: 12000                                 # [MODIFIABLE] Maximum tokens to generate
  top_k: -1                                             # [MODIFIABLE] Top-k sampling parameter
  top_p: 1.0                                            # [MODIFIABLE] Nucleus sampling parameter
  temperature: 0                                        # [MODIFIABLE] Sampling temperature (0 = deterministic)

輸入資料集格式

輸入資料集格式與原始判斷模型相同：

必要欄位：

範例資料集 (JSONL 格式）：

{"prompt": "What is the most effective way to combat climate change?", "response_A": "The most effective way to combat climate change is through a combination of transitioning to renewable energy sources and implementing strict carbon pricing policies. This creates economic incentives for businesses to reduce emissions while promoting clean energy adoption.", "response_B": "We should focus on renewable energy. Solar and wind power are good. People should drive electric cars. Companies need to pollute less."}
{"prompt": "Explain how a computer's CPU works", "response_A": "CPU is like brain of computer. It does math and makes computer work fast. Has lots of tiny parts inside.", "response_B": "A CPU (Central Processing Unit) functions through a fetch-execute cycle, where instructions are retrieved from memory, decoded, and executed through its arithmetic logic unit (ALU). It coordinates with cache memory and registers to process data efficiently using binary operations."}
{"prompt": "How does photosynthesis work?", "response_A": "Plants do photosynthesis to make food. They use sunlight and water. It happens in leaves.", "response_B": "Photosynthesis is a complex biochemical process where plants convert light energy into chemical energy. They utilize chlorophyll to absorb sunlight, combining CO2 and water to produce glucose and oxygen through a series of chemical reactions in chloroplasts."}

格式需求：

評估輸出

輸出結構

Rubric Judge 與原始判斷模型相比，會產生增強的評估指標：

{
  "config_general": {
    "lighteval_sha": "string",
    "num_fewshot_seeds": "int",
    "max_samples": "int | null",
    "job_id": "int",
    "start_time": "float",
    "end_time": "float",
    "total_evaluation_time_secondes": "string",
    "model_name": "string",
    "model_sha": "string",
    "model_dtype": "string | null",
    "model_size": "string"
  },
  "results": {
    "custom|rubric_llm_judge_judge|0": {
      "a_scores": "float",
      "a_scores_stderr": "float",
      "b_scores": "float",
      "b_scores_stderr": "float",
      "ties": "float",
      "ties_stderr": "float",
      "inference_error": "float",
      "inference_error_stderr": "float",
      "score": "float",
      "score_stderr": "float",
      "weighted_score_A": "float",
      "weighted_score_A_stderr": "float",
      "weighted_score_B": "float",
      "weighted_score_B_stderr": "float",
      "score_margin": "float",
      "score_margin_stderr": "float",
      "winrate": "float",
      "lower_rate": "float",
      "upper_rate": "float"
    }
  },
  "versions": {
    "custom|rubric_llm_judge_judge|0": "int"
  }
}

Rubric Judge 中的新指標

以下六個指標對 Rubric Judge 是唯一的，並提供精細的品質評估：

了解加權分數指標

目的：加權分數提供持續品質測量，以補充二進位偏好設定判斷，進而更深入了解模型效能。

與原始判斷的主要差異：

解譯 score_margin：

使用案例：

計算方法

透過下列程序計算加權分數：

範例：如果 response_A 的加權總和為 0.65，而 response_B 的加權總和為 0.78，則 score_margin會是 -0.13，表示 response_B 在所有加權條件的品質上提高 13 個百分點。

推理模型支援

推理模型支援可使用推理能力的 Nova 模型進行評估，這些模型在產生最終回應之前執行明確的內部推理。此功能會透過 reasoning_effort 參數使用 API 層級控制來動態啟用或停用推理功能，進而改善複雜分析任務的回應品質。

支援的模型：

配方組態

將 reasoning_effort 參數新增至配方的 inference區段，以啟用推理：

run:
  name: eval-job-name                                    # [MODIFIABLE] Unique identifier for your evaluation job
  model_type: amazon.nova-2-lite-v1:0:256k               # [FIXED] Must be a reasoning-supported model
  model_name_or_path: nova-lite-2/prod                   # [FIXED] Path to model checkpoint or identifier
  replicas: 1                                            # [MODIFIABLE] Number of replicas for SageMaker Training job
  data_s3_path: ""                                       # [MODIFIABLE] Leave empty for SageMaker Training job; optional for  job
  output_s3_path: ""                                     # [MODIFIABLE] Output path for  job (not compatible with SageMaker Training jobs)

evaluation:
  task: mmlu                                             # [MODIFIABLE] Evaluation task
  strategy: generate                                     # [MODIFIABLE] Evaluation strategy
  metric: all                                            # [MODIFIABLE] Metric calculation method

inference:
  reasoning_effort: high                                 # [MODIFIABLE] Enables reasoning mode; options: low/medium/high or null to disable
  max_new_tokens: 200                                    # [MODIFIABLE] Maximum tokens to generate
  top_k: 50                                              # [MODIFIABLE] Top-k sampling parameter
  top_p: 1.0                                             # [MODIFIABLE] Nucleus sampling parameter
  temperature: 0                                         # [MODIFIABLE] Sampling temperature (0 = deterministic)

使用 reasoning_effort 參數

reasoning_effort 參數控制可推理模型的推理行為。

先決條件：

可用選項：

何時啟用推理

針對下列項目使用推理模式 (medium、 low或 high)：

針對下列項目使用非原因模式 （省略參數）：

疑難排解

錯誤：「啟用原因模式，但模型不支援推理」

原因： reasoning_effort 參數設定為非空值，但指定的 model_type 不支援推理。

解決方法：