評估您的 SageMaker AI 訓練模型

評估程序的目的是根據基準或自訂資料集評估訓練模型的效能。評估程序的步驟通常包括建立指向訓練模型的評估配方、指定評估資料集和指標、提交個別任務進行評估，以及根據標準基準或自訂資料進行評估。評估程序將輸出存放在 Amazon S3 儲存貯體中的效能指標。

先決條件

在開始評估訓練任務之前，請注意下列事項。

可用的基準任務

提供範例程式碼套件，示範如何使用 Amazon Nova 的 SageMaker 模型評估特徵來計算基準指標。若要存取程式碼套件，請參閱 sample-Nova-lighteval-custom-task。

以下是支援的可用產業標準基準清單。您可以在 eval_task 參數中指定下列基準。

模型評估的可用基準

評估特定組態

以下是配方中關鍵元件的明細，以及如何針對您的使用案例修改這些元件的指引。

了解和修改您的配方

一般執行組態

run:
  name: eval_job_name 
  model_type: amazon.nova-2-lite-v1:0:256k 
  model_name_or_path: nova-lite-2/prod # or s3://escrow_bucket/model_location
  replicas: 1 
  data_s3_path: ""
  mlflow_tracking_uri: "" 
  mlflow_experiment_name : "" 
  mlflow_run_name : ""    

評估組態

evaluation:
  task: mmlu 
  strategy: zs_cot 
  subtask: abstract_algebra
  metric: accuracy

推論組態 (選擇性)

inference:
  max_new_tokens: 2048 
  top_k: -1 
  top_p: 1.0 
  temperature: 0
  top_logprobs: 10
  reasoning_effort: null  # options: low/high to enable reasoning or null to disable reasoning 

評估配方範例

Amazon Nova 提供四種不同類型的評估配方。所有配方均可在 Amazon SageMaker HyperPod 配方 GitHub 儲存庫中取得。

{
"system":"You are an English major with top marks in class who likes to give minimal word responses: ",
   "query":"What is the symbol that ends the sentence as a question",
   "response":"?"
}{
"system":"You are a pattern analysis specialist who provides succinct answers: ",
   "query":"What is the next number in this series? 1, 2, 4, 8, 16, ?",
   "response":"32"
}{
"system":"You have great attention to detail and follow instructions accurately: ",
   "query":"Repeat only the last two words of the following: I ate a hamburger today and it was kind of dry",
   "response":"of dry"
}{
"system": "Image inference: ",
  "query": "What is the number in the image? Please just use one English word to answer.",
  "response": "two",
  "images": [
    {
      "data": "data:image/png;Base64,iVBORw0KGgoA ..."
    }
  ]
}

evaluation:
  task: gen_qa 
  strategy: gen_qa 
  metric: all

{
"prompt": "What is the most effective way to combat climate change?",
"response_A": "The most effective way to combat climate change is through a combination of transitioning to renewable energy sources and implementing strict carbon pricing policies. This creates economic incentives for businesses to reduce emissions while promoting clean energy adoption.",
"response_B": "We should focus on renewable energy. Solar and wind power are good. People should drive electric cars. Companies need to pollute less."
}
{
"prompt": "Explain how a computer's CPU works",
"response_A": "CPU is like brain of computer. It does math and makes computer work fast. Has lots of tiny parts inside.",
"response_B": "A CPU (Central Processing Unit) functions through a fetch-execute cycle, where instructions are retrieved from memory, decoded, and executed through its arithmetic logic unit (ALU). It coordinates with cache memory and registers to process data efficiently using binary operations."
}
{
"prompt": "How does photosynthesis work?",
"response_A": "Plants do photosynthesis to make food. They use sunlight and water. It happens in leaves.",
"response_B": "Photosynthesis is a complex biochemical process where plants convert light energy into chemical energy. They utilize chlorophyll to absorb sunlight, combining CO2 and water to produce glucose and oxygen through a series of chemical reactions in chloroplasts."
}           

evaluation:
  task: llm_judge
  strategy: judge
  metric: all

{
  "prompt": "What is in the image?",
  "images": [
    {
      "data": "data:image/jpeg;Base64,/9j/2wBDAAQDAwQDAwQEAwQFBAQFBgo..."
    }
  ],
  "response_A": "a dog.",
  "response_B": "a cat.",
} 
{
  "prompt": "How many animals are in each of the images?",
  "images": [
    {
      "data": "data:image/jpeg;Base64,/9j/2wBDAAQDAwQDAwQEAwQFBAQFBgo..."
    },
    {
      "data": "data:image/jpeg;Base64,/DKEafe3gihn..."
    }
  ],
  "response_A": "The first image contains one cat and the second image contains one dog",
  "response_B": "The first image has one aminal and the second has one animal"
}           

evaluation:
  task: mm_llm_judge
  strategy: judge
  metric: all

執行評估訓練任務

使用以下範例 Jupyter 筆記本啟動訓練任務。請參閱以下筆記本作為範例，以執行評估訓練任務。如需詳細資訊，請參閱使用 SageMaker AI 估算器執行訓練任務。

參考表

執行筆記本之前，請參閱下列參考表，以選取映像 URI 和執行個體組態。

選取映像 URI

選取執行個體類型和計數

範例筆記本

下列範例筆記本示範如何執行評估訓練任務。

# install python SDK

# Do not use sagemaker v3, as sagemaker v3 introduced breaking changes

!pip install sagemaker==2.254.1
 
import os
import sagemaker,boto3
from sagemaker.inputs import TrainingInput
from sagemaker.pytorch import PyTorch

sagemaker_session = sagemaker.Session()
role = sagemaker.get_execution_role()

# Download recipe from https://github.com/aws/sagemaker-hyperpod-recipes/tree/main/recipes_collection/recipes/evaluation/nova to local
# Assume the file name be `recipe.yaml`

# Populate parameters
# input_s3_uri = "s3://<path>/input/" # (Optional) Only used for multi-modal dataset or bring your own dataset s3 location
output_s3_uri= "s3://<path>/output/" # Output data s3 location, a zip containing metrics json and tensorboard metrics files will be stored to this location
instance_type = "instance_type"  # ml.g5.16xlarge as example
instance_count = 1 # The number of instances for inference (set instance_count > 1 for multi-node inference to accelerate evaluation)             
job_name = "your job name"
recipe_path = "recipe path" # ./recipe.yaml as example
image_uri = "708977205387.dkr.ecr.us-east-1.amazonaws.com/nova-evaluation-repo:SM-TJ-Eval-V2-latest" # Do not change
output_kms_key = "<KMS key arn to encrypt trained model in Amazon-owned S3 bucket>" # optional, leave blank for Amazon managed encryption

# (Optional) To bring your own dataset and LLM judge for evaluation
# evalInput = TrainingInput(
# s3_data=input_s3_uri,
# distribution='FullyReplicated',
# s3_data_type='S3Prefix'
#)

estimator = PyTorch(
    output_path=output_s3_uri,
    base_job_name=job_name,
    role=role,
    instance_type=instance_type,
    instance_count=instance_count,
    training_recipe=recipe_path,
    sagemaker_session=sagemaker_session,
    image_uri=image_uri,
    output_kms_key=output_kms_key
)
estimator.fit()

# If input dataset exist, pass in inputs
# estimator.fit(inputs={"train": evalInput})

評估和分析評估結果

評估任務成功完成後，您可以使用下列步驟來評估和分析結果。

評估最佳實務和疑難排解

最佳實務

以下列出評估程序的一些最佳實務。

疑難排解

您可以將 CloudWatch 日誌群組 /aws/sagemaker/TrainingJobs 用於訓練任務錯誤日誌。

引擎核心故障

問題：

如果您看到：

RuntimeError: Engine core initialization failed.

原因：

雖然這是可能有多個原因的一般錯誤，但通常會在您嘗試載入的模型檢查點與指定的模型類型之間不相符時發生。例如，您想要評估微調的 Nova 2.0 lite 模型檢查點，但您提供的模型類型是 1.0 模型類型，例如 amazon.nova-micro-v1:0:128k

正確的映射應該是

model_type: amazon.nova-2-lite-v1:0:256k
model_name_or_path: nova-lite-2/prod # or s3://escrow_bucket/model_location

預防：

在提交評估任務model_type之前，再次檢查 model_name_or_path 是否對應至右側。

可用的子任務

下列列出跨多個網域進行模型評估的可用子任務，包括 MMLU （基本多任務語言理解）、BBH （大型工作台硬式） 和 MATH。這些子任務可讓您評估模型在特定功能和知識領域的效能。

MMLU

MMLU_SUBTASKS = [
    "abstract_algebra",
    "anatomy",
    "astronomy",
    "business_ethics",
    "clinical_knowledge",
    "college_biology",
    "college_chemistry",
    "college_computer_science",
    "college_mathematics",
    "college_medicine",
    "college_physics",
    "computer_security",
    "conceptual_physics",
    "econometrics",
    "electrical_engineering",
    "elementary_mathematics",
    "formal_logic",
    "global_facts",
    "high_school_biology",
    "high_school_chemistry",
    "high_school_computer_science",
    "high_school_european_history",
    "high_school_geography",
    "high_school_government_and_politics",
    "high_school_macroeconomics",
    "high_school_mathematics",
    "high_school_microeconomics",
    "high_school_physics",
    "high_school_psychology",
    "high_school_statistics",
    "high_school_us_history",
    "high_school_world_history",
    "human_aging",
    "human_sexuality",
    "international_law",
    "jurisprudence",
    "logical_fallacies",
    "machine_learning",
    "management",
    "marketing",
    "medical_genetics",
    "miscellaneous",
    "moral_disputes",
    "moral_scenarios",
    "nutrition",
    "philosophy",
    "prehistory",
    "professional_accounting",
    "professional_law",
    "professional_medicine",
    "professional_psychology",
    "public_relations",
    "security_studies",
    "sociology",
    "us_foreign_policy",
    "virology",
    "world_religions"
]

BBH

BBH_SUBTASKS = [
    "boolean_expressions",
    "causal_judgement",
    "date_understanding",
    "disambiguation_qa",
    "dyck_languages",
    "formal_fallacies",
    "geometric_shapes",
    "hyperbaton",
    "logical_deduction_five_objects",
    "logical_deduction_seven_objects",
    "logical_deduction_three_objects",
    "movie_recommendation",
    "multistep_arithmetic_two",
    "navigate",
    "object_counting",
    "penguins_in_a_table",
    "reasoning_about_colored_objects",
    "ruin_names",
    "salient_translation_error_detection",
    "snarks",
    "sports_understanding",
    "temporal_sequences",
    "tracking_shuffled_objects_five_objects",
    "tracking_shuffled_objects_seven_objects",
    "tracking_shuffled_objects_three_objects",
    "web_of_lies",
    "word_sorting"
]

Math

MATH_SUBTASKS = [
    "algebra",
    "counting_and_probability",
    "geometry",
    "intermediate_algebra",
    "number_theory",
    "prealgebra",
    "precalculus",

使用各種評估方法和指標來評估自訂的 Nova 模型。