评估经过 SageMaker AI 训练的模型

评估过程旨在根据基准测试或自定义数据集评测训练模型的性能。评估过程通常涉及以下步骤：创建指向训练模型的评估配方、指定评估数据集和指标、提交单独的评估作业，以及根据标准基准测试或自定义数据进行评估。评估过程将输出存储在您的 Amazon S3 存储桶中的性能指标。

先决条件

在开始评估训练作业之前，请注意具备以下内容：

可用的基准测试任务

一个可用的示例代码包，用于演示如何使用适用于 Amazon Nova 的 SageMaker AI 模型评测特征计算基准评测指标。要获取代码包，请参阅 sample-Nova-lighteval-custom-task。

下面是支持的可用行业标准基准测试列表。您可以在 eval_task 参数中指定以下基准测试。

可用的模型评测基准测试

特定于评估的配置

以下是关于配方中关键组成部分的详细介绍，以及有关如何根据您的使用案例对其进行修改的指导。

了解和修改配方

常规运行配置

run:
  name: eval_job_name
  model_type: amazon.nova-lite-v1:0:300k
  model_name_or_path: nova-lite/prod # or s3://escrow_bucket/model_location
  replicas: 1
  data_s3_path: ""
  mlflow_tracking_uri: ""
  mlflow_experiment_name : ""
  mlflow_run_name : ""    

评估配置

evaluation:
  task: mmlu
  strategy: zs_cot
  subtask: abstract_algebra
  metric: accuracy

推理配置（可选）

inference:
  max_new_tokens: 2048
  top_k: -1
  top_p: 1.0
  temperature: 0
  top_logprobs: 10
  reasoning_effort: null  # options: low/high to enable reasoning or null to disable reasoning 

对数概率输出格式

在推理设置中配置 top_logprobs 后，评估输出会在 parquet 文件中包含词元级别的对数概率。每个词元位置都会以字典形式，返回排名靠前的候选词元及其对数概率，结构如下：

{
"Ġint": {"logprob_value": -17.8125, "decoded_value": " int"},
"Ġthe": {"logprob_value": -2.345, "decoded_value": " the"}
}

每个词元条目包含：

原始分词器词元作为字典键以确保唯一性，decoded_value 则提供可读形式。

评估配方示例

Amazon Nova 提供四种不同类型的评估配方。所有配方均可在 SageMaker HyperPod 配方 GitHub 存储库中找到。

{
"system":"You are an English major with top marks in class who likes to give minimal word responses: ",
   "query":"What is the symbol that ends the sentence as a question",
   "response":"?"
}{
"system":"You are a pattern analysis specialist who provides succinct answers: ",
   "query":"What is the next number in this series? 1, 2, 4, 8, 16, ?",
   "response":"32"
}{
"system":"You have great attention to detail and follow instructions accurately: ",
   "query":"Repeat only the last two words of the following: I ate a hamburger today and it was kind of dry",
   "response":"of dry"
}{
"system": "Image inference: ",
  "query": "What is the number in the image? Please just use one English word to answer.",
  "response": "two",
  "images": [
    {
      "data": "data:image/png;Base64,iVBORw0KGgoA ..."
    }
  ]
}

evaluation:
  task: gen_qa
  strategy: gen_qa
  metric: all

{
"prompt": "What is the most effective way to combat climate change?",
"response_A": "The most effective way to combat climate change is through a combination of transitioning to renewable energy sources and implementing strict carbon pricing policies. This creates economic incentives for businesses to reduce emissions while promoting clean energy adoption.",
"response_B": "We should focus on renewable energy. Solar and wind power are good. People should drive electric cars. Companies need to pollute less."
}
{
"prompt": "Explain how a computer's CPU works",
"response_A": "CPU is like brain of computer. It does math and makes computer work fast. Has lots of tiny parts inside.",
"response_B": "A CPU (Central Processing Unit) functions through a fetch-execute cycle, where instructions are retrieved from memory, decoded, and executed through its arithmetic logic unit (ALU). It coordinates with cache memory and registers to process data efficiently using binary operations."
}
{
"prompt": "How does photosynthesis work?",
"response_A": "Plants do photosynthesis to make food. They use sunlight and water. It happens in leaves.",
"response_B": "Photosynthesis is a complex biochemical process where plants convert light energy into chemical energy. They utilize chlorophyll to absorb sunlight, combining CO2 and water to produce glucose and oxygen through a series of chemical reactions in chloroplasts."
}           

evaluation:
  task: llm_judge
  strategy: judge
  metric: all

{
  "prompt": "What is in the image?",
  "images": [
    {
      "data": "data:image/jpeg;Base64,/9j/2wBDAAQDAwQDAwQEAwQFBAQFBgo..."
    }
  ],
  "response_A": "a dog.",
  "response_B": "a cat.",
}
{
  "prompt": "How many animals are in each of the images?",
  "images": [
    {
      "data": "data:image/jpeg;Base64,/9j/2wBDAAQDAwQDAwQEAwQFBAQFBgo..."
    },
    {
      "data": "data:image/jpeg;Base64,/DKEafe3gihn..."
    }
  ],
  "response_A": "The first image contains one cat and the second image contains one dog",
  "response_B": "The first image has one aminal and the second has one animal"
}           

evaluation:
  task: mm_llm_judge
  strategy: judge
  metric: all

运行评估训练作业

使用以下示例笔记本启动训练作业。请参考下方的笔记本示例，运行评估训练作业。有关更多信息，请参阅使用 SageMaker AI 估算器运行训练作业。

参考表

在运行笔记本之前，请参阅以下参考表来选择映像 URI 和实例配置。

选择映像 URI

选择实例类型和计数

示例笔记本

以下示例笔记本演示了如何运行评估训练作业。

# install python SDK

# Do not use sagemaker v3, as sagemaker v3 introduced breaking changes

!pip install sagemaker==2.254.1

import os
import sagemaker,boto3
from sagemaker.inputs import TrainingInput
from sagemaker.pytorch import PyTorch

sagemaker_session = sagemaker.Session()
role = sagemaker.get_execution_role()

# Download recipe from https://github.com/aws/sagemaker-hyperpod-recipes/tree/main/recipes_collection/recipes/evaluation/nova to local
# Assume the file name be `recipe.yaml`

# Populate parameters
# input_s3_uri = "s3://<path>/input/" # (Optional) Only used for multi-modal dataset or bring your own dataset s3 location
output_s3_uri= "s3://<path>/output/" # Output data s3 location, a zip containing metrics json and tensorboard metrics files will be stored to this location
instance_type = "instance_type"  # ml.g5.16xlarge as example
instance_count = 1 # The number of instances for inference (set instance_count > 1 for multi-node inference to accelerate evaluation)
job_name = "your job name"
recipe_path = "recipe path" # ./recipe.yaml as example
image_uri = "708977205387.dkr.ecr.us-east-1.amazonaws.com/nova-evaluation-repo:SM-TJ-Eval-V2-latest" # Do not change
output_kms_key = "<KMS key arn to encrypt trained model in Amazon-owned S3 bucket>" # optional, leave blank for Amazon managed encryption

# (Optional) To bring your own dataset and LLM judge for evaluation
# evalInput = TrainingInput(
# s3_data=input_s3_uri,
# distribution='FullyReplicated',
# s3_data_type='S3Prefix'
#)

estimator = PyTorch(
    output_path=output_s3_uri,
    base_job_name=job_name,
    role=role,
    instance_type=instance_type,
    instance_count=instance_count,
    training_recipe=recipe_path,
    sagemaker_session=sagemaker_session,
    image_uri=image_uri,
    output_kms_key=output_kms_key
)
estimator.fit()

# If input dataset exist, pass in inputs
# estimator.fit(inputs={"train": evalInput})

评测和分析评估结果

成功完成评估作业后，您可以使用以下步骤评测和分析结果。

评估最佳实践和疑难解答

最佳实践

下面列出了评估过程的一些最佳实践。

问题排查

您可以使用 CloudWatch 日志组 /aws/sagemaker/TrainingJobs 来记录训练作业错误日志。

引擎核心故障

问题：

如遇以下报错：

RuntimeError: Engine core initialization failed.

原因：

虽然这是一个可能由多种原因引发的一般性错误，但最常见原因是：您尝试加载的模型检查点与指定的模型类型（model_type）不匹配。例如，您想要评估一个微调后的 Nova 2.0 lite 模型检查点，但指定的模型类型却是 1.0 版本（如 amazon.nova-micro-v1:0:128k）

正确的映射示例：

model_type: amazon.nova-2-lite-v1:0:256k
model_name_or_path: nova-lite-2/prod # or s3://escrow_bucket/model_location

预防措施：

提交评估作业前，务必核对 model_name_or_path 与 model_type 是否正确匹配。

可用子任务

下面列出了跨多个领域进行模型评测的可用子任务，包括 MMLU（大规模多任务语言理解）、BBH（高难度推理基准）、MATH（数学推理）和 MMMU（大规模多学科多模态理解）。这些子任务让您能够评测模型在特定功能和知识领域的表现。

MMLU

MMLU_SUBTASKS = [
    "abstract_algebra",
    "anatomy",
    "astronomy",
    "business_ethics",
    "clinical_knowledge",
    "college_biology",
    "college_chemistry",
    "college_computer_science",
    "college_mathematics",
    "college_medicine",
    "college_physics",
    "computer_security",
    "conceptual_physics",
    "econometrics",
    "electrical_engineering",
    "elementary_mathematics",
    "formal_logic",
    "global_facts",
    "high_school_biology",
    "high_school_chemistry",
    "high_school_computer_science",
    "high_school_european_history",
    "high_school_geography",
    "high_school_government_and_politics",
    "high_school_macroeconomics",
    "high_school_mathematics",
    "high_school_microeconomics",
    "high_school_physics",
    "high_school_psychology",
    "high_school_statistics",
    "high_school_us_history",
    "high_school_world_history",
    "human_aging",
    "human_sexuality",
    "international_law",
    "jurisprudence",
    "logical_fallacies",
    "machine_learning",
    "management",
    "marketing",
    "medical_genetics",
    "miscellaneous",
    "moral_disputes",
    "moral_scenarios",
    "nutrition",
    "philosophy",
    "prehistory",
    "professional_accounting",
    "professional_law",
    "professional_medicine",
    "professional_psychology",
    "public_relations",
    "security_studies",
    "sociology",
    "us_foreign_policy",
    "virology",
    "world_religions"
]

BBH

BBH_SUBTASKS = [
    "boolean_expressions",
    "causal_judgement",
    "date_understanding",
    "disambiguation_qa",
    "dyck_languages",
    "formal_fallacies",
    "geometric_shapes",
    "hyperbaton",
    "logical_deduction_five_objects",
    "logical_deduction_seven_objects",
    "logical_deduction_three_objects",
    "movie_recommendation",
    "multistep_arithmetic_two",
    "navigate",
    "object_counting",
    "penguins_in_a_table",
    "reasoning_about_colored_objects",
    "ruin_names",
    "salient_translation_error_detection",
    "snarks",
    "sports_understanding",
    "temporal_sequences",
    "tracking_shuffled_objects_five_objects",
    "tracking_shuffled_objects_seven_objects",
    "tracking_shuffled_objects_three_objects",
    "web_of_lies",
    "word_sorting"
]

数学

MATH_SUBTASKS = [
    "algebra",
    "counting_and_probability",
    "geometry",
    "intermediate_algebra",
    "number_theory",
    "prealgebra",
    "precalculus",

MMMU

            MATH_SUBTASKS = [
    "Accounting",
    "Agriculture",
    "Architecture_and_Engineering",
    "Art",
    "Art_Theory",
    "Basic_Medical_Science",
    "Biology",
    "Chemistry",
    "Clinical_Medicine",
    "Computer_Science",
    "Design",
    "Diagnostics_and_Laboratory_Medicine",
    "Economics",
    "Electronics",
    "Energy_and_Power",
    "Finance",
    "Geography",
    "History",
    "Literature",
    "Manage",
    "Marketing",
    "Materials",
    "Math",
    "Mechanical_Engineering",
    "Music",
    "Pharmacy",
    "Physics",
    "Psychology",
    "Public_Health",
    "Sociology",

            

您可通过多种评估方法与指标，对自定义 Nova 模型进行评估。