评估你的 SageMaker AI 训练模型

评估过程的目的是根据基准或自定义数据集评估训练模型的性能。评估过程通常包括创建指向训练模型的评估方案、指定评估数据集和指标、提交单独的评估任务以及根据标准基准或自定义数据进行评估的步骤。评估过程将输出存储在您的 Amazon S3 存储桶中的性能指标。

先决条件

在开始评估训练作业之前，请注意以下几点。

可用的基准测试任务

提供了一个示例代码包，用于演示如何使用 Amazon Nova 的 SageMaker 模型评估功能计算基准指标。要访问代码包，请参阅 Sample-n ova-lighteval-custom-task。

以下是支持的可用行业标准基准测试列表。您可以在eval_task参数中指定以下基准。

模型评估的可用基准

特定于评估的配置

以下是配方中关键组件的详细介绍，以及有关如何根据您的用例对其进行修改的指南。

了解和修改您的食谱

常规运行配置

run:
  name: eval_job_name 
  model_type: amazon.nova-micro-v1:0:128k 
  model_name_or_path: nova-micro/prod 
  replicas: 1 
  data_s3_path: ""

评估配置

evaluation:
  task: mmlu 
  strategy: zs_cot 
  subtask: abstract_algebra
  metric: accuracy

推理配置（可选）

inference:
  max_new_tokens: 2048 
  top_k: -1 
  top_p: 1.0 
  temperature: 0

评估配方示例

Amazon Nova 提供四种不同类型的评估方案。所有食谱都可在 Amazon SageMaker HyperPod 食谱 GitHub 库中找到。

{
"system":"You are an English major with top marks in class who likes to give minimal word responses: ",
   "query":"What is the symbol that ends the sentence as a question",
   "response":"?"
}{
"system":"You are a pattern analysis specialist who provides succinct answers: ",
   "query":"What is the next number in this series? 1, 2, 4, 8, 16, ?",
   "response":"32"
}{
"system":"You have great attention to detail and follow instructions accurately: ",
   "query":"Repeat only the last two words of the following: I ate a hamburger today and it was kind of dry",
   "response":"of dry"
} 

evaluation:
  task: gen_qa 
  strategy: gen_qa 
  metric: all

{
"prompt": "What is the most effective way to combat climate change?",
"response_A": "The most effective way to combat climate change is through a combination of transitioning to renewable energy sources and implementing strict carbon pricing policies. This creates economic incentives for businesses to reduce emissions while promoting clean energy adoption.",
"response_B": "We should focus on renewable energy. Solar and wind power are good. People should drive electric cars. Companies need to pollute less."
}
{
"prompt": "Explain how a computer's CPU works",
"response_A": "CPU is like brain of computer. It does math and makes computer work fast. Has lots of tiny parts inside.",
"response_B": "A CPU (Central Processing Unit) functions through a fetch-execute cycle, where instructions are retrieved from memory, decoded, and executed through its arithmetic logic unit (ALU). It coordinates with cache memory and registers to process data efficiently using binary operations."
}
{
"prompt": "How does photosynthesis work?",
"response_A": "Plants do photosynthesis to make food. They use sunlight and water. It happens in leaves.",
"response_B": "Photosynthesis is a complex biochemical process where plants convert light energy into chemical energy. They utilize chlorophyll to absorb sunlight, combining CO2 and water to produce glucose and oxygen through a series of chemical reactions in chloroplasts."
}           

evaluation:
  task: llm_judge
  strategy: judge
  metric: all

运行评估训练作业

使用以下示例 Jupyter 笔记本开始训练作业。有关更多信息，请参阅使用 A SageMaker I 估算器运行训练作业。

参考表

在运行笔记本之前，请参阅以下参考表来选择镜像 URI 和实例配置。

选择图片 URI

选择实例类型和计数

示例笔记本

以下示例笔记本演示了如何运行评估训练作业。

# install python SDK
!pip install sagemaker
 
import os
import sagemaker,boto3
from sagemaker.inputs import TrainingInput
from sagemaker.pytorch import PyTorch

sagemaker_session = sagemaker.Session()
role = sagemaker.get_execution_role()

# Download recipe from https://github.com/aws/sagemaker-hyperpod-recipes/tree/main/recipes_collection/recipes/evaluation/nova to local
# Assume the file name be `recipe.yaml`

# Populate parameters
# input_s3_uri = "s3://<path>/input/" # (Optional) Only used for multi-modal dataset or bring your own dataset s3 location
output_s3_uri= "s3://<path>/output/" # Output data s3 location, a zip containing metrics json and tensorboard metrics files will be stored to this location
instance_type = "instace_type"  # ml.g5.16xlarge as example
job_name = "your job name"
recipe_path = "recipe path" # ./recipe.yaml as example
image_uri = "708977205387.dkr.ecr.us-east-1.amazonaws.com/nova-evaluation-repo:SM-TJ-Eval-latest" # Do not change

# (Optional) To bring your own dataset and LLM judge for evaluation
# evalInput = TrainingInput(
# s3_data=input_s3_uri,
# distribution='FullyReplicated',
# s3_data_type='S3Prefix'
#)

estimator = PyTorch(
    output_path=output_s3_uri,
    base_job_name=job_name,
    role=role,
    instance_type=instance_type,
    training_recipe=recipe_path,
    sagemaker_session=sagemaker_session,
    image_uri = image_uri
)
estimator.fit()

# If input dataset exist, pass in inputs
# estimator.fit(inputs={"train": evalInput})

评估和分析评估结果

成功完成评估任务后，您可以使用以下步骤评估和分析结果。

评估最佳实践和故障排除

最佳实践

以下列出了评估过程的一些最佳实践。

故障排除

您可以使用 CloudWatch 日志组来记录/aws/sagemaker/TrainingJobs训练作业错误日志。

CUDA 内存不足错误

问题：

运行模型评估时，您会收到以下错误：

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate X MiB. 
GPU 0 has a total capacity of Y GiB of which Z MiB is free.

原因：

当您尝试加载需要比当前实例类型上可用的 GPU 内存更多的模型时，就会发生此错误。

解决方案：

选择具有更多 GPU 内存的实例类型。例如，如果你使用 G5.12xLarge（96 GiB GPU 内存），请升级到 G5.48xLarge（192 GiB GPU 内存）

预防：

在运行模型评估之前，请执行以下操作。

可用子任务

以下列出了跨多个领域进行模型评估的可用子任务，包括 MMLU（大规模多任务语言理解）、BBH（Big Bench Hard）、数学和 MMMU（大规模多学科多模态理解）。这些子任务允许您评估模型在特定功能和知识领域的性能。

MMLU

MMLU_SUBTASKS = [
    "abstract_algebra",
    "anatomy",
    "astronomy",
    "business_ethics",
    "clinical_knowledge",
    "college_biology",
    "college_chemistry",
    "college_computer_science",
    "college_mathematics",
    "college_medicine",
    "college_physics",
    "computer_security",
    "conceptual_physics",
    "econometrics",
    "electrical_engineering",
    "elementary_mathematics",
    "formal_logic",
    "global_facts",
    "high_school_biology",
    "high_school_chemistry",
    "high_school_computer_science",
    "high_school_european_history",
    "high_school_geography",
    "high_school_government_and_politics",
    "high_school_macroeconomics",
    "high_school_mathematics",
    "high_school_microeconomics",
    "high_school_physics",
    "high_school_psychology",
    "high_school_statistics",
    "high_school_us_history",
    "high_school_world_history",
    "human_aging",
    "human_sexuality",
    "international_law",
    "jurisprudence",
    "logical_fallacies",
    "machine_learning",
    "management",
    "marketing",
    "medical_genetics",
    "miscellaneous",
    "moral_disputes",
    "moral_scenarios",
    "nutrition",
    "philosophy",
    "prehistory",
    "professional_accounting",
    "professional_law",
    "professional_medicine",
    "professional_psychology",
    "public_relations",
    "security_studies",
    "sociology",
    "us_foreign_policy",
    "virology",
    "world_religions"
]

BBH

BBH_SUBTASKS = [
    "boolean_expressions",
    "causal_judgement",
    "date_understanding",
    "disambiguation_qa",
    "dyck_languages",
    "formal_fallacies",
    "geometric_shapes",
    "hyperbaton",
    "logical_deduction_five_objects",
    "logical_deduction_seven_objects",
    "logical_deduction_three_objects",
    "movie_recommendation",
    "multistep_arithmetic_two",
    "navigate",
    "object_counting",
    "penguins_in_a_table",
    "reasoning_about_colored_objects",
    "ruin_names",
    "salient_translation_error_detection",
    "snarks",
    "sports_understanding",
    "temporal_sequences",
    "tracking_shuffled_objects_five_objects",
    "tracking_shuffled_objects_seven_objects",
    "tracking_shuffled_objects_three_objects",
    "web_of_lies",
    "word_sorting"
]

数学

MATH_SUBTASKS = [
    "algebra",
    "counting_and_probability",
    "geometry",
    "intermediate_algebra",
    "number_theory",
    "prealgebra",
    "precalculus",
]

MMMU

MATH_SUBTASKS = [
    "Accounting",
    "Agriculture",
    "Architecture_and_Engineering",
    "Art",
    "Art_Theory",
    "Basic_Medical_Science",
    "Biology",
    "Chemistry",
    "Clinical_Medicine",
    "Computer_Science",
    "Design",
    "Diagnostics_and_Laboratory_Medicine",
    "Economics",
    "Electronics",
    "Energy_and_Power",
    "Finance",
    "Geography",
    "History",
    "Literature",
    "Manage",
    "Marketing",
    "Materials",
    "Math",
    "Mechanical_Engineering",
    "Music",
    "Pharmacy",
    "Physics",
    "Psychology",
    "Public_Health",
    "Sociology",
]