评估经过训练的模型

评估配方是一个定义如何执行 Amazon Nova 模型评测任务的 YAML 配置文件。使用此配方，您可以根据常用基准或自有的自定义数据集评估基础模型或训练模型的性能。性能指标可能存储在 Amazon S3 或 TensorBoard 中。评估提供定量指标来帮助您评估模型在各种任务中的性能，从而确定是否需要进一步自定义。

模型评测是一个离线过程，模型使用预定义的答案，按照固定的基准进行测试。模型不是实时评估的，也不是按照用户实时互动进行评估。对于实时评估，您可以在模型部署到 Amazon Bedrock 之后通过调用 Amazon Bedrock 运行时 API 进行评估。

可用的基准测试任务

一个可用的示例代码包，用于演示如何使用适用于 Amazon Nova 的 SageMaker AI 模型评测特征计算基准评测指标。要获取代码包，请参阅 sample-Nova-lighteval-custom-task。

以下是支持的可用行业标准基准测试列表。您可以在 eval_task 参数中指定以下基准测试：

以下 mmlu 子任务可用：

MMLU_SUBTASKS = [
    "abstract_algebra",
    "anatomy",
    "astronomy",
    "business_ethics",
    "clinical_knowledge",
    "college_biology",
    "college_chemistry",
    "college_computer_science",
    "college_mathematics",
    "college_medicine",
    "college_physics",
    "computer_security",
    "conceptual_physics",
    "econometrics",
    "electrical_engineering",
    "elementary_mathematics",
    "formal_logic",
    "global_facts",
    "high_school_biology",
    "high_school_chemistry",
    "high_school_computer_science",
    "high_school_european_history",
    "high_school_geography",
    "high_school_government_and_politics",
    "high_school_macroeconomics",
    "high_school_mathematics",
    "high_school_microeconomics",
    "high_school_physics",
    "high_school_psychology",
    "high_school_statistics",
    "high_school_us_history",
    "high_school_world_history",
    "human_aging",
    "human_sexuality",
    "international_law",
    "jurisprudence",
    "logical_fallacies",
    "machine_learning",
    "management",
    "marketing",
    "medical_genetics",
    "miscellaneous",
    "moral_disputes",
    "moral_scenarios",
    "nutrition",
    "philosophy",
    "prehistory",
    "professional_accounting",
    "professional_law",
    "professional_medicine",
    "professional_psychology",
    "public_relations",
    "security_studies",
    "sociology",
    "us_foreign_policy",
    "virology",
    "world_religions"
]

以下 bbh 子任务可用：

BBH_SUBTASKS = [
    "boolean_expressions",
    "causal_judgement",
    "date_understanding",
    "disambiguation_qa",
    "dyck_languages",
    "formal_fallacies",
    "geometric_shapes",
    "hyperbaton",
    "logical_deduction_five_objects",
    "logical_deduction_seven_objects",
    "logical_deduction_three_objects",
    "movie_recommendation",
    "multistep_arithmetic_two",
    "navigate",
    "object_counting",
    "penguins_in_a_table",
    "reasoning_about_colored_objects",
    "ruin_names",
    "salient_translation_error_detection",
    "snarks",
    "sports_understanding",
    "temporal_sequences",
    "tracking_shuffled_objects_five_objects",
    "tracking_shuffled_objects_seven_objects",
    "tracking_shuffled_objects_three_objects",
    "web_of_lies",
    "word_sorting"
]

以下 math 子任务可用：

MATH_SUBTASKS = [
    "algebra",
    "counting_and_probability",
    "geometry",
    "intermediate_algebra",
    "number_theory",
    "prealgebra",
    "precalculus",
]

了解配方参数

运行配置

以下是常规运行配置和所涉及参数的说明。

run:
  name: eval_job_name 
  model_type: amazon.nova-micro-v1:0:128k 
  model_name_or_path: nova-micro/prod 
  replicas: 1 
  data_s3_path: ""
  output_s3_path: s3://output_path

评估配置

以下是模型评测配置和所涉及参数的说明。

evaluation:
  task: mmlu 
  strategy: zs_cot 
  subtask: mathematics
  metric: accuracy

推理配置

以下是推理配置和所涉及参数的说明。所有参数都是可选的。

inference:
  max_new_tokens: 200 
  top_k: -1 
  top_p: 1.0 
  temperature: 0

请注意，对于 humaneval，建议使用以下推理配置：

inference:
  top_k: 1
  max_new_tokens: 1600
  temperature: 0.0

评估配方示例

Amazon Nova 提供了四种类型的评估配方，这些配方可在 HyperPod 配方 GitHub 存储库中找到。

通用文本基准测试配方

这些配方让您能够通过一套全面的纯文本基准测试来评估 Amazon Nova 模型的基本功能。它们以 xxx_general_text_benchmark_eval.yaml 格式提供。

通用多模态基准测试配方

这些配方让您能够通过一套全面的多模态基准测试来评估 Amazon Nova 模型的基本功能。它们以 xxx_general_multi_modal_benchmark_eval.yaml 格式提供。

自带数据集基准测试配方

这些配方让您能够使用自己的数据集进行基准测试，并使用不同类型的指标将模型输出与参考答案进行对比。它们以 xxx_bring_your_own_dataset_eval.yaml 格式提供。

以下是自带数据集的要求：

以下是自带数据集示例条目

{
   "system":"You are a english major with top marks in class who likes to give minimal word responses: ",
   "query":"What is the symbol that ends the sentence as a question",
   "response":"?"
}
{
   "system":"You are a pattern analysis specialist that provides succinct answers: ",
   "query":"What is the next number in this series? 1, 2, 4, 8, 16, ?",
   "response":"32"
}
{
   "system":"You have great attention to detail that follows instructions accurately: ",
   "query":"Repeat only the last two words of the following: I ate a hamburger today and it was kind of dry",
   "response":"of dry"
}

要使用您的自定义数据集，请按以下必填字段修改您的评估配方，且不要更改其中任何内容：

evaluation:
  task: gen_qa 
  strategy: gen_qa 
  metric: all

适用以下限制：

Amazon Nova LLM-as-a-Judge 基准测试配方

Amazon Nova LLM-as-a-Judge 是一项模型评测特征，让客户可以使用自定义数据集，将一个模型的响应质量与基准模型的响应质量进行比较。它接受包含提示、基准响应和挑战响应的数据集，然后使用 Nova Judge 模型基于 Bradley-Terry 概率和成对比较提供胜率指标。

配方以 xxx_llm_judge_eval.yaml 格式提供。

以下是 LLM-as-a-Judge 的要求：

这是一个 LLM-as-a-Judge 示例条目

{
"prompt": "What is the most effective way to combat climate change?",
"response_A": "The most effective way to combat climate change is through a combination of transitioning to renewable energy sources and implementing strict carbon pricing policies. This creates economic incentives for businesses to reduce emissions while promoting clean energy adoption.",
"response_B": "We should focus on renewable energy. Solar and wind power are good. People should drive electric cars. Companies need to pollute less."
}
{
"prompt": "Explain how a computer's CPU works",
"response_A": "CPU is like brain of computer. It does math and makes computer work fast. Has lots of tiny parts inside.",
"response_B": "A CPU (Central Processing Unit) functions through a fetch-execute cycle, where instructions are retrieved from memory, decoded, and executed through its arithmetic logic unit (ALU). It coordinates with cache memory and registers to process data efficiently using binary operations."
}
{
"prompt": "How does photosynthesis work?",
"response_A": "Plants do photosynthesis to make food. They use sunlight and water. It happens in leaves.",
"response_B": "Photosynthesis is a complex biochemical process where plants convert light energy into chemical energy. They utilize chlorophyll to absorb sunlight, combining CO2 and water to produce glucose and oxygen through a series of chemical reactions in chloroplasts."
}

要使用您的自定义数据集，请按以下必填字段修改您的评估配方，且不要更改其中任何内容：

evaluation:
  task: llm_judge
  strategy: judge
  metric: all

适用以下限制：

启动评估作业

下面提供了建议的评估实例类型和模型类型配置：

# Install Dependencies (Helm - https://helm.sh/docs/intro/install/)
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
rm -f ./get_helm.sh  

# Install the HyperPod CLI
git clone --recurse-submodules https://github.com/aws/sagemaker-hyperpod-cli.git
git checkout -b release_v2
cd sagemaker-hyperpod-cli
pip install .

# Verify the installation
hyperpod --help

# Connect to a HyperPod Cluster
hyperpod connect-cluster --cluster-name cluster-name


# Submit the Job using the recipe for eval
# Namespace by default should be kubeflow
hyperpod start-job [--namespace namespace] --recipe evaluation/nova/nova_micro_p5_48xl_general_text_benchmark_eval --override-parameters \
'{
    "instance_type":"p5d.48xlarge",
    "container": "708977205387.dkr.ecr.us-east-1.amazonaws.com/nova-evaluation-repo:SM-HP-Eval-latest",
    "recipes.run.name": custom-run-name,
    "recipes.run.model_type": model_type,
    "recipes.run.model_name_or_path" " model name or finetune checkpoint s3uri,
    "recipes.run.data_s3_path": s3 for input data only for genqa and llm_judge, must be full S3 path that include filename,
}'

# List jobs
hyperpod list-jobs [--namespace namespace] [--all-namespaces]

# Getting Job details
hyperpod get-job --job-name job-name [--namespace namespace] [--verbose]

# Listing Pods
hyperpod list-pods --job-name job-name --namespace namespace

# Cancel Job
hyperpod cancel-job --job-name job-name [--namespace namespace]

您还应该能够通过 Amazon EKS 集群控制台查看作业状态。

访问和分析评估结果

成功完成评估作业后，您可以参考本节中的信息访问并分析结果。根据配方中定义的 output_s3_path（例如 s3://output_path/），输出结构如下所示：

job_name/
├── eval-result/
│    └── results_[timestamp].json
│    └── inference_output.jsonl (only present for gen_qa)
│    └── details/
│        └── model/
│            └── execution-date-time/
│                └──details_task_name_#_datetime.parquet
└── tensorboard-results/
    └── eval/
        └── events.out.tfevents.[timestamp]

指标结果存储在指定的 S3 输出位置 s3://output_path/job_name/eval-result/result-timestamp.json。

Tensorboard 结果存储在 S3 路径 s3://output_path/job_name/eval-tensorboard-result/eval/event.out.tfevents.epoch+ip 中。

除 llm_judge 和 strong_reject 之外的所有推理输出都存储在 S3 路径中：s3://output_path/job_name/eval-result/details/model/taskname.parquet。

对于 gen_qa，inference_output.jsonl 文件针对每个 JSON 对象均包含以下字段：

要在 Tensorboard 中直观显示您的评估指标，请完成以下步骤：

时间序列、标量和文本可视化均可用。

我们建议您遵循以下最佳实操：

您可以在 CloudWatch 日志组 /aws/sagemaker/Clusters/cluster-id 中找到 HyperPod 作业错误日志。