評估您的訓練模型

評估配方是一種 YAML 組態檔案，可定義 Amazon Nova 模型評估任務的執行方式。使用此配方，您可以根據常見的基準或您自己的自訂資料集，來評估基礎或訓練模型的效能。指標可以存放在 Amazon S3 或 TensorBoard 中。評估會提供量化指標，協助您評估各種任務的模型效能，以判斷是否需要進一步自訂。

模型評估是一種離線程序，在程序中會根據具有預先定義的回答測試模型。系統不會即時或根據即時使用者互動來評估它們。對於即時評估，將模型部署到 Amazon Bedrock 之後，您可以透過呼叫 Amazon Bedrock 執行時期 API 來評估模型。

可用的基準任務

提供範例程式碼套件，示範如何使用 Amazon Nova 的 SageMaker AI 模型評估功能來計算基準指標。若要存取程式碼套件，請參閱 sample-Nova-lighteval-custom-task。

以下是支援的可用產業標準基準清單。您可以在 eval_task 參數中指定下列基準。

下列是可用的 mmlu 子任務：

MMLU_SUBTASKS = [
    "abstract_algebra",
    "anatomy",
    "astronomy",
    "business_ethics",
    "clinical_knowledge",
    "college_biology",
    "college_chemistry",
    "college_computer_science",
    "college_mathematics",
    "college_medicine",
    "college_physics",
    "computer_security",
    "conceptual_physics",
    "econometrics",
    "electrical_engineering",
    "elementary_mathematics",
    "formal_logic",
    "global_facts",
    "high_school_biology",
    "high_school_chemistry",
    "high_school_computer_science",
    "high_school_european_history",
    "high_school_geography",
    "high_school_government_and_politics",
    "high_school_macroeconomics",
    "high_school_mathematics",
    "high_school_microeconomics",
    "high_school_physics",
    "high_school_psychology",
    "high_school_statistics",
    "high_school_us_history",
    "high_school_world_history",
    "human_aging",
    "human_sexuality",
    "international_law",
    "jurisprudence",
    "logical_fallacies",
    "machine_learning",
    "management",
    "marketing",
    "medical_genetics",
    "miscellaneous",
    "moral_disputes",
    "moral_scenarios",
    "nutrition",
    "philosophy",
    "prehistory",
    "professional_accounting",
    "professional_law",
    "professional_medicine",
    "professional_psychology",
    "public_relations",
    "security_studies",
    "sociology",
    "us_foreign_policy",
    "virology",
    "world_religions"
]

下列是可用的 bbh 子任務：

BBH_SUBTASKS = [
    "boolean_expressions",
    "causal_judgement",
    "date_understanding",
    "disambiguation_qa",
    "dyck_languages",
    "formal_fallacies",
    "geometric_shapes",
    "hyperbaton",
    "logical_deduction_five_objects",
    "logical_deduction_seven_objects",
    "logical_deduction_three_objects",
    "movie_recommendation",
    "multistep_arithmetic_two",
    "navigate",
    "object_counting",
    "penguins_in_a_table",
    "reasoning_about_colored_objects",
    "ruin_names",
    "salient_translation_error_detection",
    "snarks",
    "sports_understanding",
    "temporal_sequences",
    "tracking_shuffled_objects_five_objects",
    "tracking_shuffled_objects_seven_objects",
    "tracking_shuffled_objects_three_objects",
    "web_of_lies",
    "word_sorting"
]

下列是可用的 math 子任務：

MATH_SUBTASKS = [
    "algebra",
    "counting_and_probability",
    "geometry",
    "intermediate_algebra",
    "number_theory",
    "prealgebra",
    "precalculus",
]

了解配方參數

執行組態

以下是一般執行組態及所涉及參數的說明。

run:
  name: eval_job_name 
  model_type: amazon.nova-micro-v1:0:128k 
  model_name_or_path: nova-micro/prod 
  replicas: 1 
  data_s3_path: ""
  output_s3_path: s3://output_path

評估組態

以下是模型評估組態及所涉及參數的說明。

evaluation:
  task: mmlu 
  strategy: zs_cot 
  subtask: mathematics
  metric: accuracy

推論組態

以下是推論組態及所涉及參數的說明。所有參數都是選用的。

inference:
  max_new_tokens: 200 
  top_k: -1 
  top_p: 1.0 
  temperature: 0

請注意，對於 humaneval，我們建議使用下列推論組態：

inference:
  top_k: 1
  max_new_tokens: 1600
  temperature: 0.0

評估配方範例

Amazon Nova 提供四種類型的評估配方，可在 HyperPod 配方 GitHub 儲存庫中使用。

一般文字基準配方

這些配方可讓您在一套完整的純文字基準之間評估 Amazon Nova 模型的基本功能。它們會以格式 xxx_general_text_benchmark_eval.yaml 提供。

一般多模態基準配方

這些配方可讓您在一套完整的多模態基準之間評估 Amazon Nova 模型的基本功能。它們會以格式 xxx_general_multi_modal_benchmark_eval.yaml 提供。

使用自有資料集基準配方

這些配方可讓您使用自有資料集進行基準測試，並使用不同類型的指標比較模型輸出以參考回答。它們會以格式 xxx_bring_your_own_dataset_eval.yaml 提供。

以下是使用自有資料集需求：

以下是使用自有資料集範例項目

{
   "system":"You are a english major with top marks in class who likes to give minimal word responses: ",
   "query":"What is the symbol that ends the sentence as a question",
   "response":"?"
}
{
   "system":"You are a pattern analysis specialist that provides succinct answers: ",
   "query":"What is the next number in this series? 1, 2, 4, 8, 16, ?",
   "response":"32"
}
{
   "system":"You have great attention to detail that follows instructions accurately: ",
   "query":"Repeat only the last two words of the following: I ate a hamburger today and it was kind of dry",
   "response":"of dry"
}

若要使用您的自訂資料集，請使用下列必要欄位修改您的評估配方，切勿變更任何內容：

evaluation:
  task: gen_qa 
  strategy: gen_qa 
  metric: all

有下列限制：

Amazon Nova LLM 即評審基準配方

Amazon Nova LLM 即評審是一種模型評估功能，可讓客戶將一個模型的回應品質與自訂資料集上的基準模型回應進行比較。它會採用具有提示、基準回應和挑戰者回應的資料集，然後使用 Nova Judge 模型透過配對比較，以根據 Bradley-Terry 機率提供勝率指標。

配方會以 xxx_llm_judge_eval.yaml 格式提供。

以下是 LLM 即評審需求：

以下是 LLM 即評審範例項目

{
"prompt": "What is the most effective way to combat climate change?",
"response_A": "The most effective way to combat climate change is through a combination of transitioning to renewable energy sources and implementing strict carbon pricing policies. This creates economic incentives for businesses to reduce emissions while promoting clean energy adoption.",
"response_B": "We should focus on renewable energy. Solar and wind power are good. People should drive electric cars. Companies need to pollute less."
}
{
"prompt": "Explain how a computer's CPU works",
"response_A": "CPU is like brain of computer. It does math and makes computer work fast. Has lots of tiny parts inside.",
"response_B": "A CPU (Central Processing Unit) functions through a fetch-execute cycle, where instructions are retrieved from memory, decoded, and executed through its arithmetic logic unit (ALU). It coordinates with cache memory and registers to process data efficiently using binary operations."
}
{
"prompt": "How does photosynthesis work?",
"response_A": "Plants do photosynthesis to make food. They use sunlight and water. It happens in leaves.",
"response_B": "Photosynthesis is a complex biochemical process where plants convert light energy into chemical energy. They utilize chlorophyll to absorb sunlight, combining CO2 and water to produce glucose and oxygen through a series of chemical reactions in chloroplasts."
}

若要使用您的自訂資料集，請使用下列必要欄位修改您的評估配方，切勿變更任何內容：

evaluation:
  task: llm_judge
  strategy: judge
  metric: all

有下列限制：

啟動評估任務

以下提供建議的評估執行個體類型和模型類型組態：

# Install Dependencies (Helm - https://helm.sh/docs/intro/install/)
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
rm -f ./get_helm.sh  

# Install the HyperPod CLI
git clone --recurse-submodules https://github.com/aws/sagemaker-hyperpod-cli.git
git checkout -b release_v2
cd sagemaker-hyperpod-cli
pip install .

# Verify the installation
hyperpod --help

# Connect to a HyperPod Cluster
hyperpod connect-cluster --cluster-name cluster-name


# Submit the Job using the recipe for eval
# Namespace by default should be kubeflow
hyperpod start-job [--namespace namespace] --recipe evaluation/nova/nova_micro_p5_48xl_general_text_benchmark_eval --override-parameters \
'{
    "instance_type":"p5d.48xlarge",
    "container": "708977205387.dkr.ecr.us-east-1.amazonaws.com/nova-evaluation-repo:SM-HP-Eval-latest",
    "recipes.run.name": custom-run-name,
    "recipes.run.model_type": model_type,
    "recipes.run.model_name_or_path" " model name or finetune checkpoint s3uri,
    "recipes.run.data_s3_path": s3 for input data only for genqa and llm_judge, must be full S3 path that include filename,
}'

# List jobs
hyperpod list-jobs [--namespace namespace] [--all-namespaces]

# Getting Job details
hyperpod get-job --job-name job-name [--namespace namespace] [--verbose]

# Listing Pods
hyperpod list-pods --job-name job-name --namespace namespace

# Cancel Job
hyperpod cancel-job --job-name job-name [--namespace namespace]

您也應能夠透過 Amazon EKS 叢集主控台檢視任務狀態。

存取和分析評估結果

評估任務成功完成後，您可以使用本節中的資訊來存取和分析結果。根據配方中定義的 output_s3_path (例如 s3://output_path/)，輸出結構如下：

job_name/
├── eval-result/
│    └── results_[timestamp].json
│    └── inference_output.jsonl (only present for gen_qa)
│    └── details/
│        └── model/
│            └── execution-date-time/
│                └──details_task_name_#_datetime.parquet
└── tensorboard-results/
    └── eval/
        └── events.out.tfevents.[timestamp]

指標結果會存放在指定的 S3 輸出位置 s3://output_path/job_name/eval-result/result-timestamp.json。

Tensorboard 結果會存放在 S3 路徑 s3://output_path/job_name/eval-tensorboard-result/eval/event.out.tfevents.epoch+ip 中。

llm_judge 和 strong_reject 除外，所有的推論輸出都存放在 S3 路徑：s3://output_path/job_name/eval-result/details/model/taskname.parquet。

對於 gen_qa，inference_output.jsonl 檔案包含每個 JSON 物件的下列欄位：

若要在 Tensorboard 中視覺化您的評估指標，請完成下列步驟：

時間序列、純量和文字視覺化可供使用。

建議遵循下列最佳實務：

您可以在 CloudWatch 日誌群組 /aws/sagemaker/Clusters/cluster-id 中找到 HyperPod 任務錯誤日誌。