Mengevaluasi model terlatih SageMaker AI Anda

Tujuan dari proses evaluasi adalah untuk menilai kinerja model terlatih terhadap tolok ukur atau kumpulan data khusus. Proses evaluasi biasanya melibatkan langkah-langkah untuk membuat resep evaluasi yang menunjuk ke model terlatih, menentukan kumpulan data dan metrik evaluasi, menyerahkan pekerjaan terpisah untuk evaluasi, dan mengevaluasi terhadap tolok ukur standar atau data khusus. Proses evaluasi akan menampilkan metrik kinerja yang disimpan di bucket Amazon S3 Anda.

Prasyarat

Sebelum Anda memulai pekerjaan pelatihan evaluasi, perhatikan hal berikut.

Tugas benchmark yang tersedia

Paket kode sampel tersedia yang menunjukkan cara menghitung metrik tolok ukur menggunakan fitur evaluasi SageMaker model untuk Amazon Nova. Untuk mengakses paket kode, lihat Sampel-N ova-lighteval-custom-task.

Berikut adalah daftar tolok ukur standar industri yang tersedia yang didukung. Anda dapat menentukan tolok ukur berikut dalam eval_task parameter.

Tolok ukur yang tersedia untuk evaluasi model

Evaluasi konfigurasi spesifik

Di bawah ini adalah rincian komponen kunci dalam resep dan panduan tentang cara memodifikasinya untuk kasus penggunaan Anda.

Memahami dan memodifikasi resep Anda

Konfigurasi run umum

run:
  name: eval_job_name 
  model_type: amazon.nova-2-lite-v1:0:256k 
  model_name_or_path: nova-lite-2/prod # or s3://escrow_bucket/model_location
  replicas: 1 
  data_s3_path: ""
  mlflow_tracking_uri: "" 
  mlflow_experiment_name : "" 
  mlflow_run_name : ""    

Konfigurasi evaluasi

evaluation:
  task: mmlu 
  strategy: zs_cot 
  subtask: abstract_algebra
  metric: accuracy

Konfigurasi inferensi (opsional)

inference:
  max_new_tokens: 2048 
  top_k: -1 
  top_p: 1.0 
  temperature: 0
  top_logprobs: 10
  reasoning_effort: null  # options: low/high to enable reasoning or null to disable reasoning 

Contoh resep evaluasi

Amazon Nova menyediakan empat jenis resep evaluasi yang berbeda. Semua resep tersedia di GitHub repositori SageMaker HyperPod resep Amazon.

{
"system":"You are an English major with top marks in class who likes to give minimal word responses: ",
   "query":"What is the symbol that ends the sentence as a question",
   "response":"?"
}{
"system":"You are a pattern analysis specialist who provides succinct answers: ",
   "query":"What is the next number in this series? 1, 2, 4, 8, 16, ?",
   "response":"32"
}{
"system":"You have great attention to detail and follow instructions accurately: ",
   "query":"Repeat only the last two words of the following: I ate a hamburger today and it was kind of dry",
   "response":"of dry"
}{
"system": "Image inference: ",
  "query": "What is the number in the image? Please just use one English word to answer.",
  "response": "two",
  "images": [
    {
      "data": "data:image/png;Base64,iVBORw0KGgoA ..."
    }
  ]
}

evaluation:
  task: gen_qa 
  strategy: gen_qa 
  metric: all

{
"prompt": "What is the most effective way to combat climate change?",
"response_A": "The most effective way to combat climate change is through a combination of transitioning to renewable energy sources and implementing strict carbon pricing policies. This creates economic incentives for businesses to reduce emissions while promoting clean energy adoption.",
"response_B": "We should focus on renewable energy. Solar and wind power are good. People should drive electric cars. Companies need to pollute less."
}
{
"prompt": "Explain how a computer's CPU works",
"response_A": "CPU is like brain of computer. It does math and makes computer work fast. Has lots of tiny parts inside.",
"response_B": "A CPU (Central Processing Unit) functions through a fetch-execute cycle, where instructions are retrieved from memory, decoded, and executed through its arithmetic logic unit (ALU). It coordinates with cache memory and registers to process data efficiently using binary operations."
}
{
"prompt": "How does photosynthesis work?",
"response_A": "Plants do photosynthesis to make food. They use sunlight and water. It happens in leaves.",
"response_B": "Photosynthesis is a complex biochemical process where plants convert light energy into chemical energy. They utilize chlorophyll to absorb sunlight, combining CO2 and water to produce glucose and oxygen through a series of chemical reactions in chloroplasts."
}           

evaluation:
  task: llm_judge
  strategy: judge
  metric: all

{
  "prompt": "What is in the image?",
  "images": [
    {
      "data": "data:image/jpeg;Base64,/9j/2wBDAAQDAwQDAwQEAwQFBAQFBgo..."
    }
  ],
  "response_A": "a dog.",
  "response_B": "a cat.",
} 
{
  "prompt": "How many animals are in each of the images?",
  "images": [
    {
      "data": "data:image/jpeg;Base64,/9j/2wBDAAQDAwQDAwQEAwQFBAQFBgo..."
    },
    {
      "data": "data:image/jpeg;Base64,/DKEafe3gihn..."
    }
  ],
  "response_A": "The first image contains one cat and the second image contains one dog",
  "response_B": "The first image has one aminal and the second has one animal"
}           

evaluation:
  task: mm_llm_judge
  strategy: judge
  metric: all

Menjalankan pekerjaan pelatihan evaluasi

Mulai pekerjaan pelatihan menggunakan contoh notebook Jupyter berikut. Silakan lihat buku catatan di bawah ini sebagai contoh untuk menjalankan pekerjaan pelatihan evaluasi. Untuk informasi selengkapnya, lihat Menggunakan estimator SageMaker AI untuk menjalankan pekerjaan pelatihan.

Tabel referensi

Sebelum menjalankan buku catatan, lihat tabel referensi berikut untuk memilih URI gambar dan konfigurasi instance.

Memilih URI gambar

Memilih jenis dan hitungan instans

Notebook sampel

Contoh notebook berikut menunjukkan bagaimana menjalankan pekerjaan pelatihan evaluasi.

# install python SDK

# Do not use sagemaker v3, as sagemaker v3 introduced breaking changes

!pip install sagemaker==2.254.1
 
import os
import sagemaker,boto3
from sagemaker.inputs import TrainingInput
from sagemaker.pytorch import PyTorch

sagemaker_session = sagemaker.Session()
role = sagemaker.get_execution_role()

# Download recipe from https://github.com/aws/sagemaker-hyperpod-recipes/tree/main/recipes_collection/recipes/evaluation/nova to local
# Assume the file name be `recipe.yaml`

# Populate parameters
# input_s3_uri = "s3://<path>/input/" # (Optional) Only used for multi-modal dataset or bring your own dataset s3 location
output_s3_uri= "s3://<path>/output/" # Output data s3 location, a zip containing metrics json and tensorboard metrics files will be stored to this location
instance_type = "instance_type"  # ml.g5.16xlarge as example
instance_count = 1 # The number of instances for inference (set instance_count > 1 for multi-node inference to accelerate evaluation)             
job_name = "your job name"
recipe_path = "recipe path" # ./recipe.yaml as example
image_uri = "708977205387.dkr.ecr.us-east-1.amazonaws.com/nova-evaluation-repo:SM-TJ-Eval-V2-latest" # Do not change
output_kms_key = "<KMS key arn to encrypt trained model in Amazon-owned S3 bucket>" # optional, leave blank for Amazon managed encryption

# (Optional) To bring your own dataset and LLM judge for evaluation
# evalInput = TrainingInput(
# s3_data=input_s3_uri,
# distribution='FullyReplicated',
# s3_data_type='S3Prefix'
#)

estimator = PyTorch(
    output_path=output_s3_uri,
    base_job_name=job_name,
    role=role,
    instance_type=instance_type,
    instance_count=instance_count,
    training_recipe=recipe_path,
    sagemaker_session=sagemaker_session,
    image_uri=image_uri,
    output_kms_key=output_kms_key
)
estimator.fit()

# If input dataset exist, pass in inputs
# estimator.fit(inputs={"train": evalInput})

Menilai dan menganalisis hasil evaluasi

Setelah pekerjaan evaluasi Anda selesai dengan sukses, Anda dapat menilai dan menganalisis hasilnya menggunakan langkah-langkah berikut.

Evaluasi praktik terbaik dan pemecahan masalah

Praktik terbaik

Berikut ini mencantumkan beberapa praktik terbaik untuk proses evaluasi.

Pemecahan masalah

Anda dapat menggunakan grup CloudWatch log /aws/sagemaker/TrainingJobs untuk melatih log kesalahan pekerjaan.

Kegagalan inti mesin

Masalah:

Jika Anda melihat:

RuntimeError: Engine core initialization failed.

Penyebab:

Meskipun ini adalah kesalahan umum yang dapat memiliki banyak penyebab, biasanya terjadi ketika ada ketidakcocokan antara pos pemeriksaan model yang Anda coba muat dan jenis model yang ditentukan. Misalnya Anda ingin mengevaluasi pos pemeriksaan model Nova 2.0 lite yang disetel dengan baik tetapi tipe model yang Anda berikan adalah tipe model 1.0. misalnya amazon.nova-micro-v1:0:128k

Pemetaan yang benar harus

model_type: amazon.nova-2-lite-v1:0:256k
model_name_or_path: nova-lite-2/prod # or s3://escrow_bucket/model_location

Pencegahan:

Periksa kembali apakah model_name_or_path dipetakan ke kanan model_type sebelum mengirimkan pekerjaan evaluasi.

Subtugas yang tersedia

Berikut daftar subtugas yang tersedia untuk evaluasi model di beberapa domain termasuk MMLU (Massive Multitask Language Understanding), BBH (Big Bench Hard), dan MATH. Subtugas ini memungkinkan Anda menilai kinerja model Anda pada kemampuan dan bidang pengetahuan tertentu.

MMLU

MMLU_SUBTASKS = [
    "abstract_algebra",
    "anatomy",
    "astronomy",
    "business_ethics",
    "clinical_knowledge",
    "college_biology",
    "college_chemistry",
    "college_computer_science",
    "college_mathematics",
    "college_medicine",
    "college_physics",
    "computer_security",
    "conceptual_physics",
    "econometrics",
    "electrical_engineering",
    "elementary_mathematics",
    "formal_logic",
    "global_facts",
    "high_school_biology",
    "high_school_chemistry",
    "high_school_computer_science",
    "high_school_european_history",
    "high_school_geography",
    "high_school_government_and_politics",
    "high_school_macroeconomics",
    "high_school_mathematics",
    "high_school_microeconomics",
    "high_school_physics",
    "high_school_psychology",
    "high_school_statistics",
    "high_school_us_history",
    "high_school_world_history",
    "human_aging",
    "human_sexuality",
    "international_law",
    "jurisprudence",
    "logical_fallacies",
    "machine_learning",
    "management",
    "marketing",
    "medical_genetics",
    "miscellaneous",
    "moral_disputes",
    "moral_scenarios",
    "nutrition",
    "philosophy",
    "prehistory",
    "professional_accounting",
    "professional_law",
    "professional_medicine",
    "professional_psychology",
    "public_relations",
    "security_studies",
    "sociology",
    "us_foreign_policy",
    "virology",
    "world_religions"
]

BBH

BBH_SUBTASKS = [
    "boolean_expressions",
    "causal_judgement",
    "date_understanding",
    "disambiguation_qa",
    "dyck_languages",
    "formal_fallacies",
    "geometric_shapes",
    "hyperbaton",
    "logical_deduction_five_objects",
    "logical_deduction_seven_objects",
    "logical_deduction_three_objects",
    "movie_recommendation",
    "multistep_arithmetic_two",
    "navigate",
    "object_counting",
    "penguins_in_a_table",
    "reasoning_about_colored_objects",
    "ruin_names",
    "salient_translation_error_detection",
    "snarks",
    "sports_understanding",
    "temporal_sequences",
    "tracking_shuffled_objects_five_objects",
    "tracking_shuffled_objects_seven_objects",
    "tracking_shuffled_objects_three_objects",
    "web_of_lies",
    "word_sorting"
]

Matematika

MATH_SUBTASKS = [
    "algebra",
    "counting_and_probability",
    "geometry",
    "intermediate_algebra",
    "number_theory",
    "prealgebra",
    "precalculus",

Evaluasi model Nova Anda yang disesuaikan menggunakan berbagai metode dan metrik evaluasi.