RFT evaluation

What is RFT evaluation?

RFT Evaluation allows you to assess your model's performance using custom reward functions before, during, or after reinforcement learning training. Unlike standard evaluations that use pre-defined metrics, RFT Evaluation lets you define your own success criteria through a Lambda function that scores model outputs based on your specific requirements.

Why evaluate with RFT?

Evaluation is crucial to determine whether the RL fine-tuning process has:

When to use RFT evaluation

Use RFT Evaluation in these scenarios:

Data format requirements

Input data structure

RFT evaluation input data must follow the OpenAI Reinforcement Fine-Tuning format. Each example is a JSON object containing:

Data format example

{  
  "messages": [  
    {  
      "role": "user",  
      "content": [  
        {  
          "type": "text",  
          "text": "Solve for x. Return only JSON like {\"x\": <number>}. Equation: 2x + 5 = 13"  
        }  
      ]  
    }  
  ],  
  "reference_answer": {  
    "x": 4  
  }  
}

Current limitations

Preparing your evaluation recipe

Sample notebook

For a complete example, see Evaluation notebooks.

Sample recipe configuration

run:  
  name: nova-lite-rft-eval-job    
  model_type: amazon.nova-lite-v1:0:300k    
  model_name_or_path: s3://escrow_bucket/model_location # [MODIFIABLE] S3 path to your model or model identifier  
  replicas: 1 # [MODIFIABLE] For SageMaker Training jobs only; fixed for HyperPod jobs  
  data_s3_path: "" # [REQUIRED FOR HYPERPOD] Leave empty for SageMaker Training jobs and use TrainingInput in sagemaker python SDK  
  output_s3_path: "" # [REQUIRED] Output artifact S3 path for evaluation results  
  
evaluation:  
  task: rft_eval # [FIXED] Do not modify  
  strategy: rft_eval # [FIXED] Do not modify  
  metric: all # [FIXED] Do not modify  
  
# Inference Configuration  
inference:  
  max_new_tokens: 8192 # [MODIFIABLE] Maximum tokens to generate  
  top_k: -1 # [MODIFIABLE] Top-k sampling parameter  
  top_p: 1.0 # [MODIFIABLE] Nucleus sampling parameter  
  temperature: 0 # [MODIFIABLE] Sampling temperature (0 = deterministic)  
  top_logprobs: 0 # [MODIFIABLE] Set between 1-20 to enable logprobs output  
  
# =============================================================================  
# Bring Your Own Reinforcement Learning Environment  
# =============================================================================  
rl_env:  
  reward_lambda_arn: arn:aws:lambda:<region>:<account_id>:function:<reward-function-name>

Preset reward functions

Two preset reward functions (prime_code and prime_math) from the open source verl library are available as a Lambda layer for easy integration with your RFT Lambda functions.

Overview

These preset functions provide out-of-the-box evaluation capabilities for:

Quick setup

prime_code function

Evaluates Python code generation tasks by executing code against test cases and measuring correctness.

Example input dataset format

{"messages":[{"role":"user","content":"Write a function that returns the sum of two numbers."}],"reference_answer":{"inputs":["3\n5","10\n-2","0\n0"],"outputs":["8","8","0"]}}
{"messages":[{"role":"user","content":"Write a function to check if a number is even."}],"reference_answer":{"inputs":["4","7","0","-2"],"outputs":["True","False","True","True"]}}

Key features

prime_math function

Evaluates mathematical reasoning and problem-solving capabilities with symbolic math support.

Input format

{"messages":[{"role":"user","content":"What is the derivative of x^2 + 3x?."}],"reference_answer":"2*x + 3"}

Key features

Data format requirements

For code evaluation

For math evaluation

Best practices

Error handling

Both functions include robust error handling for:

Creating your reward function

Lambda ARN requirements

Your Lambda ARN must follow this format:

"arn:aws:lambda:*:*:function:*SageMaker*"

If the Lambda does not have this naming scheme, the job will fail with this error:

[ERROR] Unexpected error: lambda_arn must contain one of: ['SageMaker', 'sagemaker', 'Sagemaker'] when running on SMHP platform (Key: lambda_arn)

Lambda request format

Your Lambda function receives data in this format:

[  
  {  
    "id": "sample-001",  
    "messages": [  
      {  
        "role": "user",  
        "content": [  
          {  
            "type": "text",  
            "text": "Do you have a dedicated security team?"  
          }  
        ]  
      },  
      {  
        "role": "nova_assistant",  
        "content": [  
          {  
            "type": "text",  
            "text": "As an AI developed by Company, I don't have a dedicated security team..."  
          }  
        ]  
      }  
    ],  
    "reference_answer": {  
      "compliant": "No",  
      "explanation": "As an AI developed by Company, I do not have a traditional security team..."  
    }  
  }  
]

Lambda response format

Your Lambda function must return data in this format:

[  
  {  
    "id": "sample-001",  
    "aggregate_reward_score": 0.75,  
    "metrics_list": [  
      {  
        "name": "accuracy",  
        "value": 0.85,  
        "type": "Metric"  
      },  
      {  
        "name": "fluency",  
        "value": 0.90,  
        "type": "Reward"  
      }  
    ]  
  }  
]

Response fields

IAM permissions

Required permissions

Your SageMaker execution role must have permissions to invoke your Lambda function. Add this policy to your SageMaker execution role:

{  
  "Version": "2012-10-17",		 	 	   
  "Statement": [  
    {  
      "Effect": "Allow",  
      "Action": [  
        "lambda:InvokeFunction"  
      ],  
      "Resource": "arn:aws:lambda:region:account-id:function:function-name"  
    }  
  ]  
}

Lambda execution role

Your Lambda function's execution role needs basic Lambda execution permissions:

{  
  "Version": "2012-10-17",		 	 	   
  "Statement": [  
    {  
      "Effect": "Allow",  
      "Action": [  
        "logs:CreateLogGroup",  
        "logs:CreateLogStream",  
        "logs:PutLogEvents"  
      ],  
      "Resource": "arn:aws:logs:*:*:*"  
    }  
  ]  
}

If your Lambda function accesses other AWS services (e.g., S3 for reference data, DynamoDB for logging), add those permissions to the Lambda execution role.

Executing the evaluation job

Understanding evaluation results

Output format

The evaluation job outputs results to your specified S3 location in JSONL format. Each line contains the evaluation results for one sample:

{  
  "id": "sample-001",  
  "aggregate_reward_score": 0.75,  
  "metrics_list": [  
    {  
      "name": "accuracy",  
      "value": 0.85,  
      "type": "Metric"  
    },  
    {  
      "name": "fluency",  
      "value": 0.90,  
      "type": "Reward"  
    }  
  ]  
}

Interpreting results

Aggregate reward score

Individual metrics

Performance benchmarks

What constitutes "good" performance depends on your use case:

Troubleshooting

Common issues

Debug checklist

Best practices

Next steps

After completing RFT evaluation: