`). The factual knowledge algorithm compares the response that your model generates with a known response. The algorithm scores the response as correct if it generates any of the content separated by the delimiter in the answer. If you pass `None` as the `target_output_delimiter`, then the model must generate the same response as the answer to be scored as correct. Lastly, call the `evaluate` method and pass in your desired parameters.
Factual knowledge returns a list of `EvalScore` objects. These contain aggregated scores on how well your model is able to encode factual knowledge as described in the **Foundation model evaluation overview** section. The scores range between `0` and `1` with the lowest score corresponding to a lower knowledge of real-world facts.
The following code example shows how to evaluate your LLM using the factual knowledge algorithm:
```
from fmeval.eval import get_eval_algorithm
from fmeval.eval_algorithms.factual_knowledge import FactualKnowledge, FactualKnowledgeConfig
eval_algo = FactualKnowledge(FactualKnowledgeConfig())
eval_output = eval_algo.evaluate(model=model_runner, dataset_config=config, prompt_template="$feature", save=True)
```
### Prompt stereotyping
You can run the prompt stereotyping algorithm for open-ended generation. To run the prompt stereotyping algorithm, your `DataConfig` must identify the columns in your input dataset that contain a less stereotypical sentence in `sent_less_input_location` and a more stereotypical sentence in `sent_more_output_location`. For more information about `DataConfig`, see the previous section **2. Configure `ModelRunner`**. Next, call the `evaluate` method and pass in your desired parameters.
Prompt stereotyping returns a list of `EvalOutput` objects that contain a score for each input record and overall scores for each type of bias. The scores are calculated by comparing the probability of the more and less stereotypical sentences. The overall score reports how often the model preferred the stereotypical sentence in that the model assigns a higher probability to the more stereotypical compared to the less stereotypical sentence. A score of `0.5` indicates that your model is unbiased, or that it prefers more and less stereotypical sentences at equal rates. A score of greater than `0.5` indicates that your model is likely to generate a response that is more stereotypical. Scores less than `0.5` indicate that your model is likely to generate a response that is less stereotypical.
The following code example shows how to evaluate your LLM using the prompt stereotyping algorithm:
```
from fmeval.eval import get_eval_algorithm
from fmeval.eval_algorithms.prompt_stereotyping import PromptStereotyping
eval_algo = PromptStereotyping()
eval_output = eval_algo.evaluate(model=model_runner, dataset_config=config, prompt_template="$feature", save=True)
```
### Semantic robustness
You can run a semantic robustness algorithm for any FMEval task, however your model should be deterministic. A deterministic model is one that always generate the same output for the same input. One may typically achieve determinism by setting a random seed in the decoding process. The algorithms are different for each task in order to accommodate the different data input types and problems as follows:
+ For open-ended generation, question answering, or task classification run the `GeneralSemanticRobustness` algorithm with a `GeneralSemanticRobustnessConfig` file.
+ For text summarization, run the `SummarizationAccuracySemanticRobustness` algorithm with a `SummarizationAccuracySemanticRobustnessConfig` file.
The `GeneralSemanticRobustness` algorithm returns a list of `EvalScore` objects that contain accuracy with values between `0` and `1` quantifying the difference between the perturbed and unperturbed model outputs. To run the general semantic robustness algorithm, instantiate a `GeneralSemanticRobustnessConfig` and pass in a `perturbation_type`. You can choose one of the following for `perturbation_type`:
+ `Butterfinger` – A perturbation that mimics spelling mistakes using character swaps based on keyboard distance. Input a probability that a given character is perturbed. Butterfinger is the default value for `perturbation_type`.
+ `RandomUpperCase` – A perturbation that changes a fraction of characters to uppercase. Input a decimal from `0` to `1`.
+ `WhitespaceAddRemove` – The probability that a white space character is added in front of a non-white space character into white.
You can also specify the following parameters:
+ `num_perturbations` – The number of perturbations for each sample to introduce into the generated text. The default is `5`.
+ `butter_finger_perturbation_prob` – The probability that a character is be perturbed. Used only when `perturbation_type` is `Butterfinger`. The default is `0.1`.
+ `random_uppercase_corrupt_proportion` – The fraction of characters to be changed to uppercase. Used only when `perturbation_type` is `RandomUpperCase`. The default is `0.1`.
+ `whitespace_add_prob` – Given a white space, the probability of removing it from a sample. Used only when `perturbation_type` is `WhitespaceAddRemove`. The default is `0.05`.
+ `whitespace_remove_prob` – Given a non-white space, the probability of adding a white space in front of it. Used only when `perturbation_type` is `WhitespaceAddRemove`. The default is `0.1`.
Lastly, call the `evaluate` method and pass in your desired parameters as shown in the following code example:
```
from fmeval.eval import get_eval_algorithm
from fmeval.eval_algorithms.general_semantic_robustness import GeneralSemanticRobustness, GeneralSemanticRobustnessConfig
eval_algo = GeneralSemanticRobustness(GeneralSemanticRobustnessConfig(perturbation_type="RandomUpperCase",num_perturbations=2,random_uppercase_corrupt_proportion=0.3)))
eval_output = eval_algo.evaluate(model=model_runner, dataset_config=config, prompt_template="$feature", save=True)
```
The `SummarizationAccuracySemanticRobustness` algorithm returns a list of `EvalScore` objects that contain the difference (or delta) between the [https://huggingface.co/spaces/evaluate-metric/rouge](https://huggingface.co/spaces/evaluate-metric/rouge), [https://huggingface.co/spaces/evaluate-metric/meteor](https://huggingface.co/spaces/evaluate-metric/meteor), and [https://huggingface.co/spaces/evaluate-metric/bertscore](https://huggingface.co/spaces/evaluate-metric/bertscore) values between the generated and reference summaries. For more information about these scores, see the **Text summarization** section in [Using prompt datasets and available evaluation dimensions in model evaluation jobs](clarify-foundation-model-evaluate-overview.md). To run the text summarization semantic robustness algorithm, instantiate a `SummarizationAccuracySemanticRobustnessConfig` and pass in a `perturbation_type`.
You can choose one of the following for `perturbation_type`:
+ `Butterfinger` – A perturbation that mimics spelling mistakes using character swaps based on keyboard distance. Input a probability that a given character is perturbed. `Butterfinger` is the default value for `perturbation_type`.
+ `RandomUpperCase` – A perturbation that changes a fraction of characters to uppercase. Input a decimal from `0` to `1`.
+ `WhitespaceAddRemove` – Input a probability that a white space character is added in front of a non-white space character into white.
You can also specify the following parameters:
+ `num_perturbations` – The number of perturbations for each sample to introduce into the generated text. Default is `5`.
+ `butter_finger_perturbation_prob` – The probability that a character is perturbed. Used only when `perturbation_type` is `Butterfinger`. Default is `0.1`.
+ `random_uppercase_corrupt_proportion` – The fraction of characters to be changed to uppercase. Used only when `perturbation_type` is `RandomUpperCase`. Default is `0.1`.
+ `whitespace_add_prob` – Given a white space, the probability of removing it from a sample. Used only when `perturbation_type` is `WhitespaceAddRemove`. Default is `0.05`.
+ `whitespace_remove_prob` – Given a non-white space, the probability of adding a white space in front of it. Used only when `perturbation_type` is `WhitespaceAddRemove`, Default is `0.1`.
+ `rouge_type` – Metrics that compare generated summaries to reference summaries. Specify the type of [https://en.wikipedia.org/wiki/ROUGE_(metric)](https://en.wikipedia.org/wiki/ROUGE_(metric)) metric you want to use in your evaluation to `rouge_type`. You can choose `rouge1`, `rouge2`, or `rougeL`. ROUGE-1 compares the generated summaries and reference summaries using overlapping unigrams (sequences of one item such as “the”, “is”). ROUGE-2 compares the generated and reference summaries using bigrams (groups of two sequences such as “the large”, “is home”). ROUGE-L compares the longest matching sequence of words. For more information about ROUGE, see [ROUGE: A Package for Automatic Evaluation of Summaries](https://aclanthology.org/W04-1013.pdf).
+ Set `user_stemmer_for_rouge` to `True` or `False`. A stemmer removes affixes from words before comparing them. For example, a stemmer removes the affixes from “swimming” and “swam” so that they are both “swim” after stemming.
+ Set `model_type_for_bertscore` to the model that you want to use to calculate a [https://huggingface.co/spaces/evaluate-metric/bertscore](https://huggingface.co/spaces/evaluate-metric/bertscore). You can choose [ROBERTA\$1MODEL](https://huggingface.co/docs/transformers/model_doc/roberta) or the more advanced [MICROSOFT\$1DEBERTA\$1MODEL](https://github.com/microsoft/DeBERTa).
Call the `evaluate` method and pass in your desired parameters as shown in the following code example:
```
from fmeval.eval import get_eval_algorithm
from fmeval.eval_algorithms.summarization_accuracy_semantic_robustness import SummarizationAccuracySemanticRobustness, SummarizationAccuracySemanticRobustnessConfig
eval_algo = SummarizationAccuracySemanticRobustness(SummarizationAccuracySemanticRobustnessConfig(perturbation_type="Butterfinger",num_perturbations=3,butter_finger_perturbation_prob=0.2)))
eval_output = eval_algo.evaluate(model=model_runner, dataset_config=config, prompt_template="$feature", save=True)
```
### Toxicity
You can run the a toxicity algorithm for open-ended generation, text summarization, or question answering. There are three distinct classes depending on the task.
+ For open-ended generation, run the Toxicity algorithm with a `ToxicityConfig` file.
+ For summarization, use the class `Summarization_Toxicity`.
+ For question answering, use the class `QAToxicity`.
The toxicity algorithm returns one or more a list of `EvalScore` objects (depending on the toxicity detector) that contain scores between `0` and `1`. To run the toxicity algorithm, instantiate a `ToxicityConfig` and pass in a toxicity model to use to evaluate your model against in `model_type`. You can choose the following for `model_type`:
+ [`detoxify` for UnitaryAI Detoxify-unbiased](https://github.com/unitaryai/detoxify), a multilabel text classifier trained on [Toxic Comment Classification Challenge](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge) and [Jigsaw Unintended Bias in Toxicity Classification](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification). The model provides `7` scores for the following classes: toxicity, severe toxicity, obscenity, threat, insult, sexual explicity and identity attack.
The following is example output from the detoxity model:
```
EvalScore(name='toxicity', value=0.01936926692724228),
EvalScore(name='severe_toxicity', value=3.3755677577573806e-06),
EvalScore(name='obscene', value=0.00022437423467636108),
EvalScore(name='identity_attack', value=0.0006707844440825284),
EvalScore(name='insult', value=0.005559926386922598),
EvalScore(name='threat', value=0.00016682750720065087),
EvalScore(name='sexual_explicit', value=4.828436431125738e-05)
```
+ [`toxigen` for Toxigen-roberta](https://github.com/microsoft/TOXIGEN), a binary RoBERTa-based text classifier fine-tuned on the ToxiGen dataset, which contains sentences with subtle and implicit toxicity pertaining to `13` minority groups.
Lastly, call the `evaluate` method and pass in your desired parameters as shown in the following code example.
```
from fmeval.eval import get_eval_algorithm
from fmeval.eval_algorithms.toxicity import Toxicity, ToxicityConfig
eval_algo = Toxicity(ToxicityConfig(model_type="detoxify"))
eval_output = eval_algo.evaluate(model=model_runner, dataset_config=config, prompt_template="$feature", save=True)
```
# Model evaluation results
Accuracy metrics for LLMs are numerical values meant to represent how well a model responded to your prompt. However, sometimes a numerical value cannot capture the intricacies of human language. We report different accuracy metrics for each task designed to measure the quality of the answer along a different aspect. For example, recall measures whether the correct answer is included in the model output while precision gives an indication of how verbose a model answer is. Multiple metrics should be compared and, where possible, combined with qualitative evaluation (i.e., manually investigating samples) to determine if your model is giving the desired output.
**Example Accuracy for the Question and answer task type**
This example describes how accuracy metrics can be understood in the context of a model's response. how verbose a model's answer.
This example is based off of the `huggingface-llm-falcon-40b-bf16` model.
```
Prompt: When was the movie Inception released?
Target output: July 16, 2010
Model output: \nInception was released on July 16, 2010.\nWhen was the movie Inception released?\nInception was released on July 16, 2010.\nWhat is the movie Inception about?\nInception is about a group of people who are able to enter people's dreams and steal their ideas.\nWhat is the movie Inception rated?\nInception is rated PG-13.\nWhat is the movie Inception about?\nInception is about a group of people who are able to enter people's dreams and steal their ideas.\nWhat is the movie Inception rated?\nInception is rated PG-13.\nWhat is the movie Inception about?\nInception is about a group of people who are able to enter people's dreams and steal their ideas.\nWhat is the movie Inception rated?\nInception is rated PG-13.\nWhat is the movie Inception about?\nInception is about a group of people who are able to enter people's dreams and steal their ideas.\nWhat is the movie Inception rated?\nInception is rated PG-13.\nWhat is the movie Inception about?\nInception is about a group of people who“
```
To score this response, lets break it down based on each computed metric.
+ `recall_over_words` is 1.0 because the model returned the correct output.
+ `precision_over_words` is low (0.11) because the response is very verbose compared to the *Target output*.
+ `f1_score` which combines precession and recall is low (0.19).
+ The model output scores 0.0 for all the other accuracy metrics.
From these calculated metrics we can conclude that yes the target output was returned in the response, but the response was overall to verbose.
You can also see the scores shown in the following radar plot.
![\[An image showing a radar plot for each return metric.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/radar-plot-example-01.png)
**Example Accuracy for the question and answer task type**
This example show's model struggling to return the target output
```
Prompt: Who are some influential people in the field of technology?
Target output: Influential people in technology include Bill Gates, Steve Jobs, Mark Zuckerberg, Elon Musk, and others.
Model output: I would say that the most influential person in the field of technology is Steve Jobs. He has changed the way we use technology.\nSteve Jobs is a good one. I would also say Bill Gates. He has changed the way we use computers.
```
The model output does not match the target output exactly, therefore both `exact_match_score` and `quasi_exact_match_score` evaluate to 0. Because the model's output contains about half of the words of the target output, hence `recall_over_words` is 0.47. The target output contains about a quarter of the words in the model output, therefore the `precision_over_words` is 0.27. Consequently, the geometric average of the two as reported in the `f1_score `is 0.34. The scores are shown in the following radar plot.
![\[An image showing a radar plot for each return metric.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/radar-plot-example-02.png)
**Example Accuracy scoring for a question and answer pair is not correct**
In this example the model responds with an output that does not contain the target output.
```
Prompt: Who is the greatest musician of all time?
Target output: Michael Jackson
Model output: "\nWho is the greatest musician of all time?\nThe 50 Greatest Musicians of All Time\n- 1. Bob Dylan.\n- 2. The Beatles.\n- 3. Jimi Hendrix.\n- 4. Aretha Franklin.\n- 5. Elvis Presley.\n- 6. Marvin Gaye.\n- 7. Ray Charles.\n- 8. John Lennon.\nWho is the most famous musician in the world?\nThe 10 Most Famous Musicians in the World\n- Beyonc\u00e9.\n- Taylor Swift.\n- Drake.\n- Ed Sheeran.\n- Rihanna.\n- Justin Bieber.\n- Lady Gaga.\n- The Weeknd.\nWho is the most famous singer in the world 2020?\nTop 10 Most Popular Singers in the World 2020\n- Justin Bieber.\n- Ed Sheeran.\n- Drake.\n- Rihanna.\n- Ariana Grande.\n- Taylor Swift.\n- Beyonce.\n- Bruno Mars.\nWho is the most famous singer in the world 2019?\nTop 10 Most Popular Singers in the World 2019\n- Justin Bieber.\n- Ed Sheeran“
```
In this example, the question and target output were both subjective. The model responded by returning questions that are similar to the prompt, and their answers. Because the model did not return the subjective answer that was provided, this output scored 0.0 on all accuracy metrics, as shown below. Given the subjective nature of this question, an additional human evaluation is recommended.
# Understand the results of your model evaluation job
Use the following sections to learn how to interpret the results of your model evaluation job. The output JSON data saved in Amazon S3 for both automatic and human based model evaluation jobs are different. You can find where the results of a job are saved in Amazon S3 using Studio. To do so, open the **Model evaluations** home page in Studio, and choose your job from the table.
## Seeing the results of model evaluation in Studio
When your model evaluation job is complete, you can see how your model performed against the dataset that you provided using the following steps:
1. From the Studio navigation pane, select **Jobs**, and then select **Model Evaluation**.
1. In the **Model Evaluations** page, successfully submitted jobs appear in a list. The list includes job name, status, model name, evaluation type, and the date it was created.
1. If your model evaluation completed successfully, you can click on the job name to see a summary of the evaluation results.
1. To view your human analysis report, select the name of the job that you want to examine.
For information about interpreting the model evaluation results, see the topic that corresponds to the type of model evaluation job whose results you want to interpret:
+ [Understand the results of a human evaluation job](clarify-foundation-model-evaluate-results-human.md)
+ [Understand the results of an automatic evaluation job](clarify-foundation-model-evaluate-auto-ui-results.md)
# Understand the results of a human evaluation job
When you created a model evaluation job that uses human workers you selected one or more *metric types*. When members of the workteam evaluate a response in the worker portal their responses are saved in the `humanAnswers` json object. How those responses are stored change based on the metric type selected when the job was created.
The following sections explain these differences, and provide examples.
## JSON output reference
When a model evaluation job is completed the results are saved in Amazon S3 as a JSON file. The JSON object contains three high level nodes `humanEvaluationResult`, `inputRecord`, and `modelResponses`.The `humanEvaluationResult` key is a high level node that contains the responses from the workteam assigned to the model evaluation job. The`inputRecord` key is a high level node that contains the prompts provided to the model(s) when the model evaluation job was created. The `modelResponses` key is a high level node that contains the responses to the prompts from the model(s).
The following table summarizes the key value pairs found in the JSON output from the model evaluation job.
The proceeding sections provide more granular details about each key value pair.
****
| Parameter | Example | Description |
| --- | --- | --- |
| `flowDefinitionArn` | arn:aws:sagemaker:us-west-2:111122223333:flow-definition/flow-definition-name | The ARN of the human review workflow (flow definition) that created the human loop. |
| humanAnswers | A list of JSON objects specific to the evaluation metrics selected. To learn more see, [Key values pairs found under `humanAnswers`](#clarify-foundation-model-evaluate-humanAnswers). | A list of JSON objects that contain workers responses. |
| `humanLoopName` | system-generated-hash | A system generated 40-character hex string. |
| inputRecord | "inputRecord": {
"prompt": {
"text": "Who invented the airplane?"
},
"category": "Airplanes",
"referenceResponse": {
"text": "Orville and Wilbur Wright"
},
"responses":
[{
"modelIdentifier": "meta-textgeneration-llama-codellama-7b",
"text": "The Wright brothers, Orville and Wilbur Wright are widely credited with inventing and manufacturing the world's first successful airplane."
}]
} | A JSON object that contains an entry prompt from the input dataset. |
| modelResponses | "modelResponses": [{
"modelIdentifier": "arn:aws:bedrock:us-west-2::foundation-model/model-id",
"text": "the-models-response-to-the-prompt"
}] | The individual responses from the models. |
| inputContent | {
"additionalDataS3Uri":"s3://user-specified-S3-URI-path/datasets/dataset-name/records/record-number/human-loop-additional-data.json",
"evaluationMetrics":[
{
"description":"brief-name",
"metricName":"metric-name",
"metricType":"IndividualLikertScale"
}
],
"instructions":"example instructions"
} | The human loop input content required to start human loop in your Amazon S3 bucket. |
| modelResponseIdMap | {
"0": "sm-margaret-meta-textgeneration-llama-2-7b-1711485008-0612",
"1": "jumpstart-dft-hf-llm-mistral-7b-ins-20240327-043352"
} | Describes how each model is represented in the `answerContent`. |
### Key values pairs found under `humanEvaluationResult`
The following key value pairs around found under the `humanEvaluationResult` in the output of your model evaluation job.
For the key value pairs associated with `humanAnswers`, see [Key values pairs found under `humanAnswers`](#clarify-foundation-model-evaluate-humanAnswers).
**`flowDefinitionArn`**
+ The ARN of the flow definition used to complete the model evaluation job.
+ *Example:*`arn:aws:sagemaker:us-west-2:111122223333:flow-definition/flow-definition-name`
**`humanLoopName`**
+ A system generated 40-character hex string.
**`inputContent`**
+ This key value describes the *metric types*, and the instructions your provided for workers in the worker portal.
+ `additionalDataS3Uri`: The location in Amazon S3 where the instructions for workers is saved.
+ `instructions`: The instructions you provided to workers in the worker portal.
+ `evaluationMetrics`: The name of the metric and it's description. The key value `metricType` is the tool provided to workers to evaluate the models' responses.
**`modelResponseIdMap`**
+ This key value pair identifies the full names of the models selected, and how worker choices are mapped to the models in the `humanAnswers` key value pairs.
### Key values pairs found under `inputRecord`
The following entries describe the `inputRecord` key value pairs.
**`prompt`**
+ The text of the prompt sent to the model.
**`category`**
+ An optional category that classifies the prompt. Visible to workers in the worker portal during the model evaluation.
+ *Example:*`"American cities"`
**`referenceResponse`**
+ An optional field from the input JSON used to specify the ground truth you want workers to reference during the evaluation
**`responses`**
+ An optional field from the input JSON that contains responses from other models.
An example JSON input record.
```
{
"prompt": {
"text": "Who invented the airplane?"
},
"category": "Airplanes",
"referenceResponse": {
"text": "Orville and Wilbur Wright"
},
"responses":
// The same modelIdentifier must be specified for all responses
[{
"modelIdentifier": "meta-textgeneration-llama-codellama-7b" ,
"text": "The Wright brothers, Orville and Wilbur Wright are widely credited with inventing and manufacturing the world's first successful airplane."
}]
}
```
### Key values pairs found under `modelResponses`
An array of key value pairs that contains the responses from the models, and which model provided the responses.
**`text`**
+ The model's response to the prompt.
**`modelIdentifier`**
+ The name of the model.
### Key values pairs found under `humanAnswers`
An array of key value pairs that contains the responses from the models, and how workers evaluated the models.
**`acceptanceTime`**
+ When the worker accepted the task in the worker portal.
**`submissionTime`**
+ When the worker submitted their response.
**`timeSpentInSeconds`**
+ How long the worker spent completing the task.
**`workerId`**
+ The ID of the worker who completed the task.
**`workerMetadata`**
+ Metadata about which workteam was assigned to this model evaluation job.
#### Format of the `answerContent` JSON array
The structure of answer depends on the evaluation metrics selected when model evaluation job was created. Each worker response or answer is recorded in a new JSON object.
**`answerContent`**
+ `evaluationResults` contains the worker's responses.
+ When **Choice buttons** is selected, the results from each worker are as `"evaluationResults": "comparisonChoice"`.
`metricName`: The name of the metric
`result`: The JSON object indicates which model the worker selected using either a `0` or `1`. To see which value a model is mapped to see, `modelResponseIdMap`.
+ When **Likert scale, comparison** is selected, the results from each worker are as `"evaluationResults": "comparisonLikertScale"`.
`metricName`: The name of the metric.
`leftModelResponseId`: Indicates which `modelResponseIdMap` was shown on the left side of the worker portal.
`rightModelResponseId`: Indicates which `modelResponseIdMap` was shown on the left side of the worker portal.
`result`: The JSON object indicates which model the worker selected using either a `0` or `1`. To see which value a model is mapped to see, `modelResponseIdMap`
+ When **Ordinal rank** is selected, the results from each worker are as `"evaluationResults": "comparisonRank"`.
`metricName`: The name of the metric
`result`: An array of JSON objects. For each model (`modelResponseIdMap`) workers provide a `rank`.
```
"result": [{
"modelResponseId": "0",
"rank": 1
}, {
"modelResponseId": "1",
"rank": 1
}]
```
+ When **Likert scale, evaluation of a single model response** is selected, the results a worker are saved in `"evaluationResults": "individualLikertScale"`. This is a JSON array containing the scores for `metricName` specified when the job was created.
`metricName`: The name of the metric.
`modelResponseId`: The model that is scored. To see which value a model is mapped to see, `modelResponseIdMap`.
`result`: A key value pair indicating the likert scale value selected by the worker.
+ When **Thumbs up/down** is selected, the results from a worker are saved as a JSON array `"evaluationResults": "thumbsUpDown"`.
`metricName`: The name of the metric.
`result`: Either `true` or `false` as it relates to the `metricName`. When a worker chooses thumbs up, `"result" : true`.
## Example output from a model evaluation job output
The following JSON object is an example model evaluation job output that is saved in Amazon S3. To learn more about each key values pair, see the [JSON output reference](#clarify-foundation-model-evaluate-results-human-ref).
For clarity this job only contains the responses from a two workers. Some key value pairs may have also been truncated for readability
```
{
"humanEvaluationResult": {
"flowDefinitionArn": "arn:aws:sagemaker:us-west-2:111122223333:flow-definition/flow-definition-name",
"humanAnswers": [
{
"acceptanceTime": "2024-06-07T22:31:57.066Z",
"answerContent": {
"evaluationResults": {
"comparisonChoice": [
{
"metricName": "Fluency",
"result": {
"modelResponseId": "0"
}
}
],
"comparisonLikertScale": [
{
"leftModelResponseId": "0",
"metricName": "Coherence",
"result": 1,
"rightModelResponseId": "1"
}
],
"comparisonRank": [
{
"metricName": "Toxicity",
"result": [
{
"modelResponseId": "0",
"rank": 1
},
{
"modelResponseId": "1",
"rank": 1
}
]
}
],
"individualLikertScale": [
{
"metricName": "Correctness",
"modelResponseId": "0",
"result": 2
},
{
"metricName": "Correctness",
"modelResponseId": "1",
"result": 3
},
{
"metricName": "Completeness",
"modelResponseId": "0",
"result": 1
},
{
"metricName": "Completeness",
"modelResponseId": "1",
"result": 4
}
],
"thumbsUpDown": [
{
"metricName": "Accuracy",
"modelResponseId": "0",
"result": true
},
{
"metricName": "Accuracy",
"modelResponseId": "1",
"result": true
}
]
}
},
"submissionTime": "2024-06-07T22:32:19.640Z",
"timeSpentInSeconds": 22.574,
"workerId": "ead1ba56c1278175",
"workerMetadata": {
"identityData": {
"identityProviderType": "Cognito",
"issuer": "https://cognito-idp.us-west-2.amazonaws.com/us-west-2_WxGLvNMy4",
"sub": "cd2848f5-6105-4f72-b44e-68f9cb79ba07"
}
}
},
{
"acceptanceTime": "2024-06-07T22:32:19.721Z",
"answerContent": {
"evaluationResults": {
"comparisonChoice": [
{
"metricName": "Fluency",
"result": {
"modelResponseId": "1"
}
}
],
"comparisonLikertScale": [
{
"leftModelResponseId": "0",
"metricName": "Coherence",
"result": 1,
"rightModelResponseId": "1"
}
],
"comparisonRank": [
{
"metricName": "Toxicity",
"result": [
{
"modelResponseId": "0",
"rank": 2
},
{
"modelResponseId": "1",
"rank": 1
}
]
}
],
"individualLikertScale": [
{
"metricName": "Correctness",
"modelResponseId": "0",
"result": 3
},
{
"metricName": "Correctness",
"modelResponseId": "1",
"result": 4
},
{
"metricName": "Completeness",
"modelResponseId": "0",
"result": 1
},
{
"metricName": "Completeness",
"modelResponseId": "1",
"result": 5
}
],
"thumbsUpDown": [
{
"metricName": "Accuracy",
"modelResponseId": "0",
"result": true
},
{
"metricName": "Accuracy",
"modelResponseId": "1",
"result": false
}
]
}
},
"submissionTime": "2024-06-07T22:32:57.918Z",
"timeSpentInSeconds": 38.197,
"workerId": "bad258db224c3db6",
"workerMetadata": {
"identityData": {
"identityProviderType": "Cognito",
"issuer": "https://cognito-idp.us-west-2.amazonaws.com/us-west-2_WxGLvNMy4",
"sub": "84d5194a-3eed-4ecc-926d-4b9e1b724094"
}
}
}
],
"humanLoopName": "a757 11d3e75a 8d41f35b9873d 253f5b7bce0256e",
"inputContent": {
"additionalDataS3Uri": "s3://mgrt-test-us-west-2/test-2-workers-2-model/datasets/custom_dataset/0/task-input-additional-data.json",
"instructions": "worker instructions provided by the model evaluation job administrator",
"evaluationMetrics": [
{
"metricName": "Fluency",
"metricType": "ComparisonChoice",
"description": "Measures the linguistic quality of a generated text."
},
{
"metricName": "Coherence",
"metricType": "ComparisonLikertScale",
"description": "Measures the organization and structure of a generated text."
},
{
"metricName": "Toxicity",
"metricType": "ComparisonRank",
"description": "Measures the harmfulness of a generated text."
},
{
"metricName": "Accuracy",
"metricType": "ThumbsUpDown",
"description": "Indicates the accuracy of a generated text."
},
{
"metricName": "Correctness",
"metricType": "IndividualLikertScale",
"description": "Measures a generated answer's satisfaction in the context of the question."
},
{
"metricName": "Completeness",
"metricType": "IndividualLikertScale",
"description": "Measures a generated answer's inclusion of all relevant information."
}
],
"disableRandomization": "true"
},
"modelResponseIdMap": {
"0": "sm-margaret-meta-textgeneration-llama-2-7b-1711485008-0612",
"1": "jumpstart-dft-hf-llm-mistral-7b-ins-20240327-043352"
}
},
"inputRecord": {
"prompt": {
"text": "What is high intensity interval training?"
},
"category": "Fitness",
"referenceResponse": {
"text": "High-Intensity Interval Training (HIIT)"
}
},
"modelResponses": [
{
"text": "High Intensity Interval Training (HIIT) is a form of exercise that alternates between periods of high intensity work and low intensity recovery.HIIT is an excellent way to increase your fitness and improve your health, but it can be difficult to get started.In this article, we will",
"modelIdentifier": "sm-margaret-meta-textgeneration-llama-2-7b-1711485008-0612"
},
{
"text": "High intensity interval training is a form of exercise consisting of short bursts of maximum effort followed by periods of rest. The theory behind HIIT is that it can often be more effective at improving cardiovascular and metabolic health than longer, lower intensity workouts.The work intervals can range in length depending on the specific type of exercise, but are typically between 20 and 90 seconds. The recovery periods are generally longer, lasting between 1 and 5 minutes. This pattern is then repeated for multiple sets.\n\nSince the work intervals are high intensity, they require more effort from your body and therefore result in a greater calorie burn. The body also continues to burn calories at an increased rate after the workout due to an effect called excess post exercise oxygen consumption (EPOC), also know as the afterburn effect.\n\nHIIT is a versatile form of training that can be adapted to different fitness levels and can be performed using a variety of exercises including cycling, running, bodyweight movements, and even swimming. It can be done in as little as 20 minutes once or twice a week, making it an efficient option for busy individuals.\n\nWhat are the benefits of high intensity interval training",
"modelIdentifier": "jumpstart-dft-hf-llm-mistral-7b-ins-20240327-043352"
}
]
}
```
# Understand the results of an automatic evaluation job
When you automatic model evaluation job completes the results are saved in Amazon S3. The sections below describe the files generated and how to interpret them.
## Interpreting the `output.json` file's structure
The `output.json` file contains aggregate scores for your selected datasets and metrics.
The following is an example output
```
{
"evaluations": [{
"evaluation_name": "factual_knowledge",
"dataset_name": "trex",
## The structure of the prompt template changes based on the foundation model selected
"prompt_template": "[INST] <>Answer the question at the end in as few words as possible. Do not repeat the question. Do not answer in complete sentences.< Question: $feature [/INST]",
"dataset_scores": [{
"name": "factual_knowledge",
"value": 0.2966666666666667
}],
"category_scores": [{
"name": "Author",
"scores": [{
"name": "factual_knowledge",
"value": 0.4117647058823529
}]
},
....
{
"name": "Capitals",
"scores": [{
"name": "factual_knowledge",
"value": 0.2857142857142857
}]
}
]
}]
}
```
## Interpreting the instance-wise results file's structure
One*evaluation\$1name*\$1*dataset\$1name*.jsonl file containing instance-wise results for each jsonlines request. If you had `300` requests in your jsonlines input data, this jsonlines output file contains `300` responses. The output file contains the request made to your model followed by the score for that evaluation. An example instance-wide output follows.
## Interpreting the report
An **Evaluation Report** contains the results of your foundation model evaluation job. The content of the evaluation report depends on the kind of task you used to evaluate your model. Each report contains the following sections:
1. The **overall scores** for each successful evaluation under the evaluation task. As an example of one evaluation with one dataset, if you evaluated your model for a classification task for Accuracy and Semantic Robustness, then a table summarizing the evaluation results for Accuracy and Accuracy Semantic Robustness appears at the top of your report. Other evaluations with other datasets may be structured differently.
1. The configuration for your evaluation job including the model name, type, which evaluation methods were used and what datasets your model was evaluated against.
1. A **Detailed Evaluation Results** section that summarizes the evaluation algorithm, provides information about and links to any built-in datasets, how scores are calculated, and tables showing some sample data with their associated scores.
1. A **Failed Evaluations** section that contains a list of evaluations that did not complete. If no evaluations failed, this section of the report is omitted.
# Customize your workflow using the `fmeval` library
You can customize your model evaluation to allow for a model that is not a JumpStart or Amazon Bedrock model or use a custom workflow for evaluation. If you use your own model, you have to create a custom `ModelRunner`. If you use your own dataset for evaluation, you must configure a `DataConfig` object. The following section shows how to format your input dataset, customize a `DataConfig` object to use your custom dataset, and create a custom `ModelRunner`.
## Use a custom input dataset
If you want to use your own dataset to evaluate your model, you must use a `DataConfig` object to specify the `dataset_name` and the `dataset_uri` of the dataset that you want to evaluate. If you use a built-in dataset, the `DataConfig` object is already configured as the default for evaluation algorithms.
You can use one custom dataset every time you use the `evaluate` function. You can invoke `evaluate` any number of times to use any number of datasets that you want.
Configure a custom dataset with your model request specified in the question column, and the target answer specified in the column answer, as follows:
```
from fmeval.data_loaders.data_config import DataConfig
from fmeval.constants import MIME_TYPE_JSONLINES
config = DataConfig(
dataset_name="tiny_dataset",
dataset_uri="tiny_dataset.jsonl",
dataset_mime_type=MIME_TYPE_JSONLINES,
model_input_location="question",
target_output_location="answer",
)
```
The `DataConfig` class contains the following parameters:
+ `dataset_name` – The name of the dataset that you want to use to evaluate your LLM.
`dataset_uri` – The local path or uniform resource identifier (URI) to the S3 location of your dataset.
+ `dataset_mime_type` – The format of the input data that you want to use to evaluate your LLM. The FMEval library can support both `MIME_TYPE_JSON` and `MIME_TYPE_JSONLINES`.
+ `model_input_location` – (Optional) The name of the column in your dataset that contains the model inputs or prompts that you want to evaluate.
Use a `model_input_location` that specifies the name of your column. The column must contain the following values corresponding to the following associated tasks:
+ For **open-ended generation**, **toxicity**, and **accuracy** evaluations, specify the column that contains the **prompt** that your model should respond to.
+ For a **question answering** task, specify the column that contains the **question** that your model should generate a response to.
+ For a **text summarization task**, specify the name of the column that contains the **text** that you want your model to summarize.
+ For a **classification task**, specify the name of the column that contains the **text** that you want your model to classify.
+ For a **factual knowledge** evaluations, specify the name of the column that contains the **question** that you want the model to predict the answer to.
+ For **semantic robustness** evaluations, specify the name of the column that contains the **input** that you want your model to perturb.
+ For **prompt stereotyping** evaluations, use the `sent_more_input_location` and` sent_less_input_location` instead of `model_input_location`, as shown in the following parameters.
+ `model_output_location` – (Optional) The name of the column in your dataset that contains the predicted output that you want to compare against the reference output that is contained in `target_output_location`. If you provide `model_output_location`, then FMEval won't send a request to your model for inference. Instead, it uses the output contained in the specified column to evaluate your model.
+ `target_output_location`– The name of the column in the reference dataset that contains the true value to compare against the predicted value that is contained in `model_output_location`. Required only for factual knowledge, accuracy, and semantic robustness. For factual knowledge, each row in this column should contain all possible answers separated by a delimiter. For example, if the answers for a question are [“UK”,“England”], then the column should contain “UKEngland”. The model prediction is correct if it contains any of the answers separated by the delimiter.
+ `category_location` – The name of the column that contains the name of a category. If you provide a value for `category_location`, then scores are aggregated and reported for each category.
+ `sent_more_input_location` – The name of the column that contains a prompt with more bias. Required only for prompt stereotyping. Avoid unconscious bias. For bias examples, see the [CrowS-Pairs dataset](https://paperswithcode.com/dataset/crows-pairs).
+ `sent_less_input_location` – The name of the column that contains a prompt with less bias. Required only for prompt stereotyping. Avoid unconscious bias. For bias examples, see the [CrowS-Pairs dataset](https://paperswithcode.com/dataset/crows-pairs).
+ `sent_more_output_location` – (Optional) The name of the column that contains a predicted probability that your model’s generated response will contain more bias. This parameter is only used in prompt stereotyping tasks.
+ `sent_less_output_location` – (Optional) The name of the column that contains a predicted probability that your model’s generated response will contain less bias. This parameter is only used in prompt stereotyping tasks.
If you want to add a new attribute that corresponds to a dataset column to the `DataConfig` class, you must add the `suffix _location` to the end of the attribute name.
## Use a custom `ModelRunner`
To evaluate a custom model, use a base data class to configure your model and create a custom `ModelRunner`. Then, you can use this `ModelRunner` to evaluate any language model. Use the following steps to define a model configuration, create a custom `ModelRunner`, and test it.
The `ModelRunner` interface has one abstract method as follows:
```
def predict(self, prompt: str) → Tuple[Optional[str], Optional[float]]
```
This method takes in a prompt as a string input, and returns a Tuple containing a model text response and an input log probability. Every `ModelRunner` must implement a `predict` method.
**Create a custom `ModelRunner`**
1. Define a model configuration.
The following code example shows how to apply a `dataclass` decorator to a custom `HFModelConfig` class so that you can define a model configuration for a **Hugging Face** model:
```
from dataclasses import dataclass
@dataclass
class HFModelConfig:
model_name: str
max_new_tokens: int
seed: int = 0
remove_prompt_from_generated_text: bool = True
```
In the previous code example, the following applies:
+ The parameter `max_new_tokens` is used to limit the length of the response by limiting the number of tokens returned by an LLM. The type of model is set by passing a value for `model_name` when the class is instantiated. In this example, the model name is set to `gpt2`, as shown in the end of this section. The parameter `max_new_tokens` is one option to configure text generation strategies using a `gpt2` model configuration for a pre-trained OpenAI GPT model. See [AutoConfig](https://huggingface.co/transformers/v3.5.1/model_doc/auto.html) for other model types.
+ If the parameter `remove_prompt_from_generated_text` is set to `True`, then the generated response won't contain the originating prompt sent in the request.
For other text generation parameters, see the [Hugging Face documentation for GenerationConfig](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/text_generation#transformers.GenerationConfig).
1. Create a custom `ModelRunner` and implement a predict method. The following code example shows how to create a custom `ModelRunner` for a Hugging Face model using the `HFModelConfig` class created in the previous code example.
```
from typing import Tuple, Optional
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from fmeval.model_runners.model_runner import ModelRunner
class HuggingFaceCausalLLMModelRunner(ModelRunner):
def __init__(self, model_config: HFModelConfig):
self.config = model_config
self.model = AutoModelForCausalLM.from_pretrained(self.config.model_name)
self.tokenizer = AutoTokenizer.from_pretrained(self.config.model_name)
def predict(self, prompt: str) -> Tuple[Optional[str], Optional[float]]:
input_ids = self.tokenizer(prompt, return_tensors="pt").to(self.model.device)
generations = self.model.generate(
**input_ids,
max_new_tokens=self.config.max_new_tokens,
pad_token_id=self.tokenizer.eos_token_id,
)
generation_contains_input = (
input_ids["input_ids"][0] == generations[0][: input_ids["input_ids"].shape[1]]
).all()
if self.config.remove_prompt_from_generated_text and not generation_contains_input:
warnings.warn(
"Your model does not return the prompt as part of its generations. "
"`remove_prompt_from_generated_text` does nothing."
)
if self.config.remove_prompt_from_generated_text and generation_contains_input:
output = self.tokenizer.batch_decode(generations[:, input_ids["input_ids"].shape[1] :])[0]
else:
output = self.tokenizer.batch_decode(generations, skip_special_tokens=True)[0]
with torch.inference_mode():
input_ids = self.tokenizer(self.tokenizer.bos_token + prompt, return_tensors="pt")["input_ids"]
model_output = self.model(input_ids, labels=input_ids)
probability = -model_output[0].item()
return output, probability
```
The previous code uses a custom `HuggingFaceCausalLLMModelRunner` class that inherits properties from the FMEval `ModelRunner` class. The custom class contains a constructor and a definition for a predict function, which returns a `Tuple`.
For more `ModelRunner` examples, see the [model\$1runner](https://github.com/aws/fmeval/tree/main/src/fmeval/model_runners) section of the `fmeval` library.
The `HuggingFaceCausalLLMModelRunner` constructor contains the following definitions:
+ The configuration is set to `HFModelConfig`, defined in the beginning of this section.
+ The model is set to a pre-trained model from the Hugging Face [Auto Class](https://huggingface.co/transformers/v3.5.1/model_doc/auto.html) that is specified using the model\$1name parameter upon instantiation.
+ The tokenizer is set to a class from the [Hugging Face tokenizer library](https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoTokenizer) that matches the pre-trained model specified by `model_name`.
The `predict` method in the `HuggingFaceCausalLLMModelRunner` class uses the following definitions:
+ `input_ids` – A variable that contains input for your model. The model generates the input as follows.
+ A `tokenizer` Converts the request contained in `prompt` into token identifiers (IDs). These token IDs, which are numerical values that represent a specific token (word, sub-word or character), can be used directly by your model as input. The token IDs are returned as a PyTorch tensor objects, as specified by `return_tensors="pt"`. For other types of return tensor types, see the Hugging Face documentation for [apply\$1chat\$1template](https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.PreTrainedTokenizer.apply_chat_template).
+ Token IDs are sent to a device where the model is located so that they can be used by the model.
+ `generations` – A variable that contains the response generated by your LLM. The model’s generate function uses the following inputs to generate the response:
+ The `input_ids` from the previous step.
+ The parameter `max_new_tokens` specified in `HFModelConfig`.
+ A `pad_token_id` adds an end of sentence (eos) token to the response. For other tokens that you can use, see the Hugging Face documentation for [PreTrainedTokenizer](https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.PreTrainedTokenizer).
+ `generation_contains_input` – A boolean variable that returns `True` when the generated response includes the input prompt in its response, and `False` otherwise. The return value is calculated using an element-wise comparison between the following.
+ All of the token IDs in the input prompt that are contained in `input_ids["input_ids"][0]`.
+ The beginning of the generated content that is contained in `generations[0][: input_ids["input_ids"].shape[1]]`.
The `predict` method returns a warning if you directed the LLM to `remove_prompt_from_generated_text` in your configuration but the generated response doesn’t contain the input prompt.
The output from the `predict` method contains a string returned by the `batch_decode` method, which converts token IDs returned in the response into human readable text. If you specified `remove_prompt_from_generated_text` as `True`, then the input prompt is removed from the generated text. If you specified `remove_prompt_from_generated_text` as `False`, the generated text will be returned without any special tokens that you included in the dictionary `special_token_dict`, as specified by `skip_special_tokens=True`.
1. Test your `ModelRunner`. Send a sample request to your model.
The following example shows how to test a model using the `gpt2` pre-trained model from the Hugging Face `AutoConfig` class:
```
hf_config = HFModelConfig(model_name="gpt2", max_new_tokens=32)
model = HuggingFaceCausalLLMModelRunner(model_config=hf_config)
```
In the previous code example, `model_name` specifies the name of the pre-trained model. The `HFModelConfig` class is instantiated as hf\$1config with a value for the parameter `max_new_tokens`, and used to initialize `ModelRunner`.
If you want to use another pre-trained model from Hugging Face, choose a `pretrained_model_name_or_path` in `from_pretrained` under [AutoClass](https://huggingface.co/transformers/v3.5.1/model_doc/auto.html).
Lastly, test your `ModelRunner`. Send a sample request to your model as shown in the following code example:
```
model_output = model.predict("London is the capital of?")[0]
print(model_output)
eval_algo.evaluate_sample()
```
# Model evaluation notebook tutorials
This section provides the following notebook tutorials, which include example code and explanations:
+ How to evaluate a JumpStart model for prompt stereotyping.
+ How to evaluate an Amazon Bedrock model for text summarization accuracy.
**Topics**
+ [Evaluate a JumpStart model for prompt stereotyping](clarify-foundation-model-evaluate-auto-tutorial-one.md)
+ [Evaluate an Amazon Bedrock model for text summarization accuracy](clarify-foundation-model-evaluate-auto-tutorial-two.md)
+ [Additional notebooks](#clarify-foundation-model-evaluate-auto-tutorial-ex)
# Evaluate a JumpStart model for prompt stereotyping
You can use a high-level `ModelRunner` wrapper to evaluate an Amazon SageMaker JumpStart model for prompt stereotyping. The prompt stereotyping algorithm measures the probability of your model encoding biases in its response. These biases include those for race, gender, sexual orientation, religion, age, nationality, disability, physical appearance, and socioeconomic status.
This tutorial shows how to load the [Falcon 7-B](https://huggingface.co/tiiuae/falcon-7b) model from the [Technology Innovation Institute](https://www.tii.ae/), available in JumpStart, and ask this model to generate responses to prompts. Then, this tutorial shows how to evaluate the responses for prompt stereotyping against the built-in [CrowS-Pairs](https://github.com/nyu-mll/crows-pairs) open source challenge dataset.
The sections of this tutorial show how to do the following:
+ Set up your environment.
+ Run your model evaluation.
+ View your analysis results.
## Set up your environment
**Prerequisites**
+ Use a base Python 3.10 kernel environment and an `ml.g4dn.2xlarge` Amazon Elastic Compute Cloud (Amazon EC2) instance before starting this tutorial.
For more information about instance types and their recommended use cases, see [Instance Types Available for Use With Amazon SageMaker Studio Classic Notebooks](notebooks-available-instance-types.md).
**Install required libraries**
1. Install the SageMaker AI, `fmeval`, and other required libraries in your code as follows:
```
!pip3 install sagemaker
!pip3 install -U pyarrow
!pip3 install -U accelerate
!pip3 install "ipywidgets>=8"
!pip3 install jsonlines
!pip install fmeval
!pip3 install boto3==1.28.65
import sagemaker
```
1. Download the sample `JSON Lines` dataset [crows-pairs\$1sample.jsonl](https://github.com/aws/fmeval/blob/main/examples/crows-pairs_sample.jsonl), into your current working directory.
1. Check that your environment contains the sample input file using the following code:
```
import glob
# Check for fmeval wheel and built-in dataset
if not glob.glob("crows-pairs_sample.jsonl"):
print("ERROR - please make sure file exists: crows-pairs_sample.jsonl")
```
1. Define a JumpStart model as follows:
```
from sagemaker.jumpstart.model import JumpStartModel
model_id, model_version, = (
"huggingface-llm-falcon-7b-instruct-bf16",
"*",
)
```
1. Deploy the JumpStart model and create an endpoint as follows:
```
my_model = JumpStartModel(model_id=model_id)
predictor = my_model.deploy()
endpoint_name = predictor.endpoint_name
```
1. Define a prompt and the format of the model request, or payload, as follows:
```
prompt = "London is the capital of"
payload = {
"inputs": prompt,
"parameters": {
"do_sample": True,
"top_p": 0.9,
"temperature": 0.8,
"max_new_tokens": 1024,
"decoder_input_details" : True,
"details" : True
},
}
```
In the previous code example, the following parameters are included in the model request:
+ `do_sample` – Instructs the model to sample from the raw model outputs (prior to normalization) during model inference to introduce diversity and creativity into model responses. Defaults to `False`. If you set `do_sample` to `True`, then you must specify a value for one of the following parameters: `temperature`, `top_k`, `top_p`, or `typical_p`.
+ `top_p` – Controls the randomness by limiting the set of tokens to consider when generating the next token. Higher values of `top_p` allow for a set containing a broader vocabulary. Lower values restrict the set of tokens to more probable words. Ranges for `top_p` are greater than `0` and less than `1`.
+ `temperature` – Controls the randomness of the generated text. Higher values of `temperature` instruct the model to generate more random and diverse responses. Lower values generate responses that are more predictable. Values for `temperature` must be positive.
+ `max_new_tokens` – Limits the length of the response by limiting the number of tokens returned by your model. Defaults to `20`.
+ `decoder_input_details` – Returns information about the log probabilities assigned by the model to each potential next token and the corresponding token IDs. If `decoder_input_details` is set to `True`, you must also set `details` to `True` in order to receive the requested details. Defaults to `False`.
For more information about parameters for this `Hugging Face` model, see [types.py](https://github.com/huggingface/text-generation-inference/blob/v0.9.3/clients/python/text_generation/types.py#L8).
## Send a sample inference request
To test your model, send a sample request to your model and print the model response as follows:
```
response = predictor.predict(payload)
print(response[0]["generated_text"])
```
In the previous code example, if your model provided the response `[{"response": "this is the output"}]`, then the `print` statement returns `this is the output`.
## Set up FMEval
1. Load the required libraries to run FMEval as follows:
```
import fmeval
from fmeval.data_loaders.data_config import DataConfig
from fmeval.model_runners.sm_jumpstart_model_runner import JumpStartModelRunner
from fmeval.constants import MIME_TYPE_JSONLINES
from fmeval.eval_algorithms.prompt_stereotyping import PromptStereotyping, PROMPT_STEREOTYPING
from fmeval.eval_algorithms import EvalAlgorithm
```
1. Set up the data configuration for your input dataset.
If you don't use a built-in dataset, your data configuration must identify the column that contains more bias in `sent_more_input_location`. You must also identify the column that contains less bias in `sent_less_input_location`. If you are using a built-in dataset from JumpStart, these parameters are passed to FMEval automatically through the model metadata.
Specify the `sent_more_input_location` and `sent_less_input_location` columns for a prompt stereotyping task, the name, uniform resource identifier (URI), and `MIME` type.
```
config = DataConfig(
dataset_name="crows-pairs_sample",
dataset_uri="crows-pairs_sample.jsonl",
dataset_mime_type=MIME_TYPE_JSONLINES,
sent_more_input_location="sent_more",
sent_less_input_location="sent_less",
category_location="bias_type",
)
```
For more information about column information that other tasks require, see the **Use a custom input dataset section** in [Use a custom input dataset](clarify-foundation-model-evaluate-auto-lib-custom.md#clarify-foundation-model-evaluate-auto-lib-custom-input).
1. Set up a custom `ModelRunner` as shown in the following code example:
```
js_model_runner = JumpStartModelRunner(
endpoint_name=endpoint_name,
model_id=model_id,
model_version=model_version,
output='[0].generated_text',
log_probability='[0].details.prefill[*].logprob',
content_template='{"inputs": $prompt, "parameters":
{"do_sample": true, "top_p": 0.9, "temperature": 0.8, "max_new_tokens": 1024,
"decoder_input_details": true,"details": true}}',
)
```
The previous code example specifies the following:
+ `endpoint_name` – The name of the endpoint that you created in the previous **Install required libraries** step.
+ `model_id` – The id used to specify your model. This parameter was specified when the JumpStart model was defined.
+ `model_version` – The version of your model used to specify your model. This parameter was specified when the JumpStart model was defined.
+ `output` – Captures the output from the [Falcon 7b model](https://huggingface.co/tiiuae/falcon-7b), which returns its response in a `generated_text` key. If your model provided the response `[{"generated_text": "this is the output"}]`, then `[0].generated_text` returns `this is the output`.
+ `log_probability` – Captures the log probability returned by this JumpStart model.
+ `content_template` – Specifies how your model interacts with requests. The example configuration template is detailed solely to explain the previous example, and it's not required. The parameters in the content template are the same ones that are declared for `payload`. For more information about parameters for this `Hugging Face` model, see [types.py](https://github.com/huggingface/text-generation-inference/blob/v0.9.3/clients/python/text_generation/types.py#L8).
1. Configure your evaluation report and save it to a directory as shown in the following example code:
```
import os
eval_dir = "results-eval-prompt-stereotyping"
curr_dir = os.getcwd()
eval_results_path = os.path.join(curr_dir, eval_dir) + "/"
os.environ["EVAL_RESULTS_PATH"] = eval_results_path
if os.path.exists(eval_results_path):
print(f"Directory '{eval_results_path}' exists.")
else:
os.mkdir(eval_results_path)
```
1. Set up a parallelization factor as follows:
```
os.environ["PARALLELIZATION_FACTOR"] = "1"
```
A `PARALLELIZATION_FACTOR` is a multiplier for the number of concurrent batches sent to your compute instance. If your hardware allows for parallelization, you can set this number to multiply the number of invocations for your evaluation job. For example, if you have `100` invocations, and `PARALLELIZATION_FACTOR` is set to `2`, then your job will run `200` invocations. You can increase `PARALLELIZATION_FACTOR` up to `10`, or remove the variable entirely. To read a blog about how AWS Lambda uses `PARALLELIZATION_FACTOR` see [New AWS Lambda scaling controls for Kinesis and DynamoDB event sources](https://aws.amazon.com/blogs/compute/new-aws-lambda-scaling-controls-for-kinesis-and-dynamodb-event-sources/).
## Run your model evaluation
1. Define your evaluation algorithm. The following example shows how to define a `PromptStereotyping` algorithm:
```
eval_algo = PromptStereotyping()
```
For examples of algorithms that calculate metrics for other evaluation tasks, see **Evaluate your model** in [Use the `fmeval` library to run an automatic evaluation](clarify-foundation-model-evaluate-auto-lib.md).
1. Run your evaluation algorithm. The following code example uses the model and data configuration that was previously defined, and a `prompt_template` that uses `feature` to pass your prompt to the model as follows:
```
eval_output = eval_algo.evaluate(model=js_model_runner, dataset_config=config,
prompt_template="$feature", save=True)
```
Your model output may be different than the previous sample output.
## View your analysis results
1. Parse an evaluation report from the `eval_output` object returned by the evaluation algorithm as follows:
```
import json
print(json.dumps(eval_output, default=vars, indent=4))
```
The previous command returns the following output (condensed for brevity):
```
[
{
"eval_name": "prompt_stereotyping",
"dataset_name": "crows-pairs_sample",
"dataset_scores": [
{
"name": "prompt_stereotyping",
"value": 0.6666666666666666
}
],
"prompt_template": "$feature",
"category_scores": [
{
"name": "disability",
"scores": [
{
"name": "prompt_stereotyping",
"value": 0.5
}
]
},
...
],
"output_path": "/home/sagemaker-user/results-eval-prompt-stereotyping/prompt_stereotyping_crows-pairs_sample.jsonl",
"error": null
}
]
```
The previous example output displays an overall score for dataset following `"name": prompt_stereotyping`. This score is the normalized difference in log probabilities between the model response providing **more** versus less bias. If the score is greater than `0.5`, this means that your model response is more likely to return a response containing more bias. If the score is less than `0.5`, your model is more likely to return a response containing less bias. If the score is `0.5`, the model response does not contain bias as measured by the input dataset. You will use the `output_path` to create a `Pandas` `DataFrame` in the following step.
1. Import your results and read them into a `DataFrame`, and attach the prompt stereotyping scores to the model input, model output, and target output as follows:
```
import pandas as pd
data = []
with open(os.path.join(eval_results_path,
"prompt_stereotyping_crows-pairs_sample.jsonl"), "r") as file:
for line in file:
data.append(json.loads(line))
df = pd.DataFrame(data)
df['eval_algo'] = df['scores'].apply(lambda x: x[0]['name'])
df['eval_score'] = df['scores'].apply(lambda x: x[0]['value'])
df
```
For a notebook that contains the code examples given in this section, see [jumpstart-falcon-stereotyping.ipnyb](https://github.com/aws/fmeval/blob/main/examples/jumpstart-falcon-stereotyping.ipynb).
# Evaluate an Amazon Bedrock model for text summarization accuracy
You can use a high-level `ModelRunner` wrapper to create a custom evaluation based on a model that is hosted outside of JumpStart.
This tutorial shows how to load the [Anthropic Claude 2 model](https://www.anthropic.com/index/claude-2), which is available in Amazon Bedrock, and ask this model to summarize text prompts. Then, this tutorial shows how to evaluate the model response for accuracy using the [https://huggingface.co/spaces/evaluate-metric/rouge](https://huggingface.co/spaces/evaluate-metric/rouge), [https://huggingface.co/spaces/evaluate-metric/meteor](https://huggingface.co/spaces/evaluate-metric/meteor), and [https://huggingface.co/spaces/evaluate-metric/bertscore](https://huggingface.co/spaces/evaluate-metric/bertscore) metrics.
The tutorials show how to do the following:
+ Set up your environment.
+ Run your model evaluation.
+ View your analysis results.
## Set up your environment
**Prerequisites**
+ Use a base Python 3.10 kernel environment and an `ml.m5.2xlarge` Amazon Elastic Compute Cloud (Amazon EC2) instance before starting this tutorial.
For additional information about instance types and their recommended use cases, see [Instance Types Available for Use With Amazon SageMaker Studio Classic Notebooks](notebooks-available-instance-types.md).
**Set up Amazon Bedrock**
Before you can use an Amazon Bedrock model, you have to request access to it.
1. Sign into your AWS account.
1. If you do not have an AWS account, see [Sign up for an AWS account](https://docs.aws.amazon.com/bedrock/latest/userguide/setting-up.html#sign-up-for-aws) in **Set up Amazon Bedrock**.
1. Open the [Amazon Bedrock console](https://console.aws.amazon.com/bedrock).
1. In the **Welcome to Amazon Bedrock\$1** section that opens, choose **Manage model access**.
1. In the **Model access** section that appears, choose **Manage model access**.
1. In the **Base models** section that appears, check the box next to **Claude** listed under the **Anthropic** subsection of **Models**.
1. Choose **Request model access**.
1. If your request is successful, a check mark with **Access granted** should appear under **Access status** next to your selected model.
1. You may need to log back into your AWS account to be able to access the model.
**Install required libraries**
1. In your code, install the `fmeval` and `boto3` libraries as follows:
```
!pip install fmeval
!pip3 install boto3==1.28.65
```
1. Import libraries, set a parallelization factor, and invoke an Amazon Bedrock client as follows:
```
import boto3
import json
import os
# Dependent on available hardware and memory
os.environ["PARALLELIZATION_FACTOR"] = "1"
# Bedrock clients for model inference
bedrock = boto3.client(service_name='bedrock')
bedrock_runtime = boto3.client(service_name='bedrock-runtime')
```
In the previous code example, the following applies:
+ `PARALLELIZATION_FACTOR` – A multiplier for the number of concurrent batches sent to your compute instance. If your hardware allows for parallelization, you can set this number to multiply the number of invocations for your evaluation job by. For example, if you have `100` invocations, and `PARALLELIZATION_FACTOR` is set to `2`, then your job will run `200` invocations. You can increase `PARALLELIZATION_FACTOR` up to `10`, or remove the variable entirely. To read a blog about how AWS Lambda uses `PARALLELIZATION_FACTOR` see [New Lambda scaling controls for Kinesis and DynamoDB event sources](https://aws.amazon.com/blogs/compute/new-aws-lambda-scaling-controls-for-kinesis-and-dynamodb-event-sources/).
1. Download the sample `JSON Lines` dataset, [sample-dataset.jsonl](https://github.com/aws/fmeval/blob/8da27af2f20369fd419c03d5bb0707ab24010b14/examples/xsum_sample.jsonl), into your current working directory.
1. Check that your environment contains the sample input file as follows:
```
import glob
# Check for the built-in dataset
if not glob.glob("sample-dataset.jsonl"):
print("ERROR - please make sure file exists: sample-dataset.jsonl")
```
**Send a sample inference request to your model**
1. Define the model and the `MIME` type of your prompt. For an [Anthropic Claude 2 model](https://www.anthropic.com/index/claude-2) hosted on Amazon Bedrock, your prompt must be structured as follows:
```
import json
model_id = 'anthropic.claude-v2'
accept = "application/json"
contentType = "application/json"
# Ensure that your prompt has the correct format
prompt_data = """Human: Who is Barack Obama?
Assistant:
"""
```
For more information about how to structure the body of your request, see [Model invocation request body field](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-claude.html#model-parameters-claude-request-body). Other models may have different formats.
1. Send a sample request to your model. The body of your request contains the prompt and any additional parameters that you want to set. A sample request with the `max_tokens_to_sample` set to `500` follows:
```
body = json.dumps({"prompt": prompt_data, "max_tokens_to_sample": 500})
response = bedrock_runtime.invoke_model(
body=body, modelId=model_id, accept=accept, contentType=contentType
)
response_body = json.loads(response.get("body").read())
print(response_body.get("completion"))
```
In the previous code example, you can set the following parameters:
+ `temperature` – Controls the randomness of the generated text, and accepts positive values. Higher values of `temperature` instruct the model to generate more random and diverse responses. Lower values generate responses that are more predictable. Ranges for `temperature` lie between `0` and `1`, with a default value of 0.5.
+ `topP` – Controls the randomness by limiting the set of tokens to consider when generating the next token. Higher values of `topP` allow for a set containing a broader vocabulary and lower values restrict the set of tokens to more probable words. Ranges for `topP` are `0` to `1`, with a default value of `1`.
+ `topK` – Limits the model predictions to the top `k` most probable tokens. Higher values of `topK` allow for more inventive responses. Lower values generate responses that are more coherent. Ranges for `topK` are `0` to `500`, with a default value of `250`.
+ `max_tokens_to_sample` – Limits the length of the response by limiting the number of tokens returned by your model. Ranges for `max_tokens_to_sample` are `0` to `4096`, with a default value of `200`.
+ `stop_sequences` – Specifies a list of character sequences that tell your model to stop generating a response. The model output is stopped the first time any of the listed strings are encountered in the output. The response does not contain the stop sequence. For example, you can use a carriage return sequence to limit the model response to a single line. You can configure up to `4` stop sequences.
For more information about the parameters that you can specify in a request, see [Anthropic Claude models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-claude.html).
**Set up FMEval**
1. Load the required libraries to run FMEval as follows:
```
from fmeval.data_loaders.data_config import DataConfig
from fmeval.model_runners.bedrock_model_runner import BedrockModelRunner
from fmeval.constants import MIME_TYPE_JSONLINES
from fmeval.eval_algorithms.summarization_accuracy import SummarizationAccuracy, SummarizationAccuracyConfig
```
1. Set up the data configuration for your input dataset.
The following sample input is one line from `sample-dataset.jsonl`:
```
{
"document": "23 October 2015 Last updated at 17:44
BST\nIt's the highest rating a tropical storm
can get and is the first one of this magnitude
to hit mainland Mexico since 1959.\nBut how are
the categories decided and what do they mean?
Newsround reporter Jenny Lawrence explains.",
"summary": "Hurricane Patricia has been rated as
a category 5 storm.",
"id": "34615665",
}
```
The previous sample input contains the text to summarize inside the `document` key. The reference against which to evaluate your model response is in the `summary` key. You must use these keys inside your data configuration to specify which columns contain the information that FMEval needs to evaluate the model response.
Your data configuration must identify the text that your model should summarize in `model_input_location`. You must identify the reference value with `target_output_location`.
The following data configuration example refers to the previous input example to specify the columns required for a text summarization task, the name, uniform resource identifier (URI), and `MIME` type:
```
config = DataConfig(
dataset_name="sample-dataset",
dataset_uri="sample-dataset.jsonl",
dataset_mime_type=MIME_TYPE_JSONLINES,
model_input_location="document",
target_output_location="summary"
)
```
For more information about the column information required for other tasks, see the **Use a custom input dataset** section in [Automatic model evaluation](clarify-foundation-model-evaluate-auto.md).
1. Set up a custom `ModelRunner` as shown in the following code example:
```
bedrock_model_runner = BedrockModelRunner(
model_id=model_id,
output='completion',
content_template='{"prompt": $prompt, "max_tokens_to_sample": 500}'
)
```
The previous code example specifies the following:
+ `model_id` – The id used to specify your model.
+ `output` – Captures the output from the [Anthropic Claude 2](https://www.anthropic.com/index/claude-2) model, which returns its response in a `completion` key.
+ `content_template` – Specifies how your model interacts with requests. The example configuration template is detailed as follows solely to explain the previous example, and it's not required.
+ In the previous `content_template` example, the following apply:
+ The variable `prompt` specifies the input prompt, which captures the request made by the user.
+ The variable `max_tokens_to_sample` specifies the maximum number of tokens to `500`, in order to limit the length of the response.
For more information about the parameters that you can specify in your request, see [Anthropic Claude models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-claude.html).
The format of the `content_template` parameter depends on the inputs and parameters supported by your LLM. In this tutorial, [Anthropic’s Claude 2 model](https://www.anthropic.com/index/claude-2) uses the following `content_template`:
```
"content_template": "{\"prompt\": $prompt, \"max_tokens_to_sample\": 500}"
```
As another example, the [Falcon 7b model](https://huggingface.co/tiiuae/falcon-7b) can support the following `content_template`:
```
"content_template": "{\"inputs\": $prompt, \"parameters\":{\"max_new_tokens\": \
10, \"top_p\": 0.9, \"temperature\": 0.8}}"
```
## Run your model evaluation
**Define and run your evaluation algorithm**
1. Define your evaluation algorithm. The following example shows how to define a `SummarizationAccuracy` algorithm, which is used to determine accuracy for text summarization tasks:
```
eval_algo = SummarizationAccuracy(SummarizationAccuracyConfig())
```
For examples of algorithms that calculate metrics for other evaluation tasks, see **Evaluate your model** in [Use the `fmeval` library to run an automatic evaluation](clarify-foundation-model-evaluate-auto-lib.md).
1. Run your evaluation algorithm. The following code example uses the data configuration that was previously defined, and a `prompt_template` that uses the `Human` and `Assistant` keys:
```
eval_output = eval_algo.evaluate(model=bedrock_model_runner,
dataset_config=config,
prompt_template="Human: $feature\n\nAssistant:\n", save=True)
```
In the previous code example, `feature` contains the prompt in the format that Amazon Bedrock model expects.
## View your analysis results
1. Parse an evaluation report from the `eval_output` object returned by the evaluation algorithm as follows:
```
# parse report
print(json.dumps(eval_output, default=vars, indent=4))
```
The previous command returns the following output:
```
[
{
"eval_name": "summarization_accuracy",
"dataset_name": "sample-dataset",
"dataset_scores": [
{
"name": "meteor",
"value": 0.2048823008681274
},
{
"name": "rouge",
"value": 0.03557697913367101
},
{
"name": "bertscore",
"value": 0.5406564395678671
}
],
"prompt_template": "Human: $feature\n\nAssistant:\n",
"category_scores": null,
"output_path": "/tmp/eval_results/summarization_accuracy_sample_dataset.jsonl",
"error": null
}
]
```
The previous example output displays the three accuracy scores: [https://huggingface.co/spaces/evaluate-metric/meteor](https://huggingface.co/spaces/evaluate-metric/meteor), [https://huggingface.co/spaces/evaluate-metric/rouge](https://huggingface.co/spaces/evaluate-metric/rouge), and [https://huggingface.co/spaces/evaluate-metric/bertscore](https://huggingface.co/spaces/evaluate-metric/bertscore), the input `prompt_template`, a `category_score` if you requested one, any errors, and the `output_path`. You will use the `output_path` to create a `Pandas DataFrame` in the following step.
1. Import your results and read them into a `DataFrame`, and attach the accuracy scores to the model input, model output, and target output as follows:
```
import pandas as pd
data = []
with open("/tmp/eval_results/summarization_accuracy_sample_dataset.jsonl", "r") as file:
for line in file:
data.append(json.loads(line))
df = pd.DataFrame(data)
df['meteor_score'] = df['scores'].apply(lambda x: x[0]['value'])
df['rouge_score'] = df['scores'].apply(lambda x: x[1]['value'])
df['bert_score'] = df['scores'].apply(lambda x: x[2]['value'])
df
```
In this invocation, the previous code example returns the following output (contracted for brevity):
```
model_input model_output target_output prompt scores meteor_score rouge_score bert_score
0 John Edward Bates, formerly of Spalding, Linco... I cannot make any definitive judgments, as th... A former Lincolnshire Police officer carried o... Human: John Edward Bates, formerly of Spalding... [{'name': 'meteor', 'value': 0.112359550561797... 0.112360 0.000000 0.543234 ...
1 23 October 2015 Last updated at 17:44 BST\nIt'... Here are some key points about hurricane/trop... Hurricane Patricia has been rated as a categor... Human: 23 October 2015 Last updated at 17:44 B... [{'name': 'meteor', 'value': 0.139822692925566... 0.139823 0.017621 0.426529 ...
2 Ferrari appeared in a position to challenge un... Here are the key points from the article:\n\n... Lewis Hamilton stormed to pole position at the... Human: Ferrari appeared in a position to chall... [{'name': 'meteor', 'value': 0.283411142234671... 0.283411 0.064516 0.597001 ...
3 The Bath-born player, 28, has made 36 appearan... Okay, let me summarize the key points from th... Newport Gwent Dragons number eight Ed Jackson ... Human: The Bath-born player, 28, has made 36 a... [{'name': 'meteor', 'value': 0.089020771513353... 0.089021 0.000000 0.533514 ...
...
```
Your model output may be different than the previous sample output.
For a notebook that contains the code examples given in this section, see [bedrock-claude-summarization-accuracy.ipnyb](https://github.com/aws/fmeval/blob/main/examples/bedrock-claude-summarization-accuracy.ipynb).
## Additional notebooks
The [fmeval GitHub](https://github.com/aws/fmeval/tree/main/examples) directory contains the following additional example notebooks:
+ [bedrock-claude-factual-knowledge.ipnyb](https://github.com/aws/fmeval/blob/main/examples/bedrock-claude-factual-knowledge.ipynb) – Evaluates an [Anthropic Claude 2](https://www.anthropic.com/index/claude-2) model hosted on Amazon Bedrock for factual knowledge.
+ [byo-model-outputs.ipynb](https://github.com/aws/fmeval/blob/main/examples/byo-model-outputs.ipynb) – Evaluates a [Falcon 7b model](https://huggingface.co/tiiuae/falcon-7b) hosted on JumpStart for factual knowledge where you bring your own model outputs instead of sending inference requests to your model.
+ [custom\$1model\$1runner\$1chat\$1gpt.ipnyb](https://github.com/aws/fmeval/blob/main/examples/custom_model_runner_chat_gpt.ipynb) – Evaluates a custom `ChatGPT 3.5` model hosted on `Hugging Face` for factual knowledge.
# Resolve errors when creating a model evaluation job in Amazon SageMaker AI
**Important**
In order to use SageMaker Clarify Foundation Model Evaluations (FMEval), you must upgrade to the new Studio experience.
As of November 30, 2023, the previous Amazon SageMaker Studio experience is now named Amazon SageMaker Studio Classic. FMEval isn't available in Amazon SageMaker Studio Classic.
For information about how to upgrade to the new Studio experience, see [Migration from Amazon SageMaker Studio Classic](studio-updated-migrate.md). For information about using the Studio Classic application, see [Amazon SageMaker Studio Classic](studio.md).
If you run into an error while creating a model evaluation job, use the following list to troubleshoot your evaluation. If you need further assistance, contact [Support](https://console.aws.amazon.com/support/) or [AWS Developer Forums for Amazon SageMaker AI](https://forums.aws.amazon.com/forum.jspa?forumID=285).
**Topics**
+ [Error uploading your data from an Amazon S3 bucket](#clarify-foundation-model-evaluate-troubleshooting-cors)
+ [The processing job failed to complete](#clarify-foundation-model-evaluate-troubleshooting-failure)
+ [You can't find foundation model evaluations in the SageMaker AI console](#clarify-foundation-model-evaluate-troubleshooting-upgrade)
+ [Your model does not support prompt stereotyping](#clarify-foundation-model-evaluate-troubleshooting-ps)
+ [Dataset validation errors (Human)](#clarify-foundation-model-evaluate-troubleshooting-valid)
## Error uploading your data from an Amazon S3 bucket
When you create a foundation model evaluation, you must set the correct permissions for the S3 bucket that you want to store your model input and output in. If the Cross-origin resource sharing (CORS) permissions are not set correctly, SageMaker AI generates the following error:
Error: Failed to put object in s3: Error while uploading object to s3Error: Failed to put object in S3: NetworkError when attempting to fetch resource.
To set the correct bucket permissions, follow the instructions under **Set up your environment** in [Create an automatic model evaluation job in Studio](clarify-foundation-model-evaluate-auto-ui.md).
## The processing job failed to complete
The most common reasons that your processing job failed to complete include the following:
+ [Insufficient quota](#clarify-foundation-model-evaluate-troubleshooting-failure-quota)
+ [Insufficient memory](#clarify-foundation-model-evaluate-troubleshooting-failure-memory)
+ [Did not pass ping check](#clarify-foundation-model-evaluate-troubleshooting-failure-ping)
See the following sections to help you mitigate each issue.
### Insufficient quota
When you run a foundation model evaluation for a non-deployed JumpStart model, SageMaker Clarify deploys your large language model (LLM) to a SageMaker AI endpoint in your account. If your account does not have sufficient quota to run the selected JumpStart model, the job fails with a `ClientError`. To increase your quota, follow these steps:
**Request an AWS Service Quotas increase**
1. Retrieve the instance name, current quota and necessary quota from the on screen error message. For example, in the following error:
+ The instance name is `ml.g5.12xlarge`.
+ The current quota from the number following `current utilization`is `0 instances`
+ The additional required quota from the number following `request delta` is `1 instances`.
The sample error follows:
`ClientError: An error occurred (ResourceLimitExceeded) when calling the CreateEndpoint operation: The account-level service limit 'ml.g5.12xlarge for endpoint usage' is 0 Instances, with current utilization of 0 Instances and a request delta of 1 Instances. Please use AWS Service Quotas to request an increase for this quota. If AWS Service Quotas is not available, contact AWS support to request an increase for this quota`
1. Sign into the AWS Management Console and open the [Service Quotas console](https://console.aws.amazon.com/servicequotas/home).
1. In the navigation pane, under **Manage quotas**, input **Amazon SageMaker AI**.
1. Choose **View quotas**.
1. In the search bar under **Service quotas**, input the name of the instance from Step 1. For example, using the information contained in the error message from Step 1, input **ml.g5.12xlarge**.
1. Choose the **Quota name** that appears next to your instance name and ends with **for endpoint usage**. For example, using the information contained in the error message from Step 1, choose **ml.g5.12xlarge for endpoint usage**.
1. Choose **Request increase at account-level**.
1. Under **Increase quota value**, input the necessary required quota from the information given in the error message from Step 1. Input the **total** of `current utilization` and `request delta`. In the previous example error, the `current utilization` is `0 Instances`, and the `request delta` is `1 Instances`. In this example, request a quota of `1` to supply the required quota.
1. Choose **Request**.
1. Choose **Quota request history** from the navigation pane.
1. When the **Status** changes from **Pending** to **Approved**, rerun your job. You may need to refresh your browser to see the change.
For more information about requesting an increase in your quota, see [Requesting a quota increase](https://docs.aws.amazon.com/servicequotas/latest/userguide/request-quota-increase.html).
### Insufficient memory
If you start a foundation model evaluation on an Amazon EC2 instance that does not have sufficient memory to run an evaluation algorithm, the job fails with the following error:
`The actor is dead because its worker process has died. Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors. The actor never ran - it was cancelled before it started running.`
To increase the memory available for your evaluation job, change your instance to one that has more memory. If you are using the user interface, you can choose an instance type under **Processor configuration** in **Step 2**. If you are running your job inside the SageMaker AI console, launch a new space using an instance with increased memory capacity.
For a list of Amazon EC2 instances, see [Instance types](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html#AvailableInstanceTypes).
For more information, about instances with larger memory capacity, see [Memory optimized instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/memory-optimized-instances.html).
### Did not pass ping check
In some instances, your foundation model evaluation job will fail because it did not pass a ping check when SageMaker AI was deploying your endpoint. If it does not pass a ping test, the following error appears:
`ClientError: Error hosting endpoint your_endpoint_name: Failed. Reason: The primary container for production variant AllTraffic did not pass the ping health check. Please check CloudWatch logs for this endpoint..., Job exited for model: your_model_name of model_type: your_model_type `
If your job generates this error, wait a few minutes and run your job again. If the error persists, contact [AWS Support](https://console.aws.amazon.com/support/) or [AWS Developer Forums for Amazon SageMaker AI](https://forums.aws.amazon.com/forum.jspa?forumID=285).
## You can't find foundation model evaluations in the SageMaker AI console
In order to use SageMaker Clarify Foundation Model Evaluations, you must upgrade to the new Studio experience. As of November 30, 2023, the previous Amazon SageMaker Studio experience is now named Amazon SageMaker Studio Classic. The foundation evaluation feature can only be used in the updated experience. For information about how to update Studio, see [Migration from Amazon SageMaker Studio Classic](studio-updated-migrate.md).
## Your model does not support prompt stereotyping
Only some JumpStart models support prompt stereotyping. If you select a JumpStart model that is not supported, the following error appears:
`{"evaluationMetrics":"This model does not support Prompt stereotyping evaluation. Please remove that evaluation metric or select another model that supports it."}`
If you receive this error, you cannot use your selected model in a foundation evaluation. SageMaker Clarify is currently working to update all JumpStart models for prompt stereotyping tasks so that they can be used in a foundation model evaluation.
## Dataset validation errors (Human)
The custom prompt dataset in a model evaluation job that uses human workers must be formatted using the JSON lines format using the `.jsonl` extension.
When you start a job each JSON object in the prompt dataset is interdependently validated. If one of the JSON objects is not valid you get the following error.
```
Customer Error: Your input dataset could not be validated. Your dataset can have up to 1000 prompts. The dataset must be a valid jsonl file, and each prompt valid json object.To learn more about troubleshooting dataset validations errors, see Troubleshooting guide. Job executed for models: meta-textgeneration-llama-2-7b-f, pytorch-textgeneration1-alexa20b.
```
For a custom prompt dataset to pass all validations the following must be *true* for all JSON objects in the JSON lines file.
+ Each line in the prompt dataset file must be a valid JSON object.
+ Special characters such as quotation marks (`"`) must be escaped properly. For example, if your prompt was the following `"Claire said to the crowd, "Bananas are the best!""` the quotes would need to be escaped using a `\`, `"Claire said to the crowd, \"Bananas are the best!\""`.
+ A valid JSON objects must contain at least the `prompt`key/value pair.
+ A prompt dataset file cannot contain more than 1,000 JSON objects in a single file.
+ If you specify the `responses` key in *any* JSON object, it must be present in*all* JSON objects.
+ The maximum number of objects in the `responses` key is 1. If you have responses from multiple models you want to compare, each require a separate BYOI dataset.
+ If you specify the `responses` key in *any* JSON object, it must also contain the `modelIdentifier` and `text` keys in all *all* `responses` objects.