# Model evaluation task types in Amazon Bedrock Model evaluation task types In a model evaluation job, an evaluation task type is a task you want the model to perform based on information in your prompts. You can choose one task type per model evaluation job. The following table summarizes available tasks types for automatic model evaluations, built-in datasets, and relevant metrics for each task type. **Available built-in datasets for automatic model evaluation jobs in Amazon Bedrock** [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/sagemaker-unified-studio/latest/userguide/model-evaluation-tasks.html) **Topics** + [ # General text generation for model evaluation in Amazon Bedrock ](model-evaluation-tasks-general-text.md) + [ # Text summarization for model evaluation in Amazon Bedrock ](model-evaluation-tasks-text-summary.md) + [ # Question and answer for model evaluation in Amazon Bedrock ](model-evaluation-tasks-question-answer.md) + [ # Text classification for model evaluation in Amazon Bedrock ](model-evaluation-text-classification.md) # General text generation for model evaluation in Amazon Bedrock General text generation General text generation is a task used by applications that include chatbots. The responses generated by a model to general questions are influenced by the correctness, relevance, and bias contained in the text used to train the model. **Important** For general text generation, there is a known system issue that prevents Cohere models from completing the toxicity evaluation successfully. The following built-in datasets contain prompts that are well-suited for use in general text generation tasks. **Bias in Open-ended Language Generation Dataset (BOLD)** The Bias in Open-ended Language Generation Dataset (BOLD) is a dataset that evaluates fairness in general text generation, focusing on five domains: profession, gender, race, religious ideologies, and political ideologies. It contains 23,679 different text generation prompts. **RealToxicityPrompts** RealToxicityPrompts is a dataset that evaluates toxicity. It attempts to get the model to generate racist, sexist, or otherwise toxic language. This dataset contains 100,000 different text generation prompts. **T-Rex : A Large Scale Alignment of Natural Language with Knowledge Base Triples (TREX)** TREX is dataset consisting of Knowledge Base Triples (KBTs) extracted from Wikipedia. KBTs are a type of data structure used in natural language processing (NLP) and knowledge representation. They consist of a subject, predicate, and object, where the subject and object are linked by a relation. An example of a Knowledge Base Triple (KBT) is "George Washington was the president of the United States". The subject is "George Washington", the predicate is "was the president of", and the object is "the United States". **WikiText2** WikiText2 is a HuggingFace dataset that contains prompts used in general text generation. The following table summarizes the metrics calculated, and recommended built-in dataset that are available for automatic model evaluation jobs. **Available built-in datasets for general text generation in Amazon Bedrock** [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/sagemaker-unified-studio/latest/userguide/model-evaluation-tasks-general-text.html) To learn more about how the computed metric for each built-in dataset is calculated, see [Review a model model evaluation job in Amazon Bedrock](model-evaluation-report.md) # Text summarization for model evaluation in Amazon Bedrock Text summarization Text summarization is used for tasks including creating summaries of news, legal documents, academic papers, content previews, and content curation. The ambiguity, coherence, bias, and fluency of the text used to train the model as well as information loss, accuracy, relevance, or context mismatch can influence the quality of responses. **Important** For text summarization, there is a known system issue that prevents Cohere models from completing the toxicity evaluation successfully. The following built-in dataset is supported for use with the task summarization task type. **Gigaword** The Gigaword dataset consists of news article headlines. This dataset is used in text summarization tasks. The following table summarizes the metrics calculated, and recommended built-in dataset. **Available built-in datasets for text summarization in Amazon Bedrock** [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/sagemaker-unified-studio/latest/userguide/model-evaluation-tasks-text-summary.html) To learn more about how the computed metric for each built-in dataset is calculated, see [Review a model model evaluation job in Amazon Bedrock](model-evaluation-report.md) # Question and answer for model evaluation in Amazon Bedrock Question and answer Question and answer is used for tasks including generating automatic help-desk responses, information retrieval, and e-learning. If the text used to train the foundation model contains issues including incomplete or inaccurate data, sarcasm or irony, the quality of responses can deteriorate. **Important** For question and answer, there is a known system issue that prevents Cohere models from completing the toxicity evaluation successfully. The following built-in datasets are recommended for use with the question andg answer task type. **BoolQ** BoolQ is a dataset consisting of yes/no question and answer pairs. The prompt contains a short passage, and then a question about the passage. This dataset is recommended for use with question and answer task type. **Natural Questions** Natural questions is a dataset consisting of real user questions submitted to Google search. **TriviaQA** TriviaQA is a dataset that contains over 650K question-answer-evidence-triples. This dataset is used in question and answer tasks. The following table summarizes the metrics calculated, and recommended built-in dataset. **Available built-in datasets for the question and answer task type in Amazon Bedrock** [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/sagemaker-unified-studio/latest/userguide/model-evaluation-tasks-question-answer.html) To learn more about how the computed metric for each built-in dataset is calculated, see [Review a model model evaluation job in Amazon Bedrock](model-evaluation-report.md) # Text classification for model evaluation in Amazon Bedrock Text classification Text classification is used to categorize text into pre-defined categories. Applications that use text classification include content recommendation, spam detection, language identification and trend analysis on social media. Imbalanced classes, ambiguous data, noisy data, and bias in labeling are some issues that can cause errors in text classification. **Important** For text classification, there is a known system issue that prevents Cohere models from completing the toxicity evaluation successfully. The following built-in datasets are recommended for use with the text classification task type. **Women's E-Commerce Clothing Reviews** Women's E-Commerce Clothing Reviews is a dataset that contains clothing reviews written by customers. This dataset is used in text classification tasks. The following table summarizes the metrics calculated, and recommended built-in datasets. **Available built-in datasets in Amazon Bedrock** [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/sagemaker-unified-studio/latest/userguide/model-evaluation-text-classification.html) To learn more about how the computed metric for each built-in dataset is calculated, see [Review a model model evaluation job in Amazon Bedrock](model-evaluation-report.md)