As traduções são geradas por tradução automática. Em caso de conflito entre o conteúdo da tradução e da versão original em inglês, a versão em inglês prevalecerá. # Personalização do modelo de peso aberto Esta seção mostra o processo de introdução à personalização do modelo de peso aberto. **Topics** + [Pré-requisitos](model-customize-open-weight-prereq.md) + [Criação de ativos para personalização de modelos na interface do usuário](model-customize-open-weight-create-assets-ui.md) + [Envio de trabalhos de personalização do modelo de IA](model-customize-open-weight-job.md) + [Envio de trabalho de avaliação de modelo](model-customize-open-weight-evaluation.md) + [Implantação de modelos](model-customize-open-weight-deployment.md) + [Amostras de conjuntos de dados e avaliadores](model-customize-open-weight-samples.md) # Pré-requisitos Antes de começar, conclua os seguintes pré-requisitos: + Integre-se a um domínio de SageMaker IA com acesso ao Studio. Caso não tenha permissões para definir o Studio como a experiência padrão do seu domínio, entre em contato com seu administrador. Para obter mais informações, consulte a [visão geral do domínio Amazon SageMaker AI](https://docs.aws.amazon.com/sagemaker/latest/dg/gs-studio-onboard.html). + Atualize o AWS CLI seguindo as etapas em [Instalando a AWS CLI versão atual](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv1.html#install-tool-bundled). + Em sua máquina local, execute `aws configure` e forneça suas credenciais da AWS . Para obter informações sobre AWS credenciais, consulte [Entendendo e obtendo suas AWS credenciais](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html). ## Permissões obrigatórias do IAM SageMaker A personalização do modelo de IA exige a adição de permissões apropriadas à execução do seu domínio de SageMaker IA. Para fazer isso, você pode criar uma política de permissões do IAM embutida e anexá-la à função do IAM. Para obter informações sobre como adicionar políticas, consulte [Adicionar e remover permissões de identidade do IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html) no *Guia do usuário do AWS Identity and Access Management*. ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowNonAdminStudioActions", "Effect": "Allow", "Action": [ "sagemaker:CreatePresignedDomainUrl", "sagemaker:DescribeDomain", "sagemaker:DescribeUserProfile", "sagemaker:DescribeSpace", "sagemaker:ListSpaces", "sagemaker:DescribeApp", "sagemaker:ListApps" ], "Resource": [ "arn:aws:sagemaker:*:*:domain/*", "arn:aws:sagemaker:*:*:user-profile/*", "arn:aws:sagemaker:*:*:app/*", "arn:aws:sagemaker:*:*:space/*" ] }, { "Sid": "LambdaListPermissions", "Effect": "Allow", "Action": [ "lambda:ListFunctions" ], "Resource": [ "*" ] }, { "Sid": "LambdaPermissionsForRewardFunction", "Effect": "Allow", "Action": [ "lambda:CreateFunction", "lambda:DeleteFunction", "lambda:InvokeFunction", "lambda:GetFunction" ], "Resource": [ "arn:aws:lambda:*:*:function:*SageMaker*", "arn:aws:lambda:*:*:function:*sagemaker*", "arn:aws:lambda:*:*:function:*Sagemaker*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "LambdaLayerForAWSSDK", "Effect": "Allow", "Action": [ "lambda:GetLayerVersion" ], "Resource": [ "arn:aws:lambda:*:336392948345:layer:AWSSDK*" ] }, { "Sid": "SageMakerPublicHubPermissions", "Effect": "Allow", "Action": [ "sagemaker:ListHubContents" ], "Resource": [ "arn:aws:sagemaker:*:aws:hub/SageMakerPublicHub" ] }, { "Sid": "SageMakerHubPermissions", "Effect": "Allow", "Action": [ "sagemaker:ListHubs", "sagemaker:ListHubContents", "sagemaker:DescribeHubContent", "sagemaker:DeleteHubContent", "sagemaker:ListHubContentVersions", "sagemaker:Search" ], "Resource": [ "*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "JumpStartAccess", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::jumpstart*" ] }, { "Sid": "ListMLFlowOperations", "Effect": "Allow", "Action": [ "sagemaker:ListMlflowApps", "sagemaker:ListMlflowTrackingServers" ], "Resource": [ "*" ] }, { "Sid": "MLFlowAccess", "Effect": "Allow", "Action": [ "sagemaker:UpdateMlflowApp", "sagemaker:DescribeMlflowApp", "sagemaker:CreatePresignedMlflowAppUrl", "sagemaker:CallMlflowAppApi", "sagemaker-mlflow:*" ], "Resource": [ "arn:aws:sagemaker:*:*:mlflow-app/*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "BYODataSetS3Access", "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetObject", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::*SageMaker*", "arn:aws:s3:::*Sagemaker*", "arn:aws:s3:::*sagemaker*" ] }, { "Sid": "AllowHubPermissions", "Effect": "Allow", "Action": [ "sagemaker:ImportHubContent" ], "Resource": [ "arn:aws:sagemaker:*:*:hub/*", "arn:aws:sagemaker:*:*:hub-content/*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "PassRoleForSageMaker", "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::*:role/service-role/AmazonSageMaker-ExecutionRole-*" ], "Condition": { "StringEquals": { "iam:PassedToService": "sagemaker.amazonaws.com", "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "PassRoleForAWSLambda", "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::*:role/service-role/AmazonSageMaker-ExecutionRole-*" ], "Condition": { "StringEquals": { "iam:PassedToService": "lambda.amazonaws.com", "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "PassRoleForBedrock", "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::*:role/service-role/AmazonSageMaker-ExecutionRole-*" ], "Condition": { "StringEquals": { "iam:PassedToService": "bedrock.amazonaws.com", "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "TrainingJobRun", "Effect": "Allow", "Action": [ "sagemaker:CreateTrainingJob", "sagemaker:DescribeTrainingJob", "sagemaker:ListTrainingJobs" ], "Resource": [ "arn:aws:sagemaker:*:*:training-job/*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "ModelPackageAccess", "Effect": "Allow", "Action": [ "sagemaker:CreateModelPackage", "sagemaker:DescribeModelPackage", "sagemaker:ListModelPackages", "sagemaker:CreateModelPackageGroup", "sagemaker:DescribeModelPackageGroup", "sagemaker:ListModelPackageGroups", "sagemaker:CreateModel" ], "Resource": [ "arn:aws:sagemaker:*:*:model-package-group/*", "arn:aws:sagemaker:*:*:model-package/*", "arn:aws:sagemaker:*:*:model/*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "TagsPermission", "Effect": "Allow", "Action": [ "sagemaker:AddTags", "sagemaker:ListTags" ], "Resource": [ "arn:aws:sagemaker:*:*:model-package-group/*", "arn:aws:sagemaker:*:*:model-package/*", "arn:aws:sagemaker:*:*:hub/*", "arn:aws:sagemaker:*:*:hub-content/*", "arn:aws:sagemaker:*:*:training-job/*", "arn:aws:sagemaker:*:*:model/*", "arn:aws:sagemaker:*:*:endpoint/*", "arn:aws:sagemaker:*:*:endpoint-config/*", "arn:aws:sagemaker:*:*:pipeline/*", "arn:aws:sagemaker:*:*:inference-component/*", "arn:aws:sagemaker:*:*:action/*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "LogAccess", "Effect": "Allow", "Action": [ "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:GetLogEvents" ], "Resource": [ "arn:aws:logs:*:*:log-group*", "arn:aws:logs:*:*:log-group:/aws/sagemaker/TrainingJobs:log-stream:*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "BedrockDeploy", "Effect": "Allow", "Action": [ "bedrock:CreateModelImportJob" ], "Resource": [ "arn:aws:bedrock:*:*:*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "BedrockOperations", "Effect": "Allow", "Action": [ "bedrock:GetModelImportJob", "bedrock:GetImportedModel", "bedrock:ListProvisionedModelThroughputs", "bedrock:ListCustomModelDeployments", "bedrock:ListCustomModels", "bedrock:ListModelImportJobs", "bedrock:GetEvaluationJob", "bedrock:CreateEvaluationJob", "bedrock:InvokeModel" ], "Resource": [ "arn:aws:bedrock:*:*:evaluation-job/*", "arn:aws:bedrock:*:*:imported-model/*", "arn:aws:bedrock:*:*:model-import-job/*", "arn:aws:bedrock:*:*:foundation-model/*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "BedrockFoundationModelOperations", "Effect": "Allow", "Action": [ "bedrock:GetFoundationModelAvailability", "bedrock:ListFoundationModels" ], "Resource": [ "*" ] }, { "Sid": "SageMakerPipelinesAndLineage", "Effect": "Allow", "Action": [ "sagemaker:ListActions", "sagemaker:ListArtifacts", "sagemaker:QueryLineage", "sagemaker:ListAssociations", "sagemaker:AddAssociation", "sagemaker:DescribeAction", "sagemaker:AddAssociation", "sagemaker:CreateAction", "sagemaker:CreateContext", "sagemaker:DescribeTrialComponent" ], "Resource": [ "arn:aws:sagemaker:*:*:artifact/*", "arn:aws:sagemaker:*:*:action/*", "arn:aws:sagemaker:*:*:context/*", "arn:aws:sagemaker:*:*:action/*", "arn:aws:sagemaker:*:*:model-package/*", "arn:aws:sagemaker:*:*:context/*", "arn:aws:sagemaker:*:*:pipeline/*", "arn:aws:sagemaker:*:*:experiment-trial-component/*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "ListOperations", "Effect": "Allow", "Action": [ "sagemaker:ListInferenceComponents", "sagemaker:ListWorkforces" ], "Resource": [ "*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "SageMakerInference", "Effect": "Allow", "Action": [ "sagemaker:DescribeInferenceComponent", "sagemaker:CreateEndpoint", "sagemaker:CreateEndpointConfig", "sagemaker:DescribeEndpoint", "sagemaker:DescribeEndpointConfig", "sagemaker:ListEndpoints" ], "Resource": [ "arn:aws:sagemaker:*:*:inference-component/*", "arn:aws:sagemaker:*:*:endpoint/*", "arn:aws:sagemaker:*:*:endpoint-config/*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "SageMakerPipelines", "Effect": "Allow", "Action": [ "sagemaker:DescribePipelineExecution", "sagemaker:ListPipelineExecutions", "sagemaker:ListPipelineExecutionSteps", "sagemaker:CreatePipeline", "sagemaker:UpdatePipeline", "sagemaker:StartPipelineExecution" ], "Resource": [ "arn:aws:sagemaker:*:*:pipeline/*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } } ] } ``` Se você anexou o [AmazonSageMakerFullAccessPolicy](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonSageMakerFullAccess.html)à sua função de execução, poderá adicionar esta política reduzida: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "LambdaListPermissions", "Effect": "Allow", "Action": [ "lambda:ListFunctions" ], "Resource": [ "*" ] }, { "Sid": "LambdaPermissionsForRewardFunction", "Effect": "Allow", "Action": [ "lambda:CreateFunction", "lambda:DeleteFunction", "lambda:InvokeFunction", "lambda:GetFunction" ], "Resource": [ "arn:aws:lambda:*:*:function:*SageMaker*", "arn:aws:lambda:*:*:function:*sagemaker*", "arn:aws:lambda:*:*:function:*Sagemaker*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "LambdaLayerForAWSSDK", "Effect": "Allow", "Action": [ "lambda:GetLayerVersion" ], "Resource": [ "arn:aws:lambda:*:336392948345:layer:AWSSDK*" ] }, { "Sid": "S3Access", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::*SageMaker*", "arn:aws:s3:::*Sagemaker*", "arn:aws:s3:::*sagemaker*", "arn:aws:s3:::jumpstart*" ] }, { "Sid": "PassRoleForSageMakerAndLambdaAndBedrock", "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::*:role/service-role/AmazonSageMaker-ExecutionRole-*" ], "Condition": { "StringEquals": { "iam:PassedToService": [ "lambda.amazonaws.com", "bedrock.amazonaws.com" ], "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "BedrockDeploy", "Effect": "Allow", "Action": [ "bedrock:CreateModelImportJob" ], "Resource": [ "*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "BedrockOperations", "Effect": "Allow", "Action": [ "bedrock:GetModelImportJob", "bedrock:GetImportedModel", "bedrock:ListProvisionedModelThroughputs", "bedrock:ListCustomModelDeployments", "bedrock:ListCustomModels", "bedrock:ListModelImportJobs", "bedrock:GetEvaluationJob", "bedrock:CreateEvaluationJob", "bedrock:InvokeModel" ], "Resource": [ "arn:aws:bedrock:*:*:evaluation-job/*", "arn:aws:bedrock:*:*:imported-model/*", "arn:aws:bedrock:*:*:model-import-job/*", "arn:aws:bedrock:*:*:foundation-model/*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "BedrockFoundationModelOperations", "Effect": "Allow", "Action": [ "bedrock:GetFoundationModelAvailability", "bedrock:ListFoundationModels" ], "Resource": [ "*" ] } ] } ``` Em seguida, você deve clicar em **Editar Política de Confiança** e substituí-la pela política a seguir e, em seguida, clicar em **Atualizar Política**. ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole" }, { "Effect": "Allow", "Principal": { "Service": "sagemaker.amazonaws.com" }, "Action": "sts:AssumeRole" }, { "Effect": "Allow", "Principal": { "Service": "bedrock.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` # Criação de ativos para personalização de modelos na interface do usuário Você pode criar e gerenciar o conjunto de dados e os ativos do avaliador que você pode usar para personalização do modelo na interface do usuário. ## Ativos Selecione **Ativos** no painel esquerdo e na interface do usuário do Amazon SageMaker Studio e, em seguida, selecione **Conjuntos de dados.** ![\[Uma imagem contendo o acesso à personalização do modelo.\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/screenshot-open-model-16.png) Escolha **Carregar conjunto de dados** para adicionar o conjunto de dados que você usará em seus trabalhos de personalização do modelo. Ao escolher o **formato de entrada de dados obrigatório**, você pode acessar uma referência do formato do conjunto de dados a ser usado. ![\[Uma imagem contendo o acesso à personalização do modelo.\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/screenshot-open-model-15.png) ## Avaliadores Você também pode adicionar **funções de recompensa** e **solicitações de recompensa para seus trabalhos de** personalização do Reinforcement Learning. ![\[Uma imagem contendo o acesso à personalização do modelo.\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/screenshot-open-model-14.png) A interface do usuário também fornece orientação sobre o formato necessário para a função de recompensa ou solicitação de recompensa. ![\[Uma imagem contendo o acesso à personalização do modelo.\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/screenshot-open-model-13.png) ## Ativos para personalização de modelos usando o SDK AWS Você também pode usar o SDK SageMaker AI Python para criar ativos. Veja um exemplo de trecho de código abaixo: ``` from sagemaker.ai_registry.air_constants import REWARD_FUNCTION, REWARD_PROMPT from sagemaker.ai_registry.dataset import DataSet, CustomizationTechnique from sagemaker.ai_registry.evaluator import Evaluator # Creating a dataset example dataset = DataSet.create( name="sdkv3-gen-ds2", source="s3://sample-test-bucket/datasets/training-data/jamjee-sft-ds1.jsonl", # or use local filepath as source. customization_technique=CustomizationTechnique.SFT ) # Refreshes status from hub dataset.refresh() pprint(dataset.__dict__) # Creating an evaluator. Method : Lambda evaluator = Evaluator.create( name = "sdk-new-rf11", source="arn:aws:lambda:us-west-2:<>:function:8", type=REWARD_FUNCTION ) # Creating an evaluator. Method : Bring your own code evaluator = Evaluator.create( name = "eval-lambda-test", source="/path_to_local/eval_lambda_1.py", type = REWARD_FUNCTION ) # Optional wait, by default we have wait = True during create call. evaluator.wait() evaluator.refresh() pprint(evaluator) ``` # Envio de trabalhos de personalização do modelo de IA O recurso de personalização do modelo de SageMaker IA pode ser acessado na página de modelos do Amazon SageMaker Studio no painel esquerdo. Você também pode encontrar a página Ativos, onde você pode criar e gerenciar seus conjuntos de dados e avaliadores de personalização de modelos. ![\[Uma imagem contendo o acesso à personalização do modelo.\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/screenshot-open-model-12.png) Para iniciar o envio de um trabalho de personalização de modelo, selecione a opção Modelos para acessar a guia Modelos básicos do Jumpstart: ![\[Uma imagem contendo como escolher o modelo básico.\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/screenshot-open-model-11.png) Você pode clicar diretamente em Personalizar modelo no cartão do modelo ou pesquisar qualquer modelo do Meta que esteja interessado em personalizar. ![\[Uma imagem contendo o cartão do modelo e como escolher o modelo a ser personalizado.\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/screenshot-open-model-10.png) Ao clicar no cartão do modelo, você pode acessar a página de detalhes do modelo e iniciar o trabalho de personalização clicando em Personalizar modelo e, em seguida, selecionando Personalizar com interface para iniciar a configuração do seu trabalho RLVR. ![\[Uma imagem contendo como iniciar o trabalho de personalização.\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/screenshot-open-model-9.png) Em seguida, você pode inserir o nome do modelo personalizado, selecionar a técnica de personalização do modelo a ser usada e configurar os hiperparâmetros do seu trabalho: ![\[Uma imagem contendo uma seleção de técnicas de personalização do modelo.\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/screenshot-open-model-8.png) ![\[Uma imagem contendo uma seleção de técnicas de personalização do modelo.\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/screenshot-open-model-7.png) ## Envio de trabalhos de personalização do modelo de IA usando SDK Você também pode usar o SDK SageMaker AI Python para enviar um trabalho de personalização de modelo: ``` # Submit a DPO model customization job from sagemaker.modules.train.dpo_trainer import DPOTrainer from sagemaker.modules.train.common import TrainingType trainer = DPOTrainer( model=BASE_MODEL, training_type=TrainingType.LORA, model_package_group_name=MODEL_PACKAGE_GROUP_NAME, training_dataset=TRAINING_DATASET, s3_output_path=S3_OUTPUT_PATH, sagemaker_session=sagemaker_session, role=ROLE_ARN ) ``` ## Monitorando seu trabalho de personalização Imediatamente após enviar seu trabalho, você será redirecionado para a página de trabalho de treinamento de personalização de modelos. ![\[Uma imagem contendo uma seleção de técnicas de personalização do modelo.\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/screenshot-open-model-6.png) Depois que o trabalho for concluído, você poderá acessar a página de detalhes do modelo personalizado clicando no botão Ir para **modelo personalizado** no canto superior direito. ![\[Uma imagem contendo uma seleção de técnicas de personalização do modelo.\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/screenshot-open-model-5.png) Na página de detalhes do modelo personalizado, você pode continuar trabalhando com seu modelo personalizado da seguinte forma: 1. Verificando informações sobre desempenho, localização dos artefatos gerados, hiperparâmetros de configuração de treinamento e registros de treinamento. 1. Inicie um trabalho de avaliação com um conjunto de dados diferente (personalização contínua). 1. Implante o modelo usando endpoints de inferência de SageMaker IA ou Amazon Bedrock Custom Model Import. ![\[Uma imagem contendo uma seleção de técnicas de personalização do modelo.\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/screenshot-open-model-4.png) # Envio de trabalho de avaliação de modelo Esta seção descreve a avaliação aberta do modelo personalizado. Ele ajuda você a começar por um passo a passo do processo de envio de trabalhos de avaliação. Recursos adicionais são fornecidos para casos de uso mais avançados de envio de trabalhos de avaliação. **Topics** + [Conceitos básicos](model-customize-evaluation-getting-started.md) + [Tipos de avaliação e envio de trabalhos](model-customize-evaluation-types.md) + [Formatos de métricas de avaliação](model-customize-evaluation-metrics-formats.md) + [Formatos de conjunto de dados compatíveis para Bring-Your-Own-Dataset tarefas (BYOD)](model-customize-evaluation-dataset-formats.md) + [Avalie com pontuadores predefinidos e personalizados](model-customize-evaluation-preset-custom-scorers.md) # Conceitos básicos ## Envie um Job de avaliação por meio do SageMaker Studio ### Etapa 1: Navegue até a avaliação a partir do seu cartão modelo Depois de personalizar seu modelo, navegue até a página de avaliação a partir do cartão do modelo. Para obter informações sobre treinamento em modelos personalizados de peso aberto: [https://docs.aws.amazon.com/sagemaker/latest/dg/model- .html customize-open-weight-job](https://docs.aws.amazon.com/sagemaker/latest/dg/model-customize-open-weight-job.html) SageMaker visualiza seu modelo personalizado na guia Meus modelos: ![\[Página do cartão de modelo registrado\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/getting-started-registered-model-card.png) Escolha Exibir versão mais recente e, em seguida, escolha Avaliar: ![\[Página de personalização do modelo\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/getting-started-evaluate-from-model-card.png) ### Etapa 2: Envie seu trabalho de avaliação Escolha o botão Enviar e envie seu trabalho de avaliação. Isso envia um trabalho mínimo de benchmark de MMLU. Para obter informações sobre os tipos de trabalho de avaliação suportados, consulte[Tipos de avaliação e envio de trabalhos](model-customize-evaluation-types.md). ![\[Página de envio de trabalhos de avaliação\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/getting-started-benchmark-submission.png) ### Etapa 3: Acompanhe o progresso do seu trabalho de avaliação O progresso do seu trabalho de avaliação é monitorado na guia Etapas de avaliação: ![\[Seu progresso no trabalho de avaliação\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/getting-started-benchmark-tracking.png) ### Etapa 4: Veja os resultados do seu trabalho de avaliação Os resultados do seu trabalho de avaliação são visualizados na guia Resultados da avaliação: ![\[Suas métricas de avaliação do trabalho\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/getting-started-benchmark-results.png) ### Etapa 5: veja suas avaliações concluídas Seu trabalho de avaliação concluído é exibido em Avaliações do seu cartão modelo: ![\[Seus trabalhos de avaliação concluídos\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/getting-started-benchmark-completed-model-card.png) ## Envie seu trabalho de avaliação por meio do SageMaker Python SDK ### Etapa 1: Crie seu BenchMarkEvaluator Passe seu modelo treinado registrado, o local de saída do AWS S3 e o ARN do MLFlow recurso e, em seguida, `BenchMarkEvaluator` inicialize-o. ``` from sagemaker.train.evaluate import BenchMarkEvaluator, Benchmark evaluator = BenchMarkEvaluator( benchmark=Benchmark.MMLU, model="arn:aws:sagemaker:::model-package//", s3_output_path="s3:////eval/", mlflow_resource_arn="arn:aws:sagemaker:::mlflow-tracking-server/", evaluate_base_model=False ) ``` ### Etapa 2: Envie seu trabalho de avaliação Chame o `evaluate()` método para enviar o trabalho de avaliação. ``` execution = evaluator.evaluate() ``` ### Etapa 3: Acompanhe o progresso do seu trabalho de avaliação Chame o `wait()` método de execução para obter uma atualização ao vivo do progresso do trabalho de avaliação. ``` execution.wait(target_status="Succeeded", poll=5, timeout=3600) ``` ### Etapa 4: Veja os resultados do seu trabalho de avaliação Chame o `show_results()` método para exibir os resultados do seu trabalho de avaliação. ``` execution.show_results() ``` # Tipos de avaliação e envio de trabalhos ## Avaliação comparativa com conjuntos de dados padronizados Use o tipo de avaliação de referência para avaliar a qualidade do seu modelo em conjuntos de dados de referência padronizados, incluindo conjuntos de dados populares como MMLU e BBH. | Referência | Conjunto de dados personalizado suportado | Modalidades | Description | Metrics | Estratégia | Subtarefa disponível | | --- | --- | --- | --- | --- | --- | --- | | mmlu | Não | Texto | Compreensão de linguagem multitarefa: testa o conhecimento em 57 disciplinas. | accuracy | zs\$1cot | Sim | | mmlu\$1pro | Não | Texto | MMLU (subconjunto profissional): concentra-se em domínios profissionais como direito, medicina, contabilidade e engenharia. | accuracy | zs\$1cot | Não | | bbh | Não | Texto | Tarefas de raciocínio avançado: um conjunto de problemas desafiadores que testam habilidades cognitivas e de resolução de problemas de nível mais alto. | accuracy | fs\$1cot | Sim | | gpqa | Não | Texto | Resposta a perguntas de física geral: avalia a compreensão de conceitos de física e habilidades relacionadas à resolução de problemas. | accuracy | zs\$1cot | Não | | math | Não | Texto | Resolução de problemas matemáticos: mede o raciocínio matemático em vários tópicos, como álgebra, cálculo e problemas de palavras. | exact\$1match | zs\$1cot | Sim | | strong\$1reject | Não | Texto | Tarefa de controle de qualidade — testa a capacidade do modelo de detectar e rejeitar conteúdo impróprio, prejudicial ou incorreto. | deflection | zs | Sim | | ifeval | Não | Texto | Avaliação de seguimento de instruções: avalia com que precisão um modelo segue as instruções dadas e conclui as tarefas conforme as especificações. | accuracy | zs | Não | Para obter mais informações sobre formatos BYOD, consulte[Formatos de conjunto de dados compatíveis para Bring-Your-Own-Dataset tarefas (BYOD)](model-customize-evaluation-dataset-formats.md). ### Subtarefas disponíveis A seguir, listamos as subtarefas disponíveis para avaliação do modelo em vários domínios, incluindo MMLU (Massive Multitask Language Understanding), BBH (Big Bench Hard) e MATH. StrongReject Essas subtarefas permitem avaliar o desempenho do modelo em capacidades e áreas de conhecimento específicas. **Subtarefas do MMLU** ``` MMLU_SUBTASKS = [ "abstract_algebra", "anatomy", "astronomy", "business_ethics", "clinical_knowledge", "college_biology", "college_chemistry", "college_computer_science", "college_mathematics", "college_medicine", "college_physics", "computer_security", "conceptual_physics", "econometrics", "electrical_engineering", "elementary_mathematics", "formal_logic", "global_facts", "high_school_biology", "high_school_chemistry", "high_school_computer_science", "high_school_european_history", "high_school_geography", "high_school_government_and_politics", "high_school_macroeconomics", "high_school_mathematics", "high_school_microeconomics", "high_school_physics", "high_school_psychology", "high_school_statistics", "high_school_us_history", "high_school_world_history", "human_aging", "human_sexuality", "international_law", "jurisprudence", "logical_fallacies", "machine_learning", "management", "marketing", "medical_genetics", "miscellaneous", "moral_disputes", "moral_scenarios", "nutrition", "philosophy", "prehistory", "professional_accounting", "professional_law", "professional_medicine", "professional_psychology", "public_relations", "security_studies", "sociology", "us_foreign_policy", "virology", "world_religions" ] ``` **Subtarefas do BBH** ``` BBH_SUBTASKS = [ "boolean_expressions", "causal_judgement", "date_understanding", "disambiguation_qa", "dyck_languages", "formal_fallacies", "geometric_shapes", "hyperbaton", "logical_deduction_five_objects", "logical_deduction_seven_objects", "logical_deduction_three_objects", "movie_recommendation", "multistep_arithmetic_two", "navigate", "object_counting", "penguins_in_a_table", "reasoning_about_colored_objects", "ruin_names", "salient_translation_error_detection", "snarks", "sports_understanding", "temporal_sequences", "tracking_shuffled_objects_five_objects", "tracking_shuffled_objects_seven_objects", "tracking_shuffled_objects_three_objects", "web_of_lies", "word_sorting" ] ``` **Subtarefas matemáticas** ``` MATH_SUBTASKS = [ "algebra", "counting_and_probability", "geometry", "intermediate_algebra", "number_theory", "prealgebra", "precalculus" ] ``` **StrongReject Subtarefas** ``` STRONG_REJECT_SUBTASKS = [ "gcg_transfer_harmbench", "gcg_transfer_universal_attacks", "combination_3", "combination_2", "few_shot_json", "dev_mode_v2", "dev_mode_with_rant", "wikipedia_with_title", "distractors", "wikipedia", "style_injection_json", "style_injection_short", "refusal_suppression", "prefix_injection", "distractors_negated", "poems", "base64", "base64_raw", " base64_input_only", "base64_output_only", "evil_confidant", "aim", "rot_13", "disemvowel", "auto_obfuscation", "auto_payload_splitting", "pair", "pap_authority_endorsement", "pap_evidence_based_persuasion", "pap_expert_endorsement", "pap_logical_appeal", "pap_misrepresentation" ] ``` ### Envie seu trabalho de referência ------ #### [ SageMaker Studio ] ![\[Uma configuração mínima para benchmarking por meio do Studio SageMaker\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/benchmark-submission-sagemaker-studio.png) ------ #### [ SageMaker Python SDK ] ``` from sagemaker.train.evaluate import get_benchmarks from sagemaker.train.evaluate import BenchMarkEvaluator Benchmark = get_benchmarks() # Create evaluator with MMLU benchmark evaluator = BenchMarkEvaluator( benchmark=Benchmark.MMLU, model="arn:aws:sagemaker:::model-package//", s3_output_path="s3:////", evaluate_base_model=False ) execution = evaluator.evaluate() ``` Para obter mais informações sobre o envio de trabalhos de avaliação por meio do SageMaker Python SDK, consulte: [https://sagemaker.readthedocs.io/en/stable/model\$1customization/evaluation.html](https://sagemaker.readthedocs.io/en/stable/model_customization/evaluation.html) ------ ## Avaliação do Large Language Model as a Judge (LLMAJ) Use a avaliação LLM-as-a-judge (LLMAJ) para aproveitar outro modelo de fronteira para avaliar as respostas do modelo de destino. Você pode usar os modelos AWS Bedrock como juízes chamando a `create_evaluation_job` API para iniciar o trabalho de avaliação. Para obter mais informações sobre os modelos de juízes suportados, consulte: [https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html) Você pode usar dois formatos de métrica diferentes para definir a avaliação: + **Métricas integradas: aproveite as métricas** integradas do AWS Bedrock para analisar a qualidade das respostas de inferência do seu modelo. Para obter mais informações, consulte: [https://docs.aws.amazon.com/bedrock/latest/userguide/model- evaluation-type-judge-prompt .html](https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-type-judge-prompt.html) + **Métricas personalizadas:** defina suas próprias métricas personalizadas no formato de métrica personalizada do Bedrock Evaluation para analisar a qualidade das respostas de inferência do seu modelo usando suas próprias instruções. Para obter mais informações, consulte: [https://docs.aws.amazon.com/bedrock/latest/userguide/model- evaluation-custom-metrics-prompt -formats.html](https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-custom-metrics-prompt-formats.html) ### Envie uma tarefa LLMAJ com métricas integradas ------ #### [ SageMaker Studio ] ![\[Uma configuração mínima para benchmarking do LLMAJ por meio do Studio SageMaker\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/llmaj-as-judge-submission-sagemaker-studio.png) ------ #### [ SageMaker Python SDK ] ``` from sagemaker.train.evaluate import LLMAsJudgeEvaluator evaluator = LLMAsJudgeEvaluator( model="arn:aws:sagemaker:::model-package//", evaluator_model="", dataset="s3:////.jsonl", builtin_metrics=["", ""], s3_output_path="s3:////", evaluate_base_model=False ) execution = evaluator.evaluate() ``` Para obter mais informações sobre o envio de trabalhos de avaliação por meio do SageMaker Python SDK, consulte: [https://sagemaker.readthedocs.io/en/stable/model\$1customization/evaluation.html](https://sagemaker.readthedocs.io/en/stable/model_customization/evaluation.html) ------ ### Envie um trabalho LLMAJ com métricas personalizadas Defina suas métricas personalizadas: ``` { "customMetricDefinition": { "name": "PositiveSentiment", "instructions": ( "You are an expert evaluator. Your task is to assess if the sentiment of the response is positive. " "Rate the response based on whether it conveys positive sentiment, helpfulness, and constructive tone.\n\n" "Consider the following:\n" "- Does the response have a positive, encouraging tone?\n" "- Is the response helpful and constructive?\n" "- Does it avoid negative language or criticism?\n\n" "Rate on this scale:\n" "- Good: Response has positive sentiment\n" "- Poor: Response lacks positive sentiment\n\n" "Here is the actual task:\n" "Prompt: {{prompt}}\n" "Response: {{prediction}}" ), "ratingScale": [ {"definition": "Good", "value": {"floatValue": 1}}, {"definition": "Poor", "value": {"floatValue": 0}} ] } } ``` Para obter mais informações, consulte: [https://docs.aws.amazon.com/bedrock/latest/userguide/model- evaluation-custom-metrics-prompt -formats.html](https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-custom-metrics-prompt-formats.html) ------ #### [ SageMaker Studio ] ![\[Faça o upload da métrica personalizada por meio de Métricas personalizadas > Adicionar métricas personalizadas\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/custom-llmaj-metrics-submission-sagemaker-studio.png) ------ #### [ SageMaker Python SDK ] ``` evaluator = LLMAsJudgeEvaluator( model="arn:aws:sagemaker:::model-package//", evaluator_model="", dataset="s3:////.jsonl", custom_metrics=custom_metric_dict = { "customMetricDefinition": { "name": "PositiveSentiment", "instructions": ( "You are an expert evaluator. Your task is to assess if the sentiment of the response is positive. " "Rate the response based on whether it conveys positive sentiment, helpfulness, and constructive tone.\n\n" "Consider the following:\n" "- Does the response have a positive, encouraging tone?\n" "- Is the response helpful and constructive?\n" "- Does it avoid negative language or criticism?\n\n" "Rate on this scale:\n" "- Good: Response has positive sentiment\n" "- Poor: Response lacks positive sentiment\n\n" "Here is the actual task:\n" "Prompt: {{prompt}}\n" "Response: {{prediction}}" ), "ratingScale": [ {"definition": "Good", "value": {"floatValue": 1}}, {"definition": "Poor", "value": {"floatValue": 0}} ] } }, s3_output_path="s3:////", evaluate_base_model=False ) ``` ------ ## Pontuadores personalizados Defina sua própria função de pontuação personalizada para iniciar um trabalho de avaliação. O sistema fornece dois marcadores integrados: Prime math e Prime code. Você também pode trazer sua própria função de marcador. Você pode copiar o código da função do marcador diretamente ou trazer sua própria definição de função Lambda usando o ARN associado. Por padrão, os dois tipos de pontuador produzem resultados de avaliação que incluem métricas padrão, como pontuação na F1, ROUGE e BLEU. Para obter mais informações sobre pontuadores integrados e personalizados e seus respectivos requisitos/contratos, consulte. [Avalie com pontuadores predefinidos e personalizados](model-customize-evaluation-preset-custom-scorers.md) ### Registre seu conjunto de dados Traga seu próprio conjunto de dados para um marcador personalizado registrando-o como um conjunto de dados do SageMaker Hub Content. ------ #### [ SageMaker Studio ] No Studio, faça o upload do seu conjunto de dados usando a página dedicada de conjuntos de dados. ![\[Conjunto de dados de avaliação registrado no Studio SageMaker\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/dataset-registration-sagemaker-studio.png) ------ #### [ SageMaker Python SDK ] No SDK do SageMaker Python, faça o upload do seu conjunto de dados usando a página dedicada de conjuntos de dados. ``` from sagemaker.ai_registry.dataset import DataSet dataset = DataSet.create( name="your-bring-your-own-dataset", source="s3:////.jsonl" ) dataset.refresh() ``` ------ ### Envie um trabalho de pontuação integrado ------ #### [ SageMaker Studio ] ![\[Selecione entre execuções de código ou respostas matemáticas para obter pontuação personalizada integrada\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/builtin-scorer-submission-sagemaker-studio.png) ------ #### [ SageMaker Python SDK ] ``` from sagemaker.train.evaluate import CustomScorerEvaluator from sagemaker.train.evaluate import get_builtin_metrics BuiltInMetric = get_builtin_metrics() evaluator_builtin = CustomScorerEvaluator( evaluator=BuiltInMetric.PRIME_MATH, dataset="arn:aws:sagemaker:::hub-content//DataSet/your-bring-your-own-dataset/", model="arn:aws:sagemaker:::model-package//", s3_output_path="s3:////", evaluate_base_model=False ) execution = evaluator.evaluate() ``` Selecione a partir de `BuiltInMetric.PRIME_MATH` ou `BuiltInMetric.PRIME_CODE` para a pontuação incorporada. ------ ### Envie um trabalho de pontuação personalizado Defina uma função de recompensa personalizada. Para obter mais informações, consulte [Pontuadores personalizados (traga suas próprias métricas)](model-customize-evaluation-preset-custom-scorers.md#model-customize-evaluation-custom-scorers-byom). **Registre a função de recompensa personalizada** ------ #### [ SageMaker Studio ] ![\[Navegando até SageMaker Studio > Assets > Evaluator > Criar avaliador > Criar função de recompensa\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/custom-scorer-submission-sagemaker-studio.png) ![\[Envie o trabalho de avaliação do Custom Scorer referenciando a função de recompensa predefinida registrada em Custom Scorer > Custom Scorer > Custom metrics\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/custom-scorer-benchmark-submission-sagemaker-studio.png) ------ #### [ SageMaker Python SDK ] ``` from sagemaker.ai_registry.evaluator import Evaluator from sagemaker.ai_registry.air_constants import REWARD_FUNCTION evaluator = Evaluator.create( name = "your-reward-function-name", source="/path_to_local/custom_lambda_function.py", type = REWARD_FUNCTION ) ``` ``` evaluator = CustomScorerEvaluator( evaluator=evaluator, dataset="s3:////.jsonl", model="arn:aws:sagemaker:::model-package//", s3_output_path="s3:////", evaluate_base_model=False ) execution = evaluator.evaluate() ``` ------ # Formatos de métricas de avaliação Avaliando a qualidade do seu modelo nesses formatos métricos: + Resumo da avaliação do modelo + MLFlow + TensorBoard ## Resumo da avaliação do modelo Ao enviar seu trabalho de avaliação, você especifica um local de saída do AWS S3. SageMaker carrega automaticamente o arquivo.json de resumo da avaliação para o local. O caminho do resumo do benchmark para o S3 é o seguinte: ``` s3:////output/output//eval_results/ ``` **Passe a localização do AWS S3** ------ #### [ SageMaker Studio ] ![\[Passar para a localização do artefato de saída (URI AWS S3)\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/s3-output-path-submission-sagemaker-studio.png) ------ #### [ SageMaker Python SDK ] ``` evaluator = BenchMarkEvaluator( benchmark=Benchmark.MMLU, model="arn:aws:sagemaker:::model-package//", s3_output_path="s3:////eval/", evaluate_base_model=False ) execution = evaluator.evaluate() ``` ------ Leia-o diretamente do local `.json` do AWS S3 ou visualizado automaticamente na interface do usuário: ``` { "results": { "custom|gen_qa_gen_qa|0": { "rouge1": 0.9152812653966208, "rouge1_stderr": 0.003536439199232507, "rouge2": 0.774569918517409, "rouge2_stderr": 0.006368825746765958, "rougeL": 0.9111255645823356, "rougeL_stderr": 0.003603841524881021, "em": 0.6562150055991042, "em_stderr": 0.007948251702846893, "qem": 0.7522396416573348, "qem_stderr": 0.007224355240883467, "f1": 0.8428757602152095, "f1_stderr": 0.005186300690881584, "f1_score_quasi": 0.9156170336744968, "f1_score_quasi_stderr": 0.003667700152375464, "bleu": 100.00000000000004, "bleu_stderr": 1.464411857851008 }, "all": { "rouge1": 0.9152812653966208, "rouge1_stderr": 0.003536439199232507, "rouge2": 0.774569918517409, "rouge2_stderr": 0.006368825746765958, "rougeL": 0.9111255645823356, "rougeL_stderr": 0.003603841524881021, "em": 0.6562150055991042, "em_stderr": 0.007948251702846893, "qem": 0.7522396416573348, "qem_stderr": 0.007224355240883467, "f1": 0.8428757602152095, "f1_stderr": 0.005186300690881584, "f1_score_quasi": 0.9156170336744968, "f1_score_quasi_stderr": 0.003667700152375464, "bleu": 100.00000000000004, "bleu_stderr": 1.464411857851008 } } } ``` ![\[Exemplos de métricas de desempenho para um benchmark gen-qa personalizado visualizado no Studio SageMaker\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/gen-qa-metrics-visualization-sagemaker-studio.png) ## MLFlow registro **Forneça o ARN SageMaker MLFlow do seu recurso** SageMaker O Studio usa o MLFlow aplicativo padrão que é provisionado em cada domínio do Studio quando você usa o recurso de personalização do modelo pela primeira vez. SageMaker O Studio usa o ARN associado ao MLflow aplicativo padrão no envio do trabalho de avaliação. Você também pode enviar seu trabalho de avaliação e fornecer explicitamente um ARN de MLFlow recurso para transmitir métricas ao server/app rastreamento associado para análise em tempo real. **SageMaker SDK para Python** ``` evaluator = BenchMarkEvaluator( benchmark=Benchmark.MMLU, model="arn:aws:sagemaker:::model-package//", s3_output_path="s3:////eval/", mlflow_resource_arn="arn:aws:sagemaker:::mlflow-tracking-server/", evaluate_base_model=False ) execution = evaluator.evaluate() ``` Visualização métrica no nível do modelo e no nível do sistema: ![\[Erro e precisão do nível do modelo de amostra para a tarefa de benchmarking MMLU\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/model-metrics-mlflow.png) ![\[Exemplo de métricas integradas para a tarefa de benchmarking do LLMAJ\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/llmaj-metrics-mlflow.png) ![\[Amostra de métricas em nível de sistema para a tarefa de benchmarking do MMLU\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/system-metrics-mlflow.png) ## TensorBoard Envie seu trabalho de avaliação com um local de saída do AWS S3. SageMaker carrega automaticamente um TensorBoard arquivo para o local. SageMaker carrega o TensorBoard arquivo para o AWS S3 no seguinte local: ``` s3:////output/output//tensorboard_results/eval/ ``` **Passe a localização AWS S3 da seguinte forma** ------ #### [ SageMaker Studio ] ![\[Passar para a localização do artefato de saída (URI AWS S3)\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/s3-output-path-submission-sagemaker-studio.png) ------ #### [ SageMaker Python SDK ] ``` evaluator = BenchMarkEvaluator( benchmark=Benchmark.MMLU, model="arn:aws:sagemaker:::model-package//", s3_output_path="s3:////eval/", evaluate_base_model=False ) execution = evaluator.evaluate() ``` ------ **Amostra de métricas em nível de modelo** ![\[SageMaker TensorBoard exibindo resultados de um trabalho de benchmarking\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/metrics-in-tensorboard.png) # Formatos de conjunto de dados compatíveis para Bring-Your-Own-Dataset tarefas (BYOD) O marcador personalizado e os tipos de LLM-as-judge avaliação exigem um arquivo JSONL de conjunto de dados personalizado localizado no S3. AWS Você deve fornecer o arquivo como um arquivo JSON Lines, aderindo a um dos seguintes formatos compatíveis. Os exemplos neste documento foram expandidos para maior clareza. Cada formato tem suas próprias nuances, mas, no mínimo, todos exigem um prompt do usuário. **Campos obrigatórios** | Campo | Obrigatório | | --- | --- | | Prompt do usuário | Sim | | Prompt do sistema | Não | | Verdade fundamental | Somente para o Custom Scorer | | Categoria | Não | **1. Formato OpenAI** ``` { "messages": [ { "role": "system", # System prompt (looks for system role) "content": "You are a helpful assistant." }, { "role": "user", # Query (looks for user role) "content": "Hello!" }, { "role": "assistant", # Ground truth (looks for assistant role) "content": "Hello to you!" } ] } ``` **2. SageMaker **Avaliação ``` { "system":"You are an English major with top marks in class who likes to give minimal word responses: ", "query":"What is the symbol that ends the sentence as a question", "response":"?", # Ground truth "category": "Grammar" } ``` **3. HuggingFace Conclusão rápida** Os formatos padrão e conversacional são suportados. ``` # Standard { "prompt" : "What is the symbol that ends the sentence as a question", # Query "completion" : "?" # Ground truth } # Conversational { "prompt": [ { "role": "user", # Query (looks for user role) "content": "What is the symbol that ends the sentence as a question" } ], "completion": [ { "role": "assistant", # Ground truth (looks for assistant role) "content": "?" } ] } ``` **4. HuggingFace Preferência** Support tanto para formato padrão (string) quanto para formato conversacional (matriz de mensagens). ``` # Standard: {"prompt": "text", "chosen": "text", "rejected": "text"} { "prompt" : "The sky is", # Query "chosen" : "blue", # Ground truth "rejected" : "green" } # Conversational: { "prompt": [ { "role": "user", # Query (looks for user role) "content": "What color is the sky?" } ], "chosen": [ { "role": "assistant", # Ground truth (looks for assistant role) "content": "It is blue." } ], "rejected": [ { "role": "assistant", "content": "It is green." } ] } ``` **5. Formato Verl** O formato Verl (formatos atual e antigo) é compatível com casos de uso de aprendizado por reforço. Documentos Verl para referência: [https://verl.readthedocs.io/en/latest/preparation/prepare\$1data.html](https://verl.readthedocs.io/en/latest/preparation/prepare_data.html) Os usuários do formato VERL normalmente não fornecem uma resposta verdadeira. Se você quiser fornecer um de qualquer maneira, use um dos campos `extra_info.answer` ou`reward_model.ground_truth`; `extra_info` tem precedência. SageMaker preserva os seguintes campos específicos do VERL como metadados, se presentes: + `id` + `data_source` + `ability` + `reward_model` + `extra_info` + `attributes` + `difficulty` ``` # Newest VERL format where `prompt` is an array of messages. { "data_source": "openai/gsm8k", "prompt": [ { "content": "You are a helpful math tutor who explains solutions to questions step-by-step.", "role": "system" }, { "content": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? Let's think step by step and output the final answer after \"####\".", "role": "user" } ], "ability": "math", "extra_info": { "answer": "Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nNatalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.\n#### 72", "index": 0, "question": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?", "split": "train" }, "reward_model": { "ground_truth": "72" # Ignored in favor of extra_info.answer } } # Legacy VERL format where `prompt` is a string. Also supported. { "data_source": "openai/gsm8k", "prompt": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? Let's think step by step and output the final answer after \"####\".", "extra_info": { "answer": "Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nNatalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.\n#### 72", "index": 0, "question": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?", "split": "train" } } ``` # Avalie com pontuadores predefinidos e personalizados Ao usar o tipo de avaliação Custom Scorer, o SageMaker Evaluation suporta dois pontuadores integrados (também chamados de “funções de recompensa”) Prime Math e Prime Code retirados da biblioteca de treinamento [volcengine/verl RL](https://github.com/volcengine/verl), ou seu próprio marcador personalizado implementado como uma função Lambda. ## Marcadores integrados **Matemática principal** O principal marcador matemático espera um conjunto de dados JSONL personalizado de entradas contendo uma pergunta matemática como a prompt/query e a resposta correta como verdade fundamental. O conjunto de dados pode ser qualquer um dos formatos compatíveis mencionados em[Formatos de conjunto de dados compatíveis para Bring-Your-Own-Dataset tarefas (BYOD)](model-customize-evaluation-dataset-formats.md). Exemplo de entrada do conjunto de dados (expandido para maior clareza): ``` { "system":"You are a math expert: ", "query":"How many vertical asymptotes does the graph of $y=\\frac{2}{x^2+x-6}$ have?", "response":"2" # Ground truth aka correct answer } ``` **Código Prime** O marcador de código principal espera um conjunto de dados JSONL personalizado de entradas contendo um problema de codificação e casos de teste especificados no campo. `metadata` Estruture os casos de teste com o nome da função esperado para cada entrada, entradas de amostra e saídas esperadas. Exemplo de entrada do conjunto de dados (expandido para maior clareza): ``` { "system":"\\nWhen tackling complex reasoning tasks, you have access to the following actions. Use them as needed to progress through your thought process.\\n\\n[ASSESS]\\n\\n[ADVANCE]\\n\\n[VERIFY]\\n\\n[SIMPLIFY]\\n\\n[SYNTHESIZE]\\n\\n[PIVOT]\\n\\n[OUTPUT]\\n\\nYou should strictly follow the format below:\\n\\n[ACTION NAME]\\n\\n# Your action step 1\\n\\n# Your action step 2\\n\\n# Your action step 3\\n\\n...\\n\\nNext action: [NEXT ACTION NAME]\\n\\n", "query":"A number N is called a factorial number if it is the factorial of a positive integer. For example, the first few factorial numbers are 1, 2, 6, 24, 120,\\nGiven a number N, the task is to return the list/vector of the factorial numbers smaller than or equal to N.\\nExample 1:\\nInput: N = 3\\nOutput: 1 2\\nExplanation: The first factorial number is \\n1 which is less than equal to N. The second \\nnumber is 2 which is less than equal to N,\\nbut the third factorial number is 6 which \\nis greater than N. So we print only 1 and 2.\\nExample 2:\\nInput: N = 6\\nOutput: 1 2 6\\nExplanation: The first three factorial \\nnumbers are less than equal to N but \\nthe fourth factorial number 24 is \\ngreater than N. So we print only first \\nthree factorial numbers.\\nYour Task: \\nYou don't need to read input or print anything. Your task is to complete the function factorialNumbers() which takes an integer N as an input parameter and return the list/vector of the factorial numbers smaller than or equal to N.\\nExpected Time Complexity: O(K), Where K is the number of factorial numbers.\\nExpected Auxiliary Space: O(1)\\nConstraints:\\n1<=N<=10^{18}\\n\\nWrite Python code to solve the problem. Present the code in \\n```python\\nYour code\\n```\\nat the end.", "response": "", # Dummy string for ground truth. Provide a value if you want NLP metrics like ROUGE, BLEU, and F1. ### Define test cases in metadata field "metadata": { "fn_name": "factorialNumbers", "inputs": ["5"], "outputs": ["[1, 2]"] } } ``` ## Pontuadores personalizados (traga suas próprias métricas) Personalize totalmente seu fluxo de trabalho de avaliação de modelos com uma lógica de pós-processamento personalizada que permite calcular métricas personalizadas de acordo com suas necessidades. Você deve implementar seu marcador personalizado como uma função AWS Lambda que aceita respostas do modelo e retorna pontuações de recompensa. ### Exemplo de carga útil de entrada Lambda Seu AWS Lambda personalizado espera entradas no formato OpenAI. Exemplo: ``` { "id": "123", "messages": [ { "role": "user", "content": "Do you have a dedicated security team?" }, { "role": "assistant", "content": "As an AI developed by Amazon, I do not have a dedicated security team..." } ], "reference_answer": { "compliant": "No", "explanation": "As an AI developed by Company, I do not have a traditional security team..." } } ``` ### Exemplo de carga útil de saída Lambda O contêiner de SageMaker avaliação espera que suas respostas do Lambda sigam este formato: ``` { "id": str, # Same id as input sample "aggregate_reward_score": float, # Overall score for the sample "metrics_list": [ # OPTIONAL: Component scores { "name": str, # Name of the component score "value": float, # Value of the component score "type": str # "Reward" or "Metric" } ] } ``` ### Definição personalizada de Lambda [Encontre um exemplo de um marcador personalizado totalmente implementado com entrada de amostra e saída esperada em: https://docs.aws.amazon.com/sagemaker/ latest/dg/nova - .html\$1 -example implementing-reward-functions nova-reward-llm-judge](https://docs.aws.amazon.com/sagemaker/latest/dg/nova-implementing-reward-functions.html#nova-reward-llm-judge-example) Use o esqueleto a seguir como ponto de partida para sua própria função. ``` def lambda_handler(event, context): return lambda_grader(event) def lambda_grader(samples: list[dict]) -> list[dict]: """ Args: Samples: List of dictionaries in OpenAI format Example input: { "id": "123", "messages": [ { "role": "user", "content": "Do you have a dedicated security team?" }, { "role": "assistant", "content": "As an AI developed by Company, I do not have a dedicated security team..." } ], # This section is the same as your training dataset "reference_answer": { "compliant": "No", "explanation": "As an AI developed by Company, I do not have a traditional security team..." } } Returns: List of dictionaries with reward scores: { "id": str, # Same id as input sample "aggregate_reward_score": float, # Overall score for the sample "metrics_list": [ # OPTIONAL: Component scores { "name": str, # Name of the component score "value": float, # Value of the component score "type": str # "Reward" or "Metric" } ] } """ ``` ### Campos de entrada e saída **Campos de entrada** | Campo | Description | Notas adicionais | | --- | --- | --- | | id | Identificador exclusivo para a amostra | Ecoou de volta na saída. Formato de string | | mensagens | Histórico de bate-papo solicitado no formato OpenAI | Matriz de objetos de mensagem | | mensagens [] .role | Palestrante da mensagem | Valores comuns: “usuário”, “assistente”, “sistema” | | mensagens [] .content | Conteúdo de texto da mensagem | String simples | | metadata | Informações de formato livre para auxiliar na avaliação | Objeto; campos opcionais transmitidos dos dados de treinamento | **Campos de saída** **Campos de saída** | Campo | Description | Notas adicionais | | --- | --- | --- | | id | Mesmo identificador da amostra de entrada | Deve corresponder à entrada | | pontuação\$1remuneração\$1agregada | Pontuação geral da amostra | Flutuação (por exemplo, 0,0—1,0 ou intervalo definido pela tarefa) | | lista\$1de\$1métricas | Pontuações de componentes que compõem o agregado | Matriz de objetos métricos | ### Permissões obrigatórias Certifique-se de que a função de SageMaker execução que você usa para executar a avaliação tenha permissões do AWS Lambda. ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "lambda:InvokeFunction" ], "Resource": "arn:aws:lambda:region:account-id:function:function-name" } ] } ``` Certifique-se de que sua função de execução AWS da Função Lambda tenha permissões básicas de execução do Lambda, bem como permissões adicionais que você possa precisar para qualquer chamada downstream. AWS ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": "arn:aws:logs:*:*:*" } ] } ``` # Implantação de modelos Na página de detalhes dos modelos personalizados, você também pode implantar seu modelo personalizado usando endpoints de inferência de SageMaker IA ou Amazon Bedrock. ![\[Uma imagem contendo uma seleção de técnicas de personalização do modelo.\]](http://docs.aws.amazon.com/pt_br/sagemaker/latest/dg/images/screenshot-open-model-1.png) # Amostras de conjuntos de dados e avaliadores ## Ajuste fino supervisionado (SFT) + Nome: TAT-QA + Licença: CC-BY-4.0 + Link: [https://huggingface. co/datasets/next-tat/TAT-QA](https://huggingface.co/datasets/next-tat/TAT-QA) + Pré-processamento - Formatação **Uma amostra** ``` { "prompt": "Given a table and relevant text descriptions, answer the following question.\n\nTable:\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
20192018
$'000$'000
Revenue from external customers
Australia144,621129,431
New Zealand13,0368,912
Total157,657138,343
\n\nParagraphs:\n 4. SEGMENT INFORMATION\n\n During the 2019 and 2018 financial years, the Group operated wholly within one business segment being the operation and management of storage centres in Australia and New Zealand.\n\n The Managing Director is the Group\u2019s chief operating decision maker and monitors the operating results on a portfolio wide basis. Monthly management reports are evaluated based upon the overall performance of NSR consistent with the presentation within the consolidated financial statements. The Group\u2019s financing (including finance costs and finance income) are managed on a Group basis and not allocated to operating segments.\n\n The operating results presented in the statement of profit or loss represent the same segment information as reported in internal management information.\n\n The revenue information above excludes interest income and is based on the location of storage centres.\n\nQuestion: How many business segments were present in 2019 and 2018?\n\nAnswer:", "completion": "one", "data_idx": "2951" } ``` ## Otimização direta de preferências (DPO) + Nome: Ultrafeedback + Licença: MIT + Link: [https://huggingface. co/datasets/openbmb/UltraFeedback](https://huggingface.co/datasets/openbmb/UltraFeedback) + Etapas de pré-processamento: formatação e subamostragem. **Uma amostra** ``` { "source": "evol_instruct", "prompt": "Can you write a C++ program that prompts the user to enter the name of a country and checks if it borders the Mediterranean Sea? Here's some starter code to help you out:\n#include \n#include \nusing namespace std;\nint main() {\n string country;\n // prompt user for input\n cout << \"Enter the name of a country: \";\n cin >> country;\n // check if country borders the Mediterranean Sea\n // [C++ code]\n return 0;\n}", "chosen": "Here's a C++ program that prompts the user to enter the name of a country and checks if it borders the Mediterranean Sea:\n\n#include \n#include \n#include \n#include \n#include \n\nusing namespace std;\n\nint main() {\n // store countries and their bordering seas in a map\n map> countries;\n countries[\"Algeria\"] = {\"Mediterranean Sea\", \"North African Coast\"};\n countries[\"France\"] = {\"Mediterranean Sea\", \"English Channel\"};\n countries[\"Italy\"] = {\"Mediterranean Sea\", \"Adriatic Sea\"};\n countries[\"Malta\"] = {\"Mediterranean Sea\", \"North African Coast\"};\n countries[\"Spain\"] = {\"Mediterranean Sea\", \"Atlantic Ocean\"};\n countries[\"Tunisia\"] = {\"Mediterranean Sea\", \"North African Coast\"};\n\n string country;\n cout << \"Enter the name of a country: \";\n cin >> country;\n\n // check if country is in the map\n if (countries.find(country)!= countries.end()) {\n // check if country borders the Mediterranean Sea\n set borders = countries.at(country);\n if (borders.find(\"Mediterranean Sea\")!= borders.end()) {\n cout << country << \" borders the Mediterranean Sea.\" << endl;\n } else {\n cout << country << \" does not border the Mediterranean Sea.\" << endl;\n }\n } else {\n cout << country << \" is not a recognized country.\" << endl;\n }\n\n return 0;\n}\n\nIn this program, we first define a map called countries that stores each country's name as a key and a set of its surrounding seas as the value. We use a set to store each sea name so that we can easily check if a country borders a certain sea.\n\nWe then present the user with a prompt to enter the name of a country. We use a while loop to ensure that the user enters a valid country name. We check if the country is in the map, and if so, we retrieve its bordering seas from the map.\n\nFinally, we check if the Mediterranean Sea is among the set of borders. If it is, we inform the user that the country borders the Mediterranean Sea. If not, we inform the user that the country does not border the Mediterranean Sea.", "chosen-rating": 5.0, "chosen-model": "starchat", "rejected": "Sure, here is the program using the C++11 algorithm \"cds::algorithm::GreaterEqual\":\n#include \n#include \n#include \n#include \n#include \n\nusing namespace std;\n\nint main() {\n string country;\n cout << \"Enter the name of a country: \";\n cin >> country;\n std::vector vec;\n vec.push_back(country);\n size_t index = std::find_if(vec.begin(), vec.end(), [](const string& s) {\n return std::any_of(s.begin(), s.end(), [](const char& c) {\n return c == '}}';\n });\n if (index != vec.end()) {\n if ((*index)[0] == 'M') {\n cout << \"Country is bordered by the Mediterranean Sea.\" << endl;\n } else {\n cout << \"Country does not border the Mediterranean Sea.\" << endl;\n }\n } else {\n cout << \"Country is not found.\" << endl;\n }\n return 0;\n}", "rejected-rating": 1.25, "rejected-model": "pythia-12b" } ``` ## Aprendizagem por reforço a partir do feedback de IA (RLAIF) - Julgamento em pares **Conjunto de dados de entrada** [Conjunto de dados de origem: https://github.com/WeOpenML/ PandaLM](https://github.com/WeOpenML/PandaLM) **Uma amostra** ``` { "data_source": "WeOpenML/PandaLM", "prompt": [ { "role": "user", "content": "Below are two responses for a given task. The task is defined by the Instruction with an Input that provides further context. Evaluate the responses and generate a reference answer for the task.\n\n ### Instruction:\nCompare the given products.\n\n### Input:\niPhone 11 and Google Pixel 4\n\n ### Response 1:\nThe iPhone 11 has a larger screen size and a longer battery life than the Google Pixel 4.\n\n ### Response 2:\nThe iPhone 11 and Google Pixel 4 are both flagship smartphones released in 2018. The iPhone 11 has a 6.1-inch LCD display, while the Google Pixel 4 has a 5.7-inch OLED display. The iPhone 11 has an A13 Bionic chipset, while the Google Pixel 4 has a Qualcomm Snapdragon 845 chipset. The iPhone 11 has a dual-camera system, while the Google Pixel 4 has a single camera system. The iPhone 11 has a longer battery life, while the Google Pixel 4 has a faster processor.\n\n### Evaluation:\n" } ], "ability": "pairwise-judging", "reward_model": { "style": "llmj", "ground_truth": "2\n\n### Reason: Response 2 provides a more detailed and comprehensive comparison of the two products, including their specifications and features. Response 1 only mentions two aspects of the products and does not provide as much information.\n\n### Reference: The iPhone 11 and Google Pixel 4 are both flagship smartphones released in 2018. The iPhone 11 has a 6.1-inch LCD display, while the Google Pixel 4 has a 5.7-inch OLED display. The iPhone 11 has an A13 Bionic chipset, while the Google Pixel 4 has a Qualcomm Snapdragon 845 chipset. The iPhone 11 has a dual-camera system, while the Google Pixel 4 has a single camera system. The iPhone 11 has a longer battery life, while the Google Pixel 4 has a faster processor." }, "extra_info": { "split": "train", "index": 0, "raw_output_sequence": "2\n\n### Reason: Response 2 provides a more detailed and comprehensive comparison of the two products, including their specifications and features. Response 1 only mentions two aspects of the products and does not provide as much information.\n\n### Reference: The iPhone 11 and Google Pixel 4 are both flagship smartphones released in 2018. The iPhone 11 has a 6.1-inch LCD display, while the Google Pixel 4 has a 5.7-inch OLED display. The iPhone 11 has an A13 Bionic chipset, while the Google Pixel 4 has a Qualcomm Snapdragon 845 chipset. The iPhone 11 has a dual-camera system, while the Google Pixel 4 has a single camera system. The iPhone 11 has a longer battery life, while the Google Pixel 4 has a faster processor.\n", "llmj": { "question": "Below are two responses for a given task. The task is defined by the Instruction with an Input that provides further context. Evaluate the responses and generate a reference answer for the task.\n\n### Instruction:\nCompare the given products.\n\n### Input:\niPhone 11 and Google Pixel 4\n\n### Response 1:\nThe iPhone 11 has a larger screen size and a longer battery life than the Google Pixel 4.\n\n### Response 2:\nThe iPhone 11 and Google Pixel 4 are both flagship smartphones released in 2018. The iPhone 11 has a 6.1-inch LCD display, while the Google Pixel 4 has a 5.7-inch OLED display. The iPhone 11 has an A13 Bionic chipset, while the Google Pixel 4 has a Qualcomm Snapdragon 845 chipset. The iPhone 11 has a dual-camera system, while the Google Pixel 4 has a single camera system. The iPhone 11 has a longer battery life, while the Google Pixel 4 has a faster processor.\n\n### Evaluation:\n", "ground_truth": "2\n\n### Reason: Response 2 provides a more detailed and comprehensive comparison of the two products, including their specifications and features. Response 1 only mentions two aspects of the products and does not provide as much information.\n\n### Reference: The iPhone 11 and Google Pixel 4 are both flagship smartphones released in 2018. The iPhone 11 has a 6.1-inch LCD display, while the Google Pixel 4 has a 5.7-inch OLED display. The iPhone 11 has an A13 Bionic chipset, while the Google Pixel 4 has a Qualcomm Snapdragon 845 chipset. The iPhone 11 has a dual-camera system, while the Google Pixel 4 has a single camera system. The iPhone 11 has a longer battery life, while the Google Pixel 4 has a faster processor.", "document_in_context": null }, "sample_size": 1980 } } ``` ## RLAIF - Cadeia de pensamento **Conjunto de dados de entrada** Dados de origem: [https://huggingface. co/datasets/thesven/gsm8k-reasoning/tree/main/data](https://huggingface.co/datasets/thesven/gsm8k-reasoning/tree/main/data) **Uma amostra** ``` { "data_source": "openai/gsm8k", "prompt": [ { "role": "system", "content": "\nYou are an AI assistant that uses a Chain of Thought (CoT) approach with reflection to answer queries.\n\nFollow these steps:\n- Think through the problem step by step within the ‹thinking> tags.\n- Reflect on your thinking\n\nto check for any errors or improvements within the ‹reflection› tags.\n- Make any necessary adjustments based on your reflection.\n- Provide your final, concise answer within the ‹output> tags.\n\nImportant: The and ‹reflection› sections are for your internal reasoning process only.\nDo not include any part of the final answer in these sections.\nThe actual response to the query must be entirely contained within the ‹output› tags.\n\nUse the following format for your response:\n\n[Your initial thought process goes here]\n\n[Your step-by-step reasoning goes here. This is your internal thought process, not the final answer. You can create as many reasoning steps as necessary in your process.]\n\n\n‹reflection>\n[Your reflection on your reasoning, checking for errors or improvements. You can create as many reflection steps as necessary in your process.]\n\n\n\n[Any adjustments to your thinking based on your reflection]\n\n\n\n[Your final, concise answer to the query. This is the only part that will be shown to the user.]\n\n" }, { "role": "user", "content": "A craft store makes a third of its sales in the fabric section, a quarter of its sales in the jewelry section, and the rest in the stationery section. They made 36 sales today. How many sales were in the stationery section?" } ], "ability": "chain-of-thought", "reward_model": { "style": "llmj-cot", "ground_truth": "Thus, there were 36 - 12 - 9 = <<36-12-9=15>>15 sales in the stationery section." }, "extra_info": { "split": "train", "index": 0, "question": "A craft store makes a third of its sales in the fabric section, a quarter of its sales in the jewelry section, and the rest in the stationery section. They made 36 sales today. How many sales were in the stationery section?", "short_answer": "15", "model_output": "\nTo find the number of sales in the stationery section, I need to first calculate the number of sales in the fabric and jewelry sections. The store makes a third of its sales in the fabric section, which is 1/3 of 36 sales. I can calculate this as 36 * 1/3 = 12 sales in the fabric section. The store makes a quarter of its sales in the jewelry section, which is 1/4 of 36 sales. I can calculate this as 36 * 1/4 = 9 sales in the jewelry section. Since the rest of the sales are in the stationery section, I can subtract the sales from the other two sections from the total number of sales to find the number of sales in the stationery section.\n\n\n\n1. Calculate the number of sales in the fabric section: 36 * 1/3 = 12\n2. Calculate the number of sales in the jewelry section: 36 * 1/4 = 9\n3. Calculate the number of sales in the stationery section: 36 - 12 - 9 = 15\n\n\n‹reflection>\nUpon reviewing my reasoning, I noticed that my calculations seem correct, but I should ensure that I understand the question correctly. The question asks for the number of sales in the stationery section, and I believe I have correctly calculated that. However, I should double-check my math to make sure it's accurate.\n\n\n\nUpon re-examining my math, I am confident that my calculations are correct. The fabric section makes up 1/3 of the sales, the jewelry section makes up 1/4, and the rest is the stationery section. My math checks out, and I am confident in my answer.\n\n\n\n15\n" } } ``` ## RLAIF - Fidelidade **Conjunto de dados de entrada** Fonte: [https://huggingface. co/datasets/rajpurkar/squad\$1v2/blob/main/squad\$1v2/train](https://huggingface.co/datasets/rajpurkar/squad_v2/blob/main/squad_v2/train-00000-of-00001.parquet)-00000-de-00001. parquet **Uma amostra** ``` { "data_source": "squad_v2", "prompt": [ { "role": "system", "content": "You are a helpful assistant that answers questions based on the provided context. Only use information from the context." }, { "role": "user", "content": "Context: Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debut album, Dangerously in Love (2003), which established her as a solo artist worldwide, earned five Grammy Awards and featured the Billboard Hot 100 number-one singles \"Crazy in Love\" and \"Baby Boy\".\n\nQuestion: When did Beyonce start becoming popular?" } ], "ability": "faithfulness", "reward_model": { "style": "llmj-faithfulness", "ground_truth": "Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debut album, Dangerously in Love (2003), which established her as a solo artist worldwide, earned five Grammy Awards and featured the Billboard Hot 100 number-one singles \"Crazy in Love\" and \"Baby Boy\"." }, "extra_info": { "question": "When did Beyonce start becoming popular?", "split": "train", "index": 0 } } ``` ## RLAIF - Resumo **Conjunto de dados de entrada** [Fonte: Conjunto de dados gsm8k limpo https://huggingface. co/datasets/thesven/gsm8k-reasoning/tree/main/data](https://huggingface.co/datasets/thesven/gsm8k-reasoning/tree/main/data) **Uma amostra** ``` { "data_source": "cnn_dailymail", "prompt": [ { "role": "system", "content": "You are a helpful assistant that creates concise, accurate summaries of news articles. Focus on the key facts and main points." }, { "role": "user", "content": "Summarize the following article:\n\nLONDON, England (Reuters) -- Harry Potter star Daniel Radcliffe gains access to a reported £20 million ($41.1 million) fortune as he turns 18 on Monday, but he insists the money won't cast a spell on him. Daniel Radcliffe as Harry Potter in \"Harry Potter and the Order of the Phoenix\" To the disappointment of gossip columnists around the world, the young actor says he has no plans to fritter his cash away on fast cars, drink and celebrity parties. \"I don't plan to be one of those people who, as soon as they turn 18, suddenly buy themselves a massive sports car collection or something similar,\" he told an Australian interviewer earlier this month. \"I don't think I'll be particularly extravagant. \"The things I like buying are things that cost about 10 pounds -- books and CDs and DVDs.\" At 18, Radcliffe will be able to gamble in a casino, buy a drink in a pub or see the horror film \"Hostel: Part II,\" currently six places below his number one movie on the UK box office chart. Details of how he'll mark his landmark birthday are under wraps. His agent and publicist had no comment on his plans. \"I'll definitely have some sort of party,\" he said in an interview. \"Hopefully none of you will be reading about it.\" Radcliffe's earnings from the first five Potter films have been held in a trust fund which he has not been able to touch. Despite his growing fame and riches, the actor says he is keeping his feet firmly on the ground. \"People are always looking to say 'kid star goes off the rails,'\" he told reporters last month. \"But I try very hard not to go that way because it would be too easy for them.\" His latest outing as the boy wizard in \"Harry Potter and the Order of the Phoenix\" is breaking records on both sides of the Atlantic and he will reprise the role in the last two films. Watch I-Reporter give her review of Potter's latest » . There is life beyond Potter, however. The Londoner has filmed a TV movie called \"My Boy Jack,\" about author Rudyard Kipling and his son, due for release later this year. He will also appear in \"December Boys,\" an Australian film about four boys who escape an orphanage. Earlier this year, he made his stage debut playing a tortured teenager in Peter Shaffer's \"Equus.\" Meanwhile, he is braced for even closer media scrutiny now that he's legally an adult: \"I just think I'm going to be more sort of fair game,\" he told Reuters. E-mail to a friend . Copyright 2007 Reuters. All rights reserved.This material may not be published, broadcast, rewritten, or redistributed." } ], "ability": "summarization", "reward_model": { "style": "llmj-summarization", "ground_truth": "Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday .\nYoung actor says he has no plans to fritter his cash away .\nRadcliffe's earnings from first five Potter films have been held in trust fund ." }, "extra_info": { "question": "LONDON, England (Reuters) -- Harry Potter star Daniel Radcliffe gains access to a reported £20 million ($41.1 million) fortune as he turns 18 on Monday, but he insists the money won't cast a spell on him. Daniel Radcliffe as Harry Potter in \"Harry Potter and the Order of the Phoenix\" To the disappointment of gossip columnists around the world, the young actor says he has no plans to fritter his cash away on fast cars, drink and celebrity parties. \"I don't plan to be one of those people who, as soon as they turn 18, suddenly buy themselves a massive sports car collection or something similar,\" he told an Australian interviewer earlier this month. \"I don't think I'll be particularly extravagant. \"The things I like buying are things that cost about 10 pounds -- books and CDs and DVDs.\" At 18, Radcliffe will be able to gamble in a casino, buy a drink in a pub or see the horror film \"Hostel: Part II,\" currently six places below his number one movie on the UK box office chart. Details of how he'll mark his landmark birthday are under wraps. His agent and publicist had no comment on his plans. \"I'll definitely have some sort of party,\" he said in an interview. \"Hopefully none of you will be reading about it.\" Radcliffe's earnings from the first five Potter films have been held in a trust fund which he has not been able to touch. Despite his growing fame and riches, the actor says he is keeping his feet firmly on the ground. \"People are always looking to say 'kid star goes off the rails,'\" he told reporters last month. \"But I try very hard not to go that way because it would be too easy for them.\" His latest outing as the boy wizard in \"Harry Potter and the Order of the Phoenix\" is breaking records on both sides of the Atlantic and he will reprise the role in the last two films. Watch I-Reporter give her review of Potter's latest » . There is life beyond Potter, however. The Londoner has filmed a TV movie called \"My Boy Jack,\" about author Rudyard Kipling and his son, due for release later this year. He will also appear in \"December Boys,\" an Australian film about four boys who escape an orphanage. Earlier this year, he made his stage debut playing a tortured teenager in Peter Shaffer's \"Equus.\" Meanwhile, he is braced for even closer media scrutiny now that he's legally an adult: \"I just think I'm going to be more sort of fair game,\" he told Reuters. E-mail to a friend . Copyright 2007 Reuters. All rights reserved.This material may not be published, broadcast, rewritten, or redistributed.", "split": "train", "index": 0, "source_id": "42c027e4ff9730fbb3de84c1af0d2c50" } } ``` ## RLAIF - Prompt personalizado Neste exemplo, usamos [RLAIF - Cadeia de pensamento](#model-customize-open-weight-samples-rlaif2) para discutir como um prompt jinja personalizado pode substituir um dos prompts predefinidos. **Abaixo está um exemplo de um prompt personalizado para CoT:** ``` You are an expert logical reasoning evaluator specializing in Chain-of-Thought (CoT) analysis. Given: A problem prompt and a model's reasoning-based response. Goal: Assess the quality and structure of logical reasoning, especially for specialized domains (law, medicine, finance, etc.). Scoring rubric (start at 0.0, then add or subtract): Core Components: Structural Completeness (0.3 max) - Clear problem statement: +0.05 - Defined variables/terminology: +0.05 - Organized given information: +0.05 - Explicit proof target: +0.05 - Step-by-step reasoning: +0.05 - Clear conclusion: +0.05 Logical Quality (0.4 max) - Valid logical flow: +0.1 - Proper use of if-then relationships: +0.1 - Correct application of domain principles: +0.1 - No logical fallacies: +0.1 Technical Accuracy (0.3 max) - Correct use of domain terminology: +0.1 - Accurate application of domain rules: +0.1 - Proper citation of relevant principles: +0.1 Critical Deductions: A. Invalid logical leap: -0.3 B. Missing critical steps: -0.2 C. Incorrect domain application: -0.2 D. Unclear/ambiguous reasoning: -0.1 Additional Instructions: - Verify domain-specific terminology and principles - Check for logical consistency throughout - Ensure conclusions follow from premises - Flag potential domain-specific compliance issues - Consider regulatory/professional standards where applicable Return EXACTLY this JSON (no extra text): { "score": , "component_scores": { "structural_completeness": , "logical_quality": , "technical_accuracy": }, "steps_present": { "problem_statement": , "variable_definitions": , "given_information": , "proof_target": , "step_reasoning": , "conclusion": }, "reasoning": "", "domain_flags": [""] } ### (Prompt field from dataset) Problem Prompt: {{ prompt }} Model's Response: {{ model_output }} ### Ground truth (if applicable): {{ ground_truth }} ``` ## Aprendizagem por reforço a partir de recompensas verificáveis (RLVR) - Exact Match **Conjunto de dados de entrada** Fonte: [https://huggingface. co/datasets/openai/gsm8k](https://huggingface.co/datasets/openai/gsm8k) **Amostra** ``` { "data_source": "openai/gsm8k", "prompt": [ { "content": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? Let\'s think step by step and output the final answer after \\"####\\".", "role": "user" } ], "ability": "math", "reward_model": { "ground_truth": "72", "style": "rule" }, "extra_info": { "answer": "Natalia sold 48\\/2 = <<48\\/2=24>>24 clips in May.\\nNatalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.\\n#### 72", "index": 0, "question": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?", "split": "train" } } ``` ## RLVR - Execução de código **Conjunto de dados de entrada** Fonte: [https://huggingface. co/datasets/open-r1/codeforces](https://huggingface.co/datasets/open-r1/codeforces) **Amostra** ``` { "data_source": "codeforces", "prompt": [ { "content": "\nWhen tackling complex reasoning tasks, you have access to the following actions. Use them as needed to progress through your thought process.\n\n[ASSESS]\n\n[ADVANCE]\n\n[VERIFY]\n\n[SIMPLIFY]\n\n[SYNTHESIZE]\n\n[PIVOT]\n\n[OUTPUT]\n\nYou should strictly follow the format below:\n\n[ACTION NAME]\n\n# Your action step 1\n\n# Your action step 2\n\n# Your action step 3\n\n...\n\nNext action: [NEXT ACTION NAME]\n\n", "role": "system" }, { "content": "Title: Zebras\n\nTime Limit: None seconds\n\nMemory Limit: None megabytes\n\nProblem Description:\nOleg writes down the history of the days he lived. For each day he decides if it was good or bad. Oleg calls a non-empty sequence of days a zebra, if it starts with a bad day, ends with a bad day, and good and bad days are alternating in it. Let us denote bad days as 0 and good days as 1. Then, for example, sequences of days 0, 010, 01010 are zebras, while sequences 1, 0110, 0101 are not.\n\nOleg tells you the story of days he lived in chronological order in form of string consisting of 0 and 1. Now you are interested if it is possible to divide Oleg's life history into several subsequences, each of which is a zebra, and the way it can be done. Each day must belong to exactly one of the subsequences. For each of the subsequences, days forming it must be ordered chronologically. Note that subsequence does not have to be a group of consecutive days.\n\nInput Specification:\nIn the only line of input data there is a non-empty string *s* consisting of characters 0 and 1, which describes the history of Oleg's life. Its length (denoted as |*s*|) does not exceed 200<=000 characters.\n\nOutput Specification:\nIf there is a way to divide history into zebra subsequences, in the first line of output you should print an integer *k* (1<=\u2264<=*k*<=\u2264<=|*s*|), the resulting number of subsequences. In the *i*-th of following *k* lines first print the integer *l**i* (1<=\u2264<=*l**i*<=\u2264<=|*s*|), which is the length of the *i*-th subsequence, and then *l**i* indices of days forming the subsequence. Indices must follow in ascending order. Days are numbered starting from 1. Each index from 1 to *n* must belong to exactly one subsequence. If there is no way to divide day history into zebra subsequences, print -1.\n\nSubsequences may be printed in any order. If there are several solutions, you may print any of them. You do not have to minimize nor maximize the value of *k*.\n\nDemo Input:\n['0010100\\n', '111\\n']\n\nDemo Output:\n['3\\n3 1 3 4\\n3 2 5 6\\n1 7\\n', '-1\\n']\n\nNote:\nnone\n\nWrite Python code to solve the problem. Present the code in \n```python\nYour code\n```\nat the end.", "role": "user" } ], "ability": "code", "reward_model": { "ground_truth": "{\"inputs\": [\"0010100\", \"111\", \"0\", \"1\", \"0101010101\", \"010100001\", \"000111000\", \"0101001000\", \"0000001000\", \"0101\", \"000101110\", \"010101010\", \"0101001010\", \"0100101100\", \"0110100000\", \"0000000000\", \"1111111111\", \"0010101100\", \"1010000\", \"0001110\", \"0000000000011001100011110101000101000010010111000100110110000011010011110110001100100001001001010010\", \"01010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010\", \"0010011100000000\"], \"outputs\": [\"3\\n1 1\\n5 2 3 4 5 6\\n1 7\", \"-1\", \"1\\n1 1\", \"-1\", \"-1\", \"-1\", \"3\\n3 1 6 7\\n3 2 5 8\\n3 3 4 9\", \"4\\n5 1 2 3 4 5\\n3 6 7 8\\n1 9\\n1 10\", \"8\\n1 1\\n1 2\\n1 3\\n1 4\\n1 5\\n3 6 7 8\\n1 9\\n1 10\", \"-1\", \"-1\", \"1\\n9 1 2 3 4 5 6 7 8 9\", \"2\\n5 1 2 3 4 5\\n5 6 7 8 9 10\", \"2\\n5 1 2 3 8 9\\n5 4 5 6 7 10\", \"-1\", \"10\\n1 1\\n1 2\\n1 3\\n1 4\\n1 5\\n1 6\\n1 7\\n1 8\\n1 9\\n1 10\", \"-1\", \"2\\n3 1 8 9\\n7 2 3 4 5 6 7 10\", \"-1\", \"-1\", \"22\\n1 1\\n1 2\\n1 3\\n1 4\\n1 5\\n1 6\\n1 7\\n1 8\\n7 9 24 25 26 27 28 29\\n7 10 13 14 17 18 23 30\\n11 11 12 15 16 19 22 31 32 33 34 35\\n3 20 21 36\\n3 37 46 47\\n9 38 39 40 45 48 57 58 75 76\\n17 41 42 43 44 49 50 51 54 55 56 59 72 73 74 77 80 81\\n9 52 53 60 71 78 79 82 83 84\\n7 61 64 65 66 67 70 85\\n5 62 63 68 69 86\\n3 87 88 89\\n3 90 91 92\\n5 93 94 95 96 97\\n3 98 99 100\", \"1\\n245 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 ...\", \"8\\n3 1 8 9\\n5 2 3 4 7 10\\n3 5 6 11\\n1 12\\n1 13\\n1 14\\n1 15\\n1 16\"]}", "style": "rule" }, "extra_info": { "index": 49, "split": "train" } } ``` **Função de recompensa** Função de recompensa: [https://github.com/volcengine/verl/tree/main/verl/utils/reward\$1score/prime\$1code](https://github.com/volcengine/verl/tree/main/verl/utils/reward_score/prime_code) ## RLVR - Resposta matemática **Conjunto de dados de entrada** [Fonte: Conjunto de dados gsm8k limpo https://huggingface. co/datasets/thesven/gsm8k-reasoning/tree/main/data](https://huggingface.co/datasets/thesven/gsm8k-reasoning/tree/main/data) **Amostra** ``` [ { "content": "\nYou are an AI assistant that uses a Chain of Thought (CoT) approach with reflection to answer queries...", "role": "system" }, { "content": "A craft store makes a third of its sales in the fabric section, a quarter of its sales in the jewelry section, and the rest in the stationery section. They made 36 sales today. How many sales were in the stationery section?", "role": "user" }, { "content": "\nTo find the number of sales in the stationery section, I need to first calculate the number of sales in the fabric and jewelry sections...\n\n\n\n1. Calculate the number of sales in the fabric section: 36 * 1/3 = 12\n2. Calculate the number of sales in the jewelry section: 36 * 1/4 = 9\n3. Calculate the number of sales in the stationery section: 36 - 12 - 9 = 15\n\n\n\nUpon reviewing my reasoning, I noticed that my calculations seem correct...\n\n\n\nUpon re-examining my math, I am confident that my calculations are correct...\n\n\n\n15\n", "role": "assistant" } ] ``` **Cálculo da recompensa** Função de recompensa: [https://github.com/volcengine/verl/blob/main/verl/utils/reward\$1score/gsm8k.py](https://github.com/volcengine/verl/blob/main/verl/utils/reward_score/gsm8k.py) ## RLVR - Lambda personalizada **Conjunto de dados de entrada** [Fonte: Conjunto de dados gsm8k limpo https://huggingface. co/datasets/thesven/gsm8k-reasoning/tree/main/data](https://huggingface.co/datasets/thesven/gsm8k-reasoning/tree/main/data) **Amostra** ``` [ { "content": "\nYou are an AI assistant that uses a Chain of Thought (CoT) approach with reflection to answer queries.\n\nFollow these steps:\n- Think through the problem step by step within the ‹thinking> tags.\n- Reflect on your thinking\n\nto check for any errors or improvements within the ‹reflection› tags.\n- Make any necessary adjustments based on your reflection.\n- Provide your final, concise answer within the ‹output> tags.\n\nImportant: The and ‹reflection› sections are for your internal reasoning process only.\nDo not include any part of the final answer in these sections.\nThe actual response to the query must be entirely contained within the ‹output› tags.\n\nUse the following format for your response:\n\n[Your initial thought process goes here]\n\n[Your step-by-step reasoning goes here. This is your internal thought process, not the final answer. You can create as many reasoning steps as necessary in your process.]\n\n\n‹reflection>\n[Your reflection on your reasoning, checking for errors or improvements. You can create as many reflection steps as necessary in your process.]\n\n\n\n[Any adjustments to your thinking based on your reflection]\n\n\n\n[Your final, concise answer to the query. This is the only part that will be shown to the user.]\n\n", "role": "system" }, { "content": "A craft store makes a third of its sales in the fabric section, a quarter of its sales in the jewelry section, and the rest in the stationery section. They made 36 sales today. How many sales were in the stationery section?", "role": "user" }, { "content": "\nTo find the number of sales in the stationery section, I need to first calculate the number of sales in the fabric and jewelry sections. The store makes a third of its sales in the fabric section, which is 1/3 of 36 sales. I can calculate this as 36 * 1/3 = 12 sales in the fabric section. The store makes a quarter of its sales in the jewelry section, which is 1/4 of 36 sales. I can calculate this as 36 * 1/4 = 9 sales in the jewelry section. Since the rest of the sales are in the stationery section, I can subtract the sales from the other two sections from the total number of sales to find the number of sales in the stationery section.\n\n\n\n1. Calculate the number of sales in the fabric section: 36 * 1/3 = 12\n2. Calculate the number of sales in the jewelry section: 36 * 1/4 = 9\n3. Calculate the number of sales in the stationery section: 36 - 12 - 9 = 15\n\n\n‹reflection>\nUpon reviewing my reasoning, I noticed that my calculations seem correct, but I should ensure that I understand the question correctly. The question asks for the number of sales in the stationery section, and I believe I have correctly calculated that. However, I should double-check my math to make sure it's accurate.\n\n\n\nUpon re-examining my math, I am confident that my calculations are correct. The fabric section makes up 1/3 of the sales, the jewelry section makes up 1/4, and the rest is the stationery section. My math checks out, and I am confident in my answer.\n\n\n\n15\n", "role": "assistant" } ] ``` **Exemplo de cálculo de recompensa** ``` # RLVR Evaluator for OSS # lambda_grader.py import json import re import uuid from typing import Any, Dict, List def custom_reward(assistant_answer: str, ground_truth: str) -> float: """ Add custom reward computation here Example:- Reward = fraction of ground-truth words that are correct in the correct position. Example: gt: "the cat sat" ans: "the dog sat" word-by-word: "the" == "the" -> correct "dog" != "cat" -> wrong "sat" == "sat" -> correct correct = 2 out of 3 -> reward = 2/3 ≈ 0.67 """ ans_words = assistant_answer.strip().lower().split() gt_words = ground_truth.strip().lower().split() if not gt_words: return 0.0 correct = 0 for aw, gw in zip(ans_words, gt_words): if aw == gw: correct += 1 return correct / len(gt_words) # Lambda utility functions def _ok(body: Any, code: int = 200) -> Dict[str, Any]: return { "statusCode": code, "headers": { "content-type": "application/json", "access-control-allow-origin": "*", "access-control-allow-methods": "POST,OPTIONS", "access-control-allow-headers": "content-type", }, "body": json.dumps(body), } def _assistant_text(sample: Dict[str, Any]) -> str: """Extract assistant text from sample messages.""" for m in reversed(sample.get("messages", [])): if m.get("role") == "assistant": return (m.get("content") or "").strip() return "" def _sample_id(sample: Dict[str, Any]) -> str: """Generate or extract sample ID.""" if isinstance(sample.get("id"), str) and sample["id"]: return sample["id"] return str(uuid.uuid4()) def _ground_truth(sample: Dict[str, Any]) -> str: """Extract ground truth from sample or metadata if available""" if isinstance(sample.get("reference_answer"), str) and sample["reference_answer"]: return sample["reference_answer"].strip() md = sample.get("metadata") or {} gt = md.get("reference_answer", None) or md.get("ground_truth", None) if gt is None: return "" return str(gt).strip() def _score_and_metrics(sample: Dict[str, Any]) -> Dict[str, Any]: sid = _sample_id(sample) solution_text = _assistant_text(sample) # Extract ground truth gt = _ground_truth(sample) metrics_list: List[Dict[str, Any]] = [] # Custom rlvr scoring if solution_text and gt: # Compute score reward_score = custom_reward( assistant_answer=solution_text, ground_truth=gt ) # Add detailed metrics metrics_list.append({ "name": "custom_reward_score", "value": float(reward_score), "type": "Reward" }) # The aggregate reward score is the custom reward score aggregate_score = float(reward_score) else: # No solution text or ground truth - default to 0 aggregate_score = 0.0 metrics_list.append({ "name": "default_zero", "value": 0.0, "type": "Reward" }) print("detected score", { "id": sid, "aggregate_reward_score": float(aggregate_score), "metrics_list": metrics_list, }) return { "id": sid, "aggregate_reward_score": float(aggregate_score), "metrics_list": metrics_list, } def lambda_handler(event, context): """AWS Lambda handler for custom reward lambda grading.""" # CORS preflight if event.get("requestContext", {}).get("http", {}).get("method") == "OPTIONS": return _ok({"ok": True}) # Body may be a JSON string (API GW/Function URL) or already a dict (Invoke) raw = event.get("body") or "{}" try: body = json.loads(raw) if isinstance(raw, str) else raw except Exception as e: return _ok({"error": f"invalid JSON body: {e}"}, 400) # Accept top-level list, {"batch":[...]}, or single sample object if isinstance(body, dict) and isinstance(body.get("batch"), list): samples = body["batch"] else: return _ok({ "error": "Send a sample object, or {'batch':[...]} , or a top-level list of samples." }, 400) try: results = [_score_and_metrics(s) for s in samples] except Exception as e: return _ok({"error": f"Custom scoring failed: {e}"}, 500) return _ok(results) ``` **Exemplo de código da função de recompensa** ``` # RLVR Evaluator for OSS # lambda_grader.py import json import re import uuid from typing import Any, Dict, List def custom_reward(assistant_answer: str, ground_truth: str) -> float: """ Add custom reward computation here Example:- Reward = fraction of ground-truth words that are correct in the correct position. Example: gt: "the cat sat" ans: "the dog sat" word-by-word: "the" == "the" -> correct "dog" != "cat" -> wrong "sat" == "sat" -> correct correct = 2 out of 3 -> reward = 2/3 ≈ 0.67 """ ans_words = assistant_answer.strip().lower().split() gt_words = ground_truth.strip().lower().split() if not gt_words: return 0.0 correct = 0 for aw, gw in zip(ans_words, gt_words): if aw == gw: correct += 1 return correct / len(gt_words) # Lambda utility functions def _ok(body: Any, code: int = 200) -> Dict[str, Any]: return { "statusCode": code, "headers": { "content-type": "application/json", "access-control-allow-origin": "*", "access-control-allow-methods": "POST,OPTIONS", "access-control-allow-headers": "content-type", }, "body": json.dumps(body), } def _assistant_text(sample: Dict[str, Any]) -> str: """Extract assistant text from sample messages.""" for m in reversed(sample.get("messages", [])): if m.get("role") == "assistant": return (m.get("content") or "").strip() return "" def _sample_id(sample: Dict[str, Any]) -> str: """Generate or extract sample ID.""" if isinstance(sample.get("id"), str) and sample["id"]: return sample["id"] return str(uuid.uuid4()) def _ground_truth(sample: Dict[str, Any]) -> str: """Extract ground truth from sample or metadata if available""" if isinstance(sample.get("reference_answer"), str) and sample["reference_answer"]: return sample["reference_answer"].strip() md = sample.get("metadata") or {} gt = md.get("reference_answer", None) or md.get("ground_truth", None) if gt is None: return "" return str(gt).strip() def _score_and_metrics(sample: Dict[str, Any]) -> Dict[str, Any]: sid = _sample_id(sample) solution_text = _assistant_text(sample) # Extract ground truth gt = _ground_truth(sample) metrics_list: List[Dict[str, Any]] = [] # Custom rlvr scoring if solution_text and gt: # Compute score reward_score = custom_reward( assistant_answer=solution_text, ground_truth=gt ) # Add detailed metrics metrics_list.append({ "name": "custom_reward_score", "value": float(reward_score), "type": "Reward" }) # The aggregate reward score is the custom reward score aggregate_score = float(reward_score) else: # No solution text or ground truth - default to 0 aggregate_score = 0.0 metrics_list.append({ "name": "default_zero", "value": 0.0, "type": "Reward" }) print("detected score", { "id": sid, "aggregate_reward_score": float(aggregate_score), "metrics_list": metrics_list, }) return { "id": sid, "aggregate_reward_score": float(aggregate_score), "metrics_list": metrics_list, } def lambda_handler(event, context): """AWS Lambda handler for custom reward lambda grading.""" # CORS preflight if event.get("requestContext", {}).get("http", {}).get("method") == "OPTIONS": return _ok({"ok": True}) # Body may be a JSON string (API GW/Function URL) or already a dict (Invoke) raw = event.get("body") or "{}" try: body = json.loads(raw) if isinstance(raw, str) else raw except Exception as e: return _ok({"error": f"invalid JSON body: {e}"}, 400) # Accept top-level list, {"batch":[...]}, or single sample object if isinstance(body, dict) and isinstance(body.get("batch"), list): samples = body["batch"] else: return _ok({ "error": "Send a sample object, or {'batch':[...]} , or a top-level list of samples." }, 400) try: results = [_score_and_metrics(s) for s in samples] except Exception as e: return _ok({"error": f"Custom scoring failed: {e}"}, 500) return _ok(results) ``` **Exemplo de solicitação de recompensa** ``` You are an expert RAG response evaluator specializing in faithfulness and relevance assessment. Given: Context documents, a question, and response statements. Goal: Evaluate both statement-level faithfulness and overall response relevance to the question. Scoring rubric (start at 0.0, then add or subtract): Core Components: Faithfulness Assessment (0.6 max) Per statement evaluation: - Direct support in context: +0.2 - Accurate inference from context: +0.2 - No contradictions with context: +0.2 Deductions: - Hallucination: -0.3 - Misrepresentation of context: -0.2 - Unsupported inference: -0.1 Question Relevance (0.4 max) - Direct answer to question: +0.2 - Appropriate scope/detail: +0.1 - Proper context usage: +0.1 Deductions: - Off-topic content: -0.2 - Implicit/meta responses: -0.2 - Missing key information: -0.1 Critical Flags: A. Complete hallucination B. Context misalignment C. Question misinterpretation D. Implicit-only responses Additional Instructions: - Evaluate each statement independently - Check for direct textual support - Verify logical inferences - Assess answer completeness - Flag any unsupported claims Return EXACTLY this JSON (no extra text): { "statements_evaluation": [ { "statement": "", "verdict": <0 or 1>, "reason": "", "context_support": "" } ], "overall_assessment": { "question_addressed": <0 or 1>, "reasoning": "", "faithfulness_score": <0.0-1.0>, "relevance_score": <0.0-1.0> }, "flags": [""] } ## Current Evaluation Task ### Context {{ ground_truth }} ### Question {{ extra_info.question }} ### Model's Response {{ model_output }} ```