Configurando seus HyperPod clusters para implantação de modelos

Configure o kubectl para se conectar ao cluster recém-criado orquestrado pelo HyperPod cluster Amazon EKS. Especifique a região e o nome do HyperPod cluster.

export HYPERPOD_CLUSTER_NAME=<hyperpod-cluster-name>
export REGION=<region>

# S3 bucket where tls certificates will be uploaded
BUCKET_NAME="<Enter name of your s3 bucket>" # This should be bucket name, not URI

export EKS_CLUSTER_NAME=$(aws --region $REGION sagemaker describe-cluster --cluster-name $HYPERPOD_CLUSTER_NAME \
  --query 'Orchestrator.Eks.ClusterArn' --output text | \
  cut -d'/' -f2)
aws eks update-kubeconfig --name $EKS_CLUSTER_NAME --region $REGION

Defina as variáveis de ambiente padrão.

LB_CONTROLLER_POLICY_NAME="AWSLoadBalancerControllerIAMPolicy-$HYPERPOD_CLUSTER_NAME"
LB_CONTROLLER_ROLE_NAME="aws-load-balancer-controller-$HYPERPOD_CLUSTER_NAME"
S3_MOUNT_ACCESS_POLICY_NAME="S3MountpointAccessPolicy-$HYPERPOD_CLUSTER_NAME"
S3_CSI_ROLE_NAME="SM_HP_S3_CSI_ROLE-$HYPERPOD_CLUSTER_NAME"
KEDA_OPERATOR_POLICY_NAME="KedaOperatorPolicy-$HYPERPOD_CLUSTER_NAME"
KEDA_OPERATOR_ROLE_NAME="keda-operator-role-$HYPERPOD_CLUSTER_NAME"
PRESIGNED_URL_ACCESS_POLICY_NAME="PresignedUrlAccessPolicy-$HYPERPOD_CLUSTER_NAME"
HYPERPOD_INFERENCE_ACCESS_POLICY_NAME="HyperpodInferenceAccessPolicy-$HYPERPOD_CLUSTER_NAME"
HYPERPOD_INFERENCE_ROLE_NAME="HyperpodInferenceRole-$HYPERPOD_CLUSTER_NAME"
HYPERPOD_INFERENCE_SA_NAME="hyperpod-inference-operator-controller"
HYPERPOD_INFERENCE_SA_NAMESPACE="hyperpod-inference-system"
JUMPSTART_GATED_ROLE_NAME="JumpstartGatedRole-$HYPERPOD_CLUSTER_NAME"
FSX_CSI_ROLE_NAME="AmazonEKSFSxLustreCSIDriverFullAccess-$HYPERPOD_CLUSTER_NAME"

Extraia o nome do cluster do Amazon EKS do ARN do cluster, atualize o kubeconfig local e verifique a conectividade listando todos os pods nos namespaces.

kubectl get pods --all-namespaces

(Opcional) Instale o plug-in do dispositivo NVIDIA para habilitar o suporte à GPU no cluster.

#Install nvidia device plugin
kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.14.5/nvidia-device-plugin.yml
# Verify that GPUs are visible to k8s
kubectl get nodes -o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia.com/gpu

Reúna os identificadores de AWS recursos essenciais e ARNs necessários para configurar as integrações de serviços entre os componentes do Amazon EKS, SageMaker AI e IAM.

%%bash -x

export ACCOUNT_ID=$(aws --region $REGION sts get-caller-identity --query 'Account' --output text)
export OIDC_ID=$(aws --region $REGION eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)
export EKS_CLUSTER_ROLE=$(aws eks --region $REGION describe-cluster --name $EKS_CLUSTER_NAME --query 'cluster.roleArn' --output text)

Associe um OIDCidentity provedor do IAM ao seu cluster EKS.

eksctl utils associate-iam-oidc-provider --region=$REGION --cluster=$EKS_CLUSTER_NAME --approve

Crie os documentos JSON da política de confiança e da política de permissão necessários para a função IAM do operador de HyperPod inferência. Essas políticas permitem a comunicação segura entre serviços entre o Amazon EKS, a SageMaker IA e outros AWS serviços.

bash
# Create trust policy JSON
cat << EOF > trust-policy.json
{
    "Version": "2012-10-17",		 	 	 
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "sagemaker.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/oidc.eks.${REGION}.amazonaws.com/id/${OIDC_ID}"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringLike": {
                    "oidc.eks.${REGION}.amazonaws.com/id/${OIDC_ID}:aud": "sts.amazonaws.com",
                    "oidc.eks.${REGION}.amazonaws.com/id/${OIDC_ID}:sub": "system:serviceaccount:*:*"
                }
            }
        }
    ]
}
EOF

# Create permission policy JSON
cat << EOF > permission-policy.json
{
    "Version": "2012-10-17",		 	 	 
    "Statement": [
        {
        "Sid": "S3Access",
        "Effect": "Allow",
        "Action": [
            "s3:Get*",
            "s3:List*",
            "s3:Describe*",
            "s3:PutObject"
        ],
        "Resource": [
            "*"
        ]
        },
        {
        "Sid": "ECRAccess",
        "Effect": "Allow",
        "Action": [
            "ecr:GetAuthorizationToken",
            "ecr:BatchCheckLayerAvailability",
            "ecr:GetDownloadUrlForLayer",
            "ecr:GetRepositoryPolicy",
            "ecr:DescribeRepositories",
            "ecr:ListImages",
            "ecr:DescribeImages",
            "ecr:BatchGetImage",
            "ecr:GetLifecyclePolicy",
            "ecr:GetLifecyclePolicyPreview",
            "ecr:ListTagsForResource",
            "ecr:DescribeImageScanFindings"
        ],
        "Resource": [
            "*"
        ]
        },
        {
        "Sid": "EC2Access",
        "Effect": "Allow",
        "Action": [
            "ec2:AssignPrivateIpAddresses",
            "ec2:AttachNetworkInterface",
            "ec2:CreateNetworkInterface",
            "ec2:DeleteNetworkInterface",
            "ec2:DescribeInstances",
            "ec2:DescribeTags",
            "ec2:DescribeNetworkInterfaces",
            "ec2:DescribeInstanceTypes",
            "ec2:DescribeSubnets",
            "ec2:DetachNetworkInterface",
            "ec2:ModifyNetworkInterfaceAttribute",
            "ec2:UnassignPrivateIpAddresses",
            "ec2:CreateTags",
            "ec2:DescribeInstances",
            "ec2:DescribeInstanceTypes",
            "ec2:DescribeRouteTables",
            "ec2:DescribeSecurityGroups",
            "ec2:DescribeSubnets",
            "ec2:DescribeVolumes",
            "ec2:DescribeVolumesModifications",
            "ec2:DescribeVpcs",
            "ec2:CreateVpcEndpointServiceConfiguration",
            "ec2:DeleteVpcEndpointServiceConfigurations",
            "ec2:DescribeVpcEndpointServiceConfigurations",
            "ec2:ModifyVpcEndpointServicePermissions"
        ],
        "Resource": [
            "*"
        ]
        },
        {
        "Sid": "EKSAuthAccess",
        "Effect": "Allow",
        "Action": [
            "eks-auth:AssumeRoleForPodIdentity"
        ],
        "Resource": [
            "*"
        ]
        },
        {
        "Sid": "EKSAccess",
        "Effect": "Allow",
        "Action": [
            "eks:AssociateAccessPolicy",
            "eks:Describe*",
            "eks:List*",
            "eks:AccessKubernetesApi"
        ],
        "Resource": [
            "*"
        ]
        },
        {
        "Sid": "ApiGatewayAccess",
        "Effect": "Allow",
        "Action": [
            "apigateway:POST",
            "apigateway:GET",
            "apigateway:PUT",
            "apigateway:PATCH",
            "apigateway:DELETE",
            "apigateway:UpdateRestApiPolicy"
        ],
        "Resource": [
            "arn:aws:apigateway:*::/vpclinks",
            "arn:aws:apigateway:*::/vpclinks/*",
            "arn:aws:apigateway:*::/restapis",
            "arn:aws:apigateway:*::/restapis/*"
        ]
        },
        {
        "Sid": "ElasticLoadBalancingAccess",
        "Effect": "Allow",
        "Action": [
            "elasticloadbalancing:CreateLoadBalancer",
            "elasticloadbalancing:DescribeLoadBalancers",
            "elasticloadbalancing:DescribeLoadBalancerAttributes",
            "elasticloadbalancing:DescribeListeners",
            "elasticloadbalancing:DescribeListenerCertificates",
            "elasticloadbalancing:DescribeSSLPolicies",
            "elasticloadbalancing:DescribeRules",
            "elasticloadbalancing:DescribeTargetGroups",
            "elasticloadbalancing:DescribeTargetGroupAttributes",
            "elasticloadbalancing:DescribeTargetHealth",
            "elasticloadbalancing:DescribeTags",
            "elasticloadbalancing:DescribeTrustStores",
            "elasticloadbalancing:DescribeListenerAttributes"
        ],
        "Resource": [
            "*"
        ]
        },
        {
        "Sid": "SageMakerAccess",
        "Effect": "Allow",
        "Action": [
            "sagemaker:*"
        ],
        "Resource": [
            "*"
        ]
        },
        {
        "Sid": "AllowPassRoleToSageMaker",
        "Effect": "Allow",
        "Action": [
            "iam:PassRole"
        ],
        "Resource": "arn:aws:iam::*:role/*",
        "Condition": {
            "StringEquals": {
            "iam:PassedToService": "sagemaker.amazonaws.com"
            }
        }
        },
        {
        "Sid": "AcmAccess",
        "Effect": "Allow",
        "Action": [
            "acm:ImportCertificate",
            "acm:DeleteCertificate"
        ],
        "Resource": [
            "*"
        ]
        }
    ]
}
EOF

Crie um perfil de execução para o operador de inferência.

aws iam create-policy --policy-name $HYPERPOD_INFERENCE_ACCESS_POLICY_NAME --policy-document file://permission-policy.json
export policy_arn="arn:aws:iam::${ACCOUNT_ID}:policy/$HYPERPOD_INFERENCE_ACCESS_POLICY_NAME"

aws iam create-role --role-name $HYPERPOD_INFERENCE_ROLE_NAME --assume-role-policy-document file://trust-policy.json
 
aws iam put-role-policy --role-name $HYPERPOD_INFERENCE_ROLE_NAME --policy-name InferenceOperatorInlinePolicy --policy-document file://permission-policy.json

Faça o download e crie a política do IAM necessária para o AWS Load Balancer Controller gerenciar os balanceadores de carga de aplicativos e os balanceadores de carga de rede em seu cluster EKS.

%%bash -x 

export ALBController_IAM_POLICY_NAME=HyperPodInferenceALBControllerIAMPolicy

curl -o AWSLoadBalancerControllerIAMPolicy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.13.0/docs/install/iam_policy.json
aws iam create-policy --policy-name $ALBController_IAM_POLICY_NAME --policy-document file://AWSLoadBalancerControllerIAMPolicy.json

Crie uma conta de serviço do IAM que vincule a conta de serviço do Kubernetes à política do IAM, permitindo que o AWS Load Balancer Controller assuma AWS as permissões necessárias por meio do IRSA (IAM Roles for Service Accounts).

%%bash -x 

export ALB_POLICY_ARN="arn:aws:iam::$ACCOUNT_ID:policy/$ALBController_IAM_POLICY_NAME"

# Create IAM service account with gathered values
eksctl create iamserviceaccount \
  --approve \
  --override-existing-serviceaccounts \
  --name=aws-load-balancer-controller \
  --namespace=kube-system \
  --cluster=$EKS_CLUSTER_NAME \
  --attach-policy-arn=$ALB_POLICY_ARN \
  --region=$REGION

# Print the values for verification
echo "Cluster Name: $EKS_CLUSTER_NAME"
echo "Region: $REGION"
echo "Policy ARN: $ALB_POLICY_ARN"

Aplique Tags (kubernetes.io.role/elb) a todas as sub-redes no cluster Amazon EKS (públicas e privadas).

export VPC_ID=$(aws --region $REGION eks describe-cluster --name $EKS_CLUSTER_NAME --query 'cluster.resourcesVpcConfig.vpcId' --output text)

# Add Tags
aws ec2 describe-subnets \
  --filters "Name=vpc-id,Values=${VPC_ID}" "Name=map-public-ip-on-launch,Values=true" \
  --query 'Subnets[*].SubnetId' --output text | \
tr '\t' '\n' | \
xargs -I{} aws ec2 create-tags --resources {} --tags Key=kubernetes.io/role/elb,Value=1

# Verify Tags are added
aws ec2 describe-subnets \
  --filters "Name=vpc-id,Values=${VPC_ID}" "Name=map-public-ip-on-launch,Values=true" \
  --query 'Subnets[*].SubnetId' --output text | \
  tr '\t' '\n' |
xargs -n1 -I{} aws ec2 describe-tags --filters "Name=resource-id,Values={}" "Name=key,Values=kubernetes.io/role/elb" --query "Tags[0].Value" --output text

Crie um namespace para o KEDA e o Cert Manager.

kubectl create namespace keda
kubectl create namespace cert-manager

Crie um endpoint da VPC do Amazon S3.

aws ec2 create-vpc-endpoint \
  --vpc-id ${VPC_ID} \
  --vpc-endpoint-type Gateway \
  --service-name "com.amazonaws.${REGION}.s3" \
  --route-table-ids $(aws ec2 describe-route-tables --filters "Name=vpc-id,Values=${VPC_ID}" --query 'RouteTables[].Associations[].RouteTableId' --output text | tr ' ' '\n' | sort -u | tr '\n' ' ')

Configure o acesso ao armazenamento do S3:

(Opcional) Configure o acesso ao FSx armazenamento. Crie uma conta de serviço do IAM para o driver Amazon FSx CSI. Essa conta de serviço será usada pelo driver FSx CSI para interagir com o FSx serviço da Amazon em nome do seu cluster.

%%bash -x 


eksctl create iamserviceaccount \
  --name fsx-csi-controller-sa \
  --namespace kube-system \
  --cluster $EKS_CLUSTER_NAME \
  --attach-policy-arn arn:aws:iam::aws:policy/AmazonFSxFullAccess \
  --approve \
  --role-name FSXLCSI-${EKS_CLUSTER_NAME}-${REGION} \
  --region $REGION

Crie a política de confiança e a política de permissões.

# Create trust policy
cat <<EOF > /tmp/keda-trust-policy.json
{
    "Version": "2012-10-17",		 	 	 
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::$ACCOUNT_ID:oidc-provider/oidc.eks.$REGION.amazonaws.com/id/$OIDC_ID"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringLike": {
                    "oidc.eks.$REGION.amazonaws.com/id/$OIDC_ID:sub": "system:serviceaccount:kube-system:keda-operator",
                    "oidc.eks.$REGION.amazonaws.com/id/$OIDC_ID:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}
EOF
 # Create permissions policy
cat <<EOF > /tmp/keda-policy.json
{
    "Version": "2012-10-17",		 	 	 
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:GetMetricData",
                "cloudwatch:GetMetricStatistics",
                "cloudwatch:ListMetrics"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "aps:QueryMetrics",
                "aps:GetLabels",
                "aps:GetSeries",
                "aps:GetMetricMetadata"
            ],
            "Resource": "*"
        }
    ]
}
EOF
 # Create the role
aws iam create-role \
    --role-name keda-operator-role \
    --assume-role-policy-document file:///tmp/keda-trust-policy.json
 # Create the policy
KEDA_POLICY_ARN=$(aws iam create-policy \
    --policy-name KedaOperatorPolicy \
    --policy-document file:///tmp/keda-policy.json \
    --query 'Policy.Arn' \
    --output text)
 # Attach the policy to the role
aws iam attach-role-policy \
    --role-name keda-operator-role \
    --policy-arn $KEDA_POLICY_ARN

Se você estiver usando modelos fechados, crie um perfil do IAM para acessar acessá-los.

Instale o operador de HyperPod inferência. Essa etapa reúne os identificadores de recursos necessários da AWS e gera o comando de instalação do Helm com os parâmetros de configuração apropriados.

Acesse o gráfico do leme em https://github.com/aws/sagemaker-hyperpod-cli/tree/main/helm_chart.

git clone https://github.com/aws/sagemaker-hyperpod-cli
cd sagemaker-hyperpod-cli
cd helm_chart/HyperPodHelmChart
helm dependencies update charts/inference-operator

%%bash -x

HYPERPOD_INFERENCE_ROLE_ARN=$(aws iam get-role --role-name=$HYPERPOD_INFERENCE_ROLE_NAME --query "Role.Arn" --output text)
echo $HYPERPOD_INFERENCE_ROLE_ARN
 
S3_CSI_ROLE_ARN=$(aws iam get-role --role-name=$S3_CSI_ROLE_NAME --query "Role.Arn" --output text)
echo $S3_CSI_ROLE_ARN

HYPERPOD_CLUSTER_ARN=$(aws sagemaker describe-cluster --cluster-name $HYPERPOD_CLUSTER_NAME --query "ClusterArn")

# Verify values
echo "Cluster Name: $EKS_CLUSTER_NAME"
echo "Execution Role: $HYPERPOD_INFERENCE_ROLE_ARN"
echo "Hyperpod ARN: $HYPERPOD_CLUSTER_ARN"
# Run the the HyperPod inference operator installation. 

helm install hyperpod-inference-operator charts/inference-operator \
-n kube-system \
--set region=$REGION \
 --set eksClusterName=$EKS_CLUSTER_NAME \
--set hyperpodClusterArn=$HYPERPOD_CLUSTER_ARN \
--set executionRoleArn=$HYPERPOD_INFERENCE_ROLE_ARN \
--set s3.serviceAccountRoleArn=$S3_CSI_ROLE_ARN \
--set s3.node.serviceAccount.create=false \
--set keda.podIdentity.aws.irsa.roleArn="arn:aws:iam::$ACCOUNT_ID:role/keda-operator-role" \
--set tlsCertificateS3Bucket="s3://$BUCKET_NAME" \
--set alb.region=$REGION \
--set alb.clusterName=$EKS_CLUSTER_NAME \
--set alb.vpcId=$VPC_ID
 
# For JumpStart Gated Model usage, Add
# --set jumpstartGatedModelDownloadRoleArn=$UMPSTART_GATED_ROLE_ARN

Configure as anotações da conta de serviço para integração com o IAM. Essa anotação permite que a conta de serviço do operador assuma as permissões necessárias do IAM para gerenciar endpoints de inferência e interagir com os serviços da AWS.

%%bash -x 

EKS_CLUSTER_ROLE_NAME=$(echo $EKS_CLUSTER_ROLE | sed 's/.*\///')

# Annotate service account
kubectl annotate serviceaccount hyperpod-inference-operator-controller-manager \
  -n hyperpod-inference-system \
  eks.amazonaws.com/role-arn=arn:aws:iam::${ACCOUNT_ID}:role/${EKS_CLUSTER_ROLE_NAME} \
  --overwrite

Crie um arquivo de configuração de implantação do modelo. Isso cria um arquivo de manifesto do Kubernetes que define uma implantação de JumpStart modelo para o HyperPod operador de inferência. A configuração especifica como implantar um modelo pré-treinado da Amazon SageMaker JumpStart como um endpoint de inferência em seu cluster Amazon EKS.

cat <<EOF>> simple_model_install.yaml
---
apiVersion: inference.sagemaker.aws.amazon.com/v1
kind: JumpStartModel
metadata:
  name: testing-deployment-bert
  namespace: default
spec:
  model:
    modelId: "huggingface-eqa-bert-base-cased"
  sageMakerEndpoint:
    name: "hp-inf-ep-for-testing"
  server:
    instanceType: "ml.c5.2xlarge"
  environmentVariables:
    - name: SAMPLE_ENV_VAR
      value: "sample_value"
  maxDeployTimeInSeconds: 1800
EOF

Implante o modelo e limpe o arquivo de configuração. Essa etapa cria o recurso do JumpStart modelo e remove o arquivo de configuração temporário para manter um espaço de trabalho limpo.

%%bash -x 

kubectl create -f simple_model_install.yaml
rm -rfv simple_model_install.yaml

Verifique se o modelo está instalado e funcionando. Essa verificação garante que o operador possa assumir com êxito AWS as permissões para gerenciar endpoints de inferência.

%%bash

# Get the service account details
kubectl get serviceaccount -n hyperpod-inference-system

# Check if the service account has theAWSannotations
kubectl describe serviceaccount hyperpod-inference-operator-controller-manager -n hyperpod-inference-system

Em Configurações de implantação, JumpStart recomendará uma instância para implantação. Você pode modificar essas configurações, se necessário.