kubectl を使用してローカル NVMe ストレージからモデルをデプロイする

InferenceEndpointConfig YAML ファイルを作成します。プレースホルダー値を実際のリソース識別子に置き換えます。

cat <<EOF> deploy_nvme_k8s_volume.yaml
apiVersion: inference.sagemaker.aws.amazon.com/v1
kind: InferenceEndpointConfig
metadata:
  name: nvme-k8s-volume
  namespace: default
spec:
  endpointName: nvme-k8s-volume
  modelName: Qwen2.5-VL-7B-Instruct
  invocationEndpoint: v1/chat/completions
  replicas: 1
  modelSourceConfig:
    modelSourceType: kubernetesVolume
  kubernetes:
    volumes:
      - name: model-weights
        hostPath:
          path: /opt/dlami/nvme/<YOUR_MODEL>
          type: Directory
  loadBalancer:
    healthCheckPath: /health
  worker:
    image: lmcache/vllm-openai:latest
    args:
      - /opt/ml/model
      - --max-model-len
      - "15000"
      - --tensor-parallel-size
      - "1"
    modelInvocationPort:
      containerPort: 8000
      name: http
    modelVolumeMount:
      name: model-weights
      mountPath: /opt/ml/model
    resources:
      limits:
        nvidia.com/gpu: "1"
      requests:
        cpu: "6"
        memory: 30Gi
        nvidia.com/gpu: "1"
    environmentVariables:
      - name: PYTHONHASHSEED
        value: "123"
      - name: VLLM_REQUEST_TIMEOUT
        value: "600"
EOF

をデプロイしますInferenceEndpointConfig。

kubectl apply -f deploy_nvme_k8s_volume.yaml

デプロイのステータスを確認します。

kubectl describe InferenceEndpointConfig nvme-k8s-volume -n default

クラスターの OIDC プロバイダー ID を取得します。

aws eks describe-cluster --name <CLUSTER_NAME> --region <REGION> \
  --query "cluster.identity.oidc.issuer" --output text

IAM 信頼ポリシーを作成します。プレースホルダー値を置き換えてtrust-policy.json、以下を として保存します。

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.<REGION>.amazonaws.com/id/<OIDC_ID>"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "oidc.eks.<REGION>.amazonaws.com/id/<OIDC_ID>:sub": "system:serviceaccount:<NAMESPACE>:<SA_NAME>",
        "oidc.eks.<REGION>.amazonaws.com/id/<OIDC_ID>:aud": "sts.amazonaws.com"
      }
    }
  }]
}

IAM ロールを作成し、モデルバケットのスコープ付き Amazon S3 読み取りポリシーをアタッチします。

aws iam create-role --role-name <ROLE_NAME> \
  --assume-role-policy-document file://trust-policy.json

aws iam put-role-policy --role-name <ROLE_NAME> \
  --policy-name S3ModelReadAccess \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::<YOUR_BUCKET>",
        "arn:aws:s3:::<YOUR_BUCKET>/<YOUR_MODEL_PREFIX>/*"
      ]
    }]
  }'

IRSA 注釈を使用して Kubernetes サービスアカウントを作成します。

kubectl create sa <SA_NAME> -n <NAMESPACE>
kubectl annotate sa <SA_NAME> -n <NAMESPACE> \
  eks.amazonaws.com/role-arn=arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>

InferenceEndpointConfig YAML ファイルを作成します。プレースホルダー値を実際のリソース識別子に置き換えます。

cat <<EOF> deploy_nvme_k8s_volume_fallback.yaml
apiVersion: inference.sagemaker.aws.amazon.com/v1
kind: InferenceEndpointConfig
metadata:
  name: nvme-k8s-volume-fallback
  namespace: default
spec:
  endpointName: nvme-k8s-volume-fallback
  modelName: Qwen2.5-VL-7B-Instruct
  invocationEndpoint: v1/chat/completions
  replicas: 1
  modelSourceConfig:
    modelSourceType: kubernetesVolume
  kubernetes:
    serviceAccountName: <YOUR_SERVICE_ACCOUNT>
    initContainers:
      - name: smart-loader
        image: public.ecr.aws/aws-cli/aws-cli:latest
        command: ["/bin/bash", "-c"]
        args:
          - |
            if [ "$(ls -A /model)" ]; then
              echo "NVMe hit — model already present ($(du -sh /model | cut -f1))"
            else
              echo "NVMe miss — downloading from S3"
              aws s3 sync s3://<YOUR_BUCKET>/<YOUR_MODEL>/ /model/
            fi
        volumeMounts:
          - name: model-weights
            mountPath: /model
    volumes:
      - name: model-weights
        hostPath:
          path: /opt/dlami/nvme/<YOUR_MODEL>
          type: DirectoryOrCreate
  loadBalancer:
    healthCheckPath: /health
  worker:
    image: lmcache/vllm-openai:latest
    args:
      - /opt/ml/model
      - --max-model-len
      - "15000"
      - --tensor-parallel-size
      - "1"
    modelInvocationPort:
      containerPort: 8000
      name: http
    modelVolumeMount:
      name: model-weights
      mountPath: /opt/ml/model
    resources:
      limits:
        nvidia.com/gpu: "1"
      requests:
        cpu: "6"
        memory: 30Gi
        nvidia.com/gpu: "1"
    environmentVariables:
      - name: PYTHONHASHSEED
        value: "123"
EOF

をデプロイしますInferenceEndpointConfig。

kubectl apply -f deploy_nvme_k8s_volume_fallback.yaml

デプロイのステータスを確認します。

kubectl describe InferenceEndpointConfig nvme-k8s-volume-fallback -n default

InferenceEndpointConfig YAML ファイルを作成します。プレースホルダー値を実際のリソース識別子に置き換えます。

cat <<EOF> deploy_nvme_s3_prefetch_fallback.yaml
apiVersion: inference.sagemaker.aws.amazon.com/v1
kind: InferenceEndpointConfig
metadata:
  name: nvme-s3-prefetch-fallback
  namespace: default
spec:
  endpointName: nvme-s3-prefetch-fallback
  modelName: Qwen2.5-VL-7B-Instruct
  invocationEndpoint: v1/chat/completions
  replicas: 1
  modelSourceConfig:
    modelSourceType: s3
    s3Storage:
      bucketName: <YOUR_BUCKET>
      region: <YOUR_REGION>
    prefetchEnabled: true
  kubernetes:
    serviceAccountName: <YOUR_SERVICE_ACCOUNT>
    initContainers:
      - name: smart-loader
        image: public.ecr.aws/aws-cli/aws-cli:latest
        command: ["/bin/bash", "-c"]
        args:
          - |
            # Check NVMe first, fall back to S3 mount, then copy to RAM
            if [ "$(ls -A /nvme)" ]; then
              echo "NVMe hit ($(du -sh /nvme | cut -f1))"
              echo "Copying model from NVMe to RAM..."
              cp -r /nvme/* /model/
            else
              echo "NVMe miss — copying from S3 mount to NVMe, then NVMe to RAM"
              cp -r /s3-model/* /nvme/
              cp -r /nvme/* /model/
            fi
            echo "Done. $(du -sh /model | cut -f1) in RAM."
        volumeMounts:
          - name: model-weights
            mountPath: /s3-model
          - name: nvme-cache
            mountPath: /nvme
          - name: model-weights-copy
            mountPath: /model
    volumes:
      - name: nvme-cache
        hostPath:
          path: /opt/dlami/nvme/<YOUR_MODEL>
          type: DirectoryOrCreate
  loadBalancer:
    healthCheckPath: /health
  worker:
    image: lmcache/vllm-openai:latest
    args:
      - /opt/ml/model
      - --max-model-len
      - "15000"
      - --tensor-parallel-size
      - "1"
    modelInvocationPort:
      containerPort: 8000
      name: http
    modelVolumeMount:
      name: model-weights
      mountPath: /opt/ml/model
    resources:
      limits:
        nvidia.com/gpu: "1"
      requests:
        cpu: "6"
        memory: 30Gi
        nvidia.com/gpu: "1"
    environmentVariables:
      - name: PYTHONHASHSEED
        value: "123"
      - name: VLLM_REQUEST_TIMEOUT
        value: "600"
EOF

をデプロイしますInferenceEndpointConfig。

kubectl apply -f deploy_nvme_s3_prefetch_fallback.yaml

デプロイのステータスを確認します。

kubectl describe InferenceEndpointConfig nvme-s3-prefetch-fallback -n default

プリロードポッド YAML ファイルを作成します。プレースホルダー値を実際のリソース識別子に置き換えます。

cat <<EOF> nvme-s3-copy.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nvme-s3-copy
  namespace: default
spec:
  nodeName: <TARGET_NODE>
  restartPolicy: Never
  containers:
    - name: s3-copy
      image: public.ecr.aws/aws-cli/aws-cli:latest
      command: ["/bin/bash", "-c"]
      args:
        - |
          echo "=== Starting S3 sync to NVMe ==="
          aws s3 sync s3://<YOUR_BUCKET>/<YOUR_MODEL>/ /nvme/<YOUR_MODEL>/ --region <YOUR_REGION>
          echo "=== Sync complete ==="
          ls -la /nvme/<YOUR_MODEL>/
          du -sh /nvme/<YOUR_MODEL>/
          echo "=== Done ==="
      volumeMounts:
        - name: nvme-storage
          mountPath: /nvme
  serviceAccountName: default
  volumes:
    - name: nvme-storage
      hostPath:
        path: /opt/dlami/nvme
        type: Directory
EOF

ポッドを適用し、同期の進行状況をモニタリングします。

kubectl apply -f nvme-s3-copy.yaml
kubectl get pod nvme-s3-copy -w
kubectl logs nvme-s3-copy -f

同期が完了したら、ポッドをクリーンアップします。

kubectl delete pod nvme-s3-copy