部署大模型推理服務

1. 根據以下內容創建PVC。具體操作，請參見創建PVC。

   參數	示例
   名稱	nas-test。
   分配模式	選擇使用掛載點域名創建。
   總量	20Gi。
   掛載點域名	08cxxxxxxec-wxxxxxxcn-hangzhou.nas.aliyuncs.com

2. 使用kubectl連接到ACS集群。具體操作，請參見獲取集群KubeConfig并通過kubectl工具連接集群。

3. 使用以下內容，創建your-deployment.yaml。

   說明

   關于以下示例中使用的GPU卡具體型號，請參見GPU型號及驅動版本。

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: my-deployment
     name: my-deployment
     namespace: default
   spec:
     progressDeadlineSeconds: 600
     replicas: 1
     revisionHistoryLimit: 10
     selector:
       matchLabels:
         app: my-deployment
     template:
       metadata:
         labels:
           # 指定GPU型號為example-model，請按實際情況填寫，如T4
           alibabacloud.com/gpu-model-series: example-model
           # 指定compute-class為gpu類型
           alibabacloud.com/compute-class: gpu
           app: my-deployment
       spec:
         containers:
           - command:
             - sh
             - -c
             - python3 -m vllm.entrypoints.openai.api_server --model /mnt/Qwen2.5-7B-Instruct --trust-remote-code --tensor-parallel-size 1  --disable-custom-all-reduce
             image: egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/llm-inference:vllm0.5.4-deepgpu-llm24.7-pytorch2.4.0-cuda12.4-ubuntu22.04
             imagePullPolicy: IfNotPresent #Always
             name: my-deployment
             resources:
               limits:
                 cpu: 16
                 memory: 64Gi
                 nvidia.com/gpu: "1"
               requests:
                 cpu: 16
                 memory: 64Gi
                 nvidia.com/gpu: "1"
             terminationMessagePath: /dev/termination-log
             terminationMessagePolicy: File
             volumeMounts:
               - mountPath: /mnt
                 name: nas-test
               - mountPath: /dev/shm
                 name: cache-volume
         dnsPolicy: ClusterFirst
         restartPolicy: Always
         schedulerName: default-scheduler
         securityContext: {}
         terminationGracePeriodSeconds: 30
         volumes:
           - name: nas-test
             persistentVolumeClaim:
               # PVC的聲明
               claimName: nas-test
           - name: cache-volume
             emptyDir:
               medium: Memory
               sizeLimit: 64G
   ---
   apiVersion: v1
   kind: Service
   metadata:
     annotations:
       service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "internet"
       service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version: ipv4
     labels:
       app: my-deployment
     name: svc-llm
     namespace: default
   spec:
     externalTrafficPolicy: Local
     ports:
     - name: serving
       port: 8000
       protocol: TCP
       targetPort: 8000
     selector:
       app: my-deployment
     type: LoadBalancer

4. 執行以下命令，部署大模型推理任務和對應的service。

   kubectl apply -f your-deployment.yaml

5. 由于鏡像較大，您可以通過執行以下命令持續關注模型部署情況，大約需要20分鐘左右。

   kubectl logs -f my-deployment-787b8xxxxx-xxxxx

   當出現以下格式的輸出時，說明模型已經部署成功。

   INFO:     Started server process [2]
   INFO:     Waiting for application startup.
   INFO:     Application startup complete.
   INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
   INFO 12-13 12:39:11 metrics.py:406] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
   INFO 12-13 12:39:21 metrics.py:406] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
   INFO 12-13 12:39:31 metrics.py:406] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
   INFO 12-13 12:39:41 metrics.py:406] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.

6. 執行以下命令，查看部署的Service。

   kubectl get svc

   預期輸出：

   NAME         TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)    AGE
   kubernetes   ClusterIP      10.0.0.1       <none>           443/TCP    3h38m
   svc-llm      LoadBalancer   10.0.143.103   112.xxx.xxx.177  8000/TCP   58s

   其中EXTERNAL-IP為Service對外公開的IP。記錄此IP，方便后續進行測試。