RHEL VM's SEV-SNP attests failed on VMware ESXi

[dulili]

Describe the bug

Attest SEV-SNP RHEL VM failed with below log
[root@localhost kbs]# /usr/local/bin/kbs-client --url https://trusteeserver:8080/ --cert-file /root/trustee/kbs/host.crt attest --tee-key-file test/tee_key.pem > test/attestation_token [2025-02-24T09:49:49Z WARN kbs_protocol::client::rcar_client] RCAR handshake failed: RcarHandshake("error sending request for url (https://trusteeserver:8080/kbs/v0/auth)"), retry 1... [2025-02-24T09:49:50Z WARN kbs_protocol::client::rcar_client] RCAR handshake failed: RcarHandshake("error sending request for url (https://trusteeserver:8080/kbs/v0/auth)"), retry 2... [2025-02-24T09:49:51Z WARN kbs_protocol::client::rcar_client] RCAR handshake failed: RcarHandshake("error sending request for url (https://trusteeserver:8080/kbs/v0/auth)"), retry 3... [2025-02-24T09:49:52Z WARN kbs_protocol::client::rcar_client] RCAR handshake failed: RcarHandshake("error sending request for url (https://trusteeserver:8080/kbs/v0/auth)"), retry 4... Error: RCAR handshake failed: Unable to get token. RCAR handshake retried 5 times. Final attempt failed with: RcarHandshake("error sending request for url (https://trusteeserver:8080/kbs/v0/auth)") [root@localhost kbs]#

How to reproduce

Install a RHEL 10 VM on VMware's ESXi, and enable the SEV-SNP feature, boot up the guest.
Set up Trustee Server and Client on the same VM.
Compile install snp-attesters from source code. Then try to attest it:
/usr/local/bin/kbs-client --url https://trusteeserver:8080 --cert-file /root/trustee/kbs/host.crt attest --tee-key-file test/tee_key.pem > test/attestation_token

CoCo version information

trustee main branch RHEL-10.0

What TEE are you seeing the problem on

Snp

Failing command and relevant log output

Server log
root@trusteeserver:~/trustee/kbs# /usr/local/bin/kbs --config-file config/kbs-config.toml
[2025-02-25T06:43:54Z INFO  kbs] Using config file config/kbs-config.toml
[2025-02-25T06:43:54Z WARN  kbs::admin] insecure admin APIs are enabled
[2025-02-25T06:43:54Z INFO  attestation_service::rvps] launch a built-in RVPS.
[2025-02-25T06:43:54Z INFO  attestation_service::token::ear_broker] Loading default AS policy "ear_default_policy.rego"
[2025-02-25T06:43:54Z INFO  attestation_service::token::ear_broker] No Token Signer key in config file, create an ephemeral key and without CA pubkey cert
[2025-02-25T06:43:54Z INFO  kbs::api_server] Starting HTTPS server at [127.0.0.1:8080, 10.72.139.82:8080]
[2025-02-25T06:43:54Z INFO  actix_server::builder] starting 1 workers
[2025-02-25T06:43:54Z INFO  actix_server::server] Actix runtime found; starting in Actix runtime
[2025-02-25T06:43:54Z INFO  actix_server::server] starting service: "actix-web-service-127.0.0.1:8080", workers: 1, listening on: 127.0.0.1:8080
[2025-02-25T06:43:54Z INFO  actix_server::server] starting service: "actix-web-service-10.72.139.82:8080", workers: 1, listening on: 10.72.139.82:8080
[2025-02-25T06:46:57Z INFO  actix_web::middleware::logger] 10.72.139.82 "POST /kbs/v0/auth HTTP/1.1" 200 74 "-" "attestation-agent-kbs-client/0.1.0" 0.000217
[2025-02-25T06:46:59Z ERROR kbs::error] AttestationError(RcarAttestFailed { source: verify TEE evidence failed
    
    Caused by:
        Verifier evaluate failed: Unable to fetch VCEK from URL: 404 })
[2025-02-25T06:46:59Z INFO  actix_web::middleware::logger] 10.72.139.82 "POST /kbs/v0/attest HTTP/1.1" 401 140 "-" "attestation-agent-kbs-client/0.1.0" 1.216494
[2025-02-25T06:47:00Z INFO  actix_web::middleware::logger] 10.72.139.82 "POST /kbs/v0/auth HTTP/1.1" 200 74 "-" "attestation-agent-kbs-client/0.1.0" 0.000148
[2025-02-25T06:47:05Z ERROR kbs::error] AttestationError(RcarAttestFailed { source: verify TEE evidence failed
    
    Caused by:
        Verifier evaluate failed: Unable to fetch VCEK from URL: 404 })
[2025-02-25T06:47:05Z INFO  actix_web::middleware::logger] 10.72.139.82 "POST /kbs/v0/attest HTTP/1.1" 401 140 "-" "attestation-agent-kbs-client/0.1.0" 1.114948
[2025-02-25T06:47:06Z INFO  actix_web::middleware::logger] 10.72.139.82 "POST /kbs/v0/auth HTTP/1.1" 200 74 "-" "attestation-agent-kbs-client/0.1.0" 0.000114
[2025-02-25T06:47:07Z ERROR kbs::error] AttestationError(RcarAttestFailed { source: verify TEE evidence failed
    
    Caused by:
        Verifier evaluate failed: Unable to fetch VCEK from URL: 429 })
[2025-02-25T06:47:07Z INFO  actix_web::middleware::logger] 10.72.139.82 "POST /kbs/v0/attest HTTP/1.1" 401 140 "-" "attestation-agent-kbs-client/0.1.0" 0.971738
[2025-02-25T06:47:08Z INFO  actix_web::middleware::logger] 10.72.139.82 "POST /kbs/v0/auth HTTP/1.1" 200 74 "-" "attestation-agent-kbs-client/0.1.0" 0.000083
[2025-02-25T06:47:13Z ERROR kbs::error] AttestationError(RcarAttestFailed { source: verify TEE evidence failed
    
    Caused by:
        Verifier evaluate failed: Unable to fetch VCEK from URL: 429 })
[2025-02-25T06:47:13Z INFO  actix_web::middleware::logger] 10.72.139.82 "POST /kbs/v0/attest HTTP/1.1" 401 140 "-" "attestation-agent-kbs-client/0.1.0" 0.960700
[2025-02-25T06:47:14Z INFO  actix_web::middleware::logger] 10.72.139.82 "POST /kbs/v0/auth HTTP/1.1" 200 74 "-" "attestation-agent-kbs-client/0.1.0" 0.000083
[2025-02-25T06:47:15Z ERROR kbs::error] AttestationError(RcarAttestFailed { source: verify TEE evidence failed
    
    Caused by:
        Verifier evaluate failed: Unable to fetch VCEK from URL: 404 })
[2025-02-25T06:47:15Z INFO  actix_web::middleware::logger] 10.72.139.82 "POST /kbs/v0/attest HTTP/1.1" 401 140 "-" "attestation-agent-kbs-client/0.1.0" 1.122145

[fitzthum]

This error arises when Trustee requests the VCEK for the attesting node from the AMD KDS. Currently we only support finding VCEKs for Milan nodes. Two things that could be happening are 1) this node is using a VLEK, which results in a malformed request to the KDS. I'm not sure if VMWare does this. 2) You are on non-Milan (i.e. Turin) node or some strange virtual env.

I think @AdithyaKrishnan has been working on extending this support. Here's another use-case.

[dulili]

@fitzthum , indeed the testing ESXi host use the AMD EPYC 9124, it's not Milan , it's Genoa. As you said Trustee only support Milan , that's the reason why failed, maybe I can try it on the Milan nodes with Trustee.
Thanks for your explanation!

[tylerfanelli]

The updated attestation reports from AMD have the generation (Milan, Genoa, etc) encoded within them. Once the updated firmware lands, this should be resolvable by fetching certs based on the report.