RFC: integrate Spiffe ID and mTLS into KBS

[Xynnn007]

Background

The Confidential Containers (CoCo) community's Trustee sub-project utilizes the Key Broker Service (KBS) component to provide remote attestation services, aiming to address the implementation of remote attestation and confidential resource distribution. This functionality serves not only CoCo but also other confidential virtualization scenarios. The existing KBS component is a web service that offers remote attestation interface, plug-in-based confidential resource access interface, and access control based on KBS resource policy capabilities.

In the original architecture of KBS, remote attestation is achieved via the RCAR protocol over HTTP, allowing clients to conduct remote attestation. Remote attestation services such as CoCoAS and ITA return results in the form of JWT, which contains the claims of the TCB as well as of the client’s elliptic curve public key. Confidential resource access relies on enforcement of KBS resource policy and JWE payload encryption using the client’s elliptic curve public key, which, although provides access control and protects data confidentiality, still presents issues.

1. Forward secrecy. JWE does not provides perfect forward secrecy. Once the tee private key is lacked the historical data stream can be decrypted.
2. Confidential resource policies filter based on specific resource paths and the client's JWT claims, leading to varying policies for different remote attestation services.
3. Current KBS has the potential to improve the architecture design by decoupling modules based on remote proof authentication, client authorization, and access control. These modules correspond to the core functions of KBS.
4. Current KBS protocol relies on cookie thus binding to HTTP. The requirement is mostly for session id.

Design and Architecture

The new proposed architecture will include the following changes:

1. Communication between the client and KBS must use HTTPS (mTLS).
2. Every client (the kbs client, s.t. confidential VM / CoCo Pod) needs to be specified a Spiffe ID. It would do great help to workload/trust subdomain management by the user.
3. Every client needs to obtain a X.509 public cert SVID (Spiffe Verifiable Identity Document) for remote attestation and verify successfully before gaining access. In contrast, previous design returns an attestation token directly from the backend attestation service.
4. Resource access policy is simplified to a unified format based on Spiffe ID, reducing the need for clients to adapt to different remote attestation services.
5. We decouple session id from http cookie to message level.

This is the overall design diagram

• CA is a plugin. Different kinds of implementation can be integrated. It helps the KBS to certify certs and also verify the endorsement of a given client cert.
• AuthN is a wrapper of current attestation module. Similar with De-couple KBS session management from attestation service #729 proposed.
• Access Control is a wrapper of current kbs resource policy.

KBS protocol updates

The original KBS protocol is

1. C-->S     Request := {}

2. C<--S     Challenge := { Nonce }

3. C-->S     Attestation := { Ev(Nonce, pubkey), pubkey }

4. C<--S     Response := { Token(pubkey, Ev) }

The new one is

1. C-->S     Request := { spiffe_subdomain_id }

2. C<--S     Challenge := { Nonce, session_id }

3. C-->S     Attestation := { Ev(Nonce, pubkey-csr(spiffe_subdomain_id)), pubkey-csr(spiffe_subdomain_id), session-id }

4. C<--S     Response := { pubkey-crt(spiffe_id) }

Note that

1. KBS will return a public key cert rather than attestation tokens to the client in the final step. The client's Spiffe ID is included inside the public key x509 cert's SAN field.
2. The key pair is generated inside the guest.

KBS architecture changes

The basic KBS architecture will be as shown in the figure.

1. We will put the remote attestation and the interaction with the backend attestation as the content of the authentication submodule.
2. In addition, we will add a plug-in CA module to support certificate issuance and verification.
3. In addition, the original KBS resource policy module is expanded to an access control module to manage and filter plugin access requests from the client side.

KBS resource policy changes

Before this, KBS resource policy may need to understand the underlying Attestation Token format. For example, when using the EAR format token, you will encounter the following sample policy

package policy
import rego.v1

default allow = false

allow if {
    input["submods"]["cpu"]["ear.status"] == "affirming"
}

The resource policy composer needs to know that the attestation token format.

with the new design, the resource policy focus on access control based on client identity

{
    "allows": {
        "spiffe://domain/usergroup1/*": [
            "resource/path1",
            "resource/path2/*"
        ],
        "spiffe://domain/user2": [
            "resource/default/key/ssh-demo"
        ]
    }
}

The use of wildcards provides flexibility and configurability for concepts such as subdomains and user groups.

Attestation Service Changes

We do not need to do any change upon Attestation Service.

Lifetime of SVID

Note that SVID is a x509 cert with Spiffe ID in it.

1. The guest will generate an EC key pair. Generate a CSR with the public key and spiffe id
2. the KBS will call the backend CA to sign the csr to generate the SVID. The SVID will have expired time
3. the KBS sends back the SVID to the guest
4. The guest uses the SVID to access secret or other things.
5. If the SVID is expired, the guest needs to do another round of RCAR attestation process to get a new one. During the whole lifetime of the guest, the Spiffe ID will not change.

At the same time, because SVID is in x509 format, it has a wide range of application scenarios, such as implementing encrypted service mesh based on Pod identity. (like istio)

A typical user experience

1. Before deploying the actual pod, the user plans how to manage the client, such as which subdomains to divide into and what permissions each has
2. The user uses kbs-client to register the user and the attestation policy ids that need to be passed in kbs. The input is (spiffe_id, attestation_policy_ids). That means "A client TEE that claims to be a user must be verified against the AS policies with IDs."
3. The user configures the kbs resource access policy
4. The user configures the client id through initdata and deploys the pod
5. Remote attestation occurs, and kbs verifies whether the tee tcb of the source client meets the attributes of the as declaration. If so, it issues a certificate.
6. kbs responds to resource access requests and filters according to the rules defined by the policy

Potential Usages

Restrictions

1. This framework does not fully solve the problem of "a group of VMs with the same TCB simultaneously claiming and attesting themselves to with a same spiffe id". Because we only define some solutions from the software perspective, it is easy to cause factory attacks. Unless the remote attestation report of the hardware contains information about the specific hardware ID (such as s390x), the location and uniqueness of the VM can be guaranteed. Otherwise, pure software cannot avoid forgery. So, we can further alleviate this problem by registering once, or resolve this by introducing the definition of hardware platform identity in the remote attestation process. Although these methods are compatible with the currently proposed framework, they are not the main issues to be solved by the current framework and can be gradually supplemented in the future.

2. The introduction of this framework may cause incompatibility with existing trustees and earlier guest image logic (mainly for KBS protocol). This is also an open that this RFC hopes to discuss: are such destructive changes still allowed before 1.0, or does this new architecture need to be implemented in an independent project rather than based on the existing KBS transformation?

Related previous discussions

#175
confidential-containers/confidential-containers#131
confidential-containers/guest-components#340

[fitzthum]

I think there are at least two main topics here: 1) some kind of integration with SPIFFE/SPIRE and some way to provide an identity to guests. 2) improvements to the KBS protocol.

In theory we might be able to do both of these things together, but that might be too ambitious, and I wonder if we should try to do them separately. I have a couple of proposals for how we can relatively easily add SPIFFE support without overhauling the KBS protocol. I do agree that the KBS protocol has limitations, especially with forward secrecy and reliance on cookies, but I'm not sure if using SVIDs is the right solution.

I think there are a couple red flags in the current proposal that we would have to think through. First, it requires the client to provide a SPIFFE ID. There are a few reasons why this is tricky. First, it makes it more difficult for clients to use coco and the KBS-client even if they don't care about identity and get no benefit from it. SPIFFE is relatively widely used, but most Trustee users probably won't actually know much about it. Requiring them to set an SVID hurts usability. This is especially true for users that have a lot of guests. One of the tricky parts of identity management is assigning the correct id to each guest. In this proposal we leave that completely to the user, which is not ideal.

Another issue is that I think the resource policy in the example above is not powerful enough. We need to create a binding between an identity and the TCB. Sometimes this just means that the TCB is affirming, but we might also want to check that a certain SPIFFE ID maps to a value in the policy (such as a container image hash). Maybe you are thinking that we would move this sort of check into the attestation policy, but that's not a great approach. The attestation policy is typically going to be quite large, complex, and hardware specific. It isn't really intended for workload-specific, or even user-specific configuration. Today we hope that most users never touch the attestation policy, but it seems like here we would need to set it for just about every pod. I don't think that's a good idea, and actually I feel like we've just gotten the resource policy and attestation policy figured out, and I'm not that excited about changing it.

That said, I do see the benefit of SPIFFE-based access control. I think what might make the most sense is to allow Trustee to provision SVIDs as resources via a plugin and allow users to use other SPIFFE-based KMSes in tandem with Trustee. We could also consider a new Trustee endpoint alongside the existing ones that would give SPIFFE-based access to the existing resource backends.

I think we could implement an SVID plugin relatively easily. In the simplest case, we would just need to support plaintext init-data and pass the init-data to the plugin. I would also like to support arbitrary keys in the init-data so we could put the id at the top level. In this case the client would use the plugin to request a special SVID resource. The request would pass through the existing resource policy, allowing the admin to enforce whatever TCB bindings they want. Then the plugin would take the spiffe id from, the init-data and create an SVID from it.

A nice thing about this is that it is one step towards a more sophisticated plugin that could automatically generate SPIFFE ids and SVIDs. To do this we would use a webhook on the host-side to add k8s information to the init-data. (again it would be useful to support arbitrary keys for this). The SPIFFE plugin could then apply its own policy to generate a SPIFFE id with the appropriate selectors. In fact, we could probably implement something very similar to the native k8s SPIFFE logic, which seems like it would be cool.

[Xynnn007]

@fitzthum Yes, and sorry my response is a little long because I want to explain further about the my thoughts and considerations. Actually when I wrote this RFC I reversed the cause and effect. At the beginning, I thought that forward security required us to use TLS instead of JWE. Obviously, we need to change the KBS protocol to change the token + public key to a certificate. At the same time, I have always thinking about to optimizing the early resource policy design, because we do not need this layer of abstraction at some time - whether the remote attestation policy is passed ultimately affects whether the resource can be accessed.

I think there are a couple red flags in the current proposal that we would have to think through. First, it requires the client to provide a SPIFFE ID. There are a few reasons why this is tricky. First, it makes it more difficult for clients to use coco and the KBS-client even if they don't care about identity and get no benefit from it. SPIFFE is relatively widely used, but most Trustee users probably won't actually know much about it. Requiring them to set an SVID hurts usability. This is especially true for users that have a lot of guests. One of the tricky parts of identity management is assigning the correct id to each guest. In this proposal we leave that completely to the user, which is not ideal.

Let me try to simplify your question into how to lower the threshold for users. It is relatively simple. If the user does not write a user id on the AA side, there will be an id in the form of spiffe://default by default, and this id is registered by default, using the default policy in the AS. And this identity has * access permission, and all user identity management logic is shielded from the user. This means that previous users do not need to configure anything, and the experience is still the same. But we may need to add documentation to indicate the hidden mysteries and security risks behind the new features.

Another issue is that I think the resource policy in the example above is not powerful enough. We need to create a binding between an identity and the TCB. Sometimes this just means that the TCB is affirming, but we might also want to check that a certain SPIFFE ID maps to a value in the policy (such as a container image hash). Maybe you are thinking that we would move this sort of check into the attestation policy, but that's not a great approach. The attestation policy is typically going to be quite large, complex, and hardware specific. It isn't really intended for workload-specific, or even user-specific configuration. Today we hope that most users never touch the attestation policy, but it seems like here we would need to set it for just about every pod. I don't think that's a good idea, and actually I feel like we've just gotten the resource policy and attestation policy figured out, and I'm not that excited about changing it.

I have read this paragraph many times, trying to dig out the differences in our perspectives on solving problems. Let me try to explain it from what we are doing today. Today, our KBS consists of two logical parts. The first is the authentication phase, which uses the Attestation Policy to shield the TCB into a simple EAR Token and hand it over to the client. The second is the access control phase, based on the KBS resource policy of the EAR Token, which limits which resources can be accessed by either accessing the bare claims again or by the EAR Level.

It is easy to find that we have handled the semantics of remote attestation in both the AS side and the KBS side strategies.

Why we still have KBS resource policies involving remote attestation? It is because we implicitly assume that "users are not completely sure what they want when writing AS policies, so we leave a second line of defense for KBS to pay attention to which TCB can access specific resources."

I personally think that this is a trade-off we made before because we were not very sure whether remote attestation would be well compatible with existing computer access control and identity management systems. Today we can confidently say that remote attestation is a way to authenticate identity. My design philosophy is that remote attestation is a form of identity authentication, which is on the same level as password verification, human fingerprints, etc., so it is best to stop working when the token or access credentials are issued.

Based on this, we can completely shield the remote attestation-related logic before it reaches KBS. Based on this, KBS is still a relying party under the Rats standard, but our interpretation of relying party is not a confidential resource storage service, but an identity authentication service. The next part of KBS storage resources and access control is actually no longer within the scope of Rats definition, but is still here in software implementation. It could be a reference, but users still can decouple this part by using other opensource projects that leverages SVID for access control.

Then back to your question. You mentioned that we don't want users to face the remote attestation policy directly. The remote attestation policy is specific to the hardware architecture and so on. These are also what we are facing now, and they are not amplified or changed because of this proposal. What is important is that in the user registration phase introduced by the current solution, it is possible to clearly define which verification policies need to be used for verification for a specific registered customer. This is a TODO in the current implementation (the default policy is used). As for the writing of remote attestation policies, I think the responsibilities are clearer. In the past, the user who wrote the attestation policies also needed to pay attention to the KBS resource policy. Now he only needs to pay attention to the AS policy.

As for the KBS admin (I distinguish AS admin and KBS admin for ease), he does not need to understand the concrete attestation policy contents of AS. He only needs to make on-demand combinations of a series of attestation policy IDs, such as "NV B100", "the image is the vllm inference engine", "TDX underlying architecture", and "CoCo upper-layer software stack", and then manage his trust domain.

I think we could implement an SVID plugin relatively easily. In the simplest case, we would just need to support plaintext init-data and pass the init-data to the plugin. I would also like to support arbitrary keys in the init-data so we could put the id at the top level. In this case the client would use the plugin to request a special SVID resource. The request would pass through the existing resource policy, allowing the admin to enforce whatever TCB bindings they want. Then the plugin would take the spiffe id from, the init-data and create an SVID from it.
A nice thing about this is that it is one step towards a more sophisticated plugin that could automatically generate SPIFFE ids and SVIDs. To do this we would use a webhook on the host-side to add k8s information to the init-data. (again it would be useful to support arbitrary keys for this). The SPIFFE plugin could then apply its own policy to generate a SPIFFE id with the appropriate selectors. In fact, we could probably implement something very similar to the native k8s SPIFFE logic, which seems like it would be cool.

I tried to summarize these two paragraphs and reply together. There are mainly three points you are worried about.

1. Why not simply include spiffe id at the top level of initdata.

First of all, logically, within the scope of CoCo, spiffe id is part of initdata, and it is the content injected at startup, it's our consensus. When I wrote it, I put spiffe id on the client side of the KbsProtocol library, and then AA integrated it. From AA's perspective, this is also neater. It is considered that when other projects integrate this library, it can integrate spiffe id in any other form. Of course, we can add any other identifiers or content to initdata as you suggested, all we need to do is to update the AS policy synchronously and let it verify one more initdata field.

2. Why not design it to request SVID resources?

It was also very interesting when I designed this. Since SVID can be used as a credential for subsequent access to resources, why not get rid of JWE altogether? This avoids maintaining another communication encryption stack and can also achieve many good features such as C-S communication bidirectional encryption and forward secrecy.

3. How to adapt the native Spiffe logic. That is, the identity issuer (our KBS) defines the spiffe id.

This may be the most troublesome and controversial, and it is what I want to discuss.
Under the standard SPIFFE architecture, it is the server that publishes the SPIFFE ID, rather than the client defining it itself. You mentioned a good point, which is to add some k8s metadata to initdata through webhook, and then the KBS side determines how to create SVID. First, let's assume that these metadata are credible-because the information injected by the K8S control end under the CoCo model is not credible. Then whether it is my design or your design, we will face the same problem. How do we dynamically generate a Spiffe Id based on these external inputs? In our two scenarios, these inputs (meta information injected by k8s control, or the current client_id in my solution) are actually a "hint" for KBS to generate Spiffe ID. We can completely change client_id to expected_domain or something else. We shoud note that the only values of this value is to

a) let KBS know which AS policies to call to verify the client TEE to issue SVID
b) which Spiffe ID to issue.

We can also modify the register interface slightly, define its input as the name of a sub-trust domain and the corresponding attestation policy ids, and then let KBS generate the spiffe id under given trust domain autonomously.

It should be noted here that even the native Spire design implies some specific settings in the environment for the selector logic to dynamically identify. These specific settings include, for example, the imds content of the AWS cloud instance, and in the CoCo scenario, the "hints" mentioned above in the initdata. These processes are out-of-band for spire/spiffe, but we mention them here because we now need a complete solution design.

[fitzthum]

and sorry my response is a little long because I want to explain further about the my thoughts and considerations.

No worries. There is much discussion in our future. Before jumping into things, I guess one source of tension here is that this proposal actually comes from before the EAR work was introduced, and there are different ways of seeing things. Hopefully we can somehow reconcile this.

Today, our KBS consists of two logical parts. The first is the authentication phase, which uses the Attestation Policy to shield the TCB into a simple EAR Token and hand it over to the client. The second is the access control phase, based on the KBS resource policy of the EAR Token, which limits which resources can be accessed by either accessing the bare claims again or by the EAR Level.

This isn't how I would describe the current approach. The difference between the AS and KBS is best summed up by the two policies. The AS policy captures platform-specific and boot-specific logic about which TCB claims are significant and how specific TCB claims map onto the overall status of the TCB. We adopted the EAR token not because the spec is particularly amazing, but because it crystallizes the purpose of the AS as a thing that returns information about the TCB status (whether it is valid or not). I wouldn't exactly call that authentication, but the terminology isn't too important. Since CoCo has a fixed boot method, we can provide this policy ahead of time and most users will never need to change it. They will simply have a service that takes hw evidence and returns generic information about the Tee.

The KBS policy, on the other hand, is workload-specific, and user-specific, but generic in terms of the Tee platform. These policies are usually much smaller and much simpler. They take the EAR token and the init-data, and determine whether a request should be fulfilled. This policy is the best place to check workload-specific information in the init-data, and to assert that the EAR token represents a guest with the right general properties (especially important for multi-device stuff).

Let's be clear that your proposal would completely change this relationship. That isn't necessarily a bad thing, but I think there are some big red flags. First, it would mix platform-specific information with workload-specific information. You do mention that there could be multiple attestation policies, so maybe you could separate things a little bit, but you would still have to write workload-specific logic in terms of platform-specific TCB claims, which is not ideal. The clear separation of workload and hardware is a big advantage of the current approach.

More fundamentally, if the AS returns a SPIFFE-ID (or endorses a SPIFFE-ID that was provided to it), it would be dramatically less expressive. Today the EAR token contains a lot of information about the guest TCB (both generic and platform-specific). People can use this token for all kinds of things (including using the AS standalone). If the AS returned a SPIFFE-ID instead, we would be throwing away almost everything we know about the Tee. In theory the SPIFFE-ID and its selectors could capture some information about the TCB, but this would be much less than what we already have. Furthermore, the SPIFFE-ID has to be driven by the client, so they would only get a useful ID if they put in the work to orchestrate the ID and the policies. The current approach will return an expressive token no matter what. On top of that, you mention that there would be a default SPIFFE-ID, which is probably what most people would use. This wouldn't be expressive at all. So basically we would change the return type from being a relatively rich EAR token, to being a single id string. That does not seem like a good idea to me.

Today we can confidently say that remote attestation is a way to authenticate identity. My design philosophy is that remote attestation is a form of identity authentication, which is on the same level as password verification, human fingerprints, etc., so it is best to stop working when the token or access credentials are issued.

I am not confident about identity at all. Remote attestation is a way to verify some hardware claims. Depending on how the guest is setup and how the attestation policies are written, this might include some kind of identity, but it might not. I worry that with SPIFFE users are going to misunderstand the hardware guarantees. I think it could make sense to associate a SPIFFE-ID to certain tcb properties of a guest. A SPIFFE-ID could represent that a guest has a certain TCB, a certain image, etc. In this proposal we are leaving all the work of figuring out how an ID maps to the TCB to the user, which I think is error prone. I do think there is a place for SPIFFE in Trustee, but I worry about making it the centerpiece of the attestation flow.

Why not simply include spiffe id at the top level of initdata.

Let's address this on that issue. It isn't a strict requirement for the plugin proposal, but I do feel pretty strongly that we should allow top-level keys.

It was also very interesting when I designed this. Since SVID can be used as a credential for subsequent access to resources, why not get rid of JWE altogether? This avoids maintaining another communication encryption stack and can also achieve many good features such as C-S communication bidirectional encryption and forward secrecy.

It's true that my proposal would introduce some redundancies. As I mentioned earlier I don't love replacing the attestation token with the SVID because it seems less expressive, and because introducing the identity paradigm could be pretty confusing. Also, it would be a huge change. A SPIFFE plugin would be relatively easy to implement. ik you already have some code for this stuff, but I am hoping to reduce churn when possible.

How to adapt the native Spiffe logic. That is, the identity issuer (our KBS) defines the spiffe id.

With the plugin approach we could start with something similar to what you are proposing, where the SPIFFE-ID comes from the user. Then we could introduce more sophisticated logic to automatically generate the ID. I don't yet know the details for ID generation, but I think we would take inspiration from the SPIRE implementation. In theory any information that is accessible on the host could be added to the init-data (even IMDS stuff).

Now there is a very interesting question here about whether this information needs to be trusted. There is some information, such as the container uid (I am looking at this list of selectors), that could be corroborated from inside the guest, but other k8s info (like the pod owner) would be coming directly from the control plane. This is subtle, but my feeling is that some k8s claims need to be validated, but others don't. For instance, if we have a container image name coming from k8s, we probably want to verify that against the init-data that specifies which containers can run. With the plugin approach we would do this via the resource policy. There are other properties where consistency is more important. Basically, some things won't be part of the TCB perhaps because they are relevant later in the lifecycle of the pod, but as long as the identity doesn't change (which it won't), the identity will imply what the future behavior is. For instance we might not check the pod owner against the TCB, but later certain resources will only be granted to certain pod owners, and the id will enforce that.

This is exactly the sort of subtlety I was alluding to above with SPIFFE. There is a lot to figure out here (we didn't even discuss whether these IDs correspond to the pod or to individual containers or components), so it seems wise to implement as a plugin that can evolve and that can use all the existing features we have, rather than to overhaul Trustee and put SPIFFE in the middle.

[niteeshkd]

One thing i would like to mention here that at present the attestation token contains very useful info which is quite helpful to find the issues with attestation failure and composing rvps values etc.

[Xynnn007]

One thing i would like to mention here that at present the attestation token contains very useful info which is quite helpful to find the issues with attestation failure and composing rvps values etc.

Yes, this proposal will not modify any Attestation Token format, the logic of interaction between AS and RVPS, and the AS itself. AS still provides the same ability to accept evidence and return Attestation Token (including claims).

[Xynnn007]

@fitzthum Yes, let's try to converge at some same point.

ik you already have some code for this stuff, but I am hoping to reduce churn when possible.

This is actually the issue I am most concerned about. Because many downstream companies/orgs, such as Fedora, have already added trustee and related components to the source, and many downstream users are also using this remote attestation system. My proposal will cause some modifications (at least it will force downstream users to make adjustments), so I really hope to solve this contradiction in an appropriate way although we are still not in stable v1.0.

This isn't how I would describe the current approach. The difference between the AS and KBS is best summed up by the two policies. The AS policy captures platform-specific and boot-specific logic about which TCB claims are significant and how specific TCB claims map onto the overall status of the TCB. We adopted the EAR token not because the spec is particularly amazing, but because it crystallizes the purpose of the AS as a thing that returns information about the TCB status (whether it is valid or not). I wouldn't exactly call that authentication, but the terminology isn't too important. Since CoCo has a fixed boot method, we can provide this policy ahead of time and most users will never need to change it. They will simply have a service that takes hw evidence and returns generic information about the Tee.
The KBS policy, on the other hand, is workload-specific, and user-specific, but generic in terms of the Tee platform. These policies are usually much smaller and much simpler. They take the EAR token and the init-data, and determine whether a request should be fulfilled. This policy is the best place to check workload-specific information in the init-data, and to assert that the EAR token represents a guest with the right general properties (especially important for multi-device stuff).

This is quite right. What I proposed aligns with most of your idea. The tee evidence consists three parts (hardware evidence, software digests(kernel, ovmf, ...), workload-specific info). AS helps to turn the first two parts into a generic status (Apprisal.TrustTier). And the KBS will use the TrustTier and workload-specific info to filter requests.

Note that initdata is just a carrier of the workload-specific info, what we care about is the workload-specific info itself but not initdata.
It should be made clear that workload-specific content in my semantics does not include the content in claims, because claims almost always contain the content of the first two.

Let's be clear that your proposal would completely change this relationship. That isn't necessarily a bad thing, but I think there are some big red flags. First, it would mix platform-specific information with workload-specific information. You do mention that there could be multiple attestation policies, so maybe you could separate things a little bit, but you would still have to write workload-specific logic in terms of platform-specific TCB claims, which is not ideal. The clear separation of workload and hardware is a big advantage of the current approach.

There may be a misunderstanding here, I mixed platform information with workload-specific information. In fact, it is not the case like I stated.

My idea is to add workload-specific configuration information to initdata to distinguish different workloads/Pods when the underlying hardware and software metrics are the same. I guess you have same idea with me. This begs the question, what configurations can we add?

We know that there are only two types of information: one is from the untrusted end, and the other is from the user. Although both are sent through the untrusted k8s tunnel, the information from the user can obviously be anticipated and verified by the user in advance, but the user on the untrusted end cannot know it.

So when I designed it, I didn't consider the untrusted and unpredictable values, but only the expected values ​​from the user to be embedded in the remote attesstation report. These metadata don't need to be listed in detail, a single simple value is enough, and we can complete the mapping on the KBS side.

More fundamentally, if the AS returns a SPIFFE-ID (or endorses a SPIFFE-ID that was provided to it), it would be dramatically less expressive. Today the EAR token contains a lot of information about the guest TCB (both generic and platform-specific). People can use this token for all kinds of things (including using the AS standalone). If the AS returned a SPIFFE-ID instead, we would be throwing away almost everything we know about the Tee. In theory the SPIFFE-ID and its selectors could capture some information about the TCB, but this would be much less than what we already have. Furthermore, the SPIFFE-ID has to be driven by the client, so they would only get a useful ID if they put in the work to orchestrate the ID and the policies. The current approach will return an expressive token no matter what. On top of that, you mention that there would be a default SPIFFE-ID, which is probably what most people would use. This wouldn't be expressive at all. So basically we would change the return type from being a relatively rich EAR token, to being a single id string. That does not seem like a good idea to me.

The KBS returns Spiffe-id rather than AS. We will do nothing upon the AS as its functionality is clearly independent and modular enough. I may not have written it clearly in the spec. KBS calls AS to verify claims. Different AS plugins behave differently. CoCoAS returns an EAR Token to KBS, and KBS checks whether the trustTier of all submods in it is affirming and then issues an SVID.

I assume that your later concern about SVID instead of Attestation Token is based on the misleading fact that AS returns an SVID instead of Attestation Token, so I will not reply for now and wait for your confirmation.

I am not confident about identity at all. Remote attestation is a way to verify some hardware claims. Depending on how the guest is setup and how the attestation policies are written, this might include some kind of identity, but it might not. I worry that with SPIFFE users are going to misunderstand the hardware guarantees. I think it could make sense to associate a SPIFFE-ID to certain tcb properties of a guest. A SPIFFE-ID could represent that a guest has a certain TCB, a certain image, etc. In this proposal we are leaving all the work of figuring out how an ID maps to the TCB to the user, which I think is error prone. I do think there is a place for SPIFFE in Trustee, but I worry about making it the centerpiece of the attestation flow.

All the considerations will lead to the following of your ideas of selectors or so.

Now there is a very interesting question here about whether this information needs to be trusted. There is some information, such as the container uid (I am looking at this list of selectors), that could be corroborated from inside the guest, but other k8s info (like the pod owner) would be coming directly from the control plane. This is subtle, but my feeling is that some k8s claims need to be validated, but others don't.
Basically, some things won't be part of the TCB perhaps because they are relevant later in the lifecycle of the pod, but as long as the identity doesn't change (which it won't), the identity will imply what the future behavior is. For instance we might not check the pod owner against the TCB, but later certain resources will only be granted to certain pod owners, and the id will enforce that.
This is exactly the sort of subtlety I was alluding to above with SPIFFE. There is a lot to figure out here (we didn't even discuss whether these IDs correspond to the pod or to individual containers or components)

When we only care about the first-level users of CoCo, that is, the workload provider, the Spiffe ID mentioned in this RFC is for Pod. We can easily add a standard Spiffe interface to AA to provide the ability to issue container spiffe SVID to the container. (The simplest way is to let AA expose the interface and let ASR do the same http forwarding, so that the logic inside the container can obtain its own SVID and trust bundle by accessing a specific URL)

This actually includes two layers of selector logic:

1. The first layer is to grant Pod-level SVID through remote attestation (of course, we can add some k8s pod-related information on this basis, but this information is directly obtained inside the guest, and may not necessarily be obtained through initdata. This is related to the basic framework of how Kata combines with Spiffe that I mentioned in the meeting). The attestation requirement at this level is actually whether the CVM black box meets expectations. If it meets expectations, he will run his own workload in it.
2. The second layer is to rely on the logic inside the guest, through the link you shared, or even Kata's own ContainerId and other information, to issue a sub-SVID for the container.

This proposal focuses on the first layer, but the second layer is easy to support based on this framework.

For instance, if we have a container image name coming from k8s, we probably want to verify that against the init-data that specifies which containers can run. With the plugin approach we would do this via the resource policy. There are other properties where consistency is more important.

Due to the two layers described before, the first layer will result in the pod accepts a container security policy that we have supported, or a kata policy. Both of them could control what container can be run inside the guest.

So directly in the next layer, when a container wants to obtain its own SVID, the container must have been allowed to run. We do not need to verify its validity again, but only need to obtain its other metadata through the selector to assist in issuing the SVID.

[fitzthum]

This is quite right. What I proposed aligns with most of your idea. The tee evidence consists three parts (hardware evidence, software digests(kernel, ovmf, ...), workload-specific info). AS helps to turn the first two parts into a generic status (Apprisal.TrustTier). And the KBS will use the TrustTier and workload-specific info to filter requests.

The way I see it is that SPIFFE is a good fit for the workload portion of things, but it's not a great way to describe the HW TCB. Mapping the HW TCB onto a SPIFFE-ID is complicated. I think it makes more sense to keep the two separate. The KBS policy can take both an EAR token (to validate the TCB) and a SPIFFE-ID (to validate the workload). In fact, we already support this (at least we would if we allowed arbitrary keys). Imagine a policy like this

package policy
import rego.v1

default allow = false

allow if {
    input["submods"]["cpu"]["ear.status"] == "affirming"
    input["submods"]["cpu"]["ear.annotated-evidence"]["init-data"]["spiffe-id"] == "spiffe://domain/user2"
}

This really isn't much different from what you are suggesting as the new KBS policy, but it allows policies that have much more control over the TCB, and we don't have to worry about mapping SPIFFE-IDs to TCB claims at all. Also, if someone doesn't want to use SPIFFE, they can just stick their own ID scheme into the init-data/policy. And did I mention that we can do this today with very minimal changes?

The KBS returns Spiffe-id rather than AS. We will do nothing upon the AS as its functionality is clearly independent and modular enough. I may not have written it clearly in the spec. KBS calls AS to verify claims. Different AS plugins behave differently. CoCoAS returns an EAR Token to KBS, and KBS checks whether the trustTier of all submods in it is affirming and then issues an SVID.

I got a bit confused because you mentioned specifying multiple policies in the AS, which the EAR broker doesn't support. It seems like your approach would want some of these policies to be workload specific (you mention "NV B100", "the image is the vllm inference engine", "TDX underlying architecture", and "CoCo upper-layer software stack" above) which does change the role of the AS.

Maybe more importantly I don't think it's sufficient to hard-code the evaluation logic of the EAR token (i.e check that the trust tier is affirming). A user might want to check other aspects, such as which platform is being used (at least that it's not the sample attester), which devices are present, etc. I think we really need a policy for validating the EAR token.

Let's skip the discussion of the hierarchical approach for. now. We can figure that out later probably and I think you're on the right track with allowing the AA to generate container IDs.

One drawback of what I proposed with the plugin is that it wouldn't be useful for releasing resources to guests with a given SPIFFE-ID. Instead, the purpose would be to generate SPIFFE-IDs that a guest could use to get resources from someplace else. I think you like the idea of having our KBS policy in terms of SPIFFE-IDs, which I can understand. To support this we would need to move the generation of the ID forward. When the ID is provided manually by the user, it is part of the init-data at policy evaluation time, but if we generate the ID automatically with a plugin, that would happen after the policy is evaluated, which might not be ideal. So now I'm thinking that if we do automatic generation of the SPIFFE-ID, we should do it earlier. We could introduce an init-data pre-processing interface, but another option would be to simply do it on the host. In fact, could we just use the native SPIRE code on the host to generate a SPIFFE-iD and put it in the init-data? If the user wants to check any TCB properties relative to the SPIFFE-ID (i.e do the allowed images match the image name), they can easily do that via the KBS policy.

Anyway, to me it seems like we can get most of what we want with SPIFFE with minimal changes (not including the issues with the KBS protocol)

[fitzthum]

Btw another thing to think about is that if you really like the new policy interface you describe above, we could probably implement something like that as a separate KBS policy engine. It could read the SPIFFE-ID from the init-data automatically and do something with the EAR token based on some minimal settings.

[Xynnn007]

The way I see it is that SPIFFE is a good fit for the workload portion of things, but it's not a great way to describe the HW TCB. Mapping the HW TCB onto a SPIFFE-ID is complicated.

Spiffe semantics is for cloud-native scenarios and it surely is good fit for workload things, but also for the entity who provisions SVID for workloads. In the current solution, it describes the capabilities of pod nodes, and the SVID is more like the agent's in this design, while the SVID of workload in Spiffe is more like SVIDs of specific container provisioned by the agent.

Thus, the idea of this proposal is to help the client (now it is the Attestation Agent, named client in KBS context, and named agent in spire context) have ability prove that the current Pod VM environment is fine then get SVID of the pod from KBS, and then can further issue a workload SVID to the concrete container. At the same time, due to the particularity of the kata/coco architecture, the Pod itself becomes an entity (VM), which will request some VM-level resources, such as keys, image policies, auths, so we can also use the agent's SVID for access control.

The example you showed is very intuitive, but it also describes Pod-level SVID and its access control.

Soon you will encounter same problems:

1. How to let the Pod level retain its own private key to issue the SVID of the lower-level container
2. How to perform access control in the semantics of pure Spiffe ID

From the sample strategy alone, we could infer that our two solutions are somehow different, so it is surely that very minimal changes are required as you said. In your design, you don’t care about these two parts, but only care about giving SVID. As for how to use it and do other things next, it seems out-of-scope. My solution is to provide a relatively complete (although inevitably simple) solution, providing solutions to the above two problems within an expected extensible framework.

I got a bit confused because you mentioned specifying multiple policies in the AS, which the EAR broker doesn't support. It seems like your approach would want some of these policies to be workload specific (you mention "NV B100", "the image is the vllm inference engine", "TDX underlying architecture", and "CoCo upper-layer software stack" above) which does change the role of the AS.

We could make it support, e.g. use ',' to separate the policy ids as a workaround. Another is that the RFC is going to consider extending this to an array.

I feel like our understandings of workload are a little different. The examples I give here are all at the Pod VM level. Workload refers to the concrete containers, right?
In the CoCo scenario, except for the container workload, the underlying CPU (s.t. TDX underlying architecture, it's tdx-module mr_seam, guest firmware mr_td, etc) and supporting software stack (s.t. CoCo upper-layer software stack, like rootfs hash in kernel cmdline, kernel hash in evenlog, etc.) are the same.

This does not change the role of the AS. Currently, the AS verifies the endorsement of evidence and providing policy verification for claims.

Maybe more importantly I don't think it's sufficient to hard-code the evaluation logic of the EAR token (i.e check that the trust tier is affirming). A user might want to check other aspects, such as which platform is being used (at least that it's not the sample attester), which devices are present, etc. I think we really need a policy for validating the EAR token.

In my initial design ( about 1.5 years ago, when we did not have EAR), there will be two policy engines on the KBS side, one responsible for the AuthS module to process the return value of AS, and the other is used in the KBS access control module to filter Spiffe ID, as shown in the figure.

But with the introduction of EAR, I discovered its powerful expressiveness (we can use a simple affirming to represent Ok). In addition, our API design for AS ensures that it has sufficient capabilities to verify the integrity of any plaintext data bound to Evidence, such as initdata. I omitted this layer in the latest proposal. As for the TEE type you mentioned, I think it can be done at the AS policy level, just bind it to the corresponding AS policy during the Spiffe ID registration phase. This still ensures that people who write KBS policies do not need too much knowledge background on remote attestation, or even TEE types.

In addition, let me try to capture our differences: our designs are actually two partially overlapping but largely different designs aiming to resolve different problems. I still think my design has practical value, and I am willing to contribute the relevant code to the community at the right place.

[Xynnn007]

CC @bpradipt, might be interested in this topic - and welcome to your always inspiring views

[bpradipt]

@Xynnn007 I'm still going through it, but it's taking some time. My current focus is on the spiffe ID part.
Is the intent to implement SPIFFE in Trustee as a plugin?

[Xynnn007]

Is the intent to implement SPIFFE in Trustee as a plugin?

In this proposal no, it's aim to use a more native way s.t. after the KBS protocol the returned thing is SVID, while currently is the JWT-format attestation token with public key inside as claim.

[bpradipt]

@Xynnn007 looking at your demo in the CoCo meeting and reading through this RFC, it appears that switching to SVID will make it easier to write resource access policies. Its a useful addition to Trustee.

Any thoughts, on how SVID generation via Trustee can integrate with an existing SPIRE installation in the environment already issuing SVIDs for the workloads ?

[Xynnn007]

Any thoughts, on how SVID generation via Trustee can integrate with an existing SPIRE installation in the environment already issuing SVIDs for the workloads ?

If the cluster is already running and SVID has been issued and is working, introducing Trustee might cause some problems (more tight verification logic would let some node go?).

But go back to the question itself, Trustee and the RCAR protocol it provides can be used as part of Node Attestation and Node Selection, which we could integrate KBS+AS logic into Spire-Server & KBS-Client logic into Spire-Agent. In that case, trustee is not responsible for generating SVID but only provides more context for node attestation. The original SPIRE ecosystem still works.

Btw, It is true that trustee itself provides access control and identity authentication (remote attestation is a kind of identity authentication) capabilities, and the spire system provides identity authentication capabilities, so it is very easy for the trustee function to be absorbed by spire as a submodule at the abstract level during design.

So, there might be two design ways according to my observation

1. respect the spire ecosystem. trustee would be a plugin.
2. use a simplified spiffe application, just as proposed in this proposal. We use the spiffe ids conceptions.