[Umbrella] [Karmada config && certificates] secret and path naming convention #6051

chaosi-zju · 2025-01-15T10:40:12Z

Task one. [Karmada Config] secret and path naming convention

Naming convention:

secretName: ${karmada_instance_name}-${component}-config
volumeName: karmada-config
volumeMountPath: /etc/karmada/config
fileName (secretField) : karmada.config

${karmada_instance_name} defaults to value karmada, but theoretically it also could be karmada-xxx or xxx-karmada.

Needed PRs:

local up: standardize the naming of karmada config in local up method #5679 @chaosi-zju
karmadactl: standardize the naming of karmada config in karmadactl method #5797 @chaosi-zju
helm: @help-wanted
karmada-operator: Standardize the naming of karmada config in Karmada Operator #6082 @seanlaii

Examples:

Karmada config

1. karmada-aggregated-apiserver

secretName: karmada-aggregated-apiserver-config
volumeName: karmada-config
volumeMountPath: /etc/karmada/config
command:
  - --kubeconfig=/etc/karmada/config/karmada.config
  - --authentication-kubeconfig=/etc/karmada/config/karmada.config
  - --authorization-kubeconfig=/etc/karmada/config/karmada.config

2. karmada-controller-manager

secretName: karmada-controller-manager-config
volumeName: karmada-config
volumeMountPath: /etc/karmada/config
command:
  - --kubeconfig=/etc/karmada/config/karmada.config

3. karmada-scheduler

secretName: karmada-scheduler-config
volumeName: karmada-config
volumeMountPath: /etc/karmada/config
command:
  - --kubeconfig=/etc/karmada/config/karmada.config

4. karmada-descheduler

secretName: karmada-descheduler-config
volumeName: karmada-config
volumeMountPath: /etc/karmada/config
command:
  - --kubeconfig=/etc/karmada/config/karmada.config

5. karmada-metrics-adapter

secretName: karmada-metrics-adapter-config
volumeName: karmada-config
volumeMountPath: /etc/karmada/config
command:
  - --kubeconfig=/etc/karmada/config/karmada.config
  - --authentication-kubeconfig=/etc/karmada/config/karmada.config
  - --authorization-kubeconfig=/etc/karmada/config/karmada.config

6. karmada-search

secretName: karmada-search-config
volumeName: karmada-config
volumeMountPath: /etc/karmada/config
command:
  - --kubeconfig=/etc/karmada/config/karmada.config
  - --authentication-kubeconfig=/etc/karmada/config/karmada.config
  - --authorization-kubeconfig=/etc/karmada/config/karmada.config

7. karmada-webhook

secretName: karmada-webhook-config
volumeName: karmada-config
volumeMountPath: /etc/karmada/config
command:
  - --kubeconfig=/etc/karmada/config/karmada.config

8. kube-controller-manager

secretName: kube-controller-manager-config
volumeName: karmada-config
volumeMountPath: /etc/karmada/config
command:
  - --kubeconfig=/etc/karmada/config/karmada.config
  - --authentication-kubeconfig=/etc/karmada/config/karmada.config
  - --authorization-kubeconfig=/etc/karmada/config/karmada.config

Task two. [Karmada Certificate] secret and path naming convention

Naming convention:

Server Certificate:
- secretName: ${karmada_instance_name}-${component}-cert
- volumeName: server-cert
- volumeMountPath: /etc/karmada/pki/server
- fileName (secretField) : ca.crt、tls.crt、tls.key
Client Certificate:
- Secret: ${karmada_instance_name}-${component}-${server}-client-cert
- Volume: ${server}-client-cert
- Volume mount path: /etc/karmada/pki/${server}-client
- fileName (secretField) : ca.crt、tls.crt、tls.key

${karmada_instance_name} defaults to value karmada, but theoretically it also could be karmada-xxx or xxx-karmada.

Needed PRs:

lcoal up: standardize the naming of karmada secrets in local up method #5423 @chaosi-zju
karmadactl: @help-wanted
helm: @help-wanted
karmada-operator: /assign @husnialhamdani

Examples:

karmada certificates

1. karmada-etcd

command:
  - --cert-file=/etc/karmada/pki/server/tls.crt
  - --key-file=/etc/karmada/pki/server/tls.key
  - --trusted-ca-file=/etc/karmada/pki/server/ca.crt
volumeMounts:
  - name: server-cert
    mountPath: /etc/karmada/pki/server
  - name: etcd-client-cert
    mountPath: /etc/karmada/pki/etcd-client
volumes:
  - name: server-cert
    secret:
      secretName: etcd-cert
  - name: etcd-client-cert
    secret:
      secretName: etcd-etcd-client-cert

2. karmada-apiserver

command:
  - --etcd-cafile=/etc/karmada/pki/etcd-client/ca.crt
  - --etcd-certfile=/etc/karmada/pki/etcd-client/tls.crt
  - --etcd-keyfile=/etc/karmada/pki/etcd-client/tls.key
  - --service-account-key-file=/etc/karmada/pki/service-account-key-pair/sa.pub
  - --service-account-signing-key-file=/etc/karmada/pki/service-account-key-pair/sa.key
  - --proxy-client-cert-file=/etc/karmada/pki/front-proxy-client/tls.crt
  - --proxy-client-key-file=/etc/karmada/pki/front-proxy-client/tls.key
  - --requestheader-client-ca-file=/etc/karmada/pki/front-proxy-client/ca.crt
  - --tls-cert-file=/etc/karmada/pki/server/tls.crt
  - --tls-private-key-file=/etc/karmada/pki/server/tls.key
  - --client-ca-file=/etc/karmada/pki/server/ca.crt
volumeMounts:
  - name: server-cert
    mountPath: /etc/karmada/pki/server
    readOnly: true
  - name: etcd-client-cert
    mountPath: /etc/karmada/pki/etcd-client
    readOnly: true
  - name: front-proxy-client-cert
    mountPath: /etc/karmada/pki/front-proxy-client
    readOnly: true
  - name: service-account-key-pair
    mountPath: /etc/karmada/pki/service-account-key-pair
    readOnly: true
volumes:
  - name: server-cert
    secret:
      secretName: karmada-apiserver-cert
  - name: etcd-client-cert
    secret:
      secretName: karmada-apiserver-etcd-client-cert
  - name: front-proxy-client-cert
    secret:
      secretName: karmada-apiserver-front-proxy-client-cert
  - name: service-account-key-pair
    secret:
      secretName: karmada-apiserver-service-account-key-pair

3. karmada-aggregated-apiserver

command:
  - --etcd-cafile=/etc/karmada/pki/etcd-client/ca.crt
  - --etcd-certfile=/etc/karmada/pki/etcd-client/tls.crt
  - --etcd-keyfile=/etc/karmada/pki/etcd-client/tls.key
  - --tls-cert-file=/etc/karmada/pki/server/tls.crt
  - --tls-private-key-file=/etc/karmada/pki//server/tls.key
volumeMounts:
  - name: server-cert
    mountPath: /etc/karmada/pki/server
    readOnly: true
  - name: etcd-client-cert
    mountPath: /etc/karmada/pki/etcd-client
    readOnly: true
volumes:
  - name: server-cert
    secret:
      secretName: karmada-aggregated-apiserver-cert
  - name: etcd-client-cert
    secret:
      secretName: karmada-aggregated-apiserver-etcd-client-cert

4. karmada-scheduler

command:
  - --scheduler-estimator-ca-file=/etc/karmada/pki/scheduler-estimator-client/ca.crt
  - --scheduler-estimator-cert-file=/etc/karmada/pki/scheduler-estimator-client/tls.crt
  - --scheduler-estimator-key-file=/etc/karmada/pki/scheduler-estimator-client/tls.key
volumeMounts:
  - name: scheduler-estimator-client-cert
    mountPath: /etc/karmada/pki/scheduler-estimator-client
    readOnly: true
volumes:
  - name: scheduler-estimator-client-cert
    secret:
      secretName: karmada-scheduler-scheduler-estimator-client-cert

5. karmada-descheduler

command:
  - --scheduler-estimator-ca-file=/etc/karmada/pki/scheduler-estimator-client/ca.crt
  - --scheduler-estimator-cert-file=/etc/karmada/pki/scheduler-estimator-client/tls.crt
  - --scheduler-estimator-key-file=/etc/karmada/pki/scheduler-estimator-client/tls.key
volumeMounts:
  - name: scheduler-estimator-client-cert
    mountPath: /etc/karmada/pki/scheduler-estimator-client
    readOnly: true
volumes:
  - name: scheduler-estimator-client-cert
    secret:
      secretName: karmada-scheduler-scheduler-estimator-client-cert

6. karmada-scheduler-estimator

command:
  - --grpc-auth-cert-file=/etc/karmada/pki/server/tls.crt
  - --grpc-auth-key-file=/etc/karmada/pki/server/tls.key
  - --grpc-client-ca-file=/etc/karmada/pki/server/ca.crt
volumeMounts:
  - name: server-cert
    mountPath: /etc/karmada/pki/server
    readOnly: true
volumes:
  - name: server-cert
    secret:
      secretName: karmada-metrics-adapter-cert

7. karmada-metrics-adapter

command:
  - --client-ca-file=/etc/karmada/pki/server/ca.crt
  - --tls-cert-file=/etc/karmada/pki/server/tls.crt
  - --tls-private-key-file=/etc/karmada/pki/server/tls.key
volumeMounts:
  - name: server-cert
    mountPath: /etc/karmada/pki/server
    readOnly: true
volumes:
  - name: server-cert
    secret:
      secretName: karmada-metrics-adapter-cert

8. karmada-search

command:
  - --etcd-cafile=/etc/karmada/pki/etcd-client/ca.crt
  - --etcd-certfile=/etc/karmada/pki/etcd-client/tls.crt
  - --etcd-keyfile=/etc/karmada/pki/etcd-client/tls.key
  - --tls-cert-file=/etc/karmada/pki/server/tls.crt
  - --tls-private-key-file=/etc/karmada/pki/server/tls.key
volumeMounts:
  - name: server-cert
    mountPath: /etc/karmada/pki/server
    readOnly: true
  - name: etcd-client-cert
    mountPath: /etc/karmada/pki/etcd-client
    readOnly: true
volumes:
  - name: server-cert
    secret:
      secretName: karmada-search-cert
  - name: etcd-client-cert
    secret:
      secretName: karmada-search-etcd-client-cert

9. karmada-webhook

command:
  - --cert-dir=/etc/karmada/pki/server
volumeMounts:
  - name: server-cert
    mountPath: /etc/karmada/pki/server
    readOnly: true
volumes:
  - name: server-cert
    secret:
      secretName: karmada-webhook-cert

10. kube-controller-manager

# --client-ca-file verifies the cert of its client like kubelet and other controller
# --cluster-signing-key-file is used for signing certificates
# --root-ca-file is stored in service account type secret
command:
  - --client-ca-file=/etc/karmada/pki/ca/tls.crt
  - --cluster-signing-cert-file=/etc/karmada/pki/ca/tls.crt
  - --cluster-signing-key-file=/etc/karmada/pki/ca/tls.key
  - --root-ca-file=/etc/karmada/pki/ca/tls.crt
  - --service-account-private-key-file=/etc/karmada/pki/service-account-key-pair/sa.key
volumeMounts:
  - name: ca-cert
    mountPath: /etc/karmada/pki/ca
    readOnly: true
  - name: service-account-key-pair
    mountPath: /etc/karmada/pki/service-account-key-pair
    readOnly: true
volumes:
  - name: ca-cert
    secret:
      secretName: kube-controller-manager-ca-cert
  - name: service-account-key-pair
    secret:
      secretName: kube-controller-manager-service-account-key-pair

11. karmada-interpreter-webhook-example

command:
  - --cert-dir=/etc/karmada/pki/server
volumeMounts:
  - name: server-cert
    mountPath: /etc/karmada/pki/server
    readOnly: true
volumes:
  - name: server-cert
    secret:
      secretName: karmada-interpreter-webhook-example-cert

Legacy issue

We initially hoped to standardize the secret name to a fixed name during this refactoring, using a format like karmada-scheduler-config (following the ${component}-config convention). However, a few users expressed the need to install two Karmada instances within the same namespace, which result in two sets of secrets, preventing us from establishing a fixed secret name.

Resolution: in helm or operator, component name is prefixed with karmada_instance_name, like karmada-xxx-scheduler, so its secret name is defined as ${karmada_instance_name}-${component}-config

cert_rotation_controller of karmada-agent has hard-coded karmada config secret name, so if we rename the secret name of karmada-agent, it would affect the upgrade of karmada-agent.

karmada/pkg/controllers/certificate/cert_rotation_controller.go

Lines 59 to 60 in ce41488

    
           // KarmadaKubeconfigName is the name of the secret containing karmada-agent certificate. 
        
           KarmadaKubeconfigName = "karmada-kubeconfig"

Resolution: this time the karmada-agent is not involved, only the control-plane components is changed.

Related website docs may need updation
- export-kubeconfig in helm

The text was updated successfully, but these errors were encountered:

chaosi-zju · 2025-01-16T02:42:00Z

/help

karmada-bot · 2025-01-16T02:42:02Z

@chaosi-zju:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

seanlaii · 2025-01-18T03:45:36Z

Hi @chaosi-zju , could I work on the karmada-operator update in task one?

chaosi-zju · 2025-01-20T01:40:07Z

Hi @chaosi-zju , could I work on the karmada-operator update in task one?

Of cource, welcome ! the sub task is assigned to you~

husnialhamdani · 2025-01-21T09:07:52Z

Can I work on this?

chaosi-zju · 2025-01-21T11:33:01Z

Can I work on this?

Of cource you can! We very much welcome community partners to participate in.

What sub task you would like to do first?

husnialhamdani · 2025-01-21T11:38:38Z

Can I work on this?

Of cource you can! We very much welcome community partners to participate in.

What sub task you would like to do first?

Let me take the operator one

chaosi-zju · 2025-01-21T12:10:12Z

Let me take the operator one

the operator in task one has already assgined to seanlaii

so, I will assign the operator in task two for you, ok?

husnialhamdani · 2025-01-21T12:14:16Z

Let me take the operator one

the operator in task one has already assgined to seanlaii

so, I will assign the operator in task two for you, ok?

Sure

chaosi-zju · 2025-01-21T12:37:06Z

Sure

ok, thanks!

actually, in this sub task we need to do just two things:

the yaml manifest of each component is listed in operator/pkg/controlplane/ directory, we shall improve the yaml manifest according to above example
the certificate is generated in NewCertTask and stored to secret in NewUploadCertsTask, we will rename the secret and the field according to above new naming convention.

If you have any questions, feel free to ask me, or refer to #5423, best wishes.

chaosi-zju added the good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. label Jan 15, 2025

chaosi-zju mentioned this issue Jan 15, 2025

[Umbrella] standardize the naming of karmada secrets across different installation methods #5363

Closed

chaosi-zju changed the title ~~【Karmada config && certificates】secret and path naming convention~~ [Umbrella] [Karmada config && certificates] secret and path naming convention Jan 15, 2025

karmada-bot added the help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. label Jan 16, 2025

RainbowMango removed the good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. label Jan 16, 2025

chaosi-zju mentioned this issue Jan 16, 2025

standardize the naming of karmada config in karmadactl method #5797

Merged

seanlaii linked a pull request Jan 24, 2025 that will close this issue

Standardize the naming of karmada config in Karmada Operator #6082

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Umbrella] [Karmada config && certificates] secret and path naming convention #6051

[Umbrella] [Karmada config && certificates] secret and path naming convention #6051

chaosi-zju commented Jan 15, 2025 •

edited

Loading

1. karmada-aggregated-apiserver

2. karmada-controller-manager

3. karmada-scheduler

4. karmada-descheduler

5. karmada-metrics-adapter

6. karmada-search

7. karmada-webhook

8. kube-controller-manager

1. karmada-etcd

2. karmada-apiserver

3. karmada-aggregated-apiserver

4. karmada-scheduler

5. karmada-descheduler

6. karmada-scheduler-estimator

7. karmada-metrics-adapter

8. karmada-search

9. karmada-webhook

10. kube-controller-manager

11. karmada-interpreter-webhook-example

chaosi-zju commented Jan 16, 2025

karmada-bot commented Jan 16, 2025

seanlaii commented Jan 18, 2025 •

edited

Loading

chaosi-zju commented Jan 20, 2025 •

edited

Loading

husnialhamdani commented Jan 21, 2025

chaosi-zju commented Jan 21, 2025

husnialhamdani commented Jan 21, 2025

chaosi-zju commented Jan 21, 2025

husnialhamdani commented Jan 21, 2025

chaosi-zju commented Jan 21, 2025

[Umbrella] [Karmada config && certificates] secret and path naming convention #6051

[Umbrella] [Karmada config && certificates] secret and path naming convention #6051

Comments

chaosi-zju commented Jan 15, 2025 • edited Loading

Task one. [Karmada Config] secret and path naming convention

1. karmada-aggregated-apiserver

2. karmada-controller-manager

3. karmada-scheduler

4. karmada-descheduler

5. karmada-metrics-adapter

6. karmada-search

7. karmada-webhook

8. kube-controller-manager

Task two. [Karmada Certificate] secret and path naming convention

1. karmada-etcd

2. karmada-apiserver

3. karmada-aggregated-apiserver

4. karmada-scheduler

5. karmada-descheduler

6. karmada-scheduler-estimator

7. karmada-metrics-adapter

8. karmada-search

9. karmada-webhook

10. kube-controller-manager

11. karmada-interpreter-webhook-example

Legacy issue

chaosi-zju commented Jan 16, 2025

karmada-bot commented Jan 16, 2025

seanlaii commented Jan 18, 2025 • edited Loading

chaosi-zju commented Jan 20, 2025 • edited Loading

husnialhamdani commented Jan 21, 2025

chaosi-zju commented Jan 21, 2025

husnialhamdani commented Jan 21, 2025

chaosi-zju commented Jan 21, 2025

husnialhamdani commented Jan 21, 2025

chaosi-zju commented Jan 21, 2025

chaosi-zju commented Jan 15, 2025 •

edited

Loading

seanlaii commented Jan 18, 2025 •

edited

Loading

chaosi-zju commented Jan 20, 2025 •

edited

Loading