Can't update Cloudflare records after update to 0.15.1 #5035

stefanandres · 2025-01-27T10:54:38Z

What happened:

After updating from 0.15.0 to 0.15.1 updating CF records just fails with "forbidden", even though nothing else changed.
Updating records after downgrading works again.

===== apps/Deployment external-dns/external-dns-cloudflare ======
199c199
<         image: registry.k8s.io/external-dns/external-dns:v0.15.0
---
>         image: registry.k8s.io/external-dns/external-dns:v0.15.1

{"action":"UPDATE","level":"info","msg":"Changing record.","record":"<domain>.org","time":"2025-01-27T10:27:39Z","ttl":1,"type":"CNAME","zone":"<zone>"}
{"action":"UPDATE","level":"error","msg":"failed to update record: forbidden (1002)","record":"<domain>.org","time":"2025-01-27T10:27:39Z","ttl":1,"type":"CNAME","zone":"<zone>"}
{"action":"UPDATE","level":"info","msg":"Changing record.","record":"<domain>.org","time":"2025-01-27T10:27:39Z","ttl":1,"type":"TXT","zone":"<zone>"}
{"action":"UPDATE","level":"error","msg":"failed to update record: forbidden (1002)","record":"<domain>.org","time":"2025-01-27T10:27:40Z","ttl":1,"type":"TXT","zone":"<zone>"}
{"level":"fatal","msg":"Failed to do run once: failed to submit all changes for the following zones: [<zone>]","time":"2025-01-27T10:27:40Z"}

What you expected to happen:

It should update the record

This might be related to changes in:

feat(cloudflare): support cloudflare region by @AndrewCharlesHay in feat(cloudflare): support cloudflare region #4646
fix: update with cloudflare and alibaba cloud by @mloiseleur in fix: update with cloudflare and alibaba cloud #4748

But I can't find anything obvious.

The text was updated successfully, but these errors were encountered:

ivankatliarchuk · 2025-01-27T19:01:01Z

Are you using API token or CF_API_KEY and CF_API_EMAIL?

It could be quite sensitive, but are there any useful debug logs?

lcapka · 2025-01-27T22:24:44Z

We are having the same issue. We are using CF_API_TOKEN. In the output there is not much information:

time="2025-01-27T22:17:03Z" level=info msg="Instantiating new Kubernetes client"
time="2025-01-27T22:17:03Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2025-01-27T22:17:03Z" level=info msg="Created Kubernetes client https://10.43.0.1:443"
time="2025-01-27T22:17:20Z" level=info msg="Changing record." action=UPDATE record=redacted.com ttl=1 type=CNAME zone=b71c92bd6...
time="2025-01-27T22:17:21Z" level=error msg="failed to update record: forbidden (1002)" action=UPDATE record=redacted.com ttl=1 type=CNAME zone=b71c92bd6...
time="2025-01-27T22:17:21Z" level=info msg="Changing record." action=UPDATE record=redacted.com ttl=1 type=TXT zone=b71c92bd6...
time="2025-01-27T22:17:22Z" level=error msg="failed to update record: forbidden (1002)" action=UPDATE record=redacted.com ttl=1 type=TXT zone=b71c92bd6...
time="2025-01-27T22:17:22Z" level=fatal msg="Failed to do run once: failed to submit all changes for the following zones: [b71c92bd6...]"

0x77dev · 2025-01-27T23:04:50Z

+1 here

Also using CF_API_TOKEN, and rolling back to 1.14.5 resolves the issue for me.

ivankatliarchuk · 2025-01-28T01:46:02Z

/help

k8s-ci-robot · 2025-01-28T01:46:05Z

@ivankatliarchuk:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

Why are we solving this issue?
To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
Does this issue have zero to low barrier of entry?
How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ivankatliarchuk · 2025-01-28T02:09:32Z

Share config that is failing. Is it helm or raw manifests?

For those who know how to execute go, worth to try

Clone repo
Provide kube context
Run main.go

export CF_API_TOKEN=xxxxxxxx

just an example with fake source

go run main.go \
    --provider=cloudflare \
    --domain-filter=xxxxxx.org \
    --fqdn-template="xxxxxx.org" \
    --registry=txt \
    --source=fake \
    --log-level=debug

ivankatliarchuk · 2025-01-28T10:06:46Z

As well we have official staging images available for testing https://console.cloud.google.com/gcr/images/k8s-staging-external-dns/GLOBAL/external-dns?pli=1&inv=1&invt=AboC3A

Coud someone give it a try?

stefanandres · 2025-01-29T10:39:19Z

As well we have official staging images available for testing console.cloud.google.com/gcr/images/k8s-staging-external-dns/GLOBAL/external-dns?pli=1&inv=1&invt=AboC3A

Coud someone give it a try?

The image gcr.io/k8s-staging-external-dns/external-dns:v20250128-v0.15.1-142-g9619e6b1 still fails with the same error from above.

It doesn't output anything more related to that change when using log-level=debug.

We are also using the CF_API_TOKEN

  ¦ ¦ ¦ env:
  ¦ ¦ ¦ - name: CF_API_TOKEN
  ¦ ¦ ¦ ¦ valueFrom:
  ¦ ¦ ¦ ¦ ¦ secretKeyRef:
  ¦ ¦ ¦ ¦ ¦ ¦ key: api-token
  ¦ ¦ ¦ ¦ ¦ ¦ name: cloudflare-api-token

AndrewCharlesHay · 2025-01-29T19:44:23Z

@lcapka @0x77dev @stefanandres Can you grant your Cloudflare Token to have Zone Settings: Read access and try again please?

0x77dev · 2025-01-29T20:43:41Z

@AndrewCharlesHay already tried, did not work for me

ivankatliarchuk · 2025-01-29T22:15:05Z

It works for me, but I'm not using a stock helm chart. Could someone share arguments, minimal config I'll try to dig a bit more.

onedr0p · 2025-02-01T17:07:16Z

I think the version upgrade from 0.15.0 to 0.15.1 might be a red herring, instead maybe it's related to the issues described here?
cert-manager/cert-manager#7540

@stefanandres if you downgrade external-dns back to 0.15.0 does the issue remain?

no1here0003 · 2025-02-01T17:13:43Z

I think the version upgrade from 0.15.0 to 0.15.1 might be a red herring, instead maybe it's related to the issues described here? cert-manager/cert-manager#7540

@stefanandres if you downgrade external-dns back to 0.15.0 does the issue remain?

Down grading mine to 1.15.0 fixed it for me.

onedr0p · 2025-02-01T17:20:17Z

Ignore me then 😆 I haven't had any problems using 0.15.0 or 0.15.1 with an API token with the correct scopes.

ivankatliarchuk · 2025-02-02T10:34:31Z

I think the version upgrade from 0.15.0 to 0.15.1 might be a red herring, instead maybe it's related to the issues described here? cert-manager/cert-manager#7540
@stefanandres if you downgrade external-dns back to 0.15.0 does the issue remain?

Down grading mine to 1.15.0 fixed it for me.

Was the helm chart downgraded or only the image?

stefanandres · 2025-02-02T12:02:55Z

I think the version upgrade from 0.15.0 to 0.15.1 might be a red herring, instead maybe it's related to the issues described here? cert-manager/cert-manager#7540
@stefanandres if you downgrade external-dns back to 0.15.0 does the issue remain?

Down grading mine to 1.15.0 fixed it for me.

Was the helm chart downgraded or only the image?

I upgraded both, both downgrading the image only fixes the error.

mikesmitty · 2025-02-02T16:54:11Z

It works for me, but I'm not using a stock helm chart. Could someone share arguments, minimal config I'll try to dig a bit more.

Trying with this staging image gives me this output: gcr.io/k8s-staging-external-dns/external-dns:v20250131-v0.15.1-152-g8eb8ea3a

{"level":"info","msg":"config: {APIServerURL: KubeConfig: RequestTimeout:30s DefaultTargets:[my-tunnel-guid-here.cfargotunnel.com] GlooNamespaces:[gloo-system] SkipperRouteGroupVersion:zalando.org/v1 Sources:[service ingress] Namespace: AnnotationFilter: LabelFilter: IngressClassNames:[] FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false IgnoreNonHostNetworkPods:false IgnoreIngressTLSSpec:false IgnoreIngressRulesSpec:false GatewayNamespace: GatewayLabelFilter: Compatibility: PodSourceDomain: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:cloudflare ProviderCacheTime:0s GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s GoogleZoneVisibility: DomainFilter:[my-domain-here.app] ExcludeDomains:[] RegexDomainFilter: RegexDomainExclusion: ZoneNameFilter:[] ZoneIDFilter:[] TargetNetFilter:[] ExcludeTargetNets:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType: AWSZoneTagFilter:[] AWSAssumeRole: AWSProfiles:[] AWSAssumeRoleExternalID: AWSBatchChangeSize:1000 AWSBatchChangeSizeBytes:32000 AWSBatchChangeSizeValues:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AWSZoneCacheDuration:0s AWSSDServiceCleanup:false AWSSDCreateTag:map[] AWSZoneMatchParent:false AWSDynamoDBRegion: AWSDynamoDBTable:external-dns AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: AzureActiveDirectoryAuthorityHost: AzureZonesCacheDuration:0s CloudflareProxied:true CloudflareDNSRecordsPerPage:100 CloudflareRegionKey: CoreDNSPrefix:/skydns/ AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: AkamaiEdgercPath: AkamaiEdgercSection: OCIConfigFile:/etc/kubernetes/oci.yaml OCICompartmentOCID: OCIAuthInstancePrincipal:false OCIZoneScope:GLOBAL OCIZoneCacheDuration:0s InMemoryZones:[] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://localhost:8081 PDNSServerID:localhost PDNSAPIKey: PDNSSkipTLSVerify:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:sync Registry:txt TXTOwnerID:default TXTPrefix: TXTSuffix: TXTEncryptEnabled:false TXTEncryptAESKey: TXTNewFormatOnly:false Interval:1m0s MinEventSyncInterval:5s Once:false DryRun:false UpdateEvents:false LogFormat:json MetricsAddress::7979 LogLevel:info TXTCacheInterval:0s TXTWildcardReplacement: ExoscaleEndpoint: ExoscaleAPIKey: ExoscaleAPISecret: ExoscaleAPIEnvironment:api ExoscaleAPIZone:ch-gva-2 CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: ResolveServiceLoadBalancerHostname:false RFC2136Host:[] RFC2136Port:0 RFC2136Zone:[] RFC2136Insecure:false RFC2136GSSTSIG:false RFC2136CreatePTR:false RFC2136KerberosRealm: RFC2136KerberosUsername: RFC2136KerberosPassword: RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s RFC2136LoadBalancingStrategy:disabled RFC2136BatchChangeSize:50 RFC2136UseTLS:false RFC2136SkipTLSVerify:false NS1Endpoint: NS1IgnoreSSL:false NS1MinTTLSeconds:0 TransIPAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A AAAA CNAME] ExcludeDNSRecordTypes:[] GoDaddyAPIKey: GoDaddySecretKey: GoDaddyTTL:0 GoDaddyOTE:false OCPRouterName: IBMCloudProxied:false IBMCloudConfigFile:/etc/kubernetes/ibmcloud.json TencentCloudConfigFile:/etc/kubernetes/tencent-cloud.json TencentCloudZoneType: PiholeServer: PiholePassword: PiholeTLSInsecureSkipVerify:false PluralCluster: PluralProvider: WebhookProviderURL:http://localhost:8888 WebhookProviderReadTimeout:5s WebhookProviderWriteTimeout:10s WebhookServer:false TraefikDisableLegacy:false TraefikDisableNew:false NAT64Networks:[]}","time":"2025-02-02T16:32:48Z"}
{"level":"info","msg":"Instantiating new Kubernetes client","time":"2025-02-02T16:32:48Z"}
{"level":"info","msg":"Using inCluster-config based on serviceaccount-token","time":"2025-02-02T16:32:48Z"}
{"level":"info","msg":"Created Kubernetes client https://10.96.0.1:443","time":"2025-02-02T16:32:48Z"}
{"action":"UPDATE","level":"info","msg":"Changing record.","record":"my-domain-here.app","time":"2025-02-02T16:32:50Z","ttl":1,"type":"CNAME","zone":"my-zone-id"}
{"action":"UPDATE","level":"error","msg":"failed to update record when editing region: forbidden (1002)","record":"my-domain-here.app","time":"2025-02-02T16:32:50Z","ttl":1,"type":"CNAME","zone":"my-zone-id"}
{"action":"UPDATE","level":"info","msg":"Changing record.","record":"my-domain-here.app","time":"2025-02-02T16:32:50Z","ttl":1,"type":"TXT","zone":"my-zone-id"}
{"action":"UPDATE","level":"error","msg":"failed to update record when editing region: forbidden (1002)","record":"my-domain-here.app","time":"2025-02-02T16:32:51Z","ttl":1,"type":"TXT","zone":"my-zone-id"}
{"level":"fatal","msg":"Failed to do run once: failed to submit all changes for the following zones: [my-zone-id]","time":"2025-02-02T16:32:51Z"}

Looks like it's specifically failing at the apex record which is a (flattened) CNAME for my CF tunnel. Here's the values.yaml I'm using with the v1.15.1 helm chart:

serviceMonitor:
  enabled: true
env:
  - name: CF_API_TOKEN
    valueFrom:
      secretKeyRef:
        name: external-dns-cloudflare
        key: CF_API_TOKEN

logFormat: json

policy: sync

domainFilters:
  - my-domain-here.app

provider: cloudflare

extraArgs:
  - --cloudflare-proxied
  - --default-targets=my-tunnel-guid-here.cfargotunnel.com

And the associated TXT record contents: "heritage=external-dns,external-dns/owner=default,external-dns/resource=service/external-dns/my-domain-here-app"

AndrewCharlesHay · 2025-02-02T17:19:19Z

I think the version upgrade from 0.15.0 to 0.15.1 might be a red herring, instead maybe it's related to the issues described here? cert-manager/cert-manager#7540
@stefanandres if you downgrade external-dns back to 0.15.0 does the issue remain?

Down grading mine to 1.15.0 fixed it for me.

Was the helm chart downgraded or only the image?

I upgraded both, both downgrading the image only fixes the error.

@stefanandres Are you saying you had to downgrade the image and the Helm chart for it to work?

shawnhwei · 2025-02-03T02:47:15Z

I also encountered this issue on updating the apex record. In my case it is an A record.

Downgrading to 1.15.0 fixed it.

stefanandres added the kind/bug Categorizes issue or PR as related to a bug. label Jan 27, 2025

k8s-ci-robot added the help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. label Jan 28, 2025

nea0d mentioned this issue Jan 28, 2025

Revert "fix(helm): update external-dns ( 1.15.0 → 1.15.1 )" nea0d/homelab-ops#1076

Merged

AndrewCharlesHay mentioned this issue Jan 29, 2025

fix(cloudflare): add more descriptive log message #5047

Merged

2 tasks

bhuism added a commit to bhuism/home-ops that referenced this issue Feb 1, 2025

override image for kubernetes-sigs/external-dns#5035

0366dde

AndrewCharlesHay mentioned this issue Feb 1, 2025

test(cloudflare): add scenarios on submitChanges #5054

Open

2 tasks

KyteProject added a commit to KyteProject/gitops-homelab that referenced this issue Feb 2, 2025

downgrade kubernetes-sigs/external-dns#5035

c5ee317

rtrox mentioned this issue Feb 2, 2025

External-DNS failing on Apex CNAME update rtrox/home-ops#224

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't update Cloudflare records after update to 0.15.1 #5035

Can't update Cloudflare records after update to 0.15.1 #5035

stefanandres commented Jan 27, 2025 •

edited

Loading

ivankatliarchuk commented Jan 27, 2025

lcapka commented Jan 27, 2025

0x77dev commented Jan 27, 2025

ivankatliarchuk commented Jan 28, 2025

k8s-ci-robot commented Jan 28, 2025

ivankatliarchuk commented Jan 28, 2025

ivankatliarchuk commented Jan 28, 2025 •

edited

Loading

stefanandres commented Jan 29, 2025

AndrewCharlesHay commented Jan 29, 2025

0x77dev commented Jan 29, 2025 •

edited

Loading

ivankatliarchuk commented Jan 29, 2025

onedr0p commented Feb 1, 2025 •

edited

Loading

no1here0003 commented Feb 1, 2025

onedr0p commented Feb 1, 2025 •

edited

Loading

ivankatliarchuk commented Feb 2, 2025

stefanandres commented Feb 2, 2025

mikesmitty commented Feb 2, 2025

AndrewCharlesHay commented Feb 2, 2025

shawnhwei commented Feb 3, 2025

Can't update Cloudflare records after update to 0.15.1 #5035

Can't update Cloudflare records after update to 0.15.1 #5035

Comments

stefanandres commented Jan 27, 2025 • edited Loading

ivankatliarchuk commented Jan 27, 2025

lcapka commented Jan 27, 2025

0x77dev commented Jan 27, 2025

ivankatliarchuk commented Jan 28, 2025

k8s-ci-robot commented Jan 28, 2025

Guidelines

ivankatliarchuk commented Jan 28, 2025

ivankatliarchuk commented Jan 28, 2025 • edited Loading

stefanandres commented Jan 29, 2025

AndrewCharlesHay commented Jan 29, 2025

0x77dev commented Jan 29, 2025 • edited Loading

ivankatliarchuk commented Jan 29, 2025

onedr0p commented Feb 1, 2025 • edited Loading

no1here0003 commented Feb 1, 2025

onedr0p commented Feb 1, 2025 • edited Loading

ivankatliarchuk commented Feb 2, 2025

stefanandres commented Feb 2, 2025

mikesmitty commented Feb 2, 2025

AndrewCharlesHay commented Feb 2, 2025

shawnhwei commented Feb 3, 2025

stefanandres commented Jan 27, 2025 •

edited

Loading

ivankatliarchuk commented Jan 28, 2025 •

edited

Loading

0x77dev commented Jan 29, 2025 •

edited

Loading

onedr0p commented Feb 1, 2025 •

edited

Loading

onedr0p commented Feb 1, 2025 •

edited

Loading