Implement federation to HadoopCatalog #1466

eric-maynard · 2025-04-25T20:43:20Z

Polaris recently added support for catalog federation in #1305, which allows a Polaris instance to federate requests against a given catalog to another Iceberg REST catalog This is useful for deployments with multiple Iceberg REST catalogs, but we need to introduce federation to other catalogs in order to support a more diverge range of deployments.

This implements federation to a Hadoop catalog via the new HADOOP type of ConnectionConfigInfo.

XJDKC

LGTM! Left few minor comments!

XJDKC · 2025-04-28T21:40:21Z

...main/java/org/apache/polaris/core/connection/iceberg/IcebergRestConnectionConfigInfoDpo.java

Should we name the package as rest? Also, IcebergCatalogProperties is also implemented by HadoopConnectionConfigInfoDpo, it's used by any Iceberg related connection configs, maybe we can leave it in the current package (iceberg).

My concern with that alternative is that delegation to e.g. Unity Catalog will also be over REST. So it's more like iceberg-rest, I called it iceberg but rest is not ideal either

XJDKC · 2025-04-28T21:42:06Z

...e/src/main/java/org/apache/polaris/core/connection/hadoop/HadoopConnectionConfigInfoDpo.java

+
+  public HadoopConnectionConfigInfoDpo(
+      @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+      @JsonProperty(value = "authenticationParameters", required = true) @Nonnull


QQ: What type of authentication method does HadoopCatalog typically use? I tried looking for some docs but couldn't find much.

Should we add a no-op authentication method?

We essentially just forward whatever properties into the catalog itself, which I think functions as the auth mechanism

If we extend or refactor HadoopCatalog then the auth could be more properly injectable; unfortunately right now it appears to just hard-code using HadoopFileIO as its fileIOImpl and also independently instantiates a org.apache.hadoop.fs.FileSystem, which relies on the org.apache.hadoop.conf.Configuration which appears to be null in this PR since we call the no-arg constructor of HadoopCatalog instead of the 2-arg constructor that takes a Configuration.

From what I can tell this means it'll just fallback to Hadoop's "default" Configuration loading where it would normally load things from e.g. hdfs-site.xml.

If we're going to really support running this federation mode properly, we should at least add a TODO to expand a HadoopCatalog-specific AuthenticationParameters subclass. One way would be to express similar delegation settings as StorageConfigInfo, or alternatively, HadoopCatalog is a special-case where the real storage credential is the catalog credential, so maybe we could just have something that "points" at the Catalog's StorageConfigInfo as the effective AuthenticationParameters.

I actually think supporting HadoopCatalog as a sort of "dumb" federation mode that just uses the properties you give it (and loads from hdfs-site.xml by default) is fine... if we wanted to add another module that brings in a POLARIS_HADOOP mode or something in the future that could be nice, too.

In practice, the way you would probably deploy this is by creating a second Polaris instance dedicated to wrapping the Hadoop catalog, where you can specify env variables and a hdfs-site.xml

it appears to just hard-code using HadoopFileIO

I think the properties we pass into initialize should take effect right? That would be the fallback if io-impl is not specified.

Should we restrict HadoopCatalog backed by cloud-providers only (like S3) and not HDFS clusters for phase 1, as opening up HDFS will add more cases like HDFS backed by kreberos, its not just hdfs-site.xml that Configuration contains, thoughts ?

I think the concept of federation means that Polaris itself is completely agnostic to the configuration of the remote catalog. Otherwise, when we delegate to IRCs, should we give admins some kind of knobs about which IRCs we can and cannot delegate to? Should we enforce certain settings for those IRCs?

In my view, it makes sense to just create a HadoopCatalog and pass along whatever properties the user provides when the user has configured Polaris to delegate to a Hadoop catalog.

In my view, it makes sense to just create a HadoopCatalog and pass along whatever properties the user provides when the user has configured Polaris to delegate to a Hadoop catalog.

I agree in principal for this, I was mostly coming from introducing / opening up for some use-case that polaris doesn't support in the moment itself, hence was my consideration, but totally fine if we move ahead.

I think the concept of federation means that Polaris itself is completely agnostic to the configuration of the remote catalog.

I am not sure if I fully agree with this though, my reasoning is following because Polaris can't just be a pass through, the identity that the federated catalog sees is much more powerful so Polaris should IMHO need to some down scoping or never the less make a way to provide the federated catalog the requester's identity, which in Kreberos world is way different and have to go hand in had with downstream catalog so some co-ordination is required. mostly talking of this : #1026 (comment)
May be I am making it more complicated, than its supposed to be at-least this is chain of thoughts that come to my mind, on thinking about this.

Polaris should IMHO need to some down scoping or never the less make a way to provide the federated catalog the requester's identity

I agree with you that this should be a goal. I would point out however that the same would be true for even the simplest case -- Polaris -> Polaris federation. This problem isn't unique to federation to Hadoop. It is, however, a very hard problem. What if the two systems don't have the same concept of identity, or they don't trust the same IdP?

There are cases where identity federation may unlock new solutions for us, where the two catalogs can federate to the same trusted IdP. But there are also going to be cases where this is not so.

The way Polaris -> IRC federation currently works, and the way that I think federation will have to continue working for the time being, is that Polaris simply communicates with the remote catalog via the connection config. This can be set on the catalog level today, but in the future we will be able to set this on the namespace or the table level, similar to a storage config.

Right, identity-passthrough/delegation is important long-term, but has a lot of complicated prerequisites and other dependencies on how we express federated identities and user-isolated caching of metadata or storage credentials, so in the meantime we should just clearly document what the "powerful connectionconfig" model implies.

XJDKC · 2025-04-28T21:46:05Z

polaris-core/src/main/java/org/apache/polaris/core/connection/ConnectionType.java

@@ -32,6 +32,7 @@
 public enum ConnectionType {
  NULL_TYPE(0),
  ICEBERG_REST(1),
+  HADOOP(2),


Do we plan to federate to Hadoop through a non-Iceberg Catalog?
Just curious how generic tables would work with Catalog Federation in that case.

This is for Hadoop's IcebergCatalog, but good point. For UC, we might have a separate connection for UC's IRC endpoints compared to the non-IRC ones and they might be used for different table types

XJDKC · 2025-04-28T21:50:28Z

spec/polaris-management-service.yml

+      allOf:
+        - $ref: '#/components/schemas/ConnectionConfigInfo'
+      properties:
+        warehouse:


Just want to flag that, for IcebergRestConnectionConfigInfo, we use remoteCatalogName, and it's passed as the warehouse property when initializing the catalog client. Using warehouse make more sense for HadoopConnectionConfigInfo.

dennishuo

This seems like a good starting point for simple HadoopCatalog use cases where the storage credentials can be broadly shared in the Polaris server's runtime environment, but we'll probably want to expand on the authentication parameters in a followup to provide more configurability and guardrails around more explicitly specifying storage roles/credentials.

In the meantime, we'll need the ability to specify which connection config types to allow at the service-level so that different connection types can be independently enabled/disabled.

dennishuo · 2025-04-29T04:27:22Z

...e/src/main/java/org/apache/polaris/core/connection/hadoop/HadoopConnectionConfigInfoDpo.java

+
+  public HadoopConnectionConfigInfoDpo(
+      @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+      @JsonProperty(value = "authenticationParameters", required = true) @Nonnull


If we extend or refactor HadoopCatalog then the auth could be more properly injectable; unfortunately right now it appears to just hard-code using HadoopFileIO as its fileIOImpl and also independently instantiates a org.apache.hadoop.fs.FileSystem, which relies on the org.apache.hadoop.conf.Configuration which appears to be null in this PR since we call the no-arg constructor of HadoopCatalog instead of the 2-arg constructor that takes a Configuration.

From what I can tell this means it'll just fallback to Hadoop's "default" Configuration loading where it would normally load things from e.g. hdfs-site.xml.

If we're going to really support running this federation mode properly, we should at least add a TODO to expand a HadoopCatalog-specific AuthenticationParameters subclass. One way would be to express similar delegation settings as StorageConfigInfo, or alternatively, HadoopCatalog is a special-case where the real storage credential is the catalog credential, so maybe we could just have something that "points" at the Catalog's StorageConfigInfo as the effective AuthenticationParameters.

dennishuo · 2025-04-29T04:31:16Z

spec/polaris-management-service.yml

@@ -862,6 +862,7 @@ components:
          type: string
          enum:
            - ICEBERG_REST
+            - HADOOP


Since the HadoopCatalog is highly unsafe if the set of people who have CATALOG_CREATE privileges isn't exactly equal to the set of people with superuser privileges to run the entire Catalog service itself, before we introduce this one even as an option we should probably add a ALLOWED_CATALOG_CONNECTION_TYPES or SUPPORTED_CATALOG_CONNECTION_TYPES analogous to SUPPORTED_CATALOG_STORAGE_TYPES.

Great idea, added

integration-tests/src/main/java/org/apache/polaris/service/it/env/ManagementApi.java

dennishuo · 2025-05-05T17:31:06Z

...e/src/main/java/org/apache/polaris/core/connection/hadoop/HadoopConnectionConfigInfoDpo.java

+
+  public HadoopConnectionConfigInfoDpo(
+      @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+      @JsonProperty(value = "authenticationParameters", required = true) @Nonnull


Right, identity-passthrough/delegation is important long-term, but has a lot of complicated prerequisites and other dependencies on how we express federated identities and user-isolated caching of metadata or storage credentials, so in the meantime we should just clearly document what the "powerful connectionconfig" model implies.

dennishuo · 2025-05-05T18:06:28Z

polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java

+      PolarisConfiguration.<List<String>>builder()
+          .key("SUPPORTED_CATALOG_CONNECTION_TYPES")
+          .description("The list of supported catalog connection types for federation")
+          .defaultValue(List.of(ConnectionType.ICEBERG_REST.name(), ConnectionType.HADOOP.name()))


Is it possible to make this default an empty list, so that whoever wants to enable the federation feature has to explicitly opt-in to the connection types they want?

And/or set it to an empty list in application.properties. Or if empty list doesn't work for some reason, it seems ICEBERG_REST-only is a reasonable initial default since it's the minimal-dependency one long-term with a more well-defined security boundary.

It's possible and I agree with the sentiment, but I would point to the default inclusion of FILE as a supported storage type as an important precedent here. Following this, it seems like what we should do is call out SUPPORTED_CATALOG_CONNECTION_TYPES in configuring-polaris-for-production. WDYT?

Maybe one way we can stay consistent with this precedent but also be really safe-by-default (beyond what we do for storage types even) is to add a second flag gating off support for federation altogether... something like CATALOG_FEDERATION_ENABLED?

Yes, we already have a feature gate for this: "ENABLE_CATALOG_FEDERATION"

However, at the moment enabling that only risks "broken experimental functionality" but not the security of the server, so the main concern would be if any service providers happen to already be setting ENABLE_CATALOG_FEDERATION=true but aren't paying attention to the addition of new federation types.

Perhaps we can at least include the new SUPPORTED_CATALOG_CONNECTION_TYPES in application.properties to explicitly list the two types, and maybe with a commented-out version that only includes ICEBERG_REST as well, with some comments about how the reader should choose the configuration. Or make the default one only [ICEBERG_REST] with a commented-out version that includes both [ICEBERG_REST, HADOOP].

Doesn't storage type FILE also risk the security of the server? It seems like almost the same situation there.

For now I'll update both the FeatureConfiguration default and the application.properties entry to only include ICEBERG_REST; when we add other federation types we can make a case-by-case evaluation of whether we want to enable those types by default.

dennishuo

LGTM

eric-maynard added 5 commits April 25, 2025 13:02

wip

9fb3c36

quarkus fixes

2608979

autolint

7d995ae

hadoop impl

e88b968

autolint

90cc006

github-project-automation bot added this to Basic Kanban Board Apr 25, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Apr 25, 2025

eric-maynard changed the title ~~Add polaris-wrapper config; implement polaris-wrapper.mode=hadoop~~ Add polaris-wrapper config; implement polaris.wrapper.mode=hadoop Apr 25, 2025

eric-maynard added 2 commits April 25, 2025 23:10

Refactors

cefc3ed

refactored

497eb0e

eric-maynard changed the title ~~Add polaris-wrapper config; implement polaris.wrapper.mode=hadoop~~ Implement federation to HadoopCatalog Apr 26, 2025

autolint

7ea31bc

eric-maynard marked this pull request as ready for review April 28, 2025 19:03

eric-maynard requested review from adutra, ashvina, dennishuo, dimas-b, jackye1995, jbonofre, vvcephei, collado-mike, snazy, RussellSpitzer, takidau, MonkeyCanCode, flyrain and ebyhr as code owners April 28, 2025 19:03

XJDKC approved these changes Apr 28, 2025

View reviewed changes

dennishuo requested changes Apr 29, 2025

View reviewed changes

add config

3dc4f7b

eric-maynard added 3 commits April 29, 2025 01:16

autolint

10ed9c7

resolve conflict

5746b99

stable

76fda06

eric-maynard requested a review from dennishuo April 29, 2025 08:33

adnanhemani reviewed Apr 30, 2025

View reviewed changes

integration-tests/src/main/java/org/apache/polaris/service/it/env/ManagementApi.java Outdated Show resolved Hide resolved

Remove breakpoint anchor

5067e8e

dennishuo reviewed May 5, 2025

View reviewed changes

add line to application.properties

a7854d7

eric-maynard requested review from HonahX and pingtimeout as code owners May 6, 2025 17:13

eric-maynard added 2 commits May 6, 2025 10:15

yank HADOOP

84b3eb3

pull main

66e6220

dennishuo previously approved these changes May 9, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board May 9, 2025

eric-maynard dismissed dennishuo’s stale review via dcd309f May 9, 2025 21:47

eric-maynard added 2 commits May 9, 2025 14:47

pull main

5924230

autolint

dcd309f

eric-maynard enabled auto-merge (squash) May 9, 2025 21:48

dennishuo approved these changes May 9, 2025

View reviewed changes

eric-maynard merged commit 24683d8 into apache:main May 9, 2025
6 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board May 9, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement federation to HadoopCatalog #1466

Implement federation to HadoopCatalog #1466

eric-maynard commented Apr 25, 2025 •

edited

Loading

XJDKC left a comment

XJDKC Apr 28, 2025

eric-maynard Apr 29, 2025

XJDKC Apr 28, 2025

eric-maynard Apr 29, 2025

dennishuo Apr 29, 2025

eric-maynard Apr 29, 2025 •

edited

Loading

eric-maynard Apr 29, 2025 •

edited

Loading

singhpk234 May 4, 2025

eric-maynard May 4, 2025

singhpk234 May 4, 2025

eric-maynard May 4, 2025 •

edited

Loading

dennishuo May 5, 2025

XJDKC Apr 28, 2025

eric-maynard Apr 29, 2025

XJDKC Apr 28, 2025

dennishuo left a comment

dennishuo Apr 29, 2025

dennishuo Apr 29, 2025

eric-maynard Apr 29, 2025

dennishuo May 5, 2025

dennishuo May 5, 2025

eric-maynard May 5, 2025

eric-maynard May 5, 2025

dennishuo May 6, 2025

eric-maynard May 6, 2025 •

edited

Loading

eric-maynard May 6, 2025

dennishuo left a comment

Implement federation to HadoopCatalog #1466

Implement federation to HadoopCatalog #1466

Conversation

eric-maynard commented Apr 25, 2025 • edited Loading

XJDKC left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eric-maynard Apr 29, 2025 • edited Loading

Choose a reason for hiding this comment

eric-maynard Apr 29, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eric-maynard May 4, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dennishuo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eric-maynard May 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dennishuo left a comment

Choose a reason for hiding this comment

eric-maynard commented Apr 25, 2025 •

edited

Loading

eric-maynard Apr 29, 2025 •

edited

Loading

eric-maynard Apr 29, 2025 •

edited

Loading

eric-maynard May 4, 2025 •

edited

Loading

eric-maynard May 6, 2025 •

edited

Loading