Concurrent CREATE and DELETE Operations Cause IllegalStateException

[BenjaminSSL]

Describe the bug

When performing concurrent parallel HTTP requests involving CREATE and DELETE operations on namespace, table, and view entities, an IllegalStateException is encountered.

Polaris logs the following error message:

ERROR [org.apa.pol.ser.exc.IcebergExceptionMapper] [,POLARIS] [,,,] (executor-thread-83) Unhandled exception returning INTERNAL_SERVER_ERROR: java.lang.IllegalStateException: Unable to resolve sibling entities to validate location - PATH_COULD_NOT_BE_FULLY_RESOLVED
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalog.validateNoLocationOverlap(IcebergCatalog.java:1121)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalog.createNamespaceInternal(IcebergCatalog.java:524)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalog.createNamespace(IcebergCatalog.java:500)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogHandlerWrapper.createNamespace(IcebergCatalogHandlerWrapper.java:485)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.lambda$createNamespace$0(IcebergCatalogAdapter.java:198)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.withCatalog(IcebergCatalogAdapter.java:161)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.createNamespace(IcebergCatalogAdapter.java:195)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_ClientProxy.createNamespace(Unknown Source)
	at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi.createNamespace(IcebergRestCatalogApi.java:166)
	at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.createNamespace$$superforward(Unknown Source)
	at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass$$function$$2.apply(Unknown Source)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
	at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor.timedMethod(MicrometerTimedInterceptor.java:79)
	at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
	at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
	at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.createNamespace(Unknown Source)
	at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi$quarkusrestinvoker$createNamespace_6b5544448686388164ce7bf7f7e9768cc6a8f333.invoke(Unknown Source)
	at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
	at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:141)
	at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
	at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:638)
	at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2675)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2654)
	at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1627)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1594)
	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Http reposonse body:

{
   "error":
   {
      "message":"Unable to resolve sibling entities to validate location - PATH_COULD_NOT_BE_FULLY_RESOLVED",
      "type":"IllegalStateException",
      "code":500
   }
}

To Reproduce

Setup a test with a minimum of two threads:

• Ensure that the top-level catalog and a namespace are created before running the test. And make sure the principal has the required permissions to create and delete the entities.
• The $n$ amount of threads should CREATE and DELETE the aforementioned entities in parallel and repeatedly. The thread should first create an entity and then immediately delete it.

Run the test for at a least 10 seconds.

During the test, the CREATE operation will occasionally fail with a 500 error. However, it does not fail every time or follow a specific pattern. The DELETE operation seems to work fine.

These are the endpoints I used for the test, with the host and resource being http://localhost:8181/api/catalog/v1:

• Create Namespace: POST /{prefix}/namespaces
• Delete Namespace: DELETE /{prefix}/namespaces/{namespace}
• Create Table: POST /{prefix}/namespaces/{namespace}/tables
• Delete Table: DELETE /{prefix}/namespaces/{namespace}/tables/{table}
• Create View: POST /{prefix}/namespaces/{namespace}/views
• Delete View: DELETE /{prefix}/namespaces/{namespace}/views/{view}

Actual Behavior

The CREATE operation occasionally fails with a 500 error with a IllegalStateException in the logs.

Expected Behavior

I would expect the CREATE operation to succeed even if there are concurrent operations happening on the top-level entity.

Additional context

Test environment used the in-memory store.

System information

OS: MacOS Sonoma v15.4.1
Polaris Catalog Version: 1.0.0-incubating-SNAPSHOT
Object Storage: FILE