Querying S3 Tables on Kubernetes with Trino Using IAM-Based Authentication

[soumilshah1995]

Hello,

We would like to seek assistance regarding an issue we're encountering with Trino.

We are using the following Trino Docker image:

image:
  repository: XX.dkr.ecr.us-east-1.amazonaws.com/dockerhub/trinodb/trino
  tag: "475"

Our Iceberg catalog configuration is as follows:

iceberg_preprod: |
  connector.name=iceberg
  iceberg.catalog.type=rest
  iceberg.rest-catalog.uri=https://glue.us-east-1.amazonaws.com/iceberg
  iceberg.rest-catalog.warehouse=<ACCOUNT>:s3tablescatalog/<BUCKET>
  iceberg.rest-catalog.view-endpoints-enabled=false
  iceberg.rest-catalog.sigv4-enabled=true
  iceberg.rest-catalog.signing-name=glue
  iceberg.writer-sort-buffer-size=1MB
  iceberg.allowed-extra-properties=write.metadata.delete-after-commit.enabled,write.metadata.previous-versions-max
  fs.hadoop.enabled=false
  fs.native-s3.enabled=true
  s3.region=us-east-1

We are using IAM role–based authentication. However, during deployment or when attempting queries, we are encountering the following error:

                  Use credentials provided by the REST backend for file system access
2025-04-29T11:49:17.182Z	INFO	main	Bootstrap	iceberg.rest-catalog.view-endpoints-enabled                    true              false                                                                               Enable view endpoints
2025-04-29T11:49:17.182Z	INFO	main	Bootstrap	iceberg.rest-catalog.warehouse                                 ----              636347950933:s3tablescatalog/zmp-preprod-use1-segments-export                       The warehouse location/identifier to use with the REST catalog server
2025-04-29T11:49:17.182Z	INFO	main	Bootstrap	iceberg.rest-catalog.signing-name                              execute-api       glue                                                                                AWS SigV4 signing service name
2025-04-29T11:49:18.440Z	WARN	main	com.google.inject.internal.util.LineNumbers	Failed loading line numbers. ASM is probably out of date. Further failures won't be logged.
java.lang.IllegalArgumentException: Unsupported class file major version 67
	at com.google.inject.internal.asm.$ClassReader.<init>(ClassReader.java:199)
	at com.google.inject.internal.asm.$ClassReader.<init>(ClassReader.java:180)
	at com.google.inject.internal.asm.$ClassReader.<init>(ClassReader.java:166)
	at com.google.inject.internal.asm.$ClassReader.<init>(ClassReader.java:287)
	at com.google.inject.internal.util.LineNumbers.<init>(LineNumbers.java:74)
	at com.google.inject.internal.util.StackTraceElements$1.load(StackTraceElements.java:48)
	at com.google.inject.internal.util.StackTraceElements$1.load(StackTraceElements.java:44)
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3551)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2302)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2177)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2068)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3986)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4007)
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4946)
	at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4953)
	at com.google.inject.internal.util.StackTraceElements.forMember(StackTraceElements.java:67)
	at com.google.inject.internal.SourceFormatter.formatMember(SourceFormatter.java:91)
	at com.google.inject.internal.SourceFormatter.formatInjectionPoint(SourceFormatter.java:97)
	at com.google.inject.internal.SourceFormatter.format(SourceFormatter.java:49)
	at com.google.inject.internal.GenericErrorDetail.formatDetail(GenericErrorDetail.java:26)
	at com.google.inject.spi.ErrorDetail.format(ErrorDetail.java:64)
	at com.google.inject.internal.Messages.formatMessages(Messages.java:90)
	at com.google.inject.CreationException.getMessage(CreationException.java:50)
	at java.base/java.lang.Throwable.getLocalizedMessage(Throwable.java:424)
	at java.base/java.lang.Throwable.toString(Throwable.java:523)
	at java.base/java.lang.Throwable.<init>(Throwable.java:337)
	at java.base/java.lang.Exception.<init>(Exception.java:103)
	at java.base/java.util.concurrent.ExecutionException.<init>(ExecutionException.java:90)
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:124)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:193)
	at io.airlift.concurrent.MoreFutures.getFutureValue(MoreFutures.java:193)
	at io.airlift.concurrent.MoreFutures.getFutureValue(MoreFutures.java:176)
	at io.airlift.concurrent.MoreFutures.getDone(MoreFutures.java:280)
	at io.trino.util.Executors.executeUntilFailure(Executors.java:50)
	at io.trino.connector.StaticCatalogManager.loadInitialCatalogs(StaticCatalogManager.java:155)
	at io.trino.server.Server.doStart(Server.java:152)
	at io.trino.server.Server.lambda$start$0(Server.java:90)
	at io.trino.$gen.Trino_475____20250429_114902_1.run(Unknown Source)
	at io.trino.server.Server.start(Server.java:90)
	at io.trino.server.TrinoServer.main(TrinoServer.java:37)


2025-04-29T11:49:18.521Z	ERROR	main	io.trino.server.Server	Unable to create injector, see the following errors:

1) [Guice/ErrorInjectingConstructor]: NullPointerException: s3.aws-access-key is null
  at SigV4AwsProperties.<init>(Unknown Source)
  while locating SigV4AwsProperties
  at IcebergRestCatalogModule.lambda$setup$1(IcebergRestCatalogModule.java:48)
      \_ installed by: IcebergCatalogModule -> ConditionalModule -> IcebergRestCatalogModule -> ConfigurationAwareModule$1 -> ConditionalModule -> IcebergRestCatalogModule$$Lambda/0x0000000802195000
  at TrinoIcebergRestCatalogFactory.<init>(Unknown Source)
      \_ for 5th parameter awsProperties
  while locating TrinoIcebergRestCatalogFactory
  at java.base/DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
      \_ installed by: IcebergCatalogModule -> ConditionalModule -> IcebergRestCatalogModule
  at IcebergMetadataFactory.<init>(Unknown Source)
      \_ for 4th parameter catalogFactory
  at IcebergModule.configure(IcebergModule.java:102)
  at IcebergTransactionManager.<init>(Unknown Source)
      \_ for 1st parameter metadataFactory
  at IcebergModule.configure(IcebergModule.java:75)
  at IcebergSplitManager.<init>(Unknown Source)
      \_ for 1st parameter transactionManager
  while locating IcebergSplitManager
  at IcebergModule.configure(IcebergModule.java:85)
  while locating ConnectorSplitManager annotated with interface ForClassLoaderSafe

Learn more:
  https://github.com/google/guice/wiki/ERROR_INJECTING_CONSTRUCTOR
Caused by: NullPointerException: s3.aws-access-key is null
	at java.base/Objects.requireNonNull(Objects.java:246)
	at SigV4AwsProperties.<init>(SigV4AwsProperties.java:80)
	at SigV4AwsProperties$$FastClassByGuice$$312b1a4.GUICE$TRAMPOLINE(<generated>)
	at SigV4AwsProperties$$FastClassByGuice$$312b1a4.apply(<generated>)
	at DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82)
	at ConstructorInjector.provision(ConstructorInjector.java:114)
	at ConstructorInjector.access$000(ConstructorInjector.java:33)
	at ConstructorInjector$1.call(ConstructorInjector.java:98)
	at ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:109)
	at LifeCycleModule.provision(LifeCycleModule.java:53)
	at ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:117)
	at ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
	at ConstructorInjector.construct(ConstructorInjector.java:93)
	at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
	at FactoryProxy.get(FactoryProxy.java:60)
	at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at SingletonScope$1.get(SingletonScope.java:169)
	at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
	at SingleParameterInjector.inject(SingleParameterInjector.java:40)
	at SingleParameterInjector.getAll(SingleParameterInjector.java:60)
	at ConstructorInjector.provision(ConstructorInjector.java:113)
	at ConstructorInjector.access$000(ConstructorInjector.java:33)
	at ConstructorInjector$1.call(ConstructorInjector.java:98)
	at ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:109)
	at LifeCycleModule.provision(LifeCycleModule.java:53)
	at ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:117)
	at ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
	at ConstructorInjector.construct(ConstructorInjector.java:93)
	at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
	at FactoryProxy.get(FactoryProxy.java:60)
	at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at SingletonScope$1.get(SingletonScope.java:169)
	at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
	at SingleParameterInjector.inject(SingleParameterInjector.java:40)
	at SingleParameterInjector.getAll(SingleParameterInjector.java:60)
	at ConstructorInjector.provision(ConstructorInjector.java:113)
	at ConstructorInjector.access$000(ConstructorInjector.java:33)
	at ConstructorInjector$1.call(ConstructorInjector.java:98)
	at ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:109)
	at LifeCycleModule.provision(LifeCycleModule.java:53)
	at ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:117)
	at ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
	at ConstructorInjector.construct(ConstructorInjector.java:93)
	at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
	at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at SingletonScope$1.get(SingletonScope.java:169)
	at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
	at SingleParameterInjector.inject(SingleParameterInjector.java:40)
	at SingleParameterInjector.getAll(SingleParameterInjector.java:60)
	at ConstructorInjector.provision(ConstructorInjector.java:113)
	at ConstructorInjector.access$000(ConstructorInjector.java:33)
	at ConstructorInjector$1.call(ConstructorInjector.java:98)
	at ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:109)
	at LifeCycleModule.provision(LifeCycleModule.java:53)
	at ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:117)
	at ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
	at ConstructorInjector.construct(ConstructorInjector.java:93)
	at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
	at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at SingletonScope$1.get(SingletonScope.java:169)
	at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
	at SingleParameterInjector.inject(SingleParameterInjector.java:40)
	at SingleParameterInjector.getAll(SingleParameterInjector.java:60)
	at ConstructorInjector.provision(ConstructorInjector.java:113)
	at ConstructorInjector.access$000(ConstructorInjector.java:33)
	at ConstructorInjector$1.call(ConstructorInjector.java:98)
	at ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:109)
	at LifeCycleModule.provision(LifeCycleModule.java:53)
	at ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:117)
	at ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
	at ConstructorInjector.construct(ConstructorInjector.java:93)
	at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
	at FactoryProxy.get(FactoryProxy.java:60)
	at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at SingletonScope$1.get(SingletonScope.java:169)
	at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
	at InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213)
	at InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186)
	at InternalInjectorCreator.build(InternalInjectorCreator.java:113)
	at Guice.createInjector(Guice.java:87)
	at Bootstrap.initialize(Bootstrap.java:335)
	at IcebergConnectorFactory.createConnector(IcebergConnectorFactory.java:102)
	at IcebergConnectorFactory.create(IcebergConnectorFactory.java:64)
	at DefaultCatalogFactory.createConnector(DefaultCatalogFactory.java:208)
	at DefaultCatalogFactory.createCatalog(DefaultCatalogFactory.java:124)
	at LazyCatalogFactory.createCatalog(LazyCatalogFactory.java:45)
	at StaticCatalogManager.lambda$loadInitialCatalogs$1(StaticCatalogManager.java:161)
	at java.base/FutureTask.run(FutureTask.java:328)
	at java.base/Executors$RunnableAdapter.call(Executors.java:545)
	at java.base/FutureTask.run(FutureTask.java:328)
	at DirectExecutor.execute(DirectExecutor.java:30)
	at java.base/ExecutorCompletionService.submit(ExecutorCompletionService.java:186)
	at Executors.executeUntilFailure(Executors.java:46)
	at StaticCatalogManager.loadInitialCatalogs(StaticCatalogManager.java:155)
	at Server.doStart(Server.java:152)
	at Server.lambda$start$0(Server.java:90)
	at io.trino.$gen.Trino_475____20250429_114902_1.run(Unknown Source)
	at Server.start(Server.java:90)
	at TrinoServer.main(TrinoServer.java:37)

2) [Guice/ErrorInjectingConstructor]: NullPointerException: s3.aws-access-key is null
  at SigV4AwsProperties.<init>(Unknown Source)
  while locating SigV4AwsProperties
  at IcebergRestCatalogModule.lambda$setup$1(IcebergRestCatalogModule.java:48)
      \_ installed by: IcebergCatalogModule -> ConditionalModule -> IcebergRestCatalogModule -> ConfigurationAwareModule$1 -> ConditionalModule -> IcebergRestCatalogModule$$Lambda/0x0000000802195000
  at TrinoIcebergRestCatalogFactory.<init>(Unknown Source)
      \_ for 5th parameter awsProperties
  while locating TrinoIcebergRestCatalogFactory
  at java.base/DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
      \_ installed by: IcebergCatalogModule -> ConditionalModule -> IcebergRestCatalogModule
  at IcebergMetadataFactory.<init>(Unknown Source)
      \_ for 4th parameter catalogFactory
  at IcebergModule.configure(IcebergModule.java:102)
  at IcebergTransactionManager.<init>(Unknown Source)
      \_ for 1st parameter metadataFactory
  at IcebergModule.configure(IcebergModule.java:75)
  at IcebergSplitManager.<init>(Unknown Source)
      \_ for 1st parameter transactionManager
  while locating IcebergSplitManager
  at IcebergModule.configure(IcebergModule.java:85)
  at ClassLoaderSafeConnectorSplitManager.<init>(Unknown Source)
      \_ for 1st parameter delegate
  while locating ClassLoaderSafeConnectorSplitManager
  at IcebergModule.configure(IcebergModule.java:86)
  while locating ConnectorSplitManager

Do we need to use a REST catalog URL like:
iceberg.rest-catalog.uri=https://glue.us-east-2.amazonaws.com/iceberg

or should it be:
https://s3tables/..amazonaws.com/iceberg?

REF https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-open-source.html

[J0hnG4lt]

For me, this was fixed after specifying the s3.iam-role ARN in the catalog properties.

• https://trino.io/docs/current/object-storage/file-system-s3.html#authentication

Otherwise, the catalog was defaulting to access keys.

[soumilshah1995]

I can give that a try to see if it works