ilum-core failed to start due to hive metastore

When trying to install ilum by `helm install ilum ilum/ilum --set ilum-sql.enabled=true --set ilum-hive-metastore.enabled=true --set ilum-core.hiveMetastore.enabled=true` I see the pod of ilum-core fails to start. below are logs 2025-03-30 19:12:19 Listening for transport dt_socket at address: 5005 2025-03-30 19:12:21 ((((. 2025-03-30 19:12:21 #(((((((((((((( ... ... ... ... .... ... 2025-03-30 19:12:21 ########((((( ... ... ... ... ..... .... 2025-03-30 19:12:21 #######(((( ... ... ... ... ...... ..... 2025-03-30 19:12:21 ,#######/ ... ... ... ... ... ... ...... 2025-03-30 19:12:21 ##### ... ... ... ... ... ... ... ... 2025-03-30 19:12:21 ### ... .......... .... ..... ... .... ... 2025-03-30 19:12:21 # ... .......... ...... ... .. ... 2025-03-30 19:12:21 2025-03-30 19:12:21 2025-03-30T11:12:21.797Z INFO 7 --- [ main] c.ilum.service.IlumServiceApplicationKt : Starting IlumServiceApplicationKt v6.3.1 using Java 17.0.13 with PID 7 (/ilum-service.jar started by ilum in /) 2025-03-30 19:12:21 2025-03-30T11:12:21.800Z INFO 7 --- [ main] c.ilum.service.IlumServiceApplicationKt : No active profile set, falling back to 1 default profile: "default" 2025-03-30 19:12:21 2025-03-30T11:12:21.921Z WARN 7 --- [ main] i.m.s.o.c.g.r.m.MetaInfExtensionModule : Module [groovy-all] - Unable to load extension class [org.codehaus.groovy.jsr223.ScriptExtensions] 2025-03-30 19:12:21 2025-03-30T11:12:21.921Z WARN 7 --- [ main] i.m.s.o.c.g.r.m.MetaInfExtensionModule : Module [groovy-all] - Unable to load extension class [org.codehaus.groovy.runtime.NioGroovyMethods] 2025-03-30 19:12:21 2025-03-30T11:12:21.921Z WARN 7 --- [ main] i.m.s.o.c.g.r.m.MetaInfExtensionModule : Module [groovy-all] - Unable to load extension class [org.codehaus.groovy.runtime.SqlGroovyMethods] 2025-03-30 19:12:21 2025-03-30T11:12:21.921Z WARN 7 --- [ main] i.m.s.o.c.g.r.m.MetaInfExtensionModule : Module [groovy-all] - Unable to load extension class [org.codehaus.groovy.runtime.SwingGroovyMethods] 2025-03-30 19:12:21 2025-03-30T11:12:21.921Z WARN 7 --- [ main] i.m.s.o.c.g.r.m.MetaInfExtensionModule : Module [groovy-all] - Unable to load extension class [org.codehaus.groovy.runtime.XmlGroovyMethods] 2025-03-30 19:12:21 2025-03-30T11:12:21.921Z WARN 7 --- [ main] i.m.s.o.c.g.r.m.MetaInfExtensionModule : Module [groovy-all] - Unable to load extension class [org.codehaus.groovy.jsr223.ScriptStaticExtensions] 2025-03-30 19:12:24 2025-03-30T11:12:24.251Z INFO 7 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data MongoDB repositories in DEFAULT mode. 2025-03-30 19:12:25 2025-03-30T11:12:25.434Z INFO 7 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 1178 ms. Found 14 MongoDB repository interfaces. 2025-03-30 19:12:26 2025-03-30T11:12:26.038Z INFO 7 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data Reactive MongoDB repositories in DEFAULT mode. 2025-03-30 19:12:26 2025-03-30T11:12:26.072Z INFO 7 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 34 ms. Found 1 Reactive MongoDB repository interfaces. 2025-03-30 19:12:32 2025-03-30T11:12:32.016Z INFO 7 --- [ main] c.i.s.s.j.i.group.GroupAutoPauser : GroupAutoPauser initialized with period 3600 2025-03-30 19:12:34 2025-03-30T11:12:34.126Z WARN 7 --- [ main] ocalAuthenticationConfig$$SpringCGLIB$$0 : Generating random JWK keys. Configure JWK to support scalability. 2025-03-30 19:12:35 2025-03-30T11:12:35.085Z INFO 7 --- [tLoopGroup-3-16] cloud.ilum.service.config.MongoConfig : All mongo indexes have been created. 2025-03-30 19:12:35 2025-03-30T11:12:35.572Z INFO 7 --- [ main] i.m.r.core.builder.RunnerBuilderBase : Mongock runner COMMUNITY version[5.3.4] 2025-03-30 19:12:35 2025-03-30T11:12:35.574Z INFO 7 --- [ main] i.m.r.core.builder.RunnerBuilderBase : Running Mongock with NO metadata 2025-03-30 19:12:35 2025-03-30T11:12:35.574Z WARN 7 --- [ main] i.m.r.core.builder.RunnerBuilderBase : Property transaction-enabled not provided. It will become true as default in next versions. Set explicit value to false in case transaction are not desired. 2025-03-30 19:12:35 2025-03-30T11:12:35.574Z WARN 7 --- [ main] i.m.r.core.builder.RunnerBuilderBase : Property transaction-enabled not provided, but driver is transactionable. BY DEFAULT MONGOCK WILL RUN IN TRANSACTION MODE. 2025-03-30 19:12:35 2025-03-30T11:12:35.882Z INFO 7 --- [ main] o.s.b.a.e.web.EndpointLinksResolver : Exposing 7 endpoint(s) beneath base path '/actuator' 2025-03-30 19:12:36 2025-03-30T11:12:36.365Z INFO 7 --- [ main] io.undertow : starting server: Undertow - 2.3.7.Final 2025-03-30 19:12:36 2025-03-30T11:12:36.386Z INFO 7 --- [ main] org.xnio : XNIO version 3.8.8.Final 2025-03-30 19:12:36 2025-03-30T11:12:36.406Z INFO 7 --- [ main] org.xnio.nio : XNIO NIO Implementation Version 3.8.8.Final 2025-03-30 19:12:36 2025-03-30T11:12:36.505Z INFO 7 --- [ main] org.jboss.threads : JBoss Threads version 3.5.0.Final 2025-03-30 19:12:36 2025-03-30T11:12:36.583Z INFO 7 --- [ main] o.s.b.w.e.undertow.UndertowWebServer : Undertow started on port(s) 8080 (http) 2025-03-30 19:12:36 2025-03-30T11:12:36.612Z INFO 7 --- [ scheduling-1] c.i.s.s.j.i.job.InteractiveJobManager : Starting automatic interactive jobs scaler 2025-03-30 19:12:36 2025-03-30T11:12:36.614Z INFO 7 --- [ main] c.ilum.service.IlumServiceApplicationKt : Started IlumServiceApplicationKt in 15.893 seconds (process running for 17.311) 2025-03-30 19:12:36 2025-03-30T11:12:36.629Z INFO 7 --- [ scheduling-2] c.i.s.s.j.i.job.InteractiveJobManager : Starting automatic interactive job instances cleaner, trying to delete 0 instances. 2025-03-30 19:12:36 2025-03-30T11:12:36.646Z INFO 7 --- [ scheduling-1] c.ilum.service.spark.job.JobService : Starting automatic jobs cleaner, trying to delete 0 jobs. 2025-03-30 19:12:36 2025-03-30T11:12:36.687Z INFO 7 --- [ scheduling-2] c.i.s.s.j.i.group.GroupAutoPauser : Starting automatic group pausing 2025-03-30 19:12:37 2025-03-30T11:12:37.056Z INFO 7 --- [ main] org.reflections.Reflections : Reflections took 112 ms to scan 1 urls, producing 2 keys and 2 values 2025-03-30 19:12:37 2025-03-30T11:12:37.082Z INFO 7 --- [ main] org.reflections.Reflections : Reflections took 2 ms to scan 1 urls, producing 2 keys and 2 values 2025-03-30 19:12:37 2025-03-30T11:12:37.107Z INFO 7 --- [ main] i.m.r.c.e.system.SystemUpdateExecutor : Mongock skipping the system update execution. All system change set items are already executed. 2025-03-30 19:12:37 2025-03-30T11:12:37.108Z INFO 7 --- [ main] i.m.r.core.executor.ChangeExecutorBase : PASSED OVER - {"id"="system-change-00001_before", "type"="before-execution", "author"="mongock", "class"="SystemChangeUnit00001", "method"="beforeExecution"} 2025-03-30 19:12:37 2025-03-30T11:12:37.108Z INFO 7 --- [ main] i.m.r.core.executor.ChangeExecutorBase : PASSED OVER - {"id"="system-change-00001", "type"="execution", "author"="mongock", "class"="SystemChangeUnit00001", "method"="execution"} 2025-03-30 19:12:37 2025-03-30T11:12:37.108Z INFO 7 --- [ main] i.m.r.c.e.system.SystemUpdateExecutor : Mongock has finished the system update execution 2025-03-30 19:12:37 2025-03-30T11:12:37.136Z INFO 7 --- [ main] org.reflections.Reflections : Reflections took 26 ms to scan 1 urls, producing 5 keys and 37 values 2025-03-30 19:12:37 2025-03-30T11:12:37.159Z INFO 7 --- [ main] org.reflections.Reflections : Reflections took 12 ms to scan 1 urls, producing 5 keys and 37 values 2025-03-30 19:12:37 2025-03-30T11:12:37.171Z INFO 7 --- [ main] i.m.r.c.e.o.migrate.MigrateExecutorBase : Mongock skipping the data migration. All change set items are already executed. 2025-03-30 19:12:37 2025-03-30T11:12:37.171Z INFO 7 --- [ main] i.m.r.core.executor.ChangeExecutorBase : PASSED OVER - {"id"="job-submit-time", "type"="execution", "author"="default_author", "class"="Migration001JobStatusSubmitTime", "method"="execution"} 2025-03-30 19:12:37 2025-03-30T11:12:37.171Z INFO 7 --- [ main] i.m.r.core.executor.ChangeExecutorBase : PASSED OVER - {"id"="mongo-map-key-dot-replacement", "type"="execution", "author"="default_author", "class"="Migration002MongoMapKeyDotReplacement", "method"="execution"} 2025-03-30 19:12:37 2025-03-30T11:12:37.171Z INFO 7 --- [ main] i.m.r.core.executor.ChangeExecutorBase : PASSED OVER - {"id"="cluster-storages", "type"="execution", "author"="default_author", "class"="Migration003ClusterStorages", "method"="execution"} 2025-03-30 19:12:37 2025-03-30T11:12:37.171Z INFO 7 --- [ main] i.m.r.core.executor.ChangeExecutorBase : PASSED OVER - {"id"="sql-notebooks-add-language", "type"="execution", "author"="default_author", "class"="Migration004SqlNotebookAddLanguage", "method"="execution"} 2025-03-30 19:12:37 2025-03-30T11:12:37.171Z INFO 7 --- [ main] i.m.r.core.executor.ChangeExecutorBase : PASSED OVER - {"id"="user-type", "type"="execution", "author"="default_author", "class"="Migration005UserTypeMigration", "method"="execution"} 2025-03-30 19:12:37 2025-03-30T11:12:37.171Z INFO 7 --- [ main] i.m.r.c.e.o.migrate.MigrateExecutorBase : Mongock has finished 2025-03-30 19:12:42 2025-03-30T11:12:42.838Z INFO 7 --- [ main] c.i.m.def.DefaultMetastoreInitializer : Creating default metastore 2025-03-30 19:12:42 2025-03-30T11:12:42.876Z INFO 7 --- [ main] i.m.s.o.a.h.h.m.HiveMetaStoreClient : Trying to connect to metastore with URI thrift://ilum-hive-metastore.default.svc.cluster.local:9083 2025-03-30 19:12:42 2025-03-30T11:12:42.892Z WARN 7 --- [ main] i.m.s.o.a.h.h.m.HiveMetaStoreClient : Failed to connect to the MetaStore Server... 2025-03-30 19:12:42 2025-03-30T11:12:42.892Z INFO 7 --- [ main] i.m.s.o.a.h.h.m.HiveMetaStoreClient : Waiting 1 seconds before next connection attempt. 2025-03-30 19:12:43 2025-03-30T11:12:43.922Z ERROR 7 --- [ main] cloud.ilum.metastore.hive.HiveClient : Could not connect to meta store using any of the URIs provided. Most recent failure: ilum.metastore.shaded.org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused 2025-03-30 19:12:43 at ilum.metastore.shaded.org.apache.thrift.transport.TSocket.open(TSocket.java:226) 2025-03-30 19:12:43 at ilum.metastore.shaded.org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:516) 2025-03-30 19:12:43 at ilum.metastore.shaded.org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:224) 2025-03-30 19:12:43 at ilum.metastore.shaded.org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:137) 2025-03-30 19:12:43 at cloud.ilum.metastore.hive.HiveClient.withHiveClient(HiveClient.kt:103) 2025-03-30 19:12:43 at cloud.ilum.metastore.hive.HiveClient.validate(HiveClient.kt:26) 2025-03-30 19:12:43 at cloud.ilum.metastore.hive.HiveValidator.validate(HiveValidator.kt:17) 2025-03-30 19:12:43 at cloud.ilum.metastore.validator.MetastoreValidators.validateConnection(MetastoreValidators.kt:11) 2025-03-30 19:12:43 at cloud.ilum.metastore.validator.MetastoreValidationService.validateMetastoreConnection(MetastoreValidationService.kt:37) 2025-03-30 19:12:43 at cloud.ilum.metastore.validator.MetastoreValidationService.validateMetastoreAddition(MetastoreValidationService.kt:14) 2025-03-30 19:12:43 at cloud.ilum.metastore.MetastoreService.saveMetastore(MetastoreService.kt:29) 2025-03-30 19:12:43 at cloud.ilum.metastore.def.DefaultMetastoreInitializer.initDefaultMetastore(DefaultMetastoreInitializer.kt:34) 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) 2025-03-30 19:12:43 at java.base/java.lang.reflect.Method.invoke(Unknown Source) 2025-03-30 19:12:43 at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:348) 2025-03-30 19:12:43 at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:233) 2025-03-30 19:12:43 at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:165) 2025-03-30 19:12:43 at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) 2025-03-30 19:12:43 at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) 2025-03-30 19:12:43 at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) 2025-03-30 19:12:43 at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:437) 2025-03-30 19:12:43 at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:370) 2025-03-30 19:12:43 at org.springframework.boot.context.event.EventPublishingRunListener.ready(EventPublishingRunListener.java:109) 2025-03-30 19:12:43 at org.springframework.boot.SpringApplicationRunListeners.lambda$ready$6(SpringApplicationRunListeners.java:80) 2025-03-30 19:12:43 at java.base/java.lang.Iterable.forEach(Unknown Source) 2025-03-30 19:12:43 at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118) 2025-03-30 19:12:43 at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:112) 2025-03-30 19:12:43 at org.springframework.boot.SpringApplicationRunListeners.ready(SpringApplicationRunListeners.java:80) 2025-03-30 19:12:43 at org.springframework.boot.SpringApplication.run(SpringApplication.java:331) 2025-03-30 19:12:43 at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) 2025-03-30 19:12:43 at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295) 2025-03-30 19:12:43 at cloud.ilum.service.IlumServiceApplicationKt.main(IlumServiceApplication.kt:15) 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) 2025-03-30 19:12:43 at java.base/java.lang.reflect.Method.invoke(Unknown Source) 2025-03-30 19:12:43 at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) 2025-03-30 19:12:43 at org.springframework.boot.loader.Launcher.launch(Launcher.java:95) 2025-03-30 19:12:43 at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) 2025-03-30 19:12:43 at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) 2025-03-30 19:12:43 Caused by: java.net.ConnectException: Connection refused 2025-03-30 19:12:43 at java.base/sun.nio.ch.Net.pollConnect(Native Method) 2025-03-30 19:12:43 at java.base/sun.nio.ch.Net.pollConnectNow(Unknown Source) 2025-03-30 19:12:43 at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(Unknown Source) 2025-03-30 19:12:43 at java.base/sun.nio.ch.NioSocketImpl.connect(Unknown Source) 2025-03-30 19:12:43 at java.base/java.net.SocksSocketImpl.connect(Unknown Source) 2025-03-30 19:12:43 at java.base/java.net.Socket.connect(Unknown Source) 2025-03-30 19:12:43 at ilum.metastore.shaded.org.apache.thrift.transport.TSocket.open(TSocket.java:221) 2025-03-30 19:12:43 ... 41 more 2025-03-30 19:12:43 2025-03-30 19:12:43 2025-03-30 19:12:43 ilum.metastore.shaded.org.apache.hadoop.hive.metastore.api.MetaException: Could not connect to meta store using any of the URIs provided. Most recent failure: ilum.metastore.shaded.org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused 2025-03-30 19:12:43 at ilum.metastore.shaded.org.apache.thrift.transport.TSocket.open(TSocket.java:226) 2025-03-30 19:12:43 at ilum.metastore.shaded.org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:516) 2025-03-30 19:12:43 at ilum.metastore.shaded.org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:224) 2025-03-30 19:12:43 at ilum.metastore.shaded.org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:137) 2025-03-30 19:12:43 at cloud.ilum.metastore.hive.HiveClient.withHiveClient(HiveClient.kt:103) 2025-03-30 19:12:43 at cloud.ilum.metastore.hive.HiveClient.validate(HiveClient.kt:26) 2025-03-30 19:12:43 at cloud.ilum.metastore.hive.HiveValidator.validate(HiveValidator.kt:17) 2025-03-30 19:12:43 at cloud.ilum.metastore.validator.MetastoreValidators.validateConnection(MetastoreValidators.kt:11) 2025-03-30 19:12:43 at cloud.ilum.metastore.validator.MetastoreValidationService.validateMetastoreConnection(MetastoreValidationService.kt:37) 2025-03-30 19:12:43 at cloud.ilum.metastore.validator.MetastoreValidationService.validateMetastoreAddition(MetastoreValidationService.kt:14) 2025-03-30 19:12:43 at cloud.ilum.metastore.MetastoreService.saveMetastore(MetastoreService.kt:29) 2025-03-30 19:12:43 at cloud.ilum.metastore.def.DefaultMetastoreInitializer.initDefaultMetastore(DefaultMetastoreInitializer.kt:34) 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) 2025-03-30 19:12:43 at java.base/java.lang.reflect.Method.invoke(Unknown Source) 2025-03-30 19:12:43 at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:348) 2025-03-30 19:12:43 at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:233) 2025-03-30 19:12:43 at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:165) 2025-03-30 19:12:43 at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) 2025-03-30 19:12:43 at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) 2025-03-30 19:12:43 at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) 2025-03-30 19:12:43 at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:437) 2025-03-30 19:12:43 at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:370) 2025-03-30 19:12:43 at org.springframework.boot.context.event.EventPublishingRunListener.ready(EventPublishingRunListener.java:109) 2025-03-30 19:12:43 at org.springframework.boot.SpringApplicationRunListeners.lambda$ready$6(SpringApplicationRunListeners.java:80) 2025-03-30 19:12:43 at java.base/java.lang.Iterable.forEach(Unknown Source) 2025-03-30 19:12:43 at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118) 2025-03-30 19:12:43 at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:112) 2025-03-30 19:12:43 at org.springframework.boot.SpringApplicationRunListeners.ready(SpringApplicationRunListeners.java:80) 2025-03-30 19:12:43 at org.springframework.boot.SpringApplication.run(SpringApplication.java:331) 2025-03-30 19:12:43 at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) 2025-03-30 19:12:43 at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295) 2025-03-30 19:12:43 at cloud.ilum.service.IlumServiceApplicationKt.main(IlumServiceApplication.kt:15) 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) 2025-03-30 19:12:43 at java.base/java.lang.reflect.Method.invoke(Unknown Source) 2025-03-30 19:12:43 at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) 2025-03-30 19:12:43 at org.springframework.boot.loader.Launcher.launch(Launcher.java:95) 2025-03-30 19:12:43 at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) 2025-03-30 19:12:43 at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) 2025-03-30 19:12:43 Caused by: java.net.ConnectException: Connection refused 2025-03-30 19:12:43 at java.base/sun.nio.ch.Net.pollConnect(Native Method) 2025-03-30 19:12:43 at java.base/sun.nio.ch.Net.pollConnectNow(Unknown Source) 2025-03-30 19:12:43 at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(Unknown Source) 2025-03-30 19:12:43 at java.base/sun.nio.ch.NioSocketImpl.connect(Unknown Source) 2025-03-30 19:12:43 at java.base/java.net.SocksSocketImpl.connect(Unknown Source) 2025-03-30 19:12:43 at java.base/java.net.Socket.connect(Unknown Source) 2025-03-30 19:12:43 at ilum.metastore.shaded.org.apache.thrift.transport.TSocket.open(TSocket.java:221) 2025-03-30 19:12:43 ... 41 more 2025-03-30 19:12:43 2025-03-30 19:12:43 at ilum.metastore.shaded.org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:565) ~[ilum-metastore-cli-shadow.jar!/:na] 2025-03-30 19:12:43 at ilum.metastore.shaded.org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:224) ~[ilum-metastore-cli-shadow.jar!/:na] 2025-03-30 19:12:43 at ilum.metastore.shaded.org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:137) ~[ilum-metastore-cli-shadow.jar!/:na] 2025-03-30 19:12:43 at cloud.ilum.metastore.hive.HiveClient.withHiveClient(HiveClient.kt:103) ~[ilum-metastore-cli-shadow.jar!/:na] 2025-03-30 19:12:43 at cloud.ilum.metastore.hive.HiveClient.validate(HiveClient.kt:26) ~[ilum-metastore-cli-shadow.jar!/:na] 2025-03-30 19:12:43 at cloud.ilum.metastore.hive.HiveValidator.validate(HiveValidator.kt:17) ~[ilum-metastore-cli-shadow.jar!/:na] 2025-03-30 19:12:43 at cloud.ilum.metastore.validator.MetastoreValidators.validateConnection(MetastoreValidators.kt:11) ~[ilum-metastore-cli-shadow.jar!/:na] 2025-03-30 19:12:43 at cloud.ilum.metastore.validator.MetastoreValidationService.validateMetastoreConnection(MetastoreValidationService.kt:37) ~[ilum-metastore-cli-shadow.jar!/:na] 2025-03-30 19:12:43 at cloud.ilum.metastore.validator.MetastoreValidationService.validateMetastoreAddition(MetastoreValidationService.kt:14) ~[ilum-metastore-cli-shadow.jar!/:na] 2025-03-30 19:12:43 at cloud.ilum.metastore.MetastoreService.saveMetastore(MetastoreService.kt:29) ~[ilum-metastore-cli-shadow.jar!/:na] 2025-03-30 19:12:43 at cloud.ilum.metastore.def.DefaultMetastoreInitializer.initDefaultMetastore(DefaultMetastoreInitializer.kt:34) ~[ilum-metastore-cli-shadow.jar!/:na] 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na] 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:na] 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na] 2025-03-30 19:12:43 at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na] 2025-03-30 19:12:43 at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:348) ~[spring-context-6.0.11.jar!/:6.0.11] 2025-03-30 19:12:43 at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:233) ~[spring-context-6.0.11.jar!/:6.0.11] 2025-03-30 19:12:43 at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:165) ~[spring-context-6.0.11.jar!/:6.0.11] 2025-03-30 19:12:43 at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-6.0.11.jar!/:6.0.11] 2025-03-30 19:12:43 at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-6.0.11.jar!/:6.0.11] 2025-03-30 19:12:43 at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) ~[spring-context-6.0.11.jar!/:6.0.11] 2025-03-30 19:12:43 at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:437) ~[spring-context-6.0.11.jar!/:6.0.11] 2025-03-30 19:12:43 at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:370) ~[spring-context-6.0.11.jar!/:6.0.11] 2025-03-30 19:12:43 at org.springframework.boot.context.event.EventPublishingRunListener.ready(EventPublishingRunListener.java:109) ~[spring-boot-3.1.2.jar!/:3.1.2] 2025-03-30 19:12:43 at org.springframework.boot.SpringApplicationRunListeners.lambda$ready$6(SpringApplicationRunListeners.java:80) ~[spring-boot-3.1.2.jar!/:3.1.2] 2025-03-30 19:12:43 at java.base/java.lang.Iterable.forEach(Unknown Source) ~[na:na] 2025-03-30 19:12:43 at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118) ~[spring-boot-3.1.2.jar!/:3.1.2] 2025-03-30 19:12:43 at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:112) ~[spring-boot-3.1.2.jar!/:3.1.2] 2025-03-30 19:12:43 at org.springframework.boot.SpringApplicationRunListeners.ready(SpringApplicationRunListeners.java:80) ~[spring-boot-3.1.2.jar!/:3.1.2] 2025-03-30 19:12:43 at org.springframework.boot.SpringApplication.run(SpringApplication.java:331) ~[spring-boot-3.1.2.jar!/:3.1.2] 2025-03-30 19:12:43 at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) ~[spring-boot-3.1.2.jar!/:3.1.2] 2025-03-30 19:12:43 at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295) ~[spring-boot-3.1.2.jar!/:3.1.2] 2025-03-30 19:12:43 at cloud.ilum.service.IlumServiceApplicationKt.main(IlumServiceApplication.kt:15) ~[classes!/:6.3.1] 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na] 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:na] 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na] 2025-03-30 19:12:43 at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na] 2025-03-30 19:12:43 at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[ilum-service.jar:6.3.1] 2025-03-30 19:12:43 at org.springframework.boot.loader.Launcher.launch(Launcher.java:95) ~[ilum-service.jar:6.3.1] 2025-03-30 19:12:43 at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[ilum-service.jar:6.3.1] 2025-03-30 19:12:43 at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) ~[ilum-service.jar:6.3.1] 2025-03-30 19:12:43 2025-03-30 19:12:43 2025-03-30T11:12:43.952Z ERROR 7 --- [ main] o.s.boot.SpringApplication : Application run failed 2025-03-30 19:12:43 2025-03-30 19:12:43 java.lang.reflect.UndeclaredThrowableException: Failed to invoke event listener method 2025-03-30 19:12:43 HandlerMethod details: 2025-03-30 19:12:43 Bean [cloud.ilum.metastore.def.DefaultMetastoreInitializer] 2025-03-30 19:12:43 Method [public final void cloud.ilum.metastore.def.DefaultMetastoreInitializer.initDefaultMetastore()] 2025-03-30 19:12:43 Resolved arguments: 2025-03-30 19:12:43 2025-03-30 19:12:43 at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:365) ~[spring-context-6.0.11.jar!/:6.0.11] 2025-03-30 19:12:43 at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:233) ~[spring-context-6.0.11.jar!/:6.0.11] 2025-03-30 19:12:43 at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:165) ~[spring-context-6.0.11.jar!/:6.0.11] 2025-03-30 19:12:43 at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-6.0.11.jar!/:6.0.11] 2025-03-30 19:12:43 at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-6.0.11.jar!/:6.0.11] 2025-03-30 19:12:43 at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) ~[spring-context-6.0.11.jar!/:6.0.11] 2025-03-30 19:12:43 at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:437) ~[spring-context-6.0.11.jar!/:6.0.11] 2025-03-30 19:12:43 at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:370) ~[spring-context-6.0.11.jar!/:6.0.11] 2025-03-30 19:12:43 at org.springframework.boot.context.event.EventPublishingRunListener.ready(EventPublishingRunListener.java:109) ~[spring-boot-3.1.2.jar!/:3.1.2] 2025-03-30 19:12:43 at org.springframework.boot.SpringApplicationRunListeners.lambda$ready$6(SpringApplicationRunListeners.java:80) ~[spring-boot-3.1.2.jar!/:3.1.2] 2025-03-30 19:12:43 at java.base/java.lang.Iterable.forEach(Unknown Source) ~[na:na] 2025-03-30 19:12:43 at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118) ~[spring-boot-3.1.2.jar!/:3.1.2] 2025-03-30 19:12:43 at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:112) ~[spring-boot-3.1.2.jar!/:3.1.2] 2025-03-30 19:12:43 at org.springframework.boot.SpringApplicationRunListeners.ready(SpringApplicationRunListeners.java:80) ~[spring-boot-3.1.2.jar!/:3.1.2] 2025-03-30 19:12:43 at org.springframework.boot.SpringApplication.run(SpringApplication.java:331) ~[spring-boot-3.1.2.jar!/:3.1.2] 2025-03-30 19:12:43 at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) ~[spring-boot-3.1.2.jar!/:3.1.2] 2025-03-30 19:12:43 at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295) ~[spring-boot-3.1.2.jar!/:3.1.2] 2025-03-30 19:12:43 at cloud.ilum.service.IlumServiceApplicationKt.main(IlumServiceApplication.kt:15) ~[classes!/:6.3.1] 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na] 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:na] 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na] 2025-03-30 19:12:43 at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na] 2025-03-30 19:12:43 at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[ilum-service.jar:6.3.1] 2025-03-30 19:12:43 at org.springframework.boot.loader.Launcher.launch(Launcher.java:95) ~[ilum-service.jar:6.3.1] 2025-03-30 19:12:43 at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[ilum-service.jar:6.3.1] 2025-03-30 19:12:43 at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) ~[ilum-service.jar:6.3.1] 2025-03-30 19:12:43 Caused by: cloud.ilum.metastore.exception.ValidationException: Unable to validate hive metastore connection 2025-03-30 19:12:43 at cloud.ilum.metastore.hive.HiveValidator.validate(HiveValidator.kt:19) ~[ilum-metastore-cli-shadow.jar!/:na] 2025-03-30 19:12:43 at cloud.ilum.metastore.validator.MetastoreValidators.validateConnection(MetastoreValidators.kt:11) ~[ilum-metastore-cli-shadow.jar!/:na] 2025-03-30 19:12:43 at cloud.ilum.metastore.validator.MetastoreValidationService.validateMetastoreConnection(MetastoreValidationService.kt:37) ~[ilum-metastore-cli-shadow.jar!/:na] 2025-03-30 19:12:43 at cloud.ilum.metastore.validator.MetastoreValidationService.validateMetastoreAddition(MetastoreValidationService.kt:14) ~[ilum-metastore-cli-shadow.jar!/:na] 2025-03-30 19:12:43 at cloud.ilum.metastore.MetastoreService.saveMetastore(MetastoreService.kt:29) ~[ilum-metastore-cli-shadow.jar!/:na] 2025-03-30 19:12:43 at cloud.ilum.metastore.def.DefaultMetastoreInitializer.initDefaultMetastore(DefaultMetastoreInitializer.kt:34) ~[ilum-metastore-cli-shadow.jar!/:na] 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na] 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:na] 2025-03-30 19:12:43 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na] 2025-03-30 19:12:43 at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na] 2025-03-30 19:12:43 at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:348) ~[spring-context-6.0.11.jar!/:6.0.11] 2025-03-30 19:12:43 ... 25 common frames omitted 2025-03-30 19:12:43 2025-03-30 19:12:43 2025-03-30T11:12:43.957Z INFO 7 --- [ main] io.undertow : stopping server: Undertow - 2.3.7.Final