On our (single node) Tableau Server, I was recently entrusted with doing the following tasks:
Harden the installation by following the instructions at https://help.tableau.com/current/server/en-us/security harden.htm.
Increase the associated volume's disc size (AWS EBS)
Change the server's instance type.
I had no prior Tableau experience, therefore I did the following:
tsm configuration set -k ssl.protocols -v "all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1" tsm pending-changes apply tsm configuration set -k ssl.protocols -v "all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1"
I waited for the adjustments to take effect before shutting down the server.
EBS volume size has been increased.
The EC2 instance type has been changed.
The server was restarted.
It took a long time for things to load when the server was restarted. Except for the admin panel and the tsm command, everything looked to be in order.
Running tsm status revealed:
$ tsm status
Could not connect to server. Make sure that Tableau Server is running and try again.
Okay. Checked the syslog (/var/log/syslog), which was pointed me towards tabadmincontroller as per:
systemd[1172]: Started Tableau Server Administration Controller.
tabadmincontroller_0[11354]: [11354] [INFO] 2021-11-24 13:13:47.106 +0000 : Loading configuration from /var/opt/tableau/tableau_server/data/tabsvc/services/tabadmincontroller_0.20204.20.1116.1810/bin/tabadmincontroller.runjavaservice.json
tabadmincontroller_0[11354]: [11354] [INFO] 2021-11-24 13:13:47.107 +0000 : Loading configuration from /var/opt/tableau/tableau_server/data/tabsvc/services/tabadmincontroller_0.20204.20.1116.1810/config/tabadmincontroller.runjavaservice.json
tabadmincontroller_0[11354]: [11354] [INFO] 2021-11-24 13:13:47.107 +0000 : Loading manifest from /var/opt/tableau/tableau_server/data/tabsvc/services
/tabadmincontroller_0.20204.20.1116.1810/bin/tabadmincontroller.jar
tabadmincontroller_0[11354]: [11354] [INFO] 2021-11-24 13:13:47.107 +0000 : Starting malloc_trim thread. Run every 60 sec. Heap pad MB: 1
tabadmincontroller_0[11354]: [11354] [INFO] 2021-11-24 13:13:47.109 +0000 : Loading JVM library /var/opt/tableau/tableau_server/data/tabsvc/services/tabadmincontroller_0.20204.20.1116.1810/repository/jre/lib/server/libjvm.so
tabadmincontroller_0[11354]: [11354] [INFO] 2021-11-24 13:13:47.192 +0000 : Java class name: com.tableausoftware.tabadmin.webapp.TabadminController; Method name: main; Arguments: run
nlp_0[10759]: time="2021-11-24T13:13:54.323621681Z" level=info msg="[core] grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\"" logger="pegasus/server:server" system=system
nlp_0[10759]: time="2021-11-24T13:14:05.775776702Z" level=info msg="get job counts" blockedJobCount=0 inProgressJobCount=0 logger="semanticmodel/metrics:Monitor" queuedJobCount=0 waitingJobCount=0
systemd[1172]: tabadmincontroller_0.service: Main process exited, code=exited, status=1/FAILURE
systemd[1172]: tabadmincontroller_0.service: Failed with result 'exit-code'.
systemd[1172]: tabadmincontroller_0.service: Service hold-off time over, scheduling restart.
systemd[1172]: tabadmincontroller_0.service: Scheduled restart job, restart counter is at 7.
systemd[1172]: Stopped Tableau Server Administration Controller.
systemd[1172]: Started Tableau Server Administration Controller.
With an ongoing crash/restart loop going on forever.
Checking the logs at /var/opt/tableau/tableau_server/data/tabsvc/logs/tabadmincontroller
shows the following:
2021-11-24 13:17:37.372 +0000 main : INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=ip-172-33-30-206:8591 sessionTimeout=120000 watcher=org.apache.curator.ConnectionState@3657201c
2021-11-24 13:17:37.381 +0000 main : INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2021-11-24 13:17:37.387 +0000 main : INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes
2021-11-24 13:17:37.397 +0000 main : INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
2021-11-24 13:17:37.402 +0000 main : INFO org.apache.curator.framework.imps.CuratorFrameworkImpl - Default schema
2021-11-24 13:17:37.408 +0000 main-SendThread(ip-172-33-30-206:8591) : INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server ip-address-redacted. Will not attempt to authenticate using SASL (unknown error)
20redacted:8591) : INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /ip-redacted:45082, server: ip/ip:8591
2021-11-24 13:17:37.423 +0000 main-SendThread(ip:8591) : INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server, sessionid = 0x10000005ed60050, negotiated timeout = 120000
2021-11-24 13:17:37.432 +0000 main-EventThread : INFO org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
2021-11-24 13:17:37.780 +0000 main : ERROR com.tableausoftware.telemetry.helper.Environment - [xdev-edpa-007] process_type is not present in the configuration
2021-11-24 13:17:38.298 +0000 ZKWorker-ScheduledTask-0 : INFO org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2021-11-24 13:17:38.298 +0000 ZKWorker-ScheduledTask-0 : INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=ip sessionTimeout=120000 watcher=org.apache.curator.ConnectionState@48dddf5a
2021-11-24 13:17:38.298 +0000 ZKWorker-ScheduledTask-0 : INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes
2021-11-24 13:17:38.299 +0000 ZKWorker-ScheduledTask-0 : INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
2021-11-24 13:17:38.300 +0000 ZKWorker-ScheduledTask-0 : INFO org.apache.curator.framework.imps.CuratorFrameworkImpl - Default schema
2021-11-24 13:17:38.300 +0000 ZKWorker-ScheduledTask-0-SendThread(ip:8591) : INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server ip/ip:8591. Will not attempt to authenticate using SASL (unknown error)
2021-11-24 13:17:38.301 +0000 ZKWorker-ScheduledTask-0-SendThread(ip:8591) : INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: ip:45088, server: ip/ip:8591
2021-11-24 13:17:38.303 +0000 ZKWorker-ScheduledTask-0-SendThread(ip-172-33-30-206:8591) : INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server ip-172-33-30-206/ip-address:8591, sessionid = 0x10000005ed60051, negotiated timeout = 120000
2021-11-24 13:17:38.303 +0000 ZKWorker-ScheduledTask-0-EventThread : INFO org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
2021-11-24 13:17:39.357 +0000 main : INFO org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=node0
2021-11-24 13:17:39.357 +0000 main : INFO org.eclipse.jetty.server.session - No SessionScavenger set, using defaults
2021-11-24 13:17:39.358 +0000 main : INFO org.eclipse.jetty.server.session - node0 Scavenging every 660000ms
2021-11-24 13:17:39.364 +0000 main : INFO org.eclipse.jetty.server.handler.ContextHandler - Started o.s.b.w.e.j.JettyEmbeddedWebAppContext@38e0ccf3{application,/,[file:///var/opt/tableau/tableau_server/data/tabsvc/temp/tabadmincontroller_0.20204.20.1116.1810/jetty-docbase.7353433006274726961.8850/],AVAILABLE}
2021-11-24 13:17:39.365 +0000 main : INFO org.eclipse.jetty.server.Server - Started @9503ms
2021-11-24 13:17:39.755 +0000 main : INFO org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2021-11-24 13:17:39.756 +0000 main : INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=ip-172-33-30-206:8591 sessionTimeout=120000 watcher=org.apache.curator.ConnectionState@6376534c
2021-11-24 13:17:39.756 +0000 main : INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes
2021-11-24 13:17:39.757 +0000 main : INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
2021-11-24 13:17:39.757 +0000 main : INFO org.apache.curator.framework.imps.CuratorFrameworkImpl - Default schema
2021-11-24 13:17:39.758 +0000 main-SendThread(ip-172-33-30-206:8591) : INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server ip-172-33-30-206/ip-address:8591. Will not attempt to authenticate using SASL (unknown error)
2021-11-24 13:17:39.758 +0000 main-SendThread(ip-172-33-30-206:8591) : INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /ip-address:45098, server: ip-172-33-30-206/ip-address:8591
2021-11-24 13:17:39.760 +0000 main-SendThread(ip-172-33-30-206:8591) : INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server ip-172-33-30-206/ip-address:8591, sessionid = 0x10000005ed60052, negotiated timeout = 120000
2021-11-24 13:17:39.760 +0000 main-EventThread : INFO org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
2021-11-24 13:17:50.803 +0000 main : WARN org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'webSecurityConfig' defined in URL [jar:file:/opt/tableau/tableau_server/packages/bin.20204.20.1116.1810/tabadmincontroller.jar!/com/tableausoftware/tabadmin/webapp/WebSecurityConfig.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cookieAuthenticationService' defined in URL [jar:file:/opt/tableau/tableau_server/packages/bin.20204.20.1116.1810/tabadmincontroller.jar!/com/tableausoftware/tabadmin/webapp/auth/CookieAuthenticationService.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.tableausoftware.tabadmin.webapp.auth.CookieAuthenticationService]: Constructor threw exception; nested exception is org.springframework.core.convert.ConversionFailedException: Failed to convert from type [java.lang.String] to type [int] for value 'warn'; nested exception is java.lang.NumberFormatException: For input string: "warn"
2021-11-24 13:17:50.806 +0000 Curator-Framework-0 : INFO org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
2021-11-24 13:17:50.910 +0000 main : INFO org.apache.zookeeper.ZooKeeper - Session: 0x10000005ed60052 closed
2021-11-24 13:17:50.910 +0000 main-EventThread : INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10000005ed60052
2021-11-24 13:17:50.917 +0000 Curator-Framework-0 : INFO org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
2021-11-24 13:17:50.930 +0000 Thread-15 : ERROR com.tableausoftware.tabadmin.webapp.EmbeddedLicenseActivator - Caught exception in EmbeddedLicenseActivator.
org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'nativeApiInitializer': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:208) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1290) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.context.annotation.ContextAnnotationAutowireCandidateResolver$1.getTarget(ContextAnnotationAutowireCandidateResolver.java:90) ~[spring-context-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:192) ~[spring-aop-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at com.sun.proxy.$Proxy144.hasEnoughRoleLicenses(Unknown Source) ~[?:?]
at com.tableausoftware.tabadmin.webapp.impl.ProductKeyService.getLicensingState(ProductKeyService.java:373) ~[tab-tabadmin-controller-latest.jar:?]
at com.tableausoftware.tabadmin.webapp.EmbeddedLicenseActivator$1.run(EmbeddedLicenseActivator.java:43) [tabadmincontroller.jar:?]
2021-11-24 13:17:50.935 +0000 Thread-15 : ERROR com.tableausoftware.tabadmin.webapp.EmbeddedLicenseActivator - Failed to activate the OEM embedded license.
2021-11-24 13:17:51.020 +0000 main-EventThread : INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10000005ed60050
2021-11-24 13:17:51.020 +0000 main : INFO org.apache.zookeeper.ZooKeeper - Session: 0x10000005ed60050 closed
2021-11-24 13:17:51.021 +0000 main : WARN com.tableausoftware.tabadmin.agent.zookeeper.ConnectionHolder - Object ZooKeeperWorker@74b99b0e. was not a consumer of this connection.
2021-11-24 13:17:51.022 +0000 Curator-Framework-0 : INFO org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
2021-11-24 13:17:51.124 +0000 ZKWorker-ScheduledTask-0-EventThread : INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10000005ed60051
2021-11-24 13:17:51.124 +0000 main : INFO org.apache.zookeeper.ZooKeeper - Session: 0x10000005ed60051 closed
2021-11-24 13:17:51.142 +0000 main : INFO org.eclipse.jetty.server.session - node0 Stopped scavenging
2021-11-24 13:17:51.144 +0000 main : INFO org.eclipse.jetty.server.handler.ContextHandler - Stopped o.s.b.w.e.j.JettyEmbeddedWebAppContext@38e0ccf3{application,/,[file:///var/opt/tableau/tableau_server/data/tabsvc/temp/tabadmincontroller_0.20204.20.1116.1810/jetty-docbase.7353433006274726961.8850/],UNAVAILABLE}
2021-11-24 13:17:51.154 +0000 main : INFO org.springframework.boot.autoconfigure.logging.ConditionEvaluationReportLoggingListener -
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2021-11-24 13:17:51.156 +0000 main : ERROR org.springframework.boot.SpringApplication - Application run failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'webSecurityConfig' defined in URL [jar:file:/opt/tableau/tableau_server/packages/bin.20204.20.1116.1810/tabadmincontroller.jar!/com/tableausoftware/tabadmin/webapp/WebSecurityConfig.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cookieAuthenticationService' defined in URL [jar:file:/opt/tableau/tableau_server/packages/bin.20204.20.1116.1810/tabadmincontroller.jar!/com/tableausoftware/tabadmin/webapp/auth/CookieAuthenticationService.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.tableausoftware.tabadmin.webapp.auth.CookieAuthenticationService]: Constructor threw exception; nested exception is org.springframework.core.convert.ConversionFailedException: Failed to convert from type [java.lang.String] to type [int] for value 'warn'; nested exception is java.lang.NumberFormatException: For input string: "warn"
Can somebody lead me in the correct way for the next steps in this situation?
