Can't recover docker swarm from pending state

+1 vote

There was a crash and I have this issue now where it says docker swarm status is pending and the node status is UNKNOWN. This is my docker info result

swarm@swarm-manager-1:~$ docker info
Containers: 270
 Running: 0
 Paused: 0
 Stopped: 270
Images: 160
Server Version: 1.12.2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 1211
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host bridge null overlay
Swarm: pending
 NodeID: d9hq8wzz6skh9pzrxzhbckm97
 Is Manager: true
 ClusterID: 5zgab5w50qgvvep35eqcbote2
 Managers: 1
 Nodes: 2
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: HIDDEN
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-91-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 6.804 GiB
Name: swarm-manager-1
ID: AXPO:VFSV:TDT3:6X7Y:QNAO:OZJN:U23R:V5S2:FU33:WUNI:CRPK:2E2C
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

This is my docker node ls result:

ID                           HOSTNAME         STATUS   AVAILABILITY  MANAGER STATUS
9tlo3rg7tuc23xzc3am28lak1    swarm-worker-1   Unknown  Active        
d9hq8wzz6skh9pzrxzhbckm97 *  swarm-manager-1  Unknown  Active        Leader
Sep 26, 2018 in Docker by Hannah
• 14,080 points
247 views

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

First, note the node IDs

On manager node:

docker node update --availability drain $WORKER_ID

This is optional

docker swarm join-token manager

This command will give you the join command to run on each node after it's removed. I'll refer to it as $JOIN_COMMAND below. We will demote the worker once the manager re-joins.

On worker:

docker swarm leave
$JOIN_COMMAND

This node is now re-joined as a manager, but I'll continue calling it the 'worker' to avoid confusion.

On manager:

docker node rm $WORKER_ID
docker node update --availability drain $MANAGER_ID
docker swarm leave -f
$JOIN_COMMAND
docker node rm $MANAGER_ID
docker node ls

Find the worker's new id (pay attention to the hostname, not the role) -> $NEW_WORKER_ID

docker node demote $NEW_WORKER_ID

Your swarm should be refreshed - if there were more nodes, the services running on each would have migrated across the swarm when you drained each node.

answered Sep 26, 2018 by Kalgi
• 35,800 points

Related Questions In Docker

0 votes
1 answer

Docker-Swarm: Join a docker-swarm from another subnet

You need the following ports open between ...READ MORE

answered Aug 22, 2018 in Docker by Nilesh
• 6,900 points
99 views
0 votes
1 answer

Docker Swarm- Build image only when stack deployed from manager node

Use docker build to build the image. ...READ MORE

answered Aug 24, 2018 in Docker by Nilesh
• 6,900 points
24 views
0 votes
2 answers

Can't access apache karaf docker container

To access karaf shell directly you can ...READ MORE

answered Aug 6, 2018 in Docker by Nilesh
• 6,900 points
129 views
0 votes
1 answer

How do I scale in Docker Swarm Mode W/Terraform Digital Ocean Load Balancing

The solution you could build for Digital ...READ MORE

answered Jun 19, 2018 in Docker by shubham
• 6,480 points
135 views
0 votes
1 answer
+5 votes
7 answers

Docker swarm vs kubernetes

Swarm is easy handling while kn8 is ...READ MORE

answered Aug 27, 2018 in Docker by Mahesh Ajmeria
631 views
0 votes
1 answer

Managers information from workers - Docker swarm

As of now there's no way to ...READ MORE

answered Oct 8, 2018 in Docker by Kalgi
• 35,800 points
17 views
+4 votes
4 answers

How To Access a Service on Host From a Docker Container?

Adding to kalgi's answer, You can also ...READ MORE

answered Oct 16, 2018 in Docker by lina
• 8,100 points

edited Oct 16, 2018 by lina 1,999 views

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.