Slow Docker Pull

Question

Hi dear community,

We are hosting private docker registry on different remote network and experiencing some pull issues.Now overall scenario is imagine 3 different remote networks.1 and 2 networks are clients and locate on same country 3rd network is docker registry and locates on different country that clients.Now 1st client has like 936.75Mbps(Latency: 1.47 ms (0.12 ms jitter)) download speed and almost has no issues pulling docker images.But second client network has a bit slower (generally high) internet speed like 100Mbps(Latency: 1.44 ms (0.55 ms jitter)).Now this second client has pulling issues like sometimes pull starts fast on some layers then one of the layers(this layer always changes ,not same layers) starts pulling very-very slowly like Kilobayts per second and that pulls takes a lot of time to pull.If I am lucky docker will retry pulling of layer and right after this retry layer pulls fastly or I would need to restart docker service, restarting docker service helps for that particular time(but very bad idea to restart docker on production).I have 3 questions
1)What can be the reason for this problem? is there any way to full debug pull process?
2)How can I customize retry interval for docker layers?
3Is there any way to force docker retry image pull without restarting docker(image pulls are done by kubernetes so pull works somewhere in background)

MD · Answer 1 · Oct 7, 2020

Hi@Allahshukar,

As you said your docker clients are in a different country then latency is an issue. For example, if you hosted your docker in a cloud platform like AWS, then you can use the CloudFront service. It will work as edge computing. So every client will connect to the nearest cloud datacenter.

answered Oct 7, 2020 by MD
• 95,460 points

Latency is the same for both clients but there is a huge difference in speed. Now, what is the other solution than hosting a docker registry locally? what about the retry interval is it customizable? because retries help docker to be fast again it is like a problem with long sessions.

The latency between 1st client and server

Server: ORANGE FRANCE - Bordeaux (id = 29542)
        ISP: AzInTelecom LLC
    Latency:    89.65 ms   (0.08 ms jitter)
   Download:  1798.13 Mbps (data used: 2.8 GB)
     Upload:   923.92 Mbps (data used: 1.6 GB)

The latency between 2nd client and server

Server: ORANGE FRANCE - Bordeaux (id = 29542)
        ISP: Delta Telecom Ltd
    Latency:    90.35 ms   (0.44 ms jitter)
   Download:    89.14 Mbps (data used: 121.0 MB)                              
     Upload:    91.31 Mbps (data used: 109.0 MB)

commented Oct 7, 2020 by Allahshukur
• 130 points
edited Oct 7, 2020 by MD

Hi@Allahshukur,

This is what I told you. Because packets are travelling from one country to another country. You may also face a network connectivity issue. I mean it starts downloading but after some time it shows network issue. According to me, you need to use an edge computing concept. Research on the topic and see if matches your requirement or not.