We found a correlation between network traffic and memory. The next thing we looked at was network traffic. The number of hosts has increased at a pretty constant rate over time, and that didn’t account for our recent jump in Swarm memory usage. At first, it looked like there was a correlation when we only looked a few weeks back, but the picture became more clear when we zoomed out to see the rest of the year. Next, we looked at the number of Docker hosts as Swarm is also responsible for keeping track of hosts. But the graph showed no correlation so we moved on. Next we looked at the number of running containers since we knew Swarm was responsible for scheduling containers. We needed to find something that could be mapped to memory usage, like holding a file in memory. It was increasing when memory was, but CPU usage does not normally cause RAM usage, so we classified this as a symptom rather than the root cause. Next, we looked into our metric graphs alongside Swarm’s memory usage.įirst we looked at CPU usage. But within 2 minutes the memory usage was back to 8GB. The InvestigationĪt first we thought it was a memory leak, and simply redeployed the Docker Swarm container. At the end of the month, Swarm was using 8GB of memory it was time to investigate. Again, everything was normal and the number of hosts we had were increasing, so we assumed it was typical usage. Our metrics and canaries reported healthy, so we assumed Swarm was just consuming memory as needed. One day we received an alert saying Swarm was using over 1GB of memory. Swarm abstracts hosts away so you don’t need to know what host a container is running on to perform an action on them (such as stop, start, inspect, or exec). Swarm takes container create requests and finds the best host to run it on. What is Swarm?ĭocker Swarm manages multiple Docker hosts in a way that makes it look like you are running on a single Docker host.
#Com docker hyperkit consuming code
After a bit of graphing and math, we were able to locate the code behind this unexpected behavior. Our investigation led us to discover an unexpected factor that determines its memory usage. A few weeks ago we were alerted that Docker Swarm was using over 8GB of RAM.
0 kommentar(er)
