Now, instead of guessing or jumping through hoops to find the answer, we can see the tool can tell us:
"Container X is occupied 12GB on GPU #1 with Y memory utilization"

We went from blindfolded resource to actual insight.

And our question is finally answered 🥳🎉

- Nvidia tools (e.g., nvidia-smi) show processes, but not container names.
- Docker tools (e.g, docker status) show CPU and memory, but no GPU data.

We would still be blindfolded. And our question is not answered yet!

When it comes to monitoring GPU usage in containerized environment, Nvidia and Docker both of them provide good out-of-the-box tools, but they aren't compatible.

None of them can answer my simple question:
"Which container uses which GPU?"

If you're training ML models with Docker containers and Nvidia GPUs, especially on-prem, you've likely seen wasted compute.

GPUs sit idle while its occupies memory (aka. VRAM), but without observation tools, that leads to poor utilization and waised compute.

#Nvidia #GPU #Docker #MachineLearning