Samuel Karp
LightNews LightNews
samuelkarp.com
Samuel Karp
@samuelkarp.com
230 followers 140 following 32 posts
Anti fascist. Chair of the Open Container Initiative TOB. Staff Software Engineer @ Google. Opinions = mine.
Posts Replies Media Videos
samuelkarp.com
This is pretty cool: a @containerd.dev shim to run LLMs with ollama. The shim augments the image by mounting in the ollama binary, generating a Modelfile as input to ollama (based on the model bundled in the image), and setting an entry point.
Mount Ollama binary • Mount the Ollama binary from the host into the container config. Mounts append(config. Mounts, specs. Mount Source: "/usr/bin/ollama" Destination: "/usr/bin/ollama", Type: "bind". Options: []string("rbind", "ro"}, }) Mount OCI as a Modelfile • Ollama needs to read the GGUF from a Modelfile file modelPath: getTaskEnv ("OLLAMA_MODEL PATH", config.Process.Env) modelfile: fmt.Sprintf("FROM %s", modelPath) err = os.WriteFile(filepath.Join(tempDir, "Modelfile"), []byte (modelfile), 0644) if err != nil { return nil, err } config.Mounts append(config.Mounts, specs. Mount( Source: filepath.Join(tempDir, "Modelfile"), Destination: "/tmp/Modelfile", Type: "bind", Options: []string("rbind", "ro"}, }) Serve the Model Set the container args to serve the model using Ollama config.Process.Args = [] string( /bin/bash", "-c". strings.Join([]string{ "ollama serve &", "while ollama ps> /dev/null 2>&1; do sleep 1; done", "ollama create modelName + --file/tmp/Modelfile", "tail -f /dev/null", }, "\n"),
April 2, 2025 at 3:40 PM
samuelkarp.com
This talk on LLMs is demonstrating why LLMs should be used with caution; KubeCon and DockerCon are not the same conference.
Slide with the following text: Using Ollama 1. Download and install Ollama from https://ollama.com/download 2. Start Ollama server ollama serve 3. Download your model ollama pull llama3:8b 4. Use the model ollama run llama3:8b >>> What is the Kubecon? KubeCon (formerly known as DockerCon) is an annual conference that focuses on cloud-native technologies, containerization, and Kubernetes... >>>
April 2, 2025 at 3:19 PM
samuelkarp.com
We released containerd 2.0 this week, with new features, now-stable features, and deprecations.

Join me next Friday at KubeCon to learn how to prepare for 2.0 in your production environments. We'll cover what's new and discuss how to detect/remediate any impact of removed features.

sched.co/1i7qL
Promotional card for "What containerd 2.0 means for you" at KubeCon NA 2024 on November 15, 2024.
November 8, 2024 at 12:19 AM