Home/Expertise
LLMOps and self-hosted inference
Self-hosted inference with Ollama, GPU cost control, and the deployment path from a local model to a containerized service. Written from shipped work by Muhammad Huzaifa Shahbaz, AI Systems Engineer at Digital Dividend Global.
What I build
- Self-hosted LLM and speech services (Ollama, Whisper) for products that should not send every request to a hosted API.
- Quantized PyTorch models aimed at IoT hardware, where a cloud round-trip is the wrong dependency.
- The surrounding ops: Docker, Kubernetes, queues, and CI so a model is a service with logs, not a notebook.
Architecture
- At Lenaar, Ollama served self-hosted LLMs and Whisper, with GPU use tuned so inference was cheaper to run.
- ViralMe Studio runs AI inference beside a Remotion render pipeline: FastAPI and Python for model calls, Node workers, RabbitMQ for jobs, Kubernetes on Azure.
- At Digital Dividend, quantized LLMs (PyTorch, Ollama) were prototyped for on-device IoT inference. FunctionGemma is in evaluation for agent-style tool calls on that hardware.
- A public workshop for Karachi.AI walked the same arc: local inference, RAG, fine-tuning on Gemma, then deployment.
Production constraints
- Hosted APIs are simpler until cost, data residency, or offline hardware rules them out. Self-hosting trades that bill for GPU scheduling and model upgrades.
- ViralMe's published target was 99.5% uptime for the editing platform, with render jobs queued rather than run inside the request.
- A quantized model that fits on a device is not the same model that won the demo. The prototype work is explicitly an evaluation, including FunctionGemma.
Stack
Ollama · PyTorch · Whisper · Docker · Kubernetes · RabbitMQ · Azure · FastAPI
Tradeoffs
- Ollama is a fast way to run a known model. It is not a substitute for evaluation, canary deploys, or a queue when inference is slower than the user will wait.
- On-device quantization wins on privacy and connectivity and loses on quality versus the full model used in the cloud product.
Projects
- ViralMe Studio — Ollama inference next to a queued Remotion render pipeline.
External verification
Questions
Has he run self-hosted models in production systems?
Yes. Ollama served LLMs and Whisper at Lenaar, and Ollama is part of the ViralMe Studio inference path. Quantized on-device models for IoT are prototypes under evaluation.
Where is the public record of the LLMOps teaching?
The May 2026 Karachi.AI workshop on local LLMs, RAG, and Gemma fine-tuning is listed on the speaking page, with slides linked from that entry.