# LLMOps and self-hosted inference — Muhammad Huzaifa Shahbaz

Canonical page: https://mhuzaifa.com/expertise/llmops
Last updated: 2026-09-28
Person: Muhammad Huzaifa Shahbaz (https://mhuzaifa.com)

Self-hosted inference with Ollama, GPU cost control, and the deployment path from a local model to a containerized service.

## What I build

- Self-hosted LLM and speech services (Ollama, Whisper) for products that should not send every request to a hosted API.
- Quantized PyTorch models aimed at IoT hardware, where a cloud round-trip is the wrong dependency.
- The surrounding ops: Docker, Kubernetes, queues, and CI so a model is a service with logs, not a notebook.

## Architecture

- At Lenaar, Ollama served self-hosted LLMs and Whisper, with GPU use tuned so inference was cheaper to run.
- ViralMe Studio runs AI inference beside a Remotion render pipeline: FastAPI and Python for model calls, Node workers, RabbitMQ for jobs, Kubernetes on Azure.
- At Digital Dividend, quantized LLMs (PyTorch, Ollama) were prototyped for on-device IoT inference. FunctionGemma is in evaluation for agent-style tool calls on that hardware.
- A public workshop for Karachi.AI walked the same arc: local inference, RAG, fine-tuning on Gemma, then deployment.

## Production constraints

- Hosted APIs are simpler until cost, data residency, or offline hardware rules them out. Self-hosting trades that bill for GPU scheduling and model upgrades.
- ViralMe's published target was 99.5% uptime for the editing platform, with render jobs queued rather than run inside the request.
- A quantized model that fits on a device is not the same model that won the demo. The prototype work is explicitly an evaluation, including FunctionGemma.

## Stack

Ollama, PyTorch, Whisper, Docker, Kubernetes, RabbitMQ, Azure, FastAPI

## Tradeoffs

- Ollama is a fast way to run a known model. It is not a substitute for evaluation, canary deploys, or a queue when inference is slower than the user will wait.
- On-device quantization wins on privacy and connectivity and loses on quality versus the full model used in the cloud product.

## Projects

- [ViralMe Studio](https://mhuzaifa.com/portfolio/viralme-studio): Ollama inference next to a queued Remotion render pipeline.

## External verification

- [ViralMe Studio](https://viralme.studio)
- [Karachi.AI workshop slides](https://docs.google.com/presentation/d/1v4bkvlg9S9MDF4tHCmzJrHYPbudywQJbIh8pBJneslg/edit?usp=sharing)
- [Speaking page](https://mhuzaifa.com/speaking)

## Questions

### Has he run self-hosted models in production systems?

Yes. Ollama served LLMs and Whisper at Lenaar, and Ollama is part of the ViralMe Studio inference path. Quantized on-device models for IoT are prototypes under evaluation.

### Where is the public record of the LLMOps teaching?

The May 2026 Karachi.AI workshop on local LLMs, RAG, and Gemma fine-tuning is listed on the speaking page, with slides linked from that entry.