I built my own GPU cloud, badly, and it mostly works
Running my own GPU inference on rented hardware instead of an API. Cold starts, cost per hour, and whether it was worth it.
I run model inference on GPUs I rent by the hour instead of calling an API.
The control plane is a free-tier ARM box on Oracle Cloud. It holds the queue, the registry and the scheduler. The GPUs are elsewhere: a rented A10 at roughly $0.75 an hour, plus a 3090 under my desk. They join over a private network. Nothing is exposed publicly, and the control plane never needs a public IP for them.
Running the model is one command: ollama run. The work is everything around it. Which machine takes this job? Is that machine still alive? How do I stop paying for it when nothing is queued?
Cold starts
The number that matters is time to the first token.
A warm worker with the weights already in VRAM answers in a few seconds. A worker with the weights on local disk has to load them first. For a small model that takes tens of seconds. A worker that has to pull the weights takes minutes.
So the scheduler prefers a machine that already has the model. Next it tries a warm machine. Only then does it wake a cold one. Most of the engineering is in avoiding the third case.
What it costs
An A10 at $0.75/hr, run four hours a day, is about $90 a month. Comparable API usage would have been somewhere around $300. Call it $200 a month saved.
I spent something like 40 hours building it. At $200 a month, that takes a year to pay back at $60 an hour.
I also got other things out of it. I can run models nobody serves. I can fine-tune on my own data. And I know what the pipeline does, because I had to build every part of it. That last one is why I built it.
Would I recommend it
If you want to summarise documents, use an API. It’s faster, and you’ll save the 40 hours this took me to build.
If you want to understand the machinery, or you have data that can’t leave your building, this is what it takes.