Self-Hosted LLM on a VPS: CPU Benchmarks, Cost, and Privacy

A self hosted LLM can run on a CPU-only VPS and still be useful. The catch is that “it fits in RAM” and “it serves my workload well” are different tests. This report is for developers and small infrastructure teams deciding whether to run local LLM workloads on rented CPUs or buy API tokens. On a 16-vCPU, 32 GiB London VPS, small Qwen models were interactive, mixture-of-experts (MoE) models delivered surprisingly high throughput, and dense 27B models were better suited to batch jobs. At 12 hours of use per day, however, every exact Qwen model we could compare cost less through OpenRouter than the VPS itself. Self-hosting made the strongest case when privacy, control, or provider independence mattered more than token price.

Can a self hosted LLM run on a CPU VPS?

If you want to run LLM locally, a dedicated GPU is not always required. A CPU VPS can handle summarization, extraction, private document processing, low-volume assistants, and background jobs. What it cannot do is erase the cost of context.

Key takeaways

• Qwen3.5-4B offered the best balance of memory, valid output, and responsiveness for light CPU inference.
• Qwen3.6-35B-A3B delivered the fastest 8K result among the current Qwen families because its MoE design activates only part of the model per token.
• Dense 27B models fit in 32 GB RAM but belong in batch or offline workflows on this CPU.
• At 12 active hours per day, exact Qwen API equivalents cost less than the RAM-fit VPS plans we compared.
• A VPS becomes compelling when privacy boundaries, version control, offline operation, or provider independence matter more than token price.

Generation speed in tokens per second measures only response decoding. A real request must first ingest the prompt. That delay is easy to miss with a short chat message and impossible to ignore with an 8K document. In our tests, Qwen3.5-4B generated at 12.23 tok/s with an 8K-class prompt, but the complete 128-token request still took 43.07 seconds. Qwen3.8-27B generated at 2.37 tok/s and took 234.70 seconds end to end.

That makes the decision less like “Can this model load?” and more like four questions:

1. Does the quantized model and its working state fit in RAM without swap?
2. Is time to first token acceptable at the context length you actually use?
3. How many complete requests fit into the hours you plan to run it?
4. Is control of the inference boundary worth more than the API cost difference?

This is a feasibility and economics analysis, not a step-by-step tutorial. Readers searching for how to run a local LLM or how to run LLM locally still need deployment instructions for authentication, networking, and process management; this benchmark answers the decision that comes first: whether CPU self-hosting fits the intended workload at all. The correct answer depends on workload shape and trust requirements as much as model size.

CPU LLM benchmark: what we tested on Arct Cloud

Arct Cloud ran and is publishing this comparison. We conducted the CPU LLM inference benchmark on August 21, 2026, using an Arct Cloud hvm.xlarge VPS (High Performance, 16 vCPUs, 32 GB RAM) in London. The VM exposed 16 AMD Ryzen 9 9950X vCPUs, 31.3 GiB usable memory, no swap, and no GPU. Ollama 0.32.15 was bound to localhost. Only one model and one request were active at a time, using 16 CPU threads.

Each model received a discarded warm-up followed by two measured passes for a short prompt and an 8K-class prompt. Every measured request generated up to 128 tokens at temperature zero. Throughput came from Ollama's tokenizer-native prompt and generation counters; time to first token (TTFT), wall time, actual token counts, and peak process RSS were captured separately.

The tables below use medians, not a selected fastest run. We retained the aggregate measurements and request-level evidence separately so every published result could be checked against the underlying run.

There are important limits to the findings:

• These are single-request, concurrency-one measurements, not a multi-user load test.
• The prompts were deliberately repetitive benchmark inputs, not a production application corpus.
• A 128-token completion does not predict latency for longer answers.
• Two measured passes support a median comparison, not p95 or p99 latency claims.
• The campaign tested speed, memory, and two narrow output contracts; it did not establish equal model quality.
• VPS neighbors, runtime versions, quantization, prompt templates, and CPU allocation can change the result.

Qwen3, Qwen3.5, Qwen3.6, and Qwen3.8 are different model families

The model names deserve clarification before reading the results. Qwen3.5, Qwen3.6, and Qwen3.8 are distinct release families. They are not decimal labels for different sizes of the original Qwen3 release.

Ollama's repository:tag syntax makes the distinction easy to miss:

• qwen3:8b-q4_K_M is the older Qwen3 model with 8B parameters. It is not Qwen3.8.
• qwen3.5:27b-q4_K_M is Qwen3.5-27B.
• qwen3.6:27b-q4_K_M is Qwen3.6-27B.
• qwen3.6:35b-a3b-q4_K_M is the Qwen3.6 mixture-of-experts model with 35B total and about 3B active parameters.
• qwen3.8:27b-q4_K_M is Qwen3.8-27B.

This matters because neither the version number nor total parameter count predicts CPU performance. The older Qwen3-30B-A3B was several times faster than newer dense 27B releases. Qwen3.6-35B-A3B was also fast because only a subset of its weights is active for each token. A reader who treats “3.8” as “3 8B” or assumes “newer 27B” means “faster 27B” will draw the wrong hardware conclusion.

The campaign prioritized official, normally aligned Qwen weights. Matched Huihui “abliterated” packages were included where available to test whether their modified alignment also changed speed or memory behavior. Four popular open-weight alternatives—Llama 3.1 8B, Gemma 3 12B, Mistral Small 3.1 24B, and GPT-OSS 20B—provided external reference points.

We retained the exact runtime aliases, source revisions, parameter counts, advertised context limits, and alignment flags for reproducibility. Primary package pages include official Qwen3, official Qwen3.5, official Qwen3.6, official Qwen3.8-27B, and the corresponding Huihui Qwen3, Qwen3.5, Qwen3.6, and Qwen3.8 GGUF sources.

How the CPU-only LLM benchmark worked

The benchmark separated four questions that are often collapsed into one claim that a model “runs locally.”

1. Can the model load without destabilizing the VPS?

The harness loaded one model at a time and monitored the combined resident memory of the Ollama service and its llama-server runner. The machine had no swap. That was intentional: swap can prevent a clean out-of-memory failure while making transformer inference so slow that the result is no longer operationally useful.

All 24 tested packages loaded and ran within 31.3 GiB usable memory. That includes dense 27B and 32B Q4 packages. Loading successfully was only the first gate, not the final recommendation.

2. How fast does it ingest a prompt and decode an answer?

For both the short and 8K classes, the harness generated a neutral repetitive corpus, changed the first token of every measured request to defeat prefix-cache reuse, and requested up to 128 output tokens. One warm-up was discarded. Two independent measured passes remained for each model and context class.

Ollama reported four tokenizer-native counters used in the analysis:

• prompt_eval_count: actual input tokens processed;
• prompt_eval_duration: time spent evaluating those input tokens;
• eval_count: output tokens generated;
• eval_duration: time spent decoding the output.

From those counters, prompt throughput is prompt_eval_count / prompt_eval_duration, while generation throughput is eval_count / eval_duration. The harness also recorded model-load time, TTFT, wall time, host CPU utilization, minimum available memory, and peak RSS.

TTFT is the user-facing bridge between prompt throughput and generation speed. A model can display an attractive decode rate after it starts answering while leaving the user waiting minutes for a long prompt to be ingested. Wall time includes both phases and is therefore the right input for request-capacity calculations.

3. Does it obey a deterministic output contract?

Speed without usable output can create a misleading winner. Each package therefore received two narrow functional tests:

1. Extract three facts from English text and return one compact JSON object. The result had to parse as JSON and contain the expected keys and values.
2. Implement Python merge_intervals without dangerous constructs or input mutation, then pass five executable assertions covering empty input, sorting, overlap, touching intervals, and non-mutation.

These are smoke tests for serving and instruction-following failures, not a general intelligence benchmark. A 2/2 score does not prove factual reliability, coding ability, safety, or suitability for a specific business domain.

4. Is the advertised long context operationally usable?

Official Qwen3.5 models received a deterministic retrieval corpus with a unique verification key near the beginning and a question at the end. Tests progressed through approximately 32K, 128K, and 260K input targets only if the previous stage completed in ten minutes, stayed below 28 GiB runtime RSS, and left at least 4 GiB host memory available.

This progression gate prevented an advertised context window from turning into an uncontrolled memory or time experiment. It also separated the largest allocatable context from the largest context that can return an answer inside a real service budget.

Local LLM benchmarks: results from all 24 packages

The complete matrix shows why a single “best local LLM” label would be misleading. Small models can be extremely fast but fail an output constraint. Dense large models can fit but respond slowly. Sparse models can outperform much smaller dense packages. Runtime integration can invalidate an otherwise fast model.

Generation throughput

Prompt speed, memory, and output checks

GPT-OSS exposed strong raw throughput but did not reliably return a usable final-answer channel through the tested direct GGUF-to-Ollama integration. Both checks failed at the serving/template boundary. The result calls for a runtime with correct Harmony-format support and a complete retest; it is not evidence that the underlying model has poor reasoning quality.

Several patterns stand out.

First, Qwen3-0.6B's 24.22 tok/s at 8K is fast and memory-efficient, but it failed the non-mutation requirement in the Python task. It remains interesting for tightly scoped, separately validated classification or extraction—not a default merely because it tops the raw throughput table.

Second, the mid-size tier is usable on a CPU VPS. Qwen3-8B and Qwen3.5-9B produced 6.32 and 7.35 tok/s at 8K while staying below 7.3 GiB RSS. Llama 3.1 8B reached 7.00 tok/s but failed the strict JSON contract by returning alternatives and explanatory text. Gemma 3 12B was slower at 5.05 tok/s yet passed both checks.

Third, architecture overwhelmed parameter count. The older Qwen3-30B-A3B passed both checks and delivered 11.70 tok/s at 8K. Dense Qwen3-32B delivered only 1.84 tok/s. Both large packages occupied roughly 19–22 GiB, so RAM alone would not reveal the sixfold decode difference.

Fourth, a high speed number does not repair an integration failure. GPT-OSS 20B reached 13.87 tok/s at 8K, close to the fastest current Qwen package, but the tested serving path did not expose a valid final response for either contract. Operational benchmarks must validate the full runtime path rather than the model weights in isolation.

What the output checks found—and what they did not test

Eighteen of the 24 packages passed both retained checks. The individual failures were more informative than a synthetic aggregate score:

• Qwen3-0.6B and Mistral Small 3.1 sorted the caller's list in place, violating the required non-mutation behavior.
• Huihui Qwen3-14B returned tuples where the contract required lists.
• Llama 3.1 8B returned two candidate JSON objects plus explanation instead of one parseable object.
• Official Qwen3.6-35B-A3B also returned tuples rather than lists in the Python task.
• GPT-OSS 20B did not provide usable final-channel JSON or code under the tested Ollama integration.

All tested Qwen3.5 packages passed both checks. Official and Huihui Qwen3.6-27B also passed 2/2, as did both Qwen3.8-27B packages. The official and Huihui Qwen3-30B-A3B pair passed both, making the older official MoE package a conservative high-parameter candidate when strict output behavior matters.

These tests deliberately avoid an LLM-as-judge score. The JSON response was parsed and compared with expected values. The Python response was extracted, screened, executed, and checked against explicit assertions. That makes each failure reproducible and understandable.

It also makes the scope narrow. The benchmark did not measure general knowledge, hallucination rate, multilingual quality, HumanEval, tool use, retrieval-augmented generation quality, refusal safety, or production prompt-injection resistance. A model that passed 2/2 can still fail your application, and a model that missed a list-versus-tuple contract may perform well after a prompt or schema adjustment. Before deployment, replace these smoke tests with a domain-specific golden set and the exact system prompt, output schema, and documents your service will use.

The 256K context window became a compute limit before a RAM limit

Qwen3.5 advertises a 256K-class context window, but model metadata describes an architectural ceiling—not the latency users will experience on local LLM hardware.

The long-context probe placed a unique key near the beginning of synthetic filler and asked for it at the end. Actual token counts came from the runtime rather than the target label.

Qwen3.5-4B and 9B both recovered the exact key at roughly 32K, but they needed about three and four minutes. Their 128K attempts crossed the ten-minute ceiling. Qwen3.5-27B crossed it at the first 32K stage, so larger stages were not attempted.

The 9B timeout is especially useful because the process peaked at only 11.24 GiB. The 32 GiB VPS had memory headroom; it ran out of acceptable time, not RAM. A feasibility tool that looks only at memory would call this workload a fit while a user-facing service would call it unavailable.

For this CPU and runtime, 8K–16K is a more defensible normal operating range. A 32K document mode can work with explicit multi-minute expectations on 4B or 9B. Promising 128K or 256K would require a different latency budget, faster hardware, a GPU, or a workload that runs asynchronously.

Official and Huihui Qwen models were usually close in speed

Huihui's “abliterated” variants intentionally alter alignment and refusal behavior. They were included as matched engineering comparisons, not as automatic production upgrades.

The older Qwen3 Huihui packages gained between 1.6% and 6.0% in 8K decoding. For the newer families, the difference was nearly zero or negative. Huihui Qwen3.8-27B was 2.2% slower at 8K and used 1.37 GiB more RSS than official, even though it decoded faster on the short prompt.

Abliteration therefore did not create a reliable throughput or memory advantage. Choose a modified model only when its changed alignment behavior is an explicit product requirement, followed by separate security, abuse, legal, and refusal testing. For a public endpoint or ordinary business assistant, official weights are the safer baseline.

How we validated and preserved the benchmark evidence

Several issues discovered during review were corrected in the data rather than hidden in prose:

• Early identical prompts activated Ollama's prefix cache. The harness added a unique first-token nonce, appended corrected runs, and the aggregation selected the retained logical measurements.
• The first memory monitor matched the Ollama daemon but missed its llama-server runner. The process match was fixed and affected Qwen measurements were rerun.
• An unrequested language-specific task was removed. Only the English JSON and Python checks remain in the effective campaign.
• JSON answers are counted only when they parse and match the required values. Reasoning that mentions the facts does not count as a pass.
• GPT-OSS analysis-only output is rejected rather than presented as a valid final answer.
• An older explicit Huihui 8B Q4 tag was removed and the popular retained alias was rerun; its model digest remains recorded in the metadata.

The append-only evidence contains 179 records. After corrected reruns are resolved, the campaign contains 173 effective logical cases across 24 packages. We retained exact runtime aliases, pinned Hugging Face revisions, Ollama digests, GGUF checksums, model parameters, tool versions, inventory, llmfit output, and cleanup verification.

llmfit independently recognized a 16-core x86_64 host with 31.27 GiB physical RAM and correctly predicted that a quantized 27B-class model would fit. Its built-in benchmark measured official Qwen3.5-9B at 7.83 tok/s and Qwen3.5-27B at 2.50 tok/s, closely matching the custom harness's short-context medians of 7.78 and 2.52 tok/s. Its generic catalog estimates were more conservative because they referred to related derivatives and different quantizations.

That makes llmfit useful for initial self-hosted LLM hardware filtering, but not a substitute for measuring the exact package. Runtime, quantization, context length, prompt shape, and chat template all affect the result.

The campaign was automated from server bootstrap through measurement, aggregation, and cleanup so the same sequence could be repeated without changing the test protocol between models.

Inference stayed on 127.0.0.1; no model server was exposed publicly. After collection, downloaded weights and temporary caches were removed, Ollama was stopped and disabled, and port 11434 had no listener. The retained reproducibility records exclude the VPS hostname, address, and credentials.

Local LLM hardware: RAM fit, MoE speed, and monthly capacity

Peak resident memory at 8K ranged from 4.18 GiB for Qwen3.5-4B to 22.90 GiB for Qwen3.6-35B-A3B. Mapping those observed peaks to the smallest Arct Cloud plan with enough advertised RAM gives a practical starting point for local LLM hardware sizing and self hosted LLM hardware requirements:

Prices were checked against Arct Cloud's VPS pricing on August 21, 2026. This is a RAM-fit table, not a throughput promise. All speed measurements came from the 16-vCPU hvm.xlarge; smaller plans have fewer vCPUs and may be substantially slower. Leave memory headroom for the operating system, the runtime, KV cache growth, and your application instead of treating observed RSS as a hard provisioning target.

The same caution applies to self-hosted LLM hardware generally. Model-file size is not peak runtime memory, and an advertised context window is not a guarantee that the longest prompt will complete within a useful time. In this campaign, Qwen3.5-4B and 9B completed roughly 32K-token retrieval tests, but their 128K attempts exceeded a ten-minute request budget while memory remained available. Operational context can be much smaller than advertised context.

MoE models produced up to 14.24 tok/s at 8K

Nominal parameter count did not predict CPU speed. Architecture did.

The 16–18 tok/s results belong to the MoE models, not the dense 27B models. Qwen3-30B-A3B and Qwen3.6-35B-A3B store many parameters but activate about 3B for each token. Their weights still consume RAM, while their active computation resembles a much smaller model. The dense 27B packages must use all their parameters for every token, so they fit in memory but decode at roughly 2.4 tok/s on this CPU.

There is no universal winner in this table. Qwen3.5-4B uses far less RAM and remained responsive. Qwen3.6-35B-A3B led the 8K throughput measurements but used the largest plan and passed only one of two narrow output-contract checks in its tested runtime configuration. Dense 27B models may offer capabilities a smaller or sparse model does not, but their CPU economics are weak for interactive work. Benchmark the answer quality your application needs before trading speed for model size.

Twelve hours of daily inference produces fewer tokens than tok/s suggests

Assume the server generates continuously for 12 hours per day over 30 days:

30 × 12 × 3,600 = 1,296,000 active seconds per month

A decode-only ceiling multiplies those seconds by generation tok/s. It is useful as an upper bound, but it assumes prompts take no time. The end-to-end estimate divides active time by measured wall time, then counts the 128 output tokens and actual input tokens in each complete request.

Decode-only ceiling

End-to-end short-context capacity

The short inputs tokenized to a median of about 747 tokens for these packages.

End-to-end 8K capacity

The gap is the capacity-planning lesson. Qwen3.6-35B-A3B has an 18.46M-token decode ceiling at 8K, yet complete requests produce only 3.09M output tokens because prompt ingestion occupies most of the wall time. For Qwen3.5-4B, the corresponding figures are 15.85M and 3.85M. A tok/s benchmark alone can overstate useful monthly output by roughly four to six times in this workload.

These figures are optimistic in another way: they assume uninterrupted work, no queue gaps, no deployment time, no failures, and no competing services. They are a measured capacity envelope, not an SLA.

Local LLM vs cloud API: OpenRouter and DeepSeek pricing

If the only question is the cheapest way to run LLM locally, the measured answer at this utilization is an API—not a dedicated VPS. We priced the exact OpenRouter model ID where one existed, using the input and output volumes each local model could process in the 8K end-to-end scenario.

This OpenRouter pricing comparison used provider prices listed in the OpenRouter model catalog on August 21, 2026, then added OpenRouter's 5.5% credit-purchase fee. OpenRouter says it passes through inference prices without markup but charges 5.5% (with a $0.80 minimum) when credits are purchased; see its pricing and fees FAQ. Our calculation assumes a credit purchase large enough for the percentage fee to exceed that minimum. The range is the lowest to highest listed provider cost for that exact model and token mix. It excludes cache discounts, free endpoints, per-request charges, taxes, storage, egress, engineering labor, and the value of idle VPS capacity.

Qwen3.5-4B has no exact OpenRouter listing, so we do not substitute qwen3.5-flash and call it equivalent. Similar branding does not establish identical weights, quantization, behavior, or quality.

At this utilization, even the same Qwen model was cheaper through the API than its smallest RAM-fit VPS. This does not prove APIs are always cheaper. A VPS is a fixed-cost resource: its effective unit cost improves when it does other work, runs for more than the assumed 12 hours, avoids idle gaps, or replaces API features that carry separate charges. Conversely, operations, backups, monitoring, patching, and failover make self-hosting cost more than the plan price alone.

A cheaper API alternative widens the price gap

OpenRouter listed deepseek/deepseek-v4-flash-0731 at $0.08 per million input tokens and $0.18 per million output tokens, with a 1.3M-token context window, when checked for this article. Applying those rates and the same 5.5% credit fee to the six local workloads produces a range from $3.81 to $20.75 per month.

That is not an equal-quality model comparison. It answers a narrower economic question: what would it cost to process the same quantities of input and output tokens with a cheap API alternative? Model capability, latency, availability, regional routing, output behavior, and application accuracy still need separate evaluation. DeepSeek API pricing makes the token-cost advantage clearer; it does not make DeepSeek a drop-in replacement for every Qwen workload.

Why run an LLM locally? Privacy, control, and hybrid deployment

For teams asking why run LLM locally when an API costs less, the answer is control. A “private LLM” is not a binary label, however, and a broader self hosted AI stack is only as private as every layer in its trust boundary.

A VPS therefore improves control relative to sending prompts to an inference API, but it is not the same as on-premises data sovereignty. The cloud operator controls the physical host and virtualization layer. Disk encryption, TLS, access control, minimized logs, patching, backups, and key management still matter. For strict residency or regulated data, verify the provider's contractual region, subprocessors, access practices, deletion terms, and incident controls rather than inferring them from the word “self-hosted.”

For an OpenRouter privacy comparison, its documentation says prompt and response content is not stored by OpenRouter unless the user opts in, while request metadata is retained. Requests still pass to the selected model provider, whose logging and training policy may differ. OpenRouter offers per-request and account-level zero-data-retention routing, but eligible endpoints can change and ZDR is a policy constraint rather than local execution. Review OpenRouter's data collection, provider logging, and zero-data-retention controls for the workload you intend to send.

That leaves four strong reasons to choose a VPS for an LLM even when the API is cheaper:

• Sensitive prompts must remain within infrastructure you administer.
• You need a pinned model, quantization, runtime, or prompt template that a hosted endpoint may change.
• The application must keep working during an inference-provider outage or without outbound network access.
• Provider independence and predictable fixed infrastructure matter more than the lowest token bill.

A hybrid design often captures most of the value. Keep sensitive extraction, redaction, classification, and document preprocessing on the VPS; send only approved or de-identified complex work to an API model. This reduces external data exposure without forcing every difficult request through slow CPU inference.

For teams comparing LLM VPS hosting with an inference API, measured memory and latency matter more than parameter count alone. Small dense models are the simplest interactive option. MoE models offer much better CPU throughput when their full weights fit in RAM. Dense 27B models can fit but are difficult to justify for interactive CPU serving. At the measured 12-hour utilization, an API wins on token cost; a VPS for LLM workloads from Arct Cloud wins only when its added control, fixed environment, reusable compute, or privacy boundary is worth the difference.

The practical conclusion is not one universal model winner. It is a deployment rule: measure RAM, TTFT, wall time, and application quality at your real context length; price the same input/output workload through the exact API model; then decide whether the trust boundary is worth paying for.

---

Data note: All benchmark figures were checked against the retained request-level measurements and aggregate results. Arct Cloud plan prices and OpenRouter model/provider prices were rechecked on August 21, 2026. API catalogs and provider availability change. Next scheduled review: September 21, 2026.