The localllama crowd is supremely unimpressed with Intel, not just because of software issues but because they just don’t have beefy enough designs, like Apple does, and AMD will soon enough. Even the latest chips are simply not fast enough for a “smart” model, and the A770 doesn’t have enough VRAM to be worth the trouble.
They made some good contributions to runtimes, but seeing how they fired a bunch of engineers, I’m not sure that will continue.
I wouldn’t call that “large.”
Strix Halo (256 bit LPDDR5X, 40 AMD CUs) is where I’d start calling integrated graphics “large.” Intel is going to remain a laughing stock in the gaming world without bigger designs than their little 128-bit IGPs.
If they wanna abandon discrete GPUs… OK.
But they need graphics. They should make M Pro/Max-ish integrated GPUs like AMD is already planning on doing, with wide busses, instead of topping out at bottom-end configs.
They could turn around and sell them as GPU-accelerated servers too, like the market is begging for right now.
I remember when SBF news was peaking right around the time Stable Diffusion 1.5 came out, and thinking of how fundamentally gutted the entire premise of an NFT was in like a month.
Evangelists of the stuff will tell you that you can own your own digital corner of the information highway (Second Life came out in 2003, and most MMOs have housing), or that you can trade rare items with your fellow players (TF2 and Counter-Strike have been doing this forever). Then there’s this idea that you “own the item” in question more than you would otherwise (you don’t, you own a certificate that’s associated with it, and the item will vanish if the infrastructure does). Then there’s the whole “you could use a sword from one game in another game!” nonsense, which I think we can all agree was cooked up by people who don’t understand how game design works on even a fundamental level.
This is so on point for the web3 space, and parts of the AI space too.
Evangelists waltz in and berate you for not understanding how gloriously awesome their system is… without even making a cursory effort to check if it already exists, much less accumulate a deep understanding and appreciation like they expect you to do.
To actually answer this, you could look into free APIs of open source models, which have daily limits but are otherwise largely catch-free. You could even mirror endpoints on your VPS if you need to, or host “middleware” like prompt formatters and enhancers.
I say this because, as others said, you cannot actually host AI on a VPS…
To go into more detail:
Exllama is faster than llama.cpp with all other things being equal.
exllama’s quantized KV cache implementation is also far superior, and nearly lossless at Q4 while llama.cpp is nearly unusable at Q4 (and needs to be turned up to Q5_1/Q4_0 or Q8_0/Q4_1 for good quality)
With ollama specifically, you get locked out of a lot of knobs like this enhanced llama.cpp KV cache quantization, more advanced quantization (like iMatrix IQ quantizations or the ARM/AVX optimized Q4_0_4_4/Q4_0_8_8 quantizations), advanced sampling like DRY, batched inference and such.
It’s not evidence or options… it’s missing features, thats my big issue with ollama. I simply get far worse, and far slower, LLM responses out of ollama than tabbyAPI/EXUI on the same hardware, and there’s no way around it.
Also, I’ve been frustrated with implementation bugs in llama.cpp specifically, like how llama 3.1 (for instance) was bugged past 8K at launch because it doesn’t properly support its rope scaling. Ollama inherits all these quirks.
I don’t want to go into the issues I have with the ollama devs behavior though, as that’s way more subjective.
It’s less optimal.
On a 3090, I simply can’t run Command-R or Qwen 2.5 34B well at 64K-80K context with ollama. Its slow even at lower context, the lack of DRY sampling and some other things majorly hit quality.
Ollama is meant to be turnkey, and thats fine, but LLMs are extremely resource intense. Sometimes the manual setup/configuration is worth it to squeeze out every ounce of extra performance and quantization quality.
Even on CPU-only setups, you are missing out on (for instance) the CPU-optimized quantizations llama.cpp offers now, or the more advanced sampling kobold.cpp offers, or more fine grained tuning of flash attention configs, or batched inference, just to start.
And as I hinted at, I don’t like some other aspects of ollama, like how they “leech” off llama.cpp and kinda hide the association without contributing upstream, some hype and controversies in the past, and hints that they may be cooking up something commercial.
Nah, I should have mentioned it but exui is it’s own “server” like TabbyAPI.
Just run exui on the host that would normally serve tabby, and access the web ui through a browser.
If you need an API server, TabbyAPI fills that role.
Shrug did you grab an older Qwen GGUF? The series goes pretty far back, and its possible you grabbed one that doesn’t support GQA or something like that.
Doesn’t really matter though, as long as it works!
Your post is suggesting that the same models with the same parameters generate different result when run on different backends
Yes… sort of. Different backends support different quantization schemes, for both the weights and the KV cache (the context). There are all sorts of tradeoffs.
There are even more exotic weight quantization schemes (ALQM, VPTQ) that are much more VRAM efficient than llama.cpp or exllama, but I skipped mentioning them (unless somedone asked) because they’re so clunky to setup.
Different backends also support different samplers. exllama and kobold.cpp tend to be at the cutting edge of this, with things like DRY for better long-form generation or grammar.
So there are multiple ways to split models across GPUs, (layer splitting, which uses one GPU then another, expert parallelism, which puts different experts on different GPUs), but the way you’re interested in is “tensor parallelism”
This requires a lot of communication between the GPUs, and NVLink speeds that up dramatically.
It comes down to this: If you’re more interested in raw generation speed, especially with parallel calls of smaller models, and/or you don’t care about long context (with 4K being plenty), use Aphrodite. It will ultimately be faster.
But if you simply want to stuff the best/highest quality model you can at VRAM, especially at longer context (>4K), use TabbyAPI. Its tensor parallelism only works over PCIe, so it will be a bit slower, but it will still stream text much faster than you can read. It can simply hold bigger, better models at higher quality in the same 48GB VRAM pool.
It’s probably much smaller than whatever other GGUF you got, aka more tightly quantized.
Look at the filesize, thats basically how much RAM it takes.
Try this one at least, it should still leave plenty of RAM free: https://huggingface.co/bartowski/Qwen2.5-Coder-7B-Instruct-GGUF/blob/main/Qwen2.5-Coder-7B-Instruct-IQ4_XS.gguf
Try reducing the context size, and make sure Q8/Q8 flash attention is enabled with flags.
I’d link a specific GGUF quantization, but huggingface seems to be down for me!
The best hope for you is ZLUDA’s revival. It’s explicitly targeting LLM runtimes now, and RDNA1 (aka your 5600XT) is the oldest supported generation.
https://www.phoronix.com/news/ZLUDA-Third-Life
TBH you should consider using free llama/qwen APIs as well, when appropriate.
You can only allocate so much to metal backends, and if you are on (say) an 8GB Mac there won’t be much RAM left for the LLM itself.
But still, use a tighter quantization (like an IQ4 or IQ3_KM) of Qwen Coder 7B, and close as many background programs as you can. It should be small enough to fit.
deleted by creator
Also, AMD is not off the table for multi-gpu. I know some LLM runners are buying used 32GB MI100s.
One can’t offload “usable” LLMs without tons of memory bandwidth and plenty of RAM. It’s just not physically possible.
You can run small models like Phi pretty quick, but I don’t think people will be satisfied with that for copilot, even as basic autocomplete.
About 2x faster than Intel’s current IGPs is the threshold where the offloading can happen, IMO. And that’s exactly what AMD/Apple are producing.