

What I am talking about is when layers are split across GPUs. I guess this is loading the full model into each GPU to parallelize layers and do batching
What I am talking about is when layers are split across GPUs. I guess this is loading the full model into each GPU to parallelize layers and do batching
Can you try setting the num_ctx
and num_predict
using a Modelfile with ollama? https://github.com/ollama/ollama/blob/main/docs/modelfile.md#parameter
Are you using a tiny model (1.5B-7B parameters)? ollama pulls 4bit quant by default. It looks like vllm does not used quantized models by default so this is likely the difference. Tiny models are impacted more by quantization
I have no problems with changing num_ctx or num_predict
Models are computed sequentially (the output of each layer is the input into the next layer in the sequence) so more GPUs do not offer any kind of performance benefit
Ummm… did you try /set parameter num_ctx #
and /set parameter num_predict #
? Are you using a model that actually supports the context length that you desire…?
We don’t, I already have a steam deck. Touchpads.
The device in the OP is not the steam deck
Absurdly massive and yet no space for some touchpads? 🤡
Feels good to have never owned a razer (or logitech) product
My guess is an x86 32bit machine
4690k was solid! Mine is retired, though. Now I selfhost on ARM
Okie dokie then!
continues enjoying the Steam Deck
🙂
Hey that’s great that Arma III mods also are realistic! Love to see it. Star Wars Battlefront mods also came close to photorealistic at moments. I’m sure these community made-by-volunteers-in-their-free-time mods were SO expensive to develop that it spurred a handheld gaming conspiracy, right???
Those dastardly indie teenager devs!!! 🤪
Described as “the first photorealistic shooter” by reviewers, there is quite literally no other game that comes close, in terms of graphical realism.
https://www.youtube.com/watch?v=SzHfZYClTwo
https://gamerant.com/most-ultra-realistic-games-like-bodycam/
The game Bodycam was developed by 2 people, a 17 and 20 year old, and has the most realistic graphics I’ve ever seen.
And they’ve already lost lmao
You can overwrite the model by using the same name instead of creating one with a new name if it bothers you. Either way there is no duplication of the llm model file