Stop downloading the same model twice. Do this instead
tldr: Every small model tweak ships as another full copy of the weights, so you store and download terabytes twice for a few changed vectors. Ship the original plus the change instead – binary diff today, LoRA file tomorrow, per-request patch in the dream.
The current practice is horrendous and redundant and should never have happened: you change a couple of vectors and you have to redownload the entire fucking model.
Consider that for every release – you have to download the original and uncensored version.
For each model you hold the original one and the modified one.
Twice the fucking size, twice the download – terabytes again and again and again for a difference of a few vectors you need in 5% of workloads while the rest of the time it just sits there doing nothing.
Wasted disk, wasted bandwidth, wasted time.
Stop doing that. The only time a full second copy is justified is when the change is really big – a full fine tune. Everything else ships as original plus change.
I’m trying to convince everyone to stop doing that and propose better methods to solve this issue.
HF suffers under all that wasteful space we have to host, disks which would be better used for actual datasets.
All because we follow this wasteful practice. They carry the financial cost of this, and if they were not so responsible, they might have kept their independence still. It is partially both our problem and their problem that we never established the practice of shipping only the changes, and everyone resorted to shipping full copies instead. For a long time we blindly allowed that practice to become the norm. And yet we can solve it together.
Why our inference engines still do not save us
I may be wrong here. I base this on how prevalently everyone on HuggingFace ships a model copy instead of shipping vectors – I assume LoRAs and steering vector snippets, the mini components, just don’t work reliably on inference engines, else we would not see such a horrendous practice.
It is a question of why our inference engines still do not support the rich ecosystem for modifying model in place, so any modification operates as original plus the change and not another fucking copy.
Because our inference engines did not create enough tooling to support all of this, it is unreliable and questionable, and everyone resorted to just shipping a copy of weights. That should happen only for a full fine tune, not for changing a couple of vectors.
Transformers lib can do it but slow. That is fast – no one cares.
What follows is how I see it – I haven’t tested it. Snapshot of early 10 2026.
The state of engine surfaces right now:
| llama.cpp | vLLM | SGLang | |
|---|---|---|---|
| multi-LoRA | yes native | yes native | yes native |
| per-request scale | yes /lora-adapters | yes per req "model" | yes lora_path |
| dynamic hot-swap | yes POST | yes (+sec. flag) | yes /load_lora_adapter |
| ACTIVATION steering | yes control vector | no hooks only | no |
| soft prompts | no | no removed | no |
| logits/grammar control | yes GBNF, bias | yes structured_out | yes |
| adapter-vs-KV guard | ptr + alora | yes hash incl. path | no (Unified Paging pending) |
| weight-space patch | no | no | no |
That last row is the gap.
Previously in what you are evaluating wrong, I hinted that you have a control layer on the inference itself, which stands on API level. A great development, which would allow us both to save space and get more choices, more controllability and all that.
But wait, here is what I am talking about. LLMs in particular lag here, while the image generation field is much more mature in that sense – there supplying LoRAs is the actual practice everyone follows, and full weight redistribution is rarer.
How it should be done ideally
Therefore, I have a solution, I made a tool and I have outline for the actual path forward. The goal as always is to ship only one changeable part and not the whole thing.
Level 1: small change applied straight in the engine
I do not trust LoRA implementations yet. It has a limit in expressiveness, something is going to be fucked up, something strange with how it is loaded – so I do not trust this. That is why this level is not what we do first – nobody pays it enough attention for it to be viable yet.
Then you take the small part which contains the change and apply it right in the inference engine, which supports it – swappable, configurable, customizable with API and all that.
If we could just ship LoRA, a small file with exact vec changes to be used in the inference engine directly. A binary diff file could take up to 10% of total weights and that is again just a redundant copy when just a couple of floats changed!
If anyone could make it work, that would be just great. You never load a second fucking model, you load a couple megabytes of file. Then you just add it as an additional argument at start and call it a day.
But that again is just LoRA modification on start. Here the restart typically requires you to drop the cache. Most of all it is the cache drop, because I doubt you could carry the cache over to be reused. Or maybe yes – that needs testing: whether you can restart the same model, the same class with only a small modification, and yet keep the cache intact.
Same idea but if inference engine would be able to apply lora to each individual request without dropping KV cache. This way you could setup your harness to operate a model normally but when needed – supply api param to an engine to apply lora to this/next requests.
This however seems almost impossible, as a full weight adapter ruins attn scores. You modify the geometry directly, which massively shifts the cache, so it breaks. Even if it has a margin of error – I suspect that margin is exactly what makes reusing cache impossible. It reuses the cache geometry from before, and therefore adding the entire LoRA is just pointless.
So what works is possible without full weight reload, as if reading from a diff folder but with the cache flushed.
Even so the trade-off seems justified. Your full adapter changes the behavior of the model – in a long trajectory you already hold the cache from all the previous interactions under different behavior. You might even want to flush the cache and recalculate the scores, so model and cache align entirely with no artifacts negating your change.
Level 2: materialize the custom weights in memory
Then you take the same small file and materialize the custom set of weights right in memory, without materializing them on disk.
This however requires engine modification, because none of them can load from memory. You can do virtual RAM-fs, but most of them load as if from disk and copy from disk to memory. You can do it only if you really have double memory to temporarily hold both disk and engine copies.
Level 2.5: custom script that makes it on disk and kicks the engine – you can do this today
When you have your own custom script that makes it on disk and then kicks the engine. Sub-ideal when you have SSD or something, decent if HDD. But this is one you can do today already.
So first we go for brute force difference.
- Get original safetensors
- Produce modified full set of weights
- Calculate the diff against each .sf shard
- Save as clickable script, discard [2]
Thus, at any moment you can recreate the modified set of weights by storing just the original weights and a single binary file. When you need it, you create it immediately, load it and then discard after you are done.
This could be modified: instead of a full binary difference between the weight files, you have exact vectors, which you load and modify and save again. However, that requires the torch library and potentially adaptation for each model you run. That method might not be universal.
This however requires you to ship not only the safetensors file with exact changes but also a script to apply them. Increases complexity but doable and preferable. Important: for the consumer it must be one exact process – run a script -> get 2nd set of weights
Quantization works the same
- Have the original + modified on disk
- Create quant config (layer wise) – this part must not change between the two versions.
- Do both quants with same config, make diff
- Same – keep a single quant only and materialize 2nd on demand
However, the quantization here has a massive limitation: you have to quant both versions yourself with same config and potentially even same llamacpp commit. Because any kind of drift between quantizations produces a rapidly different change.
You cannot take an unsolved quant, apply a patch and think it will work. No. The original quantization and the patch must come from the same source, who did it and deployed it. However, we could potentially share just the quantization config, and if we have different strategies to quantize as opposed to just default ones, then we could share the config and you can most likely reproduce the same quant.
Also, many changes could be applied to quant anyway – like nvfp4, which typically requires effort for matrix recalibration. The shipped model is not a quant done in a couple of minutes, it is a long process, so this model becomes a full derivative in itself and not a process you can replicate. But you can still patch the change on top anyway. That also exists in parallel because many people download the quant directly and don’t bother quanting themselves.
Level 3: permanent second set from original plus patch – the tool I made
When you have originals plus a patch and you create a permanent set of weights – whether original plus patch modified, or the quant of the original plus a modified quant, which also has limitations, since you have to quant it yourself or it has to come from one place. But this is the tool we have already, and here is the tool I made for this: (model-patch).
In the ideal world, nobody would ship the same twice – only the small part…
But what I want is important: for the end-user all of this manipulation is very clear and straightforward. Have the original in a folder, place the patch nearby, click a script, it just copies. Very easy. I have the option which allows per-layer modification, but that requires the user to have python with transformers installed and all that. And pure, which operates only on binary level, but is more sensitive and can potentially fuck up. So test it yourself first, as in my method.
What new pattern unlocks in future
When we have custom scripts applicable to inference engines, to modify as I would with transformers lib.
I mean real-time LoRA modification. You keep one set of weights but dynamically apply what you want from a set of options.
What the future holds when you have transformers-level modification of parts but on a modern inference engine.
Currently you have to compromise: you want dynamic steering vectors, you have to do it slow. A false dilemma – nothing stops you from supplying a script for vLLM for example, and using that as a filter dynamically enabled with API.
That unlocks possibly the actual dream of model control.
When you could choose not only sampling params – which are getting less and less relevant each day – but also flexible behavioral modifications on demand.
Once engines start taking this seriously, it gets mature, flexible, supported everywhere – more choices, more customization, same knobs on every public API. The end goal is a mini code module you ship next to the weights – the same control you have in transformers sleeve today, but fast and toggled per request.
My suspicion here, and I have not tested it: per-request LoRA without dropping KV cache sounds great – run the model normally, pass an API param to apply the adapter to this or next requests – but a full weight adapter ruins attn scores. You modify the geometry directly, you massively shift the cache so it breaks. Even the margin of error that lets it limp along is exactly what makes reusing cache pointless, because you reuse the old geometry and the LoRA does nothing. What works is without full weight reload, as if reading from a diff folder, but with the cache flushed. Even so the trade-off feels justified: in a long trajectory you already hold cache from previous turns under different behavior, you might even want to flush and recalculate so model and cache align with no artifacts negating your change.
And the real end of this is not a predefined set. Not something you prepared beforehand and tested. You modify based on your current request – ideal is to calculate the change based on what you sent.
Stuff that should have been able to be just prompted – we would have to do here instead.
Starts with you – stop uploading redundant crap
It’s not damn rocket science. If you can run command line to start the model, you can very well solve this part too.
What I ask of you is: if you do modified models, stop uploading them as a whole. Then you can ship only the difference.
The benefits are not just space saving, but also the growth of communities, growth of reach, of variety and all that.
The model customization is still not as prevalent and not as commonplace precisely because of space and bandwidth problems.
If you learn how to supply just a small artifact, we could proliferate it at much faster speeds. Currently, if you do just a small change, you still have to upload the entire thing to Hugging Face. And everyone has to go from that.
But if you learn to do only a small patch, you could just share it in Discord, drop it in the chat, just host it yourself on any drive. This way we could reach more people, enrich the scene and generally make the practice very common.
And when the technology penetrates, it makes it resistant to shutdown. So the AI stays yours, because the tools to modify and change it how you want are everywhere and prevalent.