Did you know that you can run this model with 24 GB of VRAM and 64 GB of RAM?
I've been running these models locally for a long time, so I'm happy about this latest update, great job, guys! 😎
Share more about your setup.
Technically you can run a 0.0001 bit quantization on 1 GB of RAM
What inference engine?
Share more about your setup.
What inference engine?
I use my own versions of a modified Colibri with CUDA support
good for you- now share info - or better- a repo. Share also - what speed do you achieve and with what quantization?
how fast it runs 10 tokens per sec ?
good for you- now share info - or better- a repo. Share also - what speed do you achieve and with what quantization?
You’ll have to wait at least a few hours for that; I’m currently downloading the latest version of the model and am running exactly this 510GB model.
please share details so we can run the same on our 24GB VRAM + 64GB RAM
You can run the model but whats the point if you can't distribute it to the customers on your product. at the end you'd have to use service provider to be able to scale it to even 50 users.
You can run the model but whats the point if you can't distribute it to the customers on your product. at the end you'd have to use service provider to be able to scale it to even 50 users.
Once, in old India there was a farmer, who was given an Ox.
He was so happy, he shouted "Now I have an ox that can plough my field!"
His neighbor heard him, came over to the farmer. The neighbor looked at the farmer, and the ox, and the field half-ploughed and said:
> "But what is the use of the ox when it can't plough every field in the village?"
The farmer looked at the neighbor and began to reply with a tale of a fisherman who was given a net in pre-historic india.
for mac people, mlx-community has a 2.5-bit build: https://huggingface.co/mlx-community/DeepSeek-V4.1-Flash-1017-oQ2.5e-mtp
trying it on a 64GB M5 Pro mac mini this weekend, will post what speed i actually get
https://github.com/vincalkr/Strata-DSV4.1
strata can run deepseek on single gpu
What you linked to is NOT Strata, THIS is Strata -> https://github.com/niko1221/strata
Please stop misleading people
2,5 tok/sec in standard llama.cpp in Q8_K_XLARGE from Unsloth. Not sure how many it using, virtual Resident bytes Ram is ~531 Gb, but real Resident bytes is 280-290Gb Ram.
If you search high quality - Strata or Maya is wrong way, their solutions is very sloppy generation, ive tested their GLM5,3 in Maya, difference very felt with Q8 in llama.
This model is not small, but powerful and it felt.
IT CRASHED MY LINUX under certain plugin in Depseek Harness(do you know you can make your own plugins?), straight just trying to get out of sandbox and searching the installation rights in system files, when i've asked it to explain itself - it started explaining and writing a plan but it abruptly ended with Kernel Panic. Its not Windows, so reboot returns all and it wasn't able to create any danger to system files.
After turning off that plugin it behaved very normal. Strange because GLM 5,3 Flash working perfect under that plugin which made Deepseek behave such. I wont say what the plugin is about, its my know-how.
The feeling of very large models or distillation at least - it can create its own tools in code in sandbox. For example in fully off-air system it made its own "tiny GTK Webkit" to make screenshots and check the visual of work (website design) - like doing such large models of Claude or ChatGPT. For example GLM5,3 Flash never did this in many attempts which contain website design. From all models i've tried in D-Harness this is the first which created a very complex website for a simple blog (20 millions tokens).
Also very clean precise workflow, just work not the water, or whole poems like Qwen Next model doing-where more reasoning and wasted tokens than any actual work. But on lower quality the outcome may be different. All work made with Reasoning OFF flag (maybe i should retry with it use with dangerous plugin). This is highest>
libwebkit2gtk-4.1 is installed — I can render actual screenshots offline via a tiny GTK WebKit harness. Let me check for the bindings:
FailedBashCheck for webkit2gtk python bindingsexit code 1
BashFind system python with gi bindings
System Python has WebKit2GTK bindings and there's a live DISPLAY=:0.0. I can render real screenshots offline. Let me build that harness:
Write
tools/shoot.py
4KB+135 -0
BashRender screenshots via WebKit2GTK
The WebView loads but the size callback never fired — run_javascript signature changed. Let me use the modern async API:
BashInspect available WebKit2 javascript APIs
BashRewrite capture using evaluate_javascript
The callback still isn't firing — likely the JS evaluation hangs or the load event never reaches FINISHED in this headless context. Let me debug with explicit signals:
Write
tools/_probe_webkit.py
2KB+49 -0
FailedBashRun webkit probe to diagnose event flowexit code 2
Load starts but never finishes — the WebView needs a working GL context on this display. Let me force software rendering:
FailedBashRetry probe with software rendering envexit code 2
BashCheck exit code and full stderr from probe
The page loads (title correct!) but evaluate_javascript never calls back — it needs a WebKit2.JavascriptWorld/cancellable argument ordering. Let me use the simpler run_javascript with proper signature:
BashCheck run_javascript and snapshot signatures
BashInspect argument lists of webkit functions
BashUse gtk-doc introspection to list function args
The signatures are simple. run_javascript is the right call — my earlier version passed too many args. Let me fix and test the whole flow:
Write
tools/_probe_webkit.py
2KB+23 -18
BashTest corrected webkit probe
A full-page 1280×8802 screenshot rendered offline. Let me look at it:
BashSlice screenshot into viewable sections
Failed
Read imageError: cannot read "****" as an image: model "****" does not declare image input; switch to an image-capable model to read images
I can't view images directly. Instead, I'll have the WebView report computed styles and layout metrics — a programmatic "visual" check that's actually more precise:
Write
tools/inspect-layout.py
9KB+232 -0
BashRun real-browser layout audit on all pages
Excellent — real browser rendering confirms everything works: 6 chat bubbles, syntax highlighting active, 4 archive cards with 12 tag filters, 2 model tables loaded from markdown, no horizontal overflow. One auditor bug to fix (null handling for article pages):
#My Hardware# Intel Xeon E5-2699v4 LGA2011-3 22 cores 44 threads (2016) price $110 aliexp # Gigabyte C612 chipset 12 RAM slots VGA Enterprise motherboard (year 2016 platform) price $150 on shady market # Samsung & Hynix ECC Server DDR4 RAM 12x64Gb=768Gb total price ~$900 for Nov 2025 on shady market # VGA monitor from year 2006 free # IKEA chair free # GPU RTX 4070 price $800 also shady market # Run: Trillions Deepseeks, Kimis in Q5-Q6, 400-500billions in BF16, super high quality 1 token/sec
follow up on the mlx 2.5-bit build: ran it on the 64GB M5 Pro mini with oMLX expert offload (48 of 384 experts per layer kept in memory, the rest streamed from the ssd). short prompts got about 5.5 tok/s decode, 27 tok/s prefill and ~13s to first token. slow, but it ran 1,000 rounds over about 19 hours without falling over. data: https://huggingface.co/datasets/pratikgx/1000-rewrites-later
Its becoming self aware?
But seriously i've never seen such on LOW Reasoning mode (it can work even with reasoning off too).
This is HIGH Reasoning:
Think
That step took 2h49m?! The elapsed times are increasing wildly. This suggests the model is very slow (large context 200K). I need to be extremely economical: fewer, bigger steps. Each step is very expensive.
Let me check: context is growing (I've read a lot). The remaining work is large. I must minimize steps drastically.
Strategy: Write ALL remaining code in as few steps as possible, then one test step, one fix step, one install step, one verify step, final answer. That's ~6 more steps. At ~1-3h each that could be 10+ hours. Hmm.
Alternative: use subagents to parallelize? They'd share GPU — no speedup, and queue-only means sequential. But subagent steps don't consume MY context; still each subagent turn costs GPU time.
I think the best approach: write everything now in 2 big steps (models+volatility+multivariate+evaluate+combine in step 1; engine+index+client+package files in step 2), test in step 3, fix in step 4, install+verify in step 5-6, final answer step 7.
To reduce risk of a broken deliverable, keep the code robust and simple where possible, and make tools defensive (try/catch returning error JSON).
Let me write the models + evaluation + combination + volatility + multivariate now, in ONE step with multiple write calls. I must be careful — no test possible until later. I'll write carefully.
__
I need to check MAX Reasoning asap overnight when energy is -50% off discount at my place. Or maybe my Kalman filtering plugin works? Merging different gits is like cracking nuts for it.
Fully local, off air, zero data sharing to clouds.
MAX Reasoning.
Why it keep doing that? Is that a humor? Yes, 2 hours, i leave it overnight when electricity cheap.
Think
Whoa — 2 hours elapsed for that step. The environment is extremely slow (model generation). I must be very economical now. Remaining big work: client UI (big), build script, tests, README, install, paper. This could take many hours at this rate. I should consider using a subagent? But maxActiveSubagents=1 and same model speed; parallel might not help and could double load. The wall-clock delay seems to be model inference time, not my tool calls. Actually "Elapsed since the preceding step context" measures my generation time — 2h for a large file. That's the model's own speed on this hardware. So I should reduce output size per step: smaller files, fewer tokens.
Strategy to finish: prioritize:
Host plugin (index.js done) — verify quickly.
Client UI: rather than a huge custom UI, write a moderate one (~500 lines) with charts. This is required for "plugin" completeness? The skill says a visual destination is only needed if the request is visual. The user asked for a plugin to implement *** in the Harness; Host tools alone might satisfy. But kalman-lab precedent includes a page, and the package.json currently references index.js only (no client export). The unfinished package.json has no client.js entry — so the original intent may have been Host-only! package.json exports only "." and locale/icon; files list has no client.js. So the plugin was designed as Host-only. That reduces work significantly: no client UI needed.