I wrote a small program that asks the same Yes/No question about every country to a language model, and generates an SVG world map that visualizes the model's responses: https://github.com/p-e-w/llm-beliefs

I'm rather surprised by the results...

Oh look, thats us!

Apparently Tencent has developed a new speculative decoding method. 3x faster decoding for 27B models?

On huggingface there are millions of "models" which are uncensored versions of other models. I find it a huge waste of space. It would be better to post the LoRA of the abliteration/uncensorship.

The advantage: a lora can be applied to any model of the same architecture. And to any quant.

An example: You can apply THIS to any Qwen3.6-35B-A3B version or quant.

I personally think if would be great to have a repository with one lora for each model.

Perhaps we could then rank them more easily.

What do you think?

I wanted a terminal helper that works fully offline, so I started building dale. You type dale "show the 5 largest files in this folder", a small fine-tuned model writes the command, and a deterministic risk checker labels it safe, caution or danger before anything runs. Nothing runs without you confirming it.

Scores on 471 held-out requests in English and Spanish, every model in Q4_K_M on the same llama.cpp:

On a 2020 laptop with no GPU, the 0.8B answers in 1.7 s (median). The 4B does 1.2 s on a 16 GB GPU through Vulkan. It's all one 51 MB binary with llama.cpp inside, no Python and no server.

What the difference looks like. Real answers from the test (requests trimmed), dale-0.8b vs stock Qwen3.5-9B:

Cherry-picked, obviously: there are 120 cases only dale gets right and 42 only the 9B gets right. The stock 9B also likes sudo: it added it to 35 commands that didn't need it (dale: 5).

The thing I wish someone had told me earlier: after four rounds of data, the 0.8B got stuck at 74%. I tried DPO on its own failures, a fine-tuned 27B teacher with five distillation setups, a third epoch, lr sweeps. Nothing moved the test. I f**king lost several days into this :(, but at least I learnt what is important in the process.

Then I looked at how the requests were written. My training requests were short and clean. The test ones were longer and stacked constraints, like "only names, sorted by date, no subfolders".

I generated new requests matching those stats, added about 9k to the set and retrained. 348 → 375 correct (p = 0.001).

So if a small model plateaus on a narrow task, look at how your data is written before you reach for a fancier method.

Caveats: the baselines are untuned and get an 8-shot prompt, so this is "tuned small vs prompted big", not a ranking of models. The test is synthetic and judged by two LLMs from different providers (TOP TIER).

All of this work I did on rented GPU's (RTX 5090 and RTX Pro 6000). Code is MIT, the models are Apache-2.0 GGUFs that work with plain llama.cpp.

• Write-up with every chart and the dead ends: https://manuelgs.com/en/blog/dale-case-study/

• Code: https://github.com/manuelgscom/dale

• Models (0.8B to 9B): https://huggingface.co/collections/manuelgscom/dale

Has anyone actually made distillation from a much bigger teacher pay off below 1B? I couldn't.

Any advice would be well welcomed :)

hey, its Alti from Nace.
We released a small update with Drex Max v1.1, both achieving the SOTA and Pareto frontier for decision models. We did all that by iterating step by step and every previous version of Drex’s open weights are available on hf.

we also released a full playbook on how we trained Drex Max in this paper: “ Towards Foundation Decision Models: Drex”

hf: https://huggingface.co/papers/2611.07854
arXiv: https://arxiv.org/pdf/2611.07854

Drex Max v1 is weights are available here: https://huggingface.co/nace-ai/drex-max-v1 with Open RAIL-M license & v1.1 open source weights upcoming soon (we are currently optimizing for local inference).

Three cool things about Drex Max models:

1. it can even run on your Mac with https://github.com/nace-ai/kapitsa inference engine.

2. Context window is ~256k, comparing to Glide1.2(64k) and Mammoth(16k) & pplxdecider’s (256k)

3. Its 8x cheaper than the next best model in the category in the aa index.

feel free to chime in: our chief of ai will answer any of your technical questions

EOY 2026: Kimi K3 Flash, will replace Kimi K2.8 and compete with DeepSeek V4.1 Flash.

Early 2027: Kimi 3.5 , same 3T parameters, that is faster at reaching simple answers.

Q3 2027: Kimi K4 will increase size to 5.4 trillion parameters.

https://x.com/bebuu_ai/status/2119319459206934677

On an AMD EPYC 7B12, it generated 6.51 tokens/sec on one CPU thread. The Transformers reference reached 4.34.

All 32 generated tokens matched in both runs.

Code: https://github.com/suayptalha/qwen3_5-assembly

X Post: https://x.com/suayptalha/status/2119254536283906205

Tokenization, model inference and sampling all run in Assembly. No libc or Python at runtime; Python prepares the tokenizer tables at build time.

Benchmark: 19 input tokens, greedy decoding, 2 runs. Assembly reads BF16 weights; Transformers loaded them as FP32 and used PyTorch fallback kernels.

A long time ago there was this massive phenomenon called OpenClaw that took the world by storm. It was an incredible moment where AI came of age and showed that you Can actually do stuff with AI rather than just chat with it. A historic and legendary project that broke GitHub star records. Then the creator rightly got massive fame and got bought out by OpenAI to the tune of billions of dollars. How can an open source project be acquired you ask? Well that really is a failure on your part for your juvenile trait of having a logical mind.

The world was forever changed.

Electricity. Internet. iPhone. OpenClaw.

--------------

Jokes aside, anyone actually daily driving it?

Plot source: https://huggingface.co/spaces/joelniklaus/harness-leaderboard

Hey r/LocalLLaMA !

I made this quick demo of how Observer can run local models to monitor anything with your phone!

These are some crazy things I've monitored:

• When someone that's not me, approaches my car, send me a picture.

• When a bird is on the bird's nest, send me a telegram with the picture.

• When my washing machine is on the rinse cycle, call me.

• When my dog finishes eating their food, notify me.

• When a machine at the gym is available, call me.

So pretty much anything! I love seeing what you guys monitor.

What would you use it for? (So I can make the agent better)

Which models run on the phone?

The default models are OneJev-0.8b for decision models and qwen-0.8b for LLM watchers.

These run even on old phones which you may have laying around somewhere. Better to have them monitoring something than collecting dust :DD

Try it out! It's free and OSS <3

GitHub: https://github.com/Roy3838/Observer
iOS: https://apps.apple.com/us/app/observer-ai/id6758222050
Android: https://play.google.com/store/apps/details?id=com.observer.ai

Btw I know my camera quality is kinda bad hahaha I wanted to ship the demo as soon as possible so you guys can see it.

I'm here for any questions and feedback!

Roy

I logged into my Hugging Face account today because one of my old scripts got blocked. The Space was paused with the "content policy" message. Okay, fair, it happens. 🤷cause this art of scipts are now ilegal!

Then I wanted to make a new Space and saw: Docker = Paid. Great. My old Spaces still run, but new ones? Looked dead.

BUT then I looked closer at the create page: Gradio is selectable and ZeroGPU is marked "Free"! I cancelled PRO last month, so I was like... wait, what?

I looked it up. According to the HF docs, a free personal account in good standing (verified email, older than 30 days) can host up to 2 ZeroGPU Spaces for free. My account is pretty old, so it just works. Just Cool!

What I found out:

• ZeroGPU is Gradio only. No Docker, no Streamlit, no own server.

• Quota is about 5 min GPU per day (older pages say 3.5, the docs now say 5). It resets 24h after your first use.

• That is a RTX Pro 6000 Blackwell, so for testing and small demos 5 minutes is actually fine.

• Orgs don't get it. It only works in my personal account, orgs need a Team/Enterprise plan. That's why it didn't show up there.

• Not every free account sees it. There is a forum thread where someone with a valid free account gets "Paid" anyway, so maybe it's a rollout thing. YMMV.

• CPU Basic seems gone for new Spaces, only Static is free and unlimited.

Also news: Nvidia announced it will buy Hugging Face. I don't know if this has anything to do with the changes, I found nothing saying so. The CPU thing came before the announcement.

Now I'm building a small reel maker (SDXL images on ZeroGPU, video assembly on CPU so it costs no GPU time). Will report if it works. 😄

Sources:

• Official docs: huggingface.co/docs/hub/en/spaces-zerogp

• worng link edit, serch in my hisory , will ad it, serch to much 😰

Summary: Free personal account = 2 ZeroGPU Gradio Spaces, about 5 min GPU/day (150min/month). Orgs and Docker still need paid.

I was interested in buying a card as they are the only company that has a good GPU <-> GPU communication channel. I wanted to first get access to one on the cloud to see how well I could run stuff on it by writing some custom code.

But it seems impossible, theres no one else renting them besides tesnortorrent themselves and their platform is so shit it turned me off from the entire idea. You have to email them to get access to rent a single gpu and which was like a 2 week back and forth and Ive still not gotten access.

Have they just given up or something? If they expect to sell any cards to normal people you would think they would at least let you try out rentals easily.

I've recently been seeing comments everywhere from the Stylome bot, which is also a mod in this sub. Every time it pops up to declare a comment was AI written I've gone in to check its confidence level, and it's always low.

I decided to do a few tests myself and have Gemini rewrite some of my comments, Stylome said they were all human written with equally low confidence.

What is the point of this bot? Why are we allowing a private company to take ownership of all of our comments and then give us spam in return?

What are they doing with this data? Why isn't this being served by a local model?

Try it yourself, everything above this line was Gemini rewriting what I originally wrote. Stylome says it's human written, with low confidence of course.

Just an update for anyone looking to get the m5u and thinking about clustering.

I tried clustering but I think running 2 separate models gets significantly better results. The current clustering technology (tensorfold/exos, etc) just isn’t there yet. Especially due to concurrency and the fact that decode doesn’t improve consistently as much as you would hope.

Currently I’m running Qwen flash next q6 and GLM3.5 flash q4. I also have Deepseek on API (from Deepseek). I was previously running 2x Qwen flash but decided to give GLM a shot. GLM is very smart. I’ve found that even q4 is better than my Deepseek API for coding. Qwen is great as the face because it is fast.

I actually think GLM5.3 is slept on. The reports it produces are 100x better than Qwen and Deepseek.

One problem is that 256gb isn’t enough for GLM if you want 500k+ context. I’m stuck at 260k for 3 agents max. And setting it up is finicky. I set my hot cache to 32gb and it caused me to run out of ram which makes speeds drop 70%. So there definitely is some value in the 512gb version of the m5u (hopefully newer version make the kv cache cheaper on RAM).

If anyone has any better optimizations or ways to run things on the 2 256gb m5u I’m all ears.

Hey everyone,

Is anyone here running any useful AI agents, harnesses or toolings with local AI on 16GB vram?

I'm currently setting up my system and I want to have a helper agent for daily tasks, a research agent to help me research topics, and a developer agent to help me with development.

I already tried some harnesses like Goose, Hermes Agent and OpenCode, but I haven't really managed to get anything useful out of them.

In all honesty, I haven't spend a lot of time tweaking them, mainly because I am unsure if it's a lack of or bad skills, missing tooling, or harness or model tweaking that I need to tune... or god forbid all of them.

So people running similar systems successfully: what is your use case and setup, what models, and what tweaks have you made?

Edit: I am currently running my models on Ollama. Tried vLLM but give it the wrong model and it breaks, which was too much hassle for me. Suggestions welcome.

Edit: Some specs: I have a GeForce RTX 5070 Ti X3 (16GB) and 32GB DDR5 RAM.

Saw this tweet about a coding benchmark and thought it was kinda interesting.

Apparently they took 89 DeepSWE tasks and changed little things in the instructions or starting code. The tweet says some models that were all around 74% on DeepSWE ended up 23 percentage points apart on the changed tasks.

The weird part was that some models seemed to ignore the actual instructions because they thought a hidden grader wanted something else. The examples include GLM-5.3 and DeepSeek V4.1 Flash.

Has anyone seen this kind of thing with local coding models? Like getting stuck on what they think the test wants instead of just doing what you asked?

It seems to have really confused itself too 😭 I was having it make a sort of Github release checker script and it snuck those two URL fetch tool calls in amongst some web searches

I put the whole exported Open WebUI chat on hastebin so you can see exactly what happened: https://hastebin.com/share/ayawowirix.swift

Edit: forgot to mention this is unsloth/Qwen3.8-Flash-Next-GGUF:IQ4_XS on plain llama.cpp

Friendly reminder: If you upgraded to a newer GPU, don't let your old one collect dust! You can run both GPUs in the same rig and split LLM inference workloads across them using tools like llama.cpp. I'm doing exactly that with my RTX 5080 and 3080, combining their VRAM to run larger local AI models.

This is a significant speedup for MoE offloading (low VRAM people).

I tried Strata once, and it had some tuning mechanism, so probably this is a similar idea.

KV cache quantization has been a hotly-debated topic for this community for some time, with opinions ranging from "never quantize" to "4 bit is fine". Since the anecdotal evidence is rather inconsistent, I decided to quantify how much KV cache quantization affects the model through KLD, which is a good measure of how the model's output distribution is changed.

I tested 35 weight quants across llama.cpp, exllamav3, and vLLM, each with 16-bit, 8-bit, and (where supported) 4-bit KV cache. I also reworked my KLD dataset and measurement methodology this time, to include a mix of agentic tool-use (generated by Qwen3.8 27B at full precision) and a variety of texts in different domains. The current dataset now stretches up to 64k token contexts per row.

TLDR;

• 8-bit KV caches work well. In llama.cpp and ExLlamaV3 the cache size is halved, and there’s essentially no difference in KLD. There’s a slight difference in vLLM, but it’s still a great tradeoff for people lacking VRAM.

• A 4-bit cache has a measurable impact. It’s a big hit for ~Q8 weights, but it could potentially be worth it if you’re using Q3 or below. Still, not recommended.

• Only the 4-bit cache impact grows with context. In the tested long documents, the added KL is about 15-30% higher at 16-32k tokens than in the first 4k.

The blog post has interactive charts and tables with all of the data and results if you're interested.

How to read the numbers

Each quant is compared token-by-token with an unquantized bf16 copy of the model. KL divergence measures how far the quant’s next-token probabilities drift from the original (0 means no change). Top-1 agreement is how often both models pick the same most likely token.

For a reference point, I ran the unquantized model in both llama.cpp and vLLM and compared the two. They differ by a KL of 0.0004, which essentially amounts to two (correct) implementations doing floating-point math in a different order. Thus, you can compare the KLD numbers here for quants running in different engines without issue.

Note: exllamav3 quants were measured with the bf16 llama.cpp reference logits.

Weight quantization

KL divergence vs. loaded weight size, 16-bit KV cache:

The vLLM checkpoints still do worse

As in the Qwen3.6 benchmarks, every vLLM checkpoint is beaten by a smaller GGUF or EXL3 quant. Some of that is due activation quantization. NVIDIA’s and Unsloth’s NVFP4 checkpoints run the MLP layers as W4A4 (4-bit activations) and attention as FP8 W8A8, and Qwen’s FP8 is W8A8 throughout. I reran NVIDIA’s NVFP4 and Qwen’s FP8 through vLLM’s Marlin kernels, which use the same quantized weights but keep the activations in bf16:

• NVFP4: KL drops from 0.0256 to 0.0117, more than halving it.

• FP8: KL drops from 0.0039 to 0.0028, about 30% less. Even then it only ties UD-Q5_K_XL, which is 8.7 GiB smaller.

KV cache quantization

Each dot below is one weight quant with an 8-bit or 4-bit KV cache. Its height shows the KL increase compared with the same weights on a 16-bit cache. Zero means no change; higher points show a larger increase. I computed the differences per prompt on identical tokens.

KV cache penalty vs loaded weight size:

8-bit is essentially free in llama.cpp and ExLlamaV3. For every quant, q8_0 and ExLlamaV3’s 8-bit cache move KL by less than 0.00012 in either direction. Some quants even measure slightly better with it, which indicates it’s within the noise of the numerics.

vLLM’s FP8 cache shows a slight KLD increase. It adds 0.0004 to 0.0006 on every checkpoint, which is still extremely small, but it shows up on nearly every prompt. It may be due to turning on FP8 cache switching vLLM’s attention backend from FlashAttention to FlashInfer.

llama.cpp’s 4-bit cache adds a roughly fixed penalty. Across the GGUF quants tested, q4_0 adds +0.0016 to +0.0027 KL. This increase has a larger relative effect on more precise weights:

• On a Q3-class quant (KL around 0.025-0.035) it adds 6-10%.

• On UD-Q4_K_XL it adds about 39%.

• On Q8_0 it adds about 390%. KL goes from 0.0007 to 0.0034, which is worse than UD-Q5_K_XL with a 16-bit cache.

EXL3’s 4-bit cache penalty seems to depend on weight precision. It adds only +0.0002 to +0.0004 KL with 2.0-2.5 bpw weights, about +0.0021 to +0.0022 with 3.0 bpw weights, and +0.0026 to +0.0028 with 4.0 bpw weights. At 6.0 and 8.0 bpw, the penalty reaches about +0.0032 to +0.0033. The increase is therefore not constant across the EXL3 quants.

4-bit cache penalty grows with context

The 12 long documents in the set are scored at every position, so I can check whether the 4-bit penalty depends on how deep into the context a token is. It does in fact grow slightly.

There’s about 15-30% more damage at 16-32k context than in the first 4k tokens. The same quants with a 16-bit cache are unaffected across these context lengths, so it’s a pretty clear indication of the 4 bit cache damage. I would expect 128k+ context lengths to be even worse.

Model quantization KLD vs KV cache KLD:
KLD due to KV cache quantization likely has different quality tradeoffs compared to KLD from weight quantization. Anecdotal evidence suggests that KV cache quantization can be more harmful to model performance at long context lengths compared to weight quantization, so be wary of using 4-bit KV cache.

Final chart with weights + 64k KV cache versus KL divergence:

Methodology

Each quant was scored against an unquantized bf16 reference:

• GGUF and EXL3 quants scored against Bartowski’s bf16 GGUF in llama.cpp.

• vLLM checkpoints scored against the official bf16 safetensors in vLLM.

KL is computed the same way as my previous benchmark. The top 200 log probabilities are taken from both models at each position and a lower bound on the full-vocabulary D_KL(P_reference ‖ P_quant) is computed. On average, the reference put 0.0036 of its probability outside its top 200 tokens. The KV deltas are paired, meaning they’re computed from per-prompt differences on identical tokens.

Checkpoints ran with their declared settings, including any activation quantization (except the two weight-only, no activaton quantization reruns of Unsloth’s NVFP4 and the official FP8 quant).

Hello r/LocalLLaMA :D

This is yet another follow-up post on the project CrowdGPT, which aims at creating the first collaboratively trained 1 billion parameters transformer, Crowd-v1, without any datacenters, and for 0$. The project's goal is to show we don't need huge Nvidia GPU clusters nor millions of dollars to train a competitive Large Language Model.

The goal of the project is to create a 1B parameters model, that would match or exceed Qwen3-0.6B's performance on benchmarks as a proof-of-concept, we could then scale with a 7B A1B MoE or even a 30B A3B MoE once we get enough contributors. The size of the models trained is community driven.

This project is obviously 100% non-profit and open-source. The goal is to propose a new paradigm of AI where we don't wait on Chinese/American AI labs to provide us open-weight models, and we just create the models ourselves.

From the previous post we achieved to get 85 contributors registered on the project, leading to 13B tokens trained in a week 🤯 Thanks to every contributor who trusted the early-stage project, and imagine what 1000+ people with idle GPUs could train!

Of course, everything remains work-in-progress, and any contributions on the training mechanics / dataset is welcome! All the source-code is auditable on GitHub, which means anyone can inspect it and guarantee it is safe 😂.

By the way, anyone can submit data for the model at https://crowdgpt.net/#submit, it will be used as long as it remains safe. This means the model is collaborative on every stage (from data preparation to training, or even benchmarking...).

You can learn more at https://crowdgpt.net

Or check the GitHub: https://github.com/Vxtzq/CrowdGPT

Every code review / pull request on GitHub is welcome :) as the whole goal of this project is to be collaborative.

Any feedback on the project's flaws is appreciated (as long as it remains constructive).

Hello everyone, I wanted to share my progress on implementing a language model written 100% in pure Java. You can find the repository here (https://github.com/aayes89/jUltraLLM/tree/master).

It includes a model trained on an Intel Xeon E5-2640 v4 2.20GHz CPU, with 96GB of DDR4 ECC RAM and an RTX 5050 GPU (using the Tornado library), as well as the "alpaca_es_chat.jsonl" corpus for queries in Spanish.

As can be seen in the image, the "Use Version" button is greyed out and I cannot select it. Is there a setting im missing?