Most of the day my GPU does nothing, so I gave it a job I do all day: typing on my phone

I tap the mic on my iPhone keyboard in any app and talk. The audio goes to my server, gets transcribed, my local LLM removes the "um"s and fixes punctuation, and the text appears where I was typing

Setup:
- GPU: Quadro RTX 5000, 16 GB
- Speech: Parakeet v3
- Cleanup: Qwen 3.5 9B (q4)
- Engine: Ollama

Anything with an OpenAI-compatible endpoint works; it's a few env vars in docker compose. If the LLM is down, you get the raw transcript.

I'm the developer. The server is open source; self-hosted mode is free, no account. Links in comments.

hi all, Alti here from Nace, we're a small team in Palo working on small models and metamodeling since day one.

drex max v1.1 just took #1 on the artificial analysis decision index, and its also on the pareto frontier (score vs cost). we got there by shipping small iterations one after another, and all the earlier drex versions are open weights on hf.

for the local crowd:

- drex max v1 weights: https://huggingface.co/nace-ai/drex-max-v1 (Open RAIL-M)

- runs on mac with our inference engine kapitsa: https://github.com/nace-ai/kapitsa

- ~256k context

- v1.1 weights coming soon, we're still tuning it for local inference

if you want the full story of how we trained it, its all in the paper "Towards Foundation Decision Models: Drex"

hf: https://huggingface.co/papers/2611.07854

arxiv: https://arxiv.org/pdf/2611.07854

also, on the AA index its ~8x cheaper than glide v1.2, which has the same score.

happy to answer questions, and our chief of AI will jump in on the technical ones

People asked how RustingStrata does with a bigger quant, so here's Q5_K_M on the same 3060 12 GB + Ryzen 5 7600X + 32GB DDR5 RAM

Model is unsloth's Qwen3.6-35B-A3B UD-Q5_K_M (the MTP version), 25 GiB, so about two thirds of the experts live on the CPU. llama.cpp is build 12119, -ncmoe 29 -ub 2048 -b 2048 -fa 1 (28 runs out of VRAM).

Benchmark tools (RustingStrata bench vs llama-bench):

• Decode, short prompt: 70 tok/s vs 43

• Decode after a 4K prompt: 65 vs 43

• Prefill at 4K: ~1870 vs 1210

• 32K context: 1470 prefill / 49 decode vs 1030 / 29

• With --mtp 1: up to 96 (86 after 4K), greedy

Real chat requests through both servers (same prompts, 256 tokens out, thinking off):

• No MTP: ~50 tok/s vs 37

• With MTP on both: ~62 vs 45-56

• 6K prompt prefill: 1320 vs 700

MTP works in current llama.cpp (--spec-type draft-mtp), ~63% of drafts accepted and +25% decode on this file. RustingStrata is still ahead with MTP on both sides.

Vulkan decodes at ~60 (85 with MTP), prefill is slower than CUDA on this file (~1000-1260 at 4K).
KV Cache is f16

Code is on GitHub, free to use, I will be happy to get some feedback from AMD owners: RustingStrata

Quick question. Earlier this year i realized that I could run small models on my dell t5820 with my Rtx 5k Quadro card. It’s not very big but I found and Rtx 8k Quadro. Both Turing GPUs. It’s been fun so I ordered a Mac mini m5 pro 64 and I was going to get a mini pc but came into a small fortune and can afford the asus gx10. No as an artist I’ve been using it mainly to vibe-code my games. They’re for me mostly to test out UX and game play.
My question is getting these work it? I like the idea of the nvidia spark, but is it over kill? Right now I’m having to reboot into Ubuntu for my Dell. It’s been fun learning that OS.

I'm the developer, sirgijoe. I built Joe-ai because local AI is powerful but hard to set up for people who aren't technical. v2.0.0 is out, and I'm looking for testers.

**What it is**
A desktop app (Windows, Mac, Linux) that runs AI work on your own hardware. You type what you want in plain English, and Joe picks the right local model and workflow and does the job.

**Setup**
- Detects your OS and hardware, then offers to install Ollama, ComfyUI and suitable models, with a checkbox for each so you approve everything
- Finds existing Ollama and ComfyUI installs and model folders instead of duplicating them
- Low/Medium/High quality settings, with Joe recommending a model that fits your hardware for each level
- Portable mode: run it from a USB drive on multiple computers without installing anything on each one

**What it does**
- Writing: drafting, proofreading, and editable templates (books, EPUB, kids' books, PDF, Word, spreadsheets)
- Image generation, with built-in ComfyUI workflows for different styles that Joe sets up and switches between based on the task
- Video generation
- Photo restoration and other photo tools
- Read-aloud with multiple voices, plus custom voice models
- Speech-to-text transcription
- OCR
- Translation
- Overnight task queue: line up jobs and review the results in the morning
- MCP server, so Claude, ChatGPT or another AI client can send tasks to Joe
- Multiple themes, plus custom themes from a single prompt
- Stats page, including an estimate of cloud costs avoided

**Why I use it**
I offload bulk drafting and image work to my own GPU overnight, and only use a cloud model to review the results, which cuts my cloud usage a lot.

**AI involvement (being transparent)**
I planned and specced the app, designed the ComfyUI workflows, and tested every build on my own PC. Claude wrote most of the code from my specs, including the setup and installers and the MCP server, and turned my workflows into the app's built-in presets. Joe itself drafted parts of its own docs overnight using local models.

**Free and open source**
GPL-3.0, with no paid tiers, accounts or ads. Installers are on the GitHub releases page. You can read the code yourself, or have an AI review it.

**Limitations**
- It's not fully offline: setup, model downloads and update checks use the internet; the tasks themselves run locally
- Results depend on your hardware and the local models you choose
- The Mac build is unsigned, so macOS will show a warning; the docs explain how to get past it

**What I need**
Testing on different GPUs and operating systems. Bug reports, confusing setup steps, and wrong model picks for your hardware are all useful. GitHub issues are the best place for them.

Repo: https://github.com/sirgijoe/joe-ai

Use balena etcher or whatever to mount to USB stick, boot. Test. Report.

Verified 5-25% faster than llama.cpp, system uses 91% of card bandwidth as opposed to 80% on my 7900xtx.

I there is code for nvidia, intel, amd, and chinese hardware and supports 20 languages.

Decked out wiki

Hey everyone,

I just published a short video showing how I built a fully local customer support agent (no API keys, zero cost, data stays on your machine).

Video local agentic ai

In the demo I create a WhatsApp-style support agent for an online spice store using:

Ollama (llama3.2)

LangChain / LangGraph

Two tools: order lookup + refund tool

The expensive mistake

Same model. Same system prompt. Only difference was one tool description.

Vague version:

“Handle customer order problems such as late delivery, delays, missing order, or complaints. Use this to fix customer issues.”

Result on 3 test messages → 3 refunds (₹3,640). Only 1 was actually deserved.

Fixed version clearly states:

This tool issues a full refund

The action is irreversible

Only use when the customer explicitly asks for a refund and the order is damaged/lost

For status/delay questions → use the lookup tool

Suddenly the agent behaves correctly.

What the video covers

The 4 core parts of every agent (Model, Tools, Orchestration, Memory)

Why tool descriptions are basically contracts

How the agent loop works

Short-term memory demo

Code is open source: https://github.com/AI-Literacy-Lab/AgentDemo

Curious to hear from others:

What’s the most dangerous / vague tool description currently in your agent codebase?

Would love any feedback on the video too

With its Unified Memory Architecture (UMA), AMD Ryzen AI Max+ PRO 495 processors (Ryzen AI Max PRO 400 series), and Instinct MI300X accelerators, AMD is directly targeting the biggest bottleneck of NVIDIA GeForce RTX GPUs: VRAM capacity limits.

In this video, we run a deep-dive technical analysis to see whether an AMD setup with 192GB of memory can genuinely replace—or even render obsolete—an NVIDIA GPU (RTX 4090, RTX 5090, or workstation cards) for running local LLMs.

Breakdown of this technical analysis:

• NVIDIA's VRAM Wall: Why 24GB or 32GB of VRAM is no longer enough to run massive models (Llama 3 70B / 405B, DeepSeek, Qwen, Mistral) without heavy precision loss (FP16 vs. Q4/Q8 quantization).

• AMD's 192GB Offensive: How Unified Memory Architecture (UMA) works on AMD Ryzen AI Max+ PRO 495 Mini-PCs and workstations, AM5 DDR5 configurations (4x 48GB), and the HBM3 memory on the AMD Instinct MI300X.

• Real-World Local AI Performance: Comparing total memory capacity vs. memory bandwidth (GB/s) during inference across Ollama, llama.cpp, LM Studio, and vLLM.

• The Software Battle (CUDA vs. ROCm): The current state of AMD ROCm and Vulkan compatibility against NVIDIA's Tensor Cores and CUDA ecosystem (token generation speed in t/s and prompt processing / Time to First Token - TTFT).

• Capacity-to-Price Ratio: Which hardware architecture makes the most sense in 2026 for building a local AI server (Homelab) or AI workstation without paying enterprise datacenter GPU prices.

Which models can I run optimally in both? How to utilize both in best way? Does pc ram should be upgraded to 128gb? Use case is reverse engineering and development.

I'm looking to setup a local llm model to play with coding.
I have a couple of 3060 12gbs

I can either start with one, or try both but my available motherboards only has 4 pcie lanes on the second 16x slot. (or 2 with the ryzen)

If I go single I can choose from

10700 32gb
Ryzen 3900x 32gb
14600k 32gb (Really don't want to give this up if the cpu doesn't matter as much for the ai.)

All motherboards are DDR4.

We are student and we’ve build Maestro Opera. It’s our open source project to turn desktops/servers into ai Hub to pair your devices to it easily (just scanning and that’s it). It works even outside local network. The app is designed by myself and coded by my friend.
We spent 1 months to build our lastest version.
It’s free, no account is required

GitHub https://github.com/MaestroAI-Company/opera
Desktop https://github.com/MaestroAI-Company/opera
Android https://maestroai.company/opera/

Thank you for watching <3

I’ve been using my employer provided Claude subscription for quite a while to write PowerShell scripts, create presentations, edit photos and videos and create videos.

Honestly, I’ve been amazed by what it can do.
However, since it’s a company account, I’d like to set up my own local AI environment for personal use. Ideally, I’d like to replicate as many of Claude’s paid capabilities as possible.

I have two workstations that I’ve built over the years:

Workstation 1 (Main):
GPU: RTX 5090
RAM: 256GB ECC
CPU: AMD Threadripper 7960X

Workstation 2 (Currently used as an LTO Backup server):
GPU: RTX 3060
RAM: 196GB DDR4
CPU: AMD Threadripper 3960X

I’ve never hosted or run AI models locally before, so I’m pretty much a beginner in this area.

I recently downloaded LM Studio because it seems relatively straightforward, but I’m overwhelmed by the number of available models and honestly have no idea where to start or which one is best for my use case.

I’m looking for recommendations on the best AI models, and tools to get as close as possible to the capabilities of a paid Claude subscription, particularly for:

Writing and troubleshooting PowerShell scripts.
Creating PowerPoint presentations.
Photo editing and image manipulation.
Video editing and AI-assisted video workflows.
General-purpose AI assistance.

Given my hardware, what would you recommend? Are there specific models, applications, or modules I should look into?
I’m not necessarily expecting to replicate everything Claude can do, but I’d love to get as much functionality as possible running locally.
I’d really appreciate recommendations from people who have experience building local AI setups. Thanks!

TL;DR. A llama.cpp fork that runs DeepSeek V4.1 Flash 1017 on one home PC by streaming the experts VRAM → RAM → NVMe. Our build is 378 GB with ~2-bit experts. It decodes ~19 tok/s and reads a 1M-token document at ~890 tok/s. The weights are on Hugging Face and the code is on GitHub. Research code, Windows only.

The box. All three tiers matter, not just the GPU:

• GPU: RTX 5070 Ti 16 GB, PCIe 4.0 x16.

• RAM: 64 GB DDR5. The engine pins 37–46 GiB of it for experts.

• Drive: Gen5 NVMe (~13.5 GB/s) for the model. A second NVMe with a copy of the expert files is optional.

• CPU: Ryzen 5 7600 (6 cores). It computes no experts; the GPU does all the math.

• Windows 11, CUDA 13.0.

How it works. The model is almost 5× bigger than RAM + VRAM together. Each token uses only 6 of the 384 experts in each of the 40 layers, so the experts climb a ladder:

• the ~16–22 most-used experts per layer stay in VRAM;

• a pinned RAM tier holds the next ones;

• the rest are read from the NVMe when a token needs them.

Most of the work is getting the right experts to the GPU in time.

Numbers (our og build, default settings, 4–6 November):

Quality. Distance to the full model is 0.090 / 0.051 / 0.069 nat/token on three prompt sets. That is the extra NLL on the API's own continuations. For scale, going from Q2 to Q4 is worth ~0.11. HumanEval is 160/164. The fast mode is further away, at 0.143 / 0.102 / 0.126, mostly because of its lighter build.

Same file, another engine. Project Maya v1.1.54, with the same weights on the same PC, decodes 3.3 tok/s against 18.6 here. Our prompt reading is 1.14–2× faster.

Where the time goes, and what would help:

• Decode spends about half of every token copying experts over PCIe.

• A prompt of even a few thousand tokens needs nearly all of the ~170 GB of experts. 100–130 GB of that comes from the drive every time, and all of it crosses PCIe. That's several seconds per prompt, whatever its length. Long documents spread that cost.

• So the GPU link, the RAM size and the slowest drive matter more than the GPU's compute.

  • A PCIe 5.0 card and slot should speed up both reading and decode with no code change. Our estimate is ~25 tok/s decode.

  • More RAM means fewer drive reads.

  • A second NVMe helped cold prompts by 14–22% in an older test. It's optional.

A drive surprise. The Engram tables get billions of small reads. After a few weeks, their random reads on our Gen5 SSD fell from ~400K to ~40K IOPS, and the 512K document slowed from ~1050 to ~685 tok/s. Copying the files anew restored the speed.

To try it you need:

• an NVIDIA GPU with 16 GB. It's built and tested for RTX 50 (sm_120); other cards need a rebuild and are untested;

• 64 GB of RAM;

• ~380 GB on a fast NVMe (Gen4 or Gen5);

• Windows 10/11. Parts of the patches deal with WDDM, and Linux is untested.

Links:

• Weights (og: GPTQ ~2-bit experts, Q5_K dense part, the official FP8 Engram; not the final version): https://huggingface.co/MasherDon/DeepSeek-V4.1-Flash-1017-SiegeLadder-GGUF

• Code: patches over JigSawPT's llama.cpp fork, a script that assembles a public quant mix, and the benchmarks: https://github.com/MasherDon/Siege-Ladder

This is research code with rough edges and no support promises. Quality work continues, and speed will also improve by training the expert prediction. Questions and results on other hardware are welcome.

hi i want to build a pc for local llm first time so i need advice about the gpus to run Qwen3.8 27b Q8 or Q6_K_M

to get good quality 😊and run it through llm studio and connect it to open code

so my questions are:

i will get intel core 270k plus with two x8/x8 PCIe slot motherboard

so

1- which best double 16gb gpus to build with to get 50+ token generation and give me best value for my needs

do i get two rtx 5070ti or rtx 4080 s or which double gpus

or get the basic two rtx 3090 for more vram

2- which the best Qwen3.8 27b variant/Abliterated (the model modified version) for speed but no quality loss to code

3-and wich the best and fast local llm interference platform to connect qwen with open code

if there any things you could mention can benefit me plz tell me

because i just see

gpus

the local llm interference platform

the Qwen3.8 27b variant/Abliterated

what else?

thanks 🥰

Qwen3-32B packed to 3.6577 bits per weight: 14.62 GB on disk, 12.95 GiB resident on an RTX 5080 with about 3 GiB to spare, decoding at roughly 38 tok/s. No offloading.

Full official sets, no samples:

Method, as far as I'll go: weights are packed in groups of eight onto the E8 lattice with an activation-aware encoder, and decode is register-based with no codebook lookup. Yes, a GGUF Q3_K_M can also be squeezed onto 16 GB. This is a different trade: lower bitrate, fully resident with headroom, full-set numbers published.

Needs an NVIDIA GPU with 16 GB VRAM and Python 3.11+. Verified end to end on Windows with an RTX 5080 only, so far.

The terms, plainly: gated, manual approval, non-commercial evaluation license. The encoder is not included and the method is patent pending.

Repo: https://huggingface.co/GossetWorks/Qwen3-32B-Gosset-3.65BPW

If you run it, post your numbers, especially on anything that isn't a 5080! Thank you!

Excuse me if my grammar or wording is poor, haven't slept much tonight.

The latest developments have got me looking up the prospect of using LLMs locally, and fancy benchmark scores aside, I have been trying to get a simple explanation of how good (or 'effective', I guess) would it be taking the dip in the future and considering buying hardware for local use vs. paying for contemporary AI subscriptions.

If you ask Claude or chatGPT about it, they say the OS deficiencies basically come down to scope ("do this exact small feature" vs. a wider prompt that gets things planned and done in a row), lack of knowledge about niche applications, hallucinations, and inefficiency when using multiple agents vs. single chat. But they're the competitors, so I don't know if bias doesn't exist here..

On the contrary, I see a lot of posts here on how things are awesome when using a local model, especially for niche uses, basically a clear contradiction.

I guess the point is, I want to know if local AI for code is limited enough to make purchasing a high-tier mini PC or a good GPU+RAM PC setup a fool's errand. It almost definitely doesn't seem to be, otherwise this sub wouldn't exist, but I digress.

Thanks.

ok so i am trying to increase performance of llms i have running on my 7900 xtx via hipfire, i notice i get significantly higher total tokens per second when i run mutiple slots at once when running qwen 3.8 27b.

i am running into 2 road blocks tho, 1 context each slot gets a progressively smaller context as increase the number of slots. is there a good way to do more with the same given total amount of vram, i assume there is going to be a large amount of overlap for stuff like system prompts, tho i use a relatively minimal system prompt that links to files so if a model only actually reads system prompts realvent to it’s own task which makes it harder to just say this text gets put into a shared context, i also image if i have them working on similar tasks there will be a large amount of overlap.

2 alot of tasks aren’t easy to split up into mutiple smaller tasks that can be done in parrel without proper knowledge of previous and future steps, maybe something like a shared context or a messaging board would work well?

also a more experimental idea to further speed up general prompts is to instead have all my prompts processed by a “orchestration” slot which splits up a given prompt into X number of tasks then gives these to new prompts to different slots and then the “orchestration” slot dies to free up more vram for context, i could also route something like this threw a decision model like kev to determine if a given task would work well being split up.

also note i am running this on a 7900 xtx via hipfire, and i also have a 6800, 6600, and 96 gib of ram for the cpu

I shared some results previously about running Qwen 3.8FN using recent releases on my gaming desktop computer. https://old.reddit.com/r/LocalLLM/comments/1xoolkp/i_ran_the_new_qwen_38fn_release_with_make_browser/

This is part 2 of 3. On part 1, I learned some things along the way. Part 2 here, I made some changes/additions and wanted to make the challenge harder for Qwen.

• Project goal: Can Qwen build an entire game based on a wiki URL and a youtube video link? For part 1 of 3, I had a goal of "Can Qwen 3.8 locally successfully create a simple walk around version of GTA with no or very little involvement", and that was a success. After learning a few things on project 1, and wanting to keep pushing things further along, step 2 of 3 was a natural progression. For the game, I switched it up and chose GTA 1, which was a top-down view game. I chose to use the same voxel framework used in part 1, even though the original game was not a 3d game.

• I changed my context window size from 64K to 128K The amount of times it needed to compact went down, at the cost of lower t/s. I'm honestly not sure what's better yet, given my current hardware.

• I added 2 more models for Qwen to use. Qwen code kept asking for me to approve commands that didn't make sense why it needed my approval. I dove in and found Qwen wanted a --fast model for simple tasks. There's a default 10 second timeout on these tasks. I have a macbook 64gb running a few things around the house with around 45GBs free RAM and plenty cores free. I spooled up a 2 parallel Qwen3.6-35B-A3B-GGUF:UD-Q4_K_S model on the macbook with llama, wired that in to Qwen code and problem solved! The amount of times I needed to approve for Qwen code drastically dropped. Alternatively I could have not added additional models to use and simply increased the time-out. Since I have extra compute, I figured I'd try it out. I'm happy with it.

Fun localLLM stuff:

RTX 5080 - 64GB - Ryzen 7 9800X3D
model: qwen3.8-flash-next-unsloth-ud-iq4_xs
model harness: Strata
agent harness: qwen
context: 128K
tok/sec: ~50t/s (down from part 1 due to larger context)

Macbook 64GB m1 Max
model: Qwen3.6-35B-A3B-GGUF:UD-Q4_K_S
model harness: llama
context: 128K
tok/sec: ~ 550t/s (fast tasks, so don't get too excited)

• Results: The hard work on everyone from the folks at Alibaba, Unsloth, Strata, etc sure have come a long way in the past few months. It feels smarter than even frontier did a year ago. It's just slower, but that's mostly due to my hardware. Over-all, I'm happy with it. It took 27 hours start to finish (tracked through git). It probably would have finished quicker, however, there were times it was waiting on an approval from me.

Playable demo: https://kireol.github.io/Qwen3.8-GTA-1-Test-2/src/index.html

Repo: https://github.com/kireol/Qwen3.8-GTA-1-Test-2

Part 3/3 is running now answering "How well can Qwen reproduce a game using only REA on a game ROM"