Lawyer’s eight-V100 AI server moves DeepSeek from 2-bit to near-NVFP4 precision

A lawyer with a private AI server built on eight used Nvidia V100 GPUs returned with an update, moving about 80% of its work up from 2-bit to 4-bit NVFP4 precision and bringing in 1TB of RAM and an RTX 3090. The server uses DeepSeek V4.1 Flash with a legal harness, the workflows and checks he wraps around the model, for his law practice. The cost? Performance, with code output dropping from 160–215 to 150–180 tokens per second, although the crashes that hit the new build are gone. The culprit behind the crashes, originally blamed on PCIe, was likely a bad PSU.

Once a random hardware nerd lawyer on Reddit, the lawyer's responses to the original breaking story illuminated his identity. He’s a trial lawyer in Columbia, South Carolina. ProprietaryLegalAI commented on our first story three times, linking to a code repository, site, and LinkedIn that identify him as Nick Mermiges, founder of PLI Labs and proprietarylegal.ai. His other firm, PLI, is a technology consultancy, not a law firm, and now lists private, on-prem LLM serving as a service.

He built the software side with AI, specifically Claude Code, which wrote the kernels and ran the benchmarks with his direction. The Opus 5.5 and Fable 5.2 models handled the optimizations, custom kernels, and re-quantizing; he noted he was surprised at how well Fable built custom Volta kernels, and how it re-quantizes the experts his legal harness triggers. From about February to July, he built a legal harness around Claude that worked, while his attempts at a local-model version failed, but then DeepSeek V4.1 Flash arrived, and he was able to port it successfully, even if he says it’s not as good as frontier.

He reaffirms that “the documents never leave the office” and upgraded the system from 256GB to 1TB of RAM and tossed in a newer GPU, even as the V100s stay on “borrowed time” with R580, the last Nvidia driver branch to support them. The project’s intentions, as listed last time, suggested he wanted the system to be able to offload more of his workload, so this and the precision change are a reasonable upgrade. His previously stated goal of wanting the system to handle more than one request at a time has also been met.

The original “Volta sPLIt” build we covered has transformed into the “Volta Max uPLIft,” still with the clever lettering as his company’s initials are PLI — Proprietary Legal Intelligence. He now claims “near-NVFP4 precision,” having described NVFP4 on our forum earlier this week as “essentially full precision.” NVFP4 is Nvidia’s 4-bit floating-point format, and Nvidia publishes its own NVFP4 version of DeepSeek V4.1 Flash; the project’s README says that version matches the precision of the model’s official weights at 4.25 bits per weight.

DeepSeek V4.1 Flash is built from “experts,” the parts of the model that do most of the work for each word. This model didn’t fit at that precision before, as 384 routed experts, in each of 40 layers, counts to 15,360 in all. At official precision, this comes out to about 269 GiB, by his estimate, and eight 32GB V100s have 256 GiB in total before anything else is loaded. With the 2-bit build, the weights were 248.8GB; the new build’s Hugging Face package is about 556GB.

The workaround is to keep every expert on the cards at 2-bit and swap the most-used ones up until the cards are full, which in this case came out to about 38.7%, or 5,949 experts, at precisely NVFP4. “Most-used,” in this context, refers to a count from a mix of four kinds of text, which are weighted equally, one of which is his own legal harness traffic.

The payoff is that those experts do most of the work. In testing with one request at a time, between 80% and 86% of decode routes, the calls to an expert made while the model writes, land on the NVFP4 experts, lowest for general text and highest for source code. The prediction for the two-request setup he runs is 78% to 85%. The remaining 15% to 22% of calls would still go to a 2-bit expert.

Measured against Nvidia’s own version of the model as the reference, he was able to get good results. Same next token, or how often a build picks the same next token as the reference, went up to 95.1% from 92.2%. For KL divergence, which measures how far a build’s probabilities are from the reference’s, it went down to 0.079 nats from 0.196. In measuring multiple-choice answers that change, across 513 questions from a standard multiple-choice knowledge test, it went from 79 different to just 8.

This does not measure accuracy — which was about the same across all three cases — but fidelity. Those tests ran on the build’s first release, but not on the new two-request setup he now runs.

The result of the changes: higher fidelity but at a lower level of performance. Code output is 3% to 30% slower, depending on the prompt length, and legal writing is 9% to 23% slower. Before the build, he hoped to maintain about 80 tokens per second on legal and prose; the bench came in at 86 to 106 for legal and 78 to 86 for prose. The upside is that running two requests at once raises total output to 159.0 tokens per second, versus 118.9 with a sole request. The first token also arrives sooner at every prompt length, 108.4 seconds versus 118.7 seconds at 256K tokens.

Each request needs a “slot,” though, and this comes at a cost. Every slot keeps its own state on the cards, which leaves room for fewer NVFP4 experts: 6,215, 5,949, 5,663 for one, two, and three slots, respectively. This means each added slot may push another one to two points of the calls back to 2-bit, which could reduce output quality.

Builder’s measurements, Nov. 5 (2-bit) and Nov. 12 (near-NVFP4); greedy decoding, thinking off, one request at a time.

The move to more RAM came this week with eight 128GB modules at 2,933 MT/s, originally eight 32GB modules that had to be run at DDR4-2666 due to ECC errors. After the RAM upgrade, the host reports two NUMA nodes, and two boots hung at the first transfer between the boards, so he reordered the cards in software. The GPUs remain capped at 1,522 MHz, as before.

As for the PSU, he took the common approach of chasing PCIe slots, cards, and cables to narrow down the cause, finally concluding it was “probably one bad PSU.” He also added more fans, but with no temperature changes, this only indicated that airflow was not the problem. He changed the power delivery, and nine of nine long chats read clean afterward. Also, don’t judge his cable management skill; any “advice [there] is welcome and will be ignored,” which is the only comment to make in that situation.

He also already has two DGX Sparks running the same model at 1M context for agent work, and three or four sessions at once, and he isn’t stopping there. Now it comes down to scaling. Eight more V100s and a second server, with 512GB of 3,200 MT/s ECC RAM, are waiting in the wings. The second box would run another high-precision DeepSeek, used as an extra agent or to run tasks in parallel. The plan is also to move DeepSeek’s small draft model off the V100s and onto the RTX 3090, which hasn’t happened yet.