Running DeepSeek-V4.1-Flash-1017 Across Two NVIDIA DGX Sparks

I deployed DeepSeek-V4.1-Flash-1017 across two NVIDIA DGX Spark systems and connected its OpenAI-compatible API to Open WebUI for use on a research laboratory network. The deployment uses a 2.9-bpw EXL3 checkpoint and a two-rank TensorFold inference recipe over the two RoCE rail interfaces exposed by the ConnectX-7 interconnect.

This article separates observations from my installation logs from sanitized example commands for readers. It covers deployment, the problems encountered, routine operation, and the checks that established a working service. The inference recipe and engine are upstream projects, not my own implementations. My contribution was the hardware and network preparation, deployment, integration, troubleshooting, and validation in the laboratory environment.

The companion article evaluates PDF text and table understanding, figure-dependent questions, and the limitations of the tested document-processing workflow.

Scope and evidence: The setup and observations date to November 2026. The checked-out MiaAI-Lab two-Spark repository was at commit c7f2b85100f0a7b099c92a4e21825e30d93c9c51. Output excerpts and measured values are from the recorded deployment; some commands below are reproduction examples, not literal terminal-history transcripts. The addresses belong to the RFC 5737 documentation ranges and will not route as written. Do not apply NetworkManager or NFS changes without checking your interfaces, routes, export configuration, and recovery access.

1. Hardware and deployment topology

The upstream recipe and local configuration designated the head as rank 0. It manages both ranks and serves the API. The worker hosts rank 1 and, once the model has loaded, also runs Open WebUI. Other existing worker services were left untouched.

The two RoCE rail interfaces used separate IPv4 subnets and were configured independently of the management LAN. They were presented over the ConnectX-7 interconnect; this should not be confused with two independently benchmarked physical links. The following RFC 5737 example addresses illustrate the topology only; the actual laboratory network used different private addresses:

These are RFC 5737 documentation ranges, not routable addresses to copy into a functioning deployment. Replace all of them consistently with your own management and fabric subnets. Figure 1 omits addresses entirely so that it can be shared independently of the configuration examples.

Figure 1. Deployment architecture. Site-specific IP addresses and credentials are deliberately omitted. The web interface and API use the management network; the two inference ranks communicate over RoCE, and the worker mounts the head model cache read-only.

DGX Spark uses unified memory. The model load therefore competes with desktop applications and other processes for the same system memory. In my installation, the startup script required at least 112 GiB of MemAvailable on each node before launching. Keeping enough memory free was more important than raising concurrency.

2. Inspect both nodes before changing anything

I checked both systems rather than assuming they had identical configurations. This is a representative read-only preflight checklist, not a claim that every command was issued in precisely this order:

hostname uname -m cat /etc/os-release free -h df -h / docker --version docker info --format '{{json .Runtimes}}' nvidia-smi -q -d CLOCK ip -br -4 addr ip -4 route 

I verified that the head could reach the worker over key-based SSH and that both RoCE subnets had peer connectivity. The commands below show how to repeat part of this check:

# HEAD — use your real worker rail-1 SSH target ssh -o BatchMode=yes "USER@<WORKER-RAIL1-IP>" true ping -c 4 "<WORKER-RAIL1-IP>" 

On each Spark, the ConnectX-7 interconnect exposed two RoCE netdevices/rails. The successful startup subsequently reported rocep1s0f1,roceP2p1s0f1, using RoCE v2 GID index 3 for both ranks. These names and indices are hardware- and configuration-dependent; do not assume they will match another Spark.

The upstream requirements recommend matching GPU clock ceilings between nodes. I did not introduce a clock-setting command as part of this deployment procedure, and I did not independently benchmark the effects of clock changes. Readers should inspect their own system settings before applying tuning recommendations from the upstream documentation.

3. Configure the second RoCE rail

Rail 1 was already configured through the existing network setup. I retained it and modified the existing NetworkManager profile for rail 2 on both nodes. This avoided replacing the management network or editing an unrelated interface.

Before changing the rail-2 profile, I saved the existing profile and checked the routes. For another installation, the equivalent checks are:

# RUN ON EACH NODE, preferably from a local console nmcli -f NAME,UUID,TYPE,DEVICE connection show ip -br -4 addr ip -4 route nmcli connection show 'Wired connection 2' \ > "$HOME/rail2-before-change.txt" 

On the head, the rail-2 profile was named Wired connection 2. Its name and interface mapping must be checked separately on each host. The following is a reproduction pattern, not an exact copy of the commands originally typed; it assumes that the profile has already been mapped to the correct physical rail-2 interface:

# HEAD, if the selected profile really belongs to rail 2 sudo nmcli connection modify 'Wired connection 2' \ ipv4.method manual \ ipv4.addresses '<HEAD-RAIL2-IP>/24' \ ipv4.gateway '' \ ipv4.never-default yes \ connection.autoconnect yes # WORKER: same properties, but <WORKER-RAIL2-IP>/24 

I applied the relevant profile from a console, then checked the addresses, routes, and bidirectional reachability. The second rail was configured without a default gateway, and verification confirmed that the management default route remained in place. NetworkManager profile activation can disrupt connectivity; retain a copy of the original profile and verify the intended interface before applying changes.

The successful TensorFold log showed both rails initialized:

RDMA rank 0 peer 1 rail 0: gid_index=3 path_mtu=3 RDMA rank 0 peer 1 rail 1: gid_index=3 path_mtu=3 

The rail-2 preflight also confirmed peer reachability with MTU 1500 in this installation. These checks establish initialization and connectivity, not measured end-to-end RDMA throughput.

4. Configure model storage and read-only NFS

The recipe needs substantial local storage. Its pinned requirements specify roughly 505 GB plus a Docker image on the head, and 310 GB plus the image on the worker. On the head, I used a separate Hugging Face cache:

/home/<USER>/.cache/dsv41-hf 

The Hugging Face CLI was available from a dedicated Python environment; the installed CLI reported version 2.4.0. I used that environment's CLI for preparation rather than replacing the system Python installation.

The worker reads the head's model cache over read-only NFS during setup. On the head, I checked the existing NFS configuration and installed nfs-kernel-server. The final export was confirmed in two ways: the configuration entry existed at /etc/exports.d/dsv41.exports, and sudo exportfs -v showed it active. The following is a sanitized reproduction example, with the same export options but placeholder addresses and usernames:

# HEAD — replace user, cache path and worker fabric IP # Review existing exporters and entries before running these commands. sudo apt install nfs-kernel-server sudo install -d -m 755 /etc/exports.d printf '%s\n' \ '/home/USER/.cache/dsv41-hf <WORKER-RAIL1-IP>(ro,sync,root_squash,no_subtree_check)' \ | sudo tee /etc/exports.d/dsv41.exports sudo exportfs -rav sudo exportfs -v 

The values in angle brackets must be replaced before running the command. Existing exports should be inspected first; avoid creating a second NFS server for a cache that is already exported.

During setup, an initial attempt to write the export file failed because /etc/exports.d did not yet exist, and an early worker mount returned access denied. The final verified file contained this entry, using the deployment's actual directory and worker fabric address (redacted here):

/home/USER/.cache/dsv41-hf <WORKER-RAIL1-IP>(ro,sync,root_squash,no_subtree_check) 

exportfs -v independently confirmed that the export was active with ro, sync, root_squash, and no_subtree_check (alongside system-added options). A worker-side read-only NFSv4.2 mount succeeded. The following is a sanitized example of that verification procedure, not a literal transcript of the original command:

# WORKER — temporary verification mount; use real host/path values sudo mkdir -p /mnt/dsv41-nfs-check sudo mount -t nfs4 -o vers=4.2,ro,nosuid,nodev,noexec \ "<HEAD-RAIL1-IP>:/home/USER/.cache/dsv41-hf" \ /mnt/dsv41-nfs-check findmnt /mnt/dsv41-nfs-check ls -la /mnt/dsv41-nfs-check sudo umount /mnt/dsv41-nfs-check 

That distinction mattered: an active NFS service did not prove that the worker could read the intended export. At the verified point, the NFS cache directory was still empty because model preparation had not yet populated it. The deployment's preparation script subsequently configured and checked its Docker NFS volume.

5. Pin and prepare the upstream deployment

On the head, the cloned upstream repository was checked at the exact revision used for this installation. For reproducibility, readers can explicitly check out that commit:

# HEAD cd ~ git clone https://github.com/MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks.git cd DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks git checkout c7f2b85100f0a7b099c92a4e21825e30d93c9c51 cp scripts/local.sh.example scripts/local.sh 

My local scripts/local.sh configured the worker SSH target, dedicated cache, management API bind address, and API-key generation. I did not include machine-specific private settings in the published article. In a documentation-only topology, it looks like this:

# HEAD: scripts/local.sh — replace all example values WORKER=USER@198.51.100.13 HF_CACHE=/home/USER/.cache/dsv41-hf HOST=192.0.2.61 API_KEY=1 

API_KEY=1 is a switch enabling authentication; it is not the bearer token. The script generated a secret in ~/.local/state/dsv41-tensorfold/api-key with restricted permissions. I did not place that key in source control.

I first checked the launch configuration without starting containers:

# HEAD, in the repository DRY_RUN=1 ./start.sh 

I then explicitly prepared both machines:

# HEAD, in the repository ./scripts/prepare.sh 

Preparation finished successfully (Prepare exit status: 0). The log shows that the expected prebuilt serving image was unavailable, so prepare.sh built the image locally and transferred it to the worker with docker save | docker load. It downloaded the quantized checkpoint and Engram tables, prepared and checked the kernel kit and token map, generated rank-specific weight files, and verified the worker's NFS access. The worker also copied its Engram tables to local disk; this was not a setup in which every inference-time file remained solely on an NFS mount. My resulting image tag was:

tensorfold-dsv41:exl3-tp2-8a98ae503d85 

The build fetched bertholomus/TensorFold at commit fbbc97132cd004237d9887b8d3185896efaedc5c (a fork of the original TensorFold project), its auxiliary inputs commit 59402086381a42c74d53053443224eec2790b4c5, and the kernel kit at commit abe07c7d509a0199cc7fec15263be2aa52ac652f. It used Zig 0.17.0 and NVIDIA's nvcr.io/nvidia/pytorch:26.07-py3 base image. These pins were defined by the upstream recipe at the selected revision, rather than chosen or modified during my installation.

The model weights were pinned to revision 45ee5d0fd106c94a3d77f721e09cbb2f50e5a16b; the base-model Engram files were pinned separately to 1bae0e1088993b9b82de08f59de5be5a34e42170. Preparation requires hundreds of gigabytes of storage and substantial download time. It is not a step to repeat casually after a successful installation.

6. Start DeepSeek with a conservative memory configuration

The upstream script defaults to a larger context and concurrency. I used the following, deliberately more conservative configuration for the working deployment:

# HEAD, inside the cloned repository CONTEXT=262144 PARALLEL=1 ./start.sh 

The preflight requires 112 GiB of MemAvailable on each Spark; this is a safety requirement of the selected recipe rather than a measured memory requirement of all possible deployments. I kept the memory guard and startup checks enabled.

During a verified stop/start rehearsal, the startup log reported:

rank cache: 95.1 GiB of expert entries read whole in 20.1 s [tensorfold] serving DeepSeek-V4.1-Flash-1017 ... [start.sh] Server answered after 32s [start.sh] OK: 'OK', 2 tokens [start.sh] Memory after the load: 9.5 GiB available on this Spark, 7.8 GiB on the worker 

The API was served on port 8888 under model name DeepSeek-V4.1-Flash-1017. The script ran its smoke test through the two ranks. The memory guard remained enabled: a rank is terminated if available memory stays below 2.5 GiB for four seconds, or falls below 1 GiB.

The serving log confirmed a 262,144-token configured window, one request stream, speculative drafts enabled, and image support enabled. It did not establish accuracy across the full configured context or multi-user throughput. The default chat mode enables reasoning; individual requests can turn it off through chat_template_kwargs.

Verify the authenticated API

For my API checks I read the bearer token locally from the head's protected file and avoided printing it. The following illustrative Python client runs on the head and prints only the model response. It is not a copy of my original test script; the recorded outcome is that an authenticated Japanese completion succeeded:

import json from pathlib import Path from urllib.request import Request, urlopen api_key = ( Path.home() / ".local/state/dsv41-tensorfold/api-key" ).read_text().strip() payload = { "model": "DeepSeek-V4.1-Flash-1017", "messages": [{"role": "user", "content": "日本語で短く自己紹介してください。"}], "max_tokens": 128, "stream": False, "chat_template_kwargs": {"enable_thinking": False}, } request = Request( "http://192.0.2.61:8888/v1/chat/completions", # Replace with head API IP data=json.dumps(payload).encode("utf-8"), headers={ "Authorization": f"Bearer {api_key}", "Content-Type": "application/json", }, ) with urlopen(request, timeout=180) as response: result = json.load(response) print(result["choices"][0]["message"]["content"]) 

A Japanese chat completion succeeded through this endpoint before I introduced the browser UI.

7. Add Open WebUI without disturbing inference

I deployed Open WebUI v0.13.3 on the worker as a separate Docker container, using a persistent data volume. The original container was initially bound to 127.0.0.1:3000; the final laboratory-LAN binding was made by recreating only the UI container and retaining the data volume. It calls the head's OpenAI-compatible API; it does not load another copy of DeepSeek.

I stored the backend API key and WEBUI_SECRET_KEY in a user-owned file (~/.config/dsv41-openwebui/runtime.env, mode 600) rather than embedding secrets in a public Docker command or configuration repository. A sanitized example of the non-secret settings is:

OPENAI_API_BASE_URL=http://192.0.2.61:8888/v1 ENABLE_OPENAI_API=True ENABLE_OLLAMA_API=False WEBUI_AUTH=True DEFAULT_USER_ROLE=pending RAG_FULL_CONTEXT=True BYPASS_EMBEDDING_AND_RETRIEVAL=True RAG_ALLOWED_FILE_EXTENSIONS=pdf RAG_FILE_MAX_SIZE=10 RAG_FILE_MAX_COUNT=1 # OPENAI_API_KEY and WEBUI_SECRET_KEY are also present in the private file. # The two initial defaults above (pending / 1) were later overridden # in the Open WebUI administrator interface (user / 3). 

The original runtime.env file retained DEFAULT_USER_ROLE=pending and RAG_FILE_MAX_COUNT=1. The administrator interface was subsequently configured to enable self-registration, set Default User Role to user, and allow three PDF attachments. I verified the three-file setting again after restarting Open WebUI. The administrator settings are the effective application configuration here; the two entries shown in the environment example document their original values, not the final policy. A clean deployment should set these values in the administrator interface and verify them after restarting. I verified the settings, not a complete sign-up/login journey for a newly registered user or multi-user throughput.

With a complete private environment file, a fresh HTTP deployment can follow this Docker pattern. This is a reproduction example; do not apply it to an existing container without preserving its data volume and configuration:

# WORKER — template for a fresh installation; inspect existing containers first docker pull ghcr.io/open-webui/open-webui:v0.13.3 docker volume create dsv41-openwebui-data docker run -d \ --name dsv41-openwebui \ --restart unless-stopped \ --memory=2g --memory-swap=2g \ --cpus=2 --pids-limit=256 \ -p 192.0.2.29:3000:8080 \ --env-file "$HOME/.config/dsv41-openwebui/runtime.env" \ -v dsv41-openwebui-data:/app/backend/data \ ghcr.io/open-webui/open-webui:v0.13.3 

The address above is a non-routable placeholder. I also tested a Caddy HTTPS reverse proxy with an internal CA, but did not retain it as the chosen access path. To bind the HTTP port to the worker management address, I recreated only the Open WebUI container, preserved its data volume, and retained the previous container as a stopped rollback copy. I did not reinstall the model or reconfigure either inference rank. With --restart unless-stopped, Docker may start the UI after a host reboot before DeepSeek is ready. The DeepSeek ranks still require their own startup procedure unless an explicit supervisor has been configured.

For private or sensitive material, HTTP is a limitation: browser credentials, session data, prompts, and uploaded files are not encrypted in transit. A local model does not make an unencrypted connection secure. I used HTTP only on the intended laboratory LAN. Even on a private network, HTTP does not protect credentials or documents in transit; this configuration should not be copied to an untrusted network.

The tested PDF workflow supported questions based on extracted text and tables, but the model did not answer questions requiring values found only in raster figures. Direct PNG uploads were tested separately. Since I did not inspect the final API payload, I cannot identify the exact stage at which the figure information became unavailable. I retained the existing document-processing configuration. The detailed observations are covered in the companion evaluation article.

8. Verify service readiness and routine operations

A Docker container reporting Started does not mean its web application is ready. During a real stop/start rehearsal, docker start dsv41-openwebui returned successfully but an immediate HTTP request failed. About a minute later, the container was healthy and returned HTTP 200:

# WORKER — use your deployed management address curl -fsS "http://<WORKER-MANAGEMENT-IP>:3000/health" # {"status":true} 

I confirmed that both inference rank containers were running and that none of the monitored containers had been OOM-killed or restarted in the checks performed. docker inspect also confirmed Open WebUI's unless-stopped restart policy, 2 GiB memory limit, 2 GiB memory-plus-swap limit, two-CPU cap, management-interface port binding, and persistent data volume. After the web interface started, Open WebUI used approximately 1.0 GiB of its 2 GiB container memory limit. The worker had around 6.4–7.0 GiB MemAvailable in repeated checks, while the head had approximately 8.6 GiB. These were operating snapshots, not sustained-load or peak-memory measurements.

Stop the service

From the worker's terminal, first stop the UI and then ask the head to stop both inference ranks:

# WORKER # Schedule the stop when no users are submitting requests. docker stop dsv41-openwebui ssh "USER@<HEAD-MANAGEMENT-IP>" ' cd ~/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks && ./stop.sh ' 

The tested stop script saved separate compressed logs for rank 0 and rank 1, then removed the inference containers on both machines. The downloaded model, prepared weight files, Docker image, and Open WebUI persistent volume are preserved. Running requests are interrupted rather than drained automatically.

Start the service

Startup order matters because the model's memory preflight runs before the ranks load:

# WORKER # Keep the web interface stopped until the model is live. docker stop dsv41-openwebui 2>/dev/null || true ssh "USER@<HEAD-MANAGEMENT-IP>" ' cd ~/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks && CONTEXT=262144 PARALLEL=1 ./start.sh ' # Only after the script reports a successful smoke test: docker start dsv41-openwebui 

Then check readiness rather than assuming immediate availability:

# WORKER — replace host IP with your actual management address for attempt in $(seq 1 15); do if curl -fsS --max-time 5 \ "http://<WORKER-MANAGEMENT-IP>:3000/health"; then echo 'Open WebUI ready' break fi sleep 5 done 

The operational command explicitly retains CONTEXT=262144 PARALLEL=1; omitting them would select the repository's larger defaults. I did not change the existing unrelated containers on the worker. The readiness loop shown above is a recommended check rather than a separately measured load test.

9. Troubleshooting lessons

The server's memory guard remained active (2.5 GiB/4 s, immediate action below 1 GiB). I did not increase parallelism simply to maximize a benchmark number. The settings and behavior described here were the ones I actually operated.

10. What this deployment establishes—and what it does not

The two rank containers launched successfully, both RoCE rails initialized, the authenticated API answered Japanese requests, Open WebUI became healthy and answered HTTP health checks on the management interface, and a real shutdown/restart procedure worked. I also tested PDF-based question answering through the browser interface using synthetic evaluation material. The results are documented in the companion evaluation article. The administrator settings were verified as self-registration enabled, default role user, and a three-file PDF attachment limit retained after restart; these setting checks should not be mistaken for independently completed new-user sign-up or multi-user load tests.

This does not demonstrate correctness across every token of a 262,144-token prompt, production-grade multi-user concurrency, encrypted transport, or long-term unattended reliability. It also does not establish RDMA throughput or independently benchmark the model against other inference engines. Those are different evaluation questions.

For readers reproducing the setup, the most important practical points were: pin the repository and model versions, validate the network before model preparation, check the actual worker-side NFS mount, preserve memory headroom, keep the startup safety checks enabled, and wait for web application health after a container starts. The placeholder commands in this article are meant to be adapted and reviewed, not run without inspecting the target systems.

References

• MiaAI-Lab: DeepSeek-V4.1-Flash-1017 on 2× DGX Spark
• Pinned source revision
• Pinned installation requirements
• TensorFold (original upstream)
• TensorFold fork used by this recipe
• Open WebUI documentation

AI disclosure: This article documents my own technical setup and results. An LLM was used to help draft, organize, and edit the English text. I remain responsible for its technical accuracy.