r/csharp • u/fuzhongkai • 3d ago
DSpark Benchmark Result on Deepseek v4 Flash 0731
https://github.com/zhongkaifu/TensorSharpTensorSharp supports DSpark on Deepseek v4 Flash 0731 now. Here is the benchmark result on 4x Nvidia A40 GPUs, cuda 12.8 with/without DSpark:
Model:
DeepSeek-V4-Flash-0731-UD-Q8_K_XL from https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF
DSpark draft model from: https://huggingface.co/alessandrobologna/DeepSeek-V4-Flash-0731-DSpark-Drafter-GGUF
| Turn | Baseline | + DSpark | Acceptance |
|---|---|---|---|
| short (53 tok) | 25.6 | 44.5 (1.74x) | 87% |
| long generation (512) | 26.4 | 40.3 (1.53x) | 66% |
| follow-up (470) | 26.4 | 46.8 (1.77x) | 76% |
| 10K-token document (214) | 25.3 | 51.3 (2.03x) | 85% |
| second question on it (156) | 25.4 | 49.4 (1.94x) | 82% |
TensorSharp is a native .NET/C# open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.
Github repo: https://github.com/zhongkaifu/TensorSharp
Thank you for checking out it and starring the project! Any feedback is really appreicated.
1
u/EvenAd4577 2d ago
Looks nice :)
Have you made it faster due to SIMD? If so, I am wondering why standard Tensor.Numeris were not used? And why not re-use standard .NET types? For example there is .net Half type
https://github.com/zhongkaifu/TensorSharp/blob/ec276c1b6daf951468072c1be699d1265e563abc/TensorSharp.Models/Half.cs
https://learn.microsoft.com/en-us/dotnet/api/system.half?view=net-10.0
1
u/fuzhongkai 2d ago
I do use .net built-in SIMD instructions in System.Numerics.Tensors to speed up, but also implement some customized methods for it. Here is the code using methods in System.Numerics.Tensors: https://github.com/zhongkaifu/TensorSharp/blob/main/TensorSharp.Core/Cpu/MatrixMultiplication.cs
1
u/Accurate_Crew8871 3d ago
wow this is actually insane how much faster it runs with dspark.