I build AI systems that do more with the hardware at hand. Studying EECS + Business in UC Berkeley’s M.E.T. program; building independently across models, GPU kernels, and practical engineering.
A complete GPTQ-INT4 inference path in llama.cpp: native RDNA3 matrix instructions, custom attention and gated-delta-net kernels, and a 4B decode graph cut from 847 to 236 launches per token. Now demonstrated on Qwen3.5-4B, Qwen3.5-9B, and Qwen3.8-27B.
Qwen3.5-4B prefill · pp512 · fast
Qwen3.5-4B decode · tg128
Qwen3.8-27B decode · no MTP
Block-Hadamard rotations tame activation outliers; GPTQ quantizes the rotated weights. Custom WMMA kernels then use RDNA3’s INT4 pipeline directly, with fused operations across prefill and decode.
The pipeline scales to 27.8 billion parameters on one 20 GB card. At 27B, the reported fast result is 2,013 tok/s prefill and 50.1 tok/s decode: 2.27× and 1.53× the stock 7900 XT. The best recorded decode reaches the report’s estimated weight-bandwidth ceiling.
Fast uses 4-bit activations for 2.01× prefill throughput, with a larger quality cost. Decode always uses 8-bit activations.
Stock Qwen3.5-4B; same checkpoint, GPU, and build. Q4_K_M: 145.2 tok/s decode, +2.00% perplexity, 0.0278 KLD. WikiText-2 evaluation. Measurements and limitations in the report.
Against stock Q4_K_M on the same card, fast prefill reaches 2.01× at 4B, 2.20× at 9B, and 2.27× at 27B. At 27B, decode rises from 32.79 to 50.1 tok/s.
| Model | Prefill stock → this work | Decode stock → this work | Speedup prefill / decode |
|---|---|---|---|
| Qwen3.5-4B | 5,477 → 11,012 | 145.2 → 200.6 | 2.01× / 1.38× |
| Qwen3.5-9B | 3,005 → 6,624 | 95.9 → 126.2 | 2.20× / 1.32× |
| Qwen3.8-27B NEW | 888 → 2,013 | 32.79 → 50.1 | 2.27× / 1.53× |
All throughput is in tokens/s. The stock 27B baseline is reported in §2; the detailed §7 table compares the two INT4 presets. The 50.1 tok/s figure is the best recorded selected decode build. Fast’s perplexity change vs BF16 is +8.67% / +4.50% / +4.55% at 4B / 9B / 27B. The quality preset at 27B gives 1,218 tok/s prefill, the same 50.1 tok/s decode, and +1.27% perplexity. Full measurements: §2, §5–7 and Table 10.
At 27B, the RX 7900 XT records 50.1 tok/s decode vs the RTX 4090’s published 49.8, while delivering 2,013 tok/s prefill vs the RTX 5080’s 1,673. No MTP or speculative decoding in the report’s local results.
| GPU | Prefill tok/s | Decode tok/s |
|---|---|---|
| RTX 5090 Published · CUDA IQ4_XS | 3,213 | 81.8 |
| RTX 4090 Published · CUDA IQ4_XS | 2,510 | 49.8 |
| RX 7900 XT THIS WORKRotated INT4 · fast | 2,013 | 50.1 |
| RTX 5080 Published · CUDA IQ4_XS | 1,673 | 49.4 |
| RTX 5070 Ti Published · CUDA IQ4_XS | 1,276 | 42.1 |
| RTX 5070 No 27B figure in the report | — | — |
| RTX 3090 Published · CUDA IQ4_XS | 1,042 | 43.0 |
| RTX 5060 Ti 16 GB Published · CUDA IQ4_XS | 815 | 27.6 |
November 3 report, §2 / Table 1: local RX 7900 XT pp512 / tg128 results and the report’s cited NVIDIA figures. NVIDIA source: Local LLM Advisor. NVIDIA uses Q4 at 4B/9B and IQ4_XS at 27B; these are cross-source comparisons with different quantization, kernels, and benchmark conditions.
27B: 50.1 tok/s is the best recorded selected decode build. Fast: +4.55% perplexity and 0.0565 KLD vs BF16; quality: +1.27% and 0.0184. Peak VRAM: 15,853 MiB (fast). The 0.3 tok/s numerical lead over the published 4090 result is small; the major gain is 1.53× the stock card’s decode.
The default quality preset reaches 7,140 tok/s prefill and 200.7 tok/s decode: 1.30× and 1.38× Q4_K_M throughput at comparable measured perplexity degradation (+1.92% vs +2.00%). The fast preset reaches 11,012 tok/s prefill at +8.67% perplexity degradation. The same pipeline is demonstrated on Qwen3.5-4B, the Ossicle-4B fine-tune, Qwen3.5-9B, and Qwen3.8-27B. RDNA3 gfx1100 is the tested GPU target. Decode varies by about ±1.5 tok/s with temperature. Cross-GPU leaderboard figures in the report use different conditions and are contextual comparisons.
A 4.9B audio-language model built by grafting Qwen3-Omni’s audio encoder onto Qwen3.5-4B.
A frozen audio encoder, a trained projector, and LoRA adapters connect speech, sound, and music to the language model. Released with evaluation code, quantized checkpoints, and streaming support for long recordings.
MMAU-Pro
Two-pass system
Ossicle describes the clip, then base Qwen3.5-4B answers from that text and Ossicle’s answer. Single-pass score: 52.0%. Category prompts and sampling settings were selected on MMAU-Pro; see the evaluation notes.
57.0% on MMAU-Pro: +2.1 percentage points over the locally reproduced MOSS-Audio-4B-Instruct, and above the cited Kimi-Audio-7B and Qwen2.5-Omni-7B results.
| Model / evaluation | Score |
|---|---|
| MOSS-Audio-4B-Instruct Author-reported | 58.16 |
| Ossicle-4B THIS WORKTwo-pass system · measured | 57.0 |
| Kimi-Audio-7B Published comparison | 56.6 |
| MOSS-Audio-4B-Instruct Locally reproduced · its own prompting | 54.9 |
| Qwen2.5-Omni-7B Published comparison | 52.2 |
+2.1 pp over MOSS in the local reproduction. A higher reported Pro score than the cited 7B Kimi and Qwen results.
| Model / evaluation | Score |
|---|---|
| MOSS-Audio-4B-Instruct Published comparison in Ossicle release | 75.8 |
| Kimi-Audio-7B Published comparison in Ossicle release | 72.4 |
| Ossicle-4B · INT4 GPTQ THIS WORKvLLM · measured | 71.9 |
| Ossicle-4B · BF16 Single-pass · measured | 71.8 |
| Qwen2.5-Omni-7B Published comparison in Ossicle release | 65.6 |
+6.2 pp over the cited Qwen score in BF16. INT4 GPTQ preserves the measured score: 71.9% vs 71.8% BF16.
Sources: Ossicle release results and evaluation notes; INT4 results; MOSS author-reported comparison. External model figures follow the cited release comparison; evaluation setups and dataset versions can differ.
MMAU-Pro: Ossicle’s exact two-pass score is 56.97%, rounded to 57.0%; the single-pass model scores 52.0%. The second pass uses base Qwen3.5-4B; category-specific prompting and speech/voice-chat sampling were selected on MMAU-Pro. MOSS’s 54.9% is a local reproduction using its own prompting and the same Pro scorer; its authors report 58.16% separately. Kimi and Qwen are published comparison figures, not local reruns. On test-mini, the released BF16 model scores 71.8%, and the INT4 GPTQ checkpoint scores 71.9%; a 0.1-point difference is within sampling noise. The mini table retains the external scores cited by the Ossicle release rather than mixing in scores from other evaluation versions.
Retrieve a conversation by its strongest matching turn. Turn-level scoring lifts the same encoder from 406 to 436 all-gold passes out of 470 LongMemEval-S questions under the released evaluator; a direct annotated-ID check gives 433/470.
Tests whether unchanged learned artifacts can work across language-model hosts, then examines compiling acquired behavior into native model weights.
How should two local language models pass information on a 16 GB GPU? A comparison of six message formats finds concise prose summaries balance factual fidelity with token and memory cost.
Low-cost sensors and predictive models for agricultural emissions management on small and medium farms. Co-authored at Castleknock College and published at IEEE GESSC.
Read publication ↗A guide stick that detects wet surfaces and warns visually impaired users through audio. Built around a single NE555 timer circuit with nearly no added cost or weight.
Built 2018–19 · Patented 2020A free competitor-discovery tool trained on more than one million companies from Crunchbase, Product Hunt, YC, and Kickstarter.
Full-stack engineering · Machine learningEECS + Business at UC Berkeley, through the Management, Entrepreneurship, & Technology program.
From a timer circuit in a guide stick to GPU instructions for a language model, I’m interested in finding the smallest system that solves the problem well.
Irish patent IES87168 · IEEE GESSC publication · SciFest National Finals, Excellence in STEM (2023)
Validated a B2B agricultural-emissions platform and designed a marketplace connecting farmers with sustainable buyers.
One of approximately 30 fellows in a youth accelerator backed by Stripe and NDRC, with mentorship from Intercom and Stripe founders.
Worked in Back-End Metals at the 14 nm fab: physical vapor deposition, metal etch, and statistical process control.
I release music as SVT. Another place to experiment with structure, texture, and sound.
Listen on Spotify ↗