I build AI systems that do more with the hardware at hand. Studying EECS + Business in UC Berkeley’s M.E.T. program; building independently across models, GPU kernels, and practical engineering.
A complete rotated GPTQ-INT4 inference path in llama.cpp: native INT4 matrix instructions, custom attention and gated-delta-net kernels, and a decode graph cut from 847 to 236 launches per token.
Prefill · pp512 · fast preset
Decode · tg128 · empty context
Prefill throughput vs Q4_K_M
Block-Hadamard rotations tame activation outliers; GPTQ quantizes the rotated weights. Custom WMMA kernels then use RDNA3’s INT4 pipeline directly, with fused operations across prefill and decode.
The same pipeline also transfers to Ossicle-4B’s language backbone. One copy of the weights stays in VRAM, with peak usage of 3,358 MiB for the stock model’s quality preset.
Fast uses 4-bit activations for 2.01× prefill throughput, with a larger quality cost. Decode always uses 8-bit activations.
Stock Qwen3.5-4B; same checkpoint, GPU, and build. Q4_K_M: 145.2 tok/s decode, +2.00% perplexity, 0.0278 KLD. WikiText-2 evaluation. Measurements and limitations in the report.
On the reported figures, the RX 7900 XT delivers 9.5% more prefill throughput than the RTX 5080 and 91% of the RTX 4090’s decode throughput.
| GPU | Prefill tok/s | Decode tok/s |
|---|---|---|
| RTX 5090 Published · CUDA Q4 | 16,036 | 313.6 |
| RTX 4090 Published · CUDA Q4 | 14,882 | 220.2 |
| RX 7900 XT THIS WORKRotated INT4 · fast | 11,012 | 200.6 |
| RTX 5080 Published · CUDA Q4 | 10,057 | 200.5 |
RX 7900 XT: pp512 / tg128 from the November 2 report, §2 & §5. NVIDIA: Local LLM Advisor’s Q4 leaderboard. These are cross-source results, with different kernels, quantization, and undisclosed NVIDIA prompt settings. Fast has +8.67% perplexity vs BF16. The 0.1 tok/s decode edge is within the RX run’s ±1.5 tok/s temperature variation; the supported claim is comparable decode throughput.
The default quality preset reaches 7,140 tok/s prefill and 200.7 tok/s decode: 1.30× and 1.38× Q4_K_M throughput at comparable measured perplexity degradation (+1.92% vs +2.00%). The fast preset reaches 11,012 tok/s prefill at +8.67% perplexity degradation. Results cover two 4B checkpoints of the same architecture on RDNA3; other model sizes and GPU families are untested. Decode varies by about ±1.5 tok/s with temperature. Cross-GPU leaderboard figures in the report use different conditions and are contextual comparisons.
A 4.9B audio-language model built by grafting Qwen3-Omni’s audio encoder onto Qwen3.5-4B.
A frozen audio encoder, a trained projector, and LoRA adapters connect speech, sound, and music to the language model. Released with evaluation code, quantized checkpoints, and streaming support for long recordings.
MMAU-Pro
Two-pass system
Ossicle describes the clip, then base Qwen3.5-4B answers from that text and Ossicle’s answer. Single-pass score: 52.0%. Category prompts and sampling settings were selected on MMAU-Pro; see the evaluation notes.
57.0% on MMAU-Pro: +2.1 percentage points over the locally reproduced MOSS-Audio-4B-Instruct, and above the cited Kimi-Audio-7B and Qwen2.5-Omni-7B results.
| Model / evaluation | Score |
|---|---|
| MOSS-Audio-4B-Instruct Author-reported | 58.16 |
| Ossicle-4B THIS WORKTwo-pass system · measured | 57.0 |
| Kimi-Audio-7B Published comparison | 56.6 |
| MOSS-Audio-4B-Instruct Locally reproduced · its own prompting | 54.9 |
| Qwen2.5-Omni-7B Published comparison | 52.2 |
+2.1 pp over MOSS in the local reproduction. A higher reported Pro score than the cited 7B Kimi and Qwen results.
| Model / evaluation | Score |
|---|---|
| MOSS-Audio-4B-Instruct Published comparison in Ossicle release | 75.8 |
| Kimi-Audio-7B Published comparison in Ossicle release | 72.4 |
| Ossicle-4B · INT4 GPTQ THIS WORKvLLM · measured | 71.9 |
| Ossicle-4B · BF16 Single-pass · measured | 71.8 |
| Qwen2.5-Omni-7B Published comparison in Ossicle release | 65.6 |
+6.2 pp over the cited Qwen score in BF16. INT4 GPTQ preserves the measured score: 71.9% vs 71.8% BF16.
Sources: Ossicle release results and evaluation notes; INT4 results; MOSS author-reported comparison. External model figures follow the cited release comparison; evaluation setups and dataset versions can differ.
MMAU-Pro: Ossicle’s exact two-pass score is 56.97%, rounded to 57.0%; the single-pass model scores 52.0%. The second pass uses base Qwen3.5-4B; category-specific prompting and speech/voice-chat sampling were selected on MMAU-Pro. MOSS’s 54.9% is a local reproduction using its own prompting and the same Pro scorer; its authors report 58.16% separately. Kimi and Qwen are published comparison figures, not local reruns. On test-mini, the released BF16 model scores 71.8%, and the INT4 GPTQ checkpoint scores 71.9%; a 0.1-point difference is within sampling noise. The mini table retains the external scores cited by the Ossicle release rather than mixing in scores from other evaluation versions.
Retrieve a conversation by its strongest matching turn. Turn-level scoring lifts the same encoder from 406 to 436 all-gold passes out of 470 LongMemEval-S questions under the released evaluator; a direct annotated-ID check gives 433/470.
Tests whether unchanged learned artifacts can work across language-model hosts, then examines compiling acquired behavior into native model weights.
How should two local language models pass information on a 16 GB GPU? A comparison of six message formats finds concise prose summaries balance factual fidelity with token and memory cost.
Low-cost sensors and predictive models for agricultural emissions management on small and medium farms. Co-authored at Castleknock College and published at IEEE GESSC.
Read publication ↗A guide stick that detects wet surfaces and warns visually impaired users through audio. Built around a single NE555 timer circuit with nearly no added cost or weight.
Built 2018–19 · Patented 2020A free competitor-discovery tool trained on more than one million companies from Crunchbase, Product Hunt, YC, and Kickstarter.
Full-stack engineering · Machine learningEECS + Business at UC Berkeley, through the Management, Entrepreneurship, & Technology program.
From a timer circuit in a guide stick to GPU instructions for a language model, I’m interested in finding the smallest system that solves the problem well.
Irish patent IES87168 · IEEE GESSC publication · SciFest National Finals, Excellence in STEM (2023)
Validated a B2B agricultural-emissions platform and designed a marketplace connecting farmers with sustainable buyers.
One of approximately 30 fellows in a youth accelerator backed by Stripe and NDRC, with mentorship from Intercom and Stripe founders.
Worked in Back-End Metals at the 14 nm fab: physical vapor deposition, metal etch, and statistical process control.
I release music as SVT. Another place to experiment with structure, texture, and sound.
Listen on Spotify ↗