Resonance Neural Networks
A novel neural network architecture replacing attention mechanisms with FFT-based spectral processing. Achieves O(n log n) complexity with 3–28× speedup over transformers, 83% fewer parameters, and 260K–300K token context windows. Accepted to AAAI 2026 Workshop.
Visit website
The problem
Transformer architectures have dominated sequence modeling but suffer from O(n²) complexity in their attention mechanism, making them prohibitively expensive for long-context tasks. Models struggle with sequences beyond 8K–32K tokens, and the parameter counts required for competitive performance (often billions) make training and deployment costly. The field needed an alternative that could match or exceed transformer performance while being fundamentally more efficient.
Spectral decomposition approach
Resonance Neural Networks replace self-attention entirely with frequency-domain processing via Fast Fourier Transforms. Instead of computing pairwise token interactions, the architecture decomposes input sequences into spectral components, applies learned transformations in frequency space, and reconstructs the output — achieving O(n log n) complexity with R² > 0.95 verified scaling.
The architecture integrates holographic memory with physics-inspired interference patterns, enabling ultra-long context windows of 260K–300K tokens via hierarchical chunking. This is 10–30× longer than most transformer implementations without the quadratic memory overhead.
Architecture details
The model features 4–6× parameter efficiency compared to transformers — achieving competitive performance with 83% fewer parameters. Model configurations scale from 50M parameters (small) through 200M (medium), 500M (large), to 1–3B (XLarge), trained on the FineWebEdu 32K dataset on NVIDIA L40 GPUs.
Key innovations include multimodal support via frequency-based cross-modal fusion (vision, audio, text), large vocabulary handling (500K–1M tokens), and export to multiple formats (PyTorch, ONNX, TorchScript, quantized).
Research results
The work was accepted to the AAAI 2026 Workshop on Linear-Complexity Sequence Modeling via Spectral Decomposition — demonstrating 3–28× speedups over transformer baselines across standard benchmarks while maintaining or improving accuracy.
This is part of a broader research program at Genovo Technologies with 6 published papers, 3 focused on novel architectures outperforming transformers, all with strong mathematical and theoretical foundations including provable approximation guarantees.
Technology stack
Built with Python 3.8+, PyTorch 2.0+, torch.fft for spectral operations, NumPy, SciPy, and einops for tensor manipulation. Training infrastructure uses TensorBoard for visualization, tqdm for progress tracking, and matplotlib for analysis. The architecture is proprietary to Genovo Technologies and represents the core ML research driving our AI infrastructure products.