Quick intro
LeetArxiv is Leetcode for implementing Arxiv and other research papers.
This is part 3 in our Low-Level Language Diffusion Models from Scratch series.
Part 1: Implementing the Forward Process of a Diffusion Model from Scratch
Part 2: Implementing Rectified Flows that power Stable Diffusion
Part 3 (We are here): ChebyKAN layers for convolutions
Quick Summary of Results
Chebyshev KANs outperform MLPs on extremely non-linear functions like the Doppler effect wave.
MLPs outperform Chebyshev KANs on more structured data like MNIST.
Chebyshev KANs take a lot of time to train compared to MLPs. Honestly, it took forever lol but then again, training rectified flows here took forever.
*C and Python code is available here
Frontmatter for the paper, Chebyshev Polynomial-Based Kolmogorov-Arnold Networks: An Efficient Architecture for Nonlinear Function Approximation
1.0 Introduction
The paper, Chebyshev Polynomial-Based Kolmogorov-Arnold Networks: An Efficient Architecture for Nonlinear Function Approximation, (Sidharth et al., 2024)1 demonstrates how to perform convolutions with Kolmogorov Arnold Networks (KANs).
The authors show that chebyshev polynomials and learnable weights can be combined by einstein summation to perform convolutions.
1.1 Deep Learning with Chebyshev Polynomials
Deep learning using Chebyshev polynomial approximation was introduced in (Tang et. al, 2023)2 with the ChebNet architecture. This was further improved in (He et al, 2024)3 yielding the ChebNetII architecture. We use the Chebyshev Polynomial-Based Kolmogorov Arnold Networks introduced in (Sidharth et al., 2024).
Background
Diffusion models can be simplified to transforming random noise into stunning images using matrix transformations. This, we observed in our previous paper implementation, where we heavily referenced Ho et al. (2020) and Sohl-Dickstein et al. (2015). We implement this paper to
