From a Regular Language Model to Mixture of Experts
My beginner-friendly notes on why Mixture of Experts exists, what an expert is, and how a model chooses which experts to use.
My beginner-friendly notes on why Mixture of Experts exists, what an expert is, and how a model chooses which experts to use.
Lessons about model execution, memory placement, layer offloading, synchronization, and performance from porting llama2.c to an OpenCL device.
Building a cache from first principles to understand why cache lines, sets, ways, eviction, and way locking exist.