Mixture of Experts: Sparse Computation, Routing, and Deployment Cost
Follow a token through MoE routing to distinguish total parameters, active parameters, memory, speed, load balancing, and communication.
Follow a token through MoE routing to distinguish total parameters, active parameters, memory, speed, load balancing, and communication.