This is the official repo of "Scaling Recurrent Neural Networks to a Billion Parameters with Zero-Order Optimization" - View it on GitHub