Implementation of the specific Transformer architecture from PaLM - Scaling Language Modeling with Pathways - View it on GitHub