[ICML 2024] Fool Your (Vision and) Language Model With Embarrassingly Simple Permutations - View it on GitHub