Generating text with autoregressive language models (LMs)  is of great importance to many natural language processing (NLP)  applications. Previous solutions for this task often produce text that contains degenerative expressionsor lacks semantic consistency. Recently,introduced a new decoding method,contrastive search, based on the isotropic representation space of the language model and obtained new state of the art on various benchmarks. In addition,argued that the representations of autoregressive LMs (e.g. GPT-2)  are intrinsically anisotropic which is also shared by previous studies. Therefore, to ensure the language model follows an isotropic distribution,proposed a contrastive learning scheme, i.e.SimCTG, which calibrates the language model’s representations through additional training. In this study, we first answer the question:“Are autoregressive LMs really anisotropic?”. To this end, we extensively evaluate the isotropy of LMs across 16 languages. Surprisingly, we find that the anisotropic problemonlyexists in the two specific English GPT-2-small/medium models. On the other hand,allother evaluated LMs are isotropic which is in contrast to the conclusion drawn by previous studies. Based on our findings, we further assess the contrastive search decoding method usingoff-the-shelfLMs on four generation tasks across 16 languages. Our experimental results demonstrate that contrastive search significantly outperforms previous decoding methodswithoutany additional training. More notably, on 12 out of the 16 evaluated languages, contrastive search performs comparably with human-level performances as judged by human evaluations. Our code and other related resources are publicly available athttps://github.com/yxuansu/Contrastive_Search_Is_What_You_Need.