In this work, we present a novel framework, **CodecLM**, which systematically generates tailored high-quality data to align LLMs for different downstream tasks. A high-level overview of CodecLM is shown in Figure 1. Inspired by the principles of Encode-Decode process Kramer (1991); Kingma and Welling (2013), we leverage a strong LLM as a codec to "encode" seed instructions from our target task into instruction _metadata_ and then "decode" the metadata into tailored synthetic instructions. The metadata serves as a word-level abstraction of the input instruction distribution, including the _use case_ and _skills_ for effective instruction following. It can be automatically generated by encoding seed instructions, or directly provided by users with a high-level anticipation of the downstream task.