
VEX prefix

The VEX prefix (from "vector extensions") and VEX coding scheme are comprising an extension to the x86 and x86-64 instruction set architecture for microprocessors from Intel, AMD and others.
Features.
The VEX coding scheme allows the definition of new instructions and the extension or modification of previously existing instruction codes. This serves the following purposes:
The VEX prefix "replaces" the most commonly used instruction prefix bytes and escape codes. In many cases, the number of prefix bytes and escape bytes that are replaced is the same as the number of bytes in the VEX prefix, so that the total length of the VEX-encoded instruction is the same as the length of the legacy instruction code. In other cases, the VEX-encoded version is longer or shorter than the legacy code. In 32-bit mode VEX encoded instructions can only access the first 8 YMM/XMM registers; the encodings for the other registers would be interpreted as the legacy LDS and LES instructions that are not supported in 64-bit mode.
The two-byte VEX prefix contains the following components:
The three-byte VEX prefix additionally contains:
Technical description.
The VEX coding scheme uses a code prefix consisting of two or three bytes, which is added to existing or new instruction codes.
In x86 architecture, instructions with a memory operand may use the ModR/M byte which specifies the addressing mode. This byte has three bit fields:
The base-plus-index and scale-plus-index forms of 32-bit addressing (encoded with r/m=100 and mod <>11) require another addressing byte, the SIB byte. It has the following fields:
To use 64-bit addressing and additional registers present in the x86-64 architecture, the REX prefix has been introduced which provides additional space for encoding addressing modes. Bit-field W changes the operand size to 64 bits, R expands "reg" to 4 bits, B expands "r/m" (or "opreg" in the few opcodes such as "POP reg" which encode register number in their 3 lowest opcode bits), and X and B expand "index" and "base" in the SIB byte. However REX prefix is encoded quite inefficiently, wasting half of its 8 bits.
The VEX prefix provides a compact representation of the REX prefix, as well as various other prefixes, to expand the addressing mode, register enumeration and operand size and width:
Instructions coded with the VEX prefix can have up to four variable operands (in registers or memory) and one constant operand (immediate value). Instructions that need more than three variable operands use immediate operand bits to specify a 4th register operand (IS4 above). At most one of the operands can be a memory operand; and at most one of the operands can be an immediate constant of 4 or 8 bits. The remaining operands are registers.
The AVX instruction set is the first instruction set extension to use the VEX coding scheme. The AVX instruction set uses VEX prefix only for instructions using the SIMD XMM registers.
However, the VEX coding scheme has been used for other instruction types as well in subsequent expansion of instruction set. For example, BMI introduced VEX coded arithmetic and bit manipulation instructions that operate on general purpose registers. AVX-512 introduced 8 mask registers and added instructions to manipulate them. These instructions use VEX encoding. VEX.R, VEX.B or VEX.v3 is ignored when the field is used to encode a mask register.
The VEX prefix's initial-byte values, C4h and C5h, are the same as the opcodes of the LDS and LES instructions. These instructions are not supported in 64-bit mode. To resolve the ambiguity while in 32-bit mode, VEX's specification exploits the fact that a legal LDS or LES's ModRM byte can not be of the form "11xxxxxx" (which would specify a register operand). Various bit-fields in the VEX prefix's second byte are inverted to ensure that the byte is always of this form in 32-bit mode.
Legacy SIMD instructions with a VEX prefix added are equivalent to the same instructions without VEX prefix with the following differences:
Instructions that use the whole 256-bit YMM register should not be mixed with non-VEX instructions that leave the upper half of the register unchanged, for reasons of efficiency.