Q:

What's the deal with System/360's "USASCII" mode?

The original System/360 architecture had a bit in the Program Status Word that would select an "USASCII" mode rather than the usual EBCDIC.  Setting this bit changed how the BCD arithmetic operations worked: The sign of results would now be encoded as a nibble reading 1010 for plus and 1011 for minus (instead of 1100/1101 for EBCDIC) and the upper nibble in unpacked "zoned decimal" numbers would be stored as 0101 (instead of 1111 for EBCDIC).
This feature was never much used, and was abolished in System/370.
For EBCDIC, putting 1111 in the upper nibbles makes perfect sense: it makes the bytes printable digits. And 1100/1101 as the sign nibble in unpacked results means that if you punch the result to a card, it will look identical to the same number computed by a 1401 (even though the 1401's internal bit patterns were different).
However what is particularly ASCII-friendly about using 1010/1011/0101 instead?
The digits in ASCII have 0011 as the upper nibble, not 0101. Putting 0101 in the upper bits would produce PQRSTUVWXY on an ASCII device rather than digits. 
Likewise, there doesn't seem to be anything ASCII about the bit combinations 1010 and 1011 for signs. Naively converting a packed BCD number to ASCII characters nibble for nibble would result in digits followed by : or ; for positive and negative, hardly an improvement over < and =.
It doesn't look like producing ASCII output is made easier in any way by running the S/360 in "USASCII" mode rather than EBCDIC. So what was the point supposed to be?

A:

I found the explanation in chapters 23 and 20 of Mackenzie, Charles E, Coded Character Sets, History and Development (Addison-Wesley, 1980), which was linked in a footnote to Wikipedia's ASCII article.
In the early 1960s, 7-bit ASCII was being standardized as an communication format (the last I in the name comes from "interchange"), but this does not directly specify how this 7-bit code would be represented internally in an 8-bit computer.
Today it seems "obvious" that a 7-bit codepoint abcdefg would be represented with a leading 0 bit as 0abcdefg -- which is what the standards committee eventually decided, in the context of representing ASCII on byte-oriented magnetic tape -- but this decision had not yet been made when the System/360 architecture was defined.
IBM was championing a competing proposal which they called USASCII-8 where the codepoint abcdefg would be represented in a byte as abacdefg with an additional copy of the topmost bit inserted between the second and third bit. So the first three bits of ASCII would be encoded as a high nybble as follows:
000   becomes  0000  for  0x0_
001      ~     0001   ~   0x1_
010      ~     0100   ~   0x4_
011      ~     0101   ~   0x5_   (digits in this column!)
100      ~     1010   ~   0xA_   (first column of upper-case letters)
101      ~     1011   ~   0xB_   (second column of upper-case letters)
110      ~     1110   ~   0xE_
111      ~     1111   ~   0xF_

In this representation the 0101/1010/1011 upper-nybbles of the S/360's USASCII mode does make sense. 0101 is indeed where the digits are located. Having 1010 in the upper bits of the last digit of a positive number would make it compatible with EBCDIC (and its predecessors, such as in the 1401). Using 1011 as the negative-sign marker would not give the same graphical results as the EBCDIC choice, but at least it would be upper-case letters too, and there were proposals for how to represent ASCII as Hollerith punch combinations that would have made these signed integers punch the same as in the EBCDIC world (though textual data would not be compatible).
Why represent ASCII in this weird way, though? Mackenzie quotes some supposed advantages of this representation -- among others:

There would be space to extend the character set to an 8-bit character set with additional control characters contiguously with the ASCII control character.
The upper nybble of letters would be clearly distinguishable from a BCD digit, which would be useful for sanity checks of numeric format -- especially for the packed format used by the S/360 where a variable-length BCD field had a single sign nybble at the end which would move to a high nybble upon unpacking.
The DEL character would be represented by a byte of all ones, such that one could overpunch anything with it on 8-bit paper tape. (The analogous reasoning for 7-bit tape was the original reason why ASCII has this control character at the far end of the 7-bit codepoint space, away from the other controls).

Since the abacdefg proposal never reached standardization, the USASCII mode of the architecture never became useful.

