you can use 1/4, 2/4, 3/4 or 4/4 of DIMM slots.  typically you want to keep DIMM 1 occupied.  12 GB would be okay, either 3x4 gb or 4+8 gb.
Im thinking of upgrading my ram from 8gb ddr3 1333MHz to 12gb ddr3 1333MHz but im not sure if the motherboard would take it. I'm also wondering if all 4 ram slots are the same and if I can use 3/4.   I have an Optiplex 790 MT  
Optiplex 790 Ram Upgrade