It's not a case of picking on the SWP; merely of giving it attention commensurate with its position on the left.
I'm not sure that counter-posing the SWP to the 'Labour left', as though the only choice is between those two (equally unsuccessful) options, is very helpful.