When the number of tax returns filed by individuals falls to the level you claim I'll belive you. However right now the return count is 150,000,000 for a population of 323,000,000.  The return count does not identify the number of those return for married. Nor do the number of return filed enable you to count children.

The simplest math where 2/3 are not working would leave 33% working.  Yet the returns  numbers are greater than 46% of the population. 

Maybe conservative math has different results.