 When does taking risk = stupidity?
unohoo: The answer in my opinion lies with you. I'd try to answer these questions:What are your priorities ? 
What is most important to you ?
What if you fail, especially considering you have 3 kids and a wife to support - do you have enough savings to support the family if things dont work out ?There's nothing stupid with what you're wanting to do. As long as you can take a calculated risk, you might just want to go for it.