Adam.  You are an investment banker.  If you want to derail your career, ask for paternity leave.  If you are very well connected, the firm may overlook your leave over time.  That said, going to the office with your baby in a bjorn is an unbelievably bad piece of advice,  Your colleagues will not be impressed.