By turning on xdebug.profiler_enable_trigger, you can at least selectively run the profiler.
Even when the profiler is off, the other primary function of xdebug (detailed stack traces on errors) causes your php instance to track lots of extra information slowing down execution.