I'm following a course where the trainer recommends to identify all the vulnerabilities affecting a web application before trying to exploit them.  

While I understand the need to identify all the vulnerabilities, I don't understand why **I should wait before trying to exploit a vulnerability I just found (like a sql injection, a command injection, remote file inclusion, ...) and come back to it later?**

**Is there any reason to proceed like that?**
['https://security.stackexchange.com/questions/126705', 'https://security.stackexchange.com', 'https://security.stackexchange.com/users/70136/']
There's a chance that the result of exploiting a vulnerability early in the test could change the application. Either crashing it or just breaking certain functionality. Thus making the need to scan for additional vulnerabilities moot.

Another reason for thoroughly identifying vulnerabilities is to give the person performing the test a full understanding of the application. Perhaps that first vulnerability could be exploited even further with information gained in the later stages of the vulnerability assessment.
That sounds more like a course in how to hack someone else's site than to test yours. 

Following the suggested logic allows the perpetrator to better plan a quick attack that will let them gain control of the machine before anyone has time to notice the attempts. 

That provides no benefit for someone who is only testing their own system's security.