The show is not wrong.

1605-Port Royal is founded as a French trading settlement. 
1607-The French government revokes the trading license; everyone goes home. Port Royal is abandoned for 3 years.
****1608-Quebec City is founded****
1610-The French return to Port Royal to resume trading.
1613-The English burn down Port Royal. It remains abandoned for 19 years.
****During this time, Quebec City continues to grow; more settlers and missionaries arrive. The first children are born in Quebec City.****
1632-Port Royal is re-re-established 8 KM away from the original site.

So for 22 of its first 27 years, Port Royal was abandoned. This does not contradict the show's claim that Quebec City was the first "permanent" European settlement.