Why is there such variation in PhD length internationally? I'm an undergrad in Australia, and at this stage I want to go on to do further study in maths (or maybe physics/CS, still have a few years till then).

Should I do so, I imagine that I will want to complete a PhD overseas. Possibly at a university in the US or maybe Oxbridge in the UK.

In Australia, PhDs typically take 3 or 4 years, and I would be eligible to go straight into a PhD upon completion of my undergraduate degree (it is research focused and includes an honours year, see here).

But, much of what I have found for overseas institutions has been in the range of 5-7 years, or maybe a 3-4 year PhD with a few years doing a master's degree beforehand.

Does this indicate that the level of rigour and amount of work produced is significantly different for various PhD programs, or is it the case that some programs are more relaxed and simply spread the same amount of work over a longer period of time?
In the UK system (or similar systems, like Australia) students enter their Ph.D. program with a stronger background due to earlier specialization, and they typically leave their Ph.D. program at a less advanced level than a Ph.D. from a comparable American school.  The difference is not in how fast people learn, but rather differences in what a bachelor's degree means and what a Ph.D. degree means.

Note that on specialization for undergraduates the US is the outlier, but in terms of Ph.D. outcome it's the UK that is unusual.  For example, Chris Parks writes, "Across Europe the view prevails that the three-year UK doctorate is too short and thus of inadequate quality compared with the more common four-year doctorate."