Why is a straight line the shortest distance between two points? The first application I was shown of the calculus of variations was proving that the shortest distance between two points is a straight line. Define a functional measuring the length of a curve between two points:
$$
I(y) = \int_{x_1}^{x_2} \sqrt{1 + (y')^2}\, dx,
$$
apply the Euler-Langrange equation, and Bob's your uncle.

So far so good, but then I started thinking: That functional was derived by splitting the curve into (infinitesimal) - wait for it - straight lines, and summing them up their lengths, and each length was defined as being the Euclidean distance between its endpoints*. 

As such, it seems to me that the proof, while correct, is rather meaningless. It's an obvious consequence of the facts that (a) the Euclidean norm satisfies the triangle inequality and (b) the length of a curve was defined as a sum of Euclidean norms.

Getting slightly philosophical, I would conjecture that proving that the shortest distance between two points is a straight line is looking at things the wrong way round. Perhaps a better way would be to say that Euclidean geometry was designed to conform to our sensory experience of the physical world: the length of string joining two points is minimized by stretching the string, and at that point, it happens to look/feel straight.

I'm just wondering whether people would agree with this, and hoping that I may get some additional or deeper insights. Perhaps an interesting question to ask to try to go deeper would be: why does a stretched string look and feel straight?



*: To illustrate my point further, imagine we had chosen to define the length of a line as the Manhattan distance between its endpoints. We could integrate again, and this time it would turn out that the length of any curve between two points is the Manhattan distance between those points.
I think a more fundamental way to approach the problem is by discussing geodesic curves on the surface you call home. Remember that the geodesic equation, while equivalent to the Euler-Lagrange equation, can be derived simply by considering differentials, not extremes of integrals. The geodesic equation emerges exactly by finding the acceleration, and hence force by Newton's laws, in generalized coordinates.

See the Schaum's guide Lagrangian Dynamics by Dare A. Wells Ch. 3, or Vector and Tensor Analysis by Borisenko and Tarapov problem 10 on P. 181

So, by setting the force equal to zero, one finds that the path is the solution to the geodesic equation. So, if we define a straight line to be the one that a particle takes when no forces are on it, or better yet that an object with no forces on it takes the quickest, and hence shortest route between two points, then walla, the shortest distance between two points is the geodesic; in Euclidean space, a straight line as we know it.

In fact, on P. 51 Borisenko and Tarapov show that if the force is everywhere tangent to the curve of travel, then the particle will travel in a straight line as well. Again, even if there is a force on it, as long as the force does not have a component perpendicular to the path, a particle will travel in a straight line between two points.

Also, as far as intuition goes, this is also the path of least work.

So, if you agree with the definition of a derivative in a given metric, then you can find the geodesic curves between points. If you define derivatives differently, and hence coordinate transformations differently, then it's a whole other story.
There are several issues here:


A typo in your equation. Your $y'$ should be squared. That is $y'^2$.
(corrected).
Nothing stops you or me to use other norms. The norm 1 could have issues, since we need the functions $y$, and the "Lagrangian" to be twice continuously differentiable before deriving the Euler-Lagrange equations.
Let us assume you use an $L_4$ norm. Then define the length of a
    path as $\lim_{ | \Delta t | \to 0} \sum  \| \Delta \bf{x}_i \|_4$
    where by $\lim | \Delta t |$ I mean the size of a partition from in
    a real interval $[a,b]$ where the parametric function $\bf{x}(\it
        t)=(x(t),y(t))$ is defined. Then using the same idea of
    differentials you can say $ds = (dx^4 + dy^4)^{1/4}$ and end up with
    another equation $I(y)=\int_A^B (1 + y'^4)^{1/4} dx$. This is
    perfectly fine. You could minimize this cost function, but if you
    apply the Euler-Lagrange equation here you would get:
     \begin{equation}      \frac{d}{dx} \frac{y'^3}{(1+y'^4)^{1/4}}=0 \end{equation} 
       and then $y'^3/(1+y'^4)^{1/4}=\text{constant}\equiv C$. The
        solution in    this case is not a straight line.   So what is the
        point? 
         It is an optimization problem and the solution and interpretation changes according to the type of norm and space that
you consider. If a distance in kilometers is computed in meters we
need to  multiply by 1000 to compensate for the fact that the new metric
is now
        1/1000 of the old metric. A straight segment in a plane would be deformed (bent or stretched)
        when changing from Euclidean metric to metric such as $L_4$, to adjust to the new shortest path condition. There are two different things that change the solution of the
    problem. The curvature of the space (manifold) being used, and the
    metric that you want to impose. From differential geometry the
    metric tensor $g_{ij}$ is the natural (Riemannian) metric to impose to a curved
    space and this metric generates the geodesics of the system. That
    is, the paths of least distances on that metric. Again, you are free
    to choose another metric, but then those paths will be deformed
    accordingly.
Now, here is a question that I do not see many people (books, documents) asking. It is obvious that we are finding a minimum but this should be proven. That is, is the cost (objective) function convex? 
In other words, the solution that the shortest path between two points is straight in $L_2$ norm here in a plane is a necessary but not sufficient condition for stationarity. We still need to prove the "sufficient" part and then ask if the minimum is a local or global  minimum, if it is unique. Seems obvious but how to prove this?
Here is an interesting argument which does not require smooth constraints and the sofisticated Euler-Lagrange machinery. 
We search for the shortest path length $L(A,B)$ between two points $A$ and $B$. The distance is an invariant under translation and rotations (those are isometries). Then without loss in generality we can think that $A=(0,0)$ the origin and $B=(0,\ell)$, where $\ell$ is the straight distance between $A$ and $B$. Then the length of any segment between $A$ and $B$ is given by
\begin{equation}
L(A,B) = \int_0^{\ell} \sqrt{1+ y'^2} dx \ge \int_0^{\ell} dx = \ell.
\end{equation}
since $y'^2 \ge 0 $. So $\ell$ is a lower bound for lengths over all
qualifying segments. Even more, the straight distance is $\ell$, so
$\ell$ is the infimum, the minimum, and the shortest distance.
If you pick a metric inherited from $L_4$ norm, you would get also,
doing the same analysis that $\ell$ is a lower bound of all qualifying
paths between $A$ and $B$, but this time $\ell$ is smaller than the
possible shortest measure under this new metric.
So, metrics induced by the norm
$L_n$, $n &gt; 2$, create longer shortest paths (at least for $n$ even,
what happens if $n$ is odd?). This makes sense in the
case of the sphere, where a straight segment is shorter than a geodesic on the sphere. 


It is obvious that there is not upper bound for the distance. You give me a distance and I can always find a path which is longer. The question is: does the distance has a lower bound? The answer is "yes" since distance is always greater or equal to zero, so it has an "inf".  Still, if the infimum is unique this does not mean that the path is unique. Think of a distance on a unit sphere between two antipodes. There are an infinite number of paths which satisfy the shortest path of length $\pi$. In the case of a path on a plane we can use Euclid's postulate: There is only one line through two points. So it is unique. It rest to prove: Is the infimum the length of the line between those two points? 

For an excellent discussion on the topic of this question please check:
A course in Metric Geometry  chapter 2.