Why isn't finitism nonsense? This is a by product of this recent question, where the concept of ultrafinitism came up. I was under the impression that finitism was just "some ancient philosophical movement" in mathematics, only followed by one or two nowadays, so It sounded like a joke to me.

But then I got curious and, after reading a bit, It seems to me that the only arguments against infinite mathematics that finitists seem to have are that "there are numbers so big that we couldn't computate in a lifetime" or the naive set theory paradoxes. The former doesn't seem like a serious argument, and the latter is not a problem now that mathematics relies on consistent axioms.

Are there some (maybe arguably) good mathematical reason to deny the existence of $\infty$ or is it just a philosophical attitude? The concept of unboundedness seems pretty natural to me, so what could be a reason to avoid it? Does this attitude even make any sense?

In short, why today-finitists have a problem with $\infty$?

Edit: First of all, thank you so much for your answers (and comments), they have been enormously illuminating. :)

I didn't know that "finitism vs. infinitism" was such a polemic topic. Now I myself agree this question might look as primarily opinion-based. However, It was not my intention to open a debate about "which posture is better"; I was just meaning to ask about what specific mathematical reason (argued and not-primarily opinion based) do finitists have to reject the "infinitists" use of infinity.

Based on the two excellent answers I've already had (thank you again :) It is my understanding that their main problem with the use of $\infty$ is that it leads to mathematical results (like the Banach-Tarski paradox) which they don't recognize as true when looked through the glasses of our real-world experience.

Final edit: After reading every answer and comment (specially Asaf Karagila's) I've came to the conclussion that there are not strictly mathematical reason to avoid the use of infinite. That my specified question on the last edit has no answer, and that the motivation to stick to finitists or infinitists view of mathematics relies on how much one expects mathematics to describe each one's "real world". As Wildcard's answer is the one that clarifies that matter best to me, I am accepting it. Thank you all again for your answers and comments!
It's not known that modern set theory is consistent; in fact, by the Incompleteness Theorem, we can't ever have a system of axioms that we can prove is consistent. Which means that the only condition we can rely on for determining whether a set of axioms is "right" is whether or not it produces absurd results.

Under $ZFC$, we have different sizes of infinity - there are sets which are larger than the set of natural numbers in a precise sense. We also have a lot of weirdness involving the Axiom of Choice - for example, with the Axiom of Choice, a theorem of Banach and Tarski states that a hollow sphere can be disassembled into five pieces and then reassembled (without stretching, tearing, or otherwise deforming the pieces) into two spheres that are both identical to the first one in both size and shape. But the Axiom of Choice simply states that given a set of sets, we can "choose" one element from each set - which seems intuitively true.

A finitist's perspective on $ZFC$ is often that results like the hierarchy of infinite cardinals and the Banach-Tarski paradox are absurd - that they should count as contradictions, because they patently disagree with the intuitive picture of mathematics. The sensible conclusion is that one of the axioms of $ZFC$ is wrong. Most of them are intuitively obvious, because we can demonstrate them with finite sets - the only one we can't is Infinity, which states that there exists an infinite set. So a finitist's conclusion is to reject the Axiom of Infinity. Without that axiom, $ZFC$ becomes purely finitistic.

Now, many finitists are happy to stop here. But some are bothered by the fact that we still have an infinite collection of natural numbers; the infinite still "exists", in a sense, and gives the opportunity for the above weirdnesses to arise in the same way. So some people (including some mathematicians) subscribe to ultrafinitism and insist that there are only finitely many numbers at all. One ultrafinitist mathematician I know defines the largest integer to be the largest integer that will ever be referenced by humans.

Among mathematicians, ultrafinitists are much rarer than simple finitists. Finitists generally agree with you that "unboundedness" is a natural idea - it's essential, for example, in the definition of a limit. But they would go on to insist that this is just a formalism - that a limit, for example, is just a statement of eventual behavior, involving only finite numbers. So $\infty$ isn't an object, it's just a shorthand. This is (at least to my mind) more mathematically defensible than ultrafinitism.

EDIT: Since a lot of people seem to be having a hard time with my first sentence, I thought I'd clarify. The Incompleteness Theorem states that we cannot have a set of axioms powerful enough to express arithmetic and still be able to prove its consistency within the system. The reason I didn't include this phrase above is because it unnecessarily weakens the point. Any axiom system intended to codify all of mathematics defines the idea of "proof"; if, say, $T$ is intended to underlie all of mathematics, then by "proof" we must mean "proof in $T$". With such a $T$, we can say that $T$ cannot be proven consistent at all; because by Incompleteness, any proof of the consistency of $T$ would not be a proof from inside $T$, but $T$ is supposed to be powerful enough that all proofs are proofs from inside $T$. Thus: we can't ever have a system of axioms for mathematics that we can prove is consistent - full stop.
Many quality responses here, but ill add my perspective.

I agree that many defenses of finitism are silly or incomplete, but there is an underlying point to it that is quite valid and important.

The way I would put it, is that infinity as a concept should not be given the same epistemic status as other mathematical ideas. As has been noted elsewhere, the concept fundamentally cannot be defended in an empirical manner. A pure mathematician may shrug as to that argument; but even within the confines of mathematics, there are clear conceptual differences. Math involving infinity cannot be described in a system that is proven to be free of contradiction; incompleteness is a feature of math involving infinity; not of math or formal systems per se.

Personally, I am wholly unimpressed by the so called modern set theoretic 'solutions' to the properties of infinity known since ancient times. 'naively' speaking, youd think that 1-to-1 correspondence, and the ability to construct one set as a subset of the other, would be equally good axioms to decide on the relative size of sets. Both are observably true; and we should be willing to embrace all the conclusions that can be derived from either axiom, as truths about the real world. The 'modern' approach is to observe that you can just drop one of these axioms, thereby get rid of some obvious contradictions involving infinity, and be so enamoured with this to jump to the conclusion that therefore this must be a good idea, without much further consideration.

The Banach-Tarski 'paradox' can be seen as a rather direct result of this hasty philosophical choice. Its not a paradox of modern math of course; yet it puzzles a lot of people; because most people, physicists especially, do like to be able to assume their math has anything whatsoever to do with the real world.

The way physics should work, is that you try and identify what axioms must be true, and then try and reason about what other implications those givens have, and see if that actually holds in the real world. What infact we have is our best physicists (Feynman specifically) mocking the Banach-Tarski paradox as silly, rather than going out and trying to set up an experiment to reproduce it (and the latter goes for all physicists I know). Thats a big epistemological breakdown. Now I think Feynmans common sense is as usual spot-on; but if there are some conclusions from math that we should take less seriously than others, id like to know which one's those are; to codify that intuition, if you will.

Personally, I find the paradoxes of infinity hard to get around. If I divide a number by two an infinite number of times, is it zero? No, because non-zero-ness is conserved by halving; yet it has to be, since if it is nonzero, I divided a finite amount of times. Are there more natural numbers than even numbers? I could go on, and I find the arguments for and against usually equally compelling. Sure you can 'solve' these problems by 'clever' redefinition of terms and other trickery; but invariably this seems to involve driving a wedge between what you take to be axiomatically true, and any empirical obviousness of such axioms.

Which isn't only bad for the physical/empirical users of math; but generally lends a 'flavor' to modern mathematics that is off-putting to many. Bertrand Russel, who gave a big impetus to the idea that if we just boil down mathematics to as few and abstract axioms as possible, all will be great, actually disavowed the wisdom of doing so in his last years. And I imagine he would agree with the later ideas of Quine on epistemological holism. That is, the quality of an axiom should be decided on its interaction with the entirety of our web of beliefs; not on what pure mathematician or logicians thinks is 'elegant' or looks pretty on paper.

Personally, I am more in favor of a *para-consistent approach to infinity (*I guess thats what its called but it seems to mean many things to many people). That is, Id rather take an approach where I embrace the contradictions in infinity. Which doesn't mean you cannot do math at all; plenty of people knew how to do math throughout history while believing in contradictions of infinity, and it wasn't because they were 'naive', but because they had common sense. In fact, the principle of explosion is nothing but an artefact of so called 'classical' logic (specifically, the fact that it does not specify a deterministic rule for symbol-lookup). Its really not that hard in the 21th century to define your logical language to be robust to contradictions, while merrily continuing to perform deductions on your non-contradicting clauses.

What that means for the 'existence' of infinity, I dont know, but I am happy with that gray status until someone can show me any of these alleged paradoxes empirically. Personally, I am inclined to believe that the infinite divisibility of space is probably as much a figment of our imagination as the infinite divisibility of matter still was well into the 19th century.

This is mostly an academic debate with little practical implications in the end. I don't think I (or anyone with finitist inclination) should be expected to come to different conclusions on any key point. They just come to fewer conclusions. And I think research into say the real number continuum is quite valuable; even if we disagree as to the parity of the last digit of the square root of two. I say the answer is both even and non-even; you say that it's not a rational number and you think sidestepping the question like that makes you less 'naive'. Soit. 

But if I was the world's science Tsar, I would definitely defund any research into the continuum hypothesis, or anything involving hierarchies of infinities. Id tell these people to stop chasing the implications of Cantors hasty assumptions, and start doing something comparatively useful for society, like solving Sudoku's for instance.