What is the difference between regression and classification? What is the difference between regression and classification, when we try to generate output for a training data set $x$?

Regression and classification can work on some common problems where the response variable is respectively continuous and ordinal.
But the result is what would make us choose between the two. 
For example, simple/hard classifiers (e.g. SVM) simply try to put the example in specific class (e.g. whether the project is "profitable" or "not profitable", and doesn't account for how much), where regression can give an exact profit value as some continuous value.
However, in the case of classification, we can consider probabilistic models (e.g. logistic regression), where each class (or label) has some probability, which can be weighted by the cost associated with each label (or class), and thus give us with a final value on basis of which we can decide to put it some label or not. For instance, label $A$ has a probability of $0.3$, but the payoff is huge (e.g. 1000). However, label $B$ has probability $0.7$, but the payoff is very low (e.g. $10$). So, for maximizing the profit, we might label the example as label $A$ instead of $B$.
Note: I am still not an expert, perhaps someone would rectify if I am wrong in some part.
