If the order in a set doesn’t matter, can we change order of, say, $\Bbb{N}$? I’m given to understand that the order of the elements of a set doesn’t matter. So can I change the order of the set of natural numbers or any set of numbers ( $\mathbb{W,Z,Q,R}$ for that matter) as follows?
$$ \mathbb{N} = \{ 1,2,5,4,3,\cdots \} $$
$$ \mathbb{N}= \{3,5,\cdots,78,1,9\} $$ 

If yes, is this is really problematic? If not, then why not?

PS: I know this is a silly question, but this has been in the back of my mind for a really long time.
It's true that sets are not ordered. As to whether you can 'change' the order, you cannot change something that is not there.

However you can define any ordering on them you want. For instance, we can order the naturals the usual way $$0,1,2,3,\ldots$$ or we can define an ordering where all the even numbers come first in their usual order, then the odd numbers $$ 0,2,4,6,\ldots, 1,3,5,7,\ldots.$$ There are many, many possibilities.

Also, the ordering needs to be defined unambiguously so we know exactly the order relationship between any two elements. For instance, I don't know what you mean when you write $$ \{3,5,...,78,1,9\}.$$ It's clear that you mean  $9$ comes last (it is not a problem for an ordering to have a greatest element, even though in the two orderings I gave above, there was no greatest element), but I have no idea where $2$ goes in this ordering. If you just wrote this out of the blue, I wouldn't even be able to tell it was an ordering of the whole set of natural numbers and not just a subset.

Edit

Henning mentions an example in the comments that I think deserves mention in the answer, to reinforce the fact that there are many possibilities. Any enumeration of a countably infinite ordered set induces an order on the natural numbers. So, from the usual ordering of the rationals and an enumeration of the rationals, we get an ordering on the natural numbers that is dense, i.e. between any two numbers lies infinitely many others. We can’t even try to communicate  this ordering as a list with some ellipses.
A set does not have order, but an enumeration of a set may have, and even non-enumerable sets like the real numbers may still be orderable.

You can reorder any listing of a finite set in any sense of the word.  However, the definition of an infinite set is that it can be placed into bijection with a proper subset of itself.  An enumeration is a bijection with natural numbers.

So if your enumeration "starts" with a bijection to a proper subset, you'll be running into logical problems.  For example, ordering a listing of the naturals by starting with all odd numbers first is not going to work:

$$\{1, 3, 5, \dots, 2, 4, 6, \dots\}$$ does not work because you'll never actually get to 2: your "enumeration" will run out of steam doing just the odd numbers.