About recommendation letters University requirements might be different but most of them ask for recommendation letters. Leaving everything else (Degree, GPA, publications) they rely heavily on the recommendation letters. Not one but three recommendation letters. At least in my field (Life sciences) it's a well known fact that you have to be recommended as if you were the next "whatever genius you like".
Since the PhD positions are limited because of many reasons (from "money" to the fact that "some top universities are proud of having a 5% acceptance") students must apply to at least (say) 10 different universities/institutes to have a minimum chance. Of course, they can be realistic about their chances and apply for the less crowded opportunities. That would mean making a list of your top 20 places and applying for the 10-20. After all, if they were "the next genius" they wouldn’t have any problem with recommendation letters. So far, so good, but they still need those letters.
As far as I know, a "normal" pre-PhD person might have at most 2 different research experiences so if they are able to get one professor to write something good about them they're OK. 
However, this means that these 3 persons have to be willing to write 10 letters for you (and logging in the online system which asks them a lot of questions about them and about the student). Basically, they can: 

1) Do copy-paste and change the name with a generic great letter. This means your recommendation letter won't be about you (or will be as yours as anybody else's)

2) Write a great unique letter

Usually a PI would have tons of students asking for recommendation letters so it's a natural part of their work to do some writing but it's still a big favor. If a PI writes too many outstanding letters the system will suspect He's a fool or even worse a liar. So my guess is that they just write "great" letters.

But, how is a great letter composed? Are the writers really aware of what they have to write? is the same letter suitable for two PhD programs? What if they're in different countries (I've been reading that south American and European PIs write "too realistic" letters for US PhDs and US PIs write "too good to be true" letters for European standards)? Then, should a pre-PhD student work in both continents first and then apply? How to deal with the fact that the PIs may be unknown? How to deal with the fact that you need many different PIs writing you many different letters? How to ask your PI a recommendation letter to "leave" him/her?  

I know that it's not ONE question but for me it's part of the same problem.
Usually a PI would have tons of students asking for recommendation letters


...which is why the experienced ones will say no to anyone they don't think they can honestly give a strong recommendation.


  it's still a big favor.


No, it's not.  It's their job.


  How is a great letter composed?


Most successful reference letters follow the same general outline:


A short blurb describing how long the writer has known you, and in what capacity (student in class, independent study, senior thesis, lab slave, coauthor, squash partner, etc.)
A detailed description of why (not just whether) the writer thinks you are a strong candidate for PhD admission, and in particular your potential for independent research in direct, personal, technical, and credible detail.  This part is different for every letter.
Direct comparisons, by name, with other students the writer has worked with and/or recommended in the past.
A short blurb describing the writer's credentials and experience.



  Are the writers really aware of what they have to write?


In general, yes—after all, they read recommendation letters themselves (or they're the wrong people to ask)—but the only way to really be sure is to ask them directly.  If they are anything but confident about their ability to write you a strong and effective letter, ask someone else.


  is the same letter suitable for two PhD programs?


At least within the US, yes, definitely.


  What if they're in different countries?


Ask your references directly whether they understand the cultural expectations in those countries.  But in my experience (in computer science), these differences are shrinking rapidly.


  Then, should a pre-PhD student work in both continents first and then apply?


Nobody can tell you what you "should" do.  There's an obvious tradeoff between breadth and depth in your pre-PhD research experience.  The choice depends on which is more likely to provide evidence of your potential as an independent researcher.


  How to deal with the fact that the PIs may be unknown?


There is no such thing as an unknown PI; everyone has a professional web page, and everyone can use Google.  (Corollary: If someone doesn't have a professional web page, do not under any circumstances ask them for a letter.)  But more junior writers do need to provide more narrative detail in their letters, to make up for being less known and less experienced.


  How to deal with the fact that you need many different PIs writing you many different letters?


What is there to "deal with"?  With rare exceptions, you need three our four PIs to write one letter each, possibly with some very minor customization.  Ask them if they are willing to write you a strong letter.  (Use the word "strong" when you ask.)  If they say yes, give them everything they need to write a strong letter, including time, and then get out of the way.


  How to ask your PI a recommendation letter to "leave" him/her?


Directly—just as if the PI were actually a responsible, mature, adult human being—and far enough in advance that they can plan for your departure.  Anything else would be incredibly disrespectful.

(If they get huffy about the idea that you might pursue opportunities elsewhere, then be very happy you asked—you really don't want to work for them.)