user323: According to the Wiki:

If the blocksize is more than 250 kB, transactions get increasingly more expensive as the blocksize approaches the limit of 500 kB. Sending a transaction when the blocksize is 400 kB will cost 5 times the normal amount; sending when it's 499 kB will cost 500x, etc.

What is the exact formula that determines the transaction fee?

user575: I don't know the formula from any authoritative source, but based on the examples in the text, it is
Fee = Normal fee / (1 - Block size / 500KB)

Or equivalently (and I suspect the actual code uses this form)
Fee = Normal fee * (500 KB) / (500KB - Block size)


user940: The other answers describe the rules currently implemented in the client, but I expect those rules to be replaced once block size limits become relevant. Since those rules aren't enforced when clients verify the blockchain, they are only guidelines, and will be violated if they're against the interest of miners.
Miners will simply choose the subset of candidate transactions that respects the limits (blocksize and signature count) and maximizes the fee.
Disregarding a few minor complications (knapsack and dependent transactions) this means that miners sort the transactions by fee/transactionSize if the maximal size is the limiting factor, and fee/signatureCount if the signature count is the limiting factor.

user659: See GetMinFee() in main.h for the code which determines the minimum transaction fee.
The part you're describing is implemented as follows:
// Raise the price as the block approaches full
if (nBlockSize != 1 && nNewBlockSize >= MAX_BLOCK_SIZE_GEN/2)
{
    if (nNewBlockSize >= MAX_BLOCK_SIZE_GEN)
        return MAX_MONEY;
    nMinFee *= MAX_BLOCK_SIZE_GEN / (MAX_BLOCK_SIZE_GEN - nNewBlockSize);
}

where:

nBlockSize depends on who we are:

if we're a user making a new transaction, nBlockSize is 1
if we're a node deciding whether to relay an incoming transaction, nBlockSize is 1000
if we're a miner deciding which transactions to include in a new block, nBlockSize is the size of the block we're building before adding the new transaction

nNewBlockSize is the size of the block including the new transaction
MAX_BLOCK_SIZE_GEN is 500kB
MAX_MONEY is 21 million BTC (all the money in the world)

so it's saying:

if we're either deciding whether to relay or mine the transaction, and adding the transaction would make the block over 50% full:

never allow a transaction that makes the block 100% full
otherwise scale the fee up by a factor of 500kB / (500kB - nNewBlockSize)


Note: by rearranging the symbols, we get exactly what Meni wrote in his answer:
Fee = Normal fee / ((500kB - size) / 500kB)
         = Normal fee / (1 - size / 500kB)

323: <p>According to the <a href="https://en.bitcoin.it/wiki/Transaction_fee">Wiki</a>:</p>

<blockquote>
  <p>If the blocksize is more than 250 kB, transactions get increasingly more expensive as the blocksize approaches the limit of 500 kB. Sending a transaction when the blocksize is 400 kB will cost 5 times the normal amount; sending when it's 499 kB will cost 500x, etc.</p>
</blockquote>

<p>What is the exact formula that determines the transaction fee?</p>
 575: <p>I don't know the formula from any authoritative source, but based on the examples in the text, it is</p>

<pre><code>Fee = Normal fee / (1 - Block size / 500KB)
</code></pre>

<p>Or equivalently (and I suspect the actual code uses this form)</p>

<pre><code>Fee = Normal fee * (500 KB) / (500KB - Block size)
</code></pre>
 940: <p>The other answers describe the rules currently implemented in the client, but I expect those rules to be replaced once block size limits become relevant. Since those rules aren't enforced when clients verify the blockchain, they are only guidelines, and will be violated if they're against the interest of miners.</p>

<p>Miners will simply choose the subset of candidate transactions that respects the limits (blocksize and signature count) and maximizes the fee.</p>

<p>Disregarding a few minor complications (knapsack and dependent transactions) this means that miners sort the transactions by <code>fee/transactionSize</code> if the maximal size is the limiting factor, and <code>fee/signatureCount</code> if the signature count is the limiting factor.</p>
 659: <p>See <a href="https://github.com/bitcoin/bitcoin/blob/v0.6.0/src/main.h#L542">GetMinFee()</a> in main.h for the code which determines the minimum transaction fee.</p>

<p>The <a href="https://github.com/bitcoin/bitcoin/blob/v0.6.0/src/main.h#L575">part you're describing</a> is implemented as follows:</p>

<pre><code>// Raise the price as the block approaches full
if (nBlockSize != 1 &amp;&amp; nNewBlockSize &gt;= MAX_BLOCK_SIZE_GEN/2)
{
    if (nNewBlockSize &gt;= MAX_BLOCK_SIZE_GEN)
        return MAX_MONEY;
    nMinFee *= MAX_BLOCK_SIZE_GEN / (MAX_BLOCK_SIZE_GEN - nNewBlockSize);
}
</code></pre>

<p>where:</p>

<ul>
<li><code>nBlockSize</code> depends on who we are:
<ul>
<li>if we're a user making a new transaction, <code>nBlockSize</code> is 1</li>
<li>if we're a node deciding whether to relay an incoming transaction, <code>nBlockSize</code> is 1000</li>
<li>if we're a miner deciding which transactions to include in a new block, <code>nBlockSize</code> is the size of the block we're building before adding the new transaction</li>
</ul></li>
<li><code>nNewBlockSize</code> is the size of the block including the new transaction</li>
<li><code>MAX_BLOCK_SIZE_GEN</code> is 500kB</li>
<li><code>MAX_MONEY</code> is 21 million BTC (all the money in the world)</li>
</ul>

<p>so it's saying:</p>

<ul>
<li>if we're either deciding whether to relay or mine the transaction, and adding the transaction would make the block over 50% full:
<ul>
<li>never allow a transaction that makes the block 100% full</li>
<li>otherwise scale the fee up by a factor of <code>500kB / (500kB - nNewBlockSize)</code></li>
</ul></li>
</ul>

<p>Note: by rearranging the symbols, we get exactly what Meni wrote in his answer:</p>

<p><code>Fee</code> = <code>Normal fee / ((500kB - size) / 500kB)</code>
         = <code>Normal fee / (1 - size / 500kB)</code></p>
