Q:

Decoding base64 encoded message with a key

I am pentesting a database that where I found data which was base64 encoded. I tried the usual method of decoding base64, which didn't work. 
Later, I found in database a field called base64_custom_key. How do I decrypt a base64 encoded message with a key?

A:

Base-n systems have n different symbols in their alphabet. Base 2 is just ones and zeros, base 10 has 0 through 9. Base 64 has 64 unique characters, as defined in its RFC.
If you see a 64-character long base64_custom_key field, I would guess that's the alphabet you have to use. I'll do an example.
The normal alphabet is A-Z, a-z, 0-9, + and ,. If the base64_custom_key contains ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+, then you have to do nothing because that's the original. But because the original does not decode properly, I guess that's not it.
If base64_custom_key contains ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123459876+, (everything normal, except the 9876 part), you might have to replace 9 with 6, 8 with 7, 7 with 8 and 6 with 9, and then run the decoder. The base64 encoding of zzz is enp6, but with this custom key it would be enp9. To decode it, we need to replace the 9 with a 6 as described, and you can then decode that normally.
Does that make sense?
It might not work, I'm just guessing what this might mean and how you might need to use that custom field. It's certainly non-standard and not a secure system, if it was intended for encryption at all (though I can't imagine another purpose).

