We have a couple of applications which are very, very chatty.  There is a database call for every.  Single.  Little.  Thing.  Serving reference data again and again and again is a major part of the workload on the system.  All that scheduling of worker threads, acquiring and dropping locks, plan cache checking etc. adds up even if there is not actual disk IO.  Contention is higher because transactions have to hold locks across multiple DB calls and so throughput is much lower than it could be.  Those teams are now looking at having to buy new, very expensive DB servers because of this.

So, although the majority of the elapsed time in your system's current configuration is taken with REST API calls, ignoring performance at the DB level is storing problems for the future.

Are mutliple database calls really significant with a network call for a web API?
We have old system using .NET 3.5 (run on Windows Server 2003 32bits) to verify e-receipts from our customers (all use SHA1RSA algorithm as agreement). We store their certificates in database in Base64 string. Old code look like that:

//Store certificates
String certificateEncodeBase64 = Convert.ToBase64String(byteArrayCertificateFile);

//Verify signatures

//Hash algorithm SHA-1
HashAlgorithm hashAlgrothm = new SHA1Managed();
//Data
byte[] data = System.Text.Encoding.UTF8.GetBytes(request.DataInUTF8);
//Signature
byte[] signature = Convert.FromBase64String(request.SignatureInBase64);
//Certificate
X509Certificate2 certificate = new X509Certificate2(Convert.FromBase64String(certEncodeBase64));
//Verify using algorithm RSA
RSACryptoServiceProvider pubKey = (RSACryptoServiceProvider)certificate.PublicKey.Key    
bool vr = pubKey.VerifyData(data, hashAlgrothm, signature);


Nothing issues except the performance is not good. That's why we want to move to new system using Java 1.7 (run on Windows Server 2012 64 bits). Our new code look like that:

//Using the same DB store certificates

//Verify signatures using Bouncy Castle JCE Provider

//Hash data using SHA-1
byte[] data_ = request.getDataInUTF8().getBytes("UTF-8");
MessageDigest messageDigest_ = MessageDigest.getInstance("SHA-1", "BC");
byte[] hashed = messageDigest.digest(data);
//Signature
byte [] signature = new BASE64Decoder().decodeBuffer(request.getSignatureInBase64());
//Certificate
CertificateFactory certificateFactory = CertificateFactory.getInstance("X509", "BC");
byte[] buffer = new BASE64Decoder().decodeBuffer(certEncodeBase64);
ByteArrayInputStream byteArrayCertificateInputStream = new ByteArrayInputStream(buffer);
Certificate certificate = certificateFactory.generateCertificate(byteArrayCertificateInputStream);
//Verify using algorithm RSA
Signature verifyEngine = Signature.getInstance("RSA", "BC");
verifyEngine.initVerify(certificate);
verifyEngine.update(hashed);
result = verifyEngine.verify(signature);


And we get this problem: In our test using all verified data taken from old system, new system verify some customer's data but some customers always return fail.
For one failed example

Certificate in Base64

MIIGVzCCBD+gAwIBAgIQVAHV43oelxumKTwtrMV+ljANBgkqhkiG9w0BAQUFADBpMQswCQYDVQQGEwJWTjETMBEGA1UEChMKVk5QVCBHcm91cDEeMBwGA1UECxMVVk5QVC1DQSBUcnVzdCBOZXR3b3JrMSUwIwYDVQQDExxWTlBUIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTEyMDQxNjA1NTg0NloXDTE0MDQxNjA1NTg0NlowgcExCzAJBgNVBAYTAlZOMRIwEAYDVQQIDAlIw6AgTuG7mWkxFzAVBgNVBAcMDkhhaSBCw6AgVHLGsG5nMTMwMQYDVQQKDCpOR8OCTiBIw4BORyBUTUNQIEPDlE5HIFRIxq/GoE5HIFZJ4buGVCBOQU0xUDBOBgNVBAMMR05Hw4JOIEjDgE5HIFRNQ1AgQ8OUTkcgVEjGr8agTkcgVk4gLSBUUlVORyBUw4JNIEPDlE5HIE5HSOG7hiBUSMOUTkcgVElOMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqxx3EzTjZLaBh2ymig4aQoqZK5R/wiVnq8hGUjyG95Fvr+IZvBKrE0Rp6+yZI5yQfWhLKoXwrcfV9qwYvDaeL7BQJpCDnEu3Xih3E66o59/zx/PjSCbhA+c9kuu5irasTLuxX2iUBbBZjzDw4TLzcWeo7HAY+WgNik1YjOy8ZINnYJfB99cwpvDpAL1nw+4dX8hYaKbwJiVLf+KwGvj/v/jxTCiZo1Ob48chiw88Y2S3ak0K26euZxQ3lFYoMKs0qp2FQuK7x0B1UY3WymXc+qWFp6z7p+XiVIyr0Rv1/C8mO4AzCY2xKkfkPTkWyHay5NffbP8gJYV+J/KYO6uDEQIDAQABo4IBoDCCAZwwcAYIKwYBBQUHAQEEZDBiMDIGCCsGAQUFBzAChiZodHRwOi8vcHViLnZucHQtY2Eudm4vY2VydHMvdm5wdGNhLmNlcjAsBggrBgEFBQcwAYYgaHR0cDovL29jc3Audm5wdC1jYS52bi9yZXNwb25kZXIwHQYDVR0OBBYEFM+JjYgczFow9j23/ufgnb0BiO91MAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBmnA1dUCihWNRn3pfOJoClWsaq8wdgYDVR0gBG8wbTA0BgkrBgEEAYH6OgIwJzAlBggrBgEFBQcCARYZaHR0cDovL3B1Yi52bnB0LWNhLnZuL3JwYTA1BgsrBgEEAYH6OgICAjAmMCQGCCsGAQUFBwICMBgeFgBEAEkARAAtAEIAMQAuADAALQAyAHkwMQYDVR0fBCowKDAmoCSgIoYgaHR0cDovL2NybC52bnB0LWNhLnZuL3ZucHRjYS5jcmwwDgYDVR0PAQH/BAQDAgTwMB8GA1UdJQQYMBYGCCsGAQUFBwMCBgorBgEEAYI3CgMMMA0GCSqGSIb3DQEBBQUAA4ICAQBvkxvjAFJKM5/4wgtW2wx+Q0O2XhanV5nPv+mW2dODyner4wki+7hZQZ6LeufZ/ktNKAv2gm9Jt9uiQDVNh037WJu5haYBhxGFgFr96ZY9Cb9boPJ5p7u+k3K/4EDlmmVh4QgV/3HbKedpfvU7QtpSHyv6Fff7KaP3MzFgQZekocHaB8kRn8XL6vc+OLZp+wXeg9WYL9x0aO+JUSU5mWzPKX5tVukgciVWblVAN8jVOgZl+VgkJa10dQZP32Zk6AuBmznQJu8Oxpbrn600lyC8lj78f2tNxTgYrbw28Cx5+N5YYL7lIzsJAfnfKNTlFx9me/a/CL8b1KWx0FyjUN50tU7C2GX+Vt/lnSlvfH5gU/UKbiorqz5pyEzSocmScrcuWuzvZzHQkgxP/fsGfrAfN9cl+2GPewtzb+e2WaGoBo56t908IBDaQGlwHx5lpnM8i/8LERXv4It37SqdSe/2mtah7AvmIYkYu6REVQC1/bnfra7/gtk5i/nGA8vdxU417VgYOmQu2sYcEPd/3KhDjImiAkYd6UzFGXpZJL8/uqjL70U7++V6VObjpbsOuuZwh1Xl9h2Mbds++EnUvmP+Jb8R9m0U4dPFr8POexPJwdDGLE5aEjn4TrPgkEIaKyx+FRVLZUJzUELQODfvdGgt8eSNSTqRc+xp3/eutP56NQ==


RawData

&lt;HEADER&gt;&lt;VERSION&gt;1.0&lt;/VERSION&gt;&lt;SENDER_CODE&gt;TTSP_ICB&lt;/SENDER_CODE&gt;&lt;SENDER_NAME&gt;He Thong Thanh toan song phuong cua Ngan hang Vietinbank&lt;/SENDER_NAME&gt;&lt;RECEIVER_CODE&gt;TTSP_KBA&lt;/RECEIVER_CODE&gt;&lt;RECEIVER_NAME&gt;He thong Thanh toan song phuong cua Kho bac Nha nuoc&lt;/RECEIVER_NAME&gt;&lt;TRAN_CODE&gt;103&lt;/TRAN_CODE&gt;&lt;MSG_ID&gt;TTSP_ICB103004203763&lt;/MSG_ID&gt;&lt;MSG_REFID /&gt;&lt;SEND_DATE&gt;04-03-2014 09:03:15&lt;/SEND_DATE&gt;&lt;ORIGINAL_CODE&gt;TTSP_ICB&lt;/ORIGINAL_CODE&gt;&lt;ORIGINAL_NAME&gt;He Thong Thanh toan song phuong cua Ngan hang Vietinbank&lt;/ORIGINAL_NAME&gt;&lt;ORIGINAL_DATE&gt;04-03-2014 09:03:15&lt;/ORIGINAL_DATE&gt;&lt;ERROR_CODE /&gt;&lt;ERROR_DESC /&gt;&lt;SPARE1 /&gt;&lt;SPARE2 /&gt;&lt;SPARE3 /&gt;&lt;/HEADER&gt;&lt;BODY&gt;&lt;MT_ID&gt;1420110300335833&lt;/MT_ID&gt;&lt;SEND_BANK&gt;89201001&lt;/SEND_BANK&gt;&lt;RECEIVE_BANK&gt;89701002&lt;/RECEIVE_BANK&gt;&lt;CREATED_DATE&gt;04-03-2014 09:26:24&lt;/CREATED_DATE&gt;&lt;CREATOR&gt;DD740063&lt;/CREATOR&gt;&lt;MANAGER&gt;DD740096&lt;/MANAGER&gt;&lt;VERIFIED_DATE&gt;04-03-2014 09:29:41&lt;/VERIFIED_DATE&gt;&lt;F20&gt;1474010300001334&lt;/F20&gt;&lt;F23B&gt;CRED&lt;/F23B&gt;&lt;F26T&gt;C&lt;/F26T&gt;&lt;F32AS1&gt;04-03-2014&lt;/F32AS1&gt;&lt;F32AS2&gt;VND&lt;/F32AS2&gt;&lt;F32AS3&gt;324000000.00&lt;/F32AS3&gt;&lt;F33BS1&gt;VND&lt;/F33BS1&gt;&lt;F33BS2&gt;324000000.00&lt;/F33BS2&gt;&lt;F36 /&gt;&lt;F50P1&gt;VND101101001&lt;/F50P1&gt;&lt;F50AP2 /&gt;&lt;F50KP2&gt;HUYNH THI KIM THOA&lt;/F50KP2&gt;&lt;F51AP1S1 /&gt;&lt;F51AP1S2 /&gt;&lt;F51AP2 /&gt;&lt;F52P1S1 /&gt;&lt;F52P1S2 /&gt;&lt;F52AP2&gt;89201001&lt;/F52AP2&gt;&lt;F52DP2&gt;89201001&lt;/F52DP2&gt;&lt;F53P1S1 /&gt;&lt;F53P1S2 /&gt;&lt;F53AP2 /&gt;&lt;F53BP2 /&gt;&lt;F53DP2 /&gt;&lt;F54P1S1 /&gt;&lt;F54P1S2 /&gt;&lt;F54AP2 /&gt;&lt;F54BP2 /&gt;&lt;F54DP2 /&gt;&lt;F55P1S1 /&gt;&lt;F55P1S2 /&gt;&lt;F55AP2 /&gt;&lt;F55BP2 /&gt;&lt;F55DP2 /&gt;&lt;F56P1S1 /&gt;&lt;F56P1S2 /&gt;&lt;F56AP2 /&gt;&lt;F56CP2 /&gt;&lt;F56DP2 /&gt;&lt;F57P1S1&gt;C&lt;/F57P1S1&gt;&lt;F57P1S2 /&gt;&lt;F57AP2&gt;89701002&lt;/F57AP2&gt;&lt;F57BP2 /&gt;&lt;F57CP2 /&gt;&lt;F57DP2&gt;KBNN PHU TAN - AN GIANG&lt;/F57DP2&gt;&lt;F59P1&gt;372301048091&lt;/F59P1&gt;&lt;F59AP2 /&gt;&lt;F59P2&gt;UBND XA BINH THANH DONG#####&lt;/F59P2&gt;&lt;F70 /&gt;&lt;F71A&gt;SHA&lt;/F71A&gt;&lt;F71GS1 /&gt;&lt;F71GS2 /&gt;&lt;F72&gt;UBND XA BINH THANH DONG NOP TIEN XE CHUYEN BENH&lt;/F72&gt;&lt;F77B /&gt;&lt;F77TS1 /&gt;&lt;F77TS2 /&gt;&lt;F77TS3 /&gt;&lt;F77TS4 /&gt;&lt;F77TS5 /&gt;&lt;F77TS6 /&gt;&lt;F77TS7 /&gt;&lt;F77TS8 /&gt;&lt;F77TS9 /&gt;&lt;F77TS10 /&gt;&lt;F77TS11 /&gt;&lt;F77TS12 /&gt;&lt;F77TS13 /&gt;&lt;F77TS14 /&gt;&lt;F77TS15 /&gt;&lt;F77TS16 /&gt;&lt;F77TS17 /&gt;&lt;F77TS18 /&gt;&lt;F77TS20 /&gt;&lt;F77TS19 /&gt;&lt;/BODY&gt;


Hash data in Base64 using SHA-1

KLpuojRk329lb++8CFKTFADh+3c=


Signature in Base64

KWMVPKpQ8Pg3fUij9pND/me3orlplkbdNLWcH5rztMhIz0BjTN8CJJHiFrN1w1Qkih0lSl24zDE5nbYw/vo+EG3tSJzC+d/h7wkm6ad9LHt5Y0cxYRIKWSHJRbCjkOXcO/eE3KZaAZONGEVWt1bLkL3zERN11RAyZjFEzrGE7WbiLvp/51Cy5zBU/mLJts3waiEpqycjOOVNo59ICulwclhcPR+gVzcwh65oxBelI+XRbRN1PfEw5J48R6fTwmdtRzD+OJejMLHmiHUBErRNo+XG7Q5ovwY1AG2GfCOCYIOu2jYd5DD9ZDuLEMa54Uz7cy3UbciXG5/n+7GtLfGeHA==


For investigate further, we debug class DigestSignatureSpi that Bouncy Castle use to verify at line

try
{
    sig = cipher.processBlock(sigBytes, 0, sigBytes.length);

    expected = derEncode(hash);
}
catch (Exception e)
{
    return false;
}


sig is

[48, 33, 48, 9, 6, 5, 43, 14, 3, 2, 26, 5, 0, 4, 20, 40, -70, 110, -94, 52, 100, -33, 111, 101, 111, -17, -68, 8, 82, -109, 20, 0, -31, -5, 119]


while expected is

[40, -70, 110, -94, 52, 100, -33, 111, 101, 111, -17, -68, 8, 82, -109, 20, 0, -31, -5, 119]


This debug show sig include expected bit (last 20 bytes).

I don't know what is wrong. PLEASE HELP!!! Thank you.
