Q:

Manual & automatic validation losses are different

I use Tensorflow 1.14.0 and Keras 2.2.4. I am using the following code with a custom validation loss:
seedn=40
import numpy as np
np.random.seed(seedn)
import random
random.seed(seedn)
import tensorflow as tf
tf.set_random_seed(seedn)

from tensorflow.keras import optimizers
from tensorflow.keras import backend as K
from tensorflow.keras.layers import Input, Dense, Activation
from tensorflow.keras.models import Model, Sequential

def EVM_cal(data,prediction):
    return np.sqrt(np.mean((prediction - data) ** 2) / np.mean(data ** 2))*100

def EVMper(data, prediction):
    return K.sqrt(K.mean((prediction-data) ** 2) / K.mean(data** 2)) * 100

x_train_s=np.reshape(np.arange(1,1201),(100,12))/18000;
x_test_s=np.reshape(np.arange(1201,1801),(50,12))/18000;

model = Sequential()
model.add(Dense(8, input_shape=(x_train_s.shape[1],)))
model.add(Activation('linear'))
model.add(Dense(x_train_s.shape[1],))
model.add(Activation('linear'))

model.compile(optimizer='adam', loss=EVMper)
fittingadam = model.fit(x_train_s, x_train_s, epochs=4,
                        validation_data=(x_test_s, x_test_s),verbose=2)

Val_Loss = np.array(fittingadam.history['val_loss'])
print('Last Val Loss (Automatic)= ', Val_Loss[-1])

myEVM = EVM_cal(x_test_s,  model.predict(x_test_s))
print('Last Val Loss (Manual) =', myEVM)

The output is:
Last Val Loss (Automatic)=  89.88273468017579
Last Val Loss (Manual) = 89.90268535029907

As we see, the manual and automatic validation losses are off about 0.02. Here, I have generated the test and train set as an example. But, for my real dataset, the manual and automatic validation losses are off about 2. Why they are different? How can I fix this?

A:

Not sure, but this might be a matter of batch size during evaluation. Different from usual losses, yours mixes values from many samples together. (The mean of two batches is not equal to result of the entire data)
You can try batch_size=50 (the size of the entire test data) to test. 
You can also try to call the Keras metric yourself in a little model like this (may also be affected by batch size):
dataInput = Input((x_train_s.shape[1],))
predInput = Input((x_train_s.shape[1],))
output = Lambda(lambda x: EVMper(x[0], x[1]))([dataInput, predInput])

evaluator = Model([dataInput, predInput], output)
print(evaluator.predict([x_test_s, x_test_s]))

If the issue happens to be batch size, one solution, only for losses that are samplewise (don't mix samples as yours seems to mix), may be to leave the batch dimension for Keras to handle (must test). Don't take the mean of the batch dimension, only the other dimensions. The final shape of the loss should be (batch_size,).

