1. A method for removing noise in a video or image sequence, the method comprising: receiving, at an encoder, a current frame comprising a plurality of current pixels and a previous frame comprising a plurality of previous pixels corresponding to the current pixels, the previous pixels being temporally aligned with the current pixels according to motion estimation; computing, for each current pixel of the current pixels, a similarity term for temporal bilateral filtering using the previous pixels and the current pixels, the temporal bilateral filtering including reliability based weighting of input samples using the similarity term; reconstructing, at the encoder, a reference frame corresponding to the previous frame, the reference frame comprising a plurality of first filtered pixels spatially corresponding to the previous pixels, the plurality of first filtered pixels produced in a previous noise removal operation; determining, for each current pixel of the current pixels, a current spatial filtered sample from neighboring pixels of the current pixel, the current spatial filtered sample determined using a first spatial bilateral filter with a filter kernel within the plurality of current pixels; determining, for each current pixel of the current pixels, a previous spatial filtered sample temporally corresponding to the current spatial filtered sample, the previous spatial filtered sample determined using a second spatial bilateral filter with a filter kernel within the plurality of first filtered pixels; applying the temporal bilateral filtering to each current pixel to produce second filtered pixels, the temporal bilateral filtering using, for each current pixel, the similarity term, the current spatial filtered sample, and the previous spatial filtered sample; and storing the second filtered pixels for a subsequent noise removal operation.