OverviewI've got 2 systems over a network. The 2nd system produces some results based on the current contents of system A. How can I efficiently produce results in system B if getting the contents of system A takes a long time?Current sitationI've got a system, system A, that keeps customers (personal details, etc). The user can manually enter hits in this system.I've got another system, system B, that performs, every night, some checks on each customer. It checks if a customer appears on a restricted-persons list producing a report of hits. A user checks on this system each morning and screens manually the produced hits. If a hit is approved, it is then send via POST for saving into system A.The problemUser might have entered a manual hit in System A, which was also found automatically in System B. Solution ASend a GET request to system A, every time I present system's B overnight results to the user, to get the existing hits and filter them out of hits produced by System BIn order to ameliorate this mismatch, every time I present the results of system B in the morning, I can send a GET request to get the existing hits of a customer in System A and then perform a comparison, filtering out hits that exist in A and B's hits. This allows me to show only the new hits in System B. The user screens only the new hits that were found.A problem with this solutionI can't efficiently know beforehand how many actual new hits exist in System B. I need to GET 50,000 customers, from System A, each time I want to get the number of new hits.Solution BSend a GET request to system A, when a hit is produced by System B, overnight, to see if it's really a new hit that doesn't exist in System A. Problem with this solutionSince the GET request is sent at the time a hit is produced while checking the list and not when the results are presented to the user there might be a mismatch. The user might have not tended to the results of System B and went ahead and manually entered a hit in System A. System B would still show that manually-entered hit from System A as a new hit in System's B morning results.Is there a standard way of dealing with such problems?DetailsThe 2 systems are built on Node.js/Express framework. I have build those systems myself so I have full control over them.The systems can communicate with each other using a REST API.I'd like to keep the 2 systems decoupled, honouring the Single Responsibility Principle - i.e System A just holds it's data, System B performs automatic screenings etc. I'd specifically like to avoid intertwining functionality between each system
Efficient change comparisons across networked systems
design;architecture;rest
I've solved this with a combination of the following:HTTP CachingTo get the new hits, I send a GET request to grab the hits from system A. That GET request is cached for exactly 1 minute. Any subsequent requests are served from the cache instead of fetching over the network again. There might be a mismatch in the new hit count but it's for very brief periods of time (when the cache is stale). I can certainly live with thatBypass cache when user focuses attention and get a fresh copyWhen the user focuses on a particular hit (and is ready to transfer it), I perform a direct GET request again for that particular piece of data, bypassing the cache. This ensures that when user is actually about to make a decision/operation on a hit, he is presented with the guaranteed, non-stale piece of data.