I recently took on a contract job that will largely involve generating data feeds from my client's data and sending it to external partners, via feeds and API invocations.I've always found this particular kind of coding to be frustrating, for a number of reasons:You have to rely on specs, which often diverge from the actual implementation or simply don't go into sufficient detailYou have little to no ability to see into the external system, to ensure that your data has been accepted and interpreted properlyIf there's any kind of problem with data you've generated, it's not always straightforward to back out the changes and try againExternal testing and staging environments frequently diverge from production logic and the production databaseWhat are some best practices for doing this kind of work? Creating mocks and stubs won't necessarily resolve my problems, as my real concern is: do my feeds have the correct effect on another system, which is basically a black box? Mocks simply duplicate my assumptions.
How should I deal with external data sinks?
data structures;api design
I've done a bit of work in recent years with EDI, and it has all the problems you've mentioned.  The biggest problem is that partners don't follow their own specs and when you point it out, they say, this is a deviation but they never document it.Here's how I deal with it and manage to sleep at night:Create a framework for building and maintaining lots of unit tests.  At first you'll just test the spec plus your assumptions, but eventually (every time you find a deviation) you have to write another batch of tests to check those new facts.Your system must log everything.  It must log everything that happened, when it happened, who did it, what it received, what it wrote, where, etc.  You will need this when you go toe-to-toe with your counterpart on the other end, not just to troubleshoot, but as evidence to prove that your system is doing everything it's supposed to do.  Confirmations are particularly important to log.  These logs are invaluable when you go to write new tests.Build in the ability to retry with modifications.  When something happens, you'll want to be able to tweak the data and submit it again.Avoid intermediaries (aka 3rd parties).  Unfortunately in our case we actually have 3 different systems our data travels through between us and the ultimate destination (two of which are called value added resellers, or VARs, which is complete BS because from my perspective they add no value at all).  When something goes wrong, we get contacted by the end-user at the other end, and then we have to go up one level on our end, ask them to check, and then they go up a level and talk to their VAR, who has to talk to the other VAR, etc.  It's frustrating.  If you have the ability to communicate with the other system directly, do it.