A client has requested that we build a platform for integrating data from partner to their central data store. This will not be Big Data scale.The data from each partner will be accessible through RESTful APIs. Since RESTful APIs are not standards based (at least not currently), I divide the problem into three parts:Authentication / AuthorizationAccess PatternData StructureI see each part as having it's own internal complexities.We proposed a driver model, where each API type will have a driver, which will be responsible for taking care of these three issues for each API individually.If a new API comes along, we can develop a new driver for it.The problem is that the customer does not want to require development resources for each new API.I have been looking into ETL / Data Integration tools from the likes of Informatica, IBM, etc. and also some open source options.None seem to cover even a small part of the complexities of each.I am looking for an approach that can provide the kind of solution the customer is looking for, but am having trouble finding one.
Data integration from heterogeneous sources
rest;web api;integration;etl
The fundamental problem here is that REST is not a standard. It is an architectural style for web APIs. While there are mechanisms to make these APIs self-describing, and there are web-service description formats, most APIs do not make use of these techniques  and why should they? That's a lot of enterprisey ceremony with very little value.Even once you can represent the structure of the API, you still have to assign a meaning to the data you are receiving. That's not really something a computer can figure out for itself. You'll want to look at the API you're adapting, and write the mapping yourself.The good news is that this mapping from received data to your customer's datamodel can often be quite simple, to the point that you may be able to specify  it in a configuration file. That's still a kind of programming, though. Most importantly, these configurations need to be tested very thoroughly to avoid data loss and data corruption.If the APIs and your datamodel are all based on XML, you can use XSLT to specify most of these transformations. I'm sure a similar transformation technology exists for JSON, but it would seriously be much simpler to do the transformation in JavaScript.Since there is no one-size-fits-all solution to these problems, it's not surprising that you didn't find any suitable existing product. This is a classic case where custom software development is unavoidable. The funny thing is that it's probably cheaper and quicker to just write one adapter per API (and re-use common parts as the development team notices similarities among APIs), than it is to develop a generic adapter that can deal with anything. Such vague, overcomplicated projects carry significantly more risk, and might not ever deliver any value.What options does that leave for you? Ideally, you'd be able to convince the client to discard this dream of an universal API adapter. You can try to dress up the separate adapters as an integrated platform that can be extended to new APIs with minimal configuration code. You could write a generic adapter that is able to handle the currently needed types of APIs, but would need further development if other types are needed. That's probably a good compromise. But whatever you do, don't enter a fixed-price contract to develop that utopic universal driver  that will only ever exist in your client's imagination.