I've used a software suite that is installed in offices and on remote vessels. The installations communicate back and forth, and they do that by using a simple proprietary file format that looks something like this:/SHIP:16MILES=45213/ORDER:22943STATUS=OPENTOTAL=447.84URGENCY=HIGH/ORDERLINES:22943ITEM=3544QUANTITY=1PRICE=299.99ITEM=11269QUANTITY=5PRICE=29.57Recently, I've been writing a piece of software for a customer that saves information in the same kind of flat file format.When the file is opened, the lines are iterated over, and stuff happens to the lines (i.e. they're inserted into a database, or whatever).But it got to me on to thinking, how would this kind of file scale? (I like things being able to scale)I could of course gzip it; but how does a file format evolve from being something basic like to this, to being monolithic? What typical practises are employed when making a file format for a new piece of software? How are they typically built?Related: Is there a proper way to create a file format? and Should I encrypt files saved by my program
How are new file formats constructed?
file handling;file structure;files;delimited files
The ability to scale will depend on the specific usage.If I take your example of lines inserted in a database, the closer model is a log. An application, such as a web server, writes some data to a log. Daily (or once per hour, or any other period of time), the log is rotated, i.e. the application frees the current file and starts writing to another one. Once the file is freed, an ETL can process this file and load the transformed data into the database.If I take a different example, such as a large file (and by large, I mean several gigabytes or terabytes) which should be read in a context where any information in it should be accessed quickly, then the format would be different and will probably use pages and indexes to point to the right content; additionally, fragmentation will be a concern too if the data in the file is modified. You can find more information about this sort of usage by reading about PST file format used by Microsoft Outlook (it can often take gigabytes) or file formats used by database files.This means that the format you are actually using is maybe extremely scalable in the context in which it is used.How are they typically built?Like any data structure and any piece of software in general.Ideally, during architecture and design phase, developers think about ways they can store information in a file, given the different requirements, priorities and constraints. Then the file format can evolve to take in account new requirements, priorities and constraints, while being, if needed, backwards compatible.Examples:If a requirement in the format you've shown in your question is that values can be multiline and contain =, this brings a specific issue of a value such as 12345=PRICE=123.If a requirement is to follow the standards, then something like EDIFACT can be used instead of the current format (maybe with some metadata if needed).If the priority is to make the file readable, item and price are fine or may even be expanded to be more explicit. If the priority is to shorten the size of the file, item could become i, quantity  q, etc. Even better, the file can become:> 22943:3544,1,299.99;11269,5,29.57or be transformed to a binary format.If a constraint is to keep the data secure, cryptography will be used. If another constraint tells that some of the involved systems don't support Unicode, this is an additional problem to solve.