Q:

separate the saved output of command that process a very large compressed JSON file

Ok, let's start with the command line that i'm using:
curl --silent http://example.com/json.gz | pigz -dc | jq -r '[.name, .value] | @csv' > data.csv

CURL will download a compressed of 11.6 GB of JSON file, pigz will decompress it and write all processed output to stdout, jq will read JSON and save the output as csv file.
The problem is, the output that saved as data.csv is extremely large and after all i still need to analyze this data by using a PHP script and insert it to MYSQL in special format (the data will have very small size then)
But, i only have less than 60 GB free space left in my server, even i'm not able to decompress the full data and save it to the CSV file.
So, i got an idea, if i'm able to save the output to separated files that have different names (let's say the names are the current date or timestamp) then i can run the PHP script to process every .csv file of them and save data to db and then delete the file to free the space, not sure if this is the best way, but at least i'm trying to make it work.
So, i modified my command line to:
curl --silent http://example.com/json.gz | pigz -dc | jq -r '[.name, .value] | @csv' > `date +"%S-%M-%d-%m-%Y"`_data.csv

But, it saved it all in one file only, i thought it will save it as multi files that everyone of them have different name since the date will keep changing while the output is written.
Also, any other working solutions are welcome, thanks!

A:

Use the split command, see man-page
Simple example (10MB to STDOUT):
# dd if=/dev/zero bs=1M count=10 | split - --bytes=1M -d -a3 out
Output files (10 files with size of 1MB read from STDIN):
# stat -c "%s %n" out00*
1048576 out000
1048576 out001
1048576 out002
1048576 out003
1048576 out004
1048576 out005
1048576 out006
1048576 out007
1048576 out008
1048576 out009

Or split the saved file with split --bytes=1M -d -a3 out out
Output:
# stat -c "%s %n" out*
10485760 out
1048576 out000
1048576 out001
1048576 out002
1048576 out003
1048576 out004
1048576 out005
1048576 out006
1048576 out007
1048576 out008
1048576 out009

A:

save space with GNU split --filter
POSIX split creates output files from its input and so requires a lot of free space to store them (the size of the entire uncompressed input plus some overhead).
However, the GNU version of split has an extra --filter option which allows processing individual chunks of data in much less space as it does not need to create any temporary files:
| split -l $NUMLINES --filter='shell_command'

You can think of it like xargs -n $NUMLINES command except passing data to stdin instead of as commandline arguments.
For example, to output the md5sum of each set of (up to) 7 lines of /etc/passwd and then output the number of chunks processed:
</etc/passwd split -l7 --filter='md5sum|tee /dev/tty' |\
{ echo Processed $(wc -l) chunks; }

To modify your command to work on 10000 lines at a time, you could do something like:
curl -L --silent "$URL" |\
pigz -dc |\
jq -r '[.name, .value] | @csv' |\
split -l 10000 --filter='save2db.php'

Your filter command save2db.php should read from stdin.
If you prefer to make it read from an actual file, you can do something like:
... |\
split -l 10000 --filter='cat >TMPFILE; save2db.php TMPFILE';
rm TMPFILE

Warning: You'll need to ensure that it is safe to split your csv file on line boundaries. Some csv files contain fields with embedded literal newlines; they may become malformed if split mid-field.

