I have this input:      startdate             end date         val1    val22015-10-13 07:00:02 2015-10-19 00:00:00      45      1900in which one line specifies a date range that spans multiple days,and I want to split the range into separate time periods,each one being a subset of a day (each one on a separate line),to facilitate parallel processing of the (multi-day) range.The output should be2015-10-13 07:00:02 2015-10-13 23:59:59      45      19002015-10-14 00:00:01 2015-10-14 23:59:59      45      19002015-10-15 00:00:01 2015-10-15 23:59:59      45      19002015-10-16 00:00:01 2015-10-16 23:59:59      45      19002015-10-17 00:00:01 2015-10-17 23:59:59      45      19002015-10-18 00:00:01 2015-10-18 23:59:59      45      19002015-10-19 00:00:01 2015-10-19 00:00:00      45      1900where the data after the end time (val1 and val2)are replicated on each line.Actually the input records are coming from the hive table and the output records also will store it in split table.Modifications:date split is fine. need to split the val2 value also as per the split date.if the date diff is 2 then we would split 2 rows that should berow 1:ratio= ratio of timespent 1st day (i.e. end-start on day 1)  / val1 val2= ratio*val2row 2:ratio= ratio of timespent 1st day (i.e. end-start on day 2)  / val1 val2 = ratio*val2How can I script this?
How to split the date range into days using script
shell script;shell;hadoop
This script will do what you want(if I understand your requirements correctly).I took the liberty of extrapolating your specificationto allow the input to have one header lineand then any number of lines with date/time ranges.Ill illustrate this, and discuss it further, below.#!/bin/shif IFS= read headerthen        printf %s\n $headerelse        echo 'EOF on first line!' >&2        exit 1fiwhile read start_date start_time end_date end_time other_data           # See note, below.do        start_epoch=$(date +%s -d $start_date $start_time)  ||  {                echo Error processing start date&time $start_date $start_time >&2                exit 1        }        end_epoch=$(date +%s -d $end_date $end_time)  ||  {                echo Error processing end date&time $end_date $end_time >&2                exit 1        }        if [ $end_epoch -lt $start_epoch ]        then                echo End date&time $end_date $end_time is before start date&time $start_date $start_time >&2                # Now what?                continue        fi        ok_seq=1        # Flag: we are moving forward.        current_date=$start_date        current_time=$start_time        while [ $ok_seq -ne 0 ]        do                # Most days end at 23:59:59.                eod_time=23:59:59                eod_epoch=$(date +%s -d $current_date $eod_time)  ||  {                        # This should never happen.                        echo Error processing end-of-day date&time $current_date $eod_time >&2                        exit 1                }                if [ $end_epoch -lt $eod_epoch ]    # Were passing the end of the date/time range.                then                        if [ $current_date != $end_date ]                        then                                # Sanity check -- this should not happen.                                echo We're finishing, but the current date is $current_date and the end date is $end_date >&2                        fi                        eod_time=$end_time                        ok_seq=0                fi                                                                        # See note, below.                printf %s %s %s %s      %s\n $current_date $current_time $current_date $eod_time $other_data                # We could also use +%F for the full YYYY-mm-dd date.                current_date=$(date +%Y-%m-%d -d $current_date next day)  ||  {                        # This shouldnt happen.                        echo Error getting next day after $current_date >&2                        exit 1                }                current_time=00:00:01        donedoneDiscussion:Read the header line.If this fails, abort the script.If it succeeds, write the line to the output.If(asyour question shows) you dont want the header in your output,remove the printf%s\n$header statement.As mentioned above:loop, reading start/end/value lines from the inputuntil we reach the end of the input (or get a fatal error).If you dont want to do this,remove the while, the do, and the corresponding done.Read the start date, start time, end date, end time, and other data.other_data includes everything after the end time,i.e., val1 and val2 (and all the space between them).Use the date +%s -d date/time string commandto convert arbitrary date/time strings toUnix epoch times the number of seconds since 1970-01-01 00:00:00 (GMT).This lets us validate input (and exit in case of error),and also gives us numbers that we can compare.(Although I suppose we could just do string comparisonon values formatted as YYYY-MM-DD HH:MM:SS.)If the end date/time is before the start date/time,skip this record and go to the next line.If youd rather do something else (like terminate) in this case,change this code.Set a flag (ok_seq) that we will useto control the loop that steps through the days.Initialize the start date/time for the first dayto be the start date/time for the entire period.On each output line, the start date and the end date are the same.On most lines, the end of day (eod) time is 23:59:59.If (same date) + 23:59:59 is greater (later) thanthe end-of-period date/time,then we are on the last day (output line) of the range.Set the eod time to the end time,and set ok_seq to 0 so we will exit the loop.Write a line of output, including the other data (val1 and val2, etc.)Compute the next days date.Set the start time to 00:00:01,which will appear on each output line except for the first.Example:$ cat input      startdate             end date         val1    val22015-10-13 07:00:02 2015-10-19 00:00:00      45      19002015-11-01 08:30:00 2015-11-05 15:00:00      42      60832015-12-27 12:00:00 2016-01-04 12:34:56      17      quux$ ./script < input      startdate             end date         val1    val22015-10-13 07:00:02 2015-10-13 23:59:59      45      19002015-10-14 00:00:01 2015-10-14 23:59:59      45      19002015-10-15 00:00:01 2015-10-15 23:59:59      45      19002015-10-16 00:00:01 2015-10-16 23:59:59      45      19002015-10-17 00:00:01 2015-10-17 23:59:59      45      19002015-10-18 00:00:01 2015-10-18 23:59:59      45      19002015-10-19 00:00:01 2015-10-19 00:00:00      45      19002015-11-01 08:30:00 2015-11-01 23:59:59      42      60832015-11-02 00:00:01 2015-11-02 23:59:59      42      60832015-11-03 00:00:01 2015-11-03 23:59:59      42      60832015-11-04 00:00:01 2015-11-04 23:59:59      42      60832015-11-05 00:00:01 2015-11-05 15:00:00      42      60832015-12-27 12:00:00 2015-12-27 23:59:59      17      quux2015-12-28 00:00:01 2015-12-28 23:59:59      17      quux2015-12-29 00:00:01 2015-12-29 23:59:59      17      quux2015-12-30 00:00:01 2015-12-30 23:59:59      17      quux2015-12-31 00:00:01 2015-12-31 23:59:59      17      quux2016-01-01 00:00:01 2016-01-01 23:59:59      17      quux2016-01-02 00:00:01 2016-01-02 23:59:59      17      quux2016-01-03 00:00:01 2016-01-03 23:59:59      17      quux2016-01-04 00:00:01 2016-01-04 12:34:56      17      quuxObserve that it has no problem rolling over,not only from one month to the next, but also from one year to the next.Note:When I wrote the above version of the script,I couldnt figure out how to capture the white spacebetween the end time and val1, so I was getting output that looked like      startdate             end date         val1    val22015-10-13 07:00:02 2015-10-13 23:59:59 45      19002015-10-14 00:00:01 2015-10-14 23:59:59 45      19002015-10-15 00:00:01 2015-10-15 23:59:59 45      1900                    so I cheated,by building the right amount of space into the printf command(before the last %s).But if you change the spacing in your input,the above version of the script will againproduce incorrectly aligned columns.I figured out how to fix it, although its a little messy.Replace the while do start_epoch= lines with:while read start_date start_time end_date other_datado        # $other_data includes end_time and all the following values.        # Break them apart:        end_time=${other_data%%[       ]*}        other_data=${other_data#$end_time}        start_epoch=where end_time has been removed from the read command,and the characters between the brackets [ and the ]are a space and a tab.So now other_data contains the spaces before val1.Then change the printf to                printf %s %s %s %s%s\n $current_date $current_time $current_date $eod_time $other_data(note that there is no space between the fourth and fifth %s).So now youre done.