Q:

Logreader is too slow to be functional

I am currently trying to build a script that reads all relevant lines from a ( or better, several) server logs and presents them in order. I am new to shell scripting and therefore make a lot of mistakes but now I am getting to a point where my script is so slow that it is not functional.
Here is my problem. I am first identifying a number of records by an errorcode. The line with that errorcode is a unstructured xml. In order to get the other lines associated with that error, I need to find another tag in the XML which holds an ID. That ID is matched on another line which has a timestamp and a thread or task number associated. And with those two values I could theoretically get all lines associated with my error, by looking for all lines with my thread number that are in "proximity" to my error message. In theory, as the file is 150MB large.
I have therefore build code that looks like this :
grep -c to get the number of errors; grep every line that has that error in it, write it in a file with sed, go over that file and find the ID, write it in an array.
Then I will do a for loop over the IDs and

find the line with the ID, the task and a timestamp using grep over the initial file
save task and timeline into variables
and then loop again with read over the original file to
find every line with my task and a timestamp that is close in seconds to my reference.

...and here it simply dies of slowness.
Doing seds in a loop of greps seems to not be ideal but i haven't found a tool to read a file from a specific point, say, only use lines x-y or something similar.
As the relevant lines do not start with either the line with my Errorcode or the line with my ID and Task, and several not relevant lines could be within them, I feel like I have to find them all using some form of grep. I know the text of the first and the last line but I have not found a way to use that to my advantage either.
Any help is appreciated, thanks.
Edit: Yes, sorry, was a bit too vague. I have now increased the performance significantly by reducing the number of external program calls. What I did before was something like:
while read Buffer; do

  TimeStamp=$(echo $Buffer | sed 's/blabla(Timestamp)blabla/\1/g')
  [ TimeStamp -ge CompareStamp ] || continue
  echo $Buffer >>./mylog
done

So, I was looking into every line and checked the timestamp in that line whether it was close in relation to my saved timestamp. That is ridiculously slow. I substituted that code with 3 greps that only compare the timestamp part of the line and see if it fits the second of my reference or the second before or after. This works, it's just super ugly. Plus I cannot guarantee that I find just lines for my reference, as the server could process more then one case with that task in 3 seconds.
My logs look like this:
timestamp first entry task blabla
timestamp blabla task blabla
timestamp blabla task blabla
timestamp blabla task reference
timestamp blabla task blabla
timestamp blabla task blabla
<xml><error>error</error></error>
timestamp blabla task blabla
timestamp last entry task blabla

I know what last and first entry is, so I could search for it. Those blocks with identical tasks are repeated before and after this one in the logfile and lines of other tasks could be within that block as well. So my first step is to get all lines with that task into a separate file, to grep less data and to not worry about other tasks.
So in normal programming, I would now read line by line and check if it is a first entry, and always save the position of my last first entry, then upon finding my reference I would now go back to the position I saved and read every line until I would find the last entry. Is there a way to do something to that effect with shell without slowing the script down to human speed again?
Edit2: Okay, here is most of it, i have just removed the regular expressions and searchstrings:
grep 'Errorcode' logfile >> ./grablog
NumOfErrors=$(grep -c 'Errorcode' grablog)

AllPrimaryReferences=($(sed -r 's/^.*(<Referencetag>)([^<]*)(<\/Referencetag>).*$/\2/g' grablog))  

j=0
for ((i=0;i<NumOfErrors;i=i+2))
do

    Reference=$(grep  'blablablabla = '${AllPrimaryRefernces[i]} logfile)
    TimeStamp=$(echo "$Reference" | sed -r 's/^ganze Zeile/timestamp/g')

    AllTasks[j]=$(echo "$Reference" | sed -r 's/ganze Zeile/Reference/g')

    grep "${AllTasks[j]}" logfile >>./tempfile

    CompTimeStamp=$(date -d "$TimeStamp" +%Y-%m-%d' '%X)
    grep 'CompTimeStamp' tempfile >>./output

    rm tempfile
    let j++
done

rm grablog
´´´

A:

There are a few parts of your script that seem wasteful, and could be rearranged in the name of efficiency:
Building an array
grep 'Errorcode' logfile >> ./grablog
NumOfErrors=$(grep -c 'Errorcode' grablog)

AllPrimaryReferences=($(sed -r 's/^.*(<Referencetag>)([^<]*)(<\/Referencetag>).*$/\2/g' grablog))  

You've used that "NumOfErrors" later as the end-condition, with a +=2, while looping through the array.  You could instead access the array's length directly via ${#arrayname[@]}.
That means you'd only have to read grablog once, or even better, can read from a pipeline, eliminating the temporary file:
AllPrimaryReferences=( $(grep 'Errorcode' logfile | sed -r '...' | uniq ) )
NumOfRefs=${#AllPrimaryReferences[@]}

The uniq should remove the duplicate entries, instead of using +=2 to skip them. I'm assuming they're always paired up next to each other (which I can't confirm as I can't see the log file).  If not, you could add a sort before the uniq, but that's also going to be slow if there are many entries.
The part of the script inside the loop is where more attention should be focused, though.
Extracting variables with sed
You're concerned that running sed twice every loop is slow.  Some ways you might avoid this are:
Combining into one sed
Can you pipe the whole line through a single sed command, that extracts both variables, into an array?
ts_task=( $(echo "$Reference" | sed 's/\(timestamp_regex\).*\(task_regex\)/\1 \2/' ) )
timestamp="${ts_task[0]}"
task="${ts_task[1]}"

Using bash substitutions
Perhaps you don't need sed at all, and can extract the substrings you need from $References with parameter expansion.  eg. to extract everything before the first space:
timestamp=${References%% *}

Using a named pipe
If you're concerned that starting sed is the slow bit, you could start it in the background, before the loop starts, and used named pipes to communicate with the main loop.  This can get fiddly, as you need to remember which processes are hanging on read/write and background them or wait for them appropriately.
Reading the entire input file, twice, each time round the loop
This is likely to be the slowest part, and where there is the most potential for improvement.
Extracting the details from the Reference
You mention, in comments, that there's only one blablabla line per reference.  That means that the first grep is looking through the whole input file, and stopping at the match; then next time round, starting from the beginning again to find the next one.
If there's only one such line for each Reference, perhaps the whole "building an array" step was unnecessary, and you can feed the loop directly:
grep 'blablablabla = ' logfile | # match each line that defines a primary reference
  sed '...' |                    # command to extract just the timestamp and taskname
  while read ts task ; do        # assign the two required variables

    # use ts and task to extract everything as before

  done

This means that the first of those two greps is now outside the loop, so it only runes once.
Filtering the results
You're dumping a subset of lines to tempfile, then running a second grep over the results.  As before, you can avoid using temporary files by combining both steps into a pipeline.
grep "$task" logfile | grep "$timestamp" >> output

or if you know enough about the full line's format
grep "$timestamp <match other part of line> $task" logfile >> output

The whole algorithm
Even with all these improvements, the bottleneck is likely to be that you're rereading the whole logfile, and examining every line in it again, for every reference/task. That's appropriate when the required lines can appear anywhere in the logfile, in any order -- it's a brute-force approach to finding every line, so it's going to take a long time.
But you've hinted that there are structures and context("the first and the last line", "first entry")  that would allow a smarter approach.  If you know something more about the structure/order of the input file, you can probably take further shortcuts to avoid the duplicated effort.
You asked how to "save the position" and "go back to the position I saved".  grep -n will report the line number of each match, and the tail(1) command can skip a number of lines from the beginning of a file, though it will still need to reread the file to find the linebreaks.  Perhaps the whole file can instead be handled in a single while read loop?

