there are many almost duplicate questions, but none gave me the answer I'm looking for, so bear with me.consider a simple server API, in which you can consume data in iterations, using a stateless token, which might look like: rQAAMTQ2MzU4MDA1MjgxM3x8fC9wZXJtaWQub3JnfHx8, so:curl -v http://ws.foo.bar/_consume?position=rQAAMTQ2MzU4MDA1MjgxM3x8fC9wZXJtaWQub3JnfHx8* Connected to ws.foo.bar (12.34.56.78) port 80 (#0)> GET /_consume?position=rQAAMTQ2MzU4MDA1MjgxM3x8fC9wZXJtaWQub3JnfHx8 HTTP/1.1> User-Agent: curl/7.35.0> Host: ws.foo.bar> Accept: */*> < HTTP/1.1 200 OK... many irrelevant headers< X-POSITION: qwAAMTQ2MzU4MDA1MTIxOHx8fC9wZXJtaWQub3JnfHx8< Connection: close< Date: Wed, 01 Jun 2016 15:02:42 GMT{ datadatadatadatadatadatadata... }output the data to stdout, and headers (along with other verbose info), goes to stderr. one of these headers is X-POSITION, storing the next position I should supply as query parameter for the next chunk of data. I tried to come up with a one-liner I could just hit up button and execute again to retrieve the next chunk of dataI tried the following (assume initial position: 1foo2BAR3baz4QUUX5blah):$ POS=1foo2BAR3baz4QUUX5blah$ POS=$( curl -v http://ws.foo.bar/_consume?position=$POS 2>&1 | grep X-POSITION | awk '{print $3}' )which obviously won't print to screen the data since it's being swallowed by grep, but also, POS only changes once, and is not usable again. curl complains on the second invocation:* Illegal characters found in URL* Closing connection -1curl: (3) Illegal characters found in URLthough it seems that POS got the new value:$ echo $POSrQAAMTQ2MzU4MDA1MjgxM3x8fC9wZXJtaWQub3JnfHx8I suspect maybe EOF?in any case, even if I solve this, I still want to not ignore stdout.tried this solution:$ POS=`( curl -v http://ws.foo.bar/_consume?position=$POS 3>&1 1>&2- 2>&3- ) | grep X-POSITION | awk '{print $3}'`but it seems not to work either. not sure what I did wrong, or if this is the right approach...
Storing stderr output in a variable without modifing stdout, and doing so repeatedly
shell script;curl;http
The faulty character in the $POS parameter is EOL. HTTP uses CRLF as line ending. That gives for example:< X-POSITION: xxxxxxxx\r\nThat gives field 3 in awk to be xxxxxxxx\r.By using print in awk you also re-introduce the last new-line, \n, but as your expression is not quoted, this is lost.You can see this by doing something like:curl -v http://ws.foo.bar/_consume?position=$POS 2>&1 | cat -vThose ^Ms at end of lines indicate \r.Or:printf %s $pos | xxd00000000: 7251 4141 4d54 5132 4d7a 5534 4d44 4131  rQAAMTQ2MzU4MDA100000010: 4d6a 6778 4d33 7838 6643 3977 5a58 4a74  MjgxM3x8fC9wZXJt00000020: 6157 5175 6233 4a6e 6648 7838 0d         aWQub3JnfHx8.That last 0d is a CR. (Optionally do ascii on command prompt.)There is also no need to mix grep and awk as awk mathces fine on its own.To get on the right track this could be a start:pos=$(curl -v http://foo.x/pos=$pos 2>&1 | awk -vRS=\r\n '/^< X-POSITION:/{printf %s, $3}')Here one set the RS, or record-separator, in awk to CRLF,... but this would only give you the token, not the content.Assuming you do not need to actually print the content to screen, but save it to file one way could be:pos=$(curl -sD - -o $pos.out http://foo.x/?position=$pos | awk -vRS=\r\n '/^X-POSITION:/{printf %s, $2}')Herre we redirect header data to stdout by -D - and save content to file by -o $pos.out.Another advantage with this is that you only parse the header data.