<commit_before>#lang scribble/doc
@(require "mz.rkt" scribble/scheme
          (for-syntax racket/base)
          (for-label racket/generator
                     racket/generic
                     compatibility/mlist))

@(define (info-on-seq where what)
   @margin-note{See @secref[where] for information on using @|what| as
                sequences.})

@title[#:style 'toc #:tag "sequences+streams"]{Sequences and Streams}

@tech{Sequences} and @tech{streams} abstract over iteration of elements in a
collection. Sequences allow iteration with @racket[for] macros or with sequence
operations such as @racket[sequence-map]. Streams are functional sequences that
can be used either in a generic way or a stream-specific way. @tech{Generators}
are closely related stateful objects that can be converted to a sequence and
vice-versa.

@local-table-of-contents[]

@; ======================================================================
@section[#:tag "sequences"]{Sequences}

@(define sequence-evaluator
   (let ([evaluator (make-base-eval)])
     (evaluator '(require racket/generic racket/list racket/stream racket/sequence
                          racket/contract))
     evaluator))

@guideintro["sequences"]{sequences}

A @deftech{sequence} encapsulates an ordered collection of values.
The elements of a sequence can be extracted with one of the
@racket[for] syntactic forms, with the procedures returned by
@racket[sequence-generate], or by converting the sequence into a
@tech{stream}.

The sequence datatype overlaps with many other datatypes.  Among
built-in datatypes, the sequence datatype includes the following:

@itemize[

 @item{exact nonnegative integers (see below)}

 @item{strings (see @secref["strings"])}

 @item{byte strings (see @secref["bytestrings"])}

 @item{lists (see @secref["pairs"])}

 @item{mutable lists (see @secref["mpairs"])}

 @item{vectors (see @secref["vectors"])}

 @item{flvectors (see @secref["flvectors"])}

 @item{fxvectors (see @secref["fxvectors"])}

 @item{hash tables (see @secref["hashtables"])}

 @item{dictionaries (see @secref["dicts"])}

 @item{sets (see @secref["sets"])}

 @item{input ports (see @secref["ports"])}

 @item{streams (see @secref["streams"])}

]

An @tech{exact number} @racket[_k] that is a non-negative
@tech{integer} acts as a sequence similar to @racket[(in-range _k)],
except that @racket[_k] by itself is not a @tech{stream}.

Custom sequences can be defined using structure type properties.  The
easiest method to define a custom sequence is to use the
@racket[gen:stream] @tech{generic interface}. Streams are a suitable
abstraction for data structures that are directly iterable.  For
example, a list is directly iterable with @racket[first] and
@racket[rest]. On the other hand, vectors are not directly iterable:
iteration has to go through an index. For data structures that are not
directly iterable, the @deftech{iterator} for the data structure can
be defined to be a stream (e.g., a structure containing the index of a
vector).

For example, unrolled linked lists (represented as a list of vectors)
themselves do not fit the stream abstraction, but have index-based
iterators that can be represented as streams:

@examples[#:eval sequence-evaluator
  (struct unrolled-list-iterator (idx lst)
    #:methods gen:stream
    [(define (stream-empty? iter)
       (define lst (unrolled-list-iterator-lst iter))
       (or (null? lst)
           (and (>= (unrolled-list-iterator-idx iter)
                    (vector-length (first lst)))
                (null? (rest lst)))))
     (define (stream-first iter)
       (vector-ref (first (unrolled-list-iterator-lst iter))
                   (unrolled-list-iterator-idx iter)))
     (define (stream-rest iter)
       (define idx (unrolled-list-iterator-idx iter))
       (define lst (unrolled-list-iterator-lst iter))
       (if (>= idx (sub1 (vector-length (first lst))))
           (unrolled-list-iterator 0 (rest lst))
           (unrolled-list-iterator (add1 idx) lst)))])

  (define (make-unrolled-list-iterator ul)
    (unrolled-list-iterator 0 (unrolled-list-lov ul)))

  (struct unrolled-list (lov)
    #:property prop:sequence
    make-unrolled-list-iterator)

  (define ul1 (unrolled-list '(#(cracker biscuit) #(cookie scone))))
  (for/list ([x ul1]) x)
]

The @racket[prop:sequence] property provides more flexibility in
specifying iteration, such as when a pre-processing step is needed to
prepare the data for iteration.  The @racket[make-do-sequence]
function creates a sequence given a thunk that returns procedures to
implement a sequence, and the @racket[prop:sequence] property can be
associated with a structure type to implement its implicit conversion
to a sequence.

For most sequence types, extracting elements from a sequence has no
side-effect on the original sequence value; for example, extracting
the sequence of elements from a list does not change the list.  For
other sequence types, each extraction implies a side effect; for
example, extracting the sequence of bytes from a port causes the bytes
to be read from the port. A sequence's state may either span all uses
of the sequence, as for a port, or it may be confined to each distinct
time that a sequence is @deftech{initiate}d by a @racket[for] form,
@racket[sequence->stream], @racket[sequence-generate], or
@racket[sequence-generate*]. Concretely, the thunk passed to
@racket[make-do-sequence] is called to @tech{initiate} the sequence
each time the sequence is used.

Individual elements of a sequence typically correspond to single
values, but an element may also correspond to multiple values.  For
example, a hash table generates two values---a key and its value---for
each element in the sequence.

@; ----------------------------------------------------------------------
@subsection{Sequence Predicate and Constructors}

@defproc[(sequence? [v any/c]) boolean?]{
  Returns @racket[#t] if @racket[v] can be used as a @tech{sequence},
  @racket[#f] otherwise.

@interaction[#:eval sequence-evaluator
  (sequence? 42)
  (sequence? '(a b c))
  (sequence? "word")
  (sequence? #\x)]}

@defproc*[([(in-range [end number?]) stream?]
           [(in-range [start number?] [end number?] [step number? 1]) stream?])]{
  Returns a sequence (that is also a @tech{stream}) whose elements are
  numbers.  The single-argument case @racket[(in-range end)] is
  equivalent to @racket[(in-range 0 end 1)].  The first number in the
  sequence is @racket[start], and each successive element is generated
  by adding @racket[step] to the previous element.  The sequence stops
  before an element that would be greater or equal to @racket[end] if
  @racket[step] is non-negative, or less or equal to @racket[end] if
  @racket[step] is negative.  @speed[in-range "number"]

  Example: gaussian sum
  @interaction[#:eval sequence-evaluator
    (for/sum ([x (in-range 10)]) x)]

  Example: sum of even numbers
  @interaction[#:eval sequence-evaluator
    (for/sum ([x (in-range 0 100 2)]) x)]
}


@defproc[(in-naturals [start exact-nonnegative-integer? 0]) stream?]{
  Returns an infinite sequence (that is also a @tech{stream}) of exact
  integers starting with @racket[start], where each element is one
  more than the preceding element.  @speed[in-naturals "integer"]

  @interaction[#:eval sequence-evaluator
    (for/list ([k (in-naturals)]
               [x (in-range 10)])
      (list k x))]
}


@defproc[(in-list [lst list?]) stream?]{
  Returns a sequence (that is also a @tech{stream}) that is equivalent
  to using @racket[lst] directly as a sequence.
  @info-on-seq["pairs" "lists"]
  @speed[in-list "list"]

  @interaction[#:eval sequence-evaluator
    (for/list ([x (in-list '(3 1 4))])
      `(,x ,(* x x)))]
}

@defproc[(in-mlist [mlst mlist?]) sequence?]{
  Returns a sequence equivalent to @racket[mlst].
  @info-on-seq["mpairs" "mutable lists"]
  @speed[in-mlist "mutable list"]

  @interaction[#:eval sequence-evaluator
    (for/list ([x (in-mlist (mcons "RACKET" (mcons "LANG" '())))])
      (string-length x))]
}

@defproc[(in-vector [vec vector?]
                    [start exact-nonnegative-integer? 0]
                    [stop (or/c exact-integer? #f) #f]
                    [step (and/c exact-integer? (not/c zero?)) 1])
         sequence?]{
  Returns a sequence equivalent to @racket[vec] when no optional
  arguments are supplied.

  @info-on-seq["vectors" "vectors"]

  The optional arguments @racket[start], @racket[stop], and
  @racket[step] are analogous to @racket[in-range], except that a
  @racket[#f] value for @racket[stop] is equivalent to
  @racket[(vector-length vec)].  That is, the first element in the
  sequence is @racket[(vector-ref vec start)], and each successive
  element is generated by adding @racket[step] to index of the
  previous element.  The sequence stops before an index that would be
  greater or equal to @racket[end] if @racket[step] is non-negative,
  or less or equal to @racket[end] if @racket[step] is negative.

  If @racket[start] is not a valid index, or @racket[stop] is not in
  [-1, @racket[(vector-length vec)]] then the
  @exnraise[exn:fail:contract].  If @racket[start] is less than
  @racket[stop] and @racket[step] is negative, then the
  @exnraise[exn:fail:contract:mismatch].  Similarly, if @racket[start]
  is more than @racket[stop] and @racket[step] is positive, then the
  @exnraise[exn:fail:contract:mismatch].

  @speed[in-vector "vector"]

  @interaction[#:eval sequence-evaluator
    (define (histogram vector-of-words)
      (define a-hash (make-hash))
      (for ([word (in-vector vector-of-words)])
        (hash-set! a-hash word (add1 (hash-ref a-hash word 0))))
      a-hash)
    (histogram #("hello" "world" "hello" "sunshine"))]
}

@defproc[(in-string [str string?]
                    [start exact-nonnegative-integer? 0]
                    [stop (or/c exact-integer? #f) #f]
                    [step (and/c exact-integer? (not/c zero?)) 1])
         sequence?]{
  Returns a sequence equivalent to @racket[str] when no optional
  arguments are supplied.

  @info-on-seq["strings" "strings"]

  The optional arguments @racket[start], @racket[stop], and
  @racket[step] are as in @racket[in-vector].

  @speed[in-string "string"]

  @interaction[#:eval sequence-evaluator
    (define (line-count str)
      (for/sum ([ch (in-string str)])
        (if (char=? #\newline ch) 1 0)))
    (line-count "this string\nhas\nthree \nnewlines")]
}

@defproc[(in-bytes [bstr bytes?]
                   [start exact-nonnegative-integer? 0]
                   [stop (or/c exact-integer? #f) #f]
                   [step (and/c exact-integer? (not/c zero?)) 1])
         sequence?]{
  Returns a sequence equivalent to @racket[bstr] when no optional
  arguments are supplied.

  @info-on-seq["bytestrings" "byte strings"]

  The optional arguments @racket[start], @racket[stop], and
  @racket[step] are as in @racket[in-vector].

  @speed[in-bytes "byte string"]

  @interaction[#:eval sequence-evaluator
    (define (has-eof? bs)
      (for/or ([ch (in-bytes bs)])
        (= ch 0)))
    (has-eof? #"this byte string has an \0embedded zero byte")
    (has-eof? #"this byte string does not")]
}

@defproc[(in-port [r (input-port? . -> . any/c) read]
                  [in input-port? (current-input-port)])
         sequence?]{
  Returns a sequence whose elements are produced by calling @racket[r]
  on @racket[in] until it produces @racket[eof].}

@defproc[(in-input-port-bytes [in input-port?]) sequence?]{
  Returns a sequence equivalent to @racket[(in-port read-byte in)].}

@defproc[(in-input-port-chars [in input-port?]) sequence?]{
  Returns a sequence whose elements are read as characters from
  @racket[in] (equivalent to @racket[(in-port read-char in)]).}

@defproc[(in-lines [in input-port? (current-input-port)]
                   [mode (or/c 'linefeed 'return 'return-linefeed 'any 'any-one) 'any])
         sequence?]{
  Returns a sequence equivalent to
  @racket[(in-port (lambda (p) (read-line p mode)) in)].  Note that
  the default mode is @racket['any], whereas the default mode of
  @racket[read-line] is @racket['linefeed].}

@defproc[(in-bytes-lines [in input-port? (current-input-port)]
                         [mode (or/c 'linefeed 'return 'return-linefeed 'any 'any-one) 'any])
         sequence?]{
  Returns a sequence equivalent to
  @racket[(in-port (lambda (p) (read-bytes-line p mode)) in)].  Note
  that the default mode is @racket['any], whereas the default mode of
  @racket[read-bytes-line] is @racket['linefeed].}

@defproc[(in-hash [hash hash?]) sequence?]{
  Returns a sequence equivalent to @racket[hash].

  @examples[
    (define table (hash 'a 1 'b 2))
    (for ([(key value) (in-hash table)])
      (printf "key: ~a value: ~a\n" key value))]

  @info-on-seq["hashtables" "hash tables"]
}

@defproc[(in-hash-keys [hash hash?]) sequence?]{
  Returns a sequence whose elements are the keys of @racket[hash].

  @examples[
    (define table (hash 'a 1 'b 2))
    (for ([key (in-hash-keys table)])
      (printf "key: ~a\n" key))]
}

@defproc[(in-hash-values [hash hash?]) sequence?]{
  Returns a sequence whose elements are the values of @racket[hash].

  @examples[
    (define table (hash 'a 1 'b 2))
    (for ([value (in-hash-values table)])
      (printf "value: ~a\n" value))]
}

@defproc[(in-hash-pairs [hash hash?]) sequence?]{
  Returns a sequence whose elements are pairs, each containing a key
  and its value from @racket[hash] (as opposed to using @racket[hash]
  directly as a sequence to get the key and value as separate values
  for each element).

  @examples[
    (define table (hash 'a 1 'b 2))
    (for ([key+value (in-hash-pairs table)])
      (printf "key and value: ~a\n" key+value))]
}

@defproc[(in-directory [dir (or/c #f path-string?) #f]
                       [use-dir? ((and/c path? complete-path?) . -> . any/c)
                                 (lambda (dir-path) #t)])
         sequence?]{
  Returns a sequence that produces all of the paths for files,
  directories, and links within @racket[dir], except for the
  contents of any directory for which @racket[use-dir?] returns
  @racket[#f]. If @racket[dir] is not
  @racket[#f], then every produced path starts with @racket[dir] as
  its prefix.  If @racket[dir] is @racket[#f], then paths in and
  relative to the current directory are produced.

  An @racket[in-directory] sequence traverses nested subdirectories
  recursively (filtered by @racket[use-dir?]).
  To generate a sequence that includes only the immediate
  content of a directory, use the result of @racket[directory-list] as
  a sequence.

@examples[
    (code:comment "Given a directory tree:")
    (code:comment "")
    (code:comment " /example")
    (code:comment "   +-- a")
    (code:comment "   |   |")
    (code:comment "   |   +-- alpha")
    (code:comment "   |   +-- apple")
    (code:comment "   |")
    (code:comment "   +-- b")
    (code:comment "   |   |")
    (code:comment "   |   +-- beta")
    (code:comment "   |")
    (code:comment "   +-- c")
    (code:comment "")
    (eval:alts
      (parameterize ([current-directory "/example"])
        (for ([p (in-directory)])
          (printf "~a\n" p)))
      (for ([p (in-list '("a"
                          "a/alpha"
                          "a/apple"
                          "b"
                          "b/beta"
                          "c"))])
        (printf "~a\n" p)))
    (eval:alts
      (for ([p (in-directory "/example")])
        (printf "~a\n" p))
      (for ([p (in-list '("/example/a"
                          "/example/a/alpha"
                          "/example/a/apple"
                          "/example/b"
                          "/example/b/beta"<commit_msg>Go back to using the nicer unicode picture of the directory now that Matthew added support for those characters to Scribble<commit_after>#lang scribble/doc
@(require "mz.rkt" scribble/scheme
          (for-syntax racket/base)
          (for-label racket/generator
                     racket/generic
                     compatibility/mlist))

@(define (info-on-seq where what)
   @margin-note{See @secref[where] for information on using @|what| as
                sequences.})

@title[#:style 'toc #:tag "sequences+streams"]{Sequences and Streams}

@tech{Sequences} and @tech{streams} abstract over iteration of elements in a
collection. Sequences allow iteration with @racket[for] macros or with sequence
operations such as @racket[sequence-map]. Streams are functional sequences that
can be used either in a generic way or a stream-specific way. @tech{Generators}
are closely related stateful objects that can be converted to a sequence and
vice-versa.

@local-table-of-contents[]

@; ======================================================================
@section[#:tag "sequences"]{Sequences}

@(define sequence-evaluator
   (let ([evaluator (make-base-eval)])
     (evaluator '(require racket/generic racket/list racket/stream racket/sequence
                          racket/contract))
     evaluator))

@guideintro["sequences"]{sequences}

A @deftech{sequence} encapsulates an ordered collection of values.
The elements of a sequence can be extracted with one of the
@racket[for] syntactic forms, with the procedures returned by
@racket[sequence-generate], or by converting the sequence into a
@tech{stream}.

The sequence datatype overlaps with many other datatypes.  Among
built-in datatypes, the sequence datatype includes the following:

@itemize[

 @item{exact nonnegative integers (see below)}

 @item{strings (see @secref["strings"])}

 @item{byte strings (see @secref["bytestrings"])}

 @item{lists (see @secref["pairs"])}

 @item{mutable lists (see @secref["mpairs"])}

 @item{vectors (see @secref["vectors"])}

 @item{flvectors (see @secref["flvectors"])}

 @item{fxvectors (see @secref["fxvectors"])}

 @item{hash tables (see @secref["hashtables"])}

 @item{dictionaries (see @secref["dicts"])}

 @item{sets (see @secref["sets"])}

 @item{input ports (see @secref["ports"])}

 @item{streams (see @secref["streams"])}

]

An @tech{exact number} @racket[_k] that is a non-negative
@tech{integer} acts as a sequence similar to @racket[(in-range _k)],
except that @racket[_k] by itself is not a @tech{stream}.

Custom sequences can be defined using structure type properties.  The
easiest method to define a custom sequence is to use the
@racket[gen:stream] @tech{generic interface}. Streams are a suitable
abstraction for data structures that are directly iterable.  For
example, a list is directly iterable with @racket[first] and
@racket[rest]. On the other hand, vectors are not directly iterable:
iteration has to go through an index. For data structures that are not
directly iterable, the @deftech{iterator} for the data structure can
be defined to be a stream (e.g., a structure containing the index of a
vector).

For example, unrolled linked lists (represented as a list of vectors)
themselves do not fit the stream abstraction, but have index-based
iterators that can be represented as streams:

@examples[#:eval sequence-evaluator
  (struct unrolled-list-iterator (idx lst)
    #:methods gen:stream
    [(define (stream-empty? iter)
       (define lst (unrolled-list-iterator-lst iter))
       (or (null? lst)
           (and (>= (unrolled-list-iterator-idx iter)
                    (vector-length (first lst)))
                (null? (rest lst)))))
     (define (stream-first iter)
       (vector-ref (first (unrolled-list-iterator-lst iter))
                   (unrolled-list-iterator-idx iter)))
     (define (stream-rest iter)
       (define idx (unrolled-list-iterator-idx iter))
       (define lst (unrolled-list-iterator-lst iter))
       (if (>= idx (sub1 (vector-length (first lst))))
           (unrolled-list-iterator 0 (rest lst))
           (unrolled-list-iterator (add1 idx) lst)))])

  (define (make-unrolled-list-iterator ul)
    (unrolled-list-iterator 0 (unrolled-list-lov ul)))

  (struct unrolled-list (lov)
    #:property prop:sequence
    make-unrolled-list-iterator)

  (define ul1 (unrolled-list '(#(cracker biscuit) #(cookie scone))))
  (for/list ([x ul1]) x)
]

The @racket[prop:sequence] property provides more flexibility in
specifying iteration, such as when a pre-processing step is needed to
prepare the data for iteration.  The @racket[make-do-sequence]
function creates a sequence given a thunk that returns procedures to
implement a sequence, and the @racket[prop:sequence] property can be
associated with a structure type to implement its implicit conversion
to a sequence.

For most sequence types, extracting elements from a sequence has no
side-effect on the original sequence value; for example, extracting
the sequence of elements from a list does not change the list.  For
other sequence types, each extraction implies a side effect; for
example, extracting the sequence of bytes from a port causes the bytes
to be read from the port. A sequence's state may either span all uses
of the sequence, as for a port, or it may be confined to each distinct
time that a sequence is @deftech{initiate}d by a @racket[for] form,
@racket[sequence->stream], @racket[sequence-generate], or
@racket[sequence-generate*]. Concretely, the thunk passed to
@racket[make-do-sequence] is called to @tech{initiate} the sequence
each time the sequence is used.

Individual elements of a sequence typically correspond to single
values, but an element may also correspond to multiple values.  For
example, a hash table generates two values---a key and its value---for
each element in the sequence.

@; ----------------------------------------------------------------------
@subsection{Sequence Predicate and Constructors}

@defproc[(sequence? [v any/c]) boolean?]{
  Returns @racket[#t] if @racket[v] can be used as a @tech{sequence},
  @racket[#f] otherwise.

@interaction[#:eval sequence-evaluator
  (sequence? 42)
  (sequence? '(a b c))
  (sequence? "word")
  (sequence? #\x)]}

@defproc*[([(in-range [end number?]) stream?]
           [(in-range [start number?] [end number?] [step number? 1]) stream?])]{
  Returns a sequence (that is also a @tech{stream}) whose elements are
  numbers.  The single-argument case @racket[(in-range end)] is
  equivalent to @racket[(in-range 0 end 1)].  The first number in the
  sequence is @racket[start], and each successive element is generated
  by adding @racket[step] to the previous element.  The sequence stops
  before an element that would be greater or equal to @racket[end] if
  @racket[step] is non-negative, or less or equal to @racket[end] if
  @racket[step] is negative.  @speed[in-range "number"]

  Example: gaussian sum
  @interaction[#:eval sequence-evaluator
    (for/sum ([x (in-range 10)]) x)]

  Example: sum of even numbers
  @interaction[#:eval sequence-evaluator
    (for/sum ([x (in-range 0 100 2)]) x)]
}


@defproc[(in-naturals [start exact-nonnegative-integer? 0]) stream?]{
  Returns an infinite sequence (that is also a @tech{stream}) of exact
  integers starting with @racket[start], where each element is one
  more than the preceding element.  @speed[in-naturals "integer"]

  @interaction[#:eval sequence-evaluator
    (for/list ([k (in-naturals)]
               [x (in-range 10)])
      (list k x))]
}


@defproc[(in-list [lst list?]) stream?]{
  Returns a sequence (that is also a @tech{stream}) that is equivalent
  to using @racket[lst] directly as a sequence.
  @info-on-seq["pairs" "lists"]
  @speed[in-list "list"]

  @interaction[#:eval sequence-evaluator
    (for/list ([x (in-list '(3 1 4))])
      `(,x ,(* x x)))]
}

@defproc[(in-mlist [mlst mlist?]) sequence?]{
  Returns a sequence equivalent to @racket[mlst].
  @info-on-seq["mpairs" "mutable lists"]
  @speed[in-mlist "mutable list"]

  @interaction[#:eval sequence-evaluator
    (for/list ([x (in-mlist (mcons "RACKET" (mcons "LANG" '())))])
      (string-length x))]
}

@defproc[(in-vector [vec vector?]
                    [start exact-nonnegative-integer? 0]
                    [stop (or/c exact-integer? #f) #f]
                    [step (and/c exact-integer? (not/c zero?)) 1])
         sequence?]{
  Returns a sequence equivalent to @racket[vec] when no optional
  arguments are supplied.

  @info-on-seq["vectors" "vectors"]

  The optional arguments @racket[start], @racket[stop], and
  @racket[step] are analogous to @racket[in-range], except that a
  @racket[#f] value for @racket[stop] is equivalent to
  @racket[(vector-length vec)].  That is, the first element in the
  sequence is @racket[(vector-ref vec start)], and each successive
  element is generated by adding @racket[step] to index of the
  previous element.  The sequence stops before an index that would be
  greater or equal to @racket[end] if @racket[step] is non-negative,
  or less or equal to @racket[end] if @racket[step] is negative.

  If @racket[start] is not a valid index, or @racket[stop] is not in
  [-1, @racket[(vector-length vec)]] then the
  @exnraise[exn:fail:contract].  If @racket[start] is less than
  @racket[stop] and @racket[step] is negative, then the
  @exnraise[exn:fail:contract:mismatch].  Similarly, if @racket[start]
  is more than @racket[stop] and @racket[step] is positive, then the
  @exnraise[exn:fail:contract:mismatch].

  @speed[in-vector "vector"]

  @interaction[#:eval sequence-evaluator
    (define (histogram vector-of-words)
      (define a-hash (make-hash))
      (for ([word (in-vector vector-of-words)])
        (hash-set! a-hash word (add1 (hash-ref a-hash word 0))))
      a-hash)
    (histogram #("hello" "world" "hello" "sunshine"))]
}

@defproc[(in-string [str string?]
                    [start exact-nonnegative-integer? 0]
                    [stop (or/c exact-integer? #f) #f]
                    [step (and/c exact-integer? (not/c zero?)) 1])
         sequence?]{
  Returns a sequence equivalent to @racket[str] when no optional
  arguments are supplied.

  @info-on-seq["strings" "strings"]

  The optional arguments @racket[start], @racket[stop], and
  @racket[step] are as in @racket[in-vector].

  @speed[in-string "string"]

  @interaction[#:eval sequence-evaluator
    (define (line-count str)
      (for/sum ([ch (in-string str)])
        (if (char=? #\newline ch) 1 0)))
    (line-count "this string\nhas\nthree \nnewlines")]
}

@defproc[(in-bytes [bstr bytes?]
                   [start exact-nonnegative-integer? 0]
                   [stop (or/c exact-integer? #f) #f]
                   [step (and/c exact-integer? (not/c zero?)) 1])
         sequence?]{
  Returns a sequence equivalent to @racket[bstr] when no optional
  arguments are supplied.

  @info-on-seq["bytestrings" "byte strings"]

  The optional arguments @racket[start], @racket[stop], and
  @racket[step] are as in @racket[in-vector].

  @speed[in-bytes "byte string"]

  @interaction[#:eval sequence-evaluator
    (define (has-eof? bs)
      (for/or ([ch (in-bytes bs)])
        (= ch 0)))
    (has-eof? #"this byte string has an \0embedded zero byte")
    (has-eof? #"this byte string does not")]
}

@defproc[(in-port [r (input-port? . -> . any/c) read]
                  [in input-port? (current-input-port)])
         sequence?]{
  Returns a sequence whose elements are produced by calling @racket[r]
  on @racket[in] until it produces @racket[eof].}

@defproc[(in-input-port-bytes [in input-port?]) sequence?]{
  Returns a sequence equivalent to @racket[(in-port read-byte in)].}

@defproc[(in-input-port-chars [in input-port?]) sequence?]{
  Returns a sequence whose elements are read as characters from
  @racket[in] (equivalent to @racket[(in-port read-char in)]).}

@defproc[(in-lines [in input-port? (current-input-port)]
                   [mode (or/c 'linefeed 'return 'return-linefeed 'any 'any-one) 'any])
         sequence?]{
  Returns a sequence equivalent to
  @racket[(in-port (lambda (p) (read-line p mode)) in)].  Note that
  the default mode is @racket['any], whereas the default mode of
  @racket[read-line] is @racket['linefeed].}

@defproc[(in-bytes-lines [in input-port? (current-input-port)]
                         [mode (or/c 'linefeed 'return 'return-linefeed 'any 'any-one) 'any])
         sequence?]{
  Returns a sequence equivalent to
  @racket[(in-port (lambda (p) (read-bytes-line p mode)) in)].  Note
  that the default mode is @racket['any], whereas the default mode of
  @racket[read-bytes-line] is @racket['linefeed].}

@defproc[(in-hash [hash hash?]) sequence?]{
  Returns a sequence equivalent to @racket[hash].

  @examples[
    (define table (hash 'a 1 'b 2))
    (for ([(key value) (in-hash table)])
      (printf "key: ~a value: ~a\n" key value))]

  @info-on-seq["hashtables" "hash tables"]
}

@defproc[(in-hash-keys [hash hash?]) sequence?]{
  Returns a sequence whose elements are the keys of @racket[hash].

  @examples[
    (define table (hash 'a 1 'b 2))
    (for ([key (in-hash-keys table)])
      (printf "key: ~a\n" key))]
}

@defproc[(in-hash-values [hash hash?]) sequence?]{
  Returns a sequence whose elements are the values of @racket[hash].

  @examples[
    (define table (hash 'a 1 'b 2))
    (for ([value (in-hash-values table)])
      (printf "value: ~a\n" value))]
}

@defproc[(in-hash-pairs [hash hash?]) sequence?]{
  Returns a sequence whose elements are pairs, each containing a key
  and its value from @racket[hash] (as opposed to using @racket[hash]
  directly as a sequence to get the key and value as separate values
  for each element).

  @examples[
    (define table (hash 'a 1 'b 2))
    (for ([key+value (in-hash-pairs table)])
      (printf "key and value: ~a\n" key+value))]
}

@defproc[(in-directory [dir (or/c #f path-string?) #f]
                       [use-dir? ((and/c path? complete-path?) . -> . any/c)
                                 (lambda (dir-path) #t)])
         sequence?]{
  Returns a sequence that produces all of the paths for files,
  directories, and links within @racket[dir], except for the
  contents of any directory for which @racket[use-dir?] returns
  @racket[#f]. If @racket[dir] is not
  @racket[#f], then every produced path starts with @racket[dir] as
  its prefix.  If @racket[dir] is @racket[#f], then paths in and
  relative to the current directory are produced.

  An @racket[in-directory] sequence traverses nested subdirectories
  recursively (filtered by @racket[use-dir?]).
  To generate a sequence that includes only the immediate
  content of a directory, use the result of @racket[directory-list] as
  a sequence.

@examples[
    (code:comment @#,t{Given a directory tree:})
    (code:comment @#,t{})
    (code:comment @#,t{ /example})
    (code:comment @#,t{ ├── a})
    (code:comment @#,t{ │   ├── alpha})
    (code:comment @#,t{ │   └── apple})
    (code:comment @#,t{ ├── b})
    (code:comment @#,t{ │   └── beta})
    (code:comment @#,t{ └── c})
    (code:comment @#,t{})
    (eval:alts
      (parameterize ([current-directory "/example"])
        (for ([p (in-directory)])
          (printf "~a\n" p)))
      (for ([p (in-list '("a"
                          "a/alpha"
                          "a/apple"
                          "b"
                          "b/beta"
                          "c"))])
        (printf "~a\n" p)))
    (eval:alts
      (for ([p (in-directory "/example")])
        (printf "~a\n" p))
      (for ([p (in-list '("/example/a"
                          "/example/a/alpha"
                          "/example/a/apple"
                          "/example/b"
                          "/example/b/beta"
{'fraction_non_alphanumeric': 0.1409148665819568, 'fraction_numerical': 0.0028589580686149934, 'mean_word_length': 3.2467287198165384, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2647601', 'n_tokens_mistral': 10159, 'n_tokens_neox': 9763, 'n_words': 3379}