<commit_before>SALDL(1)
=========
:doctype: manpage

NAME
----

saldl - A lightweight well-featured CLI downloader optimized for speed and
early preview.


SYNOPSIS
--------

*{manname}* <<OPTIONS,[OPTIONS]>> 'URL'


DESCRIPTION
-----------

*{manname}* is a CLI downloader based on libcurl. +

By default, it splits a download into  fixed-sized chunks and download
them in-order with multiple concurrent connections.

Many <<OPTIONS,OPTIONS>> are available to change the default behavior.


Files and Dirs
---------------

'<filename>.part.sal'::
  The incomplete/partial file that will eventually be renamed to '<filename>'.

'<filename>.ctrl.sal'::
  This file contains the minimal information required about the download
  progress to allow resuming.

'<filename>.tmp.sal/'::
  By default, *{manname}* uses temporary files to store chunks before they are
  merged. Those files are created in this dir.


Chunk Progress States
----------------------

*4*::
  Chunk already merged in '<filename>.part.sal'

*3*::
  Chunk downloaded, but still not merged. +
  +
  This state should be transient. If it's not, your system probably has
  load problems (CPU/IO/memory).

*2*::
  Chunk download is in progress.

*1*::
  Chunk is queued for download. +
  +
  This is a transient state you shouldn't see.
  Unless you have serious load/congestion problems in your system.

*0*::
  Chunk still not queued for download.

[NOTE]
=============
When resuming from a previous session, an incomplete chunk would
initially have state '0'. That doesn't mean it will be emptied and
downloaded from scratch. Sub-chunk resume will work as expected.
=============


[[OPTIONS]]
OPTIONS
---------

[[gen-opts]]
General
~~~~~~~~

*-d, --dry-run*::
  Do not download, only display information.

*--mirror-url*::
  If the mirror is valid, Assign half the connections to it. +

  A mirror URL is considered valid when compared to URL if: :::
  +
  1- Both do not have the same effective URL. +
  2- Both have the same content size. +
  3- Both support ranges. +
  4- Both are not FTP. +
  5- Both are not using content encoding. Or both do, but we will not
     decode/decompress.

*--fatal-if-invalid-mirror*::
  Fatally fail if the mirror is invalid. Only a warning is displayed by default.

*--stdout*::
  Write/Pipe output to *stdout*. +
  +
  Everything else will behave in the same way except: :::
   +
   1- The pseudo output filename "STDOUT" will be used. +
   2- Chunks will only be merged (written to stdout) in order. +
   3- Resume is disabled. +
   4- No state is saved in a '<filename>.ctrl.sal' file. +
   5- If single mode is used, timeouts and other non-fatal response
      errors will become fatal.


 ::
  If *-m/--memory-buffers* are used, saldl will not create or write to
  any files directly.

*-m, --memory-buffers*::
  Use memory buffers instead of temp files for downloading chunks. +
  This mode increases memory overhead. And we lose sub-chunk resumability.
  But it might help performance in some situations, especially when IO is
  the bottleneck.

*--read-only*::
  Don't create files or write anything to disk.
  This should be only used to test network performance.

*--no-mmap* (default mode only)::
  Read chunk files into buffers instead of using mmap(). +
  This option hurts performance and should only be used for
  debugging.
<commit_msg>man: Formatting changes<commit_after>SALDL(1)
=========
:doctype: manpage

NAME
----

saldl - A lightweight well-featured CLI downloader optimized for speed and
early preview.


SYNOPSIS
--------

*{manname}* <<OPTIONS,[OPTIONS]>> 'URL'


DESCRIPTION
-----------

*{manname}* is a CLI downloader based on libcurl. +

By default, it splits a download into  fixed-sized chunks and download
them in-order with multiple concurrent connections.

Many <<OPTIONS,OPTIONS>> are available to change the default behavior.


Files and Dirs
---------------

'<filename>.part.sal'::
  The incomplete/partial file that will eventually be renamed to '<filename>'.

'<filename>.ctrl.sal'::
  This file contains the minimal information required about the download
  progress to allow resuming.

'<filename>.tmp.sal/'::
  By default, *{manname}* uses temporary files to store chunks before they are
  merged. Those files are created in this dir.


Chunk Progress States
----------------------

*4*::
  Chunk already merged in '<filename>.part.sal'

*3*::
  Chunk downloaded, but still not merged. +
  +
  This state should be transient. If it's not, your system probably has
  load problems (CPU/IO/memory).

*2*::
  Chunk download is in progress.

*1*::
  Chunk is queued for download. +
  +
  This is a transient state you shouldn't see.
  Unless you have serious load/congestion problems in your system.

*0*::
  Chunk still not queued for download.

[NOTE]
=============
When resuming from a previous session, an incomplete chunk would
initially have state '0'. That doesn't mean it will be emptied and
downloaded from scratch. Sub-chunk resume will work as expected.
=============


[[OPTIONS]]
OPTIONS
---------

[[gen-opts]]
General
~~~~~~~~

*-d, --dry-run*::
  Do not download, only display information.

*--mirror-url*::
  If the mirror is valid, Assign half the connections to it.


  A mirror URL is considered valid when compared to URL if: :::

   1. Both do not have the same effective URL.
   2. Both have the same content size.
   3. Both support ranges.
   4. Both are not FTP.
   5. Both are not using content encoding. Or both do, but we will not
      decode/decompress.

*--fatal-if-invalid-mirror*::
  Fatally fail if the mirror is invalid. Only a warning is displayed by default.

*--stdout*::
  Write/Pipe output to *stdout*.

  Everything else will behave in the same way except: :::

   1. The pseudo output filename "STDOUT" will be used.
   2. Chunks will only be merged (written to stdout) in order.
   3. Resume is disabled.
   4. No state is saved in a '<filename>.ctrl.sal' file.
   5. If single mode is used, timeouts and other non-fatal response
      errors will become fatal.

  ::
  If *-m/--memory-buffers* are used, saldl will not create or write to
  any files directly.

*-m, --memory-buffers*::
  Use memory buffers instead of temp files for downloading chunks. +
  This mode increases memory overhead. And we lose sub-chunk resumability.
  But it might help performance in some situations, especially when IO is
  the bottleneck.

*--read-only*::
  Don't create files or write anything to disk.
  This should be only used to test network performance.

*--no-mmap* (default mode only)::
  Read chunk files into buffers instead of using mmap(). +
  This option hurts performance and should only be used for
  debugging.

{'fraction_non_alphanumeric': 0.11878824056111029, 'fraction_numerical': 0.005073869571705716, 'mean_word_length': 3.175700934579439, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 24, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3159635', 'n_tokens_mistral': 2010, 'n_tokens_neox': 1912, 'n_words': 898}