<commit_before>MyArchive
---------

MyArchive is a tool for interacting with a multitude of social networks and sites online via established APIs. It downloads and archives posts, images, favorites, etc, allowing you to maintain a central repository of your online activity on your own machine.

Tricks
++++++

* DA collections are handled as just another set of tags! In this case, we make four tags to cover all possible bases.

  * collection_name
  * da.user.(username).(favorite|gallery)
  * da.user.(username).(favorite|gallery).(collection_name)
  * da.author.(author_username)

Twitter
=======

* The Twitter API does not permit downloading of GIFs/videos. (No path returned in the API call.) We can only pull thumbnails.

* Tweets (and their files) are tagged in the following manner.
<commit_msg>A small readme update.<commit_after>MyArchive
---------

MyArchive is a tool for interacting with a multitude of social networks and sites online via established APIs. It downloads and archives posts, images, favorites, etc, allowing you to maintain a central repository of your online activity on your own machine.

Supported Data Import Sources
+++++++++++++++++++++++++++++

* Shotwell databases/files
* Raw folders and files on your filesystem
* Deviantart
* Twitter
* Planned, or in work.
  * Dreamwidth
  * LiveJournal
  * Github Gist (and probably repo mirroring)
  * Goodreads
  * LibraryThing
  * Mastodon
  * Medium
  * Reddit
  * SoFurry
  * Weasyl
  * Youtube playlists
  * Generic RSS feeds


Deviantart
==========

* DA collections are handled as just another set of tags! In this case, we make four tags to cover all possible bases.

  * collection_name
  * da.user.(username).(favorite|gallery)
  * da.user.(username).(favorite|gallery).(collection_name)
  * da.author.(author_username)

Twitter
=======

* The Twitter API does not permit downloading of GIFs/videos. (No path returned in the API call.) We can only pull thumbnails.

* Tweets (and their files) are tagged in the following manner.

{'fraction_non_alphanumeric': 0.10009910802775025, 'fraction_numerical': 0.0, 'mean_word_length': 4.111392405063291, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1810824', 'n_tokens_mistral': 589, 'n_tokens_neox': 539, 'n_words': 244}