<jupyter_start><jupyter_text>The two core components of the conda ecosystem are the package cache and the environment subfolders.  These are abstracted with `PackageInfo` and `Environment` objects respectively.Here we create "pools" of `PackageInfo` and `Environment` objects.  These objects permit easy, read-only access to various bits of metadata stored in the package cache and conda-meta/ subfolders in the environment.  We want to reuse the objects as much as we can to minimize disk I/O.  All the disk reads are currently cached with the objects, so the more objects you work with, the more RAM will be required.<jupyter_code># Create pkg_cache and environments
pkg_cache = cache.packages(root_pkgs)
envs = environment.environments(root_envs)
print(pkg_cache[:5])
print()
print(envs[:5])<jupyter_output>(PackageInfo(C:\Users\Ryan\Miniconda3\pkgs\alabaster-0.7.8-py35_0) @ 0x13dfd352828, PackageInfo(C:\Users\Ryan\Miniconda3\pkgs\anaconda-client-1.4.0-py35_0) @ 0x13dfd3527b8, PackageInfo(C:\Users\Ryan\Miniconda3\pkgs\babel-2.3.3-py35_0) @ 0x13dfd352860, PackageInfo(C:\Users\Ryan\Miniconda3\pkgs\backports-1.0-py27_0) @ 0x13dfd3526d8, PackageInfo(C:\Users\Ryan\Miniconda3\pkgs\backports-1.0-py35_0) @ 0x13dfd3527f0)

(Environment(C:\Users\Ryan\Miniconda3) @ 0x13dfd39aeb8, Environment(C:\Users\Ryan\Miniconda3\envs\adamimage) @ 0x13dfd39afd0, Environment(C:\Users\Ryan\Miniconda3\envs\ct_test1) @ 0x13dfd389080, Environment(C:\Users\Ryan\Miniconda3\envs\ct_test2) @ 0x13dfd389128, Environment(C:\Users\Ryan\Miniconda3\envs\ct_test3) @ 0x13dfd389198)<jupyter_text>PackagesConda packages all have an info/ subdirectory for storing metadata about the package.  PackageInfo provide convenient access to this metadata.<jupyter_code>pi = pkg_cache[0]
pi.index  # info/index.json
# We can access fields of index.json directly from the object.
pi.name, pi.version, pi.build
# Access to info/files
pi.files
# The full spec of the package.  This is always "name-version-build"
pi.full_spec
# We can queries against the information we have on packages
# For example, I want to find all MIT licensed packages in the cache
{pi.full_spec: pi.license for pi in pkg_cache if pi.license == 'MIT'}<jupyter_output><empty_output><jupyter_text>Environments<jupyter_code>e = envs[2]
e
# We can discover the currently activated environment
{e.path: e.activated() for e in envs}
# We can see all the packages that claim to be linked into the environment, keyed by name
e.linked_packages
# linked packages are either hard-linked, symlinked, or copied into environments.
set(chain(e.hard_linked, e.soft_linked, e.copy_linked)) ^ set(e.linked_packages.values())
# The origin channel of each package
e.package_channels
# We also have access to the history of the environment.
# The history object is an adaptation of conda's history parser.
# (note: The interface to this may change in the future)
e.history.object_log<jupyter_output><empty_output><jupyter_text>Neat stuffConvenient access to the package cache and environment metadata allows you to do some neat stuff relatively easily.Below are a few examples of some quick ideas that can be implemented with little effort.<jupyter_code># Calculate potential collisions in environments by packages claiming the same file paths
# Very quick and naive way of detecting file path collisions.
for i, p1 in enumerate(pkg_cache):
    for p2 in pkg_cache[i+1:]:
        if p1.name == p2.name:
            continue
        x = p1.files.intersection(p2.files)
        if x:
            print("{} collides with {}".format(p1, p2))
            print("\tCollisions: ", x)
# Cache Utils has some higher order, convenience functions

# See what environments a package is linked into
# Note that this is a O(n) operation where n is the sum of the installed packages in each environment you're checking.
# If you're running this for the first time, it has to read all the metadata for each environment.
# Also note, that this creates new package info objects and environment objects each run, so each run
# prompts a full scan of both the package cache and all environments.
cu.linked_environments((pkg_cache[0],), envs)
# Find which environments the latest packages are linked to.
# This example uses Versio to parse and compare PEP440 compliant version numbers
# This will exclude packages like packages like jpeg and openssl

# This loop simple creates Version objects so we can compare them later.
Versions = {}
for x in pkg_cache:
    try:
        if x.name in Versions:
            Versions[x.name].append(Version(x.version))
        else:
            Versions[x.name] = [Version(x.version)]
    except:
        print("Skipping ", x.name, x.version)
    
# sort the value lists and pick the latest versions
#pversions = {k: str(list(sorted(v))[-1]) for k, v in Versions.items()}

# sort the value lists and pick the older versions
pversions = {k: list(map(str, list(sorted(v))[:-1])) for k, v in Versions.items()}

# The most up-to-date packages are linked to which environments?
#latest_pkgs = [x for x in pkg_cache if x.name in pversions and x.version == pversions[x.name]]

# Find the environments that older packages are linked to
latest_pkgs = [x for x in pkg_cache if x.name in pversions and x.version in set(pversions[x.name])]

# Simply print the results nicely
{str(k): list(map(str, v)) for k, v in cu.linked_environments(latest_pkgs, envs).items()}
# All packages that are not linked to any environment
cu.unlinked_packages(pkg_cache, envs)
# Environment representation of root environment
e = environment.Environment(join(root_envs, 'env2'))
# Long running.  Disk intensive.
filter_pyc = lambda f: filter(lambda x: not x.endswith('.pyc'), f)
# List all files in an environment that are not hardlinked (and should be).
# Note that *.pyc files are filtered out.
not_linked = {x: tuple(filter_pyc(y)) for x, y in eu.check_hardlinked_env(envs[0]).items()}

# If you wish to see all the non-existant hardlinks, including *.pyc files, remove the filter_pyc function call
# not_linked = {x: y for x, y in eu.check_hardlinked_env(envs[0]).items()}

not_linked
# We can leverage the information in the environment's history to get packages 
# that were explicitly installed by the user.
eu.explicitly_installed(e)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08998085513720484, 'fraction_numerical': 0.02297383535417996, 'mean_word_length': 4.714676390154968, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '52201', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2016, 'n_tokens_neox': 1927, 'n_words': 715, 'path': 'Conda-tools.ipynb', 'repo_name': 'groutr/conda-tools', 'size': '6268'}