<commit_before># HighLevelBrowse

Given an LC Call Number, try to get a set of academic disciplines associated with it

## Usage

```ruby

use 'high_level_browse'

# Pull a new version of the raw data from the UM website,
# transform it into something that can be quickly searched,
# and serialize it to `hlb.json.gz` in the specified directory
hlb = HighLevelBrowse.fetch_and_save(dir: '/tmp')

# ...or just grab an already fetch_and_saved copy
hlb = HighLevelBrowse.load(dir: '/tmp')

# What HLB categories is an LC Call Number in?
hlb.topics 'hc 9112.2'
# => [["Social Sciences", "Economics"],
#     ["Social Sciences", "Social Sciences (General)"]]

# ... or use the #[] shortcut syntax

hlb['NC1766 .U52 D733 2014']
# => [["Arts", "Art History"],
#    ["Arts", "Art and Design"],
#    ["Arts", "Film and Video Studies"]]

# You can also send more than one call number at a time

hlb.topics('E 99 .S2 Y67 1993', 'PS 3565 .R5734 F67 2015')
# => [["Humanities", "American Culture"],
#     ["Humanities", "United States History"],
#     ["Social Sciences", "Native American Studies"],
#     ["Social Sciences", "Archaeology"],
#     ["Humanities", "English Language and Literature"]]

```


## Overview

The University of Michigan Library has for years maintained 
the [High Level Browse](https://www.lib.umich.edu/browse/categories/),
a mapping of call-number ranges to academic subjects. The entire 
data set is available as [1.8MB XML file](https://www.lib.umich.edu/browse/categories/xml.php)
for download.

This gem gives a relatively time-efficient way to get the set of disciplines associated
with the given callnumber or callnumbers as part of indexing MARC records into Solr. 
This mapping is used in many places in the University Library at the University of 
Michigan, including the 
[Mirlyn Catalog](https://mirlyn.lib.umich.edu/)
(exposed as "Academic Discipline" in the facets) and ejournals/databases (and even 
Librarians!) via the [Browse page](https://www.lib.umich.edu/browse). 
 
This categorization may be useful for clustering/faceting
in similar applications at other institutions. Note that the actual creation and 
maintenance of the call number ranges is done by subject specialist librarians and 
is out of scope for this gem.

## Working from the command line

There are also a couple command line applications for managing and querying the
data.

* **fetch_new_hlb** tries to grab a new copy of the data from the umich website
  and serialize it to the given directory. Useful for putting in a cron job
  to periodically update with fresh data
  
```bash

$> fetch_new_hlb

fetch_new_hlb -- get a new copy of the HLB ready for use by high_level_browse
and stick it in the given directory

   Usage: fetch_new_hlb <dir>
```

* **hlb** takes one or more callnumbers and returns a text display of the categories
  associated with them. It will stash a copy of the database in `Dir.tmpdir` (e.g., 
  `/tmp/`) if there isn't one there already, and use it on subsequent calls so things
  aren't so desperately slow.


```bash
$> hlb

hlb -- get high level browse data for an LC call number

Example:
   hlb "qa 11.33 .C4 .H3"
    or do several at once
   hlb "PN 33.4" "AC 1122.3 .C22" ...
 
# Let's try it
$> hlb "qa 11.33 .C4 .H3"
   
   Science | Mathematics
   Social Sciences | Education   
```


## A warning about (lack of) coverage

Note that not every possible valid callnumber will be necessarily be contained in any 
dicipline at all. Coverage is known to have some holes, and the ranges themselves 
sometimes cover essentially a single book in the umich collection.

Hence, this may or may not be useful at your insitution. You'll have to experiment.

## Performance
<commit_msg>Update README about tmpdir location<commit_after># HighLevelBrowse

Given an LC Call Number, try to get a set of academic disciplines associated with it

## Usage

```ruby

use 'high_level_browse'

# Pull a new version of the raw data from the UM website,
# transform it into something that can be quickly searched,
# and serialize it to `hlb.json.gz` in the specified directory
hlb = HighLevelBrowse.fetch_and_save(dir: '/tmp')

# ...or just grab an already fetch_and_saved copy
hlb = HighLevelBrowse.load(dir: '/tmp')

# What HLB categories is an LC Call Number in?
hlb.topics 'hc 9112.2'
# => [["Social Sciences", "Economics"],
#     ["Social Sciences", "Social Sciences (General)"]]

# ... or use the #[] shortcut syntax

hlb['NC1766 .U52 D733 2014']
# => [["Arts", "Art History"],
#    ["Arts", "Art and Design"],
#    ["Arts", "Film and Video Studies"]]

# You can also send more than one call number at a time

hlb.topics('E 99 .S2 Y67 1993', 'PS 3565 .R5734 F67 2015')
# => [["Humanities", "American Culture"],
#     ["Humanities", "United States History"],
#     ["Social Sciences", "Native American Studies"],
#     ["Social Sciences", "Archaeology"],
#     ["Humanities", "English Language and Literature"]]

```


## Overview

The University of Michigan Library has for years maintained 
the [High Level Browse](https://www.lib.umich.edu/browse/categories/),
a mapping of call-number ranges to academic subjects. The entire 
data set is available as [1.8MB XML file](https://www.lib.umich.edu/browse/categories/xml.php)
for download.

This gem gives a relatively time-efficient way to get the set of disciplines associated
with the given callnumber or callnumbers as part of indexing MARC records into Solr. 
This mapping is used in many places in the University Library at the University of 
Michigan, including the 
[Mirlyn Catalog](https://mirlyn.lib.umich.edu/)
(exposed as "Academic Discipline" in the facets) and ejournals/databases (and even 
Librarians!) via the [Browse page](https://www.lib.umich.edu/browse). 
 
This categorization may be useful for clustering/faceting
in similar applications at other institutions. Note that the actual creation and 
maintenance of the call number ranges is done by subject specialist librarians and 
is out of scope for this gem.

## Working from the command line

There are also a couple command line applications for managing and querying the
data.

* **fetch_new_hlb** tries to grab a new copy of the data from the umich website
  and serialize it to the given directory. Useful for putting in a cron job
  to periodically update with fresh data
  
```bash

$> fetch_new_hlb

fetch_new_hlb -- get a new copy of the HLB ready for use by high_level_browse
and stick it in the given directory

   Usage: fetch_new_hlb <dir>
```

* **hlb** takes one or more callnumbers and returns a text display of the categories
  associated with them. It will stash a copy of the database in `Dir.tmpdir` (e.g., 
  `/tmp/` on unix, or something weird under OSX) if there isn't one there already, 
  and use it on subsequent calls so things aren't so desperately slow.


```bash
$> hlb

hlb -- get high level browse data for an LC call number

Example:
   hlb "qa 11.33 .C4 .H3"
    or do several at once
   hlb "PN 33.4" "AC 1122.3 .C22" ...
 
# Let's try it
$> hlb "qa 11.33 .C4 .H3"
   
   Science | Mathematics
   Social Sciences | Education   
```


## A warning about (lack of) coverage

Note that not every possible valid callnumber will be necessarily be contained in any 
dicipline at all. Coverage is known to have some holes, and the ranges themselves 
sometimes cover essentially a single book in the umich collection.

Hence, this may or may not be useful at your insitution. You'll have to experiment.

## Performance

{'fraction_non_alphanumeric': 0.09064210385796288, 'fraction_numerical': 0.01735415832332132, 'mean_word_length': 3.9127868852459016, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 17, 'https://': 8, 'lorem ipsum': 0, 'www.': 6, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1387916', 'n_tokens_mistral': 2415, 'n_tokens_neox': 2189, 'n_words': 1037}