<commit_before>﻿# ScholarSphere

ScholarSphere is a Ruby on Rails application utilizing the Blacklight
and Hydra-head gems for integration with the search & indexing system,
Solr, and the digital asset management platform, Fedora Commons.  The
application runs on Fedora 3.5.0 and Solr 3.5.0.

ScholarSphere is being developed as part of
[Penn State's Digital Stewardship Program](http://stewardship.psu.edu/).
Development on ScholarSphere began as part of the prototype
[CAPS project](http://stewardship.psu.edu/2011/02/caps-a-curation-platform-prototype.html). Code
and documentation are freely available via [Github](http://github.com/psu-stewardship/scholarsphere).

For more information, read the [ScholarSphere development docs](https://github.com/psu-stewardship/scholarsphere/wiki).

## Install

Infrastructural components

 * Fedora (if you don't have access to an instance, use the built-in
   hydra-jetty submodule)
 * Solr (if you don't have access to an instance, use the built-in
   hydra-jetty submodule)
 * A relational database (SQLite and MySQL have been tested)
 * Redis (for processing background jobs)

Install system dependencies

 * libmysqlclient-dev (if running MySQL as RDBMS)
 * libsqlite3-dev (if running SQLite as RDBMS)
 * libmagick-dev
 * libmagickwand-dev
 * [FITS](http://code.google.com/p/fits/) -- put it in a
  directory on your PATH

Install gems

    bundle install

Copy and **edit** database, LDAP, Fedora, and Solr configs

    cp config/database.yml.sample config/database.yml
    cp config/fedora.yml.sample config/fedora.yml
    cp config/solr.yml.sample config/solr.yml
    cp config/hydra-ldap.yml.sample config/hydra-ldap.yml

(Re-)Generate the app's secret token

    rake scholarsphere:generate_secret

Create and migrate database

    rake db:create
    rake db:migrate

If you'll be *developing* ScholarSphere, setup test databases and load
fixture data

    RAILS_ENV=test rake db:create
    RAILS_ENV=test rake db:migrate
    RAILS_ENV=test rake scholarsphere:fixtures:generate
    RAILS_ENV=test rake scholarsphere:fixtures:refresh

To use the built-in Fedora and Solr instances, get the bundled hydra-jetty,
configure it, & fire it up

    git submodule init
    git submodule update
    rake jetty:config
    rake jetty:start

Start the delayed job workers (needed for characterization and audit services)

    script/delayed_job start

Run the app server

    rails server

Browse to http://localhost:3000/ and you should see ScholarSphere!

## Usage Notes

### Auditing All Datastreams

To audit the digital signatures of every version of every object in the
repository, run the following command

    script/audit_repository

You'll probably want to schedule this regularly (e.g., via cron) in production environments.
Note that this does not *force* an audit -- it respects the value of max_days_between_audits
in application.rb.  Also note that if you want to run this on any environment other than 
development, you will need to call the script with RAILS_ENV=environment in front.

### Characterize All Uncharacterized Datastreams

In the event that some objects have not undergone characterization (for whatever reason), 
there is a rake task that sweeps through the entire repository looking for objects that lack
a characterization datastream.  For each object that lacks this datastream, a CharacterizationJob
that will characterize and thumbnailize the object is queued up.<commit_msg>replaces reference to DJ in README with a reference to resque-pool<commit_after>﻿# ScholarSphere

ScholarSphere is a Ruby on Rails application utilizing the Blacklight
and Hydra-head gems for integration with the search & indexing system,
Solr, and the digital asset management platform, Fedora Commons.  The
application runs on Fedora 3.5.0 and Solr 3.5.0.

ScholarSphere is being developed as part of
[Penn State's Digital Stewardship Program](http://stewardship.psu.edu/).
Development on ScholarSphere began as part of the prototype
[CAPS project](http://stewardship.psu.edu/2011/02/caps-a-curation-platform-prototype.html). Code
and documentation are freely available via [Github](http://github.com/psu-stewardship/scholarsphere).

For more information, read the [ScholarSphere development docs](https://github.com/psu-stewardship/scholarsphere/wiki).

## Install

Infrastructural components

 * Fedora (if you don't have access to an instance, use the built-in
   hydra-jetty submodule)
 * Solr (if you don't have access to an instance, use the built-in
   hydra-jetty submodule)
 * A relational database (SQLite and MySQL have been tested)
 * Redis (for processing background jobs)

Install system dependencies

 * libmysqlclient-dev (if running MySQL as RDBMS)
 * libsqlite3-dev (if running SQLite as RDBMS)
 * libmagick-dev
 * libmagickwand-dev
 * [FITS](http://code.google.com/p/fits/) -- put it in a
  directory on your PATH

Install gems

    bundle install

Copy and **edit** database, LDAP, Fedora, and Solr configs

    cp config/database.yml.sample config/database.yml
    cp config/fedora.yml.sample config/fedora.yml
    cp config/solr.yml.sample config/solr.yml
    cp config/hydra-ldap.yml.sample config/hydra-ldap.yml

(Re-)Generate the app's secret token

    rake scholarsphere:generate_secret

Create and migrate database

    rake db:create
    rake db:migrate

If you'll be *developing* ScholarSphere, setup test databases and load
fixture data

    RAILS_ENV=test rake db:create
    RAILS_ENV=test rake db:migrate
    RAILS_ENV=test rake scholarsphere:fixtures:generate
    RAILS_ENV=test rake scholarsphere:fixtures:refresh

To use the built-in Fedora and Solr instances, get the bundled hydra-jetty,
configure it, & fire it up

    git submodule init
    git submodule update
    rake jetty:config
    rake jetty:start

Start the resque-pool workers (needed for characterization, audit, and
unzip services)

    resque-pool --daemon --environment development start

Run the app server

    rails server

Browse to http://localhost:3000/ and you should see ScholarSphere!

## Usage Notes

### Auditing All Datastreams

To audit the digital signatures of every version of every object in the
repository, run the following command

    script/audit_repository

You'll probably want to schedule this regularly (e.g., via cron) in production environments.
Note that this does not *force* an audit -- it respects the value of max_days_between_audits
in application.rb.  Also note that if you want to run this on any environment other than 
development, you will need to call the script with RAILS_ENV=environment in front.

### Characterize All Uncharacterized Datastreams

In the event that some objects have not undergone characterization (for whatever reason), 
there is a rake task that sweeps through the entire repository looking for objects that lack
a characterization datastream.  For each object that lacks this datastream, a CharacterizationJob
that will characterize and thumbnailize the object is queued up.
{'fraction_non_alphanumeric': 0.06404574696211579, 'fraction_numerical': 0.004860614724803431, 'mean_word_length': 4.3323170731707314, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3034426', 'n_tokens_mistral': 2119, 'n_tokens_neox': 1977, 'n_words': 816}