<commit_before>---------------------GENERAL INFORMATION-----------------------------
Name of publisher: British Institute of Publishing
Publishing platform?: Atypon 

RT:
JIRA: PD-44r
legacy RU:

Plugin name suggestion: BIRAtyponPlugin/ClockssBIRAtyponPlugin

Clockss? [yes/no]:		yes
GLN? [yes/no]:			yes
PLN? [and name thereof]:  n/a

TDB information:
    PARAMS - base_url
    PARAMS - volume, journal_id
    ATTRS:

Base url: http://www.birpublications.org/

Start URL(s): http://www.birpublications.org/(c)lockss/<jid>/<vol>/index.html 
    permission statement is text on the manifest page, eg:
    " CLOCKSS system has permission to ingest, preserve, and serve this Archival Unit."

a) British Journal of Radiology (bjr) Vol 86 (2013)
b) 
c) 

---------
This is an Atypon site and they have a very consistent layout.
This analysis will focus on the ways in which this site is unique among Atypon
   this is primarily about filtering and exclusion during form requests 

Crawl rules & Content Layout
Consistent Atypon layout, that is:

Journal Front Page (we won't collect, but need to know) : http://www.birpublications.org/loi/<jid>
Volume Table of Contents - no volume TOC other than manifest page/start url 
  (http://www.birpublications.org/clockss/<jid>/<vol>)
    the Journal Front page contains expandable items for each item but links only to specific issue TOCs
Issue Table of contents: http://www.birpublications.org/toc/<jid>/<vol>/<issue>

BIR on ATYPON file types & URL pattern:
   http://www.birpublications.org/doi/full/<doi1st>/<doi2nd> : full text html as available
   http://www.birpublications.org/doi/abs/<doi1st>/<doi2nd> : abstract 
   http://www.birpublications.org/doi/pdfplus/<doi1st>/<doi2nd> pdf version
   http://www.birpublications.org/doi/ref/<doi1st>/<doi2nd> references
   
   abstract, references & full text are each their own URL but they appear on the page as tabs on a single article landing page
   On the article page there are additiona tabs: figures, citedby with the corresponding URL tags:
      <base>/doi/figure/<doi1st>/<doi2nd>
      <base>/doi/citedby/<doi1st>/<doi2nd>
     
     References live in their own tab (really, a unique URL) but also have information for in-line popup as the viewer floats over the 
     reference link and a section at the bottom of the full text of the article.

Citation information (also note format options - may be RIS, Bibtex, endnote, etc)
   This is accessed from a link from any article landing page (abs/full/ref). 
   The link leads to a form for selecting type of citation. 
   We must process this and get all the options (ris, bibtex) except refworks. We will get metadata from the RIS citation version.  

    The link is in the right hand column "Download citation".
    www.birpublications.org/action/showCitFormats?doi=<doi1st>/<doi2nd>
    
Images, Figures & Tables
  Figures are available via a tab on the article page (a unique URL) that shows all the graphics associated
  with the article. Clicking on one opens the image viewer. Similarly the full text in-line images are small and clicking
  on them opens the image viewer. The actual image is at:
     http://www.birpublications.org/na101/home/literatum/publisher/bir/journals/content/img/2013/img.2013.22.issue-1/imaging.20100022/production/images/small/img-22-1-d10022-g002.gif
     
DOI pattern

a)10.1259/bjr.20130258 (British journal of radiology)
b)10.1259/imaging.20100022 (imaging)
c)10.1259/dmfr.20130129 (dmfr)



Does this site use automatically generated content?? 
  There is something that looks changeable, for css files:
    /pb/css/1402444492000/head_3_12_41.css

Notes
Anything else that you think the plugin writer needs to be aware of in terms of URL layout.

Unlike some other atypon the images for outside organizations are well-named:
"/pb/assets/raw/social_li.png"
"/pb/assets/raw/social_fb.png"
"/pb/assets/raw/social_tw.png"


- EXCLUDED:

    <base_url>/doi/citedBySection/<doi1st>/<doi2nd>
        BIR uses this new URL format for a citedBySection - we already crawl & hash filter anything
        in this section, but I was told not to collect it either.
        
     refworks format of citation downloads
 
 - INCLUDED
                   
URL NORMALIZATION:
	TBD
--------------------------URL NORMALIZATION---------------------------------------------

Atypon -
Normalize the arguments used in the citation download so that you can guess at the URL in the article iterator.
The BaseAtypon url normalizer can do this. 
It should become the format:
    http://www.bironline.com/action/downloadCitation?doi=<doi>&format=ris&include=cit
    where format can be ris, bibtex, medlars, endnote, refworks and refworks-cn
      we need to exclude refworks & refworks-cn... must look in to whether we want any other than bibtex and ris
      
normalize off the anchors (but this happens automatically) to avoid duplicated URLs

 Add to BaseAtypon url normalizer - found in BIR and in T&F
 * Some URL links for the abstract end up with:
 * (see http://www.birpublications.org/toc/bjr/85/1020)
 *     ...doi/abs/10.1259/bjr.20130630?queryID=${resultBean.queryID}
 *     which is html encoded as 
 *     ...doi/abs/10.1259/bjr.20130630?queryID=%24%7BresultBean.queryID%7D
---------------------HTML HASH FILTERING-----------------------------------------
        // handled by parent: script, sfxlink, stylesheet
        
        // toc - first top block ad
        // http://www.birpublications.org/toc/bjr/87/1044
        HtmlNodeFilters.tagWithAttributeRegex("div", "class", "literatumAd"),
        // page header: login, register, etc., and journal menu such as
        // subscribe, alerts, ...
        HtmlNodeFilters.tagWithAttributeRegex("div", "id", "pageHeader"),
        // page footer
        HtmlNodeFilters.tagWithAttributeRegex("div", "id", "pageFooter"),
        // toc - BJR logo image right below pageHeader
        HtmlNodeFilters.tagWithAttributeRegex("div", "class", 
                                              "^widget general-image"),
        // toc, abs, full, ref - menu above breadcrumbs
        HtmlNodeFilters.tagWithAttributeRegex("div",  "class", "menuXml"),
        // toc - free.gif image tied to an abs
        HtmlNodeFilters.tagWithAttributeRegex("img",  "src", "free.gif"),   
        // toc - access icon container
        HtmlNodeFilters.tagWithAttribute("td", "class", "accessIconContainer"),
        // toc - pulldown with sections - may add citedby later
        HtmlNodeFilters.tagWithAttribute("div", "class", 
                                         "publicationTooldropdownContainer"), 
        // toc - right column, current issue
        HtmlNodeFilters.tagWithAttributeRegex("div", "class", 
                                              "literatumBookIssueNavigation"),
        // toc, abs - share social media
        HtmlNodeFilters.tagWithAttributeRegex("div", "class",
                                              "general-bookmark-share"),
        // toc - right column impact factor block - no unique name found
        HtmlNodeFilters.tagWithAttributeRegex("div", "class", 
            "widget\\s+layout-one-column\\s+none\\s+widget-regular\\s+widget-border-toggle"),
        // ref - this seems unused but may get turned on
        // http://www.birpublications.org/doi/ref/10.1259/bjr.20130571
        HtmlNodeFilters.tagWithAttribute("div",  "id", "MathJax_Message"),
        // abs - right column all literatumArticleToolsWidget 
        // except Download Citation
        // http://www.birpublications.org/doi/abs/10.1259/bjr.20140472                                      
        HtmlNodeFilters.allExceptSubtree(
            HtmlNodeFilters.tagWithAttributeRegex( 
                "div", "class", "literatumArticleToolsWidget"),
                HtmlNodeFilters.tagWithAttributeRegex(
                    "a", "href", "/action/showCitFormats\\?"))   
                    
-------------------HTML CRAWL FILTERING----------------------------------------
      // handled by parent BaseAtypon:
      // prev/next issues
      // <td class="journalNavLeftTd">
      // <td class="journalNavRightTd">
      
      // toc, abs, full, ref - menu above breadcrumbs, Paper in Press links
      // to http://www.birpublications.org/toc/bjr/0/0
      // http://www.birpublications.org/toc/bjr/87/1038
      HtmlNodeFilters.tagWithAttributeRegex("div",  "class", "menuXml"),
      
      // toc - erratum section linking to Original Article
      // http://www.birpublications.org/toc/dmfr/42/8
      HtmlNodeFilters.tagWithAttribute("div", "class", "relatedLayer"),
      
      // errata page linking to Original Article
      // http://www.birpublications.org/doi/abs/10.1259/dmfr/24435203
      HtmlNodeFilters.tagWithAttribute("div", "id", "relatedContent"),
-----------------------PDF FILTER--------------------------------------------

Looks okay - no filtering needed
* pdfplus has blue triangle - probably need scrapping filter - Ly 23jan2015<commit_msg>added hash filter for Cited by<commit_after>---------------------GENERAL INFORMATION-----------------------------
Name of publisher: British Institute of Publishing
Publishing platform?: Atypon 

RT:
JIRA: PD-44r
legacy RU:

Plugin name suggestion: BIRAtyponPlugin/ClockssBIRAtyponPlugin

Clockss? [yes/no]:		yes
GLN? [yes/no]:			yes
PLN? [and name thereof]:  n/a

TDB information:
    PARAMS - base_url
    PARAMS - volume, journal_id
    ATTRS:

Base url: http://www.birpublications.org/

Start URL(s): http://www.birpublications.org/(c)lockss/<jid>/<vol>/index.html 
    permission statement is text on the manifest page, eg:
    " CLOCKSS system has permission to ingest, preserve, and serve this Archival Unit."

a) British Journal of Radiology (bjr) Vol 86 (2013)
b) 
c) 

---------
This is an Atypon site and they have a very consistent layout.
This analysis will focus on the ways in which this site is unique among Atypon
   this is primarily about filtering and exclusion during form requests 

Crawl rules & Content Layout
Consistent Atypon layout, that is:

Journal Front Page (we won't collect, but need to know) : http://www.birpublications.org/loi/<jid>
Volume Table of Contents - no volume TOC other than manifest page/start url 
  (http://www.birpublications.org/clockss/<jid>/<vol>)
    the Journal Front page contains expandable items for each item but links only to specific issue TOCs
Issue Table of contents: http://www.birpublications.org/toc/<jid>/<vol>/<issue>

BIR on ATYPON file types & URL pattern:
   http://www.birpublications.org/doi/full/<doi1st>/<doi2nd> : full text html as available
   http://www.birpublications.org/doi/abs/<doi1st>/<doi2nd> : abstract 
   http://www.birpublications.org/doi/pdfplus/<doi1st>/<doi2nd> pdf version
   http://www.birpublications.org/doi/ref/<doi1st>/<doi2nd> references
   
   abstract, references & full text are each their own URL but they appear on the page as tabs on a single article landing page
   On the article page there are additiona tabs: figures, citedby with the corresponding URL tags:
      <base>/doi/figure/<doi1st>/<doi2nd>
      <base>/doi/citedby/<doi1st>/<doi2nd>
     
     References live in their own tab (really, a unique URL) but also have information for in-line popup as the viewer floats over the 
     reference link and a section at the bottom of the full text of the article.

Citation information (also note format options - may be RIS, Bibtex, endnote, etc)
   This is accessed from a link from any article landing page (abs/full/ref). 
   The link leads to a form for selecting type of citation. 
   We must process this and get all the options (ris, bibtex) except refworks. We will get metadata from the RIS citation version.  

    The link is in the right hand column "Download citation".
    www.birpublications.org/action/showCitFormats?doi=<doi1st>/<doi2nd>
    
Images, Figures & Tables
  Figures are available via a tab on the article page (a unique URL) that shows all the graphics associated
  with the article. Clicking on one opens the image viewer. Similarly the full text in-line images are small and clicking
  on them opens the image viewer. The actual image is at:
     http://www.birpublications.org/na101/home/literatum/publisher/bir/journals/content/img/2013/img.2013.22.issue-1/imaging.20100022/production/images/small/img-22-1-d10022-g002.gif
     
DOI pattern

a)10.1259/bjr.20130258 (British journal of radiology)
b)10.1259/imaging.20100022 (imaging)
c)10.1259/dmfr.20130129 (dmfr)



Does this site use automatically generated content?? 
  There is something that looks changeable, for css files:
    /pb/css/1402444492000/head_3_12_41.css

Notes
Anything else that you think the plugin writer needs to be aware of in terms of URL layout.

Unlike some other atypon the images for outside organizations are well-named:
"/pb/assets/raw/social_li.png"
"/pb/assets/raw/social_fb.png"
"/pb/assets/raw/social_tw.png"


- EXCLUDED:

    <base_url>/doi/citedBySection/<doi1st>/<doi2nd>
        BIR uses this new URL format for a citedBySection - we already crawl & hash filter anything
        in this section, but I was told not to collect it either.
        
     refworks format of citation downloads
 
 - INCLUDED
                   
URL NORMALIZATION:
	TBD
--------------------------URL NORMALIZATION---------------------------------------------

Atypon -
Normalize the arguments used in the citation download so that you can guess at the URL in the article iterator.
The BaseAtypon url normalizer can do this. 
It should become the format:
    http://www.bironline.com/action/downloadCitation?doi=<doi>&format=ris&include=cit
    where format can be ris, bibtex, medlars, endnote, refworks and refworks-cn
      we need to exclude refworks & refworks-cn... must look in to whether we want any other than bibtex and ris
      
normalize off the anchors (but this happens automatically) to avoid duplicated URLs

 Add to BaseAtypon url normalizer - found in BIR and in T&F
 * Some URL links for the abstract end up with:
 * (see http://www.birpublications.org/toc/bjr/85/1020)
 *     ...doi/abs/10.1259/bjr.20130630?queryID=${resultBean.queryID}
 *     which is html encoded as 
 *     ...doi/abs/10.1259/bjr.20130630?queryID=%24%7BresultBean.queryID%7D
---------------------HTML HASH FILTERING-----------------------------------------
        // handled by parent: script, sfxlink, stylesheet
        
        // toc - first top block ad
        // http://www.birpublications.org/toc/bjr/87/1044
        HtmlNodeFilters.tagWithAttributeRegex("div", "class", "literatumAd"),
        // page header: login, register, etc., and journal menu such as
        // subscribe, alerts, ...
        HtmlNodeFilters.tagWithAttributeRegex("div", "id", "pageHeader"),
        // page footer
        HtmlNodeFilters.tagWithAttributeRegex("div", "id", "pageFooter"),
        // toc - BJR logo image right below pageHeader
        HtmlNodeFilters.tagWithAttributeRegex("div", "class", 
                                              "^widget general-image"),
        // toc, abs, full, ref - menu above breadcrumbs
        HtmlNodeFilters.tagWithAttributeRegex("div",  "class", "menuXml"),
        // toc - free.gif image tied to an abs
        HtmlNodeFilters.tagWithAttributeRegex("img",  "src", "free.gif"),   
        // toc - access icon container
        HtmlNodeFilters.tagWithAttribute("td", "class", "accessIconContainer"),
        // toc - pulldown with sections - may add citedby later
        HtmlNodeFilters.tagWithAttribute("div", "class", 
                                         "publicationTooldropdownContainer"), 
        // toc - right column, current issue
        HtmlNodeFilters.tagWithAttributeRegex("div", "class", 
                                              "literatumBookIssueNavigation"),
        // toc, abs - share social media
        HtmlNodeFilters.tagWithAttributeRegex("div", "class",
                                              "general-bookmark-share"),
        // toc - right column impact factor block - no unique name found
        HtmlNodeFilters.tagWithAttributeRegex("div", "class", 
            "widget\\s+layout-one-column\\s+none\\s+widget-regular\\s+widget-border-toggle"),
        // ref - this seems unused but may get turned on
        // http://www.birpublications.org/doi/ref/10.1259/bjr.20130571
        HtmlNodeFilters.tagWithAttribute("div",  "id", "MathJax_Message"),
        // abs - right column all literatumArticleToolsWidget 
        // except Download Citation
        // http://www.birpublications.org/doi/abs/10.1259/bjr.20140472                                      
        HtmlNodeFilters.allExceptSubtree(
            HtmlNodeFilters.tagWithAttributeRegex( 
                "div", "class", "literatumArticleToolsWidget"),
                HtmlNodeFilters.tagWithAttributeRegex(
                    "a", "href", "/action/showCitFormats\\?"))   
        // abs, full, ref, figure - remove Cited by include parent tag <li>
-------------------HTML CRAWL FILTERING----------------------------------------
      // handled by parent BaseAtypon:
      // prev/next issues
      // <td class="journalNavLeftTd">
      // <td class="journalNavRightTd">
      
      // toc, abs, full, ref - menu above breadcrumbs, Paper in Press links
      // to http://www.birpublications.org/toc/bjr/0/0
      // http://www.birpublications.org/toc/bjr/87/1038
      HtmlNodeFilters.tagWithAttributeRegex("div",  "class", "menuXml"),
      
      // toc - erratum section linking to Original Article
      // http://www.birpublications.org/toc/dmfr/42/8
      HtmlNodeFilters.tagWithAttribute("div", "class", "relatedLayer"),
      
      // errata page linking to Original Article
      // http://www.birpublications.org/doi/abs/10.1259/dmfr/24435203
      HtmlNodeFilters.tagWithAttribute("div", "id", "relatedContent"),
-----------------------PDF FILTER--------------------------------------------

Looks okay - no filtering needed
* pdfplus has blue triangle - probably need scrapping filter - Ly 23jan2015
{'fraction_non_alphanumeric': 0.1287643932683791, 'fraction_numerical': 0.024357838795394153, 'mean_word_length': 2.511175898931001, 'pattern_counts': {'":': 0, '<': 64, '<?xml version=': 0, '>': 64, 'https://': 0, 'lorem ipsum': 0, 'www.': 40, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 28, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6439194', 'n_tokens_mistral': 5641, 'n_tokens_neox': 5101, 'n_words': 1637}