<commit_before><!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
  <head>
    <title>xia2: Automated Data Reduction</title>
  </head>

  <body>
    <h1><img src="xia2-new-logo.gif" alt="xia2"></img>
    Automated Data Reduction</h1>

    <p>New! I have now set up a mailing list for xia2 on sourceforge.
      If you would like to subscribe to this - where announcements of new
      versions of xia2 and problems can be discussed, please go to
      <a href=https://lists.sourceforge.net/lists/listinfo/xia2-list>the list
      page</a>. Also, if you are interested in keeping up-to-date with what
      is going on in xia2 development take a look at the
      <a href="http://xia2.blogspot.com">blog</a>.</p>

    <p>xia2 is a new automated data reduction system designed to work from raw
      diffraction data and a little metadata, and produce usefully reduced
      data in a form suitable for immediately starting phasing and structure
      solution, e.g. through Mr BUMP or your favourite experimental phasing
      suite.</p>

    <p>The following steps are performed (although not strictly in
      this order)</p>

    <ul>
      <li>Integration with feedback and complex lattice selection stuff</li>
      <li>Reindexing to standard setting for spacegroup (e.g. P 2 21 21 to
        P 21 21 2)</li>
      <li>Scaling together of all data to maximise anomalous signal</li>
      <li>Estimation of number of molecules in ASU and solvent fraction
        based on sequence information (if provided)</li>
    </ul>

    <p>There is a lot of fun stuff which goes on in there - run it to find
      out more...</p>

    <p>The system is designed to cope with MAD data and data measured in
      multiple passes, or a combination of the two. The input is a therefore
      a little complex, but consists only of information you should have
      to hand anyway - here is a simple example, a more complex example
      can be found below.</p>

    <h1>Download</h1>

    <p>The current version, 0.3.5.1 can be obtained by following these
      links:</p>

    <ul>
      <li><a href="xia2-0.3.5.1.zip">Windows (as .zip file)</a></li>
      <li><a href="xia2-0.3.5.1.tar.bz2">Unix (as .tar.bz2 file)</a></li>
      <li><a href="xia2-0.3.5.1.tar.gz">Unix (as .tar.gz file)</a></li>
    </ul>

    <p>All of the files are the same, simply packed for different platforms.
    </p>

    <h1>Installation</h1>

    <p>Since everything is precompiled, the only requirement is to unpack the
      directory somewhere and edit the directory setup.bat, setup.csh or
      setup.sh appropriately, for instance setting XIA2_HOME as
      /opt/px/xia/xia-0.3.5.1 if you unpacked the tarball in /opt/px/xia.
      Note that this includes the xia-0.3.5.1 directory!</p>

    <p>All that is then needed is to source this setup file and the application
      xia2 should be available. One more recent change is that I am using
      CCTBX python, which needs to be more recent than the version included
      (currently) in CCP4 - downloading an update from cctbx.sourceforge.net
      and reading the setup file <i>after</i> the CCP4 one works fine.
      Also loading any recent PHENIX installation should work fine.</p>



    <h1>Input</h1>

    There are two ways of running xia2 - with and without an input file,
    an example of which follows below. If you just run

<pre>
xia2 /my/data/are/here
</pre>

    xia2 will do something sensible - in this case process all of the data,
    scale all measurements as if they are from a single crystal and merge
    the data from each wavelength separately. If only one wavelength is present
    xia2 will assume that the data are a native data set - to separate
    anomalous pairs provide a heavy atom (at this time it doesn't matter
    what it is...) i.e.

<pre>
xia2 -atom se /my/data/are/here
</pre>

    Other options are (type just xia2 to get this list)

<pre>
Command-line options to xia2:
[-2d] or [-3d] or [-3dii]
[-parallel 4] (say, for XDS usage)
[-resolution 2.8] (say, applies to all sweeps)
[-freer_file free.mtz]
[-quick]
[-atom se] (say)
[-reversephi]
[-migrate_data]
[-beam x,y]
</pre>

    Running ths way some assumptions are made:

    <ul>
      <li>All images are from one crystal only.</li>
      <li>The scan, if present, was the one used to select the wavelengths
        for data collection. If more than one is present, the alphabetically
        latest one will be used.</li>
      <li>The sequence file, if present, should be in the one letter form
        and all comment lines should begin with a character not in A-Z.
        Again the alphabetically latest sequence file will be used.</li>
      <li>If the sequence file is provided, and the atom is "se", then
        xia2setup will assume that you are doing a SeMet experiment
        and will provide an appropriate number of atoms per monomer - though
        <b>you will still have to uncomment this</b> to include it,
        in case it has guessed wrong.</li>
    </ul>

    <p>If you want to combine data from a number of crystals in the same
      .xinfo file, then you will need to copy out all text from
      BEGIN CRYSTAL to END CRYSTAL from one .xinfo file to the other.</p>

    <p>You should then load automatic.xinfo in your favourite editor, and
      check that the sequence looks correct and that the names are sensible
      as well as checking that the epoch numbers are set correctly and
      also that the wavelengths and beam centres are correct. If you
      provided a heavy atom there is a place to say how many to look for.</p>

    <p>Finally, if you have labelit installed xia2 will run this
      to update the beam positions. If this happens, you will see a comment
      to this effect above the BEAM records in the sweeps.</p>

    The other mechanism for running xia2 is via a <i>xinfo</i> file, which
    explains the layout of the data set to xia2 explicitly. This is helpful
    if you wish to only process a subset of the measurements, or want to
    process data for an RIP experiment. A simple example of xinfo file
    follow below, and more complex examples can be found here:

    <ul>
      <li><a href="NATIVE.xinfo">Native data</a></li>
      <li><a href="SAD.xinfo">SAD data</a></li>
      <li><a href="MAD.xinfo">MAD data</a></li>
    </ul>

    <p>More examples can be found in $XIA2_ROOT/examples. Remember - you need
      to tell xia2 where to find your data! Answers to frequently asked
      questions can be found <a href="faq.html">here</a>.</p>

    <pre>
! This is a demonstration .xinfo file which illustrates how to cope
! with a simple case - this example is a native cubic insulin data
! set measured on 14.2 at the SRS

BEGIN PROJECT DEMONSTRATION

BEGIN CRYSTAL INSULIN

BEGIN AA_SEQUENCE

! this is only really needed at the moment for assessing the solvent
! content and number of residues in the asu

GIVEQCCASVCSLYQLENYCN
FVNQHLCGSHLVEALYLVCGERGFFYTPKA

END AA_SEQUENCE

BEGIN WAVELENGTH NATIVE

! this doesn't have to be here - if it is
! not included then the values from
! the image headers will be used - however
! if it is there then it should
! be correct!

WAVELENGTH 0.979000

! in here you can also have
! F' value
! F'' value

END WAVELENGTH NATIVE

BEGIN SWEEP NATIVE
WAVELENGTH NATIVE
IMAGE insulin_1_001.img

! you will probably need to change this -
! this is the only thing which
! you will need to change for the
! demonstration data set

DIRECTORY /media/data1/graeme/demo/

! additionally you can add the following
! information - if it is wrong in the headers
! BEAM x y (mm)
! DISTANCE z (mm)

! this describes the order in which
! the sweeps were collected -
! it usually comes from the image header
! if that information is in there
! EPOCH 5

! you can also add this to only reduce
! a subset of the data
! START_END 1 30 (image numbers)

END SWEEP

END CRYSTAL INSULIN

END PROJECT DEMONSTRATION
    </pre>

    <h1>Demonstration Data: Cubic Insulin</h1>

    <p>This uses test images which are available from the links below.
      Thanks to <NAME> for providing this test data!</p>

    <ul>
      <li><a href="demo.zip">Windows (as .zip file)</a></li>
      <li><a href="demo.tar.bz2">Unix (as .tar.bz2 file)</a></li>
      <li><a href="demo.tar.gz">Unix (as .tar.gz file)</a></li>
    </ul>

    <h1>Platforms</h1>

    <p>The following platforms are supported (see notes at bottom of page
      for platform specific advice.)</p>

    <ul>
      <li>GNU/Linux on i386 (and 64 bit extensions)</li>
      <li>OS X 10.4 i386 and Power PC G4/G5</li>
      <li>Microsoft Windows 2000/XP (Vista doesn't work - problem with
        the mosflm SPOTOD file - currently investigating)</li>
    </ul>

    <h1>Supported Detector Types</h1>

    <p>The following detectors are supported:</p>

    <ul>
      <li>ADSC CCD detectors</li>
      <li>MAR CCD detectors</li>
      <li>MAR 345 image plates</li>
      <li>RAXIS IV image plates</li>
      <li>Rigaku Saturn CCDs (Binned and unbinned)</li>
      <li>Pilatus 6M detectors</li>
    </ul>

    <p>If you have another detector class which needs support, please get
      in touch.</p>

    <h1>Requirements</h1>

    <p>The only software which is actually required is CCP4 6.0.1/2 and Python
      2.4.x. On Windows the PyWin32 module is also required, to provide
      additional process control functionality. If you are using this
      on a Windows platform and wish to be able to use the integration
      functionality, you should get in touch with <NAME> for a custom
      build of Mosflm.</p>

    <p>In addition the autoindexing program Labelit is supported but the
      system will work without it. If it is installed (and it should be a
      moderately recent version) then it will be used for autoindexing.</p>

    <h1>License</h1>

    <p>This software is distributed under the BSD license for all users.
      Everyone who uses this software is invited to join the xia2bb
      mailing list. Finally, if you use this software in solving a structure
      which is published, please acknowledge it! Thanks!</p>

    <p>A copy of the license is available <a href="license.html">here
      </a>.</p>

    <p>This software depends apon CCP4 and will also make use of Labelit -
      users are reminded that it is their responsibility to have properly
      obtained licenses for this software.</p>


    <h1>Usage</h1>

    <p>Once the software is installed, the first thing you need to do is
      prepare a .xinfo file (example above) to run from. This contains a
      description of the experiment, which will allow xia2 to decide how to
      handle your data and what information to put in the MTZ headers.</p>

    <p>Essentially the input file just needs a description of the data
      which were collected from each crystal - for instance, if you measured
      three MAD wavelengths, a peak, inflection and high remote, then
      there should be three WAVELENGTH blocks in the file, containing the
      fp, fpp and wavelength values. The images that you measured from these
      should then be described in the SWEEP block, which is then assigned to
      a wavelength. This structure simplifies handling of multiple passes
      contributing to one wavelength. In the reduced output, the wavelength
      names will correspond to MTZ datasets. If you are interested in
      keeping the harvesting information, and you have more than one crystal
      to process in a single xia2 run, you will need to ensure that the
      WAVELENGTH names are globally unique within the input file.</p>

    <p>The program is actually run with:</p>

    <pre>
xia2 -xinfo project.xinfo
    </pre>

    <p>And all data reduction is performed in the working directory, with the
      a directory structure CRYSTAL/WAVELENGTH/SWEEP. Interesting log files
      can be found in the CRYSTAL/scale directory.</p>

    <h1>Acknowledgements</h1>

    <p>Development of xia2 is supported by the BBSRC e-HTPX grant and the
      EU framework 6 BioXHit initiative. xia2 makes extensive use of Phil
      Evans's program Pointless, and I would like to thank him for the
      extensive modifications made to the program. I would also like to thank
      the US structural genomics JCSG group for providing their extensive
      collection of raw diffraction data to methods developers. Finally I
      would like to acknowledge CCP4, for without their software this project
      would go nowhere!</p>

    <h1>The More Complex Example</h1>

    <p>This example has two crystals, one of which is selenomethionated,
      with two wavelengths (inflection and low remote) measured for the
      derivative and two native sweeps measured for the native sample.
      Lines beginning with # or ! will be treated as comments.</p>

    <pre>
BEGIN PROJECT TS01
BEGIN CRYSTAL 12847

BEGIN AA_SEQUENCE

MKVKKWVTQDFPMVEESATVRECLHRMRQYQTNECIVKDREGHFRGVVNKEDLLDLDLDSSVFNKVSLPD
FFVHEEDNITHALLLFLEHQEPYLPVVDEEMRLKGAVSLHDFLEALIEALAMDVPGIRFSVLLEDKPGEL
RKVVDALALSNINILSVITTRSGDGKREVLIKVDAVDEGTLIKLFESLGIKIESIEKEEGF

END AA_SEQUENCE

BEGIN WAVELENGTH NATIVE
WAVELENGTH 0.99187
END WAVELENGTH NATIVE

! high resolution native pass

BEGIN SWEEP NATIVE_HR
WAVELENGTH NATIVE
BEAM 109.0 105.0
IMAGE 12847_4_001.img
DIRECTORY /Volumes/Arthur/JCSG Data/1vr9/data/jcsg/als1/8.2.1/20050121/collection/TM0892/12847/
END SWEEP NATIVE_HR

! low resolution native pass

BEGIN SWEEP NATIVE_LR
WAVELENGTH NATIVE
BEAM 109.0 105.0
IMAGE 12847_5_001.img
DIRECTORY /Volumes/Arthur/JCSG Data/1vr9/data/jcsg/als1/8.2.1/20050121/collection/TM0892/12847/
END SWEEP NATIVE_LR

END CRYSTAL 12847

BEGIN CRYSTAL 13140

BEGIN AA_SEQUENCE

MKVKKWVTQDFPMVEESATVRECLHRMRQYQTNECIVKDREGHFRGVVNKEDLLDLDLDSSVFNKVSLPD
FFVHEEDNITHALLLFLEHQEPYLPVVDEEMRLKGAVSLHDFLEALIEALAMDVPGIRFSVLLEDKPGEL
RKVVDALALSNINILSVITTRSGDGKREVLIKVDAVDEGTLIKLFESLGIKIESIEKEEGF

END AA_SEQUENCE

BEGIN HA_INFO
ATOM SE
NUMBER_PER_MONOMER 5
END HA_INFO

BEGIN WAVELENGTH INFL
WAVELENGTH 0.979741
F' -10.0
F'' 3.2
END WAVELENGTH INFL

BEGIN WAVELENGTH LREM
WAVELENGTH 1.019859
F' -2.6
F'' 0.55
END WAVELENGTH LREM

BEGIN SWEEP INFL
WAVELENGTH INFL
BEAM 108.7 102.0
IMAGE 13140_1_E1_001.img
DIRECTORY /Volumes/Arthur/JCSG Data/1vr9/data/jcsg/als1/8.3.1/20050105/collect/TM0892/13140/
END SWEEP

BEGIN SWEEP LREM
WAVELENGTH LREM
BEAM 108.7 102.0
IMAGE 13140_1_E2_001.img
DIRECTORY /Volumes/Arthur/JCSG Data/1vr9/data/jcsg/als1/8.3.1/20050105/collect/TM0892/13140/
END SWEEP

END CRYSTAL 13140

END PROJECT TS01
    </pre>

    <p>This is a particularly complex example, but it illustrates the
      capability of the system well. There are data from two cystals, one
      native and the other selenomethionated. The native set is measured in
      two sweeps. Currently (version 0.3.5.1) no attempt is made to
      intelligently combine data from multiple crystals, however this is on
      the to-do list.</p>

    <h1>Release Notes</h1>

    <p>Changes since 0.3.5.2: dealing with huge data sets</p>
    <ul>
      <li>If single sweep, do not rebatch.</li>
      <li>Replace CCP4 Mtzdump with CCTBX-powered equivalent.</li>
      <li>If reindex for changing spacegroup name but not indices
    (i.e. no REINDEX= card) then use CCTBX-powered replacement.</li>
    </ul>

    <p>Changes since 0.3.5.1: CCP4 patch release</p>
    <ul>
      <li>Resolution limit calculation now delegated to resolutionizer.</li>
      <li>Twinning test now uses CCTBX code not sfcheck.</li>
      <li>PythonDriver added to support execution of Python code as separate 
	process.</li>
    </ul>

    <p>Changes since 0.3.5.0</p>
    <ul>
      <li>-hdr_in mode - pass in text file containing image
      header. Can also generate this file with xia2 with -hdr_out example.hdr
      </li>
      <li>-xparm_ub mode for XDS processing only, provide a UB matrix (XDS 
	XPARM file) to match the orientation to: primarily useful if you record
	data from several regions on a sample, and the sample has close-to
	symmetry (i.e. you almost certainly don't need this)</li>
      <li>Removed usage of BINSORT_SCR as this is being deprecated by CCP4.
      </li>
      <li>Allow Phil overrides on selection of autoindexing solutions, where
	indexing can give large variations in unit cell constants (e.g. low
	resolution virus data)</li>
      <li>Properly set the resolution in CAD copying free R flags.</li>
      <li>Allow access to profile grid size through the Phil interface:

	xds.parameter.profile_grid_size = 13, 13
      </li>
      <li>Implement TRUSTED_REGION through Phil as suggested by <NAME>
      </li>
      <li>Updates to process simulated data from SIM_MX</li>
      <li>More measures of anomalous signal in output.</li>
      <li>Corrected citations in output</li>
      <li>Replaced TRUNCATE with CTRUNCATE.</li>
      <li>Replace sfcheck with E^4 calculation from cctbx.</li>
      <li>Exclude grid-scan data when running automatic setup.</li>
      <li>If XDS refinement not performed, exception not warning.</li>
      <li>If -3d and sweep narrower than 10 degrees, use XDS II indexer.</li>
    </ul>

    <p>Changes since 0.3.4.0</p>
    <ul>
      <li>Additional information in output: correct directory for BINSORT_SCR,
	free space in working directory, warns if less than 200MB in 
	BINSORT_SCR directory.</li>
      <li>EXCLUDE keyword added to XINFO SWEEP block to allow
	specific resolution ranges to be excluded in processing,
	contributed by <NAME> at Monash University.</li>
      <li>Include the resolution limits at the XSCALE stage rather
	than simply in the merging.</li>
      <li>Support for 2x2 binned ADSC Quantum 270 detector.</li>
      <li>Support for '.' in file names.</li>
      <li>Support for NOIR1 detector on ALS 4.2.2</li>
      <li>Smarter 3dii mode - if 3di style indexing will give better answer, 
	it will do that...</li>
      <li>Use ctruncate in place of truncate (finally!)</li>
    </ul>

    <p>Changes since 0.3.3.4</p>
    <ul>
      <li>Use Phil to override XDS parameters.</li>
      <li>Use Pointless in place of Reindex; Aimless in place of Scala; no
	limit to number of batches which can be processed.</li>
    </ul>

    <p>Changes since 0.3.3.3</p>
    <ul>
      <li>Fixed bug in polarization correction vector for Pilatus data.</li>
      <li>Fixed bug relating to assigning spacegroup on command-line.</li>
      <li>Begun work on supporting Pilatus 300K for small molecule
        crystallography. Have now successfully processed data from Diamond
        Light Source beamline I19 EH2. N.B. Only works -3dii!</li>
    </ul>

    <p>Changes since 0.3.3.2</p>
    <ul>
      <li>Many bug fixes as a result of implementing a comprehensive testing
        `scheme.</li>
      <li>Support for full cbf format, making use of cbflib Python bindings
        pycbf now included in cctbx releases.</li>
      <li>Merged functionality from XDSIntegraterR into XDSIntegrater, removed
        duplicate code. Will make for more maintainable suite in the future.
      </li>
      <li>Improved 2d pipeline scaling, now tests 8 permutations.</li>
      <li>Improved handling of twinned data for multiple sweeps.</li>
      <li>Fixed occasional bugs with multiple sweeps, reindexing trigonal
        data sets with XDS.</li>
      <li>Removed "old" scaling modes, 2d and 3d, as these were unhelpful.</li>
    </ul>

    <p>Changes since 0.3.3.1</p>
    <ul>
      <li>-xparallel -1 option added which will split your data into 30 degree
        chunks and process with forkintegrate <i>Note well you are responsible
          for setting up forkintegrate correctly!</i>
      </li>
    </ul>

    <p>Changes since 0.3.3.0</p>
    <ul>
      <li>Handle division in updated CCTBX.</li>
      <li>Squash bug with -3d and -resolution.</li>
      <li>Squash bug with resolution and -2d, result slightly different to
        that reported.</li>
      <li>Trap weirdness with XDS reindexing, causes problems sometimes
        trying to process in C2 from P222 not P2 as you would hope!</li>
      <li>Adjustments to cope with excludively narrow sweeps i.e. all 3
        image sweeps.</li>
    </ul>

    <p>Changes since 0.3.2.0</p>
    <ul>
      <li>Support for new XDS build.</li>
      <li>Small bug fix / improvement in XDS indexer implementation.</li>
      <li>Capability to ptovide refined experimental geometry <i>via</i>
        XDS GXPARM.XDS - useful for polycrystal data reduction.</li>
      <li>Now check what you tell the program, viz:
<pre>
gw56@ws050 nonsense]$ xia2 -3d -nonsense /data/gw56/dl/Cowan/Insulin/insulin/
Traceback (most recent call last):
  File "/home/gw56/svn/xia2/Applications/xia2.py", line 45, in
    from xia2setup import write_xinfo
  File "/home/gw56/svn/xia2/Applications/xia2setup.py", line 35, in
    from Handlers.CommandLine import CommandLine
  File "/home/gw56/svn/xia2/Handlers/CommandLine.py", line 1513, in
    CommandLine.setup()
  File "/home/gw56/svn/xia2/Handlers/CommandLine.py", line 349, in setup
    raise RuntimeError, nonsense
RuntimeError: Unknown command-line options: -nonsense
</pre>
        and complain if you type in something xia2 does not understand.
      </li>
      <li>Removed a <i>lot</i> of cruft from the xia2 code base in preparation
        for some more substantial refactoring.</li>
      <li>Found bug which meant that the labelit beam centre computed in the
        setup phase was not used - mostly this is not important, unless you are
        using -3dii.</li>
      <li>Added -ice command-line argument, which will exclude measurements
        from regions which are typically where ice rings land. Will need to
        add more subtle mechanism which will allow specific regions to be
        excluded.</li>
      <li>Specifying resolution with 3d(r) pipeline now works correctly.</li>
      <li>Removed XDS version check by default, which was annoying every new
        year. If you would like to check that the XDS version is explicitly
        supported, add -check_xds_version to the command-line.</li>
      <li>Added table of scala runs to sweep name in the debug output, helpful
        for reviewing in polycrystal data reduction cases.</li>
      <li>Mosaic spread used from Mosflm is now average of all images, not the
        first or last image.</li>
      <li>Fixed header reading for new Pilatus instruments at DLS and
        elsewhere.</li>
      <li>Added capacity to pass in known globally postrefined experimental
        setup via -xparm GXPARM.XDS, only useful with XDS processing
        pipelines.</li>
      <li>Removed use of ccp4 printpeaks tool, which crashes on new pilatus
        images, replace with Mosflm for the moment. Move to replacement with
        labelit code at later stage planned.</li>
    </ul>

    <p>Changes since 0.3.1.7</p>
    <ul>
      <li>Now report the min, mean, max mosaic spread (according to the
        <emph>program definitions</emph> after integration of each sweep.</li>
      <li>Resolved problems with overriding environment.</li>
      <li>Added -serial pun for -parallel 1.</li>
      <li>Allow resolution limit to be assigned on a per-sweep basis.</li>
      <li>Handled corner case with images 200 - 250 (say) giving REBATCH
        error.</li>
      <li>Added full 8-way search for scaling models for 2d pipelines.</li>
      <li>Partial support for two-theta offset data collection (tested with
        Rigaku X-ray equipment and Diamond I19 data.)</li>
      <li>Added -min_images keyword which is helpful when analysing virus data
        with e.g. 4 images / sweep.</li>
    </ul>

    <p>Changes since 0.3.1.6</p>
    <ul>
      <li>Changes suitable for working on microcrystals - reachable through
        a -microcrystal command line flag, and -failover is probably helpful
        too.</li>
      <li>If lattice or spacegroup specified, do not run tests.</li>
      <li>2dr and 3dr now the default pipelines - use 2dold or 3dold if you
        need to get to the old pipelines.</li>
      <li>Fixed selection of known lattice in IDXREF reindexing.</li>
    </ul>

    <p>Changes since 0.3.1.0</p>
    <ul>
      <li>Fixed bug where the resolution limit was not reset in an indexer
        solution elimination.</li>
      <li>Added support for XDS from Dec 09.</li>
      <li>Fixed some of the issues found in the resolution limit determination
        when processing low resolution data.</li>
      <li>Fixed use of I/sigma limit assignment.</li>
      <li>Added command-line control of individual indexer, integrater,
        scaler.</li>
      <li>Repaired regular expression for image matching, to cope with
        images where there is e.g. 3.5 in the template. This was previously
        misinterpreted as an image name of the form setup.NNNN.</li>
      <li>Implemented new resolution limit pipelines, based on new merging code
        to give more robust control: -2dr and -3dr (recommended.) Fine
        control over the choices can be made using the -isigma, -misigma,
        -completeness and -rmerge command line options.</li>
      <li>Added interactive indexing mechanism, where user can assign images
        to use for autoindexing, e.g.:

        <pre>
Existing images for indexing: 1, 90, 180
>1, 60, 120, 180
New images for indexing: 1, 60, 120, 180
        </pre>

        Assign -interactive on the command line (Mosflm and Labelit indexers)
      </li>
    </ul>

    <p>Changes since 0.3.0.6</p>
    <ul>
      <li>Added -xparallel command line option to allow use of forkinitegrate
        on a cluster with the 3d pipeline.</li>
      <li>Allow for xia2html and (beta) ISPyB output (latter more useful for
        synchrotron sources, former handy if you want a report. </li>
      <li>Smart scaling mode switched on by default.</li>
      <li>Chef now run by default, for radiation damage analysis, e.g.
        <pre>
Group 1: 2-wedge data collection
Group 2: Single wedge
Significant radiation damage detected:
Rd analysis (12287/LREM): 15.17
Rd analysis (12287/INFL): 7.89
Rd analysis (12287/PEAK): 12.82
Conclusion: cut off after DOSE ~ 608.8
        </pre>
      </li>
      <li>Blank images now handled more gracefully.</li>
      <li>Overloaded and blank images now correctly reported in the summary
        output, along with abandoned images for the processing with
        Mosflm. Also postrefinement results.</li>
      <li>Substantially cleaned up the program output.</li>
    </ul>

    <p>Changes since 0.3.0.5</p>
    <ul>
      <li>No lattice test mode added (you can guess the command-line option)
        for tricky data sets where this perhaps falls over.</li>
      <li>Fixed side effect of changes which allowed setting of unit cell
        etc. - lattices were no longer eliminated, failed complaining
        can't eliminate only solution.</li>
      <li>Tidied up the generation of output files and so on - now only have
        the main log file and the debug trace.</li>
      <li>Added copy of the command line to program output.</li>
    </ul>

    <p>Changes since 0.3.0.4</p>
    <ul>
      <li>Now able to specify number or fraction of reflections assigned to
        the free set, rather than the default 5 percent.</li>
      <li>Added trap for sdcorrection not being refined in XDS correct if
        multiplicity rather low. Unusual case.</li>
      <li>Assigning a freer_file by definition sets this as an indexing
        reference and also copies the spacegroup assignment. This is
        what you would expect!</li>
    </ul>

    <p>Changes since 0.3.0.3</p>
    <ul>
      <li>User now able to assign cell constants: use with great care, as
        there is little in the way of nonsense trapping. This may however
        be used to handle cases where the solution you want is monoclinic
        with a pseudo-orthorhombic lattice, where the default indexing would
        select a different setting. This will not currently work with a
        Mosflm indexer. Usage is:
<pre>-cell a,b,c,alpha,beta,gamma</pre>

        and the correct symmetry should also be assigned. Can also assign
        USER_CELL a b c alpha beta gamma in the xinfo file.</li>
    </ul>

    <p>Changes since 0.3.0.0</p>
    <ul>
      <li>Now automatically determine the number of available processor
        cores.</li>
      <li>Implemented check for centring of crystallographic basis from
        autoindexing.</li>
      <li>Started to use CCTBX - this will be bundled with the release
        from now on...</li>
      <li>Now runs "smart scaling" i.e. will customise the scaling model
        used to the data. This can seriously improve the xia2 run time in
        cases where some of the default scaling models do not converge.</li>
      <li>Now also includes running of CHEF for radiation damage, which will
        slice and dice your data into dose groups, then run a correctly
        time sequenced radiation damage analysis. N.B. for data measured in
        "dose mode" the doses will be scaled to &lt; 1,000,000 to ensure that
        the output is tidy.</li>
      <li>Corner case of running XDS pipelines from data from a Rigaku setup.
        The low resolution was previously calculated to be 0.0! D'oh!</li>
      <li>Added access to reference reflection file functionality to the
        command line - -reference_reflection_file foo.mtz.</li>
      <li>Fixed the use of the -image command-line option. Now works.</li>
      <li>Removed all of the binaries from the distribution, so now
        <b>you need to be using CCP4 6.1.0 or later!</b></li>
      <li>Will correctly handle reindexing with the 3d pipeline (XDS/XSCALE)
        with trigonal spacegroups and multiple sweeps.</li>
    </ul>

    <p>Changes since 0.2.7.2</p>
    <ul>
      <li>Fixed merging statistics bug - I/sigma output were very slightly
        different.</li>
      <li>Now extend FreeR column if copying from lower resolution input
        file.</li>
      <li>Detector limits now correctly specified around the detector centre
        for Mosflm, rather than the beam centre. Only useful if your beam
        centre is a fair way from the image centre...</li>
      <li>Parallelised integration with Mosflm - divides the sweeps into an
        appropriate number of chunks and then sorts together the resulting
        reflection files. -parallel N, remember.</li>
      <li>Added a putative 'small molecule' mode, which will autoindex with
        Mosflm from a modest number of images, for smaller molecules where
        Labelit is unhappy with the number of good Bragg reflections.</li>
    </ul>

    <p>Changes since 0.2.7.0</p>
    <ul>
      <li>Fixed to work correctly with XDS and Rigaku Saturn/RAXIS
        detectors.</li>
      <li>Better determination of resolution limits.</li>
      <li>Correct merging of data from XSCALE (needs to invert the I=F*F
        scale factor...)</li>
      <li>Fixed output of scalepack unmerged output using 3d pipeline.</li>
      <li>Use fewer frames for background calculation with 3dii pipeline -
        speeds things up a fair amount.</li>
      <li>Write chef output files with correct SD correction parameters.</li>
      <li>Print SD correction factors for scala runs - if these are bigger
        than about 2 there is something properly wrong. Also print the
        same from XDS CORRECT step.</li>
    </ul>

    <p>Changes since 0.2.6.6</p>
    <ul>
      <li>All command line options now echoed to the standard output if
        -debug selected.</li>
      <li>Command line options now copied to .xinfo file when operating in
        automatic mode.</li>
      <li>XDS: low resolution limit determined from spot list from
        IDXREF.</li>
      <li>Now include explicit python version check in xia2 main program.</li>
      <li>High resolution limits for integration now assessed from Wilson
        plot rather than integration program log output.</li>
      <li>If no images are given, a more helpful error message is produced
        than before, viz:
<pre>
------------------------------------------
| No images assigned for crystal DEFAULT |
------------------------------------------
</pre>
      </li>
      <li>If images are not readable (i.e. a permissions problem) then
        warnings will be sent to the debug channel (switch on -debug
        for more information.)</li>
      <li>If in automatic mode an image is given in place of a directory,
        xia2 will now complain in a more helpful way.</li>
      <li>Resolution limits will now be based on analysis of the reflection
        files rather than the program output, since this will generally give
        a more helpful answer...</li>
      <li>Now correctly set the reindex matrix for running correct with
        multiple sweeps of data for non primitive matrix (subtle bug,
        thanks to <NAME> for helping to fix this one.)<li>
      <li>Is images are missing from a sweep, xia2 will now tell you
        this before starting processing rather than giving a strange error -
        if the images are unreadable you will also be told.</li>
    </ul>

    <p>Changes since 0.2.6.5</p>
    <ul>
      <li>Now supports CCP4 6.1 (forthcoming release) though you may need
        to add XIA2_CCP4_61=1 to your environment. This release should include
        xia2.</li>
      <li>Added -user_resolution keyword, also support for this on a
        per-wavelength basis in the xinfo file.</li>
      <li>Added support for reverse-phi data collection - either add
        reversephi to the xinfo file or -reversephi to the command line.</li>
    </ul>

    <p>Changes since 0.2.6.4</p>
    <ul>
      <li>Now handles complex image names with e.g. + in them correctly.</li>
      <li>New option for processing <pre>-3dii</pre> added - this will run
        the 3D pipeline (XDS, XSCALE) using all images for spot picking and
        autoindexing - useful for properly tricky cases.</li>
      <li>Now correctly propogate the profile information to Mosflm when not
        using Mosflm for autoindexing - this is done by running a "fake"
        autoindexing task.</li>
      <li>Use the raster information for the cell refinement test, and changed
        the selection of images for this.</li>
      <li>Added support for the output of BioXHIT XML in the LogFiles
        directory - for project tracking.</li>
      <li>Now copy the XDS log files ("LP" files) into the LogFiles
        directory - as always these are the most up-to-date versions from
        processing.</li>
      <li>Added support for latest release of XDS (June 2, 2008.)</li>
    </ul>

    <p>Changes since 0.2.6.3</p>
    <ul>
      <li>Added possibility to give a reflection file with an existing
        FreeR_flag column, which will be copied to the output MTZ:
        <pre>
xia2 -freer_file free.mtz
        </pre>
or...
        <pre>
FREER_FILE free.mtz
        </pre>
        in the xinfo file (in the same way as the reference reflection file -
        indeed, this may be the same file...)
      </li>
      <li>Added command line control of the spacegroup - note well that this
        means that the data WILL NOT be reindexed to a standard setting.</li>
      <li>Added support for ADSC Quantum 270.</li>
      <li>Intelligent selection of autoindex threshold for Mosflm.</li>
      <li>Added support for latest XDS version.</li>
      <li>Added support for pilatus 6M mini CBF images.</li>
      <li>Fixed problem with cell refinement giving negative mosaic
        spreads sometimes for 2d pipeline.</li>
    </ul>

    <p>Changes since 0.2.6.2</p>
    <ul>
      <li>Now works for images with long image numbers, as typically
        recorded on Rigaku X-Ray sets.</li>
      <li>Signed off for operation with Rigaku Saturn and RAXIS IV
        detectors.</li>
      <li>Now works with no input, e.g.
        <pre>
xia2 -project TG6623 -crystal X77788 -atom se /my/images/are/here
        </pre>
        However this relies on your image headers being accurate and the
        images having some kind of recognisable format...
      </li>
      <li>Repaired operation on ppc and intel macs - added required libraries
        to the installation and reset the library paths appropriately.</li>
      <li>Slightly improved error reporting from xinfo file errors.</li>
    </ul>


    <p>Changes since 0.2.6.1</p>
    <ul>
      <li>Now carefully select the images to use for cell refinement based
        on the orientation of the crystal lattice.</li>
      <li>Fixed numerous bugs to do with the naming of the detector class
        which changed in the previous version.</li>
    </ul>

    <p>Changes since 0.2.6.0</p>

    <ul>
      <li>Fixed bug where if the distance was read incorrectly from the
        image header (or was wrong therein) XDS would get the wrong value
        even if you had put the correct value in the .xinfo file.</li>
      <li>Now "unroll" the unmerged reflections from XSCALE and then
        merge them in their original sweeps in Scala. This should give
        a useful Rmerge vs. time plot.</li>
      <li>Including updated versions of Pointless and Scala.</li>
      <li>Includes new and more robust handling of pointgroups, lattices
        and unstable refinement of parameters during integration.</li>
      <li>Include support for a reference reflection file, which will
        ensure that the reflections are indexed in the same way and
        with the same spacegroup - useful for mutants.</li>
      <li>Now always use three wedges of images for the cell refinement
        with Mosflm as this makes the lattice elimination more reliable.
      </li>
      <li>If one of the images in a sweep is broken (e.g. corrupted)
        xia2 will tell you more helpfully rather than just crashing.</li>
      <li>Now correctly read distance from Mar 165 images. Thanks to
        <NAME> for this fix to DiffractionImage.</li>
      <li>Fixed the use of xia2 -3d with RAXIS IV detectors.</li>
      <li>Use more images for Mosflm autoindexing (three instead of two)
        as this gives uniformly better results.</li>
    </ul>

    <p>Changes since 0.2.5.2</p>

    <ul>
      <li>Now works for highly incomplete data sets, so long as you
        have either 4 times the mosaic spread or 8 images, whichever
        is the larger. This is to allow pointgroup identification
        from e.g. the first 10 images in a data set - designed to assist
        in synchrotron / lab source data collection.</li>
      <li>Now include the latest Mosflm binaries in the extras package -
        these are often better than those in the CCP4 distribution,
        as they come directly from Harry Powell's web page.</li>
      <li>Added -ehtpx_xml_out option which will write out marked
        up metadata about the data reduction, for inclusion in the
        e-HTPX data reduction portal and, perhaps, other automated
        systems. Usage:
        <pre>
xia2 -ehtpx_xml_out project.xml
        </pre>
      </li>
      <li>If data are very incomplete (e.g. less than 50% complete) then don't
        try to refine the error parameters. This is both incorrect and a
        waste of time.</li>
      <li>Renamed the output reflection files - these are now in DataFiles
        directory with names like "PROJECT_CRYSTAL_free.mtz." This makes
        them a little easier to identify.</li>
      <li>Harvesting (deposition) files now in subdirectory of Harvest
        rather than being spread around the place. These will have
        names Harvest/DepositFiles/PROJECT/WAVELENGTH.scala etc.</li>
      <li><b>Have now fixed xia2setup so it works much more sensibly -
          the decision about when a new sweep starts was a little broken
          (rounding errors issue) now fixed!</b></li>
      <li>Added -parallel flag to work when using XDS for data reduction.</li>
      <li><b>Added XDS support!</b> This is however much less mature than
        the support for Mosflm/CCP4. It is also worth noting that this
        relies on many CCP4 tools.</li>
      <li>Added selection of the "pipeline" to use through a command line
        option, either -2d for mosflm/ccp4 or -3d for xds/xscale though
        the latter still uses a number of CCP4 programs.</li>
    </ul>

    <p>Changes since 0.2.5.1 - big changes in <b>bold</b></p>
    <ul>
      <li><b>Process data as native if no heavy atom information, only one
          wavelength specified and no anomalous scattering form factors
          provided. Otherwise separate anomalous pairs for scaling.</b></li>
      <li><b>Added a -quick option to the xia2 command line, which will
          cut out many of the data reduction optimisation steps (not
          refine detector parameters, resolution or scaling parameters)
          but will still include everything else. This is to allow quick
          characterisation of data at the beamline, and perhaps map
          calculation?</b></li>
      <li>Correctly print the unmerged scalepack file name for single
        wavelength data.</li>
      <li>Perform radiation damage analysis using R merge and B factor
        as a function of <i>time</i> - note that this is not <i>batch.</i>
        This is likely to have interesting side-effects.</li>
      <li>If time of data collection not recorded in header <i>but</i>
        the time stamp for the images are correct (e.g. you have not moved
        them since collection) then the command line flag
        <pre>-trust_timestamps</pre> can be used, which will use the
        time stamps on the image files to analyse things like radiation
        damage.</li>
      <li>If you are running this on a cluster, have added an option
        to migrate the diffraction data to a local disk (e.g. /tmp.)
        To do this add <pre>-migrate_data</pre> to the command line -
        the data will be removed from the local disk once the processing
        is finished.</li>
      <li>Copy all final reflection files to a DataFiles directory.</li>
      <li>Now, when something goes horribly wrong, just write the actual
        message to the screen and the horrible stack trace gunk to a file
        called xia2.error. This should be sent in for error reports!</li>
      <li>Remerge individual wavelengths to get the merging statistics to
        the most appropriate resolution limit, rather than the furthest
        extent of any data, for the summary table.</li>
      <li>If GAIN included in SWEEP block, will be used as the default
        value for integration. E.g. GAIN 0.25.</li>
    </ul>

    <p>Changes since 0.2.5</p>
    <ul>
      <li>Fixed problem with spacegroup R3:H (naming convention problem
        - there's a surprise.)</li>
      <li>GAIN estimation temporarily removed.</li>
      <li>Pointgroup evaluation fixed - found a <b>major</b> gremlin in
        there</li>
      <li>Allow environment variables and tilde in DIRECTORY token:
        <pre>DIRECTORY $DATA/example or ~/data or %DATA%/example (win32)</pre>
      </li>
      <li>Now tracks the citations for the programs used, writing them
        to the standard output in plain text and to xia-citations.bib
        in BIBTeX format.</li>
      <li>Removed [XIA2] tokens from standard output - the reason for
        having this there no longer applies.</li>
    </ul>


    <p>Changes since 0.2.4</p>

    <ul>
      <li>xia2setup will now add the f', f'' values if a scan is available
        and has been processed by chooch</li>
      <li>GAIN now estimated by diffdump - enabling future parallelisation
        of the integration stage.</li>
      <li>For detectors formed as a mosaic of a number of tiles, with "gaps"
        in between, now mask those gapped areas in Mosflm to reduce the
        number of bad reflections.</li>
      <li>Now has a fix for the ongoing indexing problem which gave rise
        to errors like "something horrible has happened in indexing".</li>
      <li>The "best" log file for each process will now be recorded in a
        LogFiles directory - which means no more hunting around for the
        final scala log file and what not.</li>
      <li>Chaling now produces unmerged reflection files in scalepack format
        as well as merged scalepack and merged MTZ format. This is done by
        scala (if used) by recycling the final SCALES and adding
        "output polish unmerged."</li>
      <li>Now runs a twinning test using the CCP4 program sfcheck, and
        will warn you if your data look twinned.</li>
      <li>Now allow up to 100 cycles of scale refinement to cope with
        more extreme cases where data at very different resolutions
        are scaled together.</li>
    </ul>

    <p>Changes since 0.2.3</p>
    <ul>
      <!-- <li>Added support for XDS</li> -->
      <li>Added <a href="preferences.html">preferences file</a></li>
      <li>Added xia2setup program to create the .xinfo file - this will
        also run LABELIT to configure the beam centre if it is installed</li>
      <li>Added a strategy-of-data-reduction step to the pipeline</li>
      <li>Added run-time check that CCP4 and so on are available</li>
    </ul>

    <p>Changes since 0.2.2.4:</p>

    <ul>
      <li>Changed to a BSD license</li>
      <li>Added "python setup" check to xia2</li>
      <li>Relaxed criteria on isomorphism to 1% not 0.5A etc.</li>
      <li><b>Now wavelengths in wavelength record will override header
        values if provided, but do not (indeed, should not) be included
        if wavelength values are correct.</b></li>
    </ul>

    <p>Changes since 0.2.2.3:</p>

    <ul>
      <li>Created version for Power PC G4/G5</li>
      <li>Fixed .csh setup scripts</li>
      <li>If wavelength values not specified in the .xinfo file will use
        values from the image headers</li>
      <li>If loggraph output from scala is broken by having Mn(I/sigma)
        greater than 99.99 can now cope (ignores that record)</li>
      <li>Will now use a maximum of 180 degrees of data for deciding
        e.g. the point group, spacegroup and correct indexing standards -
        this helps the cases where exceptionally high redundancy data
        has been measured</li>
    </ul>

    <p>Changes since 0.2.2.2:</p>

    <ul>
      <li>Updated printheader to cope with MAR 165</li>
      <li>Cleaner error messages when data missing</li>
      <li>Added support for MAR image plate data</li>
      <li>Included more example .xinfo files in the examples directory</li>
      <li>Relaxed refinement parameters in Mosflm cell refinement</li>
    </ul>

    <h1>Platform Specific Advice</h1>

    <h2>Gfortran compiled CCP4</h2>

    <p>Recent versions of the gcc compiler suite have included a new compiler
      called "gfortran" in place of "g77". This has, unfortunately, been
      designed with buffering too high on the agenda, and the output
      from programs can sometimes be mashed. If this happens you need to
      set "GFORTRAN_UNBUFFERED_ALL" to 1 in your environment.</p>

    <h2>Linux</h2>

    <p>There should be no particular problems on this platform - assuming
      that you have a recent version of Python. Any 2.4 version should work,
      but I have mostly used 2.4.3 and 2.4.4, and would recommend them.
      If the version you have on your system by default (type "python"
      to find out what version you have)is too old, you can easily install
      a new version, perhaps in the xia directory, from
      <a href="http://www.python.org/download/releases/2.4.4/">here</a>.
      This is best installed by unpacking the tarball (tar xvfz
      Python-2.4.4.tgz) then doing ./configure /where/I/want/it then
      make, make install.</p>

    <h2>OS X.4 - X.6</h2>

    <p>The only potential problem with this platform is that the Python
      which comes as standard is 2.3 - however since you need cctbx.python
      anyway the problem has gone away.</p>

    <h2>Windows XP</h2>

    <p>Once more you will need a recent version of Python - which can be found
      <a href="http://www.python.org/download/releases/2.4.4/">here</a>,
      however, you will also need "pywin32" to provide a little more
      job control functionality - this is available from
      <a href="http://sourceforge.net/projects/pywin32/">here</a>. Both are
      simple, double-click, binary installers.</p>
    <p>There is now a version of Labelit available for Windows, which
      can be found <a href="http://adder.lbl.gov/labelit/">here.</a></p>
    <p>The mosflm binary which comes with
      <a href="http://www.mrc-lmb.cam.ac.uk/harry/imosflm/index.html">
        iMosflm</a> works well with xia2.</p>

  </body>
</html>
<commit_msg>Added link to project page<commit_after><!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
  <head>
    <title>xia2: Automated Data Reduction</title>
  </head>

  <body>
    <h1><img src="xia2-new-logo.gif" alt="xia2"></img>
    Automated Data Reduction</h1>

    <p><a href="http://sourceforge.net/projects/xia2/">Sourceforge project 
	page</a>.</p>

    <p>New! I have now set up a mailing list for xia2 on sourceforge.
      If you would like to subscribe to this - where announcements of new
      versions of xia2 and problems can be discussed, please go to
      <a href=https://lists.sourceforge.net/lists/listinfo/xia2-list>the list
      page</a>. Also, if you are interested in keeping up-to-date with what
      is going on in xia2 development take a look at the
      <a href="http://xia2.blogspot.com">blog</a>.</p>

    <p>xia2 is a new automated data reduction system designed to work from raw
      diffraction data and a little metadata, and produce usefully reduced
      data in a form suitable for immediately starting phasing and structure
      solution, e.g. through Mr BUMP or your favourite experimental phasing
      suite.</p>

    <p>The following steps are performed (although not strictly in
      this order)</p>

    <ul>
      <li>Integration with feedback and complex lattice selection stuff</li>
      <li>Reindexing to standard setting for spacegroup (e.g. P 2 21 21 to
        P 21 21 2)</li>
      <li>Scaling together of all data to maximise anomalous signal</li>
      <li>Estimation of number of molecules in ASU and solvent fraction
        based on sequence information (if provided)</li>
    </ul>

    <p>There is a lot of fun stuff which goes on in there - run it to find
      out more...</p>

    <p>The system is designed to cope with MAD data and data measured in
      multiple passes, or a combination of the two. The input is a therefore
      a little complex, but consists only of information you should have
      to hand anyway - here is a simple example, a more complex example
      can be found below.</p>

    <h1>Download</h1>

    <p>The current version, 0.3.5.1 can be obtained by following these
      links:</p>

    <ul>
      <li><a href="xia2-0.3.5.1.zip">Windows (as .zip file)</a></li>
      <li><a href="xia2-0.3.5.1.tar.bz2">Unix (as .tar.bz2 file)</a></li>
      <li><a href="xia2-0.3.5.1.tar.gz">Unix (as .tar.gz file)</a></li>
    </ul>

    <p>All of the files are the same, simply packed for different platforms.
    </p>

    <h1>Installation</h1>

    <p>Since everything is precompiled, the only requirement is to unpack the
      directory somewhere and edit the directory setup.bat, setup.csh or
      setup.sh appropriately, for instance setting XIA2_HOME as
      /opt/px/xia/xia-0.3.5.1 if you unpacked the tarball in /opt/px/xia.
      Note that this includes the xia-0.3.5.1 directory!</p>

    <p>All that is then needed is to source this setup file and the application
      xia2 should be available. One more recent change is that I am using
      CCTBX python, which needs to be more recent than the version included
      (currently) in CCP4 - downloading an update from cctbx.sourceforge.net
      and reading the setup file <i>after</i> the CCP4 one works fine.
      Also loading any recent PHENIX installation should work fine.</p>



    <h1>Input</h1>

    There are two ways of running xia2 - with and without an input file,
    an example of which follows below. If you just run

<pre>
xia2 /my/data/are/here
</pre>

    xia2 will do something sensible - in this case process all of the data,
    scale all measurements as if they are from a single crystal and merge
    the data from each wavelength separately. If only one wavelength is present
    xia2 will assume that the data are a native data set - to separate
    anomalous pairs provide a heavy atom (at this time it doesn't matter
    what it is...) i.e.

<pre>
xia2 -atom se /my/data/are/here
</pre>

    Other options are (type just xia2 to get this list)

<pre>
Command-line options to xia2:
[-2d] or [-3d] or [-3dii]
[-parallel 4] (say, for XDS usage)
[-resolution 2.8] (say, applies to all sweeps)
[-freer_file free.mtz]
[-quick]
[-atom se] (say)
[-reversephi]
[-migrate_data]
[-beam x,y]
</pre>

    Running ths way some assumptions are made:

    <ul>
      <li>All images are from one crystal only.</li>
      <li>The scan, if present, was the one used to select the wavelengths
        for data collection. If more than one is present, the alphabetically
        latest one will be used.</li>
      <li>The sequence file, if present, should be in the one letter form
        and all comment lines should begin with a character not in A-Z.
        Again the alphabetically latest sequence file will be used.</li>
      <li>If the sequence file is provided, and the atom is "se", then
        xia2setup will assume that you are doing a SeMet experiment
        and will provide an appropriate number of atoms per monomer - though
        <b>you will still have to uncomment this</b> to include it,
        in case it has guessed wrong.</li>
    </ul>

    <p>If you want to combine data from a number of crystals in the same
      .xinfo file, then you will need to copy out all text from
      BEGIN CRYSTAL to END CRYSTAL from one .xinfo file to the other.</p>

    <p>You should then load automatic.xinfo in your favourite editor, and
      check that the sequence looks correct and that the names are sensible
      as well as checking that the epoch numbers are set correctly and
      also that the wavelengths and beam centres are correct. If you
      provided a heavy atom there is a place to say how many to look for.</p>

    <p>Finally, if you have labelit installed xia2 will run this
      to update the beam positions. If this happens, you will see a comment
      to this effect above the BEAM records in the sweeps.</p>

    The other mechanism for running xia2 is via a <i>xinfo</i> file, which
    explains the layout of the data set to xia2 explicitly. This is helpful
    if you wish to only process a subset of the measurements, or want to
    process data for an RIP experiment. A simple example of xinfo file
    follow below, and more complex examples can be found here:

    <ul>
      <li><a href="NATIVE.xinfo">Native data</a></li>
      <li><a href="SAD.xinfo">SAD data</a></li>
      <li><a href="MAD.xinfo">MAD data</a></li>
    </ul>

    <p>More examples can be found in $XIA2_ROOT/examples. Remember - you need
      to tell xia2 where to find your data! Answers to frequently asked
      questions can be found <a href="faq.html">here</a>.</p>

    <pre>
! This is a demonstration .xinfo file which illustrates how to cope
! with a simple case - this example is a native cubic insulin data
! set measured on 14.2 at the SRS

BEGIN PROJECT DEMONSTRATION

BEGIN CRYSTAL INSULIN

BEGIN AA_SEQUENCE

! this is only really needed at the moment for assessing the solvent
! content and number of residues in the asu

GIVEQCCASVCSLYQLENYCN
FVNQHLCGSHLVEALYLVCGERGFFYTPKA

END AA_SEQUENCE

BEGIN WAVELENGTH NATIVE

! this doesn't have to be here - if it is
! not included then the values from
! the image headers will be used - however
! if it is there then it should
! be correct!

WAVELENGTH 0.979000

! in here you can also have
! F' value
! F'' value

END WAVELENGTH NATIVE

BEGIN SWEEP NATIVE
WAVELENGTH NATIVE
IMAGE insulin_1_001.img

! you will probably need to change this -
! this is the only thing which
! you will need to change for the
! demonstration data set

DIRECTORY /media/data1/graeme/demo/

! additionally you can add the following
! information - if it is wrong in the headers
! BEAM x y (mm)
! DISTANCE z (mm)

! this describes the order in which
! the sweeps were collected -
! it usually comes from the image header
! if that information is in there
! EPOCH 5

! you can also add this to only reduce
! a subset of the data
! START_END 1 30 (image numbers)

END SWEEP

END CRYSTAL INSULIN

END PROJECT DEMONSTRATION
    </pre>

    <h1>Demonstration Data: Cubic Insulin</h1>

    <p>This uses test images which are available from the links below.
      Thanks to <NAME> for providing this test data!</p>

    <ul>
      <li><a href="demo.zip">Windows (as .zip file)</a></li>
      <li><a href="demo.tar.bz2">Unix (as .tar.bz2 file)</a></li>
      <li><a href="demo.tar.gz">Unix (as .tar.gz file)</a></li>
    </ul>

    <h1>Platforms</h1>

    <p>The following platforms are supported (see notes at bottom of page
      for platform specific advice.)</p>

    <ul>
      <li>GNU/Linux on i386 (and 64 bit extensions)</li>
      <li>OS X 10.4 i386 and Power PC G4/G5</li>
      <li>Microsoft Windows 2000/XP (Vista doesn't work - problem with
        the mosflm SPOTOD file - currently investigating)</li>
    </ul>

    <h1>Supported Detector Types</h1>

    <p>The following detectors are supported:</p>

    <ul>
      <li>ADSC CCD detectors</li>
      <li>MAR CCD detectors</li>
      <li>MAR 345 image plates</li>
      <li>RAXIS IV image plates</li>
      <li>Rigaku Saturn CCDs (Binned and unbinned)</li>
      <li>Pilatus 6M detectors</li>
    </ul>

    <p>If you have another detector class which needs support, please get
      in touch.</p>

    <h1>Requirements</h1>

    <p>The only software which is actually required is CCP4 6.0.1/2 and Python
      2.4.x. On Windows the PyWin32 module is also required, to provide
      additional process control functionality. If you are using this
      on a Windows platform and wish to be able to use the integration
      functionality, you should get in touch with <NAME> for a custom
      build of Mosflm.</p>

    <p>In addition the autoindexing program Labelit is supported but the
      system will work without it. If it is installed (and it should be a
      moderately recent version) then it will be used for autoindexing.</p>

    <h1>License</h1>

    <p>This software is distributed under the BSD license for all users.
      Everyone who uses this software is invited to join the xia2bb
      mailing list. Finally, if you use this software in solving a structure
      which is published, please acknowledge it! Thanks!</p>

    <p>A copy of the license is available <a href="license.html">here
      </a>.</p>

    <p>This software depends apon CCP4 and will also make use of Labelit -
      users are reminded that it is their responsibility to have properly
      obtained licenses for this software.</p>


    <h1>Usage</h1>

    <p>Once the software is installed, the first thing you need to do is
      prepare a .xinfo file (example above) to run from. This contains a
      description of the experiment, which will allow xia2 to decide how to
      handle your data and what information to put in the MTZ headers.</p>

    <p>Essentially the input file just needs a description of the data
      which were collected from each crystal - for instance, if you measured
      three MAD wavelengths, a peak, inflection and high remote, then
      there should be three WAVELENGTH blocks in the file, containing the
      fp, fpp and wavelength values. The images that you measured from these
      should then be described in the SWEEP block, which is then assigned to
      a wavelength. This structure simplifies handling of multiple passes
      contributing to one wavelength. In the reduced output, the wavelength
      names will correspond to MTZ datasets. If you are interested in
      keeping the harvesting information, and you have more than one crystal
      to process in a single xia2 run, you will need to ensure that the
      WAVELENGTH names are globally unique within the input file.</p>

    <p>The program is actually run with:</p>

    <pre>
xia2 -xinfo project.xinfo
    </pre>

    <p>And all data reduction is performed in the working directory, with the
      a directory structure CRYSTAL/WAVELENGTH/SWEEP. Interesting log files
      can be found in the CRYSTAL/scale directory.</p>

    <h1>Acknowledgements</h1>

    <p>Development of xia2 is supported by the BBSRC e-HTPX grant and the
      EU framework 6 BioXHit initiative. xia2 makes extensive use of Phil
      Evans's program Pointless, and I would like to thank him for the
      extensive modifications made to the program. I would also like to thank
      the US structural genomics JCSG group for providing their extensive
      collection of raw diffraction data to methods developers. Finally I
      would like to acknowledge CCP4, for without their software this project
      would go nowhere!</p>

    <h1>The More Complex Example</h1>

    <p>This example has two crystals, one of which is selenomethionated,
      with two wavelengths (inflection and low remote) measured for the
      derivative and two native sweeps measured for the native sample.
      Lines beginning with # or ! will be treated as comments.</p>

    <pre>
BEGIN PROJECT TS01
BEGIN CRYSTAL 12847

BEGIN AA_SEQUENCE

MKVKKWVTQDFPMVEESATVRECLHRMRQYQTNECIVKDREGHFRGVVNKEDLLDLDLDSSVFNKVSLPD
FFVHEEDNITHALLLFLEHQEPYLPVVDEEMRLKGAVSLHDFLEALIEALAMDVPGIRFSVLLEDKPGEL
RKVVDALALSNINILSVITTRSGDGKREVLIKVDAVDEGTLIKLFESLGIKIESIEKEEGF

END AA_SEQUENCE

BEGIN WAVELENGTH NATIVE
WAVELENGTH 0.99187
END WAVELENGTH NATIVE

! high resolution native pass

BEGIN SWEEP NATIVE_HR
WAVELENGTH NATIVE
BEAM 109.0 105.0
IMAGE 12847_4_001.img
DIRECTORY /Volumes/Arthur/JCSG Data/1vr9/data/jcsg/als1/8.2.1/20050121/collection/TM0892/12847/
END SWEEP NATIVE_HR

! low resolution native pass

BEGIN SWEEP NATIVE_LR
WAVELENGTH NATIVE
BEAM 109.0 105.0
IMAGE 12847_5_001.img
DIRECTORY /Volumes/Arthur/JCSG Data/1vr9/data/jcsg/als1/8.2.1/20050121/collection/TM0892/12847/
END SWEEP NATIVE_LR

END CRYSTAL 12847

BEGIN CRYSTAL 13140

BEGIN AA_SEQUENCE

MKVKKWVTQDFPMVEESATVRECLHRMRQYQTNECIVKDREGHFRGVVNKEDLLDLDLDSSVFNKVSLPD
FFVHEEDNITHALLLFLEHQEPYLPVVDEEMRLKGAVSLHDFLEALIEALAMDVPGIRFSVLLEDKPGEL
RKVVDALALSNINILSVITTRSGDGKREVLIKVDAVDEGTLIKLFESLGIKIESIEKEEGF

END AA_SEQUENCE

BEGIN HA_INFO
ATOM SE
NUMBER_PER_MONOMER 5
END HA_INFO

BEGIN WAVELENGTH INFL
WAVELENGTH 0.979741
F' -10.0
F'' 3.2
END WAVELENGTH INFL

BEGIN WAVELENGTH LREM
WAVELENGTH 1.019859
F' -2.6
F'' 0.55
END WAVELENGTH LREM

BEGIN SWEEP INFL
WAVELENGTH INFL
BEAM 108.7 102.0
IMAGE 13140_1_E1_001.img
DIRECTORY /Volumes/Arthur/JCSG Data/1vr9/data/jcsg/als1/8.3.1/20050105/collect/TM0892/13140/
END SWEEP

BEGIN SWEEP LREM
WAVELENGTH LREM
BEAM 108.7 102.0
IMAGE 13140_1_E2_001.img
DIRECTORY /Volumes/Arthur/JCSG Data/1vr9/data/jcsg/als1/8.3.1/20050105/collect/TM0892/13140/
END SWEEP

END CRYSTAL 13140

END PROJECT TS01
    </pre>

    <p>This is a particularly complex example, but it illustrates the
      capability of the system well. There are data from two cystals, one
      native and the other selenomethionated. The native set is measured in
      two sweeps. Currently (version 0.3.5.1) no attempt is made to
      intelligently combine data from multiple crystals, however this is on
      the to-do list.</p>

    <h1>Release Notes</h1>

    <p>Changes since 0.3.5.2: dealing with huge data sets</p>
    <ul>
      <li>If single sweep, do not rebatch.</li>
      <li>Replace CCP4 Mtzdump with CCTBX-powered equivalent.</li>
      <li>If reindex for changing spacegroup name but not indices
    (i.e. no REINDEX= card) then use CCTBX-powered replacement.</li>
    </ul>

    <p>Changes since 0.3.5.1: CCP4 patch release</p>
    <ul>
      <li>Resolution limit calculation now delegated to resolutionizer.</li>
      <li>Twinning test now uses CCTBX code not sfcheck.</li>
      <li>PythonDriver added to support execution of Python code as separate 
	process.</li>
    </ul>

    <p>Changes since 0.3.5.0</p>
    <ul>
      <li>-hdr_in mode - pass in text file containing image
      header. Can also generate this file with xia2 with -hdr_out example.hdr
      </li>
      <li>-xparm_ub mode for XDS processing only, provide a UB matrix (XDS 
	XPARM file) to match the orientation to: primarily useful if you record
	data from several regions on a sample, and the sample has close-to
	symmetry (i.e. you almost certainly don't need this)</li>
      <li>Removed usage of BINSORT_SCR as this is being deprecated by CCP4.
      </li>
      <li>Allow Phil overrides on selection of autoindexing solutions, where
	indexing can give large variations in unit cell constants (e.g. low
	resolution virus data)</li>
      <li>Properly set the resolution in CAD copying free R flags.</li>
      <li>Allow access to profile grid size through the Phil interface:

	xds.parameter.profile_grid_size = 13, 13
      </li>
      <li>Implement TRUSTED_REGION through Phil as suggested by <NAME>
      </li>
      <li>Updates to process simulated data from SIM_MX</li>
      <li>More measures of anomalous signal in output.</li>
      <li>Corrected citations in output</li>
      <li>Replaced TRUNCATE with CTRUNCATE.</li>
      <li>Replace sfcheck with E^4 calculation from cctbx.</li>
      <li>Exclude grid-scan data when running automatic setup.</li>
      <li>If XDS refinement not performed, exception not warning.</li>
      <li>If -3d and sweep narrower than 10 degrees, use XDS II indexer.</li>
    </ul>

    <p>Changes since 0.3.4.0</p>
    <ul>
      <li>Additional information in output: correct directory for BINSORT_SCR,
	free space in working directory, warns if less than 200MB in 
	BINSORT_SCR directory.</li>
      <li>EXCLUDE keyword added to XINFO SWEEP block to allow
	specific resolution ranges to be excluded in processing,
	contributed by <NAME> at Monash University.</li>
      <li>Include the resolution limits at the XSCALE stage rather
	than simply in the merging.</li>
      <li>Support for 2x2 binned ADSC Quantum 270 detector.</li>
      <li>Support for '.' in file names.</li>
      <li>Support for NOIR1 detector on ALS 4.2.2</li>
      <li>Smarter 3dii mode - if 3di style indexing will give better answer, 
	it will do that...</li>
      <li>Use ctruncate in place of truncate (finally!)</li>
    </ul>

    <p>Changes since 0.3.3.4</p>
    <ul>
      <li>Use Phil to override XDS parameters.</li>
      <li>Use Pointless in place of Reindex; Aimless in place of Scala; no
	limit to number of batches which can be processed.</li>
    </ul>

    <p>Changes since 0.3.3.3</p>
    <ul>
      <li>Fixed bug in polarization correction vector for Pilatus data.</li>
      <li>Fixed bug relating to assigning spacegroup on command-line.</li>
      <li>Begun work on supporting Pilatus 300K for small molecule
        crystallography. Have now successfully processed data from Diamond
        Light Source beamline I19 EH2. N.B. Only works -3dii!</li>
    </ul>

    <p>Changes since 0.3.3.2</p>
    <ul>
      <li>Many bug fixes as a result of implementing a comprehensive testing
        `scheme.</li>
      <li>Support for full cbf format, making use of cbflib Python bindings
        pycbf now included in cctbx releases.</li>
      <li>Merged functionality from XDSIntegraterR into XDSIntegrater, removed
        duplicate code. Will make for more maintainable suite in the future.
      </li>
      <li>Improved 2d pipeline scaling, now tests 8 permutations.</li>
      <li>Improved handling of twinned data for multiple sweeps.</li>
      <li>Fixed occasional bugs with multiple sweeps, reindexing trigonal
        data sets with XDS.</li>
      <li>Removed "old" scaling modes, 2d and 3d, as these were unhelpful.</li>
    </ul>

    <p>Changes since 0.3.3.1</p>
    <ul>
      <li>-xparallel -1 option added which will split your data into 30 degree
        chunks and process with forkintegrate <i>Note well you are responsible
          for setting up forkintegrate correctly!</i>
      </li>
    </ul>

    <p>Changes since 0.3.3.0</p>
    <ul>
      <li>Handle division in updated CCTBX.</li>
      <li>Squash bug with -3d and -resolution.</li>
      <li>Squash bug with resolution and -2d, result slightly different to
        that reported.</li>
      <li>Trap weirdness with XDS reindexing, causes problems sometimes
        trying to process in C2 from P222 not P2 as you would hope!</li>
      <li>Adjustments to cope with excludively narrow sweeps i.e. all 3
        image sweeps.</li>
    </ul>

    <p>Changes since 0.3.2.0</p>
    <ul>
      <li>Support for new XDS build.</li>
      <li>Small bug fix / improvement in XDS indexer implementation.</li>
      <li>Capability to ptovide refined experimental geometry <i>via</i>
        XDS GXPARM.XDS - useful for polycrystal data reduction.</li>
      <li>Now check what you tell the program, viz:
<pre>
gw56@ws050 nonsense]$ xia2 -3d -nonsense /data/gw56/dl/Cowan/Insulin/insulin/
Traceback (most recent call last):
  File "/home/gw56/svn/xia2/Applications/xia2.py", line 45, in
    from xia2setup import write_xinfo
  File "/home/gw56/svn/xia2/Applications/xia2setup.py", line 35, in
    from Handlers.CommandLine import CommandLine
  File "/home/gw56/svn/xia2/Handlers/CommandLine.py", line 1513, in
    CommandLine.setup()
  File "/home/gw56/svn/xia2/Handlers/CommandLine.py", line 349, in setup
    raise RuntimeError, nonsense
RuntimeError: Unknown command-line options: -nonsense
</pre>
        and complain if you type in something xia2 does not understand.
      </li>
      <li>Removed a <i>lot</i> of cruft from the xia2 code base in preparation
        for some more substantial refactoring.</li>
      <li>Found bug which meant that the labelit beam centre computed in the
        setup phase was not used - mostly this is not important, unless you are
        using -3dii.</li>
      <li>Added -ice command-line argument, which will exclude measurements
        from regions which are typically where ice rings land. Will need to
        add more subtle mechanism which will allow specific regions to be
        excluded.</li>
      <li>Specifying resolution with 3d(r) pipeline now works correctly.</li>
      <li>Removed XDS version check by default, which was annoying every new
        year. If you would like to check that the XDS version is explicitly
        supported, add -check_xds_version to the command-line.</li>
      <li>Added table of scala runs to sweep name in the debug output, helpful
        for reviewing in polycrystal data reduction cases.</li>
      <li>Mosaic spread used from Mosflm is now average of all images, not the
        first or last image.</li>
      <li>Fixed header reading for new Pilatus instruments at DLS and
        elsewhere.</li>
      <li>Added capacity to pass in known globally postrefined experimental
        setup via -xparm GXPARM.XDS, only useful with XDS processing
        pipelines.</li>
      <li>Removed use of ccp4 printpeaks tool, which crashes on new pilatus
        images, replace with Mosflm for the moment. Move to replacement with
        labelit code at later stage planned.</li>
    </ul>

    <p>Changes since 0.3.1.7</p>
    <ul>
      <li>Now report the min, mean, max mosaic spread (according to the
        <emph>program definitions</emph> after integration of each sweep.</li>
      <li>Resolved problems with overriding environment.</li>
      <li>Added -serial pun for -parallel 1.</li>
      <li>Allow resolution limit to be assigned on a per-sweep basis.</li>
      <li>Handled corner case with images 200 - 250 (say) giving REBATCH
        error.</li>
      <li>Added full 8-way search for scaling models for 2d pipelines.</li>
      <li>Partial support for two-theta offset data collection (tested with
        Rigaku X-ray equipment and Diamond I19 data.)</li>
      <li>Added -min_images keyword which is helpful when analysing virus data
        with e.g. 4 images / sweep.</li>
    </ul>

    <p>Changes since 0.3.1.6</p>
    <ul>
      <li>Changes suitable for working on microcrystals - reachable through
        a -microcrystal command line flag, and -failover is probably helpful
        too.</li>
      <li>If lattice or spacegroup specified, do not run tests.</li>
      <li>2dr and 3dr now the default pipelines - use 2dold or 3dold if you
        need to get to the old pipelines.</li>
      <li>Fixed selection of known lattice in IDXREF reindexing.</li>
    </ul>

    <p>Changes since 0.3.1.0</p>
    <ul>
      <li>Fixed bug where the resolution limit was not reset in an indexer
        solution elimination.</li>
      <li>Added support for XDS from Dec 09.</li>
      <li>Fixed some of the issues found in the resolution limit determination
        when processing low resolution data.</li>
      <li>Fixed use of I/sigma limit assignment.</li>
      <li>Added command-line control of individual indexer, integrater,
        scaler.</li>
      <li>Repaired regular expression for image matching, to cope with
        images where there is e.g. 3.5 in the template. This was previously
        misinterpreted as an image name of the form setup.NNNN.</li>
      <li>Implemented new resolution limit pipelines, based on new merging code
        to give more robust control: -2dr and -3dr (recommended.) Fine
        control over the choices can be made using the -isigma, -misigma,
        -completeness and -rmerge command line options.</li>
      <li>Added interactive indexing mechanism, where user can assign images
        to use for autoindexing, e.g.:

        <pre>
Existing images for indexing: 1, 90, 180
>1, 60, 120, 180
New images for indexing: 1, 60, 120, 180
        </pre>

        Assign -interactive on the command line (Mosflm and Labelit indexers)
      </li>
    </ul>

    <p>Changes since 0.3.0.6</p>
    <ul>
      <li>Added -xparallel command line option to allow use of forkinitegrate
        on a cluster with the 3d pipeline.</li>
      <li>Allow for xia2html and (beta) ISPyB output (latter more useful for
        synchrotron sources, former handy if you want a report. </li>
      <li>Smart scaling mode switched on by default.</li>
      <li>Chef now run by default, for radiation damage analysis, e.g.
        <pre>
Group 1: 2-wedge data collection
Group 2: Single wedge
Significant radiation damage detected:
Rd analysis (12287/LREM): 15.17
Rd analysis (12287/INFL): 7.89
Rd analysis (12287/PEAK): 12.82
Conclusion: cut off after DOSE ~ 608.8
        </pre>
      </li>
      <li>Blank images now handled more gracefully.</li>
      <li>Overloaded and blank images now correctly reported in the summary
        output, along with abandoned images for the processing with
        Mosflm. Also postrefinement results.</li>
      <li>Substantially cleaned up the program output.</li>
    </ul>

    <p>Changes since 0.3.0.5</p>
    <ul>
      <li>No lattice test mode added (you can guess the command-line option)
        for tricky data sets where this perhaps falls over.</li>
      <li>Fixed side effect of changes which allowed setting of unit cell
        etc. - lattices were no longer eliminated, failed complaining
        can't eliminate only solution.</li>
      <li>Tidied up the generation of output files and so on - now only have
        the main log file and the debug trace.</li>
      <li>Added copy of the command line to program output.</li>
    </ul>

    <p>Changes since 0.3.0.4</p>
    <ul>
      <li>Now able to specify number or fraction of reflections assigned to
        the free set, rather than the default 5 percent.</li>
      <li>Added trap for sdcorrection not being refined in XDS correct if
        multiplicity rather low. Unusual case.</li>
      <li>Assigning a freer_file by definition sets this as an indexing
        reference and also copies the spacegroup assignment. This is
        what you would expect!</li>
    </ul>

    <p>Changes since 0.3.0.3</p>
    <ul>
      <li>User now able to assign cell constants: use with great care, as
        there is little in the way of nonsense trapping. This may however
        be used to handle cases where the solution you want is monoclinic
        with a pseudo-orthorhombic lattice, where the default indexing would
        select a different setting. This will not currently work with a
        Mosflm indexer. Usage is:
<pre>-cell a,b,c,alpha,beta,gamma</pre>

        and the correct symmetry should also be assigned. Can also assign
        USER_CELL a b c alpha beta gamma in the xinfo file.</li>
    </ul>

    <p>Changes since 0.3.0.0</p>
    <ul>
      <li>Now automatically determine the number of available processor
        cores.</li>
      <li>Implemented check for centring of crystallographic basis from
        autoindexing.</li>
      <li>Started to use CCTBX - this will be bundled with the release
        from now on...</li>
      <li>Now runs "smart scaling" i.e. will customise the scaling model
        used to the data. This can seriously improve the xia2 run time in
        cases where some of the default scaling models do not converge.</li>
      <li>Now also includes running of CHEF for radiation damage, which will
        slice and dice your data into dose groups, then run a correctly
        time sequenced radiation damage analysis. N.B. for data measured in
        "dose mode" the doses will be scaled to &lt; 1,000,000 to ensure that
        the output is tidy.</li>
      <li>Corner case of running XDS pipelines from data from a Rigaku setup.
        The low resolution was previously calculated to be 0.0! D'oh!</li>
      <li>Added access to reference reflection file functionality to the
        command line - -reference_reflection_file foo.mtz.</li>
      <li>Fixed the use of the -image command-line option. Now works.</li>
      <li>Removed all of the binaries from the distribution, so now
        <b>you need to be using CCP4 6.1.0 or later!</b></li>
      <li>Will correctly handle reindexing with the 3d pipeline (XDS/XSCALE)
        with trigonal spacegroups and multiple sweeps.</li>
    </ul>

    <p>Changes since 0.2.7.2</p>
    <ul>
      <li>Fixed merging statistics bug - I/sigma output were very slightly
        different.</li>
      <li>Now extend FreeR column if copying from lower resolution input
        file.</li>
      <li>Detector limits now correctly specified around the detector centre
        for Mosflm, rather than the beam centre. Only useful if your beam
        centre is a fair way from the image centre...</li>
      <li>Parallelised integration with Mosflm - divides the sweeps into an
        appropriate number of chunks and then sorts together the resulting
        reflection files. -parallel N, remember.</li>
      <li>Added a putative 'small molecule' mode, which will autoindex with
        Mosflm from a modest number of images, for smaller molecules where
        Labelit is unhappy with the number of good Bragg reflections.</li>
    </ul>

    <p>Changes since 0.2.7.0</p>
    <ul>
      <li>Fixed to work correctly with XDS and Rigaku Saturn/RAXIS
        detectors.</li>
      <li>Better determination of resolution limits.</li>
      <li>Correct merging of data from XSCALE (needs to invert the I=F*F
        scale factor...)</li>
      <li>Fixed output of scalepack unmerged output using 3d pipeline.</li>
      <li>Use fewer frames for background calculation with 3dii pipeline -
        speeds things up a fair amount.</li>
      <li>Write chef output files with correct SD correction parameters.</li>
      <li>Print SD correction factors for scala runs - if these are bigger
        than about 2 there is something properly wrong. Also print the
        same from XDS CORRECT step.</li>
    </ul>

    <p>Changes since 0.2.6.6</p>
    <ul>
      <li>All command line options now echoed to the standard output if
        -debug selected.</li>
      <li>Command line options now copied to .xinfo file when operating in
        automatic mode.</li>
      <li>XDS: low resolution limit determined from spot list from
        IDXREF.</li>
      <li>Now include explicit python version check in xia2 main program.</li>
      <li>High resolution limits for integration now assessed from Wilson
        plot rather than integration program log output.</li>
      <li>If no images are given, a more helpful error message is produced
        than before, viz:
<pre>
------------------------------------------
| No images assigned for crystal DEFAULT |
------------------------------------------
</pre>
      </li>
      <li>If images are not readable (i.e. a permissions problem) then
        warnings will be sent to the debug channel (switch on -debug
        for more information.)</li>
      <li>If in automatic mode an image is given in place of a directory,
        xia2 will now complain in a more helpful way.</li>
      <li>Resolution limits will now be based on analysis of the reflection
        files rather than the program output, since this will generally give
        a more helpful answer...</li>
      <li>Now correctly set the reindex matrix for running correct with
        multiple sweeps of data for non primitive matrix (subtle bug,
        thanks to <NAME> for helping to fix this one.)<li>
      <li>Is images are missing from a sweep, xia2 will now tell you
        this before starting processing rather than giving a strange error -
        if the images are unreadable you will also be told.</li>
    </ul>

    <p>Changes since 0.2.6.5</p>
    <ul>
      <li>Now supports CCP4 6.1 (forthcoming release) though you may need
        to add XIA2_CCP4_61=1 to your environment. This release should include
        xia2.</li>
      <li>Added -user_resolution keyword, also support for this on a
        per-wavelength basis in the xinfo file.</li>
      <li>Added support for reverse-phi data collection - either add
        reversephi to the xinfo file or -reversephi to the command line.</li>
    </ul>

    <p>Changes since 0.2.6.4</p>
    <ul>
      <li>Now handles complex image names with e.g. + in them correctly.</li>
      <li>New option for processing <pre>-3dii</pre> added - this will run
        the 3D pipeline (XDS, XSCALE) using all images for spot picking and
        autoindexing - useful for properly tricky cases.</li>
      <li>Now correctly propogate the profile information to Mosflm when not
        using Mosflm for autoindexing - this is done by running a "fake"
        autoindexing task.</li>
      <li>Use the raster information for the cell refinement test, and changed
        the selection of images for this.</li>
      <li>Added support for the output of BioXHIT XML in the LogFiles
        directory - for project tracking.</li>
      <li>Now copy the XDS log files ("LP" files) into the LogFiles
        directory - as always these are the most up-to-date versions from
        processing.</li>
      <li>Added support for latest release of XDS (June 2, 2008.)</li>
    </ul>

    <p>Changes since 0.2.6.3</p>
    <ul>
      <li>Added possibility to give a reflection file with an existing
        FreeR_flag column, which will be copied to the output MTZ:
        <pre>
xia2 -freer_file free.mtz
        </pre>
or...
        <pre>
FREER_FILE free.mtz
        </pre>
        in the xinfo file (in the same way as the reference reflection file -
        indeed, this may be the same file...)
      </li>
      <li>Added command line control of the spacegroup - note well that this
        means that the data WILL NOT be reindexed to a standard setting.</li>
      <li>Added support for ADSC Quantum 270.</li>
      <li>Intelligent selection of autoindex threshold for Mosflm.</li>
      <li>Added support for latest XDS version.</li>
      <li>Added support for pilatus 6M mini CBF images.</li>
      <li>Fixed problem with cell refinement giving negative mosaic
        spreads sometimes for 2d pipeline.</li>
    </ul>

    <p>Changes since 0.2.6.2</p>
    <ul>
      <li>Now works for images with long image numbers, as typically
        recorded on Rigaku X-Ray sets.</li>
      <li>Signed off for operation with Rigaku Saturn and RAXIS IV
        detectors.</li>
      <li>Now works with no input, e.g.
        <pre>
xia2 -project TG6623 -crystal X77788 -atom se /my/images/are/here
        </pre>
        However this relies on your image headers being accurate and the
        images having some kind of recognisable format...
      </li>
      <li>Repaired operation on ppc and intel macs - added required libraries
        to the installation and reset the library paths appropriately.</li>
      <li>Slightly improved error reporting from xinfo file errors.</li>
    </ul>


    <p>Changes since 0.2.6.1</p>
    <ul>
      <li>Now carefully select the images to use for cell refinement based
        on the orientation of the crystal lattice.</li>
      <li>Fixed numerous bugs to do with the naming of the detector class
        which changed in the previous version.</li>
    </ul>

    <p>Changes since 0.2.6.0</p>

    <ul>
      <li>Fixed bug where if the distance was read incorrectly from the
        image header (or was wrong therein) XDS would get the wrong value
        even if you had put the correct value in the .xinfo file.</li>
      <li>Now "unroll" the unmerged reflections from XSCALE and then
        merge them in their original sweeps in Scala. This should give
        a useful Rmerge vs. time plot.</li>
      <li>Including updated versions of Pointless and Scala.</li>
      <li>Includes new and more robust handling of pointgroups, lattices
        and unstable refinement of parameters during integration.</li>
      <li>Include support for a reference reflection file, which will
        ensure that the reflections are indexed in the same way and
        with the same spacegroup - useful for mutants.</li>
      <li>Now always use three wedges of images for the cell refinement
        with Mosflm as this makes the lattice elimination more reliable.
      </li>
      <li>If one of the images in a sweep is broken (e.g. corrupted)
        xia2 will tell you more helpfully rather than just crashing.</li>
      <li>Now correctly read distance from Mar 165 images. Thanks to
        <NAME> for this fix to DiffractionImage.</li>
      <li>Fixed the use of xia2 -3d with RAXIS IV detectors.</li>
      <li>Use more images for Mosflm autoindexing (three instead of two)
        as this gives uniformly better results.</li>
    </ul>

    <p>Changes since 0.2.5.2</p>

    <ul>
      <li>Now works for highly incomplete data sets, so long as you
        have either 4 times the mosaic spread or 8 images, whichever
        is the larger. This is to allow pointgroup identification
        from e.g. the first 10 images in a data set - designed to assist
        in synchrotron / lab source data collection.</li>
      <li>Now include the latest Mosflm binaries in the extras package -
        these are often better than those in the CCP4 distribution,
        as they come directly from <NAME>'s web page.</li>
      <li>Added -ehtpx_xml_out option which will write out marked
        up metadata about the data reduction, for inclusion in the
        e-HTPX data reduction portal and, perhaps, other automated
        systems. Usage:
        <pre>
xia2 -ehtpx_xml_out project.xml
        </pre>
      </li>
      <li>If data are very incomplete (e.g. less than 50% complete) then don't
        try to refine the error parameters. This is both incorrect and a
        waste of time.</li>
      <li>Renamed the output reflection files - these are now in DataFiles
        directory with names like "PROJECT_CRYSTAL_free.mtz." This makes
        them a little easier to identify.</li>
      <li>Harvesting (deposition) files now in subdirectory of Harvest
        rather than being spread around the place. These will have
        names Harvest/DepositFiles/PROJECT/WAVELENGTH.scala etc.</li>
      <li><b>Have now fixed xia2setup so it works much more sensibly -
          the decision about when a new sweep starts was a little broken
          (rounding errors issue) now fixed!</b></li>
      <li>Added -parallel flag to work when using XDS for data reduction.</li>
      <li><b>Added XDS support!</b> This is however much less mature than
        the support for Mosflm/CCP4. It is also worth noting that this
        relies on many CCP4 tools.</li>
      <li>Added selection of the "pipeline" to use through a command line
        option, either -2d for mosflm/ccp4 or -3d for xds/xscale though
        the latter still uses a number of CCP4 programs.</li>
    </ul>

    <p>Changes since 0.2.5.1 - big changes in <b>bold</b></p>
    <ul>
      <li><b>Process data as native if no heavy atom information, only one
          wavelength specified and no anomalous scattering form factors
          provided. Otherwise separate anomalous pairs for scaling.</b></li>
      <li><b>Added a -quick option to the xia2 command line, which will
          cut out many of the data reduction optimisation steps (not
          refine detector parameters, resolution or scaling parameters)
          but will still include everything else. This is to allow quick
          characterisation of data at the beamline, and perhaps map
          calculation?</b></li>
      <li>Correctly print the unmerged scalepack file name for single
        wavelength data.</li>
      <li>Perform radiation damage analysis using R merge and B factor
        as a function of <i>time</i> - note that this is not <i>batch.</i>
        This is likely to have interesting side-effects.</li>
      <li>If time of data collection not recorded in header <i>but</i>
        the time stamp for the images are correct (e.g. you have not moved
        them since collection) then the command line flag
        <pre>-trust_timestamps</pre> can be used, which will use the
        time stamps on the image files to analyse things like radiation
        damage.</li>
      <li>If you are running this on a cluster, have added an option
        to migrate the diffraction data to a local disk (e.g. /tmp.)
        To do this add <pre>-migrate_data</pre> to the command line -
        the data will be removed from the local disk once the processing
        is finished.</li>
      <li>Copy all final reflection files to a DataFiles directory.</li>
      <li>Now, when something goes horribly wrong, just write the actual
        message to the screen and the horrible stack trace gunk to a file
        called xia2.error. This should be sent in for error reports!</li>
      <li>Remerge individual wavelengths to get the merging statistics to
        the most appropriate resolution limit, rather than the furthest
        extent of any data, for the summary table.</li>
      <li>If GAIN included in SWEEP block, will be used as the default
        value for integration. E.g. GAIN 0.25.</li>
    </ul>

    <p>Changes since 0.2.5</p>
    <ul>
      <li>Fixed problem with spacegroup R3:H (naming convention problem
        - there's a surprise.)</li>
      <li>GAIN estimation temporarily removed.</li>
      <li>Pointgroup evaluation fixed - found a <b>major</b> gremlin in
        there</li>
      <li>Allow environment variables and tilde in DIRECTORY token:
        <pre>DIRECTORY $DATA/example or ~/data or %DATA%/example (win32)</pre>
      </li>
      <li>Now tracks the citations for the programs used, writing them
        to the standard output in plain text and to xia-citations.bib
        in BIBTeX format.</li>
      <li>Removed [XIA2] tokens from standard output - the reason for
        having this there no longer applies.</li>
    </ul>


    <p>Changes since 0.2.4</p>

    <ul>
      <li>xia2setup will now add the f', f'' values if a scan is available
        and has been processed by chooch</li>
      <li>GAIN now estimated by diffdump - enabling future parallelisation
        of the integration stage.</li>
      <li>For detectors formed as a mosaic of a number of tiles, with "gaps"
        in between, now mask those gapped areas in Mosflm to reduce the
        number of bad reflections.</li>
      <li>Now has a fix for the ongoing indexing problem which gave rise
        to errors like "something horrible has happened in indexing".</li>
      <li>The "best" log file for each process will now be recorded in a
        LogFiles directory - which means no more hunting around for the
        final scala log file and what not.</li>
      <li>Chaling now produces unmerged reflection files in scalepack format
        as well as merged scalepack and merged MTZ format. This is done by
        scala (if used) by recycling the final SCALES and adding
        "output polish unmerged."</li>
      <li>Now runs a twinning test using the CCP4 program sfcheck, and
        will warn you if your data look twinned.</li>
      <li>Now allow up to 100 cycles of scale refinement to cope with
        more extreme cases where data at very different resolutions
        are scaled together.</li>
    </ul>

    <p>Changes since 0.2.3</p>
    <ul>
      <!-- <li>Added support for XDS</li> -->
      <li>Added <a href="preferences.html">preferences file</a></li>
      <li>Added xia2setup program to create the .xinfo file - this will
        also run LABELIT to configure the beam centre if it is installed</li>
      <li>Added a strategy-of-data-reduction step to the pipeline</li>
      <li>Added run-time check that CCP4 and so on are available</li>
    </ul>

    <p>Changes since 0.2.2.4:</p>

    <ul>
      <li>Changed to a BSD license</li>
      <li>Added "python setup" check to xia2</li>
      <li>Relaxed criteria on isomorphism to 1% not 0.5A etc.</li>
      <li><b>Now wavelengths in wavelength record will override header
        values if provided, but do not (indeed, should not) be included
        if wavelength values are correct.</b></li>
    </ul>

    <p>Changes since 0.2.2.3:</p>

    <ul>
      <li>Created version for Power PC G4/G5</li>
      <li>Fixed .csh setup scripts</li>
      <li>If wavelength values not specified in the .xinfo file will use
        values from the image headers</li>
      <li>If loggraph output from scala is broken by having Mn(I/sigma)
        greater than 99.99 can now cope (ignores that record)</li>
      <li>Will now use a maximum of 180 degrees of data for deciding
        e.g. the point group, spacegroup and correct indexing standards -
        this helps the cases where exceptionally high redundancy data
        has been measured</li>
    </ul>

    <p>Changes since 0.2.2.2:</p>

    <ul>
      <li>Updated printheader to cope with MAR 165</li>
      <li>Cleaner error messages when data missing</li>
      <li>Added support for MAR image plate data</li>
      <li>Included more example .xinfo files in the examples directory</li>
      <li>Relaxed refinement parameters in Mosflm cell refinement</li>
    </ul>

    <h1>Platform Specific Advice</h1>

    <h2>Gfortran compiled CCP4</h2>

    <p>Recent versions of the gcc compiler suite have included a new compiler
      called "gfortran" in place of "g77". This has, unfortunately, been
      designed with buffering too high on the agenda, and the output
      from programs can sometimes be mashed. If this happens you need to
      set "GFORTRAN_UNBUFFERED_ALL" to 1 in your environment.</p>

    <h2>Linux</h2>

    <p>There should be no particular problems on this platform - assuming
      that you have a recent version of Python. Any 2.4 version should work,
      but I have mostly used 2.4.3 and 2.4.4, and would recommend them.
      If the version you have on your system by default (type "python"
      to find out what version you have)is too old, you can easily install
      a new version, perhaps in the xia directory, from
      <a href="http://www.python.org/download/releases/2.4.4/">here</a>.
      This is best installed by unpacking the tarball (tar xvfz
      Python-2.4.4.tgz) then doing ./configure /where/I/want/it then
      make, make install.</p>

    <h2>OS X.4 - X.6</h2>

    <p>The only potential problem with this platform is that the Python
      which comes as standard is 2.3 - however since you need cctbx.python
      anyway the problem has gone away.</p>

    <h2>Windows XP</h2>

    <p>Once more you will need a recent version of Python - which can be found
      <a href="http://www.python.org/download/releases/2.4.4/">here</a>,
      however, you will also need "pywin32" to provide a little more
      job control functionality - this is available from
      <a href="http://sourceforge.net/projects/pywin32/">here</a>. Both are
      simple, double-click, binary installers.</p>
    <p>There is now a version of Labelit available for Windows, which
      can be found <a href="http://adder.lbl.gov/labelit/">here.</a></p>
    <p>The mosflm binary which comes with
      <a href="http://www.mrc-lmb.cam.ac.uk/harry/imosflm/index.html">
        iMosflm</a> works well with xia2.</p>

  </body>
</html>

{'fraction_non_alphanumeric': 0.08004565224096981, 'fraction_numerical': 0.01718128258120238, 'mean_word_length': 2.7568757725587143, 'pattern_counts': {'":': 0, '<': 1756, '<?xml version=': 0, '>': 1758, 'https://': 2, 'lorem ipsum': 0, 'www.': 6, 'xml': 6}, 'pii_count': 78, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1821138', 'n_tokens_mistral': 29890, 'n_tokens_neox': 27999, 'n_words': 12483}