<commit_before>Pre 1.0 release:

Implement sparse iteration through values for BP [McCallum, Singh]
Redesign app.classify for better access to underlying optimize goodness. [McCallum, Passos]
In BP, for factors that have the same set of varying neighbors, cache the marginal distribution
Improve LDA speed and multi-threading [Vineet]
Packaged POS, NER, DepParser, Coref [Andrew]
Consider removing the implicit DiffList from the arguments to the TokenSpan constructor [Andrew]
Consider again making Mentions inherit from TokenSpan, with the option to simply initialize from a TokenSpan constructor argument [Andrew] [Passos: please don't!]
Rename ClearTokenizer and ClearSegmenter [Andrew]
Make DocumentAnnotator take lazy values [Andrew]
Standardize NER2 [Andrew]

Finish app.classify.Classifier command-line tool [McCallum, Vilnis]
Finish app.chain.Chain command-line tool [McCallum, Vilnis]
Wrap-up and clean up Parser, NER and POS  [Belanger]
Consider changes to generative.Collapse infrastructure  [McCallum]
Implement generative.CollapsedVariationalBayes and make sure it works for LDA  [McCallum]
Look at app.nlp.relation  [McCallum]
Make reasonable command-line tools for MALLET functionality, and document  [McCallum and others]
Sanity-check pass through all code [All]
Package documentation  [All]
Fix copyright year [McCallum]

More examples:
Tutorial examples [McCallum]
Clean up old examples
Basic models [All]
Implement TopicsOverTime (needs Beta distribution) [Bakalov]
Make an HMM example and show that BP inference and EM training work on it [Belanger]
Fixed HACKING.txt Eclipse set up

Tutorial coverage:
000 Introduction: Motivation, overview, first quick examples for feel, comparison with similar tools
010 Variables, Domains, Assignments, Proportions
020 Tensors and la package
030 Factors, Families, Templates, Models
040 Infer, Summary, Marginal, BP, Sampling, GibbsSampler, MHSampler, MeanField, MLPL, DualDecomposition 
050 Optimize package 
060 Learning
070 Directed package
080 Standard ML models: HMM, Mixture of Gaussians,
090 Parallelism, Hyperparameter optimization [DONE]
100 Classify package
110 Chain package
120 NLP package, Document, Section, Token, TokenSpan, Sentence, attr, DocumentAnnotator, pos, ner, mention, parse, lexicon, wordnet, coref
130 Topic modeling package
140 Serialization and Cubbies and Mongo

Missing?
Regression, Matrix Factorization,



DONE:
Write JUnit tests for cc.factorie.la [Vilnis?]
 Consider which println's should be transformed into Logging statements
Finish lexicon data loading management. [McCallum]
In BP, when running on chains, verify that we don't need to walk to create a tree.
Verify ChainModel actually works [Passos?,Martin]
Code review BP.  Make it fast!  [Passos]
Implement dual decomposition. [Passos]
Learning-supporting Sampler extends Infer [Passos]
No holes in Tensor dot products [Passos]
Make Tensor.foreachActiveElement faster with a macro.  After 1.0 or perhaps never. [Luke]
Strongly consider removing type argument from Domain.  It isn't needed for anything.  Decided against for now. [McCallum]
Add something like Passage to Document, indexed by char offsets [McCallum]
Remove VarAndValueGenericDomain [McCallum]
Remove unused classes from cc.factorie.util [Luke]
Reconsider naming DiscreteDimensionTensorVar and DiscreteDimensionCategoricalVar to something shorter? [McCallum]
Make Assignment inherit from Summary [Passos]  (No, just make a wrapper class.)
Substitute alternative lapack library [Alex] (Not done.)
Look at Model, Tensor, Domain serialization [Vilnis?]
Liblinear-like L1 SVM... [Martin]
Polish Piece and OptimizationDriver (Renamed Example and Trainer) [Passos, Vilnis, McCallum]
Consider Model[Context].  (Decided on ModelWithContext[Context] instead.) [McCallum]
HashDomain [Martin]
Consider renaming LabelVariable to LabeledCategoricalVariable, and having LabeledDiscreteVariable, LabeledBooleanVariable [McCallum]
DomainFromClass or make it the default?  No.  [McCallum]
Re-consider type arguments in Sampler and descendants?  No.  [McCallum]
Consider property-like facilities and its interaction with command line [McCallum, Vilnis]
 No, use scala interpreter itself for configuration as recommended online.
Consider Statistics -> Statistic, and then also having Statistic1, Statistic2, etc.
Make app.regress  [McCallum, Passos]
Make DotFamily can depend on just TensorVar, not DiscreteTensorVar.
Remove er package altogether for now [McCallum]
Remove statisticsDomain and let DotFamily.weights be abstract  [McCallum]  
Make BooleanValue be a class rather than (as it is currently) a type alias for CategoricalValue[Boolean] [McCallum]
Use _1, _2 again for the variables in AbstractAssignment2, etc.  Use value1, value2 for values. [McCallum]
Rename all "Stat" to "Statistics". [McCallum]
Finish/fix DecisionTree implementation, and make example
Remove Factor.Values
Make RealVar inherit from TensorVar (with DoubleVar replacing current one)
Make a Family.score(Tensor) method for use in higher efficiency situations (e.g. BP)
Consider moving generative.GibbsSampler up to cc.factorie?
Consider making no variable inherit from Iterable.  (Seq would come from value instead.)  Remove SeqEqualsEq and friends.
 Consider making any current Variable that *could* have more than one different value type be not a Variable.  
 Then allow Model.factors and Infer and Maximize to have apply(Iterable[Variable]) and Apply(Variable).
 Then, e.g. change "for (token <- document)" to "for (token <- document.tokens)". [done]
Re-consider if variable.value should always make an immutable copy.  [no]
Look at Gaussian.logpr, Beta.logpr, etc, and ensure these scores are correct.  [yes]
Get rid of Seq[Double] everywhere, including cc.factorie.maths.  Get rid of IndexedSeqOps completely.
Get rid of la.Vector and use la.Tensor instead.  Make corresponding changes in Factors/Templates, etc.
Make VectorFamily require "def statisticsDomain"
Consider making cc.factorie.optimize take DoubleSeq instead of Array[Double]
Get AROW Perceptron training working, and look at SampleRank interaction
Get rid of dependency on Java bibtex parser library
Implement parameter averaging

Future:
Implement beam search for BP.
Make LDA faster and multi-threaded [Bakalov]
Look at other BUGS-like tools, and see what we can reproduce easily [Bakalov, Duckworth]
Sebastian: Make new kinds of Templates; current one: factors(Variable neighbor); new: factors(arbitrary context)
 Consider naming the current Template something more specific, e.g. "NeighborTemplate"
 Template[C] { def factors(context:C): Iterable[Factors] }
 NeighborTemplate extends Template[Variable] { def unroll1(v:Variable): Iterable[Factors] }




Low Priority Coding:
Replace cc.factorie.er with something better that uses Scala 2.10 macros.
-Implement Forany2 in er
-Fix IntTerm in er
-Support inference over relations in er
Create simple infrastructure for non-relational generative Bayes nets

Testing:
Create more unit tests!
Profile everything, especially BP and LDA.

Documentation:
Clean up all examples
Put in ACE coref example<commit_msg>adding TODO to implement AROW/CW<commit_after>Pre 1.0 release:

Implement sparse iteration through values for BP [McCallum, Singh]
Redesign app.classify for better access to underlying optimize goodness. [McCallum, Passos]
In BP, for factors that have the same set of varying neighbors, cache the marginal distribution
Improve LDA speed and multi-threading [Vineet]
Packaged POS, NER, DepParser, Coref [Andrew]
Consider removing the implicit DiffList from the arguments to the TokenSpan constructor [Andrew]
Consider again making Mentions inherit from TokenSpan, with the option to simply initialize from a TokenSpan constructor argument [Andrew] [Passos: please don't!]
Rename ClearTokenizer and ClearSegmenter [Andrew]
Make DocumentAnnotator take lazy values [Andrew]
Standardize NER2 [Andrew]

Finish app.classify.Classifier command-line tool [McCallum, Vilnis]
Finish app.chain.Chain command-line tool [McCallum, Vilnis]
Wrap-up and clean up Parser, NER and POS  [Belanger]
Consider changes to generative.Collapse infrastructure  [McCallum]
Implement generative.CollapsedVariationalBayes and make sure it works for LDA  [McCallum]
Look at app.nlp.relation  [McCallum]
Make reasonable command-line tools for MALLET functionality, and document  [McCallum and others]
Sanity-check pass through all code [All]
Package documentation  [All]
Fix copyright year [McCallum]

More examples:
Tutorial examples [McCallum]
Clean up old examples
Basic models [All]
Implement TopicsOverTime (needs Beta distribution) [Bakalov]
Make an HMM example and show that BP inference and EM training work on it [Belanger]
Fixed HACKING.txt Eclipse set up

Tutorial coverage:
000 Introduction: Motivation, overview, first quick examples for feel, comparison with similar tools
010 Variables, Domains, Assignments, Proportions
020 Tensors and la package
030 Factors, Families, Templates, Models
040 Infer, Summary, Marginal, BP, Sampling, GibbsSampler, MHSampler, MeanField, MLPL, DualDecomposition 
050 Optimize package 
060 Learning
070 Directed package
080 Standard ML models: HMM, Mixture of Gaussians,
090 Parallelism, Hyperparameter optimization [DONE]
100 Classify package
110 Chain package
120 NLP package, Document, Section, Token, TokenSpan, Sentence, attr, DocumentAnnotator, pos, ner, mention, parse, lexicon, wordnet, coref
130 Topic modeling package
140 Serialization and Cubbies and Mongo

Missing?
Regression, Matrix Factorization,



DONE:
Write JUnit tests for cc.factorie.la [Vilnis?]
 Consider which println's should be transformed into Logging statements
Finish lexicon data loading management. [McCallum]
In BP, when running on chains, verify that we don't need to walk to create a tree.
Verify ChainModel actually works [Passos?,Martin]
Code review BP.  Make it fast!  [Passos]
Implement dual decomposition. [Passos]
Learning-supporting Sampler extends Infer [Passos]
No holes in Tensor dot products [Passos]
Make Tensor.foreachActiveElement faster with a macro.  After 1.0 or perhaps never. [Luke]
Strongly consider removing type argument from Domain.  It isn't needed for anything.  Decided against for now. [McCallum]
Add something like Passage to Document, indexed by char offsets [McCallum]
Remove VarAndValueGenericDomain [McCallum]
Remove unused classes from cc.factorie.util [Luke]
Reconsider naming DiscreteDimensionTensorVar and DiscreteDimensionCategoricalVar to something shorter? [McCallum]
Make Assignment inherit from Summary [Passos]  (No, just make a wrapper class.)
Substitute alternative lapack library [Alex] (Not done.)
Look at Model, Tensor, Domain serialization [Vilnis?]
Liblinear-like L1 SVM... [Martin]
Polish Piece and OptimizationDriver (Renamed Example and Trainer) [Passos, Vilnis, McCallum]
Consider Model[Context].  (Decided on ModelWithContext[Context] instead.) [McCallum]
HashDomain [Martin]
Consider renaming LabelVariable to LabeledCategoricalVariable, and having LabeledDiscreteVariable, LabeledBooleanVariable [McCallum]
DomainFromClass or make it the default?  No.  [McCallum]
Re-consider type arguments in Sampler and descendants?  No.  [McCallum]
Consider property-like facilities and its interaction with command line [McCallum, Vilnis]
 No, use scala interpreter itself for configuration as recommended online.
Consider Statistics -> Statistic, and then also having Statistic1, Statistic2, etc.
Make app.regress  [McCallum, Passos]
Make DotFamily can depend on just TensorVar, not DiscreteTensorVar.
Remove er package altogether for now [McCallum]
Remove statisticsDomain and let DotFamily.weights be abstract  [McCallum]  
Make BooleanValue be a class rather than (as it is currently) a type alias for CategoricalValue[Boolean] [McCallum]
Use _1, _2 again for the variables in AbstractAssignment2, etc.  Use value1, value2 for values. [McCallum]
Rename all "Stat" to "Statistics". [McCallum]
Finish/fix DecisionTree implementation, and make example
Remove Factor.Values
Make RealVar inherit from TensorVar (with DoubleVar replacing current one)
Make a Family.score(Tensor) method for use in higher efficiency situations (e.g. BP)
Consider moving generative.GibbsSampler up to cc.factorie?
Consider making no variable inherit from Iterable.  (Seq would come from value instead.)  Remove SeqEqualsEq and friends.
 Consider making any current Variable that *could* have more than one different value type be not a Variable.  
 Then allow Model.factors and Infer and Maximize to have apply(Iterable[Variable]) and Apply(Variable).
 Then, e.g. change "for (token <- document)" to "for (token <- document.tokens)". [done]
Re-consider if variable.value should always make an immutable copy.  [no]
Look at Gaussian.logpr, Beta.logpr, etc, and ensure these scores are correct.  [yes]
Get rid of Seq[Double] everywhere, including cc.factorie.maths.  Get rid of IndexedSeqOps completely.
Get rid of la.Vector and use la.Tensor instead.  Make corresponding changes in Factors/Templates, etc.
Make VectorFamily require "def statisticsDomain"
Consider making cc.factorie.optimize take DoubleSeq instead of Array[Double]
Get AROW Perceptron training working, and look at SampleRank interaction
Get rid of dependency on Java bibtex parser library
Implement parameter averaging

Future:
Implement beam search for BP.
Make LDA faster and multi-threaded [Bakalov]
Look at other BUGS-like tools, and see what we can reproduce easily [Bakalov, Duckworth]
Sebastian: Make new kinds of Templates; current one: factors(Variable neighbor); new: factors(arbitrary context)
 Consider naming the current Template something more specific, e.g. "NeighborTemplate"
 Template[C] { def factors(context:C): Iterable[Factors] }
 NeighborTemplate extends Template[Variable] { def unroll1(v:Variable): Iterable[Factors] }




Low Priority Coding:
Reimplement AROW and CW
Replace cc.factorie.er with something better that uses Scala 2.10 macros.
-Implement Forany2 in er
-Fix IntTerm in er
-Support inference over relations in er
Create simple infrastructure for non-relational generative Bayes nets

Testing:
Create more unit tests!
Profile everything, especially BP and LDA.

Documentation:
Clean up all examples
Put in ACE coref example
{'fraction_non_alphanumeric': 0.05930240718646922, 'fraction_numerical': 0.008842725805319671, 'mean_word_length': 5.677600749765698, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5224697', 'n_tokens_mistral': 4063, 'n_tokens_neox': 3781, 'n_words': 1799}