<commit_before>Apache Tez Change Log
=====================


Release 0.7.0: Unreleased

INCOMPATIBLE CHANGES

ALL CHANGES:
  TEZ-1941. Memory provided by *Context.getAvailableMemory needs to be setup explicitly.
  TEZ-1879. Create local UGI instances for each task and the AM, when running in LocalMode.
  TEZ-1661. LocalTaskScheduler hangs when shutdown.
  TEZ-1934. TestAMRecovery may fail due to the execution order is not determined.
  TEZ-1951. Fix general findbugs warnings in tez-dag.
  TEZ-1905. Fix findbugs warnings in tez-tests.
  TEZ-1945. Remove 2 GB memlimit restriction in MergeManager.
  TEZ-1913. Reduce deserialize cost in ValuesIterator.
  TEZ-1917. Examples should extend TezExampleBase.
  TEZ-1892. Add hashCode and equals for Vertex/VertexGroup/Edge/GroupInputEdge.
  TEZ-1904. Fix findbugs warnings in tez-runtime-library module.
  TEZ-1903. Fix findbugs warnings in tez-runtime-internal module.
  TEZ-1896. Move the default heartbeat timeout and checkinterval to TezConfiguration.
  TEZ-1923. FetcherOrderedGrouped gets into infinite loop due to memory pressure.
  TEZ-1274. Remove Key/Value type checks in IFile.
  TEZ-1912. Merge exceptions are thrown when enabling tez.runtime.shuffle.memory-to-memory.enable && tez.runtime.shuffle.memory-to-memory.segments.
  TEZ-1922. Fix comments: add UNSORTED_OUTPUT to TEZ_TASK_SCALE_MEMORY_WEIGHTED_RATIOS.
  TEZ-485. Get rid of TezTaskStatus.
  TEZ-1899. Fix findbugs warnings in tez-common module.
  TEZ-1898. Fix findbugs warnings in tez-api module.
  TEZ-1906. Fix findbugs warnings in tez-yarn-timeline-history-with-acls.
  TEZ-1767. Enable RLE in reducer side merge codepath.
  TEZ-1837. Restrict usage of Environment variables to main methods.
  TEZ-1867. Create new central dispatcher for Tez AM
  TEZ-1844. Shouldn't invoke system.exit in local mode when AM is failed to start.
  TEZ-1889. Fix test-patch to provide correct findbugs report.
  TEZ-1313. Setup pre-commit build to test submitted patches.
  TEZ-1856. Remove LocalOnFileSortedOutput, LocalMergedInput, LocalTaskOutputFiles.
  TEZ-1949. Whitelist TEZ_RUNTIME_OPTIMIZE_SHARED_FETCH for broadcast edges.
  TEZ-1593. Refactor PipelinedSorter to remove all MMAP based ByteBuffer references.

Release 0.6.0: Unreleased

INCOMPATIBLE CHANGES

ALL CHANGES:
  TEZ-1968. Tez UI - All vertices of DAG are not listed in vertices page
  TEZ-1890. tez-ui web.tar.gz also being uploaded to maven repository
  TEZ-1931. Publish tez version info to Timeline.
  TEZ-1938. Build warning duplicate jersey-json definitions
  TEZ-1910. Build fails against hadoop-2.2.0.
  TEZ-1882. Tez UI build does not work on Windows
  TEZ-1915. Add public key to KEYS
  TEZ-1907. Fix javadoc warnings in tez codebase
  TEZ-1891. Incorrect number of Javadoc warnings reported
  TEZ-1762. Lots of unit tests do not have timeout parameter set.
  TEZ-1886. remove deprecation warnings for tez-ui on the console.
  TEZ-1875. dropdown filters do not work on vertices and task attempts page.
  TEZ-1873. TestTezAMRMClient fails due to host resolution timing out.
  TEZ-1881. Setup initial test-patch script for TEZ-1313.
  TEZ-1864. move initialization code dependent on config params to App.ready.
  TEZ-1870. Time displayed in the UI is in GMT.
  TEZ-1858. Docs for deploying/using the Tez UI.
  TEZ-1642. TestAMRecovery sometimes fail.
  TEZ-1859. TestGroupedSplits has commented out test: testGzip.
  TEZ-1868. Document how to do Windows builds due to with ACL symlink build changes.
  TEZ-1872. docs/src/site/custom/project-info-report.properties needs license header.
  TEZ-1850. Enable deploy for tez-ui war.
  TEZ-1841. Remove range versions for dependencies in tez-ui.
  TEZ-1854. Failing tests due to host resolution timing out.
  TEZ-1860. mvn apache-rat:check broken for tez-ui.
  TEZ-1866. remove the "original" directory under tez-ui
  TEZ-1591. Add multiDAG session test and move TestLocalMode to tez-tests
  TEZ-1769. ContainerCompletedWhileRunningTransition should inherit from TerminatedWhileRunningTransition
  TEZ-1849. Fix tez-ui war file licensing.
  TEZ-1840. Document TezTaskOutput.
  TEZ-1576. Class level comment in {{MiniTezCluster}} ends abruptly.
  TEZ-1838. tez-ui/src/main/webapp/bower.json gets updated after compiling source code.
  TEZ-1789. Move speculator processing off the central dispatcher.
  TEZ-1610. Add additional task counters for fetchers, merger.
  TEZ-1847. Fix package name for MiniTezClusterWithTimeline.
  TEZ-1846. Build fails with package org.apache.tez.dag.history.logging.ats does not exist.
  TEZ-1696. Make Tez use the domain-based timeline ACLs.
  TEZ-1835. TestFaultTolerance#testRandomFailingTasks is timing out
  TEZ-1832. TestSecureShuffle fails with NoClassDefFoundError: org/bouncycastle/x509/X509V1CertificateGenerator
  TEZ-1672. Update jetty to use stable 7.x version - 7.6.16.v20140903.
  TEZ-1822. Docs for Timeline/ACLs/HistoryText.
  TEZ-1252. Change wording on http://tez.apache.org/team-list.html related to member confusion.
  TEZ-1805. Tez client DAG cycle detection should detect self loops
  TEZ-1816. It is possible to receive START event when DAG is failed
  TEZ-1787. Counters for speculation
  TEZ-1773. Add attempt failure cause enum to the attempt failed/killed
  history record
  TEZ-14. Support MR like speculation capabilities based on latency deviation
  from the mean
  TEZ-1733. TezMerger should sort FileChunks on size when merging
  TEZ-1738. Tez tfile parser for log parsing
  TEZ-1627. Remove OUTPUT_CONSUMABLE and related Event in TaskAttemptImpl
  TEZ-1736. Add support for Inputs/Outputs in runtime-library to generate history text data.
  TEZ-1721. Update INSTALL instructions for clarifying tez client jars
    compatibility with runtime tarball on HDFS.
  TEZ-1690. TestMultiMRInput tests fail because of user collisions.
  TEZ-1687. Use logIdentifier of Vertex for logging.
  TEZ-1737. Should add taskNum in VertexFinishedEvent.
  TEZ-1772. Failing tests post TEZ-1737.
  TEZ-1785. Remove unused snappy-java dependency.
  TEZ-1685. Remove YARNMaster which is never used.
  TEZ-1797. Create necessary content for Tez DOAP file.
  TEZ-1650. Please create a DOAP file for your TLP.
  TEZ-1697. DAG submission fails if a local resource added is already part of tez.lib.uris
  TEZ-1060 Add randomness to fault tolerance tests
  TEZ-1790. DeallocationTaskRequest may been handled before corresponding AllocationTaskRequest in local mode

TEZ-UI CHANGES (TEZ-8):
  TEZ-1799. Enable Cross Origin Support in Tez UI
  TEZ-1634. Fix compressed IFile shuffle errors
  TEZ-1615. Skeleton framework for Tez UI
  TEZ-1604. Task View for Tez UI
  TEZ-1603. Vertex View for Tez UI.
  TEZ-1720. Allow filters in all tables and also to pass in filters using url params.
  TEZ-1708. Make UI part of TEZ build process.
  TEZ-1617. Shim layer for Tez UI for use within Ambari.
  TEZ-1741. App view.
  TEZ-1751. Log view & download links in task and task attempt view.
  TEZ-1753. Queue in dags view.
  TEZ-1765. Allow dropdown lists in table filters.
  TEZ-1606. Counters View for DAG, Vertex, and Task.
  TEZ-1768. follow up jira to address minor issues in Tez-ui.
  TEZ-1783. Wrapper in standalone mode.
  TEZ-1820. Fix wrong links.

Release 0.5.4: Unreleased

ALL CHANGES:
  TEZ-1942. Number of tasks show in Tez UI with auto-reduce parallelism is misleading.
  TEZ-1962. Fix a thread leak in LocalMode.
  TEZ-1924. Tez AM does not register with AM with full FQDN causing jobs
  to fail in some environments.
  TEZ-1878. Task-specific log level override not working in certain conditions.
  TEZ-1775. Allow setting log level per logger.
  TEZ-1851. FileSystem counters do not differentiate between different FileSystems.
  TEZ-1852. Get examples to work in LocalMode.
  TEZ-1861. Fix failing test: TestOnFileSortedOutput.
  TEZ-1836. Provide better error messages when tez.runtime.io.sort.mb, spill percentage is incorrectly configured.
  TEZ-1800. Integer overflow in ExternalSorter.getInitialMemoryRequirement()

Release 0.5.3: 2014-12-10

ALL CHANGES:<commit_msg>TEZ-1931. Update CHANGES.txt to reflect final fix version.<commit_after>Apache Tez Change Log
=====================


Release 0.7.0: Unreleased

INCOMPATIBLE CHANGES

ALL CHANGES:
  TEZ-1941. Memory provided by *Context.getAvailableMemory needs to be setup explicitly.
  TEZ-1879. Create local UGI instances for each task and the AM, when running in LocalMode.
  TEZ-1661. LocalTaskScheduler hangs when shutdown.
  TEZ-1934. TestAMRecovery may fail due to the execution order is not determined.
  TEZ-1951. Fix general findbugs warnings in tez-dag.
  TEZ-1905. Fix findbugs warnings in tez-tests.
  TEZ-1945. Remove 2 GB memlimit restriction in MergeManager.
  TEZ-1913. Reduce deserialize cost in ValuesIterator.
  TEZ-1917. Examples should extend TezExampleBase.
  TEZ-1892. Add hashCode and equals for Vertex/VertexGroup/Edge/GroupInputEdge.
  TEZ-1904. Fix findbugs warnings in tez-runtime-library module.
  TEZ-1903. Fix findbugs warnings in tez-runtime-internal module.
  TEZ-1896. Move the default heartbeat timeout and checkinterval to TezConfiguration.
  TEZ-1923. FetcherOrderedGrouped gets into infinite loop due to memory pressure.
  TEZ-1274. Remove Key/Value type checks in IFile.
  TEZ-1912. Merge exceptions are thrown when enabling tez.runtime.shuffle.memory-to-memory.enable && tez.runtime.shuffle.memory-to-memory.segments.
  TEZ-1922. Fix comments: add UNSORTED_OUTPUT to TEZ_TASK_SCALE_MEMORY_WEIGHTED_RATIOS.
  TEZ-485. Get rid of TezTaskStatus.
  TEZ-1899. Fix findbugs warnings in tez-common module.
  TEZ-1898. Fix findbugs warnings in tez-api module.
  TEZ-1906. Fix findbugs warnings in tez-yarn-timeline-history-with-acls.
  TEZ-1767. Enable RLE in reducer side merge codepath.
  TEZ-1837. Restrict usage of Environment variables to main methods.
  TEZ-1867. Create new central dispatcher for Tez AM
  TEZ-1844. Shouldn't invoke system.exit in local mode when AM is failed to start.
  TEZ-1889. Fix test-patch to provide correct findbugs report.
  TEZ-1313. Setup pre-commit build to test submitted patches.
  TEZ-1856. Remove LocalOnFileSortedOutput, LocalMergedInput, LocalTaskOutputFiles.
  TEZ-1949. Whitelist TEZ_RUNTIME_OPTIMIZE_SHARED_FETCH for broadcast edges.
  TEZ-1593. Refactor PipelinedSorter to remove all MMAP based ByteBuffer references.

Release 0.6.0: Unreleased

INCOMPATIBLE CHANGES

ALL CHANGES:
  TEZ-1968. Tez UI - All vertices of DAG are not listed in vertices page
  TEZ-1890. tez-ui web.tar.gz also being uploaded to maven repository
  TEZ-1938. Build warning duplicate jersey-json definitions
  TEZ-1910. Build fails against hadoop-2.2.0.
  TEZ-1882. Tez UI build does not work on Windows
  TEZ-1915. Add public key to KEYS
  TEZ-1907. Fix javadoc warnings in tez codebase
  TEZ-1891. Incorrect number of Javadoc warnings reported
  TEZ-1762. Lots of unit tests do not have timeout parameter set.
  TEZ-1886. remove deprecation warnings for tez-ui on the console.
  TEZ-1875. dropdown filters do not work on vertices and task attempts page.
  TEZ-1873. TestTezAMRMClient fails due to host resolution timing out.
  TEZ-1881. Setup initial test-patch script for TEZ-1313.
  TEZ-1864. move initialization code dependent on config params to App.ready.
  TEZ-1870. Time displayed in the UI is in GMT.
  TEZ-1858. Docs for deploying/using the Tez UI.
  TEZ-1642. TestAMRecovery sometimes fail.
  TEZ-1859. TestGroupedSplits has commented out test: testGzip.
  TEZ-1868. Document how to do Windows builds due to with ACL symlink build changes.
  TEZ-1872. docs/src/site/custom/project-info-report.properties needs license header.
  TEZ-1850. Enable deploy for tez-ui war.
  TEZ-1841. Remove range versions for dependencies in tez-ui.
  TEZ-1854. Failing tests due to host resolution timing out.
  TEZ-1860. mvn apache-rat:check broken for tez-ui.
  TEZ-1866. remove the "original" directory under tez-ui
  TEZ-1591. Add multiDAG session test and move TestLocalMode to tez-tests
  TEZ-1769. ContainerCompletedWhileRunningTransition should inherit from TerminatedWhileRunningTransition
  TEZ-1849. Fix tez-ui war file licensing.
  TEZ-1840. Document TezTaskOutput.
  TEZ-1576. Class level comment in {{MiniTezCluster}} ends abruptly.
  TEZ-1838. tez-ui/src/main/webapp/bower.json gets updated after compiling source code.
  TEZ-1789. Move speculator processing off the central dispatcher.
  TEZ-1610. Add additional task counters for fetchers, merger.
  TEZ-1847. Fix package name for MiniTezClusterWithTimeline.
  TEZ-1846. Build fails with package org.apache.tez.dag.history.logging.ats does not exist.
  TEZ-1696. Make Tez use the domain-based timeline ACLs.
  TEZ-1835. TestFaultTolerance#testRandomFailingTasks is timing out
  TEZ-1832. TestSecureShuffle fails with NoClassDefFoundError: org/bouncycastle/x509/X509V1CertificateGenerator
  TEZ-1672. Update jetty to use stable 7.x version - 7.6.16.v20140903.
  TEZ-1822. Docs for Timeline/ACLs/HistoryText.
  TEZ-1252. Change wording on http://tez.apache.org/team-list.html related to member confusion.
  TEZ-1805. Tez client DAG cycle detection should detect self loops
  TEZ-1816. It is possible to receive START event when DAG is failed
  TEZ-1787. Counters for speculation
  TEZ-1773. Add attempt failure cause enum to the attempt failed/killed
  history record
  TEZ-14. Support MR like speculation capabilities based on latency deviation
  from the mean
  TEZ-1733. TezMerger should sort FileChunks on size when merging
  TEZ-1738. Tez tfile parser for log parsing
  TEZ-1627. Remove OUTPUT_CONSUMABLE and related Event in TaskAttemptImpl
  TEZ-1736. Add support for Inputs/Outputs in runtime-library to generate history text data.
  TEZ-1721. Update INSTALL instructions for clarifying tez client jars
    compatibility with runtime tarball on HDFS.
  TEZ-1690. TestMultiMRInput tests fail because of user collisions.
  TEZ-1687. Use logIdentifier of Vertex for logging.
  TEZ-1737. Should add taskNum in VertexFinishedEvent.
  TEZ-1772. Failing tests post TEZ-1737.
  TEZ-1785. Remove unused snappy-java dependency.
  TEZ-1685. Remove YARNMaster which is never used.
  TEZ-1797. Create necessary content for Tez DOAP file.
  TEZ-1650. Please create a DOAP file for your TLP.
  TEZ-1697. DAG submission fails if a local resource added is already part of tez.lib.uris
  TEZ-1060 Add randomness to fault tolerance tests
  TEZ-1790. DeallocationTaskRequest may been handled before corresponding AllocationTaskRequest in local mode

TEZ-UI CHANGES (TEZ-8):
  TEZ-1799. Enable Cross Origin Support in Tez UI
  TEZ-1634. Fix compressed IFile shuffle errors
  TEZ-1615. Skeleton framework for Tez UI
  TEZ-1604. Task View for Tez UI
  TEZ-1603. Vertex View for Tez UI.
  TEZ-1720. Allow filters in all tables and also to pass in filters using url params.
  TEZ-1708. Make UI part of TEZ build process.
  TEZ-1617. Shim layer for Tez UI for use within Ambari.
  TEZ-1741. App view.
  TEZ-1751. Log view & download links in task and task attempt view.
  TEZ-1753. Queue in dags view.
  TEZ-1765. Allow dropdown lists in table filters.
  TEZ-1606. Counters View for DAG, Vertex, and Task.
  TEZ-1768. follow up jira to address minor issues in Tez-ui.
  TEZ-1783. Wrapper in standalone mode.
  TEZ-1820. Fix wrong links.

Release 0.5.4: Unreleased

ALL CHANGES:
  TEZ-1931. Publish tez version info to Timeline.
  TEZ-1942. Number of tasks show in Tez UI with auto-reduce parallelism is misleading.
  TEZ-1962. Fix a thread leak in LocalMode.
  TEZ-1924. Tez AM does not register with AM with full FQDN causing jobs
  to fail in some environments.
  TEZ-1878. Task-specific log level override not working in certain conditions.
  TEZ-1775. Allow setting log level per logger.
  TEZ-1851. FileSystem counters do not differentiate between different FileSystems.
  TEZ-1852. Get examples to work in LocalMode.
  TEZ-1861. Fix failing test: TestOnFileSortedOutput.
  TEZ-1836. Provide better error messages when tez.runtime.io.sort.mb, spill percentage is incorrectly configured.
  TEZ-1800. Integer overflow in ExternalSorter.getInitialMemoryRequirement()

Release 0.5.3: 2014-12-10

ALL CHANGES:
{'fraction_non_alphanumeric': 0.06297650777908634, 'fraction_numerical': 0.06545589784912911, 'mean_word_length': 3.7313782991202347, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1484947', 'n_tokens_mistral': 5845, 'n_tokens_neox': 5168, 'n_words': 1988}