<commit_before>Trunk (unreleased changes)

  Non-backward compatible changes:

    BUGFIXES:

    IMPROVEMENTS:

  Backward compatible changes:

    BUGFIXES:

      BOOKKEEPER-145: Put notice and license file for distributed binaries in SVN (ivank)

      BOOKKEEPER-254: Bump zookeeper version in poms (ivank)

      BOOKKEEPER-72: Fix warnings issued by FindBugs (ivank)

      BOOKKEEPER-238: Add log4j.properties in conf/ for bin packages (ivank)

      bookkeeper-server/

        BOOKKEEPER-142: Parsing last log id is wrong, which may make entry log files overwritten (Sijie Gou via ivank)

        BOOKKEEPER-141: Run extracting ledger id from entry log files in GC thread to speed up bookie restart (Sijie Gou via ivank)

        BOOKKEEPER-148: Jenkins build is failing (ivank via fpj)

        BOOKKEEPER-40: BookieClientTest fails intermittantly (fpj via ivank)

	BOOKKEEPER-150: Entry is lost when recovering a ledger with not enough bookies. (Sijie Guo via ivank)

        BOOKKEEPER-153: Ledger can't be opened or closed due to zero-length metadata (Sijie Guo via ivank)

        BOOKKEEPER-23: Timeout requests (ivank)

        BOOKKEEPER-161: PerChannelBookieClient tries to reuse HashedWheelTimer, throws Exception (ivank)

        BOOKKEEPER-167: PerChannelBookieClient doesn't use ClientConfiguration (Sijie Guo via ivank)

        BOOKKEEPER-156: BookieJournalRollingTest failing (Sijie Guo via ivank)

	BOOKKEEPER-162: LedgerHandle.readLastConfirmed does not work (fpj)

        BOOKKEEPER-152: Can't recover a ledger whose current ensemble contain failed bookie. (ivank)

        BOOKKEEPER-171: ServerConfiguration can't use more than one directory for ledgers. (ivank via sijie)
	
	BOOKKEEPER-170: Bookie constructor starts a number of threads. (ivank via fpj)

        BOOKKEEPER-169: bookie hangs on reading header when encountering partial header index file (sijie via ivank)

        BOOKKEEPER-174: Bookie can't start when replaying entries whose ledger were deleted and garbage collected. (sijie via ivank)

        BOOKKEEPER-177: Index file is lost or some index pages aren't flushed. (sijie via ivank)

        BOOKKEEPER-113: NPE In BookKeeper test (fpj via ivank)

	BOOKKEEPER-176: HierarchicalBookieFailureTest Hung (ivank via fpj)

        BOOKKEEPER-180: bookie server doesn't quit when running out of disk space (sijie via ivank)

        BOOKKEEPER-185: Remove bookkeeper-server dependency on hadoop-common (ivank)

        BOOKKEEPER-184: CompactionTest failing on Jenkins (sijie via ivank)

        BOOKKEEPER-182: Entry log file is overwritten when fail to read lastLogId. (sijie via ivank)

        BOOKKEEPER-186: Bookkeeper throttling - permits is not released when read has failed from all replicas (Rakesh R via sijie)

        BOOKKEEPER-189: AbstractZkLedgerManager doesn't disregard cookies (ivank via sijie)

        BOOKKEEPER-195: HierarchicalLedgerManager doesn't consider idgen as a "specialNode" (ivank)

        BOOKKEEPER-190: Add entries would fail when number of open ledgers reaches more than openFileLimit. (sijie via ivank)

        BOOKKEEPER-194: Get correct latency for addEntry operations for JMX. (sijie via ivank)

        BOOKKEEPER-166: Bookie will not recover its journal if the length prefix of an entry is truncated (ivank)

        BOOKKEEPER-193: Ledger is garbage collected by mistake. (sijie, ivank via sijie)

        BOOKKEEPER-198: replaying entries of deleted ledgers would exhaust ledger cache. (sijie)

        BOOKKEEPER-112: Bookie Recovery on an open ledger will cause LedgerHandle#close on that ledger to fail (sijie)

        BOOKKEEPER-135: Fencing does not check the ledger masterPasswd (ivank)

	BOOKKEEPER-212: Bookie stops responding when creating and deleting many ledgers (sijie via fpj)

        BOOKKEEPER-211: Bookie fails to to start (sijie)

	BOOKKEEPER-200: Fix format and comments (fpj)

        BOOKKEEPER-216: Bookie doesn't exit with right exit code (sijie via ivank)

        BOOKKEEPER-196: Define interface between bookie and ledger storage (ivank)

        BOOKKEEPER-213: PerChannelBookieClient calls the wrong errorOut function when encountering an exception (Aniruddha via sijie)

        BOOKKEEPER-231: ZKUtil.killServer not closing the FileTxnSnapLog from ZK. (<NAME> via sijie)

        BOOKKEEPER-232: AsyncBK tests failing (umamaheswararao via ivank)

	BOOKKEEPER-229: Deleted entry log files would be garbage collected again and again. (sijie via fpj)

        BOOKKEEPER-242: Bookkeeper not able to connect other zookeeper when shutdown the zookeeper server where the BK has connected. (sijie & rakeshr via ivank)

        BOOKKEEPER-234: EntryLogger will throw NPE, if any dir does not exist or IO Errors. (umamaheswararao via ivank)

	BOOKKEEPER-235: Bad syncing in entrylogger degrades performance for many concurrent ledgers (ivank via fpj)

        BOOKKEEPER-224: Fix findbugs in bookkeeper-server component (ivank)

        BOOKKEEPER-251: Noise error message printed when scanning entry log files those have been garbage collected. (sijie via ivank)

        BOOKKEEPER-266: Review versioning documentation (ivank)

        BOOKKEEPER-258: CompactionTest failed (ivank via sijie)

      hedwig-client/

        BOOKKEEPER-217: NPE in hedwig client when enable DEBUG (sijie via ivank)

      hedwig-server/
      
        BOOKKEEPER-140: Hub server doesn't subscribe remote region correctly when a region is down. (Sijie Gou via ivank)

        BOOKKEEPER-133: Hub server should update subscription state to zookeeper when losing topic or shutting down (Sijie Gou via ivank)

        BOOKKEEPER-74: Bookkeeper Persistence Manager should give up topic on error (sijie via ivank)

        BOOKKEEPER-163: Prevent incorrect NoSuchLedgerException for readLastConfirmed. (ivank via sijie)

        BOOKKEEPER-197: HedwigConsole uses the same file to load bookkeeper client config and hub server config (sijie)

        BOOKKEEPER-56: Race condition of message handler in connection recovery in Hedwig client (sijie & Gavin Li via ivank)

        BOOKKEEPER-215: Deadlock occurs under high load (sijie via ivank)

        BOOKKEEPER-245: Intermittent failures in PersistanceManager tests (ivank)

        BOOKKEEPER-209: Typo in ServerConfiguration for READAHEAD_ENABLED (ivank)

        BOOKKEEPER-146: TestConcurrentTopicAcquisition sometimes hangs (ivank)

      bookkeeper-benchmark/
	
	BOOKKEEPER-207: BenchBookie doesn't run correctly (ivank via fpj)

        BOOKKEEPER-228: Fix the bugs in BK benchmark (umamaheswararao via ivank)

    IMPROVEMENTS:

	BOOKKEEPER-265: Review JMX documentation (sijie via fpj)

      bookkeeper-server/

        BOOKKEEPER-95: extends zookeeper JMX to monitor and manage bookie server (Sijie Guo via ivank)

        BOOKKEEPER-98: collect add/read statistics on bookie server (Sijie Guo via ivank)

	BOOKKEEPER-157:	For small packets, increasing number of bookies actually degrades performance. (ivank via fpj)

        BOOKKEEPER-165: Add versioning support for journal files (ivank)

        BOOKKEEPER-137: Do not create Ledger index files until absolutely necessary. (ivank)

	BOOKKEEPER-172: Upgrade framework for filesystem layouts (ivank via fpj)

        BOOKKEEPER-178: Delay ledger directory creation until the ledger index file was created (sijie via ivank)

        BOOKKEEPER-160: bookie server needs to do compaction over entry log files to reclaim disk space (sijie via ivank)

        BOOKKEEPER-187: Create well defined interface for LedgerCache (ivank)

        BOOKKEEPER-175: Bookie code is very coupled (ivank)

        BOOKKEEPER-188: Garbage collection code is in the wrong place (ivank via sijie)

        BOOKKEEPER-218: Provide journal manager to manage journal related operations (sijie)

	BOOKKEEPER-173: Uncontrolled number of threads in bookkeeper (sijie via fpj)

	BOOKKEEPER-241: Add documentation for bookie entry log compaction (sijie via fpj)

	BOOKKEEPER-263: ZK ledgers root path is hard coded (Aniruddha via sijie)

	BOOKKEEPER-260: Define constant for -1 (invalid entry id) (ivank via fpj)

      hedwig-server/

        BOOKKEEPER-77: Add a console client for hedwig (Sijie Guo via ivank)

        BOOKKEEPER-168: Message bounding on subscriptions (ivank)

        BOOKKEEPER-96: extends zookeeper JMX to monitor and manage hedwig server (sijie via ivank)

        BOOKKEEPER-97: collect pub/sub/consume statistics on hub server (sijie via ivank)<commit_msg>BOOKKEEPER-270: Review documentation on bookie cookie (ivank via fpj)<commit_after>Trunk (unreleased changes)

  Non-backward compatible changes:

    BUGFIXES:

    IMPROVEMENTS:

  Backward compatible changes:

    BUGFIXES:

      BOOKKEEPER-145: Put notice and license file for distributed binaries in SVN (ivank)

      BOOKKEEPER-254: Bump zookeeper version in poms (ivank)

      BOOKKEEPER-72: Fix warnings issued by FindBugs (ivank)

      BOOKKEEPER-238: Add log4j.properties in conf/ for bin packages (ivank)

      bookkeeper-server/

        BOOKKEEPER-142: Parsing last log id is wrong, which may make entry log files overwritten (Sijie Gou via ivank)

        BOOKKEEPER-141: Run extracting ledger id from entry log files in GC thread to speed up bookie restart (Sijie Gou via ivank)

        BOOKKEEPER-148: Jenkins build is failing (ivank via fpj)

        BOOKKEEPER-40: BookieClientTest fails intermittantly (fpj via ivank)

	BOOKKEEPER-150: Entry is lost when recovering a ledger with not enough bookies. (Sijie Guo via ivank)

        BOOKKEEPER-153: Ledger can't be opened or closed due to zero-length metadata (Sijie Guo via ivank)

        BOOKKEEPER-23: Timeout requests (ivank)

        BOOKKEEPER-161: PerChannelBookieClient tries to reuse HashedWheelTimer, throws Exception (ivank)

        BOOKKEEPER-167: PerChannelBookieClient doesn't use ClientConfiguration (Sijie Guo via ivank)

        BOOKKEEPER-156: BookieJournalRollingTest failing (Sijie Guo via ivank)

	BOOKKEEPER-162: LedgerHandle.readLastConfirmed does not work (fpj)

        BOOKKEEPER-152: Can't recover a ledger whose current ensemble contain failed bookie. (ivank)

        BOOKKEEPER-171: ServerConfiguration can't use more than one directory for ledgers. (ivank via sijie)
	
	BOOKKEEPER-170: Bookie constructor starts a number of threads. (ivank via fpj)

        BOOKKEEPER-169: bookie hangs on reading header when encountering partial header index file (sijie via ivank)

        BOOKKEEPER-174: Bookie can't start when replaying entries whose ledger were deleted and garbage collected. (sijie via ivank)

        BOOKKEEPER-177: Index file is lost or some index pages aren't flushed. (sijie via ivank)

        BOOKKEEPER-113: NPE In BookKeeper test (fpj via ivank)

	BOOKKEEPER-176: HierarchicalBookieFailureTest Hung (ivank via fpj)

        BOOKKEEPER-180: bookie server doesn't quit when running out of disk space (sijie via ivank)

        BOOKKEEPER-185: Remove bookkeeper-server dependency on hadoop-common (ivank)

        BOOKKEEPER-184: CompactionTest failing on Jenkins (sijie via ivank)

        BOOKKEEPER-182: Entry log file is overwritten when fail to read lastLogId. (sijie via ivank)

        BOOKKEEPER-186: Bookkeeper throttling - permits is not released when read has failed from all replicas (Rakesh R via sijie)

        BOOKKEEPER-189: AbstractZkLedgerManager doesn't disregard cookies (ivank via sijie)

        BOOKKEEPER-195: HierarchicalLedgerManager doesn't consider idgen as a "specialNode" (ivank)

        BOOKKEEPER-190: Add entries would fail when number of open ledgers reaches more than openFileLimit. (sijie via ivank)

        BOOKKEEPER-194: Get correct latency for addEntry operations for JMX. (sijie via ivank)

        BOOKKEEPER-166: Bookie will not recover its journal if the length prefix of an entry is truncated (ivank)

        BOOKKEEPER-193: Ledger is garbage collected by mistake. (sijie, ivank via sijie)

        BOOKKEEPER-198: replaying entries of deleted ledgers would exhaust ledger cache. (sijie)

        BOOKKEEPER-112: Bookie Recovery on an open ledger will cause LedgerHandle#close on that ledger to fail (sijie)

        BOOKKEEPER-135: Fencing does not check the ledger masterPasswd (ivank)

	BOOKKEEPER-212: Bookie stops responding when creating and deleting many ledgers (sijie via fpj)

        BOOKKEEPER-211: Bookie fails to to start (sijie)

	BOOKKEEPER-200: Fix format and comments (fpj)

        BOOKKEEPER-216: Bookie doesn't exit with right exit code (sijie via ivank)

        BOOKKEEPER-196: Define interface between bookie and ledger storage (ivank)

        BOOKKEEPER-213: PerChannelBookieClient calls the wrong errorOut function when encountering an exception (Aniruddha via sijie)

        BOOKKEEPER-231: ZKUtil.killServer not closing the FileTxnSnapLog from ZK. (Uma <NAME> G via sijie)

        BOOKKEEPER-232: AsyncBK tests failing (umamaheswararao via ivank)

	BOOKKEEPER-229: Deleted entry log files would be garbage collected again and again. (sijie via fpj)

        BOOKKEEPER-242: Bookkeeper not able to connect other zookeeper when shutdown the zookeeper server where the BK has connected. (sijie & rakeshr via ivank)

        BOOKKEEPER-234: EntryLogger will throw NPE, if any dir does not exist or IO Errors. (umamaheswararao via ivank)

	BOOKKEEPER-235: Bad syncing in entrylogger degrades performance for many concurrent ledgers (ivank via fpj)

        BOOKKEEPER-224: Fix findbugs in bookkeeper-server component (ivank)

        BOOKKEEPER-251: Noise error message printed when scanning entry log files those have been garbage collected. (sijie via ivank)

        BOOKKEEPER-266: Review versioning documentation (ivank)

        BOOKKEEPER-258: CompactionTest failed (ivank via sijie)

      hedwig-client/

        BOOKKEEPER-217: NPE in hedwig client when enable DEBUG (sijie via ivank)

      hedwig-server/
      
        BOOKKEEPER-140: Hub server doesn't subscribe remote region correctly when a region is down. (Sijie Gou via ivank)

        BOOKKEEPER-133: Hub server should update subscription state to zookeeper when losing topic or shutting down (Sijie Gou via ivank)

        BOOKKEEPER-74: Bookkeeper Persistence Manager should give up topic on error (sijie via ivank)

        BOOKKEEPER-163: Prevent incorrect NoSuchLedgerException for readLastConfirmed. (ivank via sijie)

        BOOKKEEPER-197: HedwigConsole uses the same file to load bookkeeper client config and hub server config (sijie)

        BOOKKEEPER-56: Race condition of message handler in connection recovery in Hedwig client (sijie & Gavin Li via ivank)

        BOOKKEEPER-215: Deadlock occurs under high load (sijie via ivank)

        BOOKKEEPER-245: Intermittent failures in PersistanceManager tests (ivank)

        BOOKKEEPER-209: Typo in ServerConfiguration for READAHEAD_ENABLED (ivank)

        BOOKKEEPER-146: TestConcurrentTopicAcquisition sometimes hangs (ivank)

      bookkeeper-benchmark/
	
	BOOKKEEPER-207: BenchBookie doesn't run correctly (ivank via fpj)

        BOOKKEEPER-228: Fix the bugs in BK benchmark (umamaheswararao via ivank)

    IMPROVEMENTS:

	BOOKKEEPER-265: Review JMX documentation (sijie via fpj)

      bookkeeper-server/

        BOOKKEEPER-95: extends zookeeper JMX to monitor and manage bookie server (Sijie Guo via ivank)

        BOOKKEEPER-98: collect add/read statistics on bookie server (Sijie Guo via ivank)

	BOOKKEEPER-157:	For small packets, increasing number of bookies actually degrades performance. (ivank via fpj)

        BOOKKEEPER-165: Add versioning support for journal files (ivank)

        BOOKKEEPER-137: Do not create Ledger index files until absolutely necessary. (ivank)

	BOOKKEEPER-172: Upgrade framework for filesystem layouts (ivank via fpj)

        BOOKKEEPER-178: Delay ledger directory creation until the ledger index file was created (sijie via ivank)

        BOOKKEEPER-160: bookie server needs to do compaction over entry log files to reclaim disk space (sijie via ivank)

        BOOKKEEPER-187: Create well defined interface for LedgerCache (ivank)

        BOOKKEEPER-175: Bookie code is very coupled (ivank)

        BOOKKEEPER-188: Garbage collection code is in the wrong place (ivank via sijie)

        BOOKKEEPER-218: Provide journal manager to manage journal related operations (sijie)

	BOOKKEEPER-173: Uncontrolled number of threads in bookkeeper (sijie via fpj)

	BOOKKEEPER-241: Add documentation for bookie entry log compaction (sijie via fpj)

	BOOKKEEPER-263: ZK ledgers root path is hard coded (Aniruddha via sijie)

	BOOKKEEPER-260: Define constant for -1 (invalid entry id) (ivank via fpj)

	BOOKKEEPER-270: Review documentation on bookie cookie (ivank via fpj)

      hedwig-server/

        BOOKKEEPER-77: Add a console client for hedwig (S<NAME>o via ivank)

        BOOKKEEPER-168: Message bounding on subscriptions (ivank)

        BOOKKEEPER-96: extends zookeeper JMX to monitor and manage hedwig server (sijie via ivank)

        BOOKKEEPER-97: collect pub/sub/consume statistics on hub server (sijie via ivank)
{'fraction_non_alphanumeric': 0.051116712585731874, 'fraction_numerical': 0.03001348261914532, 'mean_word_length': 3.528802760817627, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '557643', 'n_tokens_mistral': 5828, 'n_tokens_neox': 5169, 'n_words': 2013}