<commit_before>/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

Pig Change Log

Release 0.10.0 - Unreleased

INCOMPATIBLE CHANGES

IMPROVEMENTS

PIG-2332: JsonLoader/JsonStorage (daijy)

PIG-2328: Add builtin UDFs for building and using bloom filters (gates)

PIG-2334: Set default number of reducers for S3N filesystem (ddaniels888 via daijy)

PIG-1387: Syntactical Sugar for PIG-1385 (azaroth)

PIG-2305: Pig should log the split locations in task logs (vivekp via thejas)

PIG-2293: Pig should support a more efficient merge join against data sources that natively support point 
lookups or where the join is against large, sparse tables (aklish via daijy)

PIG-2287: add test cases for limit and sample that use expressions with 
 constants only (no scalar variables) (thejas via gates)

PIG-2092: Missing sh command from Grant shell (olgan)

PIG-2163: Improve nested cross to stream one relation (zjshen via daijy)

PIG-2249: Enable pig e2e testing on EC2 (gates)

PIG-2256: Upgrade Avro dependency to 1.5.3 (tucu00 via dvryaboy)

PIG-604: Kill the Pig job should kill all associated Hadoop Jobs (daijy)

PIG-2096: End to end tests for new Macro feature (gates)

PIG-2242: Allow the delimiter to be specified when calling TOKENIZE (markroddy via hashutosh)

PIG-2240: Allow any compression codec to be specified in AvroStorage (tomwhite via dvryaboy)

PIG-2229: Pig end-to-end tests should test local mode as well as mr mode (gates)

PIG-2235: Several files in e2e tests aren't being run (gates)

PIG-2196: Test harness should be independent of Pig (hashutosh) -- Missed few
changes in last commit.

PIG-2196: Test harness should be independent of Pig (hashutosh)

PIG-1429: Add Boolean Data Type to Pig (zjshen via daijy)

PIG-2218: Pig end-to-end tests should be accessible from top level build.xml (gates)

PIG-2176: add logical plan assumption checker  (thejas)

PIG-1631: Support to 2 level nested foreach (aniket486 via daijy)

PIG-2191: Reduce amount of log spam generated by UDFs (dvryaboy)

PIG-2200: Piggybank cannot be built from the Git mirror (dvryaboy)

PIG-2168: CubeDimensions UDF (dvryaboy)

PIG-2189: e2e test harness needs to use Pig as a source of truth (gates via daijy)

PIG-1904: Default split destination (azaroth via thejas)

PIG-2125: Make Pig work with hadoop .NEXT (daijy)

PIG-2143: Make PigStorage optionally store schema; improve docs. (dvryaboy) 

PIG-1973: UDFContext.getUDFContext usage of ThreadLocal pattern 
 is not typical (woody via thejas)

PIG-2053: PigInputFormat uses class.isAssignableFrom() where 
 instanceof is more appropriate (woody via thejas)

PIG-2161: TOTUPLE should use no-copy tuple creation (dvryaboy)

PIG-1946: HBaseStorage constructor syntax is error prone (billgraham via dvryaboy)

PIG-2001: DefaultTuple(List) constructor is inefficient, causes List.size()
 System.arraycopy() calls (though they are 0 byte copies), 
 DefaultTuple(int) constructor is a bit misleading wrt time 
 complexity (woody via thejas)

PIG-1916: Nested cross (zjshen via daijy)

PIG-2128: Generating the jar file takes a lot of time and is unnecessary when running Pig local mode (julien)

PIG-2121: e2e test harness should use ant instead of make (gates)

PIG-2142: Allow registering multiple jars from DFS via single statement (rangadi via dvryaboy)

PIG-1926: Sample/Limit should take scalar (azaroth via thejas)

PIG-1950: e2e test harness needs to be able to compare to previous version of
Pig (gates)

PIG-536: the shell script 'pig' does not work if PIG_HOME has the word 'hadoop' in it's directory (miguno via olgan)

PIG-2108 e2e test harness needs to be able to mark certain tests as ignored
		(gates)

PIG-1825: ability to turn off the write ahead log for pig's HBaseStorage (billgraham via dvryaboy)

PIG-1772: Pig 090 Documentation (chandec via olgan)

PIG-1772: Pig 090 Documentation (chandec via olgan)

PIG-1772: Pig 090 Documentation (chandec via olgan)

PIG-1824: Support import modules in Jython UDF (woody via rding)

PIG-1994: e2e test harness deployment implementation for existing cluster
(gates)

PIG-2036: [piggybank] Set header delimiter in PigStorageSchema (mmoeller via dvryaboy)

PIG-1949: e2e test harness should use bin/pig rather than calling java
directly (gates)

PIG-2026: e2e tests in eclipse classpath (azaroth via hashutosh) 

PIG-2024: Incorrect jar paths in .classpath template for eclipse (azaroth via hashutosh)

OPTIMIZATIONS

PIG-2011: Speed up TestTypedMap.java (dvryaboy)

PIG-2228: support partial aggregation in map task (thejas)

BUG FIXES

PIG-2184: Not able to provide positional reference to macro invocations (daijy)

PIG-2209: JsonMetadata fails to find schema for glob paths (daijy)

PIG-2352: e2e test harness' use of environment variables causes unintended effects between tests (gates)

PIG-2165: Need a way to deal with params and param_file in embedded pig in python (daijy)

PIG-2313: NPE in ILLUSTRATE trying to get StatusReporter in STORE (daijy)

PIG-2335: bin/pig does not work with bash 3.0 (azaroth)

PIG-2275: NullPointerException from ILLUSTRATE (daijy)

PIG-2119: DuplicateForEachColumnRewrite makes assumptions about the position of LOGGenerate in the plan (daijy)

PIG-2290: TOBAG wraps tuple parameters in another tuple (ryan.hoegg via thejas)

PIG-2288: Pig 0.9 error message not useful as compared to 0.8 in case 
 of group by (vivekp via thejas)

PIG-2316: Incorrect results for FILTER *** BY ( *** OR ***) with
 FilterLogicExpressionSimplifier optimizer turned on (knoguchi via thejas)<commit_msg>PIG-2184: Not able to provide positional reference to macro invocations<commit_after>/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

Pig Change Log

Release 0.10.0 - Unreleased

INCOMPATIBLE CHANGES

IMPROVEMENTS

PIG-2332: JsonLoader/JsonStorage (daijy)

PIG-2328: Add builtin UDFs for building and using bloom filters (gates)

PIG-2334: Set default number of reducers for S3N filesystem (ddaniels888 via daijy)

PIG-1387: Syntactical Sugar for PIG-1385 (azaroth)

PIG-2305: Pig should log the split locations in task logs (vivekp via thejas)

PIG-2293: Pig should support a more efficient merge join against data sources that natively support point 
lookups or where the join is against large, sparse tables (aklish via daijy)

PIG-2287: add test cases for limit and sample that use expressions with 
 constants only (no scalar variables) (thejas via gates)

PIG-2092: Missing sh command from Grant shell (olgan)

PIG-2163: Improve nested cross to stream one relation (zjshen via daijy)

PIG-2249: Enable pig e2e testing on EC2 (gates)

PIG-2256: Upgrade Avro dependency to 1.5.3 (tucu00 via dvryaboy)

PIG-604: Kill the Pig job should kill all associated Hadoop Jobs (daijy)

PIG-2096: End to end tests for new Macro feature (gates)

PIG-2242: Allow the delimiter to be specified when calling TOKENIZE (markroddy via hashutosh)

PIG-2240: Allow any compression codec to be specified in AvroStorage (tomwhite via dvryaboy)

PIG-2229: Pig end-to-end tests should test local mode as well as mr mode (gates)

PIG-2235: Several files in e2e tests aren't being run (gates)

PIG-2196: Test harness should be independent of Pig (hashutosh) -- Missed few
changes in last commit.

PIG-2196: Test harness should be independent of Pig (hashutosh)

PIG-1429: Add Boolean Data Type to Pig (zjshen via daijy)

PIG-2218: Pig end-to-end tests should be accessible from top level build.xml (gates)

PIG-2176: add logical plan assumption checker  (thejas)

PIG-1631: Support to 2 level nested foreach (aniket486 via daijy)

PIG-2191: Reduce amount of log spam generated by UDFs (dvryaboy)

PIG-2200: Piggybank cannot be built from the Git mirror (dvryaboy)

PIG-2168: CubeDimensions UDF (dvryaboy)

PIG-2189: e2e test harness needs to use Pig as a source of truth (gates via daijy)

PIG-1904: Default split destination (azaroth via thejas)

PIG-2125: Make Pig work with hadoop .NEXT (daijy)

PIG-2143: Make PigStorage optionally store schema; improve docs. (dvryaboy) 

PIG-1973: UDFContext.getUDFContext usage of ThreadLocal pattern 
 is not typical (woody via thejas)

PIG-2053: PigInputFormat uses class.isAssignableFrom() where 
 instanceof is more appropriate (woody via thejas)

PIG-2161: TOTUPLE should use no-copy tuple creation (dvryaboy)

PIG-1946: HBaseStorage constructor syntax is error prone (billgraham via dvryaboy)

PIG-2001: DefaultTuple(List) constructor is inefficient, causes List.size()
 System.arraycopy() calls (though they are 0 byte copies), 
 DefaultTuple(int) constructor is a bit misleading wrt time 
 complexity (woody via thejas)

PIG-1916: Nested cross (zjshen via daijy)

PIG-2128: Generating the jar file takes a lot of time and is unnecessary when running Pig local mode (julien)

PIG-2121: e2e test harness should use ant instead of make (gates)

PIG-2142: Allow registering multiple jars from DFS via single statement (rangadi via dvryaboy)

PIG-1926: Sample/Limit should take scalar (azaroth via thejas)

PIG-1950: e2e test harness needs to be able to compare to previous version of
Pig (gates)

PIG-536: the shell script 'pig' does not work if PIG_HOME has the word 'hadoop' in it's directory (miguno via olgan)

PIG-2108 e2e test harness needs to be able to mark certain tests as ignored
		(gates)

PIG-1825: ability to turn off the write ahead log for pig's HBaseStorage (billgraham via dvryaboy)

PIG-1772: Pig 090 Documentation (chandec via olgan)

PIG-1772: Pig 090 Documentation (chandec via olgan)

PIG-1772: Pig 090 Documentation (chandec via olgan)

PIG-1824: Support import modules in Jython UDF (woody via rding)

PIG-1994: e2e test harness deployment implementation for existing cluster
(gates)

PIG-2036: [piggybank] Set header delimiter in PigStorageSchema (mmoeller via dvryaboy)

PIG-1949: e2e test harness should use bin/pig rather than calling java
directly (gates)

PIG-2026: e2e tests in eclipse classpath (azaroth via hashutosh) 

PIG-2024: Incorrect jar paths in .classpath template for eclipse (azaroth via hashutosh)

OPTIMIZATIONS

PIG-2011: Speed up TestTypedMap.java (dvryaboy)

PIG-2228: support partial aggregation in map task (thejas)

BUG FIXES

PIG-2184: Not able to provide positional reference to macro invocations (xutingz via daijy)

PIG-2209: JsonMetadata fails to find schema for glob paths (daijy)

PIG-2352: e2e test harness' use of environment variables causes unintended effects between tests (gates)

PIG-2165: Need a way to deal with params and param_file in embedded pig in python (daijy)

PIG-2313: NPE in ILLUSTRATE trying to get StatusReporter in STORE (daijy)

PIG-2335: bin/pig does not work with bash 3.0 (azaroth)

PIG-2275: NullPointerException from ILLUSTRATE (daijy)

PIG-2119: DuplicateForEachColumnRewrite makes assumptions about the position of LOGGenerate in the plan (daijy)

PIG-2290: TOBAG wraps tuple parameters in another tuple (ryan.hoegg via thejas)

PIG-2288: Pig 0.9 error message not useful as compared to 0.8 in case 
 of group by (vivekp via thejas)

PIG-2316: Incorrect results for FILTER *** BY ( *** OR ***) with
 FilterLogicExpressionSimplifier optimizer turned on (knoguchi via thejas)
{'fraction_non_alphanumeric': 0.060875079264426125, 'fraction_numerical': 0.05009511731135067, 'mean_word_length': 4.565505072783414, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4278032', 'n_tokens_mistral': 4471, 'n_tokens_neox': 3984, 'n_words': 1768}