Allow dynamic first dimension for ArrayXD
{'login': 'rpowalski', 'id': 10357417, 'node_id': 'MDQ6VXNlcjEwMzU3NDE3', 'avatar_url': 'https://avatars.githubusercontent.com/u/10357417?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/rpowalski', 'html_url': 'https://github.com/rpowalski', 'followers_url': 'https://api.github.com/users/rpowalski/followers', 'following_url': 'https://api.github.com/users/rpowalski/following{/other_user}', 'gists_url': 'https://api.github.com/users/rpowalski/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/rpowalski/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/rpowalski/subscriptions', 'organizations_url': 'https://api.github.com/users/rpowalski/orgs', 'repos_url': 'https://api.github.com/users/rpowalski/repos', 'events_url': 'https://api.github.com/users/rpowalski/events{/privacy}', 'received_events_url': 'https://api.github.com/users/rpowalski/received_events', 'type': 'User', 'site_admin': False}
[]
[]
["@lhoestq, thanks for your review.\r\n\r\nI added test for `to_pylist`, I didn't do that for `to_numpy` because this method shouldn't be called for dynamic dimension ArrayXD - this method will try to make a single numpy array for the whole column which cannot be done for dynamic arrays.\r\n\r\nI dig into `to_pandas()` functionality and I found it quite difficult to implement. `PandasArrayExtensionArray` takes single np.array as an argument. It might be a bit of changes to make it work with the list of arrays. Do you mind if we exclude this work from this PR. I added an error message for the case if somebody tries to use dynamic arrays with `to_pandas`"
 '@lhoestq, I just fixed all the tests. Let me know if there is anything else to add.'
 '@lhoestq, any chance you had some time to check out this PR?\r\n'
 'Hi ! Sorry for the delay\r\n\r\nIt looks good to me ! I think the only thing missing is the support for passing a list of numpy arrays to `map` when the first dimension is dynamic.\r\n\r\nCurrently it raises an error:\r\n```python\r\nfrom datasets import *\r\nimport numpy as np\r\n\r\nfeatures=  Features({"a": Array3D(shape=(None, 5, 2), dtype="int32")})\r\nd = Dataset.from_dict({"a": [np.zeros((5,5,2)), np.zeros((2,5,2))]}, features=features)\r\nd = d.map(lambda a: {"a": np.concatenate([a]*2)}, input_columns="a")\r\nprint(d[0])\r\n```\r\nraises\r\n```python\r\nTraceback (most recent call last):\r\n  File "playground/ttest.py", line 6, in <module>\r\n    d = d.map(lambda x: {"a": np.concatenate([x]*2)}, input_columns="a")\r\n  File "/home/truent/hf/datasets/src/datasets/arrow_dataset.py", line 1932, in map\r\n    return self._map_single(\r\n  File "/home/truent/hf/datasets/src/datasets/arrow_dataset.py", line 426, in wrapper\r\n    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)\r\n  File "/home/truent/hf/datasets/src/datasets/fingerprint.py", line 406, in wrapper\r\n    out = func(self, *args, **kwargs)\r\n  File "/home/truent/hf/datasets/src/datasets/arrow_dataset.py", line 2317, in _map_single\r\n    writer.finalize()\r\n  File "/home/truent/hf/datasets/src/datasets/arrow_writer.py", line 443, in finalize\r\n    self.write_examples_on_file()\r\n  File "/home/truent/hf/datasets/src/datasets/arrow_writer.py", line 312, in write_examples_on_file\r\n    pa_array = pa.array(typed_sequence)\r\n  File "pyarrow/array.pxi", line 222, in pyarrow.lib.array\r\n  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol\r\n  File "/home/truent/hf/datasets/src/datasets/arrow_writer.py", line 108, in __arrow_array__\r\n    storage = pa.array(self.data, type.storage_dtype)\r\n  File "pyarrow/array.pxi", line 305, in pyarrow.lib.array\r\n  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array\r\n  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status\r\n  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status\r\npyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values\r\n```\r\n\r\nI think the issue is that here we don\'t cover the case where self.data is a list of numpy arrays:\r\n\r\nhttps://github.com/huggingface/datasets/blob/55fd140a63b8f03a0e72985647e498f1fc799d3f/src/datasets/arrow_writer.py#L104-L109\r\n\r\nWe should remove the `isinstance(self.data[0], np.ndarray)` part and add these lines to cover this case:\r\n\r\nhttps://github.com/huggingface/datasets/blob/55fd140a63b8f03a0e72985647e498f1fc799d3f/src/datasets/arrow_writer.py#L112-L113'
 '@lhoestq, thanks, good catch!\r\nAre you able to run this check with fixed dimension ArrayXD?\r\nfor below example\r\n```\r\nimport numpy as np\r\nfrom datasets import *\r\n\r\nfeatures = Features({"a": Array3D(shape=(2, 5, 2), dtype="int32")})\r\nd = Dataset.from_dict({"a": [np.zeros((2, 5, 2)), np.zeros((2, 5, 2))]}, features=features)\r\nd = d.map(lambda a: {"a": np.array(a) + 1}, input_columns="a")\r\nprint(d[0])\r\n```\r\n\r\nI am getting:\r\n```\r\n  File "/home/ib/datasets/src/datasets/arrow_writer.py", line 116, in __arrow_array__\r\n    if trying_type and out[0].as_py() != self.data[0]:\r\nValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()\r\n```'
 'Nevertheless, I tried to fix that. Let me know if that works.'
 '@lhoestq, just resolved the conflicts. Let me know if there is anything left to do with this PR'
 'Hi, thanks a lot for your comments.\r\nAgree, happy to contribute to this topic in future PRs'
 "Hi @rpowalski, thanks for adding this feature! \r\n\r\nI wanted to check if you are still interested in documenting this, otherwise I'd be happy to help with it :)"]
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/2891', 'html_url': 'https://github.com/huggingface/datasets/pull/2891', 'diff_url': 'https://github.com/huggingface/datasets/pull/2891.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/2891.patch', 'merged_at': datetime.datetime(2021, 10, 29, 9, 37, 17)}
Add support for dynamic first dimension for ArrayXD features. See issue [#887](https://github.com/huggingface/datasets/issues/887).
Following changes allow for `to_pylist` method of `ArrayExtensionArray` to return a list of numpy arrays where fist dimension can vary.

@lhoestq Could you suggest how you want to extend test suit. For now I added only very limited testing.
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/2891/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}