Add more compression types for `to_json`
{'login': 'bhavitvyamalik', 'id': 19718818, 'node_id': 'MDQ6VXNlcjE5NzE4ODE4', 'avatar_url': 'https://avatars.githubusercontent.com/u/19718818?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/bhavitvyamalik', 'html_url': 'https://github.com/bhavitvyamalik', 'followers_url': 'https://api.github.com/users/bhavitvyamalik/followers', 'following_url': 'https://api.github.com/users/bhavitvyamalik/following{/other_user}', 'gists_url': 'https://api.github.com/users/bhavitvyamalik/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/bhavitvyamalik/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/bhavitvyamalik/subscriptions', 'organizations_url': 'https://api.github.com/users/bhavitvyamalik/orgs', 'repos_url': 'https://api.github.com/users/bhavitvyamalik/repos', 'events_url': 'https://api.github.com/users/bhavitvyamalik/events{/privacy}', 'received_events_url': 'https://api.github.com/users/bhavitvyamalik/received_events', 'type': 'User', 'site_admin': False}
[]
[]
["@lhoestq, I looked into how to compress with `zipfile` for which few methods exist, let me know which one looks good:\r\n1. create the file in normal `wb` mode and then zip it separately\r\n2. use `ZipFile.write_str` to write file into the archive. For this we'll need to change how we're writing files from `_write` method  \r\n\r\nHow `pandas` handles it is that they have created a wrapper for standard library class `ZipFile` and allow the returned file-like handle to accept byte strings via `write` method instead of `write_str` (purpose was to change the name of function by creating that wrapper)"
 "1. sounds not ideal since it creates an intermediary file.\r\nI like pandas' approach. Is it possible to implement 2. using the pandas class ? Or maybe we can have something similar ?"
 "Definitely, @lhoestq! I've adapted that from original code and turns out it is faster than `gz` compression. Apart from that I've also added `infer` option to automatically infer compression type from `path_or_buf` given"
 'One small thing, currently I\'m assuming that user will provide compression extension in `path_or_buf`. Is it this also possible?\r\n`dataset.to_json("from_dataset.json", compression="zip")`? \r\nShould I put an `assert` to ensure the file name provided always has a compression extension?'
 'Thanks !\r\n\r\n> One small thing, currently I\'m assuming that user will provide compression extension in path_or_buf. Is it this also possible?\r\n>dataset.to_json("from_dataset.json", compression="zip")?\r\n>Should I put an assert to ensure the file name provided always has a compression extension?\r\n\r\nI think it\'s fine as it is right now :) No need to check the extension of the filename passed to `path_or_buf`.\r\n'
 '> turns out it is faster than gz compression\r\n\r\nI think the default compression level of `gzip` is 9 in python, which is very slow. Maybe we can switch to compression level 6 instead which is faster, like the `gzip` command on unix'
 "I found that `fsspec` has something that may interest you: [fsspec.open(..., compression=...)](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.open). I don't remember if we've already mentioned it or not\r\n\r\nIt also has `zip` if I understand correctly ! see https://github.com/fsspec/filesystem_spec/blob/master/fsspec/compression.py#L70\r\n\r\nSince `fsspec` is a dependency of `datasets` we can use all this :)\r\n\r\nLet me know if you prefer using `fsspec` instead (I haven't tested this yet to write compressed files). IMO it sounds pretty easy to use and it would make the code base simpler"
 "Just tried `fsspec` but I'm not able to write compressed `zip` files :/\r\n`gzip`, `xz`, `bz2` are all working fine and it's really simple (no need for `FileWriteHandler` now!)"]
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/3551', 'html_url': 'https://github.com/huggingface/datasets/pull/3551', 'diff_url': 'https://github.com/huggingface/datasets/pull/3551.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/3551.patch', 'merged_at': datetime.datetime(2022, 2, 21, 15, 58, 15)}
This PR adds `bz2`, `xz`, and `zip` (WIP) for `to_json`. I also plan to add `infer` like how `pandas` does it
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/3551/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}