<jupyter_start><jupyter_text>Using HDFS CLILet us understand how to use HDFS CLI to interact with HDFS.* Typically the cluster contain 3 types of nodes.  * Gateway nodes or client nodes or edge nodes  * Master nodes  * Worker nodes* Developers like us will typically have access to Gateway nodes or Client nodes.* We can connect to Gateway nodes or Client nodes using SSH.* Once login, we can interact with HDFS either by using `hadoop fs` or `hdfs dfs`. Both of them are aliases to each other.* `hadoop` have other subcommands than `fs` and is typically used to interact with HDFS or Map Reduce as developers.* `hdfs` have other subcommands than `dfs`. It is typically used to not only manage files in HDFS but also administrative tasks related HDFS components such as **Namenode**, **Secondary Namenode**, **Datanode** etc.* As deveopers, our scope will be limited to use `hdfs dfs` or `hadoop fs` to interact with HDFS.* Both have sub commands and each of the sub command take additional control arguments. Let us understand the structure by taking the example of `hdfs dfs -ls -l -S -r /public`.  * `hdfs` is the main command to manage all the components of HDFS.  * `dfs` is the sub command to manage files in HDFS.  * `-ls` is the file system command to list files in HDFS.  * `-l -S -r` are control arguments for `-ls` to control the run time behavior of the command.  * `/public` is the argument for the `-ls` command. It is path in HDFS. You will understad as you get into the details.<jupyter_code>%%sh

hadoop
%%sh

hadoop fs -usage
%%sh

hdfs
%%sh

hdfs dfs -usage<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07160647571606475, 'fraction_numerical': 0.0006226650062266501, 'mean_word_length': 4.0854430379746836, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '9767', 'license': 'MIT', 'n_tokens_mistral': 463, 'n_tokens_neox': 441, 'n_words': 263, 'path': '02_overview_of_hdfs/02_using_hdfs_cli.ipynb', 'repo_name': 'itversity/spark-sql', 'size': '1606'}