<jupyter_start><jupyter_text>Overriding PropertiesLet us understand how we can override the properties while running `hdfs dfs` or `hadoop fs` commands.<jupyter_code>%%HTML
<iframe width="560" height="315" src="https://www.youtube.com/embed/wsOqy0mYnM0?rel=0&amp;controls=1&amp;showinfo=0" frameborder="0" allowfullscreen></iframe><jupyter_output><empty_output><jupyter_text>* We can change any property which is not defined as final in **core-site.xml** or **hdfs-site.xml**.* We can change `blocksize` as well as `replication` while copying the files. We can also change them after copying the files as well.* We can either pass individual properties using `-D` or bunch of properties by passing xml similar to **core-site.xml** or **hdfs-site.xml** as part of `--conf`.* Let's copy a file **/data/crime/csv/rows.csv** with default values. The file is splitted into 12 blocks with 2 copies each (as our default blocksize is 128 MB and replication factor is 2).<jupyter_code>%%sh

hdfs dfs -ls /user/${USER}/crime
%%sh

hdfs dfs -rm -R -skipTrash /user/${USER}/crime
%%sh

hdfs dfs -mkdir -p /user/${USER}/crime/csv
%%sh

ls -lhtr /data/crime/csv
%%sh

hdfs dfs -put /data/crime/csv/rows.csv /user/${USER}/crime/csv
%%sh

hdfs dfs -stat %r /user/${USER}/crime/csv/rows.csv
%%sh

hdfs dfs -stat %o /user/${USER}/crime/csv/rows.csv
%%sh

hdfs dfs -stat %b /user/${USER}/crime/csv/rows.csv
%%sh

hdfs dfs -rm -R -skipTrash /user/${USER}/crime/csv/rows.csv
%%sh

hdfs dfs -Ddfs.blocksize=64M -Ddfs.replication=3 -put /data/crime/csv/rows.csv /user/${USER}/crime/csv
%%sh

hdfs dfs -stat %r /user/${USER}/crime/csv/rows.csv
%%sh

hdfs dfs -stat %o /user/${USER}/crime/csv/rows.csv
%%sh

hdfs dfs -stat %b /user/${USER}/crime/csv/rows.csv
%%sh

ls -ltr /etc/hadoop/conf/
%%sh

cat /etc/hadoop/conf/hdfs-site.xml<jupyter_output><configuration>
    
    <property>
      <name>dfs.block.access.token.enable</name>
      <value>true</value>
    </property>
    
    <property>
      <name>dfs.blockreport.initialDelay</name>
      <value>120</value>
    </property>
    
    <property>
      <name>dfs.blocksize</name>
      <value>134217728</value>
    </property>
    
    <property>
      <name>dfs.client.read.shortcircuit</name>
      <value>true</value>
    </property>
    
    <property>
      <name>dfs.client.read.shortcircuit.streams.cache.size</name>
      <value>4096</value>
    </property>
    
    <property>
      <name>dfs.client.retry.policy.enabled</name>
      <value>false</value>
    </property>
    
    <property>
      <name>dfs.cluster.administrators</name>
      <value> hdfs</value>
    </property>
    
    <property>
      <name>dfs.content-summary.limit</name>
      <value>5000</value>
    </property>
    
    <property>
      <name>dfs.datanode.address</name>
      <value>0.0.0.0:50010</value[...]
{'fraction_non_alphanumeric': 0.1631858407079646, 'fraction_numerical': 0.018053097345132742, 'mean_word_length': 3.2242152466367715, 'pattern_counts': {'":': 0, '<': 64, '<?xml version=': 0, '>': 63, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 6}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '580024', 'license': 'MIT', 'n_tokens_mistral': 1077, 'n_tokens_neox': 1035, 'n_words': 238, 'path': '02_overview_of_hdfs/16_overriding_properties.ipynb', 'repo_name': 'itversity/spark-sql', 'size': '2825'}