th. We explain how to generate a jar file with the journal manager and its dependencies, and how to put it into the classpath below.
<property> <name>dfs.namenode.bookkeeperjournal.output-buffer-size</name> <value>1024</value> </property>
<property> <name>dfs.namenode.bookkeeperjournal.ensemble-size</name> <value>3</value> </property>
<property> <name>dfs.namenode.bookkeeperjournal.quorum-size</name> <value>2</value> </property>
<property> <name>dfs.namenode.bookkeeperjournal.digestPw</name> <value>myPassword</value> </property>
<property> <name>dfs.namenode.bookkeeperjournal.zk.session.timeout</name> <value>3000</value> </property>
To generate the distribution packages for BK journal, do the following.
$ mvn clean package -Pdist
This will generate a jar with the BookKeeperJournalManager, all the dependencies needed by the journal manager, hadoop-hdfs/src/contrib/bkjournal/target/hadoop-hdfs-bkjournal-VERSION.jar
Note that the -Pdist part of the build command is important, as otherwise the dependencies would not be packaged in the jar. The dependencies included in the jar are shaded to avoid conflicts with other dependencies of the NameNode.
To run a HDFS namenode using BookKeeper as a backend, copy the bkjournal jar, generated above, into the lib directory of hdfs. In the standard distribution of HDFS, this is at $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/
cp hadoop-hdfs/src/contrib/bkjournal/target/hadoop-hdfs-bkjournal-VERSION.jar $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/
1) NameNode format command will not format the BookKeeper data automatically. We have to clean the data manually from BookKeeper cluster and create the /ledgers/available path in Zookeeper. ---- $ zkCli.sh create /ledgers 0 $ zkCli.sh create /ledgers/available 0 ---- Note: bookkeeper://zk1:2181;zk2:2181;zk3:2181/hdfsjournal The final part /hdfsjournal specifies the znode in zookeeper where ledger metadata will be stored. Administrators may set this to anything they wish.
2) Security in BookKeeper. BookKeeper does not support SASL nor SSL for connections between the NameNode and BookKeeper storage nodes.
3) Auto-Recovery of storage node failures. Work inprogress BOOKKEEPER-237. Currently we have the tools to manually recover the data from failed storage nodes.