stuartjandrews.com

Hadoop Filesystem Tools

If you're like me, you get frustrated by the amount of typing that is required to copy a file from your Hadoop filesystem to your local filesystem, e.g.:

  hdfs dfs -get hdfs://xxx/very/long/path/to/a/file \
      /yyy/very/long/path/to/a/file

Also, if you are like me, you want the directory structures of the two filesystems to be mirror-images. This means you typically have to type a common path component twice, which is redundant, time consuming, and error prone.

To address this issue (and to exercise my Bash scripting skills), I hacked together a collection of shell scripts that automate this process, together called HDFS-Tools. The HDFS-Tools simplify the management of files in your Hadoop Filesystem by helping to synchronize a local copy of the filesystem with HDFS.

How Does It Work?

To enable HDFS-Tools, one must first designate a directory to hold the root of the local copy; this is done by setting the HDFS_PREFIX environment variable. Paths relative to HDFS_PREFIX in the local copy are the same as in HDFS.

Once this is done, copying data between HDFS and your local copy is simply a matter of getting or putting a file; e.g.:

  hget <path>

HDFS-Tools deals with the task of expanding the path arguments to create the conventional command format, using the HDFS_PREFIX and your HDFS's configuration. Furthermore, with some code from rapleaf's dev blog, these commands have been augmented with filename auto-completion. Together, these features make hget, hput, etc., more convenient than using:

  hdfs dfs -get <hdfs_path> <local_path>

Say goodbye to the frustration of typing long paths in HDFS. Indeed, you rarely need to type more than the commands themselves.

Filename Auto-Completion

Auto-completion is available for hls, hget, and hput, by pressing <TAB>. There may be a delay before results are displayed, as the query to the remote HDFS is issued. When the CWD is below HDFS_PREFIX, filename auto-completion displays paths relative to CWD; otherwise, they are relative to HDFS_PREFIX. In the later case, the paths are displayed with a / prefix.

Auto-completion for directories is a little clunky because a space character is appended to the result. In order to extend the path further, you must type <backspace><TAB>.

Details

HDFS-Tools consists of the following:

hpwd : List corresponding path in HDFS. When the current working directory resides under HDFS_PREFIX, the hpwd command lists the corresponding location in HDFS. The result has the form: hdfs://host/path. The command hpwd -r lists only the path component, while hpwd -p lists only the hdfs://host/ component.

hls : List files from HDFS. hls [path ..] lists files from HDFS that correspond to path; e.g. hdfs://host/[path ..]. When the current working directory resides under HDFS_PREFIX, the path is relative to it; e.g. hdfs://host/CWD/[path ..]. A recursive directory listing is produced with a -r flag.

hget : Retrieve files from HDFS. hget [path ..] copies the corresponding files from HDFS to the local filesystem. Directories will not be created unless the -p flag is present. Local files will not be overwritten, unless the -f flag is included.

hput : Copy files to HDFS. hput [path ..] copies local files to the corresponding locations in HDFS. HDFS files will not be overwritten, unless the -f flag is included.

hconnect : Connect to a remote HDFS. hconnect opens or closes an ssh tunnel for communication with remote HDFS.

henv : This is a configuration script for HDFS-Tools auto-completion.

Notes

  • Use option -h to display help for a command, and -v for extra debugging information.
  • When the current working directory is outside of HDFS_PREFIX, HDFS-Tools behave as though they have been invoked with the current working directory set to HDFS_PREFIX.
  • One drawback of HDFS-Tools is that filename globbing is not supported, so you can not do things like hget '[io]*'.

Installation & Setup

HDFS-Tools are available on GitHub.

Note: HDFS-Tools are configured for use with Hadoop 0.21.0.

Bare Minimum

  1. Install these scripts somewhere on your path
  2. HDFS_PREFIX - Select the local directory where you wish to mirror HDFS
  3. HADOOP_CONF_DIR - Select the directory containing the active configuration, in order to lookup information on HDFS

  4. Add the following line to your .bash_profile

      source <HDFS-TOOLS>/henv

For Remote Connections

  1. HDFS_USER - Set the user name used to connect to the remote hadoop filesystem
  2. HDFS_HOST - Set the host
  3. HDFS_PORT - Set the port

hconnect opens an ssh tunnel to the remote host using ssh -ND $HDFS_PORT $HDFS_USER@$HDFS_HOST

Examples Part 1

The first set of examples demonstrate the behavior of HDFS-Tools with CWD=HDFS_PREFIX, where HDFS_PREFIX=~/Data/Hdfs-2011-08-28.

List Files

  1. -> hls

      Found 3 items
  drwxr-xr-x   - stu supergroup          0 2011-09-03 21:50 /Users
  drwxr-xr-x   - stu supergroup          0 2011-09-03 21:51 /jobtracker
  drwxr-xr-x   - stu supergroup          0 2011-09-03 21:51 /user

  2. -> hls -v user/stu

      HDFS_PREFIX=/Users/stu/Data/Hdfs-2011-08-28
  HDFS_PWD=
  HDFS_URL=/user/stu/input/hdfs-site.xml
  Found 2 items
  drwxr-xr-x   - stu supergroup          0 2011-09-03 21:45 /user/stu/input
  drwxr-xr-x   - stu supergroup          0 2011-09-03 21:51 /user/stu/output

  3. -> hls -v not/a/valid/file

      HDFS_PREFIX=/Users/stu/Data/Hdfs-2011-08-28
  HDFS_PWD=
  HDFS_URL=not/a/valid/file
  ls: Cannot access hdfs://localhost:9000//not/a/valid/file: No such file or directory.

Get Files

  1. -> hget /user/stu/output

      hget > Local path already exists /Users/stu/Data/Hdfs-2011-08-28/user/stu/output/

  2. -> hget -vf /user/stu/output

      hget > Local path already exists /Users/stu/Data/Hdfs-2011-08-28/user/stu/output/
  HDFS_PREFIX=/Users/stu/Data/Hdfs-2011-08-28
  HDFS_PWD=
  HDFS_URL=user/stu/output/
  LOCAL_URL=/Users/stu/Data/Hdfs-2011-08-28/user/stu/output/
  LOCAL_DIR=/Users/stu/Data/Hdfs-2011-08-28/user/stu

Put Files

  1. -> hput /user/stu/output

      put: Target hdfs://localhost:9000/user/stu/output is a directory

  2. -> hput -vf /user/stu/output

      HDFS_PREFIX=/Users/stu/Data/Hdfs-2011-08-28
  HDFS_PWD=
  HDFS_URL=user/stu/output
  LOCAL_URL=/Users/stu/Data/Hdfs-2011-08-28/user/stu/output
  HDFS_DIR=user/stu

Tab Completion

  1. -> hls <TAB>

      Users       jobtracker  user
  -> hls *

  2. -> hget u<TAB>

      -> hget user/stu *

  3. -> hput user/stu<TAB>

      /user/stu/input   /user/stu/output
  -> hput /user/stu/ *

  4. -> hput user/stu/<TAB>

      /user/stu/input   /user/stu/output
  -> hput /user/stu/*

Examples Part 2

When the CWD is located below HDFS_PREFIX, HDFS-Tools use relative paths. For example, with CWD=$(HDFS_PREFIX)/user/stu

  1. -> hget <TAB> 

      input   output
  -> hget *

Examples Part 3

When the CWD is not below HDFS_PREFIX, HDFS-Tools behave as though they were involked from HDFS_PREFIX. The only difference is that paths on the command line are prefixed with /. In this case, we are using CWD=~

  1. -> hls 

      Found 3 items
  drwxr-xr-x   - stu supergroup          0 2011-09-03 21:50 /Users
  drwxr-xr-x   - stu supergroup          0 2011-09-03 21:51 /jobtracker
  drwxr-xr-x   - stu supergroup          0 2011-09-03 21:51 /user

  2. -> hls <TAB> 

      /Users       /jobtracker  /user
  -> hls /*

  3. -> hput /use<TAB> 

      -> hput /user/ *

  4. -> hget /user/stu/input 

      hget > Local path already exists /Users/stu/Data/Hdfs-2011-08-28/user/stu/input

Examples Part 4

  1. -> hconnect -c

      ENABLED:  0
  RUNNING PROCESS:

  2. -> hconnect -t

      ENABLED:  0
  PID:
    ssh -ND 2600 user@email.com
  Started HDFS tunnel with PID: '7647'

  3. -> hconnect -c

      ENABLED:  1
  RUNNING PROCESS:  7647 ssh -ND 2600 user@email.com

  4. -> hconnect

      ENABLED:  1
  PID:  7647
  Stopping HDFS tunnel with PID: '7647'
  kill -9 7647

Learn More

blog comments powered by Disqus

Tags

Published

06 September 2011

by Stuart Andrews @tub78 (github) @tub78 (twitter) [About this blog]