frameworkuploader

The command collects framework JARs and uploads them to HDFS as a tarball.

The usage is as follows:

$ mapred frameworkuploader -target <target> args
Arguments

-input <classpath>

The input classpath that is searched for JAR files to be included in the tarball

-fs <filesystem>

The target file system. Defaults to the default filesystem set by fs.defaultFS

-target <target>

The target location to store the framework tarball, optionally followed by a # with the localized alias. An example is /usr/lib/framework.tar#framework. Make sure the target directory is readable by all users and is not writable by others than administrators to ensure cluster security

-blacklist <list>

Specifies a comma-separated array of regexps to filter JAR file names to exclude from the classpath. For example, this can be used to exclude test JARs or Hadoop services that are not necessary to localize

-whitelist <list>

Specifies a comma-separated array of regexps to include specific JAR files. This can be used to provide additional security, so that no external source can include malicious code in the classpath when the tool runs

-nosymlink

Used to exclude symlinks that point to the same directory. For example, the path /a/foo.jar and a symlink /a/bar.jar that points to /a/foo.jar would normally add foo.jar and bar.jar to the tarball as separate files despite them actually being the same file. This flag would make the tool exclude /a/bar.jar so only one copy of the file is added

-initialReplication <num>

The replication number that is used to create the framework tarball. It is safe to leave this value as the default 3

-finalReplication <num>

The uploader tool sets the replication factor once all blocks are collected and uploaded. If a quick initial startup is required, then it is recommended to set this value equal to the number of commissioned nodes divided by two but less than 512

-acceptableReplication <num>

The tool waits until the tarball gets replicated by this number of times before exiting. The value should be a replication number less than or equal to the value of finalReplication. An optimal value is 90% of the finalReplication value to accommodate failing nodes

-timeout <seconds>

A timeout to wait to reach acceptableReplication before the tool exits (in seconds). The tool logs an error otherwise and returns

Example:

$ mapred frameworkuploader -timeout 10
Found a mistake? Seleсt text and press Ctrl+Enter to report it