FileSystem shell
Overview
The File System (FS) shell includes various shell-like commands that directly interact with the Hadoop Distributed File System (HDFS) as well as other file systems that Hadoop supports, such as Local FS, WebHDFS, S3 FS, and others. The FS shell is invoked by:
$ bin/hadoop fs <args>All FS shell commands take path URIs as arguments.
The URI format is scheme://authority/path.
For HDFS the scheme is hdfs, and for the local FS the scheme is file.
The scheme and authority are optional.
If not specified, the default scheme specified in the configuration is used.
An HDFS file or directory such as /parent/child can be specified as hdfs://namenodehost/parent/child or simply as /parent/child (given that your configuration is set to point to hdfs://namenodehost).
Most of the commands in FS shell behave like corresponding Unix commands. The difference is described for each command. Error information is sent to stderr and the output is sent to stdout.
If HDFS is being used, hdfs dfs is a synonym.
Relative paths can be used. For HDFS, the current working directory is the HDFS home directory /user/<username> that often has to be created manually. The HDFS home directory can also be implicitly accessed, e.g., when using the HDFS trash folder, the .Trash directory in the home directory.
Commands
| Command | Description |
|---|---|
Appends sources from the local file system to the destination file system |
|
Prints the contents of a file to stdout |
|
Returns the checksum information of a file |
|
Changes group association of files |
|
Changes the permissions of files |
|
Changes the owner of files |
|
Identical to the put command |
|
Identical to the get command |
|
Counts the number of directories and files |
|
Copies files from source to destination |
|
Creates a snapshot of a snapshot table directory |
|
Deletes a snapshot from a snapshot table directory |
|
Displays free space |
|
Displays sizes of files and directories |
|
Displays a summary of file lengths |
|
Permanently deletes files in checkpoints older than the retention threshold from the trash directory, and creates a new checkpoint |
|
Finds all files that match the specified expression and applies selected actions to them |
|
Copies files to the local file system |
|
Displays the ACLs of files and directories |
|
Displays the extended attribute names and values (if any) for a file or directory |
|
Takes a source directory and a destination file as inputted and concatenates files into the destination local file |
|
Displays the first kilobyte of a file to stdout |
|
Returns usage output |
|
Returns the file stats or list of its direct children for directory |
|
Recursive version of ls |
|
Takes path URI’s as argument and creates directories |
|
Identical to the put command |
|
Moves a file/directory from HDFS to the local file system |
|
Moves files from source to destination |
|
Copies sources from the local file system to the destination file system |
|
Renames a snapshot |
|
Deletes files specified as args |
|
Deletes a directory |
|
Recursive version of rm |
|
Sets ACLs of files and directories |
|
Sets an extended attribute name and value for a file or directory |
|
Changes the replication factor of a file |
|
Prints statistics about the file/directory in the specified format |
|
Displays last kilobyte of the file to stdout |
|
Performs a test |
|
Takes a source file and outputs the file in a text format |
|
Updates the access and modification times of the file specified by the URI to the current time |
|
Creates a file of zero-length |
|
Truncates all files that match the specified file pattern to the specified length |
|
Prints help for an individual command |