Use quotas
In HDFS, the administrator can limit the size and population of directories by using quotas. There are three types of quotas that can be set:
-
Name quotas — limit the amount of names in the directory.
-
Space quotas — limit the disk space used by the directory.
-
Storage type quotas — limit the disk space of a specific type of storage.
All types of quotas work independently, which means you can configure all three types of quotas at the same time.
The configured quotas are written into the fsimage. When a NameNode starts, it prints out any existing quota violations. Setting or removing a quota creates a journal entry in the EditLogs.
Name quotas
The name quota is a limitation on the number of names in a directory, including the directory itself. If a new file or directory is created or renamed inside the quoted catalog, it will count to the quota.
When the quota has been filled, any file creation or rename operations will fail.
To set a name quota for a directory, use the command:
$ hdfs dfsadmin -setQuota <N> <directory>...<directory>Where:
-
<N>— a positive integer representing the number of names allowed in the directory; -
<directory>— a path to the directory for which to set the name quota.
To remove the name quota for a directory, use the command:
$ hdfs dfsadmin -clrQuota <directory>...<directory>Where <directory> is the path to the directory for which to remove the quotas.
Space quotas
The space quota is a limitation on the directory size. The quota is measured in blocks, including their replicas. This means that changing the replication rate will affect the quota. For example, 1 GB of data with replication rate of 3 consumes 3 GB of quota.
Since there’s no block creation, the following does not count against the quota:
-
creating a directory;
-
creating a file (if the quota is full, the file will be created but it’s going to be empty);
-
hosting the metadata.
To set the space quota for a directory, use the command:
$ hdfs dfsadmin -setSpaceQuota <N> <directory>...<directory>Where:
-
<N>— a positive integer representing the size limit for the directory in bytes. You can specify size with a binary prefix for convenience, for example: 50g for 50 GB and 2t for 2 TB etc. -
<directory>— the path to the directory for which to set the space quota.
To remove the space quota for a directory, use the command:
$ hdfs dfsadmin -clrSpaceQuota <directory>...<directory>Where <directory> is the path to the directory for which to remove the space quota.
Storage type quotas
The storage type quotas limit how much of a specific disk space can be used by a directory. When counting the occupied space, this type of quotas work similarly to the space quotas.
Before setting a storage type quota, make sure you have configured the storage policy for the directory. For information on storage policies, refer to Storage policies.
To set the storage type quota for a directory, use the command:
$ hdfs dfsadmin -setSpaceQuota <N> -storageType <storagetype> <directory>...<directory>Where:
-
<N>— a positive integer representing the space limit of the specified storage type for the directory in bytes. This limit applies to all the files under the directory tree. You can specify size with a binary prefix for convenience, for example: 50g for 50 GB and 2t for 2 TB etc. -
<storagetype>— the type of storage to put the limit on. Available storage type values areRAM_DISK,DISK,SSD,ARCHIVE. -
<directory>— the path to the directory for which to set the storage type quota.
To remove the storage type quota for a directory, use the command:
$ hdfs dfsadmin -clrSpaceQuota -storageType <storagetype> <directory>...<directory>-
<storagetype>— the type of storage to take the limit off. Available storage type values areRAM_DISK,DISK,SSD,ARCHIVE. -
<directory>— the path to the directory for which to remove the storage type quota.