Topology descriptors in Knox
Knox topology
A Knox topology is a cluster-wide configuration file that defines what cluster services are exposed via proxy, how clients can access them, and what security rules are applied to them. All the topologies are stored in the /lib/knox/conf/topologies directory as XML files.
Commonly, a topology file has the following structure:
<topology> (1)
<gateway> (2)
<provider> (3)
<role></role>
<name></name>
...
</provider>
...
</gateway>
<service> (4)
<role></role>
<url></url>
...
</service>
...
</topology>| 1 | A root level that contains all the topology settings. |
| 2 | A root level for providers that define the Knox Gateway behavior when it receives a request. |
| 3 | Provider settings. More information is present in the Providers section. |
| 4 | Service configuration. The <service> blocks contain information about services that are being proxied through Knox Gateway such as URLs or other service-specific parameters. Knox has many services defined out-of-the-box, but if a service you need is not present, you can define it manually and use it like a predefined one. |
Below is an example of a topology file for an ADH cluster that authenticates users via Kerberos and proxies API requests and UI.
<topology>
<generated>true</generated>
<gateway>
<provider>
<role>authentication</role>
<name>HadoopAuth</name>
<enabled>true</enabled>
<param>
<name>config.prefix</name>
<value>hadoop.auth.config</value>
</param>
<param>
<name>hadoop.auth.config.cookie.domain</name>
<value>ru-central1.internal</value>
</param>
<param>
<name>hadoop.auth.config.cookie.path</name>
<value>/</value>
</param>
<param>
<name>hadoop.auth.config.kerberos.keytab</name>
<value>/etc/security/keytabs/HTTP.service.keytab</value>
</param>
<param>
<name>hadoop.auth.config.kerberos.name.rules</name>
<value>DEFAULT</value>
</param>
<param>
<name>hadoop.auth.config.kerberos.principal</name>
<value>HTTP/stikhomirov-adps.ru-central1.internal@AD.RANGER-TEST</value>
</param>
<param>
<name>hadoop.auth.config.signature.secret</name>
<value>password</value>
</param>
<param>
<name>hadoop.auth.config.simple.anonymous.allowed</name>
<value>false</value>
</param>
<param>
<name>hadoop.auth.config.token.validity</name>
<value>1800</value>
</param>
<param>
<name>hadoop.auth.config.type</name>
<value>kerberos</value>
</param>
</provider>
</gateway>
<service>
<role>HDFSUI</role>
<url>http://tsn-adh-2.ru-central1.internal:14000/webhdfs/</url>
</service>
<service>
<role>HIVE</role>
<url>http://tsn-adh-2.ru-central1.internal:10002/</url>
</service>
<service>
<role>NAMENODE</role>
<url>http://tsn-adh-1.ru-central1.internal:9870</url>
<url>http://tsn-adh-3.ru-central1.internal:9870</url>
</service>
<service>
<role>RESOURCEMANAGER</role>
<url>http://tsn-adh-2.ru-central1.internal:8088/ws/</url>
</service>
<service>
<role>WEBHBASE</role>
<url>http://tsn-adh-2.ru-central1.internal:60080/</url>
</service>
<service>
<role>WEBHDFS</role>
<url>http://tsn-adh-2.ru-central1.internal:14000/</url>
</service>
</topology>
Such topology files can be created manually and pushed to Knox via it’s REST API or they can be auto-generated from a topology descriptor that can be configured in the Knox Admin UI.
Providers
Providers define how the requests are processed by Knox Gateway. There are providers available for the following use cases:
-
Authentication. Knox supports many authentication methods: LDAP, PAM, Kerberos, SSO, SSO Cookie, JSON Web Tokens, CAS, OAuth, SAML, OpenID Connect, as well as anonymous authentication.
-
Authorization. You can restrict certain users, groups, or hosts from accessing services by setting up ACLs.
-
Identity assertion. Such providers allow you to impersonate users or adjust their names into an expected format before forwards a request to an actual backend service.
-
Web application security. You can opt in to use the following technologies while proxying an incoming request to actual cluster services:
-
Cross-Site Request Forgery
-
Cross-Origin Resource Sharing
-
X-Frame-Options
-
X-Content-Type-Options
-
Strict Transport Security
-
X-XSS-Protection
-
-
High availability. With this provider, you can manually specify the failover options for certain services.
-
Host mapping. This provider can be used to dynamically adjust a hostname, port, or scheme of backend service URLs before forwarding requests. This might be useful for the load balancing scenarios.
Topology descriptor
A topology descriptor is a JSON-file from which a topology file can be automatically generated. Such files are located in the /lib/knox/conf/descriptors directory.
A common structure of a descriptor file is presented below.
{
"provider-config-ref": <provider_name>, (1)
"services": [ (2)
{
"name": <service_name>,
"urls": [
<service_urls>
],
...
},
...
]
}| 1 | Name of a provider that is configured in Knox. |
| 2 | A list of cluster services to be proxied through Knox Gateway. |
Below is an example of a topology descriptor file for an ADH cluster. A topology from the example above was automatically generated from this provider by Knox.
{
"provider-config-ref": "my-provider",
"services": [
{
"name": "HIVE",
"urls": [
"http://tsn-adh-2.ru-central1.internal:10002/"
]
},
{
"name": "RESOURCEMANAGER",
"urls": [
"http://tsn-adh-2.ru-central1.internal:8088/ws/"
]
},
{
"name": "WEBHBASE",
"urls": [
"http://tsn-adh-2.ru-central1.internal:60080/"
]
},
{
"name": "NAMENODE",
"urls": [
"http://tsn-adh-1.ru-central1.internal:9870",
"http://tsn-adh-3.ru-central1.internal:9870"
]
},
{
"name": "HDFSUI",
"urls": [
"http://tsn-adh-2.ru-central1.internal:14000/webhdfs/"
]
},
{
"name": "WEBHDFS",
"urls": [
"http://tsn-adh-2.ru-central1.internal:14000/"
]
}
]
}