Skip to content

Commit ee3f11d

Browse files
drothery-edbdrothery-edb
authored
Merge pull request #1480 from EnterpriseDB/content/hadoop_data_adapter/2.0.8/upcoming_release
Content for upcoming Hadoop FDW release 2.0.8 Former-commit-id: 6189034
2 parents 2c9a821 + 37c1867 commit ee3f11d

25 files changed

+1410
-15
lines changed

‎product_docs/docs/hadoop_data_adapter/2.0.7/02_requirements_overview.mdx

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -12,14 +12,14 @@ The Hadoop Foreign Data Wrapper is supported on the following platforms:
1212

1313
**Linux x86-64**
1414

15-
> - RHEL 8.x and 7.x
16-
> - CentOS 8.x and 7.x
17-
> - OL 8.x and 7.x
18-
> - Ubuntu 20.04 and 18.04 LTS
19-
> - Debian 10.x and 9.x
15+
- RHEL 8.x and 7.x
16+
- CentOS 8.x and 7.x
17+
- OL 8.x and 7.x
18+
- Ubuntu 20.04 and 18.04 LTS
19+
- Debian 10.x and 9.x
2020

2121
**Linux on IBM Power8/9 (LE)**
2222

23-
> - RHEL 7.x
23+
- RHEL 7.x
2424

2525
The Hadoop Foreign Data Wrapper supports use of the Hadoop file system using a HiveServer2 interface or Apache Spark using the Spark Thrift Server.

‎product_docs/docs/hadoop_data_adapter/2.0.7/05_installing_the_hadoop_data_adapter.mdx

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -293,23 +293,23 @@ The following steps will walk you through on using the EDB apt repository to ins
293293

294294
On Debian 9 and Ubuntu:
295295

296-
> ```text
297-
> sh -c 'echo "deb https://username:[email protected]/$(lsb_release -cs)-edb/ $(lsb_release -cs) main" > /etc/apt/sources.list.d/edb-$(lsb_release -cs).list'
298-
> ```
296+
```text
297+
sh -c 'echo "deb https://username:[email protected]/$(lsb_release -cs)-edb/ $(lsb_release -cs) main" > /etc/apt/sources.list.d/edb-$(lsb_release -cs).list'
298+
```
299299

300300
On Debian 10:
301301

302302
1. Set up the EDB repository:
303303

304-
> ```text
305-
> sh -c 'echo "deb [arch=amd64] https://apt.enterprisedb.com/$(lsb_release -cs)-edb/ $(lsb_release -cs) main" > /etc/apt/sources.list.d/edb-$(lsb_release -cs).list'
306-
> ```
304+
```text
305+
sh -c 'echo "deb [arch=amd64] https://apt.enterprisedb.com/$(lsb_release -cs)-edb/ $(lsb_release -cs) main" > /etc/apt/sources.list.d/edb-$(lsb_release -cs).list'
306+
```
307307

308308
1. Substitute your EDB credentials for the `username` and `password` in the following command:
309309

310-
> ```text
311-
> sh -c 'echo "machine apt.enterprisedb.com login <username> password <password>" > /etc/apt/auth.conf.d/edb.conf'
312-
> ```
310+
```text
311+
sh -c 'echo "machine apt.enterprisedb.com login <username> password <password>" > /etc/apt/auth.conf.d/edb.conf'
312+
```
313313

314314
3. Add support to your system for secure APT repositories:
315315

‎product_docs/docs/hadoop_data_adapter/2.0.8/01_whats_new.mdx

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
title: "What’s New"
3+
---
4+
5+
<div id="whats_new" class="registered_link"></div>
6+
7+
The following features are added to create Hadoop Foreign Data Wrapper `2.0.8`:
8+
9+
- Support for Hadoop version 3.2.x
10+
- Support for Hive version 3.1.x
11+
- Support for Spark version 3.0.x

‎product_docs/docs/hadoop_data_adapter/2.0.8/02_requirements_overview.mdx

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
---
2+
title: "Requirements Overview"
3+
---
4+
5+
## Supported Versions
6+
7+
The Hadoop Foreign Data Wrapper is certified with EDB Postgres Advanced Server 9.6 and above.
8+
9+
## Supported Platforms
10+
11+
The Hadoop Foreign Data Wrapper is supported on the following platforms:
12+
13+
**Linux x86-64**
14+
15+
- RHEL 8.x and 7.x
16+
- CentOS 8.x and 7.x
17+
- OL 8.x and 7.x
18+
- Ubuntu 20.04 and 18.04 LTS
19+
- Debian 10.x and 9.x
20+
21+
**Linux on IBM Power8/9 (LE)**
22+
23+
- RHEL 7.x
24+
25+
The Hadoop Foreign Data Wrapper supports use of the Hadoop file system using a HiveServer2 interface or Apache Spark using the Spark Thrift Server.

‎product_docs/docs/hadoop_data_adapter/2.0.8/03_architecture_overview.mdx

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
---
2+
title: "Architecture Overview"
3+
---
4+
5+
<div id="architecture_overview" class="registered_link"></div>
6+
7+
Hadoop is a framework that allows you to store a large data set in a distributed file system.
8+
9+
The Hadoop data wrapper provides an interface between a Hadoop file system and a Postgres database. The Hadoop data wrapper transforms a Postgres `SELECT` statement into a query that is understood by the HiveQL or Spark SQL interface.
10+
11+
![Using a Hadoop distributed file system with Postgres](images/hadoop_distributed_file_system_with_postgres.png)
12+
13+
When possible, the Foreign Data Wrapper asks the Hive or Spark server to perform the actions associated with the `WHERE` clause of a `SELECT` statement. Pushing down the `WHERE` clause improves performance by decreasing the amount of data moving across the network.

‎product_docs/docs/hadoop_data_adapter/2.0.8/04_supported_authentication_methods.mdx

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
---
2+
title: "Supported Authentication Methods"
3+
---
4+
5+
<div id="supported_authentication_methods" class="registered_link"></div>
6+
7+
The Hadoop Foreign Data Wrapper supports `NOSASL` and `LDAP` authentication modes. To use `NOSASL`, do not specify any `OPTIONS` while creating user mapping. For `LDAP` authentication mode, specify `username` and `password` in `OPTIONS` while creating user mapping.
8+
9+
## Using LDAP Authentication
10+
11+
When using the Hadoop Foreign Data Wrapper with `LDAP` authentication, you must first configure the `Hive Server` or `Spark Server` to use LDAP authentication. The configured server must provide a `hive-site.xml` file that includes the connection details for the LDAP server. For example:
12+
13+
```text
14+
<property>
15+
<name>hive.server2.authentication</name>
16+
<value>LDAP</value>
17+
<description>
18+
Expects one of [nosasl, none, ldap, kerberos, pam, custom].
19+
Client authentication types.
20+
NONE: no authentication check
21+
LDAP: LDAP/AD based authentication
22+
KERBEROS: Kerberos/GSSAPI authentication
23+
CUSTOM: Custom authentication provider
24+
(Use with property hive.server2.custom.authentication.class)
25+
PAM: Pluggable authentication module
26+
NOSASL: Raw transport
27+
</description>
28+
</property>
29+
<property>
30+
<name>hive.server2.authentication.ldap.url</name>
31+
<value>ldap://localhost</value>
32+
<description>LDAP connection URL</description>
33+
</property>
34+
<property>
35+
<name>hive.server2.authentication.ldap.baseDN</name>
36+
<value>ou=People,dc=itzgeek,dc=local</value>
37+
<description>LDAP base DN</description>
38+
</property>
39+
```
40+
41+
Then, when starting the hive server, include the path to the `hive-site.xml` file in the command. For example:
42+
43+
```text
44+
./hive --config path_to_hive-site.xml_file --service hiveServer2
45+
```
46+
47+
Where *path_to_hive-site.xml_file* specifies the complete path to the `hive‑site.xml` file.
48+
49+
When creating the user mapping, you must provide the name of a registered LDAP user and the corresponding password as options. For details, see [Create User Mapping](08_configuring_the_hadoop_data_adapter/#create-user-mapping).
50+
51+
<div id="using_nosasl_authentication" class="registered_link"></div>
52+
53+
## Using NOSASL Authentication
54+
55+
When using `NOSASL` authentication with the Hadoop Foreign Data Wrapper, set the authorization to `None`, and the authentication method to `NOSASL` on the `Hive Server` or `Spark Server`. For example, if you start the `Hive Server` at the command line, include the `hive.server2.authentication` configuration parameter in the command:
56+
57+
```text
58+
hive --service hiveserver2 --hiveconf hive.server2.authentication=NOSASL
59+
```

0 commit comments

Comments
 (0)