Skip to content

Commit 49bea5a

Browse files
seancxmaodongjoon-hyun
authored andcommitted
[SPARK-25833][SQL][DOCS] Update migration guide for Hive view compatibility
## What changes were proposed in this pull request? Both Spark and Hive support views. However in some cases views created by Hive are not readable by Spark. For example, if column aliases are not specified in view definition queries, both Spark and Hive will generate alias names, but in different ways. In order for Spark to be able to read views created by Hive, users should explicitly specify column aliases in view definition queries. Given that it's not uncommon that Hive and Spark are used together in enterprise data warehouse, this PR aims to explicitly describe this compatibility issue to help users troubleshoot this issue easily. ## How was this patch tested? Docs are manually generated and checked locally. ``` SKIP_API=1 jekyll serve ``` Closes apache#22868 from seancxmao/SPARK-25833. Authored-by: seancxmao <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 9cf9a83 commit 49bea5a

File tree

1 file changed

+15
-0
lines changed

1 file changed

+15
-0
lines changed

docs/sql-migration-guide-hive-compatibility.md

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,21 @@ Spark SQL supports the vast majority of Hive features, such as:
5151
* Explain
5252
* Partitioned tables including dynamic partition insertion
5353
* View
54+
* If column aliases are not specified in view definition queries, both Spark and Hive will
55+
generate alias names, but in different ways. In order for Spark to be able to read views created
56+
by Hive, users should explicitly specify column aliases in view definition queries. As an
57+
example, Spark cannot read `v1` created as below by Hive.
58+
59+
```
60+
CREATE VIEW v1 AS SELECT * FROM (SELECT c + 1 FROM (SELECT 1 c) t1) t2;
61+
```
62+
63+
Instead, you should create `v1` as below with column aliases explicitly specified.
64+
65+
```
66+
CREATE VIEW v1 AS SELECT * FROM (SELECT c + 1 AS inc_c FROM (SELECT 1 c) t1) t2;
67+
```
68+
5469
* All Hive DDL Functions, including:
5570
* `CREATE TABLE`
5671
* `CREATE TABLE AS SELECT`

0 commit comments

Comments
 (0)