Skip to content

Fix schema/type inference issue #261

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 22 commits into from
Closed

Fix schema/type inference issue #261

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
a275120
Fix schema/type inference issue #216
tanwanirahul Jan 29, 2016
8b4dbdd
Add testcases for #216
tanwanirahul Jan 29, 2016
b150c55
Adding a simple parse dataset for testing type/schema inference #216
tanwanirahul Jan 30, 2016
9be0313
Bit of refactoring
tanwanirahul Jan 30, 2016
ff96172
Adding an end to end test case for type inference of simple parse dat…
tanwanirahul Jan 30, 2016
d36b4a8
Revert "Bit of refactoring"
tanwanirahul Jan 30, 2016
c20b852
Fix scalastyle issue - max length 100
tanwanirahul Jan 30, 2016
18957e2
Bit of refactoring
tanwanirahul Jan 30, 2016
d41d5fb
Fix scalastyle issue - max length 100
tanwanirahul Jan 30, 2016
9a1f428
Fix scalastyle issue - max length 100
tanwanirahul Jan 30, 2016
80fbac4
Fix type
tanwanirahul Feb 1, 2016
bb510a6
Reverting the refactoring part
tanwanirahul Feb 1, 2016
2c24965
Move end to end test case to CSV suite; Fix indentation comments
tanwanirahul Feb 4, 2016
d6424ef
Merge branch 'master' of github.com:tanwanirahul/spark-csv
tanwanirahul Feb 4, 2016
890fadd
Resolving merge conflicts
tanwanirahul Feb 4, 2016
52cd74f
Resolving merge conflicts
tanwanirahul Feb 4, 2016
6f90a6b
Fix indentation issue
tanwanirahul Feb 4, 2016
3d07d82
Revert build.sbt changes
tanwanirahul Feb 4, 2016
718b467
Fix code style
tanwanirahul Feb 4, 2016
5ef29b8
Indentation
tanwanirahul Feb 5, 2016
6771184
Fix indentation issue
tanwanirahul Feb 10, 2016
7141f35
Stylic corrections
HyukjinKwon Feb 11, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions src/main/scala/com/databricks/spark/csv/util/InferSchema.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,11 @@ private[csv] object InferSchema {
mergeRowTypes)

val structFields = header.zip(rootTypes).map { case (thisHeader, rootType) =>
StructField(thisHeader, rootType, nullable = true)
val dType = rootType match {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, does this not produce Nulltype after merging Nulltypes?
It looks the test Merging Nulltypes should yeild Nulltype is not covering this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@HyukjinKwon Please elaborate. Referring to doc comments in InferSchema.scala

/**

  • Similar to the JSON schema inference.
  • [[org.apache.spark.sql.execution.datasources.json.InferSchema]]
  • 1. Infer type of each row

  • 2. Merge row types to find common type

  • 3. Replace any null types with string type
    */

All null types should be replaced with String types at the end. This is what is happening right now?
If I understand you correctly, you feel there is a need for some more tests to be added. Could you please mention what that test is suppose to test?

As far as testing merging two nulltypes returning null type, is already covered.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, sorry. Please ignore my comment.

case z: NullType => StringType
case other => other
}
StructField(thisHeader, dType, nullable = true)
}

StructType(structFields)
Expand All @@ -62,11 +66,7 @@ private[csv] object InferSchema {
first: Array[DataType],
second: Array[DataType]): Array[DataType] = {
first.zipAll(second, NullType, NullType).map { case ((a, b)) =>
val tpe = findTightestCommonType(a, b).getOrElse(StringType)
tpe match {
case _: NullType => StringType
case other => other
}
findTightestCommonType(a, b).getOrElse(NullType)
}
}

Expand All @@ -93,7 +93,6 @@ private[csv] object InferSchema {
}
}


private def tryParseInteger(field: String): DataType = if ((allCatch opt field.toInt).isDefined) {
IntegerType
} else {
Expand Down Expand Up @@ -152,6 +151,8 @@ private[csv] object InferSchema {
case (t1, t2) if t1 == t2 => Some(t1)
case (NullType, t1) => Some(t1)
case (t1, NullType) => Some(t1)
case (StringType, t2) => Some(StringType)
case (t1, StringType) => Some(StringType)

// Promote numeric types to the highest of the two and all numeric types to unlimited decimal
case (t1, t2) if Seq(t1, t2).forall(numericPrecedence.contains) =>
Expand Down
5 changes: 5 additions & 0 deletions src/test/resources/simple.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
A,B,C,D
1,,,
,1,,
,,1,
,,,1
13 changes: 12 additions & 1 deletion src/test/scala/com/databricks/spark/csv/CsvSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ abstract class AbstractCsvSuite extends FunSuite with BeforeAndAfterAll {
val tempEmptyDir = "target/test/empty/"
val commentsFile = "src/test/resources/comments.csv"
val disableCommentsFile = "src/test/resources/disable_comments.csv"
private val simpleDatasetFile = "src/test/resources/simple.csv"

val numCars = 3

Expand Down Expand Up @@ -658,7 +659,6 @@ abstract class AbstractCsvSuite extends FunSuite with BeforeAndAfterAll {
assert(results.toSeq.map(_.toSeq) === expected)
}


test("Setting comment to null disables comment support") {
val results: Array[Row] = new CsvParser()
.withDelimiter(',')
Expand Down Expand Up @@ -717,6 +717,17 @@ abstract class AbstractCsvSuite extends FunSuite with BeforeAndAfterAll {

assert(results.size === numCars)
}

test("Type/Schema inference works as expected for the simple sparse dataset.") {
val df = new CsvParser()
.withUseHeader(true)
.withInferSchema(true)
.csvFile(sqlContext, simpleDatasetFile)

assert(
df.schema.fields.map(_.dataType).deep ==
Array(IntegerType, IntegerType, IntegerType, IntegerType).deep)
}
}

class CsvSuite extends AbstractCsvSuite {
Expand Down
6 changes: 6 additions & 0 deletions src/test/scala/com/databricks/spark/csv/util/InferSchemaSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,12 @@ class InferSchemaSuite extends FunSuite {
assert(InferSchema.inferField(LongType, "2015-08 14:49:00") == StringType)
}

test("Merging Nulltypes should yeild Nulltype.") {
assert(
InferSchema.mergeRowTypes(Array(NullType),
Array(NullType)).deep == Array(NullType).deep)
}

test("Type arrays are merged to highest common type") {
assert(
InferSchema.mergeRowTypes(Array(StringType),
Expand Down