diff --git a/build.sbt b/build.sbt index 1d7d54731a..66e5bb7a0f 100644 --- a/build.sbt +++ b/build.sbt @@ -572,12 +572,13 @@ lazy val `kyo-schema` = .withoutSuffixFor(JVMPlatform) .crossType(CrossType.Full) .dependsOn(`kyo-data` % "test->test;compile->compile") + .dependsOn(`kyo-core` % "test->compile") .in(file("kyo-schema")) .withKyoTest .settings(`kyo-settings`) .jvmSettings(mimaCheck(false)) .nativeSettings(`native-settings`) - .jsSettings(`js-settings`) + .jsSettings(`js-settings`, Test / scalaJSLinkerConfig ~= (_.withModuleKind(ModuleKind.CommonJSModule))) .wasmSettings(`wasm-settings`) lazy val `kyo-core` = diff --git a/kyo-schema/README.md b/kyo-schema/README.md index a6f10ae082..8a29649c4a 100644 --- a/kyo-schema/README.md +++ b/kyo-schema/README.md @@ -31,13 +31,13 @@ Schema[User].focus(_.address.city).update(alice)(_.toUpperCase) Everything flows from `Schema[A]`, the central type that captures a type's structure at compile time. It's the single source of truth that powers serialization, validation, navigation, and conversion. -The serialization format is chosen at the call site, not baked into the type. `Json.encode(value)` and `Protobuf.encode(value)` summon the `Schema[A]` from implicit scope; a schema you reshaped or enriched only takes effect when you encode through that instance with `s.encode[Json](value)`. +The serialization format is chosen at the call site, not baked into the type. `Json.encode(value)`, `Ion.encode(value)`, and `Protobuf.encode(value)` summon the `Schema[A]` from implicit scope; a schema you reshaped or enriched only takes effect when you encode through that instance with `s.encode[Json](value)`. These are the top-level entry points: | Entry point | Purpose | |-------------|---------| -| `Json` / `Yaml` / `Protobuf` | Serialize to JSON strings, YAML documents, or Protocol Buffers bytes | +| `Json` / `Ion` / `Yaml` / `Protobuf` | Serialize to JSON strings, Ion text, YAML documents, or Protocol Buffers bytes | | `Focus` | Type-safe lens for reading, writing, and updating fields at any depth | | `Compare` | Read-only field-by-field comparison of two values | | `Modify` | Batched field mutations applied as a single unit | @@ -171,6 +171,40 @@ Json.decode[User](untrustedInput, maxDepth = 64, maxCollectionSize = 10000) Exceeding either limit returns `Result.Failure(LimitExceededException)`. `LimitExceededException` is a subtype of `DecodeException`, so the same pattern-match handles malformed input and limit breaches. +### Ion + +`Ion.encode` converts a value to Amazon Ion text. Case classes become structs, collections become lists, `Map[String, V]` becomes a struct, and `Span[Byte]` becomes an Ion blob: + +```scala +val ion: String = Ion.encode(alice) +// {id:1,name:"Alice",email:"alice@example.com",password:"secret",address:{city:"Portland",zip:"97201"}} + +Ion.decode[User](ion) +// Result.Success(alice) + +Ion.encode(Span.from("hello".getBytes("UTF-8"))) +// {{aGVsbG8=}} +``` + +The reader accepts the Ion text features most useful for schema-shaped data: unquoted or quoted field names, comments, annotations, typed nulls, blobs, long strings, and symbol values decoded as strings: + +```scala +Ion.decode[User]( + """user::{ + | id: 1, + | name: "Alice", + | email: "alice@example.com", + | password: "secret", + | address: {city: Portland, zip: "97201"}, + |}""".stripMargin +) +// Result.Success(alice) +``` + +Ion type annotations are accepted as input syntax and ignored as metadata during schema decoding. They are not preserved by `Ion.decode` or emitted by `Ion.encode`. + +`Ion.decode` and `Ion.decodeBytes` accept the same `maxDepth` and `maxCollectionSize` safety limits as `Json.decode`. + ### YAML `Yaml.decode` parses one YAML document into a typed value and returns `Result[DecodeException, A]`. For document streams, use `Yaml.decodeAll`, or pass `Yaml.DocumentIndex(n)` to target one zero-based document without decoding the whole stream. Use `Yaml.ReaderConfig` when you need document selection, stream-fragment merging, decode limits, or YAML 1.1 scalar resolution for legacy systems. @@ -1011,7 +1045,7 @@ The `Structure.Type` tree ships with a small set of operations for runtime inspe ## Custom Formats -`Json` and `Protobuf` are the built-in formats, but the serialization pipeline itself is format-agnostic. A schema describes a value as a sequence of typed events (`objectStart`, `field`, `int`, `arrayStart`, ...) and a matching sequence on the way back. A format is the code that turns those events into bytes and back. +`Json`, `Ion`, `Yaml`, and `Protobuf` are the built-in formats, but the serialization pipeline itself is format-agnostic. A schema describes a value as a sequence of typed events (`objectStart`, `field`, `int`, `arrayStart`, ...) and a matching sequence on the way back. A format is the code that turns those events into bytes and back. ### The Codec trait @@ -1077,7 +1111,7 @@ Schema[User].encode(alice)(using Lines) // Span[Byte] in the Lines format Schema[User].decode(bytes)(using Lines) // Result[DecodeException, User] ``` -For a complete example, read `JsonWriter` and `JsonReader` (or their Protobuf counterparts) in the same package: they implement the full contract. +For a complete example, read `JsonWriter` and `JsonReader`, `IonWriter` and `IonReader`, or their Protobuf counterparts in the same package: they implement the full contract. When writing a custom schema for an opaque or wrapper type, you can also construct a `Schema` instance directly using the public factories `Schema.init` (for plain schemas) and `Schema.initFocused` (when you need to track the focused type member). Both take inlined `writeFn` and `readFn` lambdas, plus an optional `getterFn`/`setterFn` pair for lens support. Abstract members must be supplied (including `fieldParse`, `matchField`, `lastFieldName`, and `captureValue`); optional overrides like `fieldBytes`, `initFields`, `clearFields`, `droppedFieldsMask`, and `release` are where real codecs recover allocation-sensitive performance. diff --git a/kyo-schema/shared/src/main/scala/kyo/Codec.scala b/kyo-schema/shared/src/main/scala/kyo/Codec.scala index 1a1c758aeb..671f0e2a44 100644 --- a/kyo-schema/shared/src/main/scala/kyo/Codec.scala +++ b/kyo-schema/shared/src/main/scala/kyo/Codec.scala @@ -9,7 +9,8 @@ import java.nio.charset.StandardCharsets * * - Pluggable: implement `newWriter` and `newReader` to support any binary or text format * - Used by [[kyo.Schema]] encode/decode methods to select the target format at the call site - * - Built-in implementations: [[Json]] (JSON) and [[Protobuf]] (Protocol Buffers wire format) + * - Built-in implementations: [[Json]] (JSON), [[Ion]] (Amazon Ion text), [[Yaml]] (YAML), and [[Protobuf]] (Protocol Buffers wire + * format) * * @see * [[Codec.Writer]] for the serialization side @@ -101,7 +102,7 @@ object Codec: * wrappers (e.g. the internal `SchemaSerializer.TransformAwareReader`) can signal that fields dropped by the schema should not * trigger [[MissingFieldException]]. * - * Default returns `0L` — no fields pre-satisfied. Overrides must return a mask with bit `i` set iff field index `i` is + * Default returns `0L`: no fields pre-satisfied. Overrides must return a mask with bit `i` set iff field index `i` is * pre-satisfied by this reader. Field index `i` corresponds to the case class constructor position (0-based). Only the low-order * `n` bits are relevant; bits beyond that are ignored by the caller. */ diff --git a/kyo-schema/shared/src/main/scala/kyo/Ion.scala b/kyo-schema/shared/src/main/scala/kyo/Ion.scala new file mode 100644 index 0000000000..7ac7e3ec8a --- /dev/null +++ b/kyo-schema/shared/src/main/scala/kyo/Ion.scala @@ -0,0 +1,109 @@ +package kyo + +/** Amazon Ion text codec instance for codec-polymorphic schema APIs. + * + * The companion object provides high-level helpers for Ion text strings and UTF-8 bytes. An `Ion` instance is useful when code works + * through the generic [[Codec]] or [[Schema.encode]] / [[Schema.decode]] APIs and needs to select Ion as the contextual codec value. + * + * Decoding accepts schema-shaped Ion text and treats Ion annotations as metadata. Encoding emits plain Ion text for the schema value, + * without preserving or synthesizing annotations. + */ +final class Ion extends Codec: + /** Creates an Ion text writer. */ + def newWriter(): Codec.Writer = kyo.internal.IonWriter() + + /** Creates an Ion text reader over UTF-8 input bytes. */ + def newReader(input: Span[Byte])(using Frame): Codec.Reader = + kyo.internal.IonReader(input) +end Ion + +/** Primary entry point for Amazon Ion text serialization. + * + * Encoding uses Ion text. Case classes become structs, collections become lists, maps become structs, byte spans become blobs, and + * options or maybes become Ion nulls when absent. Decoding accepts Ion text features that are useful for schema-shaped values, including + * unquoted field names, comments, type annotations, typed nulls, blobs, symbols as strings, and long strings. Type annotations are + * treated as Ion metadata and are not preserved in the decoded Scala value. + * + * @see + * [[kyo.Schema]] for the type-driven serialization model + * @see + * [[kyo.Json]] for JSON serialization + */ +object Ion: + /** Default maximum nesting depth for structs and lists in Ion decoding. */ + val DefaultMaxDepth: Int = Json.DefaultMaxDepth + + /** Default maximum number of entries in any single collection or struct in Ion decoding. */ + val DefaultMaxCollectionSize: Int = Json.DefaultMaxCollectionSize + + given Ion = Ion() + + /** Encodes a value of type A to an Ion text string. + * + * @param value + * the value to encode + * @return + * the Ion text representation + */ + inline def encode[A](value: A)(using schema: Schema[A], frame: Frame): String = + val w = summon[Ion].newWriter() + schema.writeTo(value, w) + w.resultString + end encode + + /** Encodes a value of type A to raw UTF-8 Ion text bytes. + * + * @param value + * the value to encode + * @return + * the Ion text bytes + */ + inline def encodeBytes[A](value: A)(using schema: Schema[A], frame: Frame): Span[Byte] = + val w = summon[Ion].newWriter() + schema.writeTo(value, w) + w.result() + end encodeBytes + + /** Decodes an Ion text string into a value of type A. + * + * @param input + * the Ion text string to decode + * @param maxDepth + * maximum nesting depth for structs and lists + * @param maxCollectionSize + * maximum number of entries in a single collection or struct + * @return + * the decoded value, or a DecodeException if the input is malformed or does not match the schema + */ + def decode[A]( + input: String, + maxDepth: Int = DefaultMaxDepth, + maxCollectionSize: Int = DefaultMaxCollectionSize + )(using ion: Ion, schema: Schema[A], frame: Frame): Result[DecodeException, A] = + val reader = ion.newReader(Span.from(input.getBytes(java.nio.charset.StandardCharsets.UTF_8))) + reader.resetLimits(maxDepth, maxCollectionSize) + Result.catching[DecodeException](schema.readFrom(reader)) + end decode + + /** Decodes raw UTF-8 Ion text bytes into a value of type A. + * + * @param input + * the raw UTF-8 Ion text bytes + * @param maxDepth + * maximum nesting depth for structs and lists + * @param maxCollectionSize + * maximum number of entries in a single collection or struct + * @return + * the decoded value, or a DecodeException if the input is malformed or does not match the schema + */ + def decodeBytes[A]( + input: Span[Byte], + maxDepth: Int = DefaultMaxDepth, + maxCollectionSize: Int = DefaultMaxCollectionSize + )(using ion: Ion, schema: Schema[A], frame: Frame): Result[DecodeException, A] = + val reader = ion.newReader(input) + reader.resetLimits(maxDepth, maxCollectionSize) + Result.catching[DecodeException](schema.readFrom(reader)) + end decodeBytes + +end Ion diff --git a/kyo-schema/shared/src/main/scala/kyo/internal/IonReader.scala b/kyo-schema/shared/src/main/scala/kyo/internal/IonReader.scala new file mode 100644 index 0000000000..8f30a07ef8 --- /dev/null +++ b/kyo-schema/shared/src/main/scala/kyo/internal/IonReader.scala @@ -0,0 +1,968 @@ +package kyo.internal + +import java.nio.charset.StandardCharsets +import kyo.* +import kyo.Codec.Reader +import scala.annotation.tailrec + +private[kyo] enum IonValue derives CanEqual: + case NullValue + case Bool(value: Boolean) + case IntNum(value: BigInt) + case DecNum(value: BigDecimal) + case FloatNum(value: Double) + case Str(value: String) + case Symbol(value: String) + case Timestamp(value: String) + case Blob(value: Span[Byte]) + case ListVal(values: Vector[IonValue]) + case StructVal(fields: Vector[(String, IonValue)]) + + def display: String = + this match + case NullValue => "null" + case Bool(_) => "bool" + case IntNum(_) => "int" + case DecNum(_) => "decimal" + case FloatNum(_) => "float" + case Str(_) => "string" + case Symbol(_) => "symbol" + case Timestamp(_) => "timestamp" + case Blob(_) => "blob" + case ListVal(_) => "list" + case StructVal(_) => "struct" +end IonValue + +final class IonReader private ( + private var raw: String, + private var parsed: Maybe[IonValue], + private var current: Maybe[IonValue], + private var _frame: Frame +) extends Reader: + import IonValue.* + + private enum Context: + case Obj(fields: Vector[(String, IonValue)], index: Int) + case Arr(values: Vector[IonValue], index: Int) + + import Context.* + + override def frame: Frame = _frame + + private var stack: List[Context] = Nil + private var parsedField: Maybe[String] = Maybe.empty + + def objectStart(): Int = + value match + case StructVal(fields) => + checkDepth() + stack = Obj(fields, 0) :: stack + fields.size + case other => mismatch("struct", other) + end objectStart + + def objectEnd(): Unit = + stack match + case Obj(_, _) :: tail => + decrementDepth() + stack = tail + case _ => + throw TypeMismatchException(Seq.empty, "struct end", "no active struct")(using _frame) + end objectEnd + + def arrayStart(): Int = + value match + case ListVal(values) => + checkDepth() + stack = Arr(values, 0) :: stack + values.size + case other => mismatch("list", other) + end arrayStart + + def arrayEnd(): Unit = + stack match + case Arr(_, _) :: tail => + decrementDepth() + stack = tail + case _ => + throw TypeMismatchException(Seq.empty, "list end", "no active list")(using _frame) + end arrayEnd + + def field(): String = + stack match + case Obj(fields, index) :: tail if index < fields.length => + val (name, fieldValue) = fields(index) + stack = Obj(fields, index + 1) :: tail + current = Maybe(fieldValue) + parsedField = Maybe(name) + name + case Obj(_, _) :: _ => + throw MissingFieldException(Seq.empty, "")(using _frame) + case _ => + throw TypeMismatchException(Seq.empty, "field", "no active struct")(using _frame) + end field + + override def fieldParse(): Unit = + discard(field()) + + override def matchField(nameBytes: Array[Byte]): Boolean = + parsedField.exists { name => + java.util.Arrays.equals(name.getBytes(StandardCharsets.UTF_8), nameBytes) + } + + override def lastFieldName(): String = + parsedField.getOrElse("") + + def hasNextField(): Boolean = + stack match + case Obj(fields, index) :: _ => index < fields.length + case _ => false + end hasNextField + + def hasNextElement(): Boolean = + stack match + case Arr(values, index) :: tail if index < values.length => + current = Maybe(values(index)) + stack = Arr(values, index + 1) :: tail + true + case Arr(_, _) :: _ => false + case _ => false + end hasNextElement + + def string(): String = + value match + case Str(v) => v + case Symbol(v) => v + case Timestamp(v) => v + case other => mismatch("string", other) + + def int(): Int = + val v = integer() + if v < BigInt(Int.MinValue) || v > BigInt(Int.MaxValue) then + throw RangeException(v.toLong, "Int", Int.MinValue.toLong, Int.MaxValue.toLong)(using _frame) + v.toInt + end int + + def long(): Long = + val v = integer() + if v < BigInt(Long.MinValue) || v > BigInt(Long.MaxValue) then + throw ParseException(Ion(), v.toString, "Long")(using _frame) + v.toLong + end long + + def float(): Float = + double().toFloat + + def double(): Double = + value match + case FloatNum(v) => v + case DecNum(v) => v.toDouble + case IntNum(v) => v.toDouble + case other => mismatch("float", other) + + def boolean(): Boolean = + value match + case Bool(v) => v + case other => mismatch("bool", other) + + def short(): Short = + val v = int() + if v < Short.MinValue || v > Short.MaxValue then + throw RangeException(v.toLong, "Short", Short.MinValue.toLong, Short.MaxValue.toLong)(using _frame) + v.toShort + end short + + def byte(): Byte = + val v = int() + if v < Byte.MinValue || v > Byte.MaxValue then + throw RangeException(v.toLong, "Byte", Byte.MinValue.toLong, Byte.MaxValue.toLong)(using _frame) + v.toByte + end byte + + def char(): Char = + val s = string() + if s.length != 1 then + throw TypeMismatchException(Seq.empty, "single character", s"string length ${s.length}")(using _frame) + s.charAt(0) + end char + + def isNil(): Boolean = + value match + case NullValue => true + case _ => false + + def skip(): Unit = () + + def mapStart(): Int = objectStart() + def mapEnd(): Unit = objectEnd() + def hasNextEntry(): Boolean = hasNextField() + + def bytes(): Span[Byte] = + value match + case Blob(v) => v + case Str(v) => + try Span.from(java.util.Base64.getDecoder.decode(v)) + catch + case e: IllegalArgumentException => + throw ParseException(Ion(), v, s"Base64 (${e.getMessage})")(using _frame) + case other => mismatch("blob", other) + end match + end bytes + + def bigInt(): BigInt = + integer() + + def bigDecimal(): BigDecimal = + value match + case DecNum(v) => v + case IntNum(v) => BigDecimal(v) + case FloatNum(v) => + if v.isNaN || v.isInfinite then mismatch("finite decimal", FloatNum(v)) + else BigDecimal(v) + case other => mismatch("decimal", other) + + def instant(): java.time.Instant = + val text = + value match + case Timestamp(v) => v + case Str(v) => v + case Symbol(v) => v + case other => mismatch("timestamp", other) + try java.time.Instant.parse(text) + catch + case e: java.time.format.DateTimeParseException => + throw ParseException(Ion(), text, s"Instant (${e.getMessage})")(using _frame) + end try + end instant + + def duration(): java.time.Duration = + val text = string() + try java.time.Duration.parse(text) + catch + case e: java.time.format.DateTimeParseException => + throw ParseException(Ion(), text, s"Duration (${e.getMessage})")(using _frame) + end try + end duration + + override def captureValue(): Reader = + val captured = new IonReader("", Maybe(value), Maybe(value), _frame) + captured.resetLimits(maxDepth, maxCollectionSize) + captured + end captureValue + + override def release(): Unit = + raw = "" + parsed = Maybe.empty + current = Maybe.empty + stack = Nil + parsedField = Maybe.empty + end release + + private def integer(): BigInt = + value match + case IntNum(v) => v + case other => mismatch("int", other) + + private def value: IonValue = + current.getOrElse { + val root = parsed.getOrElse { + val p = IonTextParser(raw, maxDepth, maxCollectionSize, _frame).parse() + parsed = Maybe(p) + raw = "" + p + } + current = Maybe(root) + root + } + end value + + private def mismatch(expected: String, actual: IonValue): Nothing = + throw TypeMismatchException(Seq.empty, expected, actual.display)(using _frame) + +end IonReader + +object IonReader: + def apply(input: Span[Byte])(using frame: Frame): IonReader = + new IonReader(new String(input.toArrayUnsafe, StandardCharsets.UTF_8), Maybe.empty, Maybe.empty, frame) + + def apply(input: String)(using frame: Frame): IonReader = + new IonReader(input, Maybe.empty, Maybe.empty, frame) +end IonReader + +final private class IonTextParser( + input: String, + maxDepth: Int, + maxCollectionSize: Int, + frame: Frame +): + import IonValue.* + + private var pos: Int = 0 + + def parse(): IonValue = + skipWhitespace() + consumeVersionMarker() + skipWhitespace() + if pos >= input.length then error("Expected Ion value") + val value = parseAnnotatedValue(0) + skipWhitespace() + if pos < input.length then error("Unexpected trailing content") + value + end parse + + private def parseAnnotatedValue(depth: Int): IonValue = + skipWhitespace() + @tailrec + def loop(): IonValue = + skipWhitespace() + val start = pos + parseAnnotationToken() match + case Some((token, tokenStart, quoted)) => + skipWhitespace() + if consume("::") then + validateAnnotationToken(token, tokenStart, quoted) + loop() + else + pos = start + parsePlainValue(depth) + end if + case None => + pos = start + parsePlainValue(depth) + end match + end loop + loop() + end parseAnnotatedValue + + private def parsePlainValue(depth: Int): IonValue = + skipWhitespace() + if pos >= input.length then error("Unexpected end of input") + peek match + case '{' => + if nextIs('{') then parseLob() + else parseStruct(depth) + case '[' => parseList(depth) + case '(' => parseSexp(depth) + case '"' => Str(parseQuoted('"')) + case '\'' => + if startsWithTripleQuote then Str(parseLongStringConcat()) + else Symbol(parseQuoted('\'')) + case _ => parseTokenValue() + end match + end parsePlainValue + + private def parseStruct(depth: Int): IonValue = + checkDepth(depth) + expect('{') + skipWhitespace() + val builder = Vector.newBuilder[(String, IonValue)] + var count = 0 + if consume('}') then StructVal(builder.result()) + else + var done = false + while !done do + val name = parseFieldName() + skipWhitespace() + expect(':') + val v = parseAnnotatedValue(depth + 1) + builder += ((name, v)) + count += 1 + checkCollection(count) + skipWhitespace() + if consume(',') then + skipWhitespace() + if consume('}') then done = true + else + expect('}') + done = true + end if + end while + StructVal(builder.result()) + end if + end parseStruct + + private def parseList(depth: Int): IonValue = + checkDepth(depth) + expect('[') + parseSequence(']', depth) + end parseList + + private def parseSexp(depth: Int): IonValue = + checkDepth(depth) + expect('(') + val builder = Vector.newBuilder[IonValue] + var count = 0 + skipWhitespace() + while !consume(')') do + builder += parseAnnotatedValue(depth + 1) + count += 1 + checkCollection(count) + skipWhitespace() + discard(consume(',')) + skipWhitespace() + end while + ListVal(builder.result()) + end parseSexp + + private def parseSequence(end: Char, depth: Int): IonValue = + val builder = Vector.newBuilder[IonValue] + var count = 0 + skipWhitespace() + if consume(end) then ListVal(builder.result()) + else + var done = false + while !done do + builder += parseAnnotatedValue(depth + 1) + count += 1 + checkCollection(count) + skipWhitespace() + if consume(',') then + skipWhitespace() + if consume(end) then done = true + else + expect(end) + done = true + end if + end while + ListVal(builder.result()) + end if + end parseSequence + + private def parseLob(): IonValue = + expect('{') + expect('{') + skipLobWhitespace() + if startsWithTripleQuote || hasChar('"') then + val text = + if startsWithTripleQuote then parseLongStringConcat() + else parseQuoted('"') + skipLobWhitespace() + expect('}') + expect('}') + Blob(clobBytes(text)) + else + val start = pos + val end = input.indexOf("}}", pos) + if end < 0 then error("Unterminated blob") + pos = end + 2 + val base64 = input.substring(start, end).filterNot(isIonWhitespace) + try Blob(Span.from(java.util.Base64.getDecoder.decode(base64))) + catch + case e: IllegalArgumentException => + throw ParseException(Ion(), base64, s"blob (${e.getMessage})", Nil, start)(using frame) + end try + end if + end parseLob + + private def clobBytes(text: String): Span[Byte] = + val bytes = new Array[Byte](text.length) + var i = 0 + while i < text.length do + val c = text.charAt(i) + if c > 0xff then error("CLOB character exceeds byte range") + bytes(i) = c.toByte + i += 1 + end while + Span.from(bytes) + end clobBytes + + private def parseFieldName(): String = + skipWhitespace() + if pos >= input.length then error("Expected field name") + val c = input.charAt(pos) + if c == '"' then parseQuoted('"') + else if c == '\'' then + if startsWithTripleQuote then parseLongStringConcat() + else parseQuoted('\'') + else + val start = pos + while pos < input.length && { + val c = input.charAt(pos) + !isIonWhitespace(c) && c != ':' + } + do + pos += 1 + end while + if pos == start then error("Expected field name") + input.substring(start, pos) + end if + end parseFieldName + + private def parseTokenValue(): IonValue = + val start = pos + while pos < input.length && !isValueDelimiter(input.charAt(pos)) do + pos += 1 + if pos == start then error("Expected value") + val token = input.substring(start, pos) + val lower = token.toLowerCase(java.util.Locale.ROOT) + lower match + case t if isNullToken(t) => NullValue + case t if t.startsWith("null.") => invalidToken(start, s"Invalid typed null '$token'") + case "true" => Bool(true) + case "false" => Bool(false) + case "nan" => FloatNum(Double.NaN) + case "+inf" => FloatNum(Double.PositiveInfinity) + case "-inf" => FloatNum(Double.NegativeInfinity) + case _ if isTimestampToken(token) => Timestamp(token) + case _ if isNumericTokenStart(token) => parseNumber(token, start) + case _ => Symbol(token) + end match + end parseTokenValue + + private def parseNumber(token: String, start: Int): IonValue = + if !isValidNumberToken(token) then + invalidToken(start, s"Invalid numeric token '$token'") + val cleaned = token.replace("_", "") + val lower = cleaned.toLowerCase(java.util.Locale.ROOT) + try + if isRadixIntegerToken(lower) then IntNum(parseBigInt(cleaned)) + else if lower.indexOf('e') >= 0 then FloatNum(cleaned.toDouble) + else if lower.indexOf('d') >= 0 || lower.indexOf('.') >= 0 then + DecNum(BigDecimal(cleaned.replace('d', 'E').replace('D', 'E'))) + else IntNum(parseBigInt(cleaned)) + catch + case _: NumberFormatException => + invalidToken(start, s"Invalid numeric token '$token'") + end try + end parseNumber + + private def parseBigInt(cleaned: String): BigInt = + val first = cleaned.charAt(0) + val sign = + if first == '-' then -1 + else 1 + val body = + if first == '-' || first == '+' then cleaned.substring(1) + else cleaned + val lower = body.toLowerCase(java.util.Locale.ROOT) + val parsed = + if lower.startsWith("0x") then BigInt(lower.drop(2), 16) + else if lower.startsWith("0b") then BigInt(lower.drop(2), 2) + else BigInt(body) + if sign < 0 then -parsed else parsed + end parseBigInt + + private def parseAnnotationToken(): Option[(String, Int, Boolean)] = + skipWhitespace() + if pos >= input.length then None + else if startsWithTripleQuote then None + else + val c = input.charAt(pos) + if c == '\'' then + val start = pos + Some((parseQuoted('\''), start, true)) + else if isIdentifierStart(c) || c == '$' then + val start = pos + while pos < input.length && { + val c = input.charAt(pos) + !isValueDelimiter(c) && c != ':' + } + do + pos += 1 + end while + if pos > start then Some((input.substring(start, pos), start, false)) + else None + else None + end if + end if + end parseAnnotationToken + + private def validateAnnotationToken(token: String, start: Int, quoted: Boolean): Unit = + val lower = token.toLowerCase(java.util.Locale.ROOT) + if !quoted then + if isNullToken(lower) || + lower == "true" || + lower == "false" || + lower == "nan" + then invalidToken(start, s"Invalid annotation token '$token'") + end if + end if + end validateAnnotationToken + + private def isRadixIntegerToken(token: String): Boolean = + val start = + if token.charAt(0) == '-' then 1 + else 0 + token.startsWith("0x", start) || token.startsWith("0b", start) + end isRadixIntegerToken + + private def isValidNumberToken(token: String): Boolean = + if token.isEmpty then false + else + val first = token.charAt(0) + val body = + if first == '-' then token.substring(1) + else if first == '+' then "" + else token + if body.isEmpty then false + else + val lower = body.toLowerCase(java.util.Locale.ROOT) + if lower.startsWith("0x") then validDigits(body.substring(2), isHexDigit) + else if lower.startsWith("0b") then validDigits(body.substring(2), isBinaryDigit) + else if hasRealMarker(body) then + validRealBody(body) + else validDecimalDigits(body, allowLeadingZeros = false) + end if + end if + end if + end isValidNumberToken + + private def validRealBody(body: String): Boolean = + val eIndex = exponentIndex(body, 'e') + val dIndex = exponentIndex(body, 'd') + if eIndex >= 0 && dIndex >= 0 then false + else + val expIndex = math.max(eIndex, dIndex) + val mantissa = + if expIndex >= 0 then body.substring(0, expIndex) + else body + val exponentOk = + if expIndex < 0 then true + else + val exponent = body.substring(expIndex + 1) + val digits = + if exponent.startsWith("+") || exponent.startsWith("-") then exponent.substring(1) + else exponent + validDigits(digits, isDecimalDigit) + val dotIndex = mantissa.indexOf('.') + val mantissaOk = + if dotIndex >= 0 then + if mantissa.indexOf('.', dotIndex + 1) >= 0 then false + else + val whole = mantissa.substring(0, dotIndex) + val frac = mantissa.substring(dotIndex + 1) + validDecimalDigits(whole, allowLeadingZeros = false) && + (frac.isEmpty || validDigits(frac, isDecimalDigit)) + else validDecimalDigits(mantissa, allowLeadingZeros = false) + mantissaOk && exponentOk + end if + end validRealBody + + private def hasRealMarker(body: String): Boolean = + var i = 0 + while i < body.length do + body.charAt(i) match + case '.' | 'e' | 'E' | 'd' | 'D' => return true + case _ => + i += 1 + end while + false + end hasRealMarker + + private def exponentIndex(body: String, marker: Char): Int = + val lower = marker.toLower + var index = -1 + var i = 0 + while i < body.length do + val c = body.charAt(i).toLower + if c == lower then + if index >= 0 then return -2 + index = i + i += 1 + end while + if index == -2 then -1 else index + end exponentIndex + + private def validDecimalDigits(text: String, allowLeadingZeros: Boolean): Boolean = + validDigits(text, isDecimalDigit) && + (allowLeadingZeros || text.length == 1 || text.charAt(0) != '0') + end validDecimalDigits + + private def validDigits(text: String, validDigit: Char => Boolean): Boolean = + if text.isEmpty then false + else + var i = 0 + var ok = true + var previousDigit = false + var previousUnderscore = false + while ok && i < text.length do + val c = text.charAt(i) + if validDigit(c) then + previousDigit = true + previousUnderscore = false + else if c == '_' then + if !previousDigit then ok = false + else + previousDigit = false + previousUnderscore = true + end if + else ok = false + end if + i += 1 + end while + ok && !previousUnderscore + end if + end validDigits + + private def isDecimalDigit(c: Char): Boolean = + c >= '0' && c <= '9' + + private def isBinaryDigit(c: Char): Boolean = + c == '0' || c == '1' + + private def isHexDigit(c: Char): Boolean = + (c >= '0' && c <= '9') || + (c >= 'a' && c <= 'f') || + (c >= 'A' && c <= 'F') + + private def parseLongStringConcat(): String = + val sb = new StringBuilder + var continue = true + while continue do + sb.append(parseLongString()) + val saved = pos + skipWhitespace() + if !startsWithTripleQuote then + pos = saved + continue = false + end while + sb.toString + end parseLongStringConcat + + private def parseLongString(): String = + expect('\'') + expect('\'') + expect('\'') + val sb = new StringBuilder + var done = false + while !done do + if pos >= input.length then error("Unterminated long string") + val c = input.charAt(pos) + if c == '\'' && startsWithTripleQuote then done = true + else if c == '\\' then + pos += 1 + appendEscape(sb) + else + sb.append(c) + pos += 1 + end if + end while + expect('\'') + expect('\'') + expect('\'') + sb.toString + end parseLongString + + private def parseQuoted(quote: Char): String = + expect(quote) + val sb = new StringBuilder + var done = false + while !done && pos < input.length do + val c = input.charAt(pos) + if c == quote then done = true + else if c == '\\' then + pos += 1 + appendEscape(sb) + else + sb.append(c) + pos += 1 + end if + end while + if pos >= input.length then error("Unterminated quoted value") + expect(quote) + sb.toString + end parseQuoted + + private def appendEscape(sb: StringBuilder): Unit = + if pos >= input.length then error("Unexpected end of escape") + input.charAt(pos) match + case '0' => sb.append('\u0000'); pos += 1 + case 'a' => sb.append('\u0007'); pos += 1 + case 'b' => sb.append('\b'); pos += 1 + case 't' => sb.append('\t'); pos += 1 + case 'n' => sb.append('\n'); pos += 1 + case 'f' => sb.append('\f'); pos += 1 + case 'r' => sb.append('\r'); pos += 1 + case 'v' => sb.append('\u000b'); pos += 1 + case '"' => sb.append('"'); pos += 1 + case '\'' => sb.append('\''); pos += 1 + case '?' => sb.append('?'); pos += 1 + case '\\' => sb.append('\\'); pos += 1 + case '/' => sb.append('/'); pos += 1 + case 'x' => + pos += 1 + sb.append(readHex(2).toChar) + case 'u' => + pos += 1 + sb.append(readHex(4).toChar) + case 'U' => + pos += 1 + appendCodePoint(sb, readHex(8)) + case '\n' => + pos += 1 + case '\r' => + pos += 1 + if pos < input.length && input.charAt(pos) == '\n' then pos += 1 + case other => + error(s"Invalid escape \\$other") + end match + end appendEscape + + private def appendCodePoint(sb: StringBuilder, codePoint: Int): Unit = + if !Character.isValidCodePoint(codePoint) then + val hex = java.lang.Long.toHexString(java.lang.Integer.toUnsignedLong(codePoint)).toUpperCase(java.util.Locale.ROOT) + error(s"Invalid Unicode code point U+$hex") + sb.appendAll(Character.toChars(codePoint)) + end appendCodePoint + + private def readHex(digits: Int): Int = + if pos + digits > input.length then error("Truncated hex escape") + var value = 0 + var i = 0 + while i < digits do + value = (value << 4) | hex(input.charAt(pos + i)) + i += 1 + pos += digits + value + end readHex + + private def hex(c: Char): Int = + if c >= '0' && c <= '9' then c - '0' + else if c >= 'a' && c <= 'f' then c - 'a' + 10 + else if c >= 'A' && c <= 'F' then c - 'A' + 10 + else error(s"Invalid hex digit '$c'") + + private def consumeVersionMarker(): Unit = + val saved = pos + if startsWith("$ion_1_0") then + pos += "$ion_1_0".length + if pos >= input.length || isValueDelimiter(input.charAt(pos)) then () + else pos = saved + end if + end consumeVersionMarker + + private def skipWhitespace(): Unit = + var continue = true + while continue && pos < input.length do + val c = input.charAt(pos) + if isIonWhitespace(c) then pos += 1 + else if c == '/' && pos + 1 < input.length then + input.charAt(pos + 1) match + case '/' => + pos += 2 + while pos < input.length && { + val c = input.charAt(pos) + c != '\n' && c != '\r' + } + do + pos += 1 + end while + case '*' => + val end = input.indexOf("*/", pos + 2) + if end < 0 then error("Unterminated block comment") + pos = end + 2 + case _ => + continue = false + end match + else continue = false + end if + end while + end skipWhitespace + + private def skipLobWhitespace(): Unit = + while pos < input.length && isIonWhitespace(input.charAt(pos)) do + pos += 1 + + private def checkDepth(depth: Int): Unit = + if depth + 1 > maxDepth then + throw LimitExceededException("Nesting depth", depth + 1, maxDepth)(using frame) + + private def checkCollection(count: Int): Unit = + if count > maxCollectionSize then + throw LimitExceededException("Collection size", count, maxCollectionSize)(using frame) + + private def isTimestampToken(token: String): Boolean = + token.length >= 10 && + token.charAt(4) == '-' && + token.charAt(7) == '-' && + isDecimalDigit(token.charAt(0)) && + isDecimalDigit(token.charAt(1)) && + isDecimalDigit(token.charAt(2)) && + isDecimalDigit(token.charAt(3)) + + private def isNullToken(token: String): Boolean = + token == "null" || isTypedNullToken(token) + + private def isTypedNullToken(token: String): Boolean = + token.length > 5 && + token.startsWith("null.") && + isNullTypeSuffix(token, 5) + end isTypedNullToken + + private def isNullTypeSuffix(token: String, offset: Int): Boolean = + token.length - offset match + case 3 => token.startsWith("int", offset) + case 4 => token.startsWith("null", offset) || token.startsWith("bool", offset) || token.startsWith("blob", offset) || + token.startsWith("clob", offset) || token.startsWith("list", offset) || token.startsWith("sexp", offset) + case 5 => token.startsWith("float", offset) + case 6 => token.startsWith("string", offset) || token.startsWith("symbol", offset) || token.startsWith("struct", offset) + case 7 => token.startsWith("decimal", offset) + case 9 => token.startsWith("timestamp", offset) + case _ => false + end isNullTypeSuffix + + private def isNumericTokenStart(token: String): Boolean = + token.nonEmpty && { + val first = token.charAt(0) + isDecimalDigit(first) || + ((first == '-' || first == '+') && token.length > 1 && isDecimalDigit(token.charAt(1))) + } + + private def isValueDelimiter(c: Char): Boolean = + isIonWhitespace(c) || c == ',' || c == ']' || c == '}' || c == ')' + + private def isIdentifierStart(c: Char): Boolean = + (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || c == '_' || c == '$' + + private def isIonWhitespace(c: Char): Boolean = + c == ' ' || c == '\t' || c == '\n' || c == '\r' || c == '\f' || c == '\u000b' + + private def consume(s: String): Boolean = + if startsWith(s) then + pos += s.length + true + else false + + private def consume(c: Char): Boolean = + if hasChar(c) then + pos += 1 + true + else false + + private def expect(c: Char): Unit = + if pos >= input.length then error(s"Expected '$c' but reached end of input") + if input.charAt(pos) != c then error(s"Expected '$c', got '${input.charAt(pos)}'") + pos += 1 + end expect + + private def startsWith(s: String): Boolean = + input.startsWith(s, pos) + + private def peek: Char = + if pos >= input.length then error("Unexpected end of input") + input.charAt(pos) + + private def hasChar(c: Char): Boolean = + pos < input.length && input.charAt(pos) == c + + private def nextIs(c: Char): Boolean = + pos + 1 < input.length && input.charAt(pos + 1) == c + + private def startsWithTripleQuote: Boolean = + pos + 2 < input.length && + input.charAt(pos) == '\'' && + input.charAt(pos + 1) == '\'' && + input.charAt(pos + 2) == '\'' + + private def error(message: String): Nothing = + val start = math.max(0, pos - 30) + val end = math.min(input.length, pos + 30) + val snippet = input.substring(start, end) + throw ParseException(Ion(), snippet, message, Nil, pos)(using frame) + end error + + private def invalidToken(start: Int, message: String): Nothing = + pos = start + error(message) + end invalidToken +end IonTextParser diff --git a/kyo-schema/shared/src/main/scala/kyo/internal/IonWriter.scala b/kyo-schema/shared/src/main/scala/kyo/internal/IonWriter.scala new file mode 100644 index 0000000000..c296861ef6 --- /dev/null +++ b/kyo-schema/shared/src/main/scala/kyo/internal/IonWriter.scala @@ -0,0 +1,228 @@ +package kyo.internal + +import java.nio.charset.StandardCharsets +import kyo.Codec.Writer +import kyo.Span + +final class IonWriter private (private val out: StringBuilder) extends Writer: + + private enum Context: + case Obj(first: Boolean) + case Arr(first: Boolean) + + import Context.* + + private var stack: List[Context] = Nil + private var afterField: Boolean = false + + // Reusable scratch buffer for Ryu float formatting (sized to RyuDouble.MaxOutputLen). + private val numberBytes = new Array[Byte](24) + + def objectStart(name: String, size: Int): Unit = + beforeValue() + out.append('{') + stack = Obj(true) :: stack + end objectStart + + def objectEnd(): Unit = + stack match + case Obj(_) :: tail => + stack = tail + out.append('}') + case other => + throw new IllegalStateException(s"IonWriter.objectEnd without object context: $other") + end match + end objectEnd + + def arrayStart(size: Int): Unit = + beforeValue() + out.append('[') + stack = Arr(true) :: stack + end arrayStart + + def arrayEnd(): Unit = + stack match + case Arr(_) :: tail => + stack = tail + out.append(']') + case other => + throw new IllegalStateException(s"IonWriter.arrayEnd without array context: $other") + end match + end arrayEnd + + def fieldBytes(nameBytes: Array[Byte], fieldId: Int): Unit = + field(new String(nameBytes, StandardCharsets.UTF_8), fieldId) + + override def field(name: String, fieldId: Int): Unit = + stack match + case Obj(first) :: tail => + if !first then out.append(',') + stack = Obj(false) :: tail + writeFieldName(name) + out.append(':') + afterField = true + case other => + throw new IllegalStateException(s"IonWriter.field without object context: $other") + end match + end field + + def string(value: String): Unit = + beforeValue() + writeQuotedString(value) + + def int(value: Int): Unit = + beforeValue() + out.append(value) + + def long(value: Long): Unit = + beforeValue() + out.append(value) + + def float(value: Float): Unit = + beforeValue() + writeFloat(value.toDouble) + + def double(value: Double): Unit = + beforeValue() + writeFloat(value) + + def boolean(value: Boolean): Unit = + beforeValue() + out.append(if value then "true" else "false") + + def short(value: Short): Unit = + beforeValue() + out.append(value.toInt) + + def byte(value: Byte): Unit = + beforeValue() + out.append(value.toInt) + + def char(value: Char): Unit = + beforeValue() + writeQuotedString(value.toString) + + def nil(): Unit = + beforeValue() + out.append("null") + + def mapStart(size: Int): Unit = objectStart("", size) + def mapEnd(): Unit = objectEnd() + + def bytes(value: Span[Byte]): Unit = + beforeValue() + out.append("{{") + out.append(java.util.Base64.getEncoder.encodeToString(value.toArray)) + out.append("}}") + end bytes + + def bigInt(value: BigInt): Unit = + beforeValue() + out.append(value.toString) + + def bigDecimal(value: BigDecimal): Unit = + beforeValue() + out.append(value.toString.replace('E', 'd').replace('e', 'd')) + + def instant(value: java.time.Instant): Unit = + beforeValue() + out.append(value.toString) + + def duration(value: java.time.Duration): Unit = + beforeValue() + writeQuotedString(value.toString) + + def result(): Span[Byte] = + Span.from(out.toString.getBytes(StandardCharsets.UTF_8)) + + override def resultString: String = out.toString + + private def beforeValue(): Unit = + if afterField then + afterField = false + else + stack match + case Arr(first) :: tail => + if !first then out.append(',') + stack = Arr(false) :: tail + case _ => () + end match + end beforeValue + + private def writeFloat(value: Double): Unit = + if java.lang.Double.isNaN(value) then out.append("nan") + else if value == Double.PositiveInfinity then out.append("+inf") + else if value == Double.NegativeInfinity then out.append("-inf") + else + // Ryu yields Java's Double.toString format deterministically across JVM, JS, and Native, + // unlike java.lang.Double.toString which diverges (for example 5.0 renders "5" on Scala.js). + val end = Ryu.RyuDouble.write(value, numberBytes, 0, numberBytes.length) + val text = new String(numberBytes, 0, end, StandardCharsets.UTF_8) + out.append(text) + if text.indexOf('E') < 0 && text.indexOf('e') < 0 then out.append("e0") + end if + end writeFloat + + private def writeFieldName(value: String): Unit = + if isIdentifier(value) && !IonWriter.ReservedSymbols.contains(value) then out.append(value) + else writeQuotedString(value) + + private def isIdentifier(value: String): Boolean = + if value.isEmpty then false + else + val first = value.charAt(0) + if !isIdentifierStart(first) then false + else + var i = 1 + var ok = true + while ok && i < value.length do + ok = isIdentifierPart(value.charAt(i)) + i += 1 + ok + end if + end if + end isIdentifier + + private def isIdentifierStart(c: Char): Boolean = + (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || c == '_' || c == '$' + + private def isIdentifierPart(c: Char): Boolean = + isIdentifierStart(c) || (c >= '0' && c <= '9') + + private def writeQuotedString(value: String): Unit = + out.append('"') + var i = 0 + while i < value.length do + val c = value.charAt(i) + c match + case '"' => out.append("\\\"") + case '\\' => out.append("\\\\") + case '\b' => out.append("\\b") + case '\f' => out.append("\\f") + case '\n' => out.append("\\n") + case '\r' => out.append("\\r") + case '\t' => out.append("\\t") + case _ => + if c < 0x20 then + out.append("\\u") + val hex = Integer.toHexString(c.toInt) + var pad = hex.length + while pad < 4 do + out.append('0') + pad += 1 + out.append(hex) + else out.append(c) + end match + i += 1 + end while + out.append('"') + end writeQuotedString + +end IonWriter + +object IonWriter: + private val ReservedSymbols: Set[String] = + Set("null", "true", "false", "nan", "+inf", "-inf") + + def apply(): IonWriter = new IonWriter(new StringBuilder(256)) +end IonWriter diff --git a/kyo-schema/shared/src/test/resources/iontestdata/LICENSE b/kyo-schema/shared/src/test/resources/iontestdata/LICENSE new file mode 100644 index 0000000000..8dada3edaf --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright {yyyy} {name of copyright owner} + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/kyo-schema/shared/src/test/resources/iontestdata/NOTICE b/kyo-schema/shared/src/test/resources/iontestdata/NOTICE new file mode 100644 index 0000000000..94d1c8db27 --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/NOTICE @@ -0,0 +1,2 @@ +Amazon Ion Tests +Copyright 2007-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved. diff --git a/kyo-schema/shared/src/test/resources/iontestdata/README.md b/kyo-schema/shared/src/test/resources/iontestdata/README.md new file mode 100644 index 0000000000..4f7d25039c --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/README.md @@ -0,0 +1,7 @@ +Vendored test data from amazon-ion/ion-tests. + +Source: https://github.com/amazon-ion/ion-tests +Commit: 721434649c97baa3d5b9c9a56c1d61fbfd0a5fd7 +License: Apache-2.0, copied in LICENSE and NOTICE in this directory. + +Only selected Ion 1.0 text files under iontestdata/good are copied here. The kyo-schema Ion codec is a schema-shaped text codec, so tests use an explicit allowlist of files and samples rather than asserting full Ion data model conformance over every upstream test vector. diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/allNulls.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/allNulls.ion new file mode 100644 index 0000000000..6d4d4ec544 --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/allNulls.ion @@ -0,0 +1,17 @@ +// Wrapped in an array to make sure dotted nulls are treated as single tokens. +[ + null, + null.null, + null.bool, + null.int, + null.float, + null.decimal, + null.timestamp, + null.string, + null.symbol, + null.blob, + null.clob, + null.struct, + null.list, + null.sexp +] diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedFalse.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedFalse.ion new file mode 100644 index 0000000000..a5e6b0489e --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedFalse.ion @@ -0,0 +1 @@ +'false'::23 diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedNan.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedNan.ion new file mode 100644 index 0000000000..68607158e8 --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedNan.ion @@ -0,0 +1 @@ +'nan'::23 diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedNegInf.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedNegInf.ion new file mode 100644 index 0000000000..225ace0937 --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedNegInf.ion @@ -0,0 +1 @@ +'-inf'::23 diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedNull.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedNull.ion new file mode 100644 index 0000000000..c2ba702f20 --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedNull.ion @@ -0,0 +1 @@ +'null'::23 diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedNullInt.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedNullInt.ion new file mode 100644 index 0000000000..6519ec5cf1 --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedNullInt.ion @@ -0,0 +1 @@ +'null.int'::23 diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedOperator.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedOperator.ion new file mode 100644 index 0000000000..4fdedef602 --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedOperator.ion @@ -0,0 +1 @@ +'@'::23 diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedPosInf.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedPosInf.ion new file mode 100644 index 0000000000..6cb448c41f --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedPosInf.ion @@ -0,0 +1 @@ +'+inf'::23 diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedTrue.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedTrue.ion new file mode 100644 index 0000000000..6b16ef0554 --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/annotationQuotedTrue.ion @@ -0,0 +1 @@ +'true'::23 diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/blobs.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/blobs.ion new file mode 100644 index 0000000000..fb06ae561d --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/blobs.ion @@ -0,0 +1,25 @@ +{{ + YSBiIGMgZCBlIGYgZyBoIGkgaiBrIGwgbSBuIG8gcCBxIHIgcyB0IHUgdiB3IHggeSB6 +}} +{{ QSB CIEM gRC B F IEYg Ry BI IEk gSi BLIE w gTS B OI E8 g UC BRI FIgUy BU IF Ug ViB XI F gg WS B a}} +{{MSAyIDMgNCA1IDYgNyA4IDkgMA +==}} +{{ + + LCAuIDsgLyBbICcgXSBcID0gLSAwIDkgOCA3IDYgNSA0IDMgMiAxIGAgfiAhIEAgIyAkICUgXiAmICogKCApIF8gKyB8IDogPCA+ID8= + + }} +{{OiBTIKUgTyAASb8=}} +{{ //79/PsAAQIDBAU= }} +{{ +A + R E +Z H i + w 3 P +E h R Y 2 + d 1 f Y u +O n K W x t + c b M 0 9 / +v 9 v 8 A +}} + {{ QSBWZXJ5IFZlcnkgVmVyeSBWZXJ5IExhcmdlIFRlc3QgQmxvYg== }} // A Very Very Very Very Large Test Blob diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/booleans.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/booleans.ion new file mode 100644 index 0000000000..da29283aaa --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/booleans.ion @@ -0,0 +1,2 @@ +true +false diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/clobs.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/clobs.ion new file mode 100644 index 0000000000..47e62493a8 --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/clobs.ion @@ -0,0 +1,34 @@ +{{"a b c d e f g h i j k l m n o p q r s t u v w x y z"}} +{{ + "A B C D E F G H I J K L M N O P Q R S T U V W X Y Z" +}} +{{ "1 2 3 4 5 6 7 8 9 0" }} +{{ ", . ; / [ ' ] \\ = - 0 9 8 7 6 5 4 3 2 1 ` ~ ! @ # $ % ^ & * ( ) _ + | : < > ?" + +}} +{{ "\0 \a \b \t \n \f \r \v \" \' \? \\\\ \/ \0\a\b\t\n\f\r\v\"\'\?\\\\\/"}} +{{"\x7f \x66 \x00 \x5a\x5b\x00\x1c\x2d\x3f\xFf"}} +{{"\x7F \x66 \x00 \x5A\x5B\x00\x1C\x2D\x3F\xfF"}} +{{'''Stuff to write on ''' + '''multiple lines ''' + '''if you want to'''}} +{{""}} +{{''''''}} +{{ +"" +}} +{{ '''concatenated''' ''' from ''' '''a single line''' }} +{{ ""}} +{{ + '''a b c d e f g h i j k l m n o p q r s t u v w x y z ''' + '''A B C D E F G H I J K L M N O P Q R S T U V W X Y Z ''' + ''', . ; / [ ' ] \\ = - 0 9 8 7 6 5 4 3 2 1 ` ~ ! @ # $ % ^ & * ( ) _ + | : < > ? ''' + '''\0 \a \b \t \n \f \r \v \" \' \? \\\\ \/ \0\a\b\t\n\f\r\v\"\'\?\\\\\/''' + '''\x7f \x66 \x00 \x5a\x5b\x00\x1c\x2d\x3f''' + '''\x7F \x66 \x00 \x5A\x5B\x00\x1C\x2D\x3F''' +}} +{{'''\ +multi-line string +with embedded\nnew line +characters\ +'''}} diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/decimal_values.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/decimal_values.ion new file mode 100644 index 0000000000..2e02a1bdbe --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/decimal_values.ion @@ -0,0 +1,62 @@ +123456.0 +123456d0 +123456d1 +123456d2 +123456d3 +123456d42 +123456d-0 +123456d-1 +123456d-2 +123456d-42 +0.123456 +1.23456 +12.3456 +123.456 +1234.56 +12345.6 +12345.60 +12345.600 +12300456.0 +123.00456 +1230.0456 +12300.456 +123.456d42 +123.456d+42 +123.456d-42 +77777.7d0007 +77777.7d-0007 +77777.7d+0007 +77777.7d00700 +77777.7d-00700 +77777.7d+00700 +-123456.0 +-123456d0 +-123456d1 +-123456d2 +-123456d3 +-123456d42 +-123456d-0 +-123456d-1 +-123456d-2 +-123456d-42 +-0.123456 +-1.23456 +-12.3456 +-123.456 +-1234.56 +-12345.6 +-12345.60 +-12345.600 +-12300456.0 +-123.00456 +-1230.0456 +-12300.456 +-123.456d42 +-123.456d+42 +-123.456d-42 +-77777.7d0007 +-77777.7d-0007 +-77777.7d+0007 +-77777.7d00700 +-77777.7d-00700 +-77777.7d+00700 diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/decimalsWithUnderscores.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/decimalsWithUnderscores.ion new file mode 100644 index 0000000000..b5eba847ed --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/decimalsWithUnderscores.ion @@ -0,0 +1,3 @@ +12_34.56_78 +12_34. +1_2_3_4.5_6_7_8 diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/fieldNameQuotedNullInt.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/fieldNameQuotedNullInt.ion new file mode 100644 index 0000000000..f217871fc7 --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/fieldNameQuotedNullInt.ion @@ -0,0 +1 @@ +{ 'null.int' : false } diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/floatSpecials.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/floatSpecials.ion new file mode 100644 index 0000000000..610f549ffd --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/floatSpecials.ion @@ -0,0 +1,5 @@ +[ + nan, + +inf, + -inf +] diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/floatsWithUnderscores.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/floatsWithUnderscores.ion new file mode 100644 index 0000000000..4bdf2e2e71 --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/floatsWithUnderscores.ion @@ -0,0 +1,3 @@ +12_34.56_78e0 +12_34e56 +1_2_3_4.5_6_7_8E90 diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/intBinary.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/intBinary.ion new file mode 100644 index 0000000000..7e8be4b5d3 --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/intBinary.ion @@ -0,0 +1,3 @@ +0b11110000 +0B010101 +-0b1111 diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/integer_values.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/integer_values.ion new file mode 100644 index 0000000000..b67e269401 --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/integer_values.ion @@ -0,0 +1,20 @@ +0 +42 +2112 +-999 +-0 +987654321 +-123456789 +0x10 +0xff +0xFF +0xA +0xAbCdEf +0x123456789 +0x1234567890abcdef +-0x1234567890ABCDEF +0x0 +-0x0 +-0xFFFF +0x00FF +-0x00FF diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/intsWithUnderscores.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/intsWithUnderscores.ion new file mode 100644 index 0000000000..fb1b7b30a8 --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/intsWithUnderscores.ion @@ -0,0 +1,57 @@ +1_2_3 +0xab_cd +0b1111_0000 +100_000 +-1_2_3 +-0xab_cd +-0b1111_0000 +-100_000 + +(1_2_3) +(0xab_cd) +(0b1111_0000) +(100_000) +(-1_2_3) +(-0xab_cd) +(-0b1111_0000) +(-100_000) + +( +1_2_3 +1_2_3 +) + +( +0xab_cd +0xab_cd +) + +( +0b1111_0000 +0b1111_0000 +) + +( +100_000 +100_000 +) + +( +-1_2_3 +-1_2_3 +) + +( +-0xab_cd +-0xab_cd +) + +( +-0b1111_0000 +-0b1111_0000 +) + +( +-100_000 +-100_000 +) diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/lists.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/lists.ion new file mode 100644 index 0000000000..e3f633eefb --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/lists.ion @@ -0,0 +1,12 @@ +[1, 2, 3, 4, 5] +[1,2,3,4,5] +[1, [2, 3], [[[5]]]] +[1, (2 3), [4, (5)]] +[true, 3.4, 3d6, 2.3e8, "string", '''multi-''' '''string''', Symbol, 'qSymbol', + {{"clob data"}}, {{YmxvYiBkYXRh}}, 1970-06-06, null.struct] +[{one:1}, 2, 3] +[0xab] +[symbol] +["string"] +['symbol'] +[+inf] diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/multipleAnnotations.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/multipleAnnotations.ion new file mode 100644 index 0000000000..8b51cd1165 --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/multipleAnnotations.ion @@ -0,0 +1 @@ +annot1::annot2::value diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/nonNulls.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/nonNulls.ion new file mode 100644 index 0000000000..0a4ebb1555 --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/nonNulls.ion @@ -0,0 +1,12 @@ +// None of these values should be null. +0 +0.0 +0d0 +0e0 +"" +'''''' +{{}} +{{""}} +[] +() +{} diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/sexps.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/sexps.ion new file mode 100644 index 0000000000..a5fa02fc81 --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/sexps.ion @@ -0,0 +1,36 @@ +(this is a sexp list) +(`~!@/%^&*-+=|;<>?. 3 -- - 4) +(+ ++ +-+ -++ - -- --- -3 - 3 -- 3 --3 ) +('+' '++' '+-+' '-++' '-' '--' '---' -3 - 3 '--' 3 '--'3 ) +(& (% -[42, 3]+(2)-)) +((())) +([]) +(null .timestamps) +(op1.op2) +(a_plus_plus_plus_operator::+++ a_3::3) +(a_plus_plus_plus_operator:: +++ a_3::3) +(a_plus_plus_plus_operator::'+++') +(a::!) +(a::#) +(a::%) +(a::&) +(a::*) +(a::+) +(a::-) +(a::.) +(a::/) +(a::;) +(a::<) +(a::=) +(a::>) +(a::?) +(a::@) +(a::^) +(a::`) +(a::|) +(a::~) +(a::...) +(a::.+) +(a::.*) +(a::._) +(a::.$) diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/strings.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/strings.ion new file mode 100644 index 0000000000..ee73743d4d --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/strings.ion @@ -0,0 +1,33 @@ +"a b c d e f g h i j k l m n o p q r s t u v w x y z" +"A B C D E F G H I J K L M N O P Q R S T U V W X Y Z" +"1 2 3 4 5 6 7 8 9 0" +", . ; / [ ' ] \\ = - 0 9 8 7 6 5 4 3 2 1 ` ~ ! @ # $ % ^ & * ( ) _ + | : < > ?" +"\0 \a \b \t \n \f \r \v \" \' \? \\\\ \/ \0\a\b\t\n\f\r\v\"\'\?\\\\\/" +"\uaa5f" +"\uabcd \ud7ff \uffff \u1234 \u4e6a \ud37b\uf4c2\u0000\x00\xff" // d7ff was deff +"\uABCD \uD7FF \uFFFF \u1234 \u4E6A \uD37B\uF4C2\u0000\x00\xff" +"\uaBcD \uD7ff \uFffF \u1234 \u4E6a \ud37B\uF4c2\u0000\x00\xff" +'''Stuff to write on ''' +'''multiple lines ''' +'''if you want to''' +"" +'''''' +"" +'''concatenated''' ''' from ''' '''a single line''' +"" +'''a b c d e f g h i j k l m n o p q r s t u v w x y z ''' +'''A B C D E F G H I J K L M N O P Q R S T U V W X Y Z ''' +''', . ; / [ ' ] \\ = - 0 9 8 7 6 5 4 3 2 1 ` ~ ! @ # $ % ^ & * ( ) _ + | : < > ? ''' +'''\0 \a \b \t \n \f \r \v \" \' \? \\\\ \/ \0\a\b\t\n\f\r\v\"\'\?\\\\\/''' +'''\uabcd \ud7ff \uffff \u1234 \u4e6a \ud37b\uf4c2\u0000\x00\xff''' +'''\uABCD \uD7FF \uFFFF \u1234 \u4E6A \uD37B\uF4C2\u0000\x00\xFF''' +'''\uaBcD \uD7ff \uFffF \u1234 \u4E6a \ud37B\uF4c2\u0000\x00\xfF''' // d7ff was deff +"" +'''\ +multi-line string +with embedded\nnew line +characters\ +''' +'''\U00001234''' +"" +'''''' diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/structFieldAnnotationsUnquotedThenQuoted.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/structFieldAnnotationsUnquotedThenQuoted.ion new file mode 100644 index 0000000000..dfa454d5c2 --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/structFieldAnnotationsUnquotedThenQuoted.ion @@ -0,0 +1,3 @@ +// Captures ION-126 + +{f:a::'b'::null} diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/structs.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/structs.ion new file mode 100644 index 0000000000..0f07d53967 --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/structs.ion @@ -0,0 +1,29 @@ +{a:b,c:42,d:{e:f,},g:3} +{'a':'b','c':42,'d':{'e':'f',},'g':3} +{"a":"b","c":42,"d":{"e":"f",},"g":3} +{a:b, c:42, d:{e:f, }, g:3} +{'a':'b', 'c':42, 'd':{'e':'f', }, 'g':3} +{"a":"b", "c":42, "d":{"e":"f", }, "g":3} +{a: b, c: 42, d: {e: f, }, g: 3, } +{'a': 'b', 'c': 42, 'd': {'e': 'f', }, 'g': 3} +{"a": "b", "c": 42, "d": {"e": "f", }, "g": 3 } +{a : b, c : 42, d : {e : f, }, g :3} +{'a' : 'b', 'c' : 42, 'd' : {'e' : 'f', }, 'g' : 3} +{"a" : "b", "c" : 42, "d" : {"e" : "f", }, "g" : 3} + +// Test lengthy field names +{ '123456789ABCDEF' : v } + +{ "123456789ABCDEF" : v } + +{ '''123456789ABCDEF''' : v } + +{ '''123456789ABCDEF''' + '''123456789ABCDEF''' : v } + +// Strange but true. +{ '''123 +455''' : v } + +{ '''123456789ABCDEF +GHI''' : v } \ No newline at end of file diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/symbolEmpty.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/symbolEmpty.ion new file mode 100644 index 0000000000..430e87a4d6 --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/symbolEmpty.ion @@ -0,0 +1,9 @@ +// Empty symbol is allowed. + +'' +{'':abc} +''::abc +''::'' +{'':''::''} +abc::'' +{'''''':abc} diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/symbols.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/symbols.ion new file mode 100644 index 0000000000..bd4597104b --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/symbols.ion @@ -0,0 +1,26 @@ +'a b c d e f g h i j k l m n o p q r s t u v w x y z' +'A B C D E F G H I J K L M N O P Q R S T U V W X Y Z' +'1 2 3 4 5 6 7 8 9 0' +', . ; / [ \' ] \\ = - 0 9 8 7 6 5 4 3 2 1 ` ~ ! @ # $ % ^ & * ( ) _ + | : < > ?' +'\0 \a \b \t \n \f \r \v \" \' \? \\\\ \/ \0\a\b\t\n\f\r\v\"\'\?\\\\\/' +'\uabcd \ud7ff \uffff \u1234 \u4e6a \ud37b\uf4c2\u0000\x00\xff' // d7ff was deff but deff isn't a legal unicode scalar +'\uABCD \uD7FF \uFFFF \u1234 \u4E6A \uD37B\uF4C2\u0000\x00\xff' +'\uaBcD \uD7ff \uFffF \u1234 \u4E6a \ud37B\uF4c2\u0000\x00\xff' +bareSymbol +BareSymbol +$bare +_bare +zzzzz +aaaaa +ZZZZZ +AAAAA +z +Z +a +A +_ +$ +_9876543210 +$3 +abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789$_ +'$99' // Symbol with text "$99", as opposed to unquoted $99, which is an out-of-range SID with unknown text. diff --git a/kyo-schema/shared/src/test/resources/iontestdata/good/whitespace.ion b/kyo-schema/shared/src/test/resources/iontestdata/good/whitespace.ion new file mode 100644 index 0000000000..28f3ec4d2e --- /dev/null +++ b/kyo-schema/shared/src/test/resources/iontestdata/good/whitespace.ion @@ -0,0 +1,14 @@ +// Be careful editing this file! +// It contains various legal whitespace characters. + +// Horizontal tab U+0009 (CTRL-I) between 1 and a: +1 a +(1 a) + +// Vertical tab U+000B (CTRL-K) between 1 and a: +1 a +(1 a) + +// Form feed U+000C (CTRL-L) between 1 and a: +1 a +(1 a) diff --git a/kyo-schema/shared/src/test/scala/kyo/IonCorpusTest.scala b/kyo-schema/shared/src/test/scala/kyo/IonCorpusTest.scala new file mode 100644 index 0000000000..646ce2fd2f --- /dev/null +++ b/kyo-schema/shared/src/test/scala/kyo/IonCorpusTest.scala @@ -0,0 +1,283 @@ +package kyo + +import java.nio.charset.StandardCharsets + +class IonCorpusTest extends kyo.test.Test[Any]: + + given CanEqual[Any, Any] = CanEqual.derived + + "vendored amazon ion-tests corpus" - { + + "resources are present with upstream license and notice" in { + for + license <- resource("/iontestdata/LICENSE") + notice <- resource("/iontestdata/NOTICE") + _ <- Kyo.foreach(IonCorpusTest.RequiredResources) { name => + resource(s"/iontestdata/good/$name").map(content => assert(content.nonEmpty)) + } + yield + assert(license.contains("Apache License")) + assert(notice.contains("Amazon Ion Tests")) + } + + "decodes typed null values from iontestdata/good/allNulls.ion" in { + resource("/iontestdata/good/allNulls.ion").map { upstream => + assert(upstream.contains("null.int")) + assert(upstream.contains("null.struct")) + + val options = Ion.decode[List[Option[Int]]](upstream).getOrThrow + assert(options.size == 14) + assert(options.forall(_ == None)) + + val maybes = Ion.decode[List[Maybe[String]]](upstream).getOrThrow + assert(maybes.size == 14) + assert(maybes.forall(_ == Maybe.empty)) + } + } + + "decodes non-null scalar and container samples from iontestdata/good/nonNulls.ion" in { + resource("/iontestdata/good/nonNulls.ion").map { upstream => + List("0", "0.0", "0d0", "0e0", "\"\"", "''''''", "{{}}", "{{\"\"}}", "[]", "()", "{}").foreach { ion => + assert(upstream.contains(ion)) + } + + assert(Ion.decode[Int]("0").getOrThrow == 0) + assert(Ion.decode[BigDecimal]("0.0").getOrThrow == BigDecimal("0.0")) + assert(Ion.decode[BigDecimal]("0d0").getOrThrow == BigDecimal(0)) + assert(Ion.decode[Double]("0e0").getOrThrow == 0.0) + assert(Ion.decode[String]("\"\"").getOrThrow == "") + assert(Ion.decode[String]("''''''").getOrThrow == "") + assert(Ion.decode[Span[Byte]]("{{}}").getOrThrow.toArray.isEmpty) + assert(Ion.decode[Span[Byte]]("{{\"\"}}").getOrThrow.toArray.isEmpty) + assert(Ion.decode[List[Int]]("[]").getOrThrow == Nil) + assert(Ion.decode[List[Int]]("()").getOrThrow == Nil) + assert(Ion.decode[Map[String, Int]]("{}").getOrThrow == Map.empty) + } + } + + "decodes integer samples from iontestdata/good/integer_values.ion and intBinary.ion" in { + val cases = List( + IonCorpusTest.IntCase("integer_values.ion", "0", BigInt(0)), + IonCorpusTest.IntCase("integer_values.ion", "-0", BigInt(0)), + IonCorpusTest.IntCase("integer_values.ion", "0x1234567890abcdef", BigInt("1234567890abcdef", 16)), + IonCorpusTest.IntCase("integer_values.ion", "-0xFFFF", BigInt(-65535)), + IonCorpusTest.IntCase("intBinary.ion", "0b11110000", BigInt(240)), + IonCorpusTest.IntCase("intBinary.ion", "0B010101", BigInt(21)), + IonCorpusTest.IntCase("intsWithUnderscores.ion", "1_2_3", BigInt(123)), + IonCorpusTest.IntCase("intsWithUnderscores.ion", "-0b1111_0000", BigInt(-240)) + ) + + Kyo.foreach(cases) { c => + resource(s"/iontestdata/good/${c.source}").map { content => + assert(content.contains(c.ion)) + assert(Ion.decode[BigInt](c.ion).getOrThrow == c.expected) + } + }.andThen(succeed) + } + + "decodes decimal samples from iontestdata/good/decimal_values.ion" in { + val cases = List( + IonCorpusTest.DecimalCase("decimal_values.ion", "123456d42", BigDecimal("123456E42")), + IonCorpusTest.DecimalCase("decimal_values.ion", "123.456d-42", BigDecimal("123.456E-42")), + IonCorpusTest.DecimalCase("decimalsWithUnderscores.ion", "12_34.56_78", BigDecimal("1234.5678")), + IonCorpusTest.DecimalCase("decimalsWithUnderscores.ion", "12_34.", BigDecimal("1234")), + IonCorpusTest.DecimalCase("decimalsWithUnderscores.ion", "1_2_3_4.5_6_7_8", BigDecimal("1234.5678")) + ) + + Kyo.foreach(cases) { c => + resource(s"/iontestdata/good/${c.source}").map { content => + assert(content.contains(c.ion)) + assert(Ion.decode[BigDecimal](c.ion).getOrThrow == c.expected) + } + }.andThen(succeed) + } + + "decodes float samples from iontestdata/good/floatSpecials.ion" in { + for + specialsRaw <- resource("/iontestdata/good/floatSpecials.ion") + underscores <- resource("/iontestdata/good/floatsWithUnderscores.ion") + yield + val specials = Ion.decode[List[Double]](specialsRaw).getOrThrow + assert(specials.size == 3) + assert(specials(0).isNaN) + assert(specials(1) == Double.PositiveInfinity) + assert(specials(2) == Double.NegativeInfinity) + + val cases = List( + "12_34.56_78e0" -> 1234.5678e0, + "12_34e56" -> 1234e56, + "1_2_3_4.5_6_7_8E90" -> 1234.5678e90 + ) + cases.foreach { (ion, expected) => + assert(underscores.contains(ion)) + assert(Ion.decode[Double](ion).getOrThrow == expected) + } + } + + "decodes string and symbol samples from iontestdata/good/strings.ion and symbols.ion" in { + for + strings <- resource("/iontestdata/good/strings.ion") + symbols <- resource("/iontestdata/good/symbols.ion") + symEmpty <- resource("/iontestdata/good/symbolEmpty.ion") + yield + assert(strings.contains("'''concatenated'''")) + assert( + Ion.decode[String]( + "'''concatenated''' ''' from ''' '''a single line'''" + ).getOrThrow == "concatenated from a single line" + ) + + val escaped = "\"\\0 \\a \\b \\t \\n \\f \\r \\v \\\" \\' \\? \\\\ \\/\"" + assert(Ion.decode[String](escaped).getOrThrow == "\u0000 \u0007 \b \t \n \f \r \u000b \" ' ? \\ /") + + assert(symbols.contains("bareSymbol")) + assert(Ion.decode[String]("bareSymbol").getOrThrow == "bareSymbol") + assert(Ion.decode[String]("'$99'").getOrThrow == "$99") + + assert(symEmpty.contains("{'':abc}")) + assert(Ion.decode[String]("''").getOrThrow == "") + assert(Ion.decode[String]("abc::''").getOrThrow == "") + assert(Ion.decode[Map[String, String]]("{'':abc}").getOrThrow == Map("" -> "abc")) + } + + "decodes blob and clob samples from iontestdata/good/blobs.ion and clobs.ion" in { + val blob = Ion.decode[Span[Byte]]( + """{{ + | YSBiIGMgZCBlIGYgZyBoIGkgaiBrIGwgbSBuIG8gcCBxIHIgcyB0IHUgdiB3IHggeSB6 + |}}""".stripMargin + ).getOrThrow + assert(new String(blob.toArray, StandardCharsets.UTF_8) == "a b c d e f g h i j k l m n o p q r s t u v w x y z") + + val clob = Ion.decode[Span[Byte]]("""{{"a b c d e f g h i j k l m n o p q r s t u v w x y z"}}""").getOrThrow + assert(new String(clob.toArray, StandardCharsets.US_ASCII) == "a b c d e f g h i j k l m n o p q r s t u v w x y z") + } + + "decodes list, sexp, and struct samples from iontestdata/good containers" in { + assert(Ion.decode[List[Int]]("[1, 2, 3, 4, 5]").getOrThrow == List(1, 2, 3, 4, 5)) + assert(Ion.decode[List[Int]]("(1_2_3)").getOrThrow == List(123)) + assert(Ion.decode[IonCorpusTest.UpstreamStruct]("{a:b,c:42,d:{e:f,},g:3}").getOrThrow == + IonCorpusTest.UpstreamStruct("b", 42, Map("e" -> "f"), 3)) + } + + "accepts upstream annotation files while treating annotations as schema metadata" in { + val annotationCases = List( + "annotationQuotedTrue.ion" -> "'true'::23", + "annotationQuotedFalse.ion" -> "'false'::23", + "annotationQuotedNan.ion" -> "'nan'::23", + "annotationQuotedNull.ion" -> "'null'::23", + "annotationQuotedNullInt.ion" -> "'null.int'::23", + "annotationQuotedOperator.ion" -> "'@'::23", + "annotationQuotedPosInf.ion" -> "'+inf'::23", + "annotationQuotedNegInf.ion" -> "'-inf'::23" + ) + + for + multi <- resource("/iontestdata/good/multipleAnnotations.ion") + _ <- Kyo.foreach(annotationCases) { case (file, ion) => + resource(s"/iontestdata/good/$file").map { content => + assert(content.trim == ion) + assert(Ion.decode[Int](ion).getOrThrow == 23) + } + } + annotatedNull <- resource("/iontestdata/good/structFieldAnnotationsUnquotedThenQuoted.ion") + quotedNullField <- resource("/iontestdata/good/fieldNameQuotedNullInt.ion") + yield + assert(multi.trim == "annot1::annot2::value") + assert(Ion.decode[String]("annot1::annot2::value").getOrThrow == "value") + assert(Ion.decode[Map[String, Option[String]]](annotatedNull).getOrThrow == Map("f" -> None)) + assert(Ion.decode[Map[String, Boolean]](quotedNullField).getOrThrow == Map("null.int" -> false)) + end for + } + + "rejects invalid annotation and numeric syntax" in { + val invalid = List( + "null::1", + "null.int::1", + "true::1", + "false::1", + "nan::1", + "null.nope", + "+1", + "0123", + "1_", + "1__2", + "0x_12", + "123_._456", + "123.456_", + "-_123.456", + "12__34.56" + ) + + invalid.foreach { ion => + assert(Ion.decode[Int](ion).isFailure) + } + succeed + } + } + + // The JS test runner's working directory is the build root, while the JVM and Native runners use the + // platform subproject directory. Resolve the repository root by walking up to the directory that holds + // build.sbt, then read the shared corpus from there. This keeps the suite cross-platform with no classpath. + private def corpusRoot(using Frame): Path < Sync = + Path.cwd.map { cwd => + cwd.ancestors.run.map { chain => + def search(candidates: List[Path]): Path < Sync = + candidates match + case Nil => cwd + case head :: tail => + (head / "build.sbt").isRegularFile.map { found => + if found then head else search(tail) + } + search(chain.toList) + } + } + + private def resource(name: String)(using Frame): String < (Sync & Abort[FileReadException]) = + corpusRoot.map { root => + (root / "kyo-schema" / "shared" / "src" / "test" / "resources" / name.stripPrefix("/")) + .read(StandardCharsets.UTF_8) + } + +end IonCorpusTest + +object IonCorpusTest: + + val RequiredResources: List[String] = + List( + "allNulls.ion", + "annotationQuotedFalse.ion", + "annotationQuotedNan.ion", + "annotationQuotedNegInf.ion", + "annotationQuotedNull.ion", + "annotationQuotedNullInt.ion", + "annotationQuotedOperator.ion", + "annotationQuotedPosInf.ion", + "annotationQuotedTrue.ion", + "blobs.ion", + "booleans.ion", + "clobs.ion", + "decimal_values.ion", + "decimalsWithUnderscores.ion", + "fieldNameQuotedNullInt.ion", + "floatSpecials.ion", + "floatsWithUnderscores.ion", + "intBinary.ion", + "integer_values.ion", + "intsWithUnderscores.ion", + "lists.ion", + "multipleAnnotations.ion", + "nonNulls.ion", + "sexps.ion", + "strings.ion", + "structFieldAnnotationsUnquotedThenQuoted.ion", + "structs.ion", + "symbolEmpty.ion", + "symbols.ion", + "whitespace.ion" + ) + + case class IntCase(source: String, ion: String, expected: BigInt) + case class DecimalCase(source: String, ion: String, expected: BigDecimal) + case class UpstreamStruct(a: String, c: Int, d: Map[String, String], g: Int) derives CanEqual + +end IonCorpusTest diff --git a/kyo-schema/shared/src/test/scala/kyo/IonTest.scala b/kyo-schema/shared/src/test/scala/kyo/IonTest.scala new file mode 100644 index 0000000000..17ac88b207 --- /dev/null +++ b/kyo-schema/shared/src/test/scala/kyo/IonTest.scala @@ -0,0 +1,192 @@ +package kyo + +class IonTest extends kyo.test.Test[Any]: + + given CanEqual[Any, Any] = CanEqual.derived + + "encode/decode" - { + + "ion encode simple case class" in { + val ion = Ion.encode[MTPerson](MTPerson("Alice", 30)) + assert(ion == """{name:"Alice",age:30}""") + } + + "ion decode simple case class" in { + val person = Ion.decode[MTPerson]("""{name:"Alice",age:30}""").getOrThrow + assert(person == MTPerson("Alice", 30)) + } + + "ion round-trip case class" in { + val person = MTPerson("Bob", 25) + val encoded = Ion.encodeBytes[MTPerson](person) + val decoded = Ion.decodeBytes[MTPerson](encoded).getOrThrow + assert(decoded == person) + } + + "ion works with nested case classes and collections" in { + val team = MTSmallTeam(MTPerson("Alice", 30), 5) + val encoded = Ion.encode(team) + assert(encoded == """{lead:{name:"Alice",age:30},size:5}""") + assert(Ion.decode[MTSmallTeam](encoded).getOrThrow == team) + + val people = List(MTPerson("Alice", 30), MTPerson("Bob", 25)) + assert(Ion.decode[List[MTPerson]](Ion.encode(people)).getOrThrow == people) + + val scores = Map("a" -> 1, "spaced key" -> 2) + assert(Ion.decode[Map[String, Int]](Ion.encode(scores)).getOrThrow == scores) + } + + "ion encodes byte spans as blobs" in { + val bytes = Span.from("To infinity... and beyond!".getBytes(java.nio.charset.StandardCharsets.UTF_8)) + val encoded = Ion.encode(bytes) + assert(encoded == "{{VG8gaW5maW5pdHkuLi4gYW5kIGJleW9uZCE=}}") + assert(Ion.decode[Span[Byte]](encoded).getOrThrow.toArray.toSeq == bytes.toArray.toSeq) + } + + "ion sealed traits use wrapper structs" in { + val shape: MTShape = MTCircle(5.0) + val encoded = Ion.encode[MTShape](shape) + assert(encoded == "{MTCircle:{radius:5.0e0}}") + assert(Ion.decode[MTShape](encoded).getOrThrow == shape) + } + + "ion discriminator schema reads annotated values" in { + given Schema[MTShape] = Schema[MTShape].discriminator("type") + + val encoded = """shape::{type:"MTRectangle",width:3.0e0,height:4.0e0}""" + val decoded = Ion.decode[MTShape](encoded).getOrThrow + assert(decoded == MTRectangle(3.0, 4.0)) + } + } + + "amazon ion-tests snippets" - { + + "decodes struct spelling variants from iontestdata/good/structs.ion" in { + val inputs = List( + "{a:b,c:42,d:{e:f,},g:3}", + "{'a':'b','c':42,'d':{'e':'f',},'g':3}", + """{"a":"b","c":42,"d":{"e":"f",},"g":3}""", + "{a: b, c: 42, d: {e: f, }, g: 3, }" + ) + + inputs.foreach { input => + val decoded = Ion.decode[IonTest.UpstreamStruct](input).getOrThrow + assert(decoded == IonTest.UpstreamStruct("b", 42, Map("e" -> "f"), 3)) + } + succeed + } + + "decodes long field names from iontestdata/good/structs.ion" in { + val input = + "{ '''123456789ABCDEF''' '''123456789ABCDEF''' : v }" + val decoded = Ion.decode[Map[String, String]](input).getOrThrow + assert(decoded == Map("123456789ABCDEF123456789ABCDEF" -> "v")) + } + + "decodes escaped and long strings from iontestdata/good/strings.ion" in { + assert(Ion.decode[String]("\"\\uABCD\"").getOrThrow == "\uABCD") + + val concat = "'''concatenated''' ''' from ''' '''a single line'''" + assert(Ion.decode[String](concat).getOrThrow == "concatenated from a single line") + + val escaped = "\"\\0 \\a \\b \\t \\n \\f \\r \\v \\\" \\' \\? \\\\ \\/\"" + assert(Ion.decode[String](escaped).getOrThrow == "\u0000 \u0007 \b \t \n \f \r \u000b \" ' ? \\ /") + } + + "decodes booleans from iontestdata/good/booleans.ion" in { + val upstream = "true\nfalse\n" + assert(upstream.linesIterator.toList == List("true", "false")) + assert(Ion.decode[Boolean]("true").getOrThrow) + assert(!Ion.decode[Boolean]("false").getOrThrow) + } + + "decodes blobs from iontestdata/good/blobs.ion" in { + val alphabet = + """{{ + | YSBiIGMgZCBlIGYgZyBoIGkgaiBrIGwgbSBuIG8gcCBxIHIgcyB0IHUgdiB3IHggeSB6 + |}}""".stripMargin + val decoded = Ion.decode[Span[Byte]](alphabet).getOrThrow + assert(new String( + decoded.toArray, + java.nio.charset.StandardCharsets.UTF_8 + ) == "a b c d e f g h i j k l m n o p q r s t u v w x y z") + + val withSlashes = "{{ //79/PsAAQIDBAU= }}" + assert( + Ion.decode[Span[Byte]](withSlashes).getOrThrow.toArray.toSeq == java.util.Base64.getDecoder.decode("//79/PsAAQIDBAU=").toSeq + ) + } + + "preserves high byte CLOB escapes from iontestdata/good/clobs.ion" in { + val decoded = Ion.decode[Span[Byte]]("""{{"\xFf"}}""").getOrThrow + assert(decoded.toArray.toSeq == Seq(0xff.toByte)) + } + + "ignores comments as whitespace like iontestdata/good/whitespace.ion" in { + val input = + """// before + |{ + | name: "Fido", /* between */ + | age: years::4, + | toys: [ball, rope,], + |} + |""".stripMargin + val decoded = Ion.decode[IonTest.Pet](input).getOrThrow + assert(decoded == IonTest.Pet("Fido", 4, List("ball", "rope"))) + } + } + + "scalar mappings" - { + + "typed nulls decode as absent optional values" in { + assert(Ion.decode[Option[Int]]("null.int").getOrThrow == None) + assert(Ion.decode[Maybe[String]]("null.string").getOrThrow == Maybe.empty) + assert(Ion.decode[Option[Int]]("0").getOrThrow == Some(0)) + } + + "decodes Ion integer and decimal syntax" in { + assert(Ion.decode[Int]("0").getOrThrow == 0) + assert(Ion.decode[Int]("0x7b").getOrThrow == 123) + assert(Ion.decode[Int]("0b1111011").getOrThrow == 123) + assert(Ion.decode[BigDecimal]("1.25d2").getOrThrow == BigDecimal("125")) + } + + "decodes Ion float specials" in { + assert(Ion.decode[Double]("+inf").getOrThrow == Double.PositiveInfinity) + assert(Ion.decode[Double]("-inf").getOrThrow == Double.NegativeInfinity) + assert(Ion.decode[Double]("nan").getOrThrow.isNaN) + } + + "rejects non-finite Ion floats for BigDecimal" in { + assert(Ion.decode[BigDecimal]("nan").isFailure) + assert(Ion.decode[BigDecimal]("+inf").isFailure) + } + + "rejects trailing content after the decoded root value" in { + assert(Ion.decode[Int]("0 1").isFailure) + } + + "rejects Unicode escapes above the valid code point range" in { + Ion.decode[String]("\"\\UFFFFFFFF\"") match + case Result.Failure(e: ParseException) => assert(e.getMessage.contains("Unicode code point")) + case other => fail(s"Expected ParseException failure, got $other") + end match + } + + "decodes timestamp tokens as instants" in { + val instant = java.time.Instant.parse("2024-01-02T03:04:05Z") + assert(Ion.decode[java.time.Instant]("2024-01-02T03:04:05Z").getOrThrow == instant) + } + + "enforces decode limits" in { + assert(Ion.decode[List[List[Int]]]("[[1]]", maxDepth = 1).isFailure) + assert(Ion.decode[List[Int]]("[1,2]", maxCollectionSize = 1).isFailure) + } + } + +end IonTest + +object IonTest: + case class UpstreamStruct(a: String, c: Int, d: Map[String, String], g: Int) derives CanEqual + case class Pet(name: String, age: Int, toys: List[String]) derives CanEqual +end IonTest