Well known mime types and Composite metadata extension (#635)

* Add encodeUnsignedMedium to NumberUtils

Signed-off-by: Simon Baslé <[email protected]>

* Add WellKnownMimeType enum with String/ID conversions

Signed-off-by: Simon Baslé <[email protected]>

* Add a flyweight capable of encoding Composite Metadata

The flyweight also exposes a method to decode Composite Metadata to a
Map.

Signed-off-by: Simon Baslé <[email protected]>

* Move flyweight into metadata package, change decodeToMap to decodeNext

Signed-off-by: Simon Baslé <[email protected]>

* Add CompositeMetadata interface with factory methods to decode/encode

The API is intended as a readonly way of listing the composite metadata
mime-buffer pairs, as well as accessing metadata pairs by mime type or
index, including the case where a mime type is associated with several
metadata entries (`List<Entry> getAll(String)`)...

Signed-off-by: Simon Baslé <[email protected]>

* Add first test for CompositeMetadata (decode smoke test)

Signed-off-by: Simon Baslé <[email protected]>

* Expose incremental decode method that decodes a single metadata entry

Signed-off-by: Simon Baslé <[email protected]>

* WellKnownMimeType parsing now avoids exceptions (except null arguments)

Instead has 2 special enums. int parsing allows detection of potentially
legit type that is still unknown to the current implementation, which
could help making it more future-proof.

Signed-off-by: Simon Baslé <[email protected]>

* rework encoding / decoding API around Entry, accommodate 3 mime case

Encoding becomes a mirror of decoding, purely based on Entry.

Accommodate 3 flavors of mime type decoding:

 - compressed well known with id matching an enum
 - uncompressed custom string
 - compressed but known as reserved for future use (future proofness)

 The later can be decoded into a special Entry that will allow re
 encoding it and transmitting it to other nodes as it was.

Signed-off-by: Simon Baslé <[email protected]>

* No need for a CompositeMetadata interface, single concrete implem

Signed-off-by: Simon Baslé <[email protected]>

* Expose the CompositeMetadataFlyweight for low-level, simplify API

 - the flyweight deals with low-level encoding/decoding with
 minimal garbage and only uses exceptions in the encoding path
 -- encoding a non-ASCII custom mime type OR
 -- encoding an out of range byte
 - the CompositeMetadata is an _option_ for a higher level
 Iterable-based API, but which instantiates new types

Signed-off-by: Simon Baslé <[email protected]>

* Remove CompositeMetadata, make Entry child of flyweight

Entry is now the only high-level abstraction, will need a way to
decode into a List<Entry>.

Signed-off-by: Simon Baslé <[email protected]>

* Add javadoc for CompositeMetadataFlyweight methods

Signed-off-by: Simon Baslé <[email protected]>

* Switch to a single Entry implementation, remove isPassthrough

Signed-off-by: Simon Baslé <[email protected]>

* Rename Entry#decodeEntry to #decode, add #decodeAll

Signed-off-by: Simon Baslé <[email protected]>

* apply Google-style formatting

Signed-off-by: Simon Baslé <[email protected]>

* Use precomputed array for WellKnownMimeType.fromId

Signed-off-by: Simon Baslé <[email protected]>

* Use precomputed Map<String,WellKnownMimeType> for fromMimeType + jmh benchmark to validate perf increase

Signed-off-by: Simon Baslé <[email protected]>

* Several custom mime metadata header encoding improvements:

 - avoid the use of a CompositeByteBuf
 - length can be safely predicted due to requirement that custom mime
 type be ASCII only
 - still using UTF8 writing and ByteBufUtil.isText(ASCII) to detect
 non-ascii text as:
   - writeAscii would not reject non-ascii chars but replace with `?`
   - ISO-8859 chars (eg. Latin-1) could be considered valid when only
   checking the number of written bytes
  - use of ByteBufUtil.writeUtf8 to simplify encoding the mime into the
  preallocated buffer
 - release the buffer if there is an encoding exception

Signed-off-by: Simon Baslé <[email protected]>

* Use Unpooled buffers in tests

Signed-off-by: Simon Baslé <[email protected]>

* Remove Entry, replace with CompositeMetadata Iterator-like abstraction

The CompositeMetadata wraps a ByteBuf and exposes a hasNext/decodeNext
API (reminiscent of the Iterator API) as a facade over the flyweight's
methods.

Each round decodeNext() is correctly invoked, the CompositeMetadata
state is updated and the getters can be used to retrieve decoded entry
components.

Signed-off-by: Simon Baslé <[email protected]>

* Add an encode flyweight method that attempts compressing String

This is a qol method that will first check if the String passed can be
mapped to a WellKnownMimeType (and thus compressed to a 1 byte
representation).

Signed-off-by: Simon Baslé <[email protected]>

* Avoid moving full buffer's readerIndex when slicing entries

This implies that the index from which to decode and slice the next
entry MUST be provided by the user. To that end, we provide a method
to compute the next entry's index, given the pair of ByteBuf slices
returned by decodeMimeAndContentBuffers.

Also renamed that method to decodeMimeAndContentBufferSlices, to make
it more obvious that it produces slices (readerIndex impact and slices
nature are mentioned in the javadoc).

CompositeMetadata now keeps the nextEntryIndex as state and uses that
for hasNext / decodeNext.

Signed-off-by: Simon Baslé <[email protected]>

* CompositeMetadata as Iterable

Previously, the implementation of CompositeMetadata resulted in a
mutable type that didn't fit in with either imperative (for) or reactive
(fromIterable) iteration strategies.  This change updates the type to
implement Iterable, and return an Iterator that lazily traverses the
ByteBuf.

There are also some improvements to the Flyweight in order to support
this iteration style.

Signed-off-by: Ben Hale <[email protected]>

* Apply googleFormat

Signed-off-by: Simon Baslé <[email protected]>

Author: simonbasle

rsocket-core/src/jmh/java/io/rsocket/metadata/WellKnownMimeTypePerf.java

@@ -0,0 +1,96 @@
+package io.rsocket.metadata;
+
+import org.openjdk.jmh.annotations.*;
+import org.openjdk.jmh.infra.Blackhole;
+
+@BenchmarkMode(Mode.Throughput)
+@Fork(value = 1)
+@Warmup(iterations = 10)
+@Measurement(iterations = 10)
+@State(Scope.Thread)
+public class WellKnownMimeTypePerf {
+
+  // this is the old values() looping implementation of fromIdentifier
+  private WellKnownMimeType fromIdValuesLoop(int id) {
+    if (id < 0 || id > 127) {
+      return WellKnownMimeType.UNPARSEABLE_MIME_TYPE;
+    }
+    for (WellKnownMimeType value : WellKnownMimeType.values()) {
+      if (value.getIdentifier() == id) {
+        return value;
+      }
+    }
+    return WellKnownMimeType.UNKNOWN_RESERVED_MIME_TYPE;
+  }
+
+  // this is the core of the old values() looping implementation of fromString
+  private WellKnownMimeType fromStringValuesLoop(String mimeType) {
+    for (WellKnownMimeType value : WellKnownMimeType.values()) {
+      if (mimeType.equals(value.getString())) {
+        return value;
+      }
+    }
+    return WellKnownMimeType.UNPARSEABLE_MIME_TYPE;
+  }
+
+  @Benchmark
+  public void fromIdArrayLookup(final Blackhole bh) {
+    // negative lookup
+    bh.consume(WellKnownMimeType.fromIdentifier(-10));
+    bh.consume(WellKnownMimeType.fromIdentifier(-1));
+    // too large lookup
+    bh.consume(WellKnownMimeType.fromIdentifier(129));
+    // first lookup
+    bh.consume(WellKnownMimeType.fromIdentifier(0));
+    // middle lookup
+    bh.consume(WellKnownMimeType.fromIdentifier(37));
+    // reserved lookup
+    bh.consume(WellKnownMimeType.fromIdentifier(63));
+    // last lookup
+    bh.consume(WellKnownMimeType.fromIdentifier(127));
+  }
+
+  @Benchmark
+  public void fromIdValuesLoopLookup(final Blackhole bh) {
+    // negative lookup
+    bh.consume(fromIdValuesLoop(-10));
+    bh.consume(fromIdValuesLoop(-1));
+    // too large lookup
+    bh.consume(fromIdValuesLoop(129));
+    // first lookup
+    bh.consume(fromIdValuesLoop(0));
+    // middle lookup
+    bh.consume(fromIdValuesLoop(37));
+    // reserved lookup
+    bh.consume(fromIdValuesLoop(63));
+    // last lookup
+    bh.consume(fromIdValuesLoop(127));
+  }
+
+  @Benchmark
+  public void fromStringMapLookup(final Blackhole bh) {
+    // unknown lookup
+    bh.consume(WellKnownMimeType.fromString("foo/bar"));
+    // first lookup
+    bh.consume(WellKnownMimeType.fromString(WellKnownMimeType.APPLICATION_AVRO.getString()));
+    // middle lookup
+    bh.consume(WellKnownMimeType.fromString(WellKnownMimeType.VIDEO_VP8.getString()));
+    // last lookup
+    bh.consume(
+        WellKnownMimeType.fromString(
+            WellKnownMimeType.MESSAGE_RSOCKET_COMPOSITE_METADATA.getString()));
+  }
+
+  @Benchmark
+  public void fromStringValuesLoopLookup(final Blackhole bh) {
+    // unknown lookup
+    bh.consume(fromStringValuesLoop("foo/bar"));
+    // first lookup
+    bh.consume(fromStringValuesLoop(WellKnownMimeType.APPLICATION_AVRO.getString()));
+    // middle lookup
+    bh.consume(fromStringValuesLoop(WellKnownMimeType.VIDEO_VP8.getString()));
+    // last lookup
+    bh.consume(
+        fromStringValuesLoop(WellKnownMimeType.MESSAGE_RSOCKET_COMPOSITE_METADATA.getString()));
+  }
+}

rsocket-core/src/main/java/io/rsocket/metadata/CompositeMetadata.java

@@ -0,0 +1,220 @@
+/*
+ * Copyright 2015-2018 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package io.rsocket.metadata;
+
+import static io.rsocket.metadata.CompositeMetadataFlyweight.computeNextEntryIndex;
+import static io.rsocket.metadata.CompositeMetadataFlyweight.decodeMimeAndContentBuffersSlices;
+import static io.rsocket.metadata.CompositeMetadataFlyweight.decodeMimeIdFromMimeBuffer;
+import static io.rsocket.metadata.CompositeMetadataFlyweight.decodeMimeTypeFromMimeBuffer;
+import static io.rsocket.metadata.CompositeMetadataFlyweight.hasEntry;
+import static io.rsocket.metadata.CompositeMetadataFlyweight.isWellKnownMimeType;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.ByteBufAllocator;
+import io.netty.buffer.CompositeByteBuf;
+import io.rsocket.metadata.CompositeMetadata.Entry;
+import java.util.Iterator;
+import reactor.util.annotation.Nullable;
+
+/**
+ * An {@link Iterable} wrapper around a {@link ByteBuf} that exposes metadata entry information at
+ * each decoding step. This is only possible on frame types used to initiate interactions, if the
+ * SETUP metadata mime type was {@link WellKnownMimeType#MESSAGE_RSOCKET_COMPOSITE_METADATA}.
+ *
+ * <p>This allows efficient incremental decoding of the entries (without moving the source's {@link
+ * io.netty.buffer.ByteBuf#readerIndex()}). The buffer is assumed to contain just enough bytes to
+ * represent one or more entries (mime type compressed or not). The decoding stops when the buffer
+ * reaches 0 readable bytes, and fails if it contains bytes but not enough to correctly decode an
+ * entry.
+ *
+ * <p>A note on future-proofness: it is possible to come across a compressed mime type that this
+ * implementation doesn't recognize. This is likely to be due to the use of a byte id that is merely
+ * reserved in this implementation, but maps to a {@link WellKnownMimeType} in the implementation
+ * that encoded the metadata. This can be detected by detecting that an entry is a {@link
+ * ReservedMimeTypeEntry}. In this case {@link Entry#getMimeType()} will return {@code null}. The
+ * encoded id can be retrieved using {@link ReservedMimeTypeEntry#getType()}. The byte and content
+ * buffer should be kept around and re-encoded using {@link
+ * CompositeMetadataFlyweight#encodeAndAddMetadata(CompositeByteBuf, ByteBufAllocator, byte,
+ * ByteBuf)} in case passing that entry through is required.
+ */
+public final class CompositeMetadata implements Iterable<Entry> {
+
+  private final boolean retainSlices;
+
+  private final ByteBuf source;
+
+  public CompositeMetadata(ByteBuf source, boolean retainSlices) {
+    this.source = source;
+    this.retainSlices = retainSlices;
+  }
+
+  @Override
+  public Iterator<Entry> iterator() {
+    return new Iterator<Entry>() {
+
+      private int entryIndex = 0;
+
+      @Override
+      public boolean hasNext() {
+        return hasEntry(CompositeMetadata.this.source, this.entryIndex);
+      }
+
+      @Override
+      public Entry next() {
+        ByteBuf[] headerAndData =
+            decodeMimeAndContentBuffersSlices(
+                CompositeMetadata.this.source,
+                this.entryIndex,
+                CompositeMetadata.this.retainSlices);
+
+        ByteBuf header = headerAndData[0];
+        ByteBuf data = headerAndData[1];
+
+        this.entryIndex = computeNextEntryIndex(this.entryIndex, header, data);
+
+        if (!isWellKnownMimeType(header)) {
+          CharSequence typeString = decodeMimeTypeFromMimeBuffer(header);
+          if (typeString == null) {
+            throw new IllegalStateException("MIME type cannot be null");
+          }
+
+          return new ExplicitMimeTimeEntry(data, typeString.toString());
+        }
+
+        byte id = decodeMimeIdFromMimeBuffer(header);
+        WellKnownMimeType type = WellKnownMimeType.fromIdentifier(id);
+
+        if (WellKnownMimeType.UNKNOWN_RESERVED_MIME_TYPE == type) {
+          return new ReservedMimeTypeEntry(data, id);
+        }
+
+        return new WellKnownMimeTypeEntry(data, type);
+      }
+    };
+  }
+
+  /** An entry in the {@link CompositeMetadata}. */
+  public interface Entry {
+
+    /**
+     * Returns the un-decoded content of the {@link Entry}.
+     *
+     * @return the un-decoded content of the {@link Entry}
+     */
+    ByteBuf getContent();
+
+    /**
+     * Returns the MIME type of the entry, if it can be decoded.
+     *
+     * @return the MIME type of the entry, if it can be decoded, otherwise {@code null}.
+     */
+    @Nullable
+    String getMimeType();
+  }
+
+  /** An {@link Entry} backed by an explicitly declared MIME type. */
+  public static final class ExplicitMimeTimeEntry implements Entry {
+
+    private final ByteBuf content;
+
+    private final String type;
+
+    public ExplicitMimeTimeEntry(ByteBuf content, String type) {
+      this.content = content;
+      this.type = type;
+    }
+
+    @Override
+    public ByteBuf getContent() {
+      return this.content;
+    }
+
+    @Override
+    public String getMimeType() {
+      return this.type;
+    }
+  }
+
+  /**
+   * An {@link Entry} backed by a {@link WellKnownMimeType} entry, but one that is not understood by
+   * this implementation.
+   */
+  public static final class ReservedMimeTypeEntry implements Entry {
+    private final ByteBuf content;
+    private final int type;
+
+    public ReservedMimeTypeEntry(ByteBuf content, int type) {
+      this.content = content;
+      this.type = type;
+    }
+
+    @Override
+    public ByteBuf getContent() {
+      return this.content;
+    }
+
+    /**
+     * {@inheritDoc} Since this entry represents a compressed id that couldn't be decoded, this is
+     * always {@code null}.
+     */
+    @Override
+    public String getMimeType() {
+      return null;
+    }
+
+    /**
+     * Returns the reserved, but unknown {@link WellKnownMimeType} for this entry. Range is 0-127
+     * (inclusive).
+     *
+     * @return the reserved, but unknown {@link WellKnownMimeType} for this entry
+     */
+    public int getType() {
+      return this.type;
+    }
+  }
+
+  /** An {@link Entry} backed by a {@link WellKnownMimeType}. */
+  public static final class WellKnownMimeTypeEntry implements Entry {
+
+    private final ByteBuf content;
+    private final WellKnownMimeType type;
+
+    public WellKnownMimeTypeEntry(ByteBuf content, WellKnownMimeType type) {
+      this.content = content;
+      this.type = type;
+    }
+
+    @Override
+    public ByteBuf getContent() {
+      return this.content;
+    }
+
+    @Override
+    public String getMimeType() {
+      return this.type.getString();
+    }
+
+    /**
+     * Returns the {@link WellKnownMimeType} for this entry.
+     *
+     * @return the {@link WellKnownMimeType} for this entry
+     */
+    public WellKnownMimeType getType() {
+      return this.type;
+    }
+  }
+}