Releases: snowplow/stream-collector
2.2.1
2.2.0
This release adds a new module that allows SQS to be configured as a sink.
Changes
Many thanks to miike for contributing to this release.
Discourse
2.1.2
2.1.1
This release fixes a bug which meant size violation and generic error (illegal querystring) bad rows were not emitted as self-describing events.
Bugfixes
- Emit valid bad rows for size violation and generic error (#113)
Misc
- Extend copyright to 2021 (#114)
Discourse: https://discourse.snowplowanalytics.com/t/stream-collector-2-1-1-released/4707
2.1.0
This release introduces three changes to how the collector works:
- Support for tracker-controlled "cookieless" mode.
- A rare scenario in which some malformed events could get dropped will now result in a bad row instead.
- It is no longer required to have a Kinesis stream running in order to deploy the collector on AWS.
Collector behaviour changes
- Do not set cookie if request has SP-Anonymous header (#90)
- Generate BadRow if querystring cannot be parsed (#73)
- Don't crash but warn if Kinesis stream and SQS queue don't exist (#100)
Dependency management
- Bump specs2-core to 4.10.5 (#106)
Under the hood
- Bump sbt to 1.4.4 (#105)
- Migrate from Travis to GH actions (#91)
- Bump to JDK 11 (#92)
- Bump base-debian to 0.2.1 (#72)
- Integrate coursier cache action (#93)
- Fix assembly merge strategy (#97)
Misc
2.0.1
This patch release overhauls the put retry logic for both SQS and Kinesis.
Patched retry logic
- Increase number of Kinesis put retries when surge protection is on (#75)
Dependency management
2.0.0 - Adding surge protection for AWS
Surge protection
While PubSub is auto-scalable, Kinesis is not, and when Kinesis can't keep up with the throughput (in case of extreme traffic spikes for instance), events accumulate in collector memory for later (infinite) retries, which can lead to OutOfMemory errors.
To address this potential issue, it is now possible to specify a SQS buffer where to write events to in case of a problem when writing to Kinesis (e.g. maximum throughput exceeded).
To activate this feature, create the SQS queues (one for enriched events and one for bad rows) and populate these lines in the configuration of the collector. Please note that writing to SQS will be retried maximum 10 times and that the size limit of an event drops to 192kb.
To send events back from SQS to Kinesis, this app can be used. It is available on Docker Hub. To run it, specify SQS_QUEUE, KINESIS_STREAM_NAME and SENTRY_DSN environment variables. The writing to Kinesis is performed with back-pressure to not overwhelm it.
Changelog
- Disable default redirect (#64)
- Bump vulnerable libs (#56)
- Implement surge protection (#57)
- Add test for extracting a URL-encoded schema from the querystring (#60)
- Add snyk monitor (#52)
- Add DockerHub credentials to .travis.yml (#49)
- Add travis integration (#59)
Breaking change
- Disable default redirect (#64)
