From 8041b586477fb4363945c1588bc52b15aa5f9cab Mon Sep 17 00:00:00 2001 From: Jack Harper Date: Mon, 30 Jun 2025 10:29:53 +0100 Subject: [PATCH 1/6] Update Datastreaming.md --- doc/specific_iocs/dae/Datastreaming.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/specific_iocs/dae/Datastreaming.md b/doc/specific_iocs/dae/Datastreaming.md index 473f53e04..adbd079df 100644 --- a/doc/specific_iocs/dae/Datastreaming.md +++ b/doc/specific_iocs/dae/Datastreaming.md @@ -13,6 +13,8 @@ The datastreaming system is being built as part of in-kind work to ESS. It will In general the system works by passing both neutron and SE data into [Kafka](https://kafka.apache.org/) and having clients that either view data live (like Mantid) or write the data to file, additional information can be found [here](http://accelconf.web.cern.ch/AccelConf/icalepcs2017/papers/tupha029.pdf) and [here](https://iopscience.iop.org/article/10.1088/1742-6596/1021/1/012013). +All data is passed into flatbuffers using [these schemas](https://github.com/ess-dmsc/streaming-data-types) - we have a tool called [saluki](https://github.com/ISISComputingGroup/saluki) which can deserialise these and make them human-readable after they've been put into Kafka. + The datastreaming layout proposed looks something like this, not including the Mantid steps or anything before event data is collected: ![](ESSDSLayout.png) From 25ee7c265d935aa53d926bd80db462c1ecd91fc2 Mon Sep 17 00:00:00 2001 From: Jack Harper Date: Mon, 30 Jun 2025 11:12:16 +0100 Subject: [PATCH 2/6] Freedom --- doc/specific_iocs/dae/Datastreaming.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/specific_iocs/dae/Datastreaming.md b/doc/specific_iocs/dae/Datastreaming.md index adbd079df..3e396e5c7 100644 --- a/doc/specific_iocs/dae/Datastreaming.md +++ b/doc/specific_iocs/dae/Datastreaming.md @@ -13,7 +13,7 @@ The datastreaming system is being built as part of in-kind work to ESS. It will In general the system works by passing both neutron and SE data into [Kafka](https://kafka.apache.org/) and having clients that either view data live (like Mantid) or write the data to file, additional information can be found [here](http://accelconf.web.cern.ch/AccelConf/icalepcs2017/papers/tupha029.pdf) and [here](https://iopscience.iop.org/article/10.1088/1742-6596/1021/1/012013). -All data is passed into flatbuffers using [these schemas](https://github.com/ess-dmsc/streaming-data-types) - we have a tool called [saluki](https://github.com/ISISComputingGroup/saluki) which can deserialise these and make them human-readable after they've been put into Kafka. +All data is passed into flatbuffers using [these schemas](https://github.com/ess-dmsc/streaming-data-types) - we have a tool called [saluki](https://github.com/ISISComputingGroup/saluki) which can deserialize these and make them human-readable after they've been put into Kafka. The datastreaming layout proposed looks something like this, not including the Mantid steps or anything before event data is collected: From 5be10fe5f52efbbb4ddc1c7df97857af5f5eb6a8 Mon Sep 17 00:00:00 2001 From: Jack Harper Date: Mon, 30 Jun 2025 11:15:48 +0100 Subject: [PATCH 3/6] add kafka to spelling list --- doc/spelling_wordlist.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/spelling_wordlist.txt b/doc/spelling_wordlist.txt index 5f72c6d4d..a7971c83e 100644 --- a/doc/spelling_wordlist.txt +++ b/doc/spelling_wordlist.txt @@ -410,6 +410,7 @@ junit jvisualvm Jython Jülich +Kafka Kammrath Kanban kbaud From 97429c90cf3357c27ec2dce0a108a21e6086c95e Mon Sep 17 00:00:00 2001 From: Jack Harper Date: Mon, 30 Jun 2025 11:17:24 +0100 Subject: [PATCH 4/6] fgdsfgdsfgdsfgdsfgd --- doc/spelling_wordlist.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/spelling_wordlist.txt b/doc/spelling_wordlist.txt index a7971c83e..ec007eecb 100644 --- a/doc/spelling_wordlist.txt +++ b/doc/spelling_wordlist.txt @@ -192,6 +192,7 @@ deenergise dehex deprioritise dereferenced +deserialize Det dev devsim From 478ef0cc23ce2f14cf8daaba161da473e29096ee Mon Sep 17 00:00:00 2001 From: Jack Harper Date: Mon, 30 Jun 2025 11:21:33 +0100 Subject: [PATCH 5/6] fdgh --- doc/spelling_wordlist.txt | 1 - 1 file changed, 1 deletion(-) diff --git a/doc/spelling_wordlist.txt b/doc/spelling_wordlist.txt index ec007eecb..2cb6c2986 100644 --- a/doc/spelling_wordlist.txt +++ b/doc/spelling_wordlist.txt @@ -411,7 +411,6 @@ junit jvisualvm Jython Jülich -Kafka Kammrath Kanban kbaud From 34639e018622cf7fc2de3eaba2ad76608f5b0e0b Mon Sep 17 00:00:00 2001 From: Jack Harper Date: Mon, 30 Jun 2025 11:26:48 +0100 Subject: [PATCH 6/6] quickly commit before tom gets back his PC --- doc/specific_iocs/dae/Datastreaming.md | 2 +- doc/spelling_wordlist.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/specific_iocs/dae/Datastreaming.md b/doc/specific_iocs/dae/Datastreaming.md index 3e396e5c7..adbd079df 100644 --- a/doc/specific_iocs/dae/Datastreaming.md +++ b/doc/specific_iocs/dae/Datastreaming.md @@ -13,7 +13,7 @@ The datastreaming system is being built as part of in-kind work to ESS. It will In general the system works by passing both neutron and SE data into [Kafka](https://kafka.apache.org/) and having clients that either view data live (like Mantid) or write the data to file, additional information can be found [here](http://accelconf.web.cern.ch/AccelConf/icalepcs2017/papers/tupha029.pdf) and [here](https://iopscience.iop.org/article/10.1088/1742-6596/1021/1/012013). -All data is passed into flatbuffers using [these schemas](https://github.com/ess-dmsc/streaming-data-types) - we have a tool called [saluki](https://github.com/ISISComputingGroup/saluki) which can deserialize these and make them human-readable after they've been put into Kafka. +All data is passed into flatbuffers using [these schemas](https://github.com/ess-dmsc/streaming-data-types) - we have a tool called [saluki](https://github.com/ISISComputingGroup/saluki) which can deserialise these and make them human-readable after they've been put into Kafka. The datastreaming layout proposed looks something like this, not including the Mantid steps or anything before event data is collected: diff --git a/doc/spelling_wordlist.txt b/doc/spelling_wordlist.txt index 2cb6c2986..8a8dbe86f 100644 --- a/doc/spelling_wordlist.txt +++ b/doc/spelling_wordlist.txt @@ -192,7 +192,7 @@ deenergise dehex deprioritise dereferenced -deserialize +deserialise Det dev devsim