Dynamically deserialize networktables structs #225

onlycs · 2025-02-21T17:37:14Z

Very simply, we get struct information by listening for all announced topics and if it is has type structschema, we store it and parse as-needed into a DynStructSchema (lib/services/struct_schemas/dyn_struct.dart). When we subscribe to or get data from a topic that uses a schema, we provide DynStructSchema along with the Uint8List to DynStruct which recursively parses struct fields and values.

I also made saving actually save to a file

Gold872

Sorry for taking so long to get to this!

I'm very impressed that you were able to get this implemented! In the past I've avoided adding struct decoding since it felt out of scope with Elastic, but with more recent (and planned) struct integration throughout WPILib it would make sense to have the ability to display (but not edit) individual struct fields.

That being said though, there's still some things I would like to have changed/fixed. Firstly, I'm a bit confused between the difference between NT4StructMeta and DynStructSchema. I'm also not sure if it entirely makes sense to have the widgets store the entire schema in its json, since it means that any time a struct field widget is created, it will assume that the schema has never changed. For example, if a widget is created with a struct schema for some type of Swerve State, but then the dashboard connects to a robot with a different struct that has the same name, the widget will always assume that it's the same which can cause invalid values to be displayed. It might make sense to instead store the name of the struct and its field, and have it looked up whenever a new value comes in.

Lastly, I would want to have unit tests for the schema parsing, decoding, and displaying. Right now that might not be easy to add since there's a lot of stuff still being changed around, but I would definitely want lots of these new features in unit tests before being merged. They can be a pain to write, but they've saved me from breaking a lot of important features.

Gold872 · 2025-03-12T18:21:38Z

lib/pages/dashboard_page.dart

    } catch (e) {
      _showJsonLoadingError(e.toString());
      return;
    }

-    await preferences.setString(PrefKeys.layout, jsonEncode(jsonData));
+    await preferences.setString(PrefKeys.layoutPath, file.path);

    setState(() => _loadLayoutFromJsonData(jsonString));
  }


The layout is stored in Shared Preferences to avoid an issue where deleting the file would delete the layout too. It could make sense to have the last path as an "initial directory" for the user, but when importing a layout I don't think it would be expected for that to turn into the layout location.

Yeah I made this change cuz I was kind of annoyed when I clicked save and elastic didn't save to the file. I can make this save to user prefs as well and load it when the json is deleted.

lib/services/log.dart

Gold872 · 2025-03-12T18:25:01Z

lib/services/nt_connection.dart

+      if (topic.type == "structschema") {
+        subscribe(topic.name).listen((data, _) {
+          if (data == null) return;
+          String schema = String.fromCharCodes(data as List<int>);


I'd prefer having some sort of type checking, since otherwise what could happen is the app suddenly breaks without any notification to the user

Type checking as in? That's kind of the only way (that I know how, at least) to check whether an NT field is a struct schema... Struct values are prefixed with struct:.

Probably misunderstood you, not really sure what you mean.

Also wouldn't the app break if we suddenly switched some published NT field from a string to a double between launches?

Also, I'm like 80% sure the code as it stands will bail out by displaying the raw bytes if the schema suddenly changes like that

Gold872 · 2025-03-12T18:26:02Z

lib/services/nt_widget_builder.dart

 });

 typedef NTModelProvider = NTWidgetModel Function({
  String dataType,
  required NTConnection ntConnection,
  double period,
  required SharedPreferences preferences,
+  required NT4StructMeta? ntStructMeta,


I don't think a nullable field should be required, since for models that don't depend on a struct subscription (any multi-topic widget), it doesn't entirely make sense for it to have to be given a struct meta

I'm not sure how to do this without the required field, the register() function would need an overhaul, probably with a seperate type for nt multitopic model providers.

I'll begin work on this

Gold872 · 2025-03-12T18:27:01Z

lib/services/shuffleboard_nt_listener.dart


-      bool lastElement = currentTopic == topic;
+      // bool lastElement = currentTopic == topic;

      if (current != null) {
        if (current.hasRow(row)) {
          current = current.getRow(row);
        } else {
-          current = current.createNewRow(
-              topic: currentTopic,
-              name: row,
-              ntTopic: (lastElement) ? nt4Topic : null);
+          current =
+              current.createNewRow(topic: currentTopic, name: row, entry: null);
        }
      } else {
        if (shuffleboardTreeRoot.hasRow(row)) {
          current = shuffleboardTreeRoot.getRow(row);
        } else {
          current = shuffleboardTreeRoot.createNewRow(
-              topic: currentTopic,
-              name: row,
-              ntTopic: (lastElement) ? nt4Topic : null);
+              topic: currentTopic, name: row, entry: null);


There isn't any issue here but I do plan on removing the Shuffleboard API support for 2026, so these changes won't be needed

Gold872 · 2025-03-12T18:27:45Z

lib/services/struct_schemas/dyn_struct.dart

+
+import 'package:dot_cast/dot_cast.dart';
+
+class DynStructField {


I prefer keeping class names less truncated, maybe rename this to DynamicStructField?

Gold872 · 2025-03-12T18:29:06Z

lib/services/struct_schemas/dyn_struct.dart

+sealed class DynStructValue {
+  final Object? anyValue;
+
+  DynStructValue({required this.anyValue});
+
+  bool get boolValue => (this as DynStructBoolean).value;
+  int get intValue => (this as DynStructInt).value;
+  int get longValue => (this as DynStructLong).value;
+  double get floatValue => (this as DynStructFloat).value;
+  double get doubleValue => (this as DynStructDouble).value;
+  String get stringValue => (this as DynStructString).value;
+  DynStructValue? get nullableValue => (this as DynStructNullable).value;
+  List<DynStructValue> get arrayValue => (this as DynStructArray).value;
+  DynStruct get structValue => (this as DynStructStruct).value;
+}


It might be possible to have this be a generic class (like DynamicStructValue<T>) instead of having to cast, and the value field can be of type T?

I come from rust and was trying to use an enum lol

There's also the problem that NT fields can only be of a certain few types. Giving T the possibility to be any type, at least in my opinion, is probably not the best, unless it can be constrained in some way I don't know about

onlycs · 2025-03-21T18:37:49Z

Thanks for reviewing it and sorry for the late reply. Comps keep me busy!

DynStructSchema is a data class that contains the schema of the struct (i.e. fields and field names) NT4StructMeta contains a NT path to the root value (containing the entire struct and all of its data), a path to the subvalue (e.g. pose.rotation.value to get a Pose2d's angle in radians), and the schema of the struct it's using to parse. Basically, schema is for the struct as a whole and meta is for the value field in the struct which is used by elastic

I'm 80% sure the code as it stands will bail out by showing raw bytes if the schemas don't line up for some reason.

onlycs · 2025-04-04T14:25:50Z

I went ahead and removed ntStructMeta from the multitopic widgets

…872-main

Gold872 · 2025-04-21T19:00:27Z

I just rebased this branch since the merges were getting very messy. I think I preserved the proper changes but I'm not entirely sure. Let me know when you would like me to re-review this, since I'd like to add this into the 2026 beta.

onlycs · 2025-04-25T12:45:29Z

i was in the middle of adding typechecking to nt types so I committed those changes too. im going to mark this as a draft until i finish fixing tests

onlycs · 2025-04-25T12:52:15Z

Also there's a problem where widgets in list layouts are too small. Dunno why that's happening; afaik I didn't change anything relating to that

onlycs added 20 commits February 19, 2025 23:45

fetch struct schemas!!

2d388ca

fetch struct schemas!!

ad8d018

store schemas somewhere useful

5278b6b

store schemas somewhere useful

a69337f

dyn struct parsing

820166f

dyn struct parsing

4d14e77

put entries to add widgets widget

b91e164

put entries to add widgets widget

38f5436

don't list out arrays

76fac0a

don't list out arrays

eaa6091

show correct types

a8b18fd

show correct types

998dbfc

good values on widget, broke json serde

220994e

good values on widget, broke json serde

95f522e

fix serde on non-struct values

b5e2413

fix serde on non-struct values

759c035

serde for struct values

2dea5b0

serde for struct values

ce0f4a6

use actual type when choosing display

4892f29

use actual type when choosing display

4a6848e

github-actions bot added the GUI Changes to Elastic's UI label Feb 21, 2025

Merge branch 'main' into main

a3f2906

github-actions bot added the workflows Changes to GitHub Workflows label Feb 21, 2025

onlycs force-pushed the main branch from cd1fc4c to a3f2906 Compare February 25, 2025 12:24

github-actions bot removed the workflows Changes to GitHub Workflows label Feb 25, 2025

onlycs added 5 commits February 28, 2025 08:35

Merge branch 'main' into main

28a510f

give treetopicentry a full nt4structmeta field

ef2d03a

give treetopicentry a full nt4structmeta field

624cf1a

fix tests

b9fe1a5

fix tests

85bffed

onlycs added 2 commits March 12, 2025 08:44

don't panic on schema parse failure

ffe41c8

don't panic on schema parse failure

6309a51

Gold872 requested changes Mar 12, 2025

View reviewed changes

onlycs added 2 commits April 4, 2025 10:24

remove ntstructmeta from multitopic widgets

d49230f

remove ntstructmeta from multitopic widgets

f8f5d26

onlycs added 5 commits April 4, 2025 10:28

rename Dyn to Dynamic, rename multi-topic to multi_topic for consistency

1361add

rename Dyn to Dynamic, rename multi-topic to multi_topic for consistency

cf0bc56

fix imports

203cc90

merge

14af00f

Merge branch 'Gold872-main'

c095859

github-actions bot added dependencies Changes to Elastic's dependencies workflows Changes to GitHub Workflows labels Apr 4, 2025

onlycs added 2 commits April 4, 2025 11:47

Merge branch 'main' of github.com:Gold872/elastic-dashboard into Gold…

a7a886b

…872-main

Merge branch 'Gold872-main'

8c91bc9

github-actions bot removed dependencies Changes to Elastic's dependencies workflows Changes to GitHub Workflows labels Apr 4, 2025

Gold872 force-pushed the main branch from 8c91bc9 to cf0bc56 Compare April 21, 2025 18:50

onlycs added 2 commits April 25, 2025 07:42

ntstruct refactor

d252acb

add type checking to nt4 types and break every unit test

1ad9391

onlycs marked this pull request as draft April 25, 2025 12:47

onlycs added 3 commits April 25, 2025 09:24

make unit tests compile

019d08d

run the import sorter

0d51dca

don't ci tests for forks

872051c

github-actions bot added the workflows Changes to GitHub Workflows label Apr 25, 2025

more dont ci tests for forks

738fe5d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamically deserialize networktables structs #225

Dynamically deserialize networktables structs #225

onlycs commented Feb 21, 2025 •

edited

Loading

Gold872 left a comment •

edited

Loading

Gold872 Mar 12, 2025

onlycs Mar 21, 2025

Gold872 Mar 12, 2025

onlycs Mar 21, 2025

onlycs Mar 21, 2025

Gold872 Mar 12, 2025

onlycs Apr 4, 2025

Gold872 Mar 12, 2025

Gold872 Mar 12, 2025

Gold872 Mar 12, 2025 •

edited

Loading

onlycs Mar 21, 2025

onlycs commented Mar 21, 2025

onlycs commented Apr 4, 2025

Gold872 commented Apr 21, 2025

onlycs commented Apr 25, 2025

onlycs commented Apr 25, 2025


		import 'package:dot_cast/dot_cast.dart';

		class DynStructField {

Dynamically deserialize networktables structs #225

Are you sure you want to change the base?

Dynamically deserialize networktables structs #225

Conversation

onlycs commented Feb 21, 2025 • edited Loading

Gold872 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Gold872 Mar 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

onlycs commented Mar 21, 2025

onlycs commented Apr 4, 2025

Gold872 commented Apr 21, 2025

onlycs commented Apr 25, 2025

onlycs commented Apr 25, 2025

onlycs commented Feb 21, 2025 •

edited

Loading

Gold872 left a comment •

edited

Loading

Gold872 Mar 12, 2025 •

edited

Loading