Issue 1217 - csv upload validation #1403

SageMar · 2024-12-05T22:39:31Z

Description

"The create meter page stops admins from entering bad values by constraining choices and/or checking before saving. It would be good if the CSV meter upload page did similar checks." After a talk with Steve, it was decided that this validation should be done in JavaScript on the back-end, so these validations are inside the uploadMeters.js file.

Done in collaboration with @cmatthews444

Partly Addresses #1217

This adds validations for the following variables:

Enabled
Disabled
Type
Timezone
Area
Cumulative
Reset
Timesort
End Only
Area Unit
Min Val
Max Val
Disabled Checks

Type of change

(Check the ones that apply by placing an "x" instead of the space in the [ ] so it becomes [x])

Note merging this changes the database configuration.
This change requires a documentation update

Checklist

(Note what you have done by placing an "x" instead of the space in the [ ] so it becomes [x]. It is hoped you do all of them.)

I have followed the OED pull request ideas
I have removed text in ( ) from the issue request
You acknowledge that every person contributing to this work has signed the OED Contributing License Agreement and each author is listed in the Description section.

Limitations

This does not cover every single validation done on a CSV file - it only covers those listed above.

…ion for checking each value towards the bottom of the uploadMeters.js file as well as calling them when the submission is sent to the server

… made sure the correct data was being sent back to the client side

…ifferent caps/lowercase

Boolean value validations

…idation

Min max validation

src/server/services/csvPipeline/uploadMeters.js

…gging still.

…idation

Merge branch 'Issue1217-csv-validation' of https://github.com/SageMar/OED into Issue1217-csv-validation

huss · 2025-01-20T15:09:20Z

@SageMar & @cmatthews444 I saw you made some changes since my comments and wanted to see where this PR is at. Are you planning to do more work on this, do you need any help or is that status something else? Thanks.

SageMar · 2025-01-20T17:52:39Z

@huss We are definitely still here finishing it up. It's been difficult to find the time between other projects this quarter, but we're aiming to fix the last errors and at least get in the validations we listed in the opening post for this PR.

When we last left off with the code, we had 3 errors still occurring which we were working to figure out what they mean. It would actually help a ton if you're able to give clarification on them as they're not quite as straightforward as the others we got. If it's easier to chat about this over Discord we can schedule a time on Friday (if you're available, I believe we can both make ourselves available any time on Friday other than 1pm PST), but if they're not too complicated to sum up here that would be great.

Meter testing files starting 'pipe100' doing 'Second meter upload where incorrectly provides meter identifier so fails' with 2
Meter testing files starting 'pipe101' doing 'Second meter with same name so fails but first meter exists' with 1 requests
Meter testing files starting 'pipe103' doing 'Uploading meters using Email. First succeeds then the second meter upload fails because it incorrectly provides meter identifier' with 2 requests

huss · 2025-01-20T22:43:15Z

@SageMar

Meter testing files starting 'pipe100' doing 'Second meter upload where incorrectly provides meter identifier so fails' with 2

Meter testing files starting 'pipe103' doing 'Uploading meters using Email. First succeeds then the second meter upload fails because it incorrectly provides meter identifier' with 2 requests

Both of these failed because it expected OED to report a success code (200) but it received an error code (400). I would debug this by adding a console.log to the code to print out the response text since that might indicate what went wrong.

Meter testing files starting 'pipe101' doing 'Second meter with same name so fails but first meter exists' with 1 requests

This one is curious because the returned error is Failed to upload meters due to internal OED Error: For meter pipe101 the area entry of 33 is invalid. Area must be a number greater than 0. but the value of 33 is greater than 0. I'm guessing you added a test that the area is valid. It may not be working as desired and/or it is returning new info so the test needs to be updated. I would first want to figure out why the test seems to have a message that is inconsistent on 33 > 0.

Do you want to work on these yourselves? I can also help (either working with you or by myself). Let me know what you prefer and if you want/need more information.

SageMar · 2025-01-24T00:37:07Z

Do you want to work on these yourselves? I can also help (either working with you or by myself). Let me know what you prefer and if you want/need more information.

@huss

I think we'll at least give it a try ourselves with that information. If we're still stuck by Tuesday, I'll reach out again! Thank you!

…ecked, now it converts strings to ints and functions

SageMar · 2025-01-24T21:16:20Z

@huss In case you were curious about what caused the issues, it seems the confusing error on pipe101 was what effected all of them. For come reason those files in particular had the number under area coming in as undefined, so using parseInt() on the value ended up fixing all three.

It passed tests and appears to be working for everything on our end, but let us know if there's anything else that needs to be changed! Thanks for always being so helpful.

huss

Thanks to @SageMar & @cmatthews444 for continued work on this issue. I reviewed/tested and made some comments. Please let me know if you need any help with this.

huss · 2025-01-30T22:03:02Z

src/server/services/csvPipeline/uploadMeters.js

+			// Verify area unit - have to parse float to ensure it can read in integer numbers like "33"
+			const areaInput = parseFloat(meter[9]);
+			if (areaInput) {
+				if (!isValidArea(areaInput)){


This is a small point but here and a few other lines don't have spaces: ){ vs ) }. Formatting with VSC should clear this up. I can do it if you prefer.

huss · 2025-01-30T22:04:52Z

src/server/services/csvPipeline/uploadMeters.js

@@ -8,17 +8,18 @@ const Meter = require('../../models/Meter');
 const readCsv = require('../pipeline-in-progress/readCsv');
 const Unit = require('../../models/Unit');
 const { normalizeBoolean } = require('./validateCsvUploadParams');
+const { type } = require('os');


I think this include may have been accidentally added as I don't think it is used.

huss · 2025-01-30T22:07:44Z

src/server/services/csvPipeline/uploadMeters.js

+ * @returns true or false
+ */
+function isValidAreaUnit(areaUnit) {
+	const validTypes = ['feet', 'meters', 'none'];


I think using the standard enum would be better. See src/server/models/Unit.js for areaUnitType. This will allow it to adjust if the allowed values are changed. Also, the error msg above with these values should be constructed by looping over the enum. Also for potential future changes the comment(s) listing the values could be updated about the enum rather than specifics.

huss · 2025-01-30T22:10:57Z

src/server/services/csvPipeline/uploadMeters.js

+ */
+function isValidTimeSort(timeSortValue) {
+	// must be one of the three values
+	if (timeSortValue == 'increasing' || timeSortValue == 'decreasing'){


Similar to above about enum, see src/server/services/csvPipeline/validateCsvUploadParams.js for MeterTimeSortTypesJS.

huss · 2025-01-30T22:11:51Z

src/server/services/csvPipeline/uploadMeters.js

+ * @returns true or false
+ */
+function isValidMeterType(meterTypeString) {
+	const validTypes = ['egauge', 'mamac', 'metasys', 'obvius', 'other'];


Same enum and see src/server/models/Meter.js for `Meter.type``.

huss · 2025-01-30T22:16:32Z

src/server/services/csvPipeline/uploadMeters.js

+ * @param {number} rowIndex - The current row index for error reporting.
+ */
+function validateBooleanFields(meter, rowIndex) {
+// all inputs that involve a true or false all bieng validated together.


bieng ->> being

huss · 2025-01-30T22:20:26Z

src/server/services/csvPipeline/uploadMeters.js

+
+	if (isNaN(minValue)){
+		// do nothing, pass it through
+	} else if (isNaN(minValue) || minValue < -9007199254740991 || minValue >= maxValue){


I think OED is allowing minValue = maxValue. This applies below.

This did raise the question in my mind that the maxValue is used before it is validated. Should they be checked for isNan, isNumber first, then min/max value and finally if min > max?

huss · 2025-01-30T22:50:18Z

src/server/services/csvPipeline/uploadMeters.js

+			const timezone = meter[5];
+			if (timezone){
+				if (!isValidTimeZone(timezone)){
+					let msg = `For meter ${meter[0]}, ${timeSortValue} is not a valid time zone.`;


I think timeSortValue should be for time zone instead.

huss · 2025-01-30T22:54:37Z

src/server/services/csvPipeline/uploadMeters.js

 					throw new CSVPipelineError(msg, undefined, 500);
 				}
 				// Need to reverse latitude & longitude because standard GPS gives in that order but a GPSPoint for the
 				// DB is longitude, latitude.
 				meter[6] = switchGPS(gpsInput);
 			}

+			// Verify area unit - have to parse float to ensure it can read in integer numbers like "33"
+			const areaInput = parseFloat(meter[9]);


Just noting you parseFloat before the check for being a number and if it is undefined.

huss · 2025-01-31T00:14:00Z

src/server/services/csvPipeline/uploadMeters.js

 					throw new CSVPipelineError(msg, undefined, 500);
 				}
 				// Need to reverse latitude & longitude because standard GPS gives in that order but a GPSPoint for the
 				// DB is longitude, latitude.
 				meter[6] = switchGPS(gpsInput);
 			}

+			// Verify area unit - have to parse float to ensure it can read in integer numbers like "33"
+			const areaInput = parseFloat(meter[9]);
+			if (areaInput) {


This check can fail in certain circumstances. I tested and 0 (zero) and -1 returns false so the area check does not happen. I also notice that 0.1 or any value with a decimal part does not work but it should. Finally, this allowed letters for numbers to pass the test. This may impact other tests you added. I did not try a fix so let me know if you have issues in resolving.

cmatthews444 · 2025-02-03T05:59:06Z

@huss We are planning on working on this later this week, will reach and let you know if we need help from there.

SageMar · 2025-02-12T19:45:12Z

@huss Do you have any time to meet with us on Friday in the Discord channel? Just looking to get a little clarification on some of the fixes. We're both available anytime after 11AM PST. Thanks!

huss · 2025-02-13T03:16:49Z

@huss Do you have any time to meet with us on Friday in the Discord channel? Just looking to get a little clarification on some of the fixes. We're both available anytime after 11AM PST. Thanks!

@SageMar Sure. Would 12:00 PT work for you?

SageMar · 2025-02-13T16:28:17Z

@SageMar Sure. Would 12:00 PT work for you?

Yup, that works great! See you then

SageMar and others added 13 commits November 27, 2024 15:51

added in the tests for area and meter type. These include a new funct…

187dde5

…ion for checking each value towards the bottom of the uploadMeters.js file as well as calling them when the submission is sent to the server

fixed issue within the added validation for area and meter type, also…

02712f0

… made sure the correct data was being sent back to the client side

Added validation for all boolean value types.

55db711

added in area unit type checking, still need to adjust to allow for d…

9722c00

…ifferent caps/lowercase

Removed old initial testing code.

67b8078

Merge pull request #1 from cmatthews444/Issue1217-csv-validation

62288b6

Boolean value validations

Merge branch 'OpenEnergyDashboard:development' into Issue1217-csv-val…

034e7b4

…idation

added in a time sort check and fixed some linter errors

4f12d03

Add validation for minmax values

f4b60f3

tweaked timedout

5f30318

Merge pull request #2 from cmatthews444/Issue1217-csv-validation

143a96d

Min max validation

Timezone validation completed

6c612b8

added time zone validation

30d213e

huss reviewed Dec 10, 2024

View reviewed changes

src/server/services/csvPipeline/uploadMeters.js Outdated Show resolved Hide resolved

SageMar and others added 4 commits January 3, 2025 11:41

clarified error messages and removed trailing spaces. Working on debu…

f6b4c1c

…gging still.

Merge branch 'OpenEnergyDashboard:development' into Issue1217-csv-val…

06751f8

…idation

Updating to match current OED vers

45ae817

Merge branch 'Issue1217-csv-validation' of https://github.com/SageMar/OED into Issue1217-csv-validation

allow empty min/max values and booleanfields

7c317f5

Fixed errors in tests! It was an issue with the way area was being ch…

4d85514

…ecked, now it converts strings to ints and functions

SageMar marked this pull request as ready for review January 24, 2025 21:11

huss requested changes Jan 31, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue 1217 - csv upload validation #1403

Issue 1217 - csv upload validation #1403

SageMar commented Dec 5, 2024 •

edited

Loading

huss commented Jan 20, 2025

SageMar commented Jan 20, 2025

huss commented Jan 20, 2025

SageMar commented Jan 24, 2025

SageMar commented Jan 24, 2025

huss left a comment

huss Jan 30, 2025

huss Jan 30, 2025

huss Jan 30, 2025

huss Jan 30, 2025

huss Jan 30, 2025

huss Jan 30, 2025

huss Jan 30, 2025

huss Jan 30, 2025

huss Jan 30, 2025

huss Jan 31, 2025

cmatthews444 commented Feb 3, 2025

SageMar commented Feb 12, 2025

huss commented Feb 13, 2025

SageMar commented Feb 13, 2025

Issue 1217 - csv upload validation #1403

Are you sure you want to change the base?

Issue 1217 - csv upload validation #1403

Conversation

SageMar commented Dec 5, 2024 • edited Loading

Description

Type of change

Checklist

Limitations

huss commented Jan 20, 2025

SageMar commented Jan 20, 2025

huss commented Jan 20, 2025

SageMar commented Jan 24, 2025

SageMar commented Jan 24, 2025

huss left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmatthews444 commented Feb 3, 2025

SageMar commented Feb 12, 2025

huss commented Feb 13, 2025

SageMar commented Feb 13, 2025

SageMar commented Dec 5, 2024 •

edited

Loading