Improving column name validation

CHANGELOG.md

@@ -17,6 +17,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Adding JavaDoc to `EntitySource.safeMapGet` [#828](https://github.com/ie3-institute/PowerSystemDataModel/issues/828)
 - Abstracting some methods in `ValidationUtils` [#852](https://github.com/ie3-institute/PowerSystemDataModel/issues/852)
 - `EmInput` should not be connected to the grid [#955](https://github.com/ie3-institute/PowerSystemDataModel/issues/955)
+- Enhancing the error message for coordinate sources with invalid column names [#670](https://github.com/ie3-institute/PowerSystemDataModel/issues/670) 
+- Allowing for additional unused columns in sources [#839](https://github.com/ie3-institute/PowerSystemDataModel/issues/839)
+- Improving column name validation to only run once per source [#849](https://github.com/ie3-institute/PowerSystemDataModel/issues/849)
 
 ## [4.1.0] - 2023-11-02
 

docs/readthedocs/models/input/participant/bm.rst

@@ -26,7 +26,7 @@ Type Model
 +---------------------+---------+------------------------------------------------------------+
 | sRated              | kVA     | Rated apparent power                                       |
 +---------------------+---------+------------------------------------------------------------+
-| cosphiRated         | --      | Rated power factor                                         |
+| cosPhiRated         | --      | Rated power factor                                         |
 +---------------------+---------+------------------------------------------------------------+
 | etaConv             | %       | Efficiency of the assets inverter                          |
 +---------------------+---------+------------------------------------------------------------+

docs/readthedocs/models/input/participant/chp.rst

@@ -28,7 +28,7 @@ Type Model
 +-------------+---------+---------------------------------------------------------+
 | sRated      | kVA     | Rated apparent power                                    |
 +-------------+---------+---------------------------------------------------------+
-| cosphiRated | --      | Rated power factor                                      |
+| cosPhiRated | --      | Rated power factor                                      |
 +-------------+---------+---------------------------------------------------------+
 | pThermal    | kW      | Rated thermal power (at rated electrical power)         |
 +-------------+---------+---------------------------------------------------------+

docs/readthedocs/models/input/participant/ev.rst

@@ -27,7 +27,7 @@ Type Model
 +-------------+----------+---------------------------------------------------------+
 | sRated      | kVA      | Rated apparent power                                    |
 +-------------+----------+---------------------------------------------------------+
-| cosphiRated | --       | Rated power factor                                      |
+| cosPhiRated | --       | Rated power factor                                      |
 +-------------+----------+---------------------------------------------------------+
 
 Entity Model

docs/readthedocs/models/input/participant/fixedfeedin.rst

@@ -25,7 +25,7 @@ Attributes, Units and Remarks
 +------------------+---------+--------------------------------------------------------------------------------------+
 | sRated           | kVA     | Rated apparent power                                                                 |
 +------------------+---------+--------------------------------------------------------------------------------------+
-| cosphiRated      | --      | Rated power factor                                                                   |
+| cosPhiRated      | --      | Rated power factor                                                                   |
 +------------------+---------+--------------------------------------------------------------------------------------+
 
 Caveats

docs/readthedocs/models/input/participant/hp.rst

@@ -24,7 +24,7 @@ Type Model
 +-------------+---------+---------------------------------------------------------+
 | sRated      | kVA     | Rated apparent power                                    |
 +-------------+---------+---------------------------------------------------------+
-| cosphiRated | --      | Rated power factor                                      |
+| cosPhiRated | --      | Rated power factor                                      |
 +-------------+---------+---------------------------------------------------------+
 | pThermal    | kW      | Rated thermal power (at rated electrical power)         |
 +-------------+---------+---------------------------------------------------------+

docs/readthedocs/models/input/participant/load.rst

@@ -30,7 +30,7 @@ Attributes, Units and Remarks
 +---------------------+---------+--------------------------------------------------------------------------------------+
 | sRated              | kVA     | Rated apparent power                                                                 |
 +---------------------+---------+--------------------------------------------------------------------------------------+
-| cosphiRated         | --      | Rated power factor                                                                   |
+| cosPhiRated         | --      | Rated power factor                                                                   |
 +---------------------+---------+--------------------------------------------------------------------------------------+
 
 Caveats

docs/readthedocs/models/input/participant/pv.rst

@@ -37,7 +37,7 @@ Detailed model of a photovoltaic power plant.
 +------------------+---------+--------------------------------------------------------------------------------------+
 | sRated           | kVA     | Rated apparent power                                                                 |
 +------------------+---------+--------------------------------------------------------------------------------------+
-| cosphiRated      | --      | Rated power factor                                                                   |
+| cosPhiRated      | --      | Rated power factor                                                                   |
 +------------------+---------+--------------------------------------------------------------------------------------+
 
 Caveats

docs/readthedocs/models/input/participant/storage.rst

@@ -26,7 +26,7 @@ Type Model
 +---------------------+---------+---------------------------------------------------------+
 | sRated              | kVA     | Rated apparent power                                    |
 +---------------------+---------+---------------------------------------------------------+
-| cosphiRated         | --      | Rated power factor                                      |
+| cosPhiRated         | --      | Rated power factor                                      |
 +---------------------+---------+---------------------------------------------------------+
 | pMax                | kW      | | Maximum permissible active power                      |
 |                     |         | | infeed or consumption                                 |

docs/readthedocs/models/input/participant/wec.rst

@@ -24,7 +24,7 @@ Type Model
 +------------------+---------+----------------------------------------------------------------------+
 | sRated           | kVA     | Rated apparent power                                                 |
 +------------------+---------+----------------------------------------------------------------------+
-| cosphiRated      | --      | Rated power factor                                                   |
+| cosPhiRated      | --      | Rated power factor                                                   |
 +------------------+---------+----------------------------------------------------------------------+
 | cpCharacteristic | --      | Wind velocity dependent :ref:`Betz factors<wec_cp_characteristic>`.  |
 +------------------+---------+----------------------------------------------------------------------+

docs/uml/main/input/SystemDatamodelConcept.puml

@@ -119,7 +119,7 @@ package models {
                     - capex: ComparableQuantity<Currency> [€]
                     - opex: ComparableQuantity<EnergyPrice> [€/MWh]
                     - sRated: ComparableQuantity<Power> [kVA]
-                    - cosphiRated: Double
+                    - cosPhiRated: Double
                 }
                 SystemParticipantTypeInput --|> AssetTypeInput
 
@@ -206,7 +206,7 @@ package models {
 
             class FixedFeedInInput {
                 - sRated: ComparableQuantity<Power> [kVA]
-                - cosphiRated: double
+                - cosPhiRated: double
             }
             FixedFeedInInput --|> SystemParticipantInput
 

src/main/java/edu/ie3/datamodel/exceptions/FactoryException.java

@@ -5,7 +5,7 @@
 */
 package edu.ie3.datamodel.exceptions;
 
-/** Is thrown, when an something went wrong during entity creation process in a EntityFactory */
+/** Is thrown, when something went wrong during entity creation process in a EntityFactory */
 public class FactoryException extends RuntimeException {
   public FactoryException(final String message, final Throwable cause) {
     super(message, cause);

src/main/java/edu/ie3/datamodel/exceptions/FailedValidationException.java

@@ -13,6 +13,10 @@ public FailedValidationException(String message, Throwable throwable) {
     super(message, throwable);
   }
 
+  public FailedValidationException(Throwable throwable) {
+    super(throwable);
+  }
+
   public FailedValidationException(String message) {
     super(message);
   }

src/main/java/edu/ie3/datamodel/exceptions/ValidationException.java

@@ -10,6 +10,10 @@ protected ValidationException(String s) {
     super(s);
   }
 
+  protected ValidationException(Throwable throwable) {
+    super(throwable);
+  }
+
   protected ValidationException(String s, Throwable throwable) {
     super(s, throwable);
   }

src/main/java/edu/ie3/datamodel/io/connectors/CouchbaseConnector.java

@@ -10,11 +10,15 @@
 import com.couchbase.client.java.Cluster;
 import com.couchbase.client.java.ClusterOptions;
 import com.couchbase.client.java.Collection;
+import com.couchbase.client.java.json.JsonObject;
 import com.couchbase.client.java.kv.GetResult;
 import com.couchbase.client.java.kv.MutationResult;
 import com.couchbase.client.java.query.QueryResult;
 import java.time.Duration;
+import java.util.HashSet;
 import java.util.List;
+import java.util.Optional;
+import java.util.Set;
 import java.util.concurrent.CompletableFuture;
 
 /**
@@ -56,6 +60,28 @@ public CouchbaseConnector(
     cluster = Cluster.connect(url, clusterOptions);
   }
 
+  /** Returns the option for a set of found fields. */
+  @SuppressWarnings("unchecked")
+  public Optional<Set<String>> getSourceFields() {
+    String query =
+        "SELECT ARRAY_DISTINCT(ARRAY_AGG(v)) AS column FROM "
+            + bucketName
+            + " b UNNEST OBJECT_NAMES(b) AS v";
+    cluster.bucket(bucketName).waitUntilReady(Duration.ofSeconds(30));
+
+    QueryResult queryResult = query(query).join();
+    JsonObject jsonObject = queryResult.rowsAsObject().get(0);
+    Object columns = jsonObject.toMap().get("column");
+
+    Set<String> set = new HashSet<>();
+
+    if (columns != null) {
+      set.addAll((List<String>) columns);
+    }
+
+    return Optional.of(set);
+  }
+
   /**
    * Return the couchbase java sdk equivalent of a session - a collection - to the previously set
    * bucket

src/main/java/edu/ie3/datamodel/io/connectors/CsvFileConnector.java

@@ -168,28 +168,22 @@ public synchronized <C extends UniqueEntity> void closeEntityWriter(Class<C> clz
    */
   public BufferedReader initReader(Class<? extends UniqueEntity> clz)
       throws FileNotFoundException, ConnectorException {
-    try {
-      Path filePath =
-          fileNamingStrategy
-              .getFilePath(clz)
-              .orElseThrow(
-                  () ->
-                      new ConnectorException(
-                          "Cannot find a naming strategy for class '"
-                              + clz.getSimpleName()
-                              + "'."));
-      return initReader(filePath);
-    } catch (ConnectorException e) {
-      throw new ConnectorException(
-          "Cannot initialize reader for entity '" + clz.getSimpleName() + "'.", e);
-    }
+    Path filePath =
+        fileNamingStrategy
+            .getFilePath(clz)
+            .orElseThrow(
+                () ->
+                    new ConnectorException(
+                        "Cannot find a naming strategy for class '" + clz.getSimpleName() + "'."));
+    return initReader(filePath);
   }
 
   /**
    * Initializes a file reader for the given file name. Use {@link
    * CsvFileConnector#initReader(Class)} for files that actually correspond to concrete entities.
    *
-   * @param filePath sub directory tree starting from base folder, including file name
+   * @param filePath path of file starting from base folder, including file name but not file
+   *     extension
    * @return the reader that contains information about the file to be read in
    * @throws FileNotFoundException if no file with the provided file name can be found
    */

src/main/java/edu/ie3/datamodel/io/connectors/InfluxDbConnector.java

@@ -38,6 +38,7 @@ public class InfluxDbConnector implements DataConnector {
         return maps;
       };
 
+  private final String databaseName;
   private final String scenarioName;
   private final InfluxDB session;
 
@@ -77,6 +78,7 @@ public InfluxDbConnector(
    */
   public InfluxDbConnector(
       InfluxDB session, String scenarioName, String databaseName, boolean createDb) {
+    this.databaseName = databaseName;
     this.scenarioName = scenarioName;
     this.session = session;
 
@@ -107,6 +109,21 @@ public InfluxDbConnector(String url, String databaseName) {
     this(url, databaseName, NO_SCENARIO, true, InfluxDB.LogLevel.NONE, BatchOptions.DEFAULTS);
   }
 
+  /** Returns the option for fields found in the source. */
+  public Optional<Set<String>> getSourceFields() {
+    QueryResult tagKeys = session.query(new Query("SHOW TAG KEYS ON " + databaseName));
+    Map<String, Set<Map<String, String>>> tagResults = parseQueryResult(tagKeys);
+
+    QueryResult fieldKeys = session.query(new Query("SHOW FIELD KEYS ON " + databaseName));
+    Map<String, Set<Map<String, String>>> fieldResults = parseQueryResult(fieldKeys);
+
+    Set<String> set = new HashSet<>();
+    tagResults.values().forEach(v -> v.stream().map(m -> m.get("tagKey")).forEach(set::add));
+    fieldResults.values().forEach(v -> v.stream().map(m -> m.get("fieldKey")).forEach(set::add));
+
+    return Optional.of(set);
+  }
+
   /**
    * Create the database of this connector if it doesn't exist yet
    *