Retrain error after adding new class

[vgeorge]

The websocket is returning the following error message when I try to retrain with a new class added to the existing ones from the base model:

{
  "message": "error",
  "data": {
    "error": "Retrain Error",
    "detailed": "Provided ``coef_`` does not match dataset. "
  }
}

This happens on a project (id 20 on staging) with five existing checkpoints created using the base model classes. The retrain starts well and goes up a 100% progress, and the error message is returned.

The retrain message payload is:

{
  "message": "error",
  "data": {
    "error": "Retrain Error",
    "detailed": "Provided ``coef_`` does not match dataset. "
  }
}


{
  "action": "model#retrain",
  "data": {
    "name": "Amanalco de Becerra",
    "classes": [
      {
        "name": "forest",
        "color": "#6CA966",
        "geometry": { "type": "FeatureCollection", "features": [] }
      },
      {
        "name": "dry_jungle",
        "color": "#B5C590",
        "geometry": { "type": "FeatureCollection", "features": [] }
      },
      {
        "name": "humid_jungle",
        "color": "#00A884",
        "geometry": { "type": "FeatureCollection", "features": [] }
      },
      {
        "name": "pasture",
        "color": "#DBD746",
        "geometry": { "type": "FeatureCollection", "features": [] }
      },
      {
        "name": "agriculture",
        "color": "#A96E2C",
        "geometry": {
          "type": "FeatureCollection",
          "features": [
            {
              "type": "Feature",
              "properties": { "class": "agriculture" },
              "geometry": {
                "type": "Polygon",
                "coordinates": [
                  [
                    [-100.040045, 19.249976],
                    [-100.04056, 19.250057],
                    [-100.040903, 19.2503],
                    [-100.040989, 19.250867],
                    [-100.040989, 19.251758],
                    [-100.040474, 19.253946],
                    [-100.040131, 19.254838],
                    [-100.040045, 19.255],
                    [-100.039959, 19.255081],
                    [-100.035152, 19.254757],
                    [-100.035067, 19.254757],
                    [-100.035067, 19.253784],
                    [-100.035238, 19.253136],
                    [-100.03541, 19.252812],
                    [-100.035667, 19.252488],
                    [-100.036354, 19.251839],
                    [-100.037384, 19.25111],
                    [-100.039616, 19.250057],
                    [-100.040045, 19.249976]
                  ]
                ]
              }
            },
            {
              "type": "Feature",
              "properties": { "class": "agriculture" },
              "geometry": {
                "type": "Polygon",
                "coordinates": [
                  [
                    [-100.030689, 19.237658],
                    [-100.034122, 19.237983],
                    [-100.036697, 19.238793],
                    [-100.037298, 19.239522],
                    [-100.037384, 19.240171],
                    [-100.037212, 19.241953],
                    [-100.037041, 19.242845],
                    [-100.03644, 19.24479],
                    [-100.025797, 19.239603],
                    [-100.025797, 19.239279],
                    [-100.02614, 19.238874],
                    [-100.026741, 19.23855],
                    [-100.0282, 19.238064],
                    [-100.030689, 19.237658]
                  ]
                ]
              }
            },
            {
              "type": "Feature",
              "properties": { "class": "agriculture" },
              "geometry": {
                "type": "Polygon",
                "coordinates": [
                  [
                    [-100.026398, 19.232229],
                    [-100.026741, 19.232472],
                    [-100.026999, 19.232877],
                    [-100.027084, 19.23312],
                    [-100.026999, 19.233363],
                    [-100.023651, 19.233931],
                    [-100.023909, 19.233282],
                    [-100.024166, 19.233039],
                    [-100.025625, 19.23231],
                    [-100.025969, 19.232229],
                    [-100.026398, 19.232229]
                  ]
                ]
              }
            }
          ]
        }
      },
      {
        "name": "urban",
        "color": "#F10100",
        "geometry": { "type": "FeatureCollection", "features": [] }
      },
      {
        "name": "without_apparent_vegetation",
        "color": "#D0F3AB",
        "geometry": { "type": "FeatureCollection", "features": [] }
      },
      {
        "name": "water",
        "color": "#486DA2",
        "geometry": { "type": "FeatureCollection", "features": [] }
      },
      {
        "name": "scrub",
        "color": "#ABC964",
        "geometry": { "type": "FeatureCollection", "features": [] }
      },
      {
        "name": "bare_soil",
        "color": "#B3AC9F",
        "geometry": { "type": "FeatureCollection", "features": [] }
      },
      {
        "name": "Clay",
        "color": "#653939",
        "geometry": {
          "type": "FeatureCollection",
          "features": [
            {
              "type": "Feature",
              "properties": { "class": "Clay" },
              "geometry": {
                "type": "Polygon",
                "coordinates": [
                  [
                    [-100.023994, 19.24714],
                    [-100.024166, 19.247302],
                    [-100.024338, 19.247626],
                    [-100.024338, 19.248274],
                    [-100.023994, 19.249246],
                    [-100.023651, 19.249733],
                    [-100.022793, 19.250543],
                    [-100.017815, 19.250381],
                    [-100.019016, 19.249084],
                    [-100.019703, 19.248598],
                    [-100.020218, 19.248274],
                    [-100.022106, 19.247464],
                    [-100.023308, 19.24714],
                    [-100.023994, 19.24714]
                  ]
                ]
              }
            },
            {
              "type": "Feature",
              "properties": { "class": "Clay" },
              "geometry": {
                "type": "Polygon",
                "coordinates": [
                  [
                    [-100.014381, 19.236443],
                    [-100.016785, 19.236929],
                    [-100.017815, 19.237415],
                    [-100.017815, 19.239036],
                    [-100.017643, 19.239684],
                    [-100.011806, 19.239036],
                    [-100.011978, 19.238712],
                    [-100.012665, 19.237739],
                    [-100.01318, 19.237091],
                    [-100.013523, 19.236767],
                    [-100.013866, 19.236605],
                    [-100.014381, 19.236443]
                  ]
                ]
              }
            },
            {
              "type": "Feature",
              "properties": { "class": "Clay" },
              "geometry": {
                "type": "Polygon",
                "coordinates": [
                  [
                    [-100.009403, 19.247464],
                    [-100.009918, 19.247626],
                    [-100.010262, 19.24795],
                    [-100.010605, 19.248436],
                    [-100.010948, 19.249084],
                    [-100.01112, 19.250057],
                    [-100.011292, 19.251191],
                    [-100.005112, 19.258484],
                    [-100.00494, 19.257674],
                    [-100.00494, 19.254919],
                    [-100.005112, 19.252488],
                    [-100.005283, 19.251515],
                    [-100.006485, 19.249246],
                    [-100.008202, 19.247626],
                    [-100.008717, 19.247464],
                    [-100.009403, 19.247464]
                  ]
                ]
              }
            }
          ]
        }
      }
    ]
  }
}

cc @ingalls @geohacker

[vgeorge]

@ingalls it looks like your recent changes corrected the problem, I was able to retrain after changing checkpoints 👍

[vgeorge]

Whoops, wrong ticket, I meant to close #84

[LanesGood]

@srmsoumya @ingalls are you able to take a look at this ticket?

[srmsoumya]

I am looking into this

[vgeorge]

@srmsoumya I did another test today and it failed at the second retrain attemp on a new project. The steps were:

• Created new project with small AOI
• Ran prediction
• Added one new class, one sample, retrain, success
• Added another class, one sample to each new class, failed

The instance log: project-83-logs.txt.zip

[LanesGood]

In Slack, @srmsoumya states:

I've successfully reproduced the retraining error locally. The issue arises when the SGD classifier, which is layered on top of the Base Neural Network, is updated to include new classes. Since the SGD classifier doesn't natively support updating its weights to accommodate new classes, we're required to retrain the SGD classifier from scratch each time we need to include additional classes.
I have a couple of questions for @ingalls and @geohacker :

• In the code at this location, does the variable classes incorporate data points from previous user iterations, or does it only include newly added points?
• Have we conducted any tests with this setup for NAIP on landcover.io? I'm encountering an error while attempting to introduce a second class there as well.

and

As a fix, we can create a new classifier everytime the user adds or modifies a class but I am not sure how we are storing the data points coming from the user.

@srmsoumya would the suggested fix permit the addition of new classes?

[srmsoumya]

@LanesGood If we store user annotations in a way that keeps track of all previous modifications and additions to the
base classes, then retraining the SGD classifier from scratch every time, should resolve the issue of adding new classes.