Pipeline issue after deleting record from source table

[Soumyabrata-Talapatra]

I am using the builtin:postgres connector to replicate data from the source table to the destination table having one integer as primary key in source and destination table. However, if i delete any record from the source table then one error i am getting:

2024-12-24T10:13:01+00:00 ERR node stopped error="node postgres_pipeline_41:postgres_source_41 stopped with error: source stream was stopped unexpectedly: error reading from source: read plugin error: failed to encode before payload: failed to marshal data with schema: could not marshal into avro: address: avro: *avro.null is unsupported for Avro string"

my pipeline yaml is:

version: 2.2

pipelines:
  - id: postgres_pipeline_41
    status: running
    name: postgres-pipeline_41
    description: Postgres pipeline configuration with column mapping
    connectors:
      - id: postgres_source_41
        type: source
        plugin: builtin:postgres
        name: my-postgres-source_41
        settings:
          url: "postgres://postgres:******@192.168.1.16:****/conduit_test_db?sslmode=disable"
          tables: "origin_table"
          cdcMode: "logrepl"
          logrepl.publicationName: "conduit_publication_41" # Choose a unique name
          logrepl.slotName: "conduit_slot_41" # Choose a unique name
               - id: postgres_dest_41
        type: destination
        plugin: builtin:postgres
        name: my-postgres-destination_41
        settings:
          url: "postgres://postgres:******@192.168.1.16:****/conduit_test_db?sslmode=disable"
          table: "destination_table"
    processors:
      - id: filter_id_ename_proc
        plugin: "postgres-processor_address_filter"
    dead-letter-queue:
        plugin: "builtin:file"    # use builtin file plugin as DLQ connector
        settings:
          path: "./dlq.out"       # route records to file ./dlq.out
        window-size: 10            # monitor last 10 records
        window-nack-threshold: 3  # tolerate up to 3 negatively acknowledged records

I have not used conduit much and my question may seem noobish. I would appreciate the help. Thanks in advance.

[hariso]

@Soumyabrata-Talapatra Thanks for reporting this! I guess you have a nullable varchar column?

[hariso]

@Soumyabrata-Talapatra The issue with nullable columns. I'm working on a fix for that in this PR. I need to handle a few edge cases, and then it should be ready to go.

However, there's another option and that is to disable schemas if you don't need those. If you don't need schemas, I'd like to ask you to try this code out: ConduitIO/conduit-connector-postgres#234.

You can try out the code by building the connector or by waiting for the PR to be merged and pulled into Conduit.

To build the connector and use it as a standalone connector, you should:

1. Clone the Postgres connector repository: gh repo clone ConduitIO/conduit-connector-postgres.
2. Run cd conduit-connector-postgres.
3. Check out the code from the branch: gh pr checkout 234.
4. Run make.
5. Copy the conduit-connector-postgres binary into Conduit's connectors directory (if it's not there, you need to create it).

If you need any help, feel free to let me know!

[Soumyabrata-Talapatra]

@hariso Thanks for your prompt reply. For empty table also I got this NULL error. So what is the timeline I should expect in which it would be resolved.

[hariso]

If you don't need the schemas, you can disable them as explained here.

If you do need the schemas, I have a partial fix (that covers non-integer types). You can try it out too if you build the Postgres connector itself.

I can't say for sure when these fixes will be merged, given that it's the holidays' and vacations' season, but it shouldn't take too much time.

For the NULL error, can you please open an issue in the Postgres connector repository? Thanks in advance!

[Soumyabrata-Talapatra]

@hariso Thanks for your prompt support. I am having tough time to make a custom connector in windows. With WSL I was able to build the binary file but may be it is not compatible with the windows system. Then tried making .exe file but that also throws error while running the pipeline. The error is(with .exe):
could not load standalone connector plugin error="failed to dispense connector specifier (tip: check if the file is a valid connector plugin binary and if you have permissions for running it): fork/exec D:\beyond\conduitio\conduit_0.12.3_Windows_x86_64\connectors/conduit-connector-postgres.exe: This version of %1 is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher." component=plugin.connector.standalone.Registry plugin_path="D:\beyond\conduitio\conduit_0.12.3_Windows_x86_64\connectors/conduit-connector-postgres"

The command I am using (alternative of make) is:
$env:GOARCH="amd64"
$env:GOOS="windows"
go build -o conduit-connector-postgres.exe

Thanks.

[hariso]

No problems, thank you for trying Conduit and the connector out!

I'm not really sure what the error is about. I've googled it, and this OP on SO has the same issue as you do: https://stackoverflow.com/questions/77214029/this-version-of-1-is-not-compatible-error. It appears to be an issue with the architecture and/or linking.

Also, try building the connector with this too: $env:CGO_ENABLED=0.