Failing to list/create/delete caches through the REST API on Coherence v14.1.1.0

[sparkcatalyst96]

Describe the bug
Attempting to create/list/delete caches through the REST API results in 404 error code.

To Reproduce
I've installed this in my K8s cluster using the Coherence Kubernetes Operator as follows:

helm repo add coherence https://oracle.github.io/coherence-operator/charts
helm repo update
helm install  \
    --namespace coherence \
    coherence-operator \
    coherence/coherence-operator

apiVersion: coherence.oracle.com/v1
kind: Coherence
metadata:
  name: storage-cluster
spec:
  replicas: 3
  role: storage
  
  image: ghcr.io/oracle/coherence-ce:14.1.1-0-20
  
  # Coherence settings
  coherence:
    cacheConfig: /coherence/config/cache-config.xml
    localPort: 7575
    localPortAdjust: 7576
    metrics:
      enabled: true
      port: 9612
      ssl:
        enabled: false
    management:
      enabled: true
      port: 30000
      ssl:
        enabled: false

  # Mount the cache config
  configMapVolumes:
    - mountPath: /coherence/config
      name: cache-config

  # JVM settings
  jvm:
      memory:
        heapSize: 4g
      args:
        - "-XX:+UseG1GC"
        - "-XX:+AlwaysPreTouch"
        - "-Dcoherence.rest.config=/coherence/config/coherence-rest-config.xml"
        - "-Dcoherence.management=all"

  # Ports configuration
  ports:
    - name: management
      port: 30000
    - name: metrics
      port: 9612
      serviceMonitor:
        enabled: true
    - name: extend
      port: 20000
    - name: rest
      port: 8080

  # Container resources
  resources:
    requests:
      memory: "6Gi"
      cpu: "2"
    limits:
      memory: "8Gi"
      cpu: "4"

  # Volume claim for persistence
  volumeClaimTemplates:
    - metadata:
        name: storage
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 50Gi
        storageClassName: "gp3"

  # Pod management
  readinessProbe:
    initialDelaySeconds: 30
    periodSeconds: 10
  
  livenessProbe:
    initialDelaySeconds: 30
    periodSeconds: 10

  # Service monitoring annotations
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9612"

apiVersion: v1
kind: ConfigMap
metadata:
  name: cache-config
  namespace: coherence
data:
  cache-config.xml: |
    <?xml version="1.0"?>
    <cache-config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                  xmlns="http://xmlns.oracle.com/coherence/coherence-cache-config"
                  xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-cache-config coherence-cache-config.xsd">
      <caching-scheme-mapping>
        <cache-mapping>
          <cache-name>*</cache-name>
          <scheme-name>distributed-scheme</scheme-name>
        </cache-mapping>
      </caching-scheme-mapping>

      <caching-schemes>
        <distributed-scheme>
          <scheme-name>distributed-scheme</scheme-name>
          <service-name>PartitionedCache</service-name>
          <backing-map-scheme>
            <local-scheme/>
          </backing-map-scheme>
          <autostart>true</autostart>
        </distributed-scheme>

        <proxy-scheme>
          <service-name>ExtendHttpProxyService</service-name>
          <acceptor-config>
            <http-acceptor>
              <local-address>
                <address>localhost</address>
                <port>8080</port>
              </local-address>
            </http-acceptor>
          </acceptor-config>
          <autostart>true</autostart>
        </proxy-scheme>
      </caching-schemes>
    </cache-config>

  coherence-rest-config.xml: |
    <?xml version="1.0"?>
    <rest-config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                 xmlns="http://xmlns.oracle.com/coherence/coherence-rest-config"
                 xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-rest-config coherence-rest-config.xsd">
      <resources>
          <resource>
            <cache-name>*</cache-name>
            <key-class>java.lang.String</key-class>
            <value-class>example.Person</value-class>
            <direct-query enabled="true"/>
          </resource>
      </resources>
    </rest-config>

I've followed the configuration instructions from the following documentation:
https://docs.oracle.com/en/middleware/standalone/coherence/14.1.1.0/develop-remote-clients/building-your-first-coherence-rest-application.html#GUID-45D33C82-D9DA-455D-8691-FA341D86CF81
https://docs.oracle.com/en/middleware/standalone/coherence/14.1.1.0/develop-remote-clients/performing-grid-operations-rest.html#GUID-5C807FF2-5F0F-4F70-8665-1741BF7A22B0

Expected behaviour
I expect to successfully retrieve, create and delete the caches in the cluster through the REST API. Instead, I get the following:

~ $ curl -v -X PUT http://storage-cluster-rest:8080/api/dist-http-example/1 \
> -H "Content-Type: application/json" \
> -d '{"value": "test1"}'
* Host storage-cluster-rest:8080 was resolved.
* IPv6: (none)
* IPv4: 172.20.51.129
*   Trying 172.20.51.129:8080...
* Connected to storage-cluster-rest (172.20.51.129) port 8080
* using HTTP/1.x
> PUT /api/dist-http-example/1 HTTP/1.1
> Host: storage-cluster-rest:8080
> User-Agent: curl/8.12.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 18
>
* upload completely sent off: 18 bytes
< HTTP/1.1 404 Not Found
< Date: Wed, 19 Feb 2025 09:15:56 GMT
< Content-length: 0
< X-content-type-options: nosniff
<
* Connection #0 to host storage-cluster-rest left intact
~ $ curl -v http://storage-cluster-rest:8080/api/dist-http-example/1.json
* Host storage-cluster-rest:8080 was resolved.
* IPv6: (none)
* IPv4: 172.20.51.129
*   Trying 172.20.51.129:8080...
* Connected to storage-cluster-rest (172.20.51.129) port 8080
* using HTTP/1.x
> GET /api/dist-http-example/1.json HTTP/1.1
> Host: storage-cluster-rest:8080
> User-Agent: curl/8.12.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 404 Not Found
< Date: Wed, 19 Feb 2025 09:16:11 GMT
< Content-length: 0
< X-content-type-options: nosniff
<
* Connection #0 to host storage-cluster-rest left intact
~ $ curl -v http://storage-cluster-rest:8080/api/caches
* Host storage-cluster-rest:8080 was resolved.
* IPv6: (none)
* IPv4: 172.20.51.129
*   Trying 172.20.51.129:8080...
* Connected to storage-cluster-rest (172.20.51.129) port 8080
* using HTTP/1.x
> GET /api/caches HTTP/1.1
> Host: storage-cluster-rest:8080
> User-Agent: curl/8.12.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 404 Not Found
< Date: Wed, 19 Feb 2025 09:16:15 GMT
< Content-length: 0
< X-content-type-options: nosniff
<
* Connection #0 to host storage-cluster-rest left intact

According to the logs, the acceptor is successfully listening for connections:

MacBook-Pro:~ georgecristiancimpoies$ kubectl logs storage-cluster-0 | grep -i acceptor
Defaulted container "coherence" out of: coherence, coherence-k8s-utils (init)
2025-02-19 09:09:55.849/5.541 Oracle Coherence CE 14.1.1.0.20 <Info> (thread=NameService:TcpAcceptor, member=n/a): TcpAcceptor now listening for connections on storage-cluster-0:7575.3
2025-02-19 09:09:56.371/6.062 Oracle Coherence CE 14.1.1.0.20 <Info> (thread=Proxy:ManagementHttpProxy:HttpAcceptor, member=1): HttpAcceptor now listening for connections on 0:0:0:0:0:0:0:0:30000
2025-02-19 09:09:56.932/6.623 Oracle Coherence CE 14.1.1.0.20 <Info> (thread=Proxy:ExtendHttpProxyService:HttpAcceptor, member=1): HttpAcceptor now listening for connections on 10.203.30.168:8080
2025-02-19 09:09:57.008/6.699 Oracle Coherence CE 14.1.1.0.20 <Info> (thread=Proxy:MetricsHttpProxy:HttpAcceptor, member=1): HttpAcceptor now listening for connections on 0:0:0:0:0:0:0:0:9612

Environment (please complete the following information):

• Coherence CE version: 14.1.1-0-20
• Java version and Java vendor: Not applicable
• Environment: Kubernetes
  Client Version: v1.32.0
  Kustomize Version: v5.5.0
  Server Version: v1.29.12-eks-2d5f260

Additional context
Add any other context about the problem here.

[sparkcatalyst96]

Hi @thegridman, I would appreciate it immensely if you could please have a look over my configuration, there's obviously something wrong that I'm doing in configuring the REST API...

Just for context, we are currently using Coherence version 3 and due to us being limited to Java 8, we are currently experimenting with Coherence CE v14.1.1.0 (AFAIK this is the last version supporting Java 8, please correct me if I'm wrong). I'm trying to do a simple PoC by installing it on our EKS cluster, enabling the REST API and performing some simple operations through the REST API before integrating it with our apps.

Thank you!

[thegridman]

I'm no expert in Coherence REST and quite frankly that part of Coherence has kind of fallen out of use as most people just build REST applications on top of Coherence using frameworks like Helidon, Spring or Micronaut. Coherence REST is not ideal for real production quality REST applications.

Having said that, it should still work. Have you actually tried running all this outside of Kubernetes by just starting Coherence locally with the same config and trying the http requests.

[sparkcatalyst96]

Hi @thegridman , thank you for the quick response! Unfortunately no, I wasn't able to test it with a sample Java app either.

Could you please confirm my understanding that there are actually 2 REST APIs, one for general cluster management and monitoring exposed over port 30000 and an Extend proxy defined according to the following acceptor:

          <proxy-scheme>
            <service-name>ExtendHttpProxyService</service-name>
            <acceptor-config>
                <http-acceptor>
                  <local-address>
                      <address>localhost</address>
                      <port>8080</port>
                  </local-address>
                </http-acceptor>
            </acceptor-config>
            <autostart>true</autostart>
          </proxy-scheme>

exposed on port 8080, which I should be able to use in order to interact with the data in the cache?

All I want to do is create/write to/read from and delete a cache over REST, that's all.

[thegridman]

Yes, there is a REST API for Coherence Management, which basically gives all the same sort of thing you can get from Coherence JMX MBeans.

What you are using is Coherence REST where you can access cache data as documented here for 14.1.1.0 https://docs.oracle.com/en/middleware/standalone/coherence/14.1.1.0/develop-remote-clients/building-your-first-coherence-rest-application.html#GUID-45D33C82-D9DA-455D-8691-FA341D86CF81

[sparkcatalyst96]

I tried grpc as an alternative but that doesn't seem to be available in the version I'm using (as I'm stuck with Java 8). Would you be able to recommend a simple example on how to test the cluster deployed on K8s with either REST or a Java client?

Thanks!

[sparkcatalyst96]

Following the documentation at https://docs.oracle.com/en/middleware/standalone/coherence/14.1.1.0/develop-remote-clients/building-your-first-coherence-rest-application.html#GUID-E2C5134A-FF8D-44B5-8777-3FE67424FAA6

and using the following:

java -cp DEV_ROOT\config;DEV_ROOT;DEV_ROOT\libs\dependencies; 
COHERENCE_HOME\coherence-rest.jar -Dcoherence.clusterport=8090 
-Dcoherence.ttl=0 
-Dcoherence.cacheconfig=DEV_ROOT\config\example-server-config.xml 
com.tangosol.net.DefaultCacheServer

and the coherence.jar (standalone install) https://www.oracle.com/middleware/technologies/coherence-archive-downloads.html

seems to be missing this class:
Error: Could not find or load main class com.tangosol.net.DefaultCacheServer

so I suspect something is definitely wrong.

[thegridman]

You are missing the Coherence jar from the class path, it appears you only have coherence-rest.jar

[aseovic]

@george-c29 @thegridman Is there anything else we need to do here, or can this issue be closed?

[sparkcatalyst96]

@aseovic I wasn't able to get the REST api working so I dropped this initiative completely for now...if you happen to have a straightforward guide on how to interact with the cache cluster solely through the REST API, that would be great. However, I tried the documentation examples listed above with no success.

Thanks!

[lsho]

@george-c29 What error did you get? Did you check the console output from the Coherence server to see that the Http server (HttpAcceptor service) is started and use the IP address and port the HttpAcceptor listens on in your Http request?

[sparkcatalyst96]

@lsho yes, I've posted a log extract in the first commend of all the acceptors that started and their listening port, so the answer is yes. However, using the examples above I wasn't able to reach it (I was getting a 404 no matter what I tried).

[lsho]

@george-c29 Your Http server is listening on:
(thread=Proxy:ExtendHttpProxyService:HttpAcceptor, member=1): HttpAcceptor now listening for connections on 10.203.30.168:8080

But your http request uses: 172.20.51.129:8080