Failed to delete microservices running on an agent with coredns containers running

[xiuyouxu]

I have a ioFog agent running with coredns/coredns:latest(Image id: ead0a4a53df8) containers on it. I deployed a microservice on that agent and then I wanted to delete the microservice and failed. The iofog-agent log gave

[08/12/2023 09:52:04.997] [SEVERE] [PMCM] [Process Manager] : com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type `com.github.dockerjava.api.model.Capability` from String "all": value not one of declared Enum instance names: [ALL, SYS_BOOT, DAC_OVERRIDE, NET_RAW, BLOCK_SUSPEND, FOWNER, IPC_LOCK, IPC_OWNER, SYS_PACCT, NET_BIND_SERVICE, WAKE_ALARM, FSETID, DAC_READ_SEARCH, SYS_CHROOT, SYS_RAWIO, SYS_ADMIN, KILL, MAC_ADMIN, SYS_RESOURCE, CHOWN, SETPCAP, SYS_PTRACE, NET_ADMIN, SETFCAP, SYS_NICE, LINUX_IMMUTABLE, AUDIT_CONTROL, LEASE, AUDIT_WRITE, SYS_MODULE, MKNOD, SYSLOG, MAC_OVERRIDE, SYS_TIME, SETGID, SETUID, SYS_TTY_CONFIG, NET_BROADCAST]
 at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 2054] (through reference chain: com.github.dockerjava.api.command.InspectContainerResponse["HostConfig"]->com.github.dockerjava.api.model.HostConfig["CapDrop"]->java.lang.Object[][0]) - Exception: Error monitoring container - Stack trace: org.eclipse.iofog.exception.AgentSystemException: Error monitoring container
        at org.eclipse.iofog.process_manager.ProcessManager.lambda$new$1(ProcessManager.java:117)
        at java.lang.Thread.run(Thread.java:748)
Caused by: javax.ws.rs.client.ResponseProcessingException: com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type `com.github.dockerjava.api.model.Capability` from String "all": value not one of declared Enum instance names: [ALL, SYS_BOOT, DAC_OVERRIDE, NET_RAW, BLOCK_SUSPEND, FOWNER, IPC_LOCK, IPC_OWNER, SYS_PACCT, NET_BIND_SERVICE, WAKE_ALARM, FSETID, DAC_READ_SEARCH, SYS_CHROOT, SYS_RAWIO, SYS_ADMIN, KILL, MAC_ADMIN, SYS_RESOURCE, CHOWN, SETPCAP, SYS_PTRACE, NET_ADMIN, SETFCAP, SYS_NICE, LINUX_IMMUTABLE, AUDIT_CONTROL, LEASE, AUDIT_WRITE, SYS_MODULE, MKNOD, SYSLOG, MAC_OVERRIDE, SYS_TIME, SETGID, SETUID, SYS_TTY_CONFIG, NET_BROADCAST]
 at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 2054] (through reference chain: com.github.dockerjava.api.command.InspectContainerResponse["HostConfig"]->com.github.dockerjava.api.model.HostConfig["CapDrop"]->java.lang.Object[][0])
        at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:873)
        at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:767)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:229)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:414)
        at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:765)
        at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:428)
        at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:324)
        at com.github.dockerjava.jaxrs.InspectContainerCmdExec.execute(InspectContainerCmdExec.java:30)
        at com.github.dockerjava.jaxrs.InspectContainerCmdExec.execute(InspectContainerCmdExec.java:13)
        at com.github.dockerjava.jaxrs.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:23)
        at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
        at com.github.dockerjava.core.command.InspectContainerCmdImpl.exec(InspectContainerCmdImpl.java:52)
        at org.eclipse.iofog.process_manager.DockerUtil.lambda$getRunningContainers$2(DockerUtil.java:363)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
        at org.eclipse.iofog.process_manager.DockerUtil.getRunningContainers(DockerUtil.java:367)
        at org.eclipse.iofog.process_manager.ProcessManager.deleteRemainingMicroservices(ProcessManager.java:199)
        at org.eclipse.iofog.process_manager.ProcessManager.lambda$new$1(ProcessManager.java:114)
        ... 1 more

docker inspect with the coredns container gave this

    "HostConfig": {
        "CapAdd": [
            "NET_BIND_SERVICE"
        ],
        "CapDrop": [
            "all"
        ],

and my docker version on that agent is

Client:
 Version:           20.10.25
 API version:       1.40
 Go version:        go1.18.1
 Git commit:        20.10.25-0ubuntu1~20.04.1
 Built:             Fri Jul 14 22:00:45 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.25
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.1
  Git commit:       20.10.25-0ubuntu1~20.04.1
  Built:            Thu Jun 29 21:55:06 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.2
  GitCommit:
 runc:
  Version:          1.1.7-0ubuntu1~20.04.1
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:

I do not know how to fix it. My work depends on ioFog deploying and deleting microservices properly and hope it can be fixed.