anna-route crashes in ebs (disk-tier) mode

[samyaditya]

I'm trying to run anna server as disk node in local mode. I changed the SERVER_TYPE in start-anna-local.sh to "ebs". anna-kvs, anna-monitor, anna-route runs as seen in ps. When I run anna-cli and do some PUT or GET, it hangs for 10 seconds and then says "Failure!". Also anna-route crashes as soon as any PUT or GET is triggered from ana-cli.

Tried changing ebs-cap from 0 to 1 & set replication->ebs to 1 in config, but no luck. Am I missing something ? Is this incorrect procedure to run server as disk node. Any help is appreciated.

[samyaditya]

Update: I am able to run anna server (route & kvs) in DISK tier node if there is another node running as MEMORY.
Question: Is it required to have at-least 1 MEMORY node along with a DISK node ?

Also, when I "PUT" something using anna-cli, it gets written on disk (./ebs_0/key). But if I restart the seed server (tier DISK), and then do "GET" of previously added key, it doesn't reads from the file & shows empty string (client gets LatticeType::NONE). How do I make it read from the stored key file ?

[cw75]

@samyaditya Yes, at least one node is required because we use that node to store all the metadata, including the key's replication factor and node membership info.

I think one possibility is that when the seed server restarts, its IP may have changed. Since we use the hash of the server ip to determine key affinity, changing the server's IP could lead to a GET request being routed to a wrong node.

Sorry for the delayed response! Since I graduated I haven't been actively working on and monitoring this repo.

[samyaditya]

Thanks for the reply @cw75 . Regarding reading from DISK tier, I'm not restarting the whole machine that's running anna. I'm just killing the anna related processes (kvs, route, monitor) & then starting them again. Also, the IP of both the nodes (docker containers) is unique & static. I added some trace prints in the code & came to know that the function supposed to read from DISK (ie. DiskLWWSerializer::get()) is never called. Instead, the data is always fetched from another node that is running as MEMORY.

Here is what I'm doing:
Running 2 nodes of anna as 1 MEMORY + 1 DISK seed node. Each node is running all 3 (monitor, route, kvs).
Client does 'PUT key value'. The pair gets synced in both nodes & after a couple of seconds gets written on disk (./ebs_0/key) in the DISK node.

Scenario 1: I stop & start all 3 anna processes in DISK node. When clients (ran on memory & disk nodes), does a 'GET key', it's fetched from the MEMORY node & correct value is printed.

Scenario 2: I stop & start all 3 anna processes in both DISK & MEMORY nodes. When clients (ran on memory & disk nodes), does a 'GET key', it tries to fetch from the MEMORY node & since it's not found, empty value is printed (client gets LatticeType::NONE).

Any idea what might be going wrong ? How to make it read from disk ?

[Yenofer]

Hello, congratulations on your great progress! I'm in some trouble, and I really need some guidance. I don't know how to run two nodes on a local Linux VM. I can only run one anna-cli now. If I enter build/cli/anna-cli conf/anna-local.yml on a new command line, I get the following error:

terminate called after throwing an instance of 'zmq::error_t'
what(): Address already in use
Aborted (core dumped)

What should I do, please? Thank you very much!