Improve C++ API for data transfers to and from streams

[Mellich]

Currently, all ACCL calls like send/recv require a reference to a BaseBuffer object as mandatory argument. When streaming is used as input or output, this requires to create and pass a BaseBuffer which is not used at all.

CCLO *ACCL::send(BaseBuffer &srcbuf, unsigned int count, unsigned int dst, unsigned int tag, communicatorId comm_id, bool from_fpga, streamFlags stream_flags, dataType compress_dtype, bool run_async, std::vector<CCLO *> waitfor)

There are different options to change/extend the API to overcome this issue:

• Pass the buffers as pointer. This would allow to use a nullptr for the argument. This would already be handled by ACCL::prepare_call but it would change the signatures for all calls.
• Create specific overloads of the calls for streaming that do not use the input buffer and implicitly set the input streaming flag
• introduce additional calls ACCL::stream_send, ACCL::stream_recv ... that explicitly support streaming and only take the required arguments as input.

I personally would favor the last option because it is most explicit. However, I am not sure if it is flexible enough to support further enhancements of ACCL in the future since we would basically remove control over the streaming flags by the user.

I would also argue for a stream_recv call because the current way of handling communication via streams as one-sided communication is ambiguous. What happens in cases, where two ranks send data to the same destination rank? The order the messages will arrive is not defined and there is no way to find out the current sender on the receiving side. With that, we need to use barriers to enforce the correct order. With the stream_recv calls we would be able to define the order how the messages are received and handled by the user kernel.

I think all this needs further discussion to find the best solution. Maybe there are also technical restrictions i am not aware of.