docs: clarity on LIST operations

[udf2457]

The docs talk of use_update_marker to reduce S3 bills, but the obvious question here is what is the frequency of the LIST calls in the first place. The docs are silent on that ?

The different commercial S3 providers provide different levels of "free" LIST calls and so it would be nice to have an accurate guess at at which point people would exceed "free" and have a good idea of likely S3 bills.

[wojas]

This is the LS config item that controls how often we do a LIST if use_update_marker is not in use:

# Check the storage for new snapshots at this interval
#storage_poll_interval: 1s

The relationship between this config item and S3 LIST requests, and how this affects costs should indeed be documented.

If I remember correctly, with AWS and use_update_marker enabled, the estimated costs for a small deployment would be around 1 USD per month per node. Please correct me if you find this is incorrect.

For AWS a LIST command is 10x more expensive than a GET command, so use_update_marker reduces the poll costs tenfold, minus the occasional LIST forced by update_marker_force_list_interval, which defaults to 5m.

[udf2457]

Thanks @wojas . But I thought the docs said use_update_marker was only for use in test projects and not really for production use ? So I guess as you say, bumping up the storage_poll_interval would be the only way to go if that is the case.

[wojas]

It's fine for production use if you are not doing cross region or datacenter S3 bucket syncing.
And even if you use such syncing, the update_marker_force_list_interval will fixup any issues with updates missed due to the update marker and files not being synced in the right order. Worst case a change will be delayed by that interval.