[I/O] HDF Data Loader not showing speedup for multi threaded

[tijyojwad]

HDF Data Loader is setup to support simultaneous loads by multiple python processes. And this functionality works as evidenced by debugging. However, the multi process support isn't leading to a speedup during training.

e.g. In the consensus sample, GPU utilization sits at around 80% using both 1 and 32 threads for data loader.

[tijyojwad]

On further investigation, it looks like the data loader time (40ms) is minimal compared to the batch processing time (39.3s) in ConsensusRNN with batch size 32. Increasing batch sizes shows a similarly network processing time heavy trend. So data loading doesn't appear to be the bottleneck, and debug print messages show multi process loading working correctly.

Keeping this open to investigate why perf for ConsensusRNN is limited to 80% despite increasing batch size.