Refactor trajectory handling #758
Description
I think we should refactor the way we handle demonstrations inside of imitation.
Skimming over the code it looks like we spend way too much LOC on supporting and converting between different trajectory formats (with or without rewards, transitions, transitions with next_obs and dones). I have the vague hunch that there is a lot of potential to reduce complexity, LOC and even improve performance by using the HuggingFace datasets library together with PyTorch dataloaders.
Originally posted by @ernestum in #651 (comment)