[feat] Add training code for InternVLA-N1 #184
Conversation
* update vln yaml; fix import agent * update habitat, using evaluator and config; env and agent is WIP * add distributed_base evaluator * Habitat env applied, distributed evaluator applied; clean evaluator and agent created * fix observation issues * update new register name; tiny fix on style * latest tested * delete temp agent; rename default evaluator for habitat * update slurm bash * update readme * fix init dist print * fix eval config; fix local rank to rank * update init distributed mode if condition * update dist for dlc * fix bug in evaluator * update distributed vln multi, episode loader * cma tested * cma tested; episode loader; torchrun; local agent tested; TODO fix isaac sim gpu setting * add vlnpe distributed script * add grscene; add new result write and resumble load data feature based on lmdb path + rank; remove TODOs and unused files * fix bugs in evaluator and dataset for distributed; n1 and rdp tested * fix comm log concurrency issue, create file with exist true * fix progress log mkdir race condition * fix comments * polish existing configs and bash * update bash align with doc * fix CI test * rename habitat_extensions * fix comments * remove useless line * [file] Update 3D Printing Files for Camera of Unitree Go2 (InternRobotics#176) * [file] Update 3d printing files for camera of Unitree Go2. * [Doc] Update readme for 3d printing models. * fix visualize image size; fix unused comment code; fixed evaluator name * bump to version v0.0.2 * update vlnmulti to VLN; update habitatVln to HabitatVLN * :Revert "Merge branch 'main' into vlnpe_refactor" This reverts commit 7a23b72, reversing changes made to ea73075. --------- Co-authored-by: Yuqiang Yang <49871992+yuqiang-yang@users.noreply.github.com>
| from .rope2d import get_rope_index_2, get_rope_index_25 | ||
|
|
||
| # Define placeholders for dataset paths | ||
| CAMBRIAN_737K = { |
There was a problem hiding this comment.
Are these datasets used in finetuning?
There was a problem hiding this comment.
Qwenvl's default placeholders for dataset paths
| import copy | ||
| import itertools | ||
| import json | ||
| import os |
There was a problem hiding this comment.
@kew6688 Check the conflicts between this file and the refactored evaluator.
There was a problem hiding this comment.
Acknowledged. Conflicts are marked. Will wait for @kew6688 's adaptation before merging.
There was a problem hiding this comment.
Adaption is ready to merge in this PR
- Fixed vlnpe internvla_n1 agent with the updated n1 model
- Merged refactored Habitat Evaluator with decoupled System 2 and Dual System evaluation functions
- Downgrade diffusers version 0.33.1 -> 0.32.2
| masked_loss = loss * mask | ||
| loss = masked_loss.sum() / mask.sum() / (loss.shape[1] * loss.shape[2]) | ||
| elif 'navdp' in self.get_system1_type(): | ||
| cur_images = traj_images.flatten(0, 1) |
There was a problem hiding this comment.
Does this model support the async navdp system1 (aka. InternVLA-N1-Preview)?
There was a problem hiding this comment.
This release just supports the inference of InternVLA-N1-Preview.
| input_ids.to(self.get_model().device) | ||
| input_ids = torch.cat([input_ids, torch.tensor([[TRAJ_START_TOKEN_INDEX]]).to(input_ids.device)], dim=1) | ||
| text_embeds = self.get_model().embed_tokens(input_ids) | ||
| with torch.no_grad(): |
There was a problem hiding this comment.
The kv cache of generat_latents has some unfixed bugs. The outputs after using kv cache are not aligned with the ones that do not use kv cache. So please make sure the past_key_values is None to disable the kv cache.
There was a problem hiding this comment.
Code related to kv cache has been removed. Support will be added back in future releases.
… in generate_traj
3 participants
✨ InternVL-N1 Dual-System Variants
📊 Extended Evaluation Support