check --repair fails with exception and OSError: [Errno 19] No such device

[DonkeeeyKong]

Hello,

On my NAS I have nightly Borg Backups to a mergerfs pool. A few days ago those failed with OSError: [Errno 30] Read-only file system. One of the drives had a very high UDMA_CRC_Error_Count, so I switched the cable and the drive has been working fine since. fsck showed no problems either, an extended S.M.A.R.T test was fine and the UDMA_CRC_Error_Count hasn't gone up anymore. My backups have been working fine now, except one which fails with borg.helpers.errors.IntegrityError: Data integrity error: Invalid segment entry size 2685978130 - too big [segment 13115, offset 88836055].

Now, when I run borg check it shows some errors:

Data integrity error: Segment entry checksum mismatch [segment 13115, offset 8]                                                                                          
finished segment check at segment 13134                                                                                                                                  
Starting repository index check                                                                                                                                          
Index object count mismatch.                                                                                                                                             
committed index: 156060 objects                                                                                                                                          
rebuilt index:   155752 objects  

It is not possible to check --repair the repository though:

Starting repository check                                                                                                                                                
Data integrity error: Segment entry checksum mismatch [segment 13115, offset 8]                                                                                          
attempting to recover /srv/mergerfs/Archiv/Backups/kvm-borg/data/13/13115                                                                                                
Local Exception                                                                                                                                                          
Traceback (most recent call last):                                                                                                                                       
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1039, in check
    objects = list(self.io.iter_objects(segment))                          
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1512, in iter_objects
    size, tag, key, data = self._read(fd, self.header_fmt, header, segment, offset,
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1611, in _read
    raise IntegrityError('Segment entry checksum mismatch [segment {}, offset {}]'.format(
borg.helpers.errors.IntegrityError: Data integrity error: Segment entry checksum mismatch [segment 13115, offset 8]
                                                                                                                                                                         
During handling of the above exception, another exception occurred:            
                                                                                    
Traceback (most recent call last):                                                                                                                                       
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 5213, in main
    exit_code = archiver.run(args)                                                  
                ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 5144, in run
    return set_ec(func(args))                                                                                                                                            
                  ^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 183, in wrapper                                                                                           
    return method(self, args, repository=repository, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                           
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 343, in do_check
    if not repository.check(repair=args.repair, save_space=args.save_space, max_duration=args.max_duration):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1044, in check
    self.io.recover_segment(segment, filename)
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1542, in recover_segment
    with mmap.mmap(src_fd.fileno(), 0, access=mmap.ACCESS_READ) as mm:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 19] No such device
                                                                                                                                                                         
Platform: Linux openmediavault 6.1.0-41-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.158-1 (2025-11-09) x86_64  
Linux: Unknown Linux                                                                
Borg: 1.2.4  Python: CPython 3.11.2 msgpack: 1.0.3 fuse: llfuse 1.4.1 [pyfuse3,llfuse]
PID: 42659  CWD: /home/user                                                                                                                                          
sys.argv: ['/usr/bin/borg', '-pv', 'check', '--repair', '/srv/mergerfs/Archiv/Backups/kvm-borg']
SSH_ORIGINAL_COMMAND: None    

Any ideas what's happening here and what I can do to save that repository?

Any help is welcome. Thanks in advance!

[ThomasWaldmann]

Well, mmap is not working, that's unexpected because it usually works.

recover_segment tries to save as much uncorrupted data from your corrupted segment files as possible (and just leaves out the corrupted data).

I will check if there were any relevant changes to mmap usage after 1.2.4:

• no changes to recover_segment in 1.2.9 (last 1.2.x release) or 1.4.x

Check that (comment right above where it crashed):

                # note: file must not be 0 size or mmap() will crash.

Update: no, there is code right above that, that avoids 0 byte files.

So, guess you have something unusal there, mmap usually works on Linux.

Maybe that mergerfs? You could try copying that repo to a known-good filesystem (like ext4 or zfs or ...) and known-good disk and try repairing it there.

[DonkeeeyKong]

Thanks for the quick help! (and for the great program also!)