This document (7018181) is provided subject to the disclaimer at the end of this document.
SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 11
NOTE: This is a live document. Content may change without further notice!
Filesystem errors are not uncommon, yet, need to be resolved to ensure a safe and stable system.
This document concentrates on errors seen with the BTRFS filesystem on SUSE Linux Enterprise
Note that at some of these errors we have to aim in two directions:
What actually caused the corruption?
Is there a bug in the btrfs tool or kernel driver to fix that corruption which prevents that?
Repairing a filesystem does not necessarily mean to recover the data, it means to fix the filesystem itself, not its content, at least not in every case.
Let’s start with some best practices first.
Whenever a BTRFS filesystem contains errors, this TID is a good starting point.
Let’s have a look on some typical errors seen in the past:
WARNING: CPU: 2 PID: 452 at ../fs/btrfs/extent-tree.c:3731 btrfs_free_reserved_data_space_noquota+0xe8/0x100 [btrfs]()
Good thing is, it’s a WARNING, not a fatal error.
WARNINGs like this one, e.g. regarding quota, typically are runtime only things that are fixed by BTRFS after the WARNING is issued. Not a bad problem.
Yet, such an issue should be reported to SUSE Support for closer examination.
If you see a message like:
BTRFS: Transaction aborted (error -2)
followed by a stack trace which looks like:
[<ffffffffa041277b>] __btrfs_abort_transaction+0x4b/0x120 [btrfs] [<ffffffffa0445f87>] __btrfs_unlink_inode+0x367/0x3c0 [btrfs] [<ffffffffa04499e7>] btrfs_unlink_inode+0x17/0x40 [btrfs] [<ffffffffa0449a76>] btrfs_unlink+0x66/0xb0 [btrfs] kernel: BTRFS warning (device sdb3): __btrfs_unlink_inode:3802: Aborting unused transaction(No such entry).
and the filesystem mounted read only, a possible way to fix this issue is to try:
mount -t btrfs -o recovery,ro /dev/<device_name> /<mount_point>
If this does not work, perform a backup then use the latest SUSE kernel and btrfs tools to start the repair procedure.
WARNING: Using ‘—repair‘ can further damage a filesystem instead of helping if it can’t fix your particular issue.
It is extremely important that you ensure a backup has been created before invoking ‘—repair‘. If any doubt open a support request first, before attempting a repair. Use this flag at your own risk. If you do not perform a backup and use an old kernel and old btrfs tool to attempt your repair, you may cause your data to be lost permanently. Use the latest quarterly update for the latest version of SUSE Linux Enterprise Server. For example, at the time of this writing that would be SLE-15-SP2-Online-x86_64-QU2-Media1.iso or SLE-15-SP2-Full-x86_64-QU2-Media1.iso.
btrfs check —repair /dev/<device_name> btrfs scrub start -Bf /dev/<device_name>
As a last resort cleaning the transaction log can be done with:
btrfs rescue zero-log /dev/<device_name>
More fatal issues are seen if the filesystem spits out tons of messages into the logs, slows down considerably or even goes readonly.
If the Root filesystem is affected, reboot the system into a independent rescue system from DVD, ISO image, USB pendrive, etc… Use the latest available rescue system, the more recent, the better.
What to do if:
- A bad tree root is found at mount time: use «-o recovery» This attempts to autocorrect that error.
- Weird ENOSPC issues seen: mount with «-o clear_cache» which will drop btrfs cache
- Quota issues prevent mounting: Needs the latest available btrfsprogs to fix that. See Section «Additional information»
- Quota issues seen during normal operation: run ‘btrfs quota rescan’
- Only if everything else fails, run ‘btrfs check‘ and research if repair could possibly fix this issue.
WARNING: If in doubt, open a case with Support. ‘btrfs check —repair‘ run with a version which can’t fix the particular issue might make things worse.
In some cases it may be necessary to use a more recent btrfs tools version from the latest service pack for the major version of SLE that you are using to repair a damaged filesystem from within that OS.
Example: For SLE15 GA, SP1 or SP2 it may work out to use the latest SP2 version of btrfsprogs.
Download the SP2 QU2 ISO image from the customer center and mount it to the /mnt directory on the system with the broken filesystem.
Extract the btrfs tool from the btrfsprogs RPM:
rpm2cpio /mnt/Module-Basesystem/x86_64/btrfsprogs-4.19.1-8.6.2.x86_64.rpm | cpio -id ./usr/sbin/btrfs
Then use ./usr/sbin/btrfs check —repair /dev/<defective btrfs device>
NOTE: This only works for a btrfs filesystem which is not mounted, it doesn’t work for the root filesystem of a running system.
To repair that, reboot the system and boot the rescue system from latest quarterly update ISO for the latest major release of SUSE Linux Enterprise Server.
Further: as said above, use with care. In doubt, run «./usr/sbin/btrfs check /dev/<defective btrfs device>» (without —repair) and send the output to SUSE Support for advice.
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented «AS IS» WITHOUT WARRANTY OF ANY KIND.
SYNOPSIS
btrfs check [options] <device>
DESCRIPTION
The filesystem checker is used to verify structural integrity of a filesystem
and attempt to repair it if requested. It is recommended to unmount the
filesystem prior to running the check, but it is possible to start checking a
mounted filesystem (see —force).
By default, btrfs check will not modify the device but you can reaffirm that
by the option —readonly.
btrfsck is an alias of btrfs check command and is now deprecated.
Warning
Do not use —repair unless you are advised to do so by a developer
or an experienced user, and then only after having accepted that no fsck
successfully repair all types of filesystem corruption. E.g. some other software
or hardware bugs can fatally damage a volume.
The structural integrity check verifies if internal filesystem objects or
data structures satisfy the constraints, point to the right objects or are
correctly connected together.
There are several cross checks that can detect wrong reference counts of shared
extents, backreferences, missing extents of inodes, directory and inode
connectivity etc.
The amount of memory required can be high, depending on the size of the
filesystem, similarly the run time. Check the modes that can also affect that.
SAFE OR ADVISORY OPTIONS
- -b|—backup
-
use the first valid set of backup roots stored in the superblock
This can be combined with —super if some of the superblocks are damaged.
- --check-data-csum
-
verify checksums of data blocks
This expects that the filesystem is otherwise OK, and is basically an offline
scrub that does not repair data from spare copies. - --chunk-root <bytenr>
-
use the given offset bytenr for the chunk tree root
- -E|—subvol-extents <subvolid>
-
show extent state for the given subvolume
- -p|—progress
-
indicate progress at various checking phases
- -Q|—qgroup-report
-
verify qgroup accounting and compare against filesystem accounting
- -r|—tree-root <bytenr>
-
use the given offset ‘bytenr’ for the tree root
- --readonly
-
(default)
run in read-only mode, this option exists to calm potential panic when users
are going to run the checker
- -s|—super <N>
-
use Nth superblock copy, valid values are 0, 1 or 2 if the
respective superblock offset is within the device sizeThis can be used to use a different starting point if some of the primary
superblock is damaged. - —clear-space-cache v1|v2
-
completely remove the free space cache of the given version
See also the clear_cache mount option.
- --clear-ino-cache
-
remove leftover items pertaining to the deprecated inode map feature
DANGEROUS OPTIONS
- --repair
-
enable the repair mode and attempt to fix problems where possible
Note
There’s a warning and 10 second delay when this option is run without
—force to give users a chance to think twice before running repair, the
warnings in documentation have shown to be insufficient. - --init-csum-tree
-
create a new checksum tree and recalculate checksums in all files
Warning
Do not blindly use this option to fix checksum mismatch problems.
- --init-extent-tree
-
build the extent tree from scratch
Warning
Do not use unless you know what you’re doing.
- --mode <MODE>
-
select mode of operation regarding memory and IO
The MODE can be one of:
- original
-
The metadata are read into memory and verified, thus the requirements are high
on large filesystems and can even lead to out-of-memory conditions. The
possible workaround is to export the block device over network to a machine
with enough memory. - lowmem
-
This mode is supposed to address the high memory consumption at the cost of
increased IO when it needs to re-read blocks. This may increase run time.
Note
lowmem mode does not work with —repair yet, and is still considered
experimental. - --force
-
allow work on a mounted filesystem and skip mount checks. Note that
this should work fine on a quiescent or read-only mounted filesystem
but may crash if the device is changed externally, e.g. by the kernel
module.Note
It is possible to run with —repair but on a mounted filesystem
that will most likely lead to a corruption unless the filesystem
is in a quiescent state which may not be possible to guarantee.This option also skips the delay and warning in the repair mode (see
—repair).
EXIT STATUS
btrfs check returns a zero exit status if it succeeds. Non zero is
returned in case of failure.
AVAILABILITY
btrfs is part of btrfs-progs. Please refer to the documentation at
https://btrfs.readthedocs.io.
SEE ALSO
mkfs.btrfs(8),
btrfs-scrub(8),
btrfs-rescue(8)
Скопирую сюда хорошее описание порядка действия для починки BTRFS. На английском.
The below are the steps I would recommend for ANY btrfs issue, smart
people reading dmesg or syslog can probably figure out which of these
steps they’d need to skip to in order to fix their particular problem.
Step 1 — boot to a suitable alternative system, such as a different
installation of openSUSE, a liveCD, or an openSUSE installation DVD.
The installation DVD for the version of openSUSE you are running is
usually the best choice as it will certainly use the same kernel/btrfs
version.
Step 2 — Go to a suitable console and make sure you do the below as root
Step 3 — Try to mount your partition to /mnt, just to confirm it’s
really broken (eg. «mount /dev/sda1 /mnt»)
Step 4 — If it mounts — are you sure it’s broken? if Yes — run «btrfs
scrub start /mnt» to scrub the system, and «btrfs scrub status /mnt»
to monitor it
Step 5 — If it doesn’t mount, try to scrub the device just in case it
works (eg. «btrfs scrub start /dev/sda1» and «btrfs scrub status
/dev/sda1» to monitor). Try mounting, if yes, you’re fixed.
Step 6 — If scrubbing is not an option or does not resolve the issue
then try «mount -o usebackuproot» instead (eg. «mount -o usebackuproot
/dev/sda1 /mnt»).
==Interlude==
All of the above steps are considered safe and should make no
destructive changes to disk, and have fixed every filesystem issue
I’ve had on btrfs in the last 5 years
Full disk issues need a different approach (basically, delete stuff
;)) documented here:
https://www.suse.com/documentation/sles-12/stor_admin/data/sect_filesystems_trouble.html
If the above doesn’t fix things for you, you can continue with the
below steps but the situation is serious enough to justify a bug
report, please!
==
Step 7 — Run «btrfs check » (eg. «btrfs check /dev/sda1»).
This isn’t going to help, but save the log somewhere, it will be
useful for the bug report.
Step 8 — Seriously consider running «btrfs restore » (eg. «btrfs restore /dev/sda1 /mnt/usbdrive»). This
won’t fix anything but it will scan the filesystem and recover
everything it can to the mounted device. This especially useful if
your btrfs issues are actually caused by failing hardware and not
btrfs fault.
Step 9 — Run «btrfs rescue super-recover » (eg. «btrfs rescue
super-recover /dev/sda1»). Then try to mount the device normally. If
it works, stop going.
Step 10 — Run «btrfs rescue zero-log » (eg. «btrfs rescue
zero-log /dev/sda1»). Then try to mount the device normally. If it
works, stop going.
Step 11 — Run «btrfs rescue chunk-recover » (eg. «btrfs rescue
chunk-recover /dev/sda1»). This will take a LONG while. Then try to
mount the device normally. If it works, stop going.
Step 12 — Don’t just consider it this time, don’t be an idiot, run
«btrfs restore » (eg. «btrfs restore
/dev/sda1 /mnt/usbdrive»).
Step 13 — Seriously, don’t be an idiot, use btrfs restore
==Danger zone=
The above tools had a small chance of making unwelcome changes, but
now you’re in the seriously suicidal territory, do not do the
following unless you’re prepared to accept the consequences of your
choice.
==
Step 14 — Now, ONLY NOW, try btrfsck aka «btrfs check —repair
» (eg. «btrfs check —repair /dev/sda1»)
There, I’m very confident the above will help you Andrei, and for
everyone else, can we please bury the nonsense that btrfs is lacking
when it comes to repair and recovery tools?
You have scrub which is safe for day to day use, the perfectly safe
usebackuproot mount option, and the various «btrfs rescue» commands
which are only moderately worrying compared to the practical Russian
roulette which is «btrfs check»
Источник
You should install smartmontools and run a long test (will take a while)
#smartctl -t long /dev/sd?
then it fails on the bad block
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 80% 682 1193046
so you have the LBA address of the block (1193046).
Then you install sg_utils and run with lba the lba address from above
# sg_verify --lba=1193046 /dev/sda
You will get a response like
# sg_verify --lba=1193046 /dev/sdb
verify (10): Fixed format, current; Sense key: Medium Error
Additional sense: Unrecovered read error
Info fld=0x123456 [1193046]
Field replaceable unit code: 228
Actual retry count: 0x008b
medium or hardware error, reported lba=0x123456
so you will know that this sector is really bad and could not been automatically put to the defective list of the micro controller of the disk.
you can check the defects list with
# sg_reassign --grown /dev/sda
>> Elements in grown defect list: 0
and if you reallocate this sector with
# sg_reassign --address=1193046 -v /dev/sda
and you check the grown defects list afterwards with
# sg_reassign --grown /dev/sdb
>> Elements in grown defect list: 1
you should see the counter grow by 1.
After this you should run
#smartctl -t long /dev/sd?
again and retry this procedure until the disk is clean and the long test runs without errors.
In this case I would use this disk for non-important stuff like a steam library or something like this. But I would replace the disk just to be sure.
But for the moment the disk should be ok.
5
5
Debian 10. Раздел с btrfs 3.6Tb для данных пользователей.
После перезагрузки вдруг перестал монтироваться, опции монтирования recovery,ro не помогали.
SMART пишет GOOD, но Victoria показывает 198-Offline_Uncorrectable красным.
btrfs check /dev/sda4 завершалась аварийно
btrfs rescue zero-log /dev/sda4 выполнился, но раздел не смонтировался
Помогло следующее: из ветки testing установил btrfs.progs версии 5
btrfs check /dev/sda4 стал нормально сыпать ошибками
btrfs check —repair /dev/sda4 стал ремонтировать.
Ждать не стал, прервал ремонт. Раздел смонтировался в ro и удалось все переписать.
Ради эксперимента запустил снова btrfs check —repair /dev/sda4 , ремонт продолжился, прождал три дня, окончания не дождался, прервал. Раздел переформатировал в btrfs.
Мои ошибки: Оказалось, что metadata и system были single. В UPS батарея тест проходила, но не держала.
PS. Есть ли возможность подключить второй диск к разделу btrfs и настроить так, чтобы metadata и system были raid1, data=single и при этом все данные писались бы исключительно на первый диск, не залезая на второй,
т.е. второй диск только для raid1 для metadata и system, а все данные только на первом диске?