I get loads of warnings when backing up my running Postfix and Courier server files like:
file has vanished: /var/kunden/mail/username/name@mymail.de/tmp/courier.lock
How do I suppress those warnings from rsync when running it from Cron /usr/bin/rsnapshot hourly?
Can I somehow exclude those dirs?
/var/kunden/mail/*/*/tmp/
the tmp folder can be deeper as well, for example:
file has vanished: /var/kunden/mail/username/name@mymail.de/.Presse/tmp/1353871473.M716135P32214_imapuid_36.test.de
file has vanished: /var/kunden/mail/username/name@mymail.de/.Presse/tmp/courier.lock
asked Aug 15, 2013 at 14:39
rubo77rubo77
27.4k42 gold badges126 silver badges197 bronze badges
Unfortunately, unlike what is described in SWdream solution, --ignore-missing-args has no impact on vanished files. It will simply ignore source arguments that doesn’t exist.
See man rsync:
--ignore-missing-args When rsync is first processing the explicitly requested source files (e.g. command-line arguments or --files-from entries), it is normally an error if the file cannot be found. This option suppresses that error, and does not try to transfer the file. This does not affect subsequent vanished-file errors if a file was initially found to be present and later is no longer there.
The «official» way of ignoring vanished file is to use this script from the official rsync source repository:
https://git.samba.org/?p=rsync.git;a=blob_plain;f=support/rsync-no-vanished;hb=HEAD
which is very similar to what @kenorb and @gilles-quenot said.
answered Mar 23, 2017 at 10:57
The reason is these files were existing while rsync is building the list of files to transfer but they are removed before transferring.
It is a warning massage, not an error. However, you should try to find out why these file was deleted, it is maybe important.
To ignore this warning, you can use —exclude option as above question or use -ignore-missing-args rsync option, it makes rsync ignores vanished files:
--ignore-missing-args ignore missing source args without error
it maybe helps.
answered Jul 27, 2016 at 3:47
SWdreamSWdream
3132 silver badges3 bronze badges
4
You can use rsync‘s exclude switch (--exclude):
$ rsync -avz --exclude '**/tmp/' source/ destination/
Specified this way --exclude '**/tmp/' will ignore any path which includes the string /tmp/. You can provide patterns to this arguments as well.
Example
$ rsync -avz --exclude '/path/to/*/tmp/' source/ destination/
Will exclude on paths of the form: /path/to/*/tmp/.
answered Aug 15, 2013 at 15:44
slm♦slm
361k114 gold badges762 silver badges869 bronze badges
6
The error means that rsync can’t find the files anymore which were existing while building the list to transfer. These vanished-file errors happens when a file was initially found to be present and later is no longer there. In some cases it also happens when the source files are corrupted or have invalid characters in the name (so fsck is advised).
Basically this is a warning, not an error, so nothing to worry about, since the state of each destination file reflects a state that the corresponding source files during the run.
If it is causing problem because of exit value is non-zero, this can be solved by the following wrapper script (source):
#!/bin/bash
(rsync "$@"; if [ $? == 24 ]; then exit 0; else exit $?; fi) 2>&1 | grep -v 'vanished'
or by the following workaround script (source):
#!/bin/sh
OUT=`/usr/bin/snapback2 2>&1`
RET=$?
if [ "$RET" != "23" -a "$RET" != "0" -a "$RET" != 24 ]; then
echo "$OUT"
exit $RET
fi
which basically exists with the same error codes as rsync if rsync fails.
This is further discussed in: Bug 3653 — Reduce the need for the «vanished files» warning
answered Feb 12, 2016 at 19:07
kenorbkenorb
20k14 gold badges137 silver badges162 bronze badges
1
Or simply (with modern bash):
#!/bin/bash
/usr/bin/rsync "$@" 2> >(grep -Ev '(file has |rsync warning: some files )vanished')
ret=$?
((ret==24)) && exit 0 || exit $ret
answered Mar 28, 2016 at 21:50
Gilles QuénotGilles Quénot
31k6 gold badges64 silver badges80 bronze badges
3
My answer might be a special use case but I felt it was worth noting, so people don’t lose data if they fall under this case.
I was getting that message for several files whilst performing a routine backup of drives via rsync.
I performed a System Check on the disk and it turns out there are issues with the drive (allocation/file corruption etc) and recommended a backup + restore + reformat.
So before you go out an suppress or ignore the message outright, it might be wise to just run a health check on your drive just to be safe.
answered Mar 3, 2018 at 0:56
1
Use this parameter:
--exclude-from="./exclude.ini"
to put the list in a file.
In the exclude.ini file write something like:
Cache
cache2/*
*.lock
/temp
Where:
- excludes ALL folders named «Cache» and his content;
- excludes the content of all «cache2» named folders but include
folders hierarchy so you will get all subfolders but empty; - excludes all files terminating with .lock (eg. some lock files of
Firefox and derivates); - excludes the content of /temp folder.
Remember that if file system is case-dipendent you may have to account for this.
answered Sep 23, 2019 at 17:28
Try mount your backup disk with nosuid,nodev,nofail,x-gvfs-show options.
Not sure which system you’re using but I believe this is related to the mount options of your disk. On Linux this happens if I set the mount options to User Session Default. It resolves when I disable it and rsync completes without any errors.
answered Aug 3, 2019 at 15:44
Max S.Max S.
1075 bronze badges
You must log in to answer this question.
Not the answer you’re looking for? Browse other questions tagged
.
Not the answer you’re looking for? Browse other questions tagged
.
Are you stuck with rsync error 24?
Errors are common while executing Rsync commands. When the Rsync command fails, it returns error codes with a predefined error description.
The error code 24 mainly happens when there is a partial transfer due to vanished source files.
At Bobcares, we often receive requests to fix the Rysnc error 24 as part of our Server Management Services.
Today, let’s get deep into this Rsync error and see how our Support Engineers fix it for our customers.
What causes Rsync error 24?
Have a vague idea about this error? Let’s discuss it in detail.
Recently, one of our customers approached us with a rsync error. He tried to take a network backup from his disk but it failed with a rsync error 24. He was not making any changes in the disk but faced this error frequently.
The exact message was:
rsync warning: some files vanished before they could be transferred (code 24) at
main.c(892) [sender=2.6.8]We’ll now take a look at the two main reasons for the error.
1. Missing source file
Rsync error 24 is a warning message that shows up due to missing source files. Rsync usually builds a file list first before doing the actual data transfer.
In other words, error 24 means the files were existing while rsync was building the list of files to transfer. But they were removed before transferring.
The files vanishing during a live backup are perfectly normal. For example, a lot of applications create short-lived temporary files. And, they will be removed later on. This particularly happens in the case of mail servers, where files containing e-mail messages are constantly moved from one directory to another.
2. Error reading file name
Similarly, the 24 error can even happen due to the presence of some special characters in the file name. This prevents Rsync from reading the file.
Or when the drive containing the files has errors, it can also create trouble for Rsync to access files. Additionally, insufficient permissions also create Rsync problems.
How we fix it?
So far, we have discussed the causes of error 24. Now, let’s see how our Support Engineers fix this error for our customers.
When we receive this error, we initially check whether all the files that we present in the original location or not. If they are found missing, we debug the reason. Mostly, this happens when the source files are moved by scripts.
In the case of temporary files, we simply omit them from the backup process using the –exclude switch.
$ rsync -avz --exclude '/home/abc/public_html/*/tmp/' source/ destination/This command will exclude the folders and files from /home/abc/public_html/*/tmp/.
If we delete any of the source files, then we make sure to remove the files from the backup process. If the files got removed by mistake, we make sure to restore the file to the original location on the source and reattempt the Rsync.
As we already saw, Rsync finds it difficult to read the file name with special characters. So our Dedicated Engineers always recommend avoiding the use of special characters while creating files or directories.
In rare cases, where the error happens due to disk drive errors, we fix it by executing a file system check on the faulty drive too.
[Need more assistance to solve the rsync error 24?- We’re available 24/7 to help you.]
Conclusion
In short, the rsync error 24 occurs when there happens a partial transfer due to vanished source files. Also, today’s write-up discussed the main causes of this error and saw how our Support Engineers fix it for our customers.
PREVENT YOUR SERVER FROM CRASHING!
Never again lose customers to poor server speed! Let us help you.
Our server experts will monitor & maintain your server 24/7 so that it remains lightning fast and secure.
GET STARTED
var google_conversion_label = «owonCMyG5nEQ0aD71QM»;
По завершению симуляции резервного копирования через rsync:
rsync -r -n -t -v --progress -s / /home/wolf
Я получаю ошибку Rsync process exit status: 24 , а точнее :
file has vanished: "/proc/10/exe"
file has vanished: "/proc/10/task/10/exe"
file has vanished: "/proc/1042/exe"
file has vanished: "/proc/1042/task/1042/exe"
file has vanished: "/proc/106/exe"
file has vanished: "/proc/106/task/106/exe"
file has vanished: "/proc/108/exe"
file has vanished: "/proc/108/task/108/exe"
file has vanished: "/proc/11/exe"
file has vanished: "/proc/11/task/11/exe"
file has vanished: "/proc/12/exe"
file has vanished: "/proc/12/task/12/exe"
file has vanished: "/proc/13/exe"
file has vanished: "/proc/13/task/13/exe"
file has vanished: "/proc/14/exe"
file has vanished: "/proc/14/task/14/exe"
file has vanished: "/proc/15/exe"
file has vanished: "/proc/15/task/15/exe"
file has vanished: "/proc/16/exe"
file has vanished: "/proc/16/task/16/exe"
file has vanished: "/proc/1645/exe"
file has vanished: "/proc/1645/task/1645/exe"
file has vanished: "/proc/169/exe"
file has vanished: "/proc/169/task/169/exe"
file has vanished: "/proc/170/exe"
file has vanished: "/proc/170/task/170/exe"
file has vanished: "/proc/171/exe"
file has vanished: "/proc/171/task/171/exe"
file has vanished: "/proc/172/exe"
file has vanished: "/proc/172/task/172/exe"
file has vanished: "/proc/176/exe"
file has vanished: "/proc/176/task/176/exe"
file has vanished: "/proc/177/exe"
file has vanished: "/proc/177/task/177/exe"
file has vanished: "/proc/178/exe"
file has vanished: "/proc/178/task/178/exe"
file has vanished: "/proc/179/exe"
file has vanished: "/proc/179/task/179/exe"
file has vanished: "/proc/185/exe"
file has vanished: "/proc/185/task/185/exe"
file has vanished: "/proc/186/exe"
file has vanished: "/proc/186/task/186/exe"
file has vanished: "/proc/187/exe"
file has vanished: "/proc/187/task/187/exe"
file has vanished: "/proc/19/exe"
file has vanished: "/proc/19/task/19/exe"
directory has vanished: "/proc/1901/task/4179"
file has vanished: "/proc/2/exe"
file has vanished: "/proc/2/task/2/exe"
file has vanished: "/proc/20/exe"
file has vanished: "/proc/20/task/20/exe"
file has vanished: "/proc/21/exe"
file has vanished: "/proc/21/task/21/exe"
file has vanished: "/proc/22/exe"
file has vanished: "/proc/22/task/22/exe"
file has vanished: "/proc/226/exe"
file has vanished: "/proc/226/task/226/exe"
file has vanished: "/proc/2261/exe"
file has vanished: "/proc/2261/task/2261/exe"
file has vanished: "/proc/228/exe"
file has vanished: "/proc/228/task/228/exe"
file has vanished: "/proc/229/exe"
file has vanished: "/proc/229/task/229/exe"
file has vanished: "/proc/24/exe"
file has vanished: "/proc/24/task/24/exe"
file has vanished: "/proc/2497/exe"
file has vanished: "/proc/2497/task/2497/exe"
file has vanished: "/proc/25/exe"
file has vanished: "/proc/25/task/25/exe"
file has vanished: "/proc/26/exe"
file has vanished: "/proc/26/task/26/exe"
file has vanished: "/proc/27/exe"
file has vanished: "/proc/27/task/27/exe"
file has vanished: "/proc/273/exe"
file has vanished: "/proc/273/task/273/exe"
file has vanished: "/proc/274/exe"
file has vanished: "/proc/274/task/274/exe"
file has vanished: "/proc/2787/exe"
file has vanished: "/proc/2787/task/2787/exe"
file has vanished: "/proc/28/exe"
file has vanished: "/proc/28/task/28/exe"
file has vanished: "/proc/2838/exe"
file has vanished: "/proc/2838/task/2838/exe"
file has vanished: "/proc/2853/exe"
file has vanished: "/proc/2853/task/2853/exe"
file has vanished: "/proc/2861/exe"
file has vanished: "/proc/2861/task/2861/exe"
file has vanished: "/proc/30/exe"
file has vanished: "/proc/30/task/30/exe"
file has vanished: "/proc/31/exe"
file has vanished: "/proc/31/task/31/exe"
file has vanished: "/proc/313/exe"
file has vanished: "/proc/313/task/313/exe"
file has vanished: "/proc/32/exe"
file has vanished: "/proc/32/task/32/exe"
file has vanished: "/proc/33/exe"
file has vanished: "/proc/33/task/33/exe"
file has vanished: "/proc/333/exe"
file has vanished: "/proc/333/task/333/exe"
file has vanished: "/proc/36/exe"
file has vanished: "/proc/36/task/36/exe"
file has vanished: "/proc/37/exe"
file has vanished: "/proc/37/task/37/exe"
file has vanished: "/proc/38/exe"
file has vanished: "/proc/38/task/38/exe"
file has vanished: "/proc/3883/exe"
file has vanished: "/proc/3883/task/3883/exe"
file has vanished: "/proc/3886/exe"
file has vanished: "/proc/3886/task/3886/exe"
file has vanished: "/proc/39/exe"
file has vanished: "/proc/39/task/39/exe"
file has vanished: "/proc/4/exe"
file has vanished: "/proc/4/task/4/exe"
file has vanished: "/proc/40/exe"
file has vanished: "/proc/40/task/40/exe"
file has vanished: "/proc/41/exe"
file has vanished: "/proc/41/task/41/exe"
file has vanished: "/proc/42/exe"
file has vanished: "/proc/42/task/42/exe"
file has vanished: "/proc/43/exe"
file has vanished: "/proc/43/task/43/exe"
file has vanished: "/proc/44/exe"
file has vanished: "/proc/44/task/44/exe"
file has vanished: "/proc/446/exe"
file has vanished: "/proc/446/task/446/exe"
file has vanished: "/proc/46/exe"
file has vanished: "/proc/46/task/46/exe"
file has vanished: "/proc/47/exe"
file has vanished: "/proc/47/task/47/exe"
file has vanished: "/proc/48/exe"
file has vanished: "/proc/48/task/48/exe"
file has vanished: "/proc/482/exe"
file has vanished: "/proc/482/task/482/exe"
file has vanished: "/proc/483/exe"
file has vanished: "/proc/483/task/483/exe"
file has vanished: "/proc/49/exe"
file has vanished: "/proc/49/task/49/exe"
file has vanished: "/proc/50/exe"
file has vanished: "/proc/50/task/50/exe"
file has vanished: "/proc/6/exe"
file has vanished: "/proc/6/task/6/exe"
file has vanished: "/proc/64/exe"
file has vanished: "/proc/64/task/64/exe"
file has vanished: "/proc/65/exe"
file has vanished: "/proc/65/task/65/exe"
file has vanished: "/proc/7/exe"
file has vanished: "/proc/7/task/7/exe"
file has vanished: "/proc/8/exe"
file has vanished: "/proc/8/task/8/exe"
file has vanished: "/proc/9/exe"
file has vanished: "/proc/9/task/9/exe"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1196) [sender=3.1.2]
Rsync process exit status: 24
Вопросы :
1. Почему это происходит ?
2. Стоит ли на эту ошибку реагировать или можно продолжить бэкапить ?
3. Исходя из второго пункта, можно ли такой бэкап использовать для восстановления системы ?
задан 12 ноя 2017 в 22:55
1
Файлы в каталогах /proc и /sys не являются нормальными файлами, они виртуальные, представляющие собой интерфейс с ядром. В связи с этим они могут изменяться при каждом чтении. У rsync’а едет крыша, когда он видит, что файлы под ним меняются.
Просто исключите каталоги /proc и /sys из бекапа.
ответ дан 12 ноя 2017 в 23:06
MikeMike
43.8k3 золотых знака33 серебряных знака66 бронзовых знаков
0
- Forum
- The Ubuntu Forum Community
- Ubuntu Official Flavours Support
- General Help
- [SOLVED] rsync: Files vanished
-
rsync: Files vanished
Hi, I do a daily backup at my office from a Ubuntu USB live session, basically I rsync the contents of one internal drive to another internal drive and then to an external drive, something like this:
Code:
rsync -av --delete /media/ubuntu/drive1/ /media/ubuntu/drive2 rsync -av --delete /media/ubuntu/drive1/ /media/ubuntu/ext_drive
transferring to the the second internal drive is always fine (the first line), but transferring to the external drive always gives me a «some files have vanished» error.
I checked all the drives for errors and there are no errors, so I don’t know what to do next.
Any help will be greatly appreciated.Thanks
-
Re: rsync: Files vanished
Are there Linux file systems on all those partitions? Is the external drive always properly unmounted before it’s unplugged?
Also note that this is not a particularly safe backup strategy. The most likely reason why you may want to use your backups is accidental deletion or corruption of a single file. You destroy those old backups after a single day, which may be before you notice you need them. Better to have versioned backups and keep them at least a few weeks.
-
Re: rsync: Files vanished
None of those partitions are Linux filesystems, they are NTFS.
I always unmount the external drive before unplugging.I have never encountered a situation in which I needed a file that I deleted from the main drive after more than one day; but it certainly can happen. I’ll consider your advice seriously, I think it’s a good strategy. Thanks
-
Re: rsync: Files vanished
Beware of —delete. It is dangerous. The —delete-after option is a little safer.
When you have to use the delete option always use —max-delete=NUM to give it a limit so that it won’t delete more than 10 or so files to prevent a disaster.
-
Re: rsync: Files vanished
Not sure if rsync logs by default but you could use the —log-file=path_to_log option then grep the file for «file has vanished:». Eg,
Code:
rsync -av --delete --log-file=path_to_write_log_to/rsync.log /media/ubuntu/drive1/ /media/ubuntu/ext_drive
Code:
grep -ir "file has vanished:" /path_to_logged_file/rsync.log
-
Re: rsync: Files vanished
Originally Posted by HermanAB
Beware of —delete. It is dangerous. The —delete-after option is a little safer.
When you have to use the delete option always use —max-delete=NUM to give it a limit so that it won’t delete more than 10 or so files to prevent a disaster.
We create and receive from outside sources a lot of very large Photoshop files that are temporary in nature. I also receive a lot of compressed files and I have to delete them (the compressed version) when I unzip the files and I don’t need the .zip anymore, otherwise the hard drives would be full of files that are completely useless or repeated. If I don’t use —delete the files are going to get transferred to the backup drives and I don’t want them there. I can keep them for a while, as suggested above, but after some time I have to delete them.
-
Re: rsync: Files vanished
Then make the max_delete larger — but do use it and set it to a reasonable average number so that the deletions will eventually catch up, but protect you against a big disaster. Otherwise if you accidentally delete a directory with a bazilion files, then all those files will also disappear from your backup before you may be able to realize what happened and recover from the backup.
(Long ago, when I supported some lawyers, it happened multiple times that a secretary would accidentally delete tens of thousands of files and on a Linux server, that happens in the blink of an eye.)
Last edited by HermanAB; February 6th, 2017 at 09:00 PM.
-
Re: rsync: Files vanished
-
Re: rsync: Files vanished
Originally Posted by leunam12
We create and receive from outside sources a lot of very large Photoshop files that are temporary in nature. I also receive a lot of compressed files and I have to delete them (the compressed version) when I unzip the files and I don’t need the .zip anymore
Use include/exclude to limit what you backup.
Code:
rsync -av source target --exclude=*.zip --delete-excluded
would backup everything except ZIP files which will be deleted. You can have multiple —exclude statements or use —exclude-from with a list of file patterns.
-
Re: rsync: Files vanished
Ok, I changed the script and I removed the —delete part, and I also decided to —exclude the «Thumbs.db» files that Windows puts everywhere, I don’t need those files. Now the problem seems to be gone.
Thanks everyone, the problem has vanished!
Originally Posted by HermanAB
Bookmarks
Bookmarks
Posting Permissions
I’m in the process of moving files from an Ubuntu server to a Snow Leopard server. The Ubuntu server has an NFS share of around 6TB which I’d like to clone to the Snow Leopard server.
I mounted the nfs share on the Snow Leopard server and then ran
rsync -av /Volumes/FromUbuntu /Volumes/LocalCopy
After it copies around 100GB it complains that files have vanished. I’m assuming that for some reason the NFS links are going stale? What could be causing this? The Ubuntu server is not crashing and there are not connectivity issues that I am aware of. I wouldn’t mind running to the rsync command over and over until the copy was finished, but it takes about 6 hours just for rsync to make it’s file list.
Any suggestions? Would it be faster to just rsync over SSH? Thanks!
(PS: I’ve tried just using ‘cp -arv’ which doesn’t seem to fail but according to the network traffic monitor on the Mac it seems to take twice as long to copy files as rsync after rsync has built its file list?)
UPDATE: I’m trying rsync over ssh from the mac server to the ubuntu server and it seems to been going much faster (it took less than two hours to create a file list and start transferring, where as when rsyncing from the nfs mount took around six hours just to build the initial file list.) It definitely looks like there is some problem with mounting the NFS share on the mac. Has anyone had problems with this? What about the other way around, mounting nfs shares on the mac on an ubuntu client? I was planning on hosting an NFS share for Ubuntu clients on the Mac, but now I’m getting nervous. Thanks for your input!