@ET-DC@eyJkeW5hbWljIjp0cnVlLCJjb250ZW50IjoicG9zdF90aXRsZSIsInNldHRpbmdzIjp7ImJlZm9yZSI6IiIsImFmdGVyIjoiIn19@[addtoany buttons="twitter", "linkedin"]
In this post I am going to describe a way to build a simple incremental backup solution using rsync and hard links. You may already be familiar with rsync but for anyone who is not, rsync is a command-line tool commonly used on Linux and other UNIX-like operating systems to copy and synchronise directories. I will assume some prior knowledge of rsync in this post so if you have not used it before there may be some parts that confuse you!
A bit of background
Before we go into the details you should understand how files are stored on the filesystem and how hard links work.All files and directories are represented in the filesystem by an inode number which is the filesystem's internal identity for the file. If you run ls -li in a directory you can see the inode numbers listed on the left:W3VzZXIxQGJhY2t1cGJveCBkaXIxXSQgbHMgLWxpCnRvdGFsIDEyOAozMzgzOTAwMiAtcnctcnctci0tLiAxIHVzZXIxIHVzZXIxIDEyOTQyIE9jdCAgMiAxNjoxNCBmaWxlMQozMzgzOTAwMyAtcnctcnctci0tLiAxIHVzZXIxIHVzZXIxIDE0MTA2IE9jdCAgMiAxNjoxNCBmaWxlMgozMzgzOTAwNCAtcnctcnctci0tLiAxIHVzZXIxIHVzZXIxIDE5MzYwIE9jdCAgMiAxNjoxNCBmaWxlMwozMzgzOTAwNSAtcnctcnctci0tLiAxIHVzZXIxIHVzZXIxIDE3MDkzIE9jdCAgMiAxNjoxNCBmaWxlNAozMzgzOTAwNiAtcnctcnctci0tLiAxIHVzZXIxIHVzZXIxIDE2MDk0IE9jdCAgMiAxNjoxNCBmaWxlNQ==A "file" as we see it by path and filename is in fact a reference to the inode and is often referred to as a "link". When you create a hard link from one file to another you are creating a separate reference (link) from a new filename to the same inode number. This is different from a “soft” or “symbolic” link (symlink) which is a reference from one location to another path in the filesystem. You can see the difference in the output of ls -li:W3VzZXIxQGJhY2t1cGJveCBkaXIxXSQgbHMgLWxpCnRvdGFsIDY0CjMzODM5MDAyIC1ydy1yLS1yLS0uIDIgdXNlcjEgdXNlcjEgMTI5NDIgT2N0ICAyIDE2OjE0IGZpbGUxCjMzODM5MDAzIC1ydy1yLS1yLS0uIDIgdXNlcjEgdXNlcjEgMTQxMDYgT2N0ICAyIDE2OjE0IGZpbGUyCjMzODM5MDAyIC1ydy1yLS1yLS0uIDIgdXNlcjEgdXNlcjEgMTI5NDIgT2N0ICAyIDE2OjE0IGhhcmRsaW5rMQozMzgzOTAwMyAtcnctci0tci0tLiAyIHVzZXIxIHVzZXIxIDE0MTA2IE9jdCAgMiAxNjoxNCBoYXJkbGluazIKMzM2OTU3NjAgbHJ3eHJ3eHJ3eC4gMSB1c2VyMSB1c2VyMSAgICAgNSBPY3QgIDIgMTY6MTUgc3ltbGluazEgLT4gZmlsZTEKMzM2OTU3NjIgbHJ3eHJ3eHJ3eC4gMSB1c2VyMSB1c2VyMSAgICAgNSBPY3QgIDIgMTY6MTUgc3ltbGluazIgLT4gZmlsZTIKWhen you edit the original file the changes are also visible in the hard-linked version:W3VzZXIxQGJhY2t1cGJveCBkaXIxXSQgbHMgLWxpCnRvdGFsIDgKMzM4MzkwMDIgLXJ3LXItLXItLS4gMiB1c2VyMSB1c2VyMSA0NyBPY3QgIDIgMTY6MTkgZmlsZTEKMzM4MzkwMDIgLXJ3LXItLXItLS4gMiB1c2VyMSB1c2VyMSA0NyBPY3QgIDIgMTY6MTkgaGFyZGxpbmsxClt1c2VyMUBiYWNrdXBib3ggZGlyMV0kIGNhdCBmaWxlMQpUaGlzIGlzIGZpbGUxClt1c2VyMUBiYWNrdXBib3ggZGlyMV0kIGNhdCBoYXJkbGluazEKVGhpcyBpcyBmaWxlMQpbdXNlcjFAYmFja3VwYm94IGRpcjFdJCBlY2hvICJhbiBleHRyYSBsaW5lIiA+PmZpbGUxClt1c2VyMUBiYWNrdXBib3ggZGlyMV0kIGNhdCBmaWxlMQpUaGlzIGlzIGZpbGUxCmFuIGV4dHJhIGxpbmUKW3VzZXIxQGJhY2t1cGJveCBkaXIxXSQgY2F0IGhhcmRsaW5rMQpUaGlzIGlzIGZpbGUxCmFuIGV4dHJhIGxpbmU=And if you edit the hard-linked file the changes are seen in the original file:W3VzZXIxQGJhY2t1cGJveCBkaXIxXSQgZWNobyAiYW5vdGhlciBleHRyYSBsaW5lIiA+PmhhcmRsaW5rMQpbdXNlcjFAYmFja3VwYm94IGRpcjFdJCBjYXQgZmlsZTEKVGhpcyBpcyBmaWxlMQphbiBleHRyYSBsaW5lCmFub3RoZXIgZXh0cmEgbGluZQpbdXNlcjFAYmFja3VwYm94IGRpcjFdJCBjYXQgaGFyZGxpbmsxClRoaXMgaXMgZmlsZTEKYW4gZXh0cmEgbGluZQphbm90aGVyIGV4dHJhIGxpbmU=Changing the ownership and permissions also affects both files:W3VzZXIxQGJhY2t1cGJveCBkaXIxXSQgbHMgLWxpCnRvdGFsIDgKMzM4MzkwMDIgLXJ3LXItLXItLS4gMiB1c2VyMSB1c2VyMSA0NyBPY3QgIDIgMTY6MTkgZmlsZTEKMzM4MzkwMDIgLXJ3LXItLXItLS4gMiB1c2VyMSB1c2VyMSA0NyBPY3QgIDIgMTY6MTkgaGFyZGxpbmsxClt1c2VyMUBiYWNrdXBib3ggZGlyMV0kIHN1ZG8gY2hvd24gcm9vdC5yb290IGZpbGUxClt1c2VyMUBiYWNrdXBib3ggZGlyMV0kIGxzIC1saQp0b3RhbCA4CjMzODM5MDAyIC1ydy1yLS1yLS0uIDIgcm9vdCAgcm9vdCAgNDcgT2N0ICAyIDE2OjE5IGZpbGUxCjMzODM5MDAyIC1ydy1yLS1yLS0uIDIgcm9vdCAgcm9vdCAgNDcgT2N0ICAyIDE2OjE5IGhhcmRsaW5rMQpbdXNlcjFAYmFja3VwYm94IGRpcjFdJCBzdWRvIGNobW9kIDA2NjYgaGFyZGxpbmsxClt1c2VyMUBiYWNrdXBib3ggZGlyMV0kIGxzIC1saQp0b3RhbCA4CjMzODM5MDAyIC1ydy1ydy1ydy0uIDIgcm9vdCAgcm9vdCAgNDcgT2N0ICAyIDE2OjE5IGZpbGUxCjMzODM5MDAyIC1ydy1ydy1ydy0uIDIgcm9vdCAgcm9vdCAgNDcgT2N0ICAyIDE2OjE5IGhhcmRsaW5rMQ==Now if we delete the original file we will see that the hard link still exists and the file content remains intact. In contrast a symlink pointing to the original file will no longer be valid:W3VzZXIxQGJhY2t1cGJveCBkaXIxXSQgbHMgLWxpCnRvdGFsIDgKMzM4MzkwMDIgLXJ3LXItLXItLS4gMiB1c2VyMSB1c2VyMSA0NyBPY3QgIDIgMTY6MTkgZmlsZTEKMzM4MzkwMDIgLXJ3LXItLXItLS4gMiB1c2VyMSB1c2VyMSA0NyBPY3QgIDIgMTY6MTkgaGFyZGxpbmsxCjMzNjk1NzYwIGxyd3hyd3hyd3guIDEgdXNlcjEgdXNlcjEgIDUgT2N0ICAyIDE2OjE1IHN5bWxpbmsxIC0+IGZpbGUxClt1c2VyMUBiYWNrdXBib3ggZGlyMV0kIHJtIC1mIGZpbGUxClt1c2VyMUBiYWNrdXBib3ggZGlyMV0kIGxzIC1saQp0b3RhbCA0CjMzODM5MDAyIC1ydy1yLS1yLS0uIDEgdXNlcjEgdXNlcjEgNDcgT2N0ICAyIDE2OjE5IGhhcmRsaW5rMQozMzY5NTc2MCBscnd4cnd4cnd4LiAxIHVzZXIxIHVzZXIxICA1IE9jdCAgMiAxNjoxNSBzeW1saW5rMSAtPiBmaWxlMQpbdXNlcjFAYmFja3VwYm94IGRpcjFdJCBjYXQgaGFyZGxpbmsxClRoaXMgaXMgZmlsZTEKYW4gZXh0cmEgbGluZQphbm90aGVyIGV4dHJhIGxpbmUKW3VzZXIxQGJhY2t1cGJveCBkaXIxXSQgY2F0IHN5bWxpbmsxCmNhdDogc3ltbGluazE6IE5vIHN1Y2ggZmlsZSBvciBkaXJlY3Rvcnk=We can even create another hard link and delete the existing one and the data still remains intact:W3VzZXIxQGJhY2t1cGJveCBkaXIxXSQgbHMgLWxpCnRvdGFsIDQKMzM4MzkwMDIgLXJ3LXItLXItLS4gMSB1c2VyMSB1c2VyMSA0NyBPY3QgIDIgMTY6MTkgaGFyZGxpbmsxClt1c2VyMUBiYWNrdXBib3ggZGlyMV0kIGxuIGhhcmRsaW5rMSBuZXdsaW5rMQpbdXNlcjFAYmFja3VwYm94IGRpcjFdJCBscyAtbGkKdG90YWwgOAozMzgzOTAwMiAtcnctci0tci0tLiAyIHVzZXIxIHVzZXIxIDQ3IE9jdCAgMiAxNjoxOSBoYXJkbGluazEKMzM4MzkwMDIgLXJ3LXItLXItLS4gMiB1c2VyMSB1c2VyMSA0NyBPY3QgIDIgMTY6MTkgbmV3bGluazEKW3VzZXIxQGJhY2t1cGJveCBkaXIxXSQgcm0gaGFyZGxpbmsxClt1c2VyMUBiYWNrdXBib3ggZGlyMV0kIGxzIC1saQp0b3RhbCA0CjMzODM5MDAyIC1ydy1yLS1yLS0uIDEgdXNlcjEgdXNlcjEgNDcgT2N0ICAyIDE2OjE5IG5ld2xpbmsxClt1c2VyMUBiYWNrdXBib3ggZGlyMV0kIGNhdCBuZXdsaW5rMQpUaGlzIGlzIGZpbGUxCmFuIGV4dHJhIGxpbmUKYW5vdGhlciBleHRyYSBsaW5lWhen you delete a file using the rm command, or any other method, what you are actually doing is just removing the link to the inode. This is why the function to delete a file in languages such as C and PHP is called "unlink". When all links to an inode have been removed the inode itself will be deleted. As long as there is at least one link pointing to it the inode and the data will remain intact.
So what does this have to do with rsync and incremental backups?
Let's say we want to create a mirror of a remote directory /home/data from a server named server1 into a local directory /backup/server1. Typically we would do something like this:cnN5bmMgLWF2IC0tZGVsZXRlIHNlcnZlcjE6L2hvbWUvZGF0YS8gL2JhY2t1cC9zZXJ2ZXIxLw==
We would then run the same command again each time we wanted to update the mirror with the latest changes from the server.
To implement a basic incremental backup system we might consider making a local copy of the previous backup before starting the rsync:
W3VzZXIxQGJhY2t1cGJveCBkaXIxXSQgY3AgLWEgL2JhY2t1cC9zZXJ2ZXIxLyAvYmFja3VwL3NlcnZlcjFPbGQv
Then we update our mirror from the remote server:
W3VzZXIxQGJhY2t1cGJveCBkaXIxXSQgcnN5bmMgLWF2IC0tZGVsZXRlIHNlcnZlcjE6L2hvbWUvZGF0YS8gL2JhY2t1cC9zZXJ2ZXIxLw==
Obviously this isn't very efficient in either time or space so we could improve this by using hard links instead, which can be done by adding the -l argument to the cp command:
IyBDcmVhdGUgYSBoYXJkLWxpbmtlZCBjbG9uZSBvZiB0aGUgY3VycmVudCBiYWNrdXAKY3AgLWFsIC9iYWNrdXAvc2VydmVyMSAvYmFja3VwL3NlcnZlcjFPbGQKIyB1cGRhdGUgb3VyIG1pcnJvciBmcm9tIHRoZSByZW1vdGUgc2VydmVyCnJzeW5jIC1hdiAtLWRlbGV0ZSBzZXJ2ZXIxOi9ob21lL2RhdGEvIC9iYWNrdXAvc2VydmVyMS8=This previous backup is preserved in /backup/server1Old and /backup/server1 will contain the entire new backup and only uses the space required for the new and changed files. This creates an efficient way to implement incremental backups, however it still has its limitations especially when dealing with large numbers of files.To improve things further we can use a feature in rsync which enables us to efficiently create hard-linked copies of a directory’s contents with only the changed files taking up space on disk. The rsync feature we need is the --link-dest argument.Taking this as a starting point:server1:/home/data: Remote source directory/backup/server1New: Destination for a new backup. Does not yet exist/backup/server1Old: Existing previous backupThe result we want in /backup/server1New is that all unchanged files are hard links to the existing files in /backup/server1Old and only the changed files are copied from the remote server and take up space in the new backup.This is exactly what the --link-dest argument does for us. It performs a normal rsync from server1:/home/data to /backup/server1New but if the file does not exist in /backup/server1New it will look at the same relative path under /backup/server1Old to see if the file has changed. If the file in /backup/server1Old is the same as the file on the remote server then instead of copying it over rsync will create a hard link from the file in /backup/server1Old into /backup/server1New.To use this we just add the "old" directory as the --link-dest argument to our rsync command:cnN5bmMgLWF2IC0tbGluay1kZXN0IC9iYWNrdXAvc2VydmVyMU9sZCBzZXJ2ZXIxOi9ob21lL2RhdGEvIC9iYWNrdXAvc2VydmVyMU5ldy8=
Here we can see the old backup directory’s contents:
W3VzZXIxQGJhY2t1cGJveCB+XSQgbHMgLWxSaSAvYmFja3VwL3NlcnZlcjFPbGQvCi9iYWNrdXAvc2VydmVyMU9sZC86CnRvdGFsIDAKNjg4NzYgZHJ3eHJ3eHIteC4gMyB1c2VyMSB1c2VyMSA1MyBPY3QgIDIgMTc6MzAgZmlsZXMKIAovYmFja3VwL3NlcnZlcjFPbGQvZmlsZXM6CnRvdGFsIDcyCjMzNjUxOTM1IGRyd3hyd3hyLXguIDIgdXNlcjEgdXNlcjEgICAgNDIgT2N0ICAyIDE3OjMwIGJhcgogICA2ODg4MiAtcnctcnctci0tLiAxIHVzZXIxIHVzZXIxIDI4ODgzIE9jdCAgMiAxNzozMCBmb28xCiAgIDY4ODgzIC1ydy1ydy1yLS0uIDEgdXNlcjEgdXNlcjEgMjc3NjMgT2N0ICAyIDE3OjMwIGZvbzIKICAgNjg4ODQgLXJ3LXJ3LXItLS4gMSB1c2VyMSB1c2VyMSAxMDQ4NyBPY3QgIDIgMTc6MzAgZm9vMwogCi9iYWNrdXAvc2VydmVyMU9sZC9maWxlcy9iYXI6CnRvdGFsIDc2CjMzNjk1NzU5IC1ydy1ydy1yLS0uIDEgdXNlcjEgdXNlcjEgMzI2MDMgT2N0ICAyIDE3OjMwIGJhcjEKMzM4Mzg5ODQgLXJ3LXJ3LXItLS4gMSB1c2VyMSB1c2VyMSAxNTMxOCBPY3QgIDIgMTc6MzAgYmFyMgozMzgzOTAwMyAtcnctcnctci0tLiAxIHVzZXIxIHVzZXIxIDI2MTIyIE9jdCAgMiAxNzozMCBiYXIz
On the server we then modify a file:
W3VzZXIxQHNlcnZlcjEgZmlsZXNdJCBlY2hvICJIZWxsbyB3b3JsZCIgPi9ob21lL2RhdGEvZmlsZXMvZm9vMw==
Now we run our incremental backup command:
W3VzZXIxQGJhY2t1cGJveCB+XSQgcnN5bmMgLWF2IC0tbGluay1kZXN0PS9iYWNrdXAvc2VydmVyMU9sZCBzZXJ2ZXIxOi9ob21lL2RhdGEvIC9iYWNrdXAvc2VydmVyMU5ldy8KcmVjZWl2aW5nIGluY3JlbWVudGFsIGZpbGUgbGlzdApjcmVhdGVkIGRpcmVjdG9yeSAvYmFja3VwL3NlcnZlcjFOZXcKZmlsZXMvZm9vMwogCnNlbnQgMTM2IGJ5dGVzICByZWNlaXZlZCAyNzIgYnl0ZXMgIDgxNi4wMCBieXRlcy9zZWMKdG90YWwgc2l6ZSBpcyAxMzAsNzAxICBzcGVlZHVwIGlzIDMyMC4zNQ==
We can see from the rsync output that only the changed file has been copied but if we list the contents of the new directory we can see it contains all of the files:
W3VzZXIxQGJhY2t1cGJveCB+XSQgbHMgLWxSaSAvYmFja3VwL3NlcnZlcjFOZXcvCi9iYWNrdXAvc2VydmVyMU5ldy86CnRvdGFsIDAKMTAxMDUxNDYwIGRyd3hyd3hyLXguIDMgdXNlcjEgdXNlcjEgNTMgT2N0ICAyIDE3OjMwIGZpbGVzCiAKL2JhY2t1cC9zZXJ2ZXIxTmV3L2ZpbGVzOgp0b3RhbCA2NAogICAgNjg4ODUgZHJ3eHJ3eHIteC4gMiB1c2VyMSB1c2VyMSAgICA0MiBPY3QgIDIgMTc6MzAgYmFyCiAgICA2ODg4MiAtcnctcnctci0tLiAyIHVzZXIxIHVzZXIxIDI4ODgzIE9jdCAgMiAxNzozMCBmb28xCiAgICA2ODg4MyAtcnctcnctci0tLiAyIHVzZXIxIHVzZXIxIDI3NzYzIE9jdCAgMiAxNzozMCBmb28yCjEwMTA1MTQ2MSAtcnctcnctci0tLiAxIHVzZXIxIHVzZXIxICAgIDEyIE9jdCAgMiAxNzo0MCBmb28zCiAKL2JhY2t1cC9zZXJ2ZXIxTmV3L2ZpbGVzL2JhcjoKdG90YWwgNzYKMzM2OTU3NTkgLXJ3LXJ3LXItLS4gMiB1c2VyMSB1c2VyMSAzMjYwMyBPY3QgIDIgMTc6MzAgYmFyMQozMzgzODk4NCAtcnctcnctci0tLiAyIHVzZXIxIHVzZXIxIDE1MzE4IE9jdCAgMiAxNzozMCBiYXIyCjMzODM5MDAzIC1ydy1ydy1yLS0uIDIgdXNlcjEgdXNlcjEgMjYxMjIgT2N0ICAyIDE3OjMwIGJhcjM=If you compare the inode numbers to the listing of /backup/server1Old above you will see that only the modified file and the directories have different inode numbers.Using du we can also see that the second backup takes up less space on disk:W3VzZXIxQGJhY2t1cGJveCB+XSQgZHUgLWNocyAvYmFja3VwL3NlcnZlcjEqCjE0MEsJL2JhY2t1cC9zZXJ2ZXIxTmV3CjEySwkvYmFja3VwL3NlcnZlcjFPbGQKMTUySwl0b3RhbA==
Putting it all together
Here is an example script that can be used to create daily incremental backups of a directory. Each backup is stored in a directory named after today’s date and it will look for yesterday’s backup to create the hard links:
IyEvYmluL2Jhc2gKIAojIFRoZSBzb3VyY2UgcGF0aCB0byBiYWNrdXAuIENhbiBiZSBsb2NhbCBvciByZW1vdGUuClNPVVJDRT1zZXJ2ZXJuYW1lOi9zb3VyY2UvZGlyLwojIFdoZXJlIHRvIHN0b3JlIHRoZSBpbmNyZW1lbnRhbCBiYWNrdXBzCkRFU1RCQVNFPS9iYWNrdXAvc2VydmVybmFtZV9kYXRhCiAKIyBXaGVyZSB0byBzdG9yZSB0b2RheSdzIGJhY2t1cApERVNUPSIkREVTVEJBU0UvJChkYXRlICslWS0lbS0lZCkiCiMgV2hlcmUgdG8gZmluZCB5ZXN0ZXJkYXkncyBiYWNrdXAKWUVTVEVSREFZPSIkREVTVEJBU0UvJChkYXRlIC1kIHllc3RlcmRheSArJVktJW0tJWQpLyIKIAojIFVzZSB5ZXN0ZXJkYXkncyBiYWNrdXAgYXMgdGhlIGluY3JlbWVudGFsIGJhc2UgaWYgaXQgZXhpc3RzCmlmIFsgLWQgIiRZRVNURVJEQVkiIF0KdGhlbgoJT1BUUz0iLS1saW5rLWRlc3QgJFlFU1RFUkRBWSIKZmkKIAojIFJ1biB0aGUgcnN5bmMKcnN5bmMgLWF2ICRPUFRTICIkU09VUkNFIiAiJERFU1QiThe beauty of doing your backups this way is that each daily backup is a full mirror of the remote directory. This means there is no complex logic required to find the latest version of a file or to find a file from a specific date, just go to the directory named with the date you want and open the file as normal. Each backup directory is completely independent of the others so if you need to free up some space you can just delete any of the backups that you no longer require. Removing a backup will not impact the backups before or after, a simple rm -rf is all you need!
Limitations
As with every backup solution this one has its limitations and you must choose a method that fits your particular use-case. Here are a few examples of limitations in this solution:
- Changes in permissions or ownership on a source file mean the file is counted as a new file so it will be copied again even if its contents have not changed. There are options in rsync to control this behaviour.
- If you move or rename a file on the source server it will count as a new file and will be copied in full even if its contents have not changed and it still has the same inode number.
- Directories themselves cannot be hard linked on most filesystems so this is not supported by rsync. For most use cases this is not a problem but if you have an enormous number of directories in the backup they will start to take a noticeable amount of space on the backup disk.
Conclusion
When it comes to using rsync for backups this is only the tip of the iceberg. There are many different options that control the behaviour of the backup process and how it determines what files to copy, link or delete. Further information about rsync can be found on their website, https://rsync.samba.org/.@ET-DC@eyJkeW5hbWljIjp0cnVlLCJjb250ZW50IjoicG9zdF9hdXRob3JfYmlvIiwic2V0dGluZ3MiOnsiYmVmb3JlIjoiUmljaGFyZCBoYXMgYSB2YXJpZWQgaGlzdG9yeSBpbiBkZXZlbG9wbWVudCwgZGV2b3BzIGFuZCBkYXRhYmFzZXMgc28gaGUgaXMgYWx3YXlzIGNvbWZvcnRhYmxlIG9uIGVpdGhlciBzaWRlIG9mIHRoZSBkZXYvb3BzIGZlbmNlLiBIaXMgcGFzdCBleHBlcmllbmNlIGluY2x1ZGVzIHdlYiBhbmQgZW1haWwgaG9zdGluZywgc29mdHdhcmUgdGVzdGluZywgYnVpbGRpbmcgZGVza3RvcCBhbmQgbW9iaWxlIGFwcHMsIG1hbmFnaW5nIGxhcmdlIENhc3NhbmRyYSBjbHVzdGVycywgYnVpbGRpbmcgYW5kIHJ1bm5pbmcgbGFyZ2Utc2NhbGUgZGlzdHJpYnV0ZWQgYXBwbGljYXRpb25zIGFuZCBtb3JlLiIsImFmdGVyIjoiIn19@
