![\"Unison\"](\"https:\/\/www.cis.upenn.edu\/~bcpierce\/unison\/Unison.gif\")
\nA Bi-directional File Synchronizer.<\/p>\n
\nA Bi-directional File Synchronizer.<\/p>\n
I tend to live on two virtual locations: My MacBook Pro running OS X and my main server KUMR.LNS.COM running Linux. \u00a0I should say, for decades before I switched to Linux, I ran various versions of *BSD on KUMR.<\/p>\n
I have a home directory on both boxes with a subfolder called “projects” that has various things I have been working on for the last 30 or so years. \u00a0I want these directories and files in both locations for access and this gives me some semblance of a backup as well. \u00a0(Of course I have other backup methods including TimeMachine and other off-site backups.)<\/p>\n
Additionally, if I am doing development, I will tend to use my Mac but OS X does have some peculiarities in how various packages like Python, etc. get installed and running package managers like home-brew may not load what I need for an environment that will be deployed on some server so I will do that work on KUMR. \u00a0(Ya… I know about containers, and VMs.).<\/p>\n
The challenge is how to keep things in sync with each other. \u00a0For quite a while I have been using Unison<\/a>, a file synchronizer that is bi-directional and uses rsync’s rather efficient method of file transfer. \u00a0I will skip describing the rsync protocol, but you can check out the paper at https:\/\/rsync.samba.org\/how-rsync-works.html<\/a>\u00a0for the details.<\/p>\n Unison is extremely efficent in working through large file collections. \u00a0I currently have about\u00a0305 GB with\u00a0215,450 files and\u00a027,888 directories just in my “project” folder. \u00a0If I was just using “rsync”, it would take a large amount of time in walking through each file, computing the hash, seeing if the hash was the same on the other server and starting up a transfer if it isn’t. \u00a0Unison will make a similar crawl of all the files once and then keep track of files via a hash of the file and put it into an archive in ~\/.unison directory. \u00a0 \u00a0This means that the first time I run Unison it may take a hours to crawl through all the files, but subsequent runs may take less than a minute to scan and transfer, depending on what was changed last.<\/p>\n If you are worried that Unison is missing anything with this system, just go into the .unison directory and delete the archives in both the local an remote servers. \u00a0Normally they start with “ar*” or “fp*” and then run unison again.<\/p>\n Unison also knows when you have just moved a file or folder with the same material. \u00a0If it sees a new file with the same hash and name of an existing file, it just moves that file as a “shortcut”. \u00a0A big win in just moving a folder with a large number of files or large files.<\/p>\n Since I have been using Unison for a while, I have had some tweaks to the Unison configuration file (~\/.unison\/default.prf). \u00a0I thought I would share mine here with some comments that detail the config file itself. \u00a0This, by no means is a complete set of options for Unison. \u00a0You can see all of it detailed in the manual<\/a>, of which I would highly suggest reviewing.<\/p>\n There is a handy command line argument called “-batch” which will avoid asking about what to do with each file it found to sync. \u00a0Normally it will figure things out by looking at the date stamp. \u00a0In some cases, it won’t know that to do. \u00a0You can see below an example where the permissions or times of a file may be in conflict with various Django files I have. \u00a0In this case I want to propagate this meta data to my remote server “KUMR”. \u00a0I would normally use the “>” key to tell it to go from left (Local) to right (KUMR)…<\/p>\n With the “-batch” command it avoids this so you can just script this to run from a cronjob if you like. \u00a0I normally run unison in batch mode using an alias for bash like:<\/p>\n Of course, if you have a situation like above, it won’t get “fixed”. \u00a0That may be fine 99.9% of the time. \u00a0Occasionally I run unison without the batch argument just to get things fully in sync.<\/p>\n But what happens if I have a thousand files like this? \u00a0Say for some reason, the modify times on a bunch of files got change on both sides. \u00a0Typically you would just use the UNIX “yes” command to tell it to send ‘>’ to the program with something like “yes \\>”. \u00a0Unison will take this input happily until it comes to the last question where it asks if you want to propagate the changes. \u00a0Then it is looking for a ‘y’ or ‘n’. \u00a0Fortunately a ‘y’ is ignored when Unison is asking what direction to propagate the files (see above). \u00a0So you can use the bash command:<\/p>\n This sends a ‘>’ and then a ‘y’ continuously into unison. \u00a0Eventually it will ask if it should propagate the changes and it will get a ‘y’.<\/p>\n I should say this command should be considered a bit “dangerous” unless you are sure, the meta data and files you are propagating are what you want on the other side.<\/p>\n Hope this gives you some insight on this rather handy tool. \u00a0Drop me a line if you have comments or questions.<\/p>\n <\/p>\n","protected":false},"excerpt":{"rendered":" A Bi-directional File Synchronizer. I tend to live on two virtual locations: My MacBook Pro running OS X and my main server KUMR.LNS.COM running Linux. \u00a0I should say, for decades before I switched to Linux, I ran various versions of *BSD on KUMR. I have a home directory on both boxes with a subfolder called … Continue reading “Syncing and backing up two desktops”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.lns.com\/index.php?rest_route=\/wp\/v2\/posts\/128"}],"collection":[{"href":"https:\/\/blog.lns.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.lns.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.lns.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.lns.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=128"}],"version-history":[{"count":18,"href":"https:\/\/blog.lns.com\/index.php?rest_route=\/wp\/v2\/posts\/128\/revisions"}],"predecessor-version":[{"id":146,"href":"https:\/\/blog.lns.com\/index.php?rest_route=\/wp\/v2\/posts\/128\/revisions\/146"}],"wp:attachment":[{"href":"https:\/\/blog.lns.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=128"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.lns.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=128"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.lns.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=128"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}# Unison preferences file\r\n# Local root directory...\r\nroot = \/Users\/pozar\/\r\n# Remote server, ssh port number and root directory...\r\nroot = ssh:\/\/lns.com:22\/\/home\/pozar\r\n# The program to use to copy files with. In this case rsync...\r\ncopyprog = rsync -aX --rsh='ssh -p 22' --inplace --compress\r\n# The program to use to copy files wit that supports partial transfers...\r\ncopyprogrest = rsync -aX --rsh='ssh -p 22' --partial --inplace --compress\r\n# maximum number of simultaneous file transfers. Good to have more than one to really use the pipe.\r\nmaxthreads = 5\r\n# synchronize resource forks and HFS meta-data. (true\/false\/default)\r\n# I'm not interested in seeing AppleDouble files on my Linux box...\r\nrsrc = false\u00a0\r\n# Filename and directories to ignore...\r\nignore = Name projects\/Mail\/.imap\r\nignore = Name .FBCIndex\r\nignore = Name .FBCLockFolder\u00a0\r\nignore = Name {Cache*,.Trash*,.VolumeIcon.icns,.HSicon,Temporary*,.Temporary*,TheFindByContentFolder}\r\nignore = Name {TheVolumeSettingsFolder,.Metadata,.filler.idsff,.Spotlight,.DS_Store,.CFUserTextEncoding}\u00a0\r\n# ignore all files with .unison somewhere in their full path\r\nignore = Path .unison\r\n# Normally ignore the giant VM Harddrive files...\r\nignore = Name Documents\/Parallels\r\n# Don't try to copy this socket...\r\nignore = Name projects\/gnupg\/S.gpg-agent\r\n# ignore = Name Music\/iTunes\r\nignore = Name .lnk\r\nignore = Name .DS_Store\r\nignore = Name Thumbs.db\r\n# Directory paths off of root to sync...\r\npath = projects\r\npath = Desktop\r\npath = Documents\r\npath = Downloads\r\n# Keep ssh open to help with long syncs such as the initial one\r\nsshargs = -C -o TCPKeepAlive=yes\r\n# synchronize modification times\r\ntimes = true\r\n# Create a local log of what happened.\r\nlog = true\r\n# Uses the modification time and length of a file as a quick check if a file has changed.\r\nfastcheck=true<\/pre>\n# unison\r\nContacting server...\r\nConnected [\/\/Tims-MBP.local\/\/Users\/pozar -> \/\/kumr\/\/home\/pozar]\r\nLooking for changes\r\n\u00a0 Waiting for changes from server\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0\r\nReconciling changes\r\nlocal\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 kumr\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0\r\nprops\u00a0 \u00a0 ====> props\u00a0 \u00a0 \u00a0 Desktop\/django\/lib\/python3.6\/site-packages\/django\/contrib\/contenttypes\/locale\/lb\/LC_MESSAGES\/django.mo\u00a0 [] >\r\nprops\u00a0 \u00a0 ====> props\u00a0 \u00a0 \u00a0 Desktop\/django\/lib\/python3.6\/site-packages\/django\/contrib\/contenttypes\/locale\/lb\/LC_MESSAGES\/django.po\u00a0 [] >\r\nprops\u00a0 \u00a0 <-?-> props\u00a0 \u00a0 \u00a0 Desktop\/django\/lib\/python3.6\/site-packages\/django\/contrib\/contenttypes\/locale\/lt\/LC_MESSAGES\/django.mo\u00a0 [] y\r\nUnrecognized command 'y': try again\u00a0 [type '?' for help]\r\nprops\u00a0 \u00a0 <-?-> props\u00a0 \u00a0 \u00a0 Desktop\/django\/lib\/python3.6\/site-packages\/django\/contrib\/contenttypes\/locale\/lt\/LC_MESSAGES\/django.mo\u00a0 []<\/pre>\n
alias unison='unison -batch'<\/pre>\n
while true; do echo \">\"; echo \"y\" ;done | unison<\/pre>\n