rsync output compressor

Rsync is a very nice tool for automating remote backups. (Specially in combination with daily snapshots (like zfs snapshot) ).
Like many others I have automated the process of running rsync on a daily basis via a cron job. Cron nicely sends me an email with the output of the rsync command.

I usually use the -v option so I can see what files have been changed. This worked nicely several years ago when I didn’t have much changes on my server. But nowadays I often receive e-mails of 10 MB or larger. That’s not very useful.

Removing the -v option is an option, but I don’t see anything anymore (perhaps a total summary).

To solve this problem I’ve hacked together an rsync-output-compressor script :)
You can find it on https://github.com/gamecreature/rsync-output-compressor

This scripts summarizes the output of rsync -v based on a given rules file. You can specify what files/folders should be explicitly mentioned and what folders/files should be grouped together.

This little script is written for ruby 1.9 and higher.

An example

For example let’s view the following output: (… = many more lines)

rsync -avz --delete user@example.com:/data /backups/remote_data
receiving incremental file list
/home/emma/public_html/important_file.txt
/home/emma/public_html/important_file1.txt
/home/emma/public_html/important_file2.txt
/home/emma/public_html/important_file3.txt
...
/home/sarah/public_html/index.html
/home/sarah/public_html/images/
...
/home/david/private/special_file.txt
/home/david/public_html/downloads/new_download.zip

Using the following filter: (compress-rules.txt)

/home/*/public_html/

results in the following output:

rsync -avz --delete user@example.com:/data /backups/remote_data | rsync-output-compressor.rb --rules compress-rules.txt
receiving incremental file list
   123    -5 /home/emma/public_html/
    40       /home/sarah/public_html/
     1       /home/david/private/special_file.txt
     2       /home/david/public_html/

The column with positive values are changed/added files and the column with negative values are the number of deleted files.

The tool has several other options like storing the original full output to an external location (option -f).

Using this script my daily emails have been reduced from 10MB to 30KB :)
And I still know what is happening with my backup.

Feel free to use and improve this little script!

share!

Ruby undocument File.basename trick

Just stumbled upon the following trick to get the basename without an extension in Ruby:
(I didn’t know you could use “.*” to remove any extension).

File.basename( "filename/sample-filename.html", ".*" )
=> "sample-filename"

Fix rails database collations

Today I discovered my Ruby on Rails development environment database migrated with the wrong character encoding. It was using the MySQL default encoding latin1.
I didn’t feel throwing my database away because it contains a lot of stuff.

I used the following snippet to convert all columns to utf8

    ActiveRecord::Base.connection.tables.each do |table|
      ActiveRecord::Base.connection.execute( "ALTER TABLE `#{table}` CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci")
    end

A copy of has been removed from the module tree but is still active

I’m working on a rails website that requires me to specify a class in an initializer.

config/initializers

spree.searcher_class = MySearcherClass

I’m currently developing this searcher class. Every time I change this class I get the following message:

A copy of MySearcherClass has been removed from the module tree but is still active

This sucks big time! Because I need to restart my rails application every time I change something.

My workaround for the moment is this:

spree.searcher_class = class.new do
    def new(*args,&block)
      return MySearcherClass.new( *args, &block )
    end
  end
end

I’m not very keen on this, but it does the trick for now :)