Go back to the main page

Create your own git server easily with Chef and the Vagrant AWS plugin: Part 3.

This article is over 2 years old. Proceed with caution.

Regards ♨ – Minimul

 

Part 3 Goal. Backup your repositories to S3

Continuing from Part 2 in where we finished creating a personal Git server using Chef and the Vagrant AWS plugin.

  1. Create a S3 bucket called git-server-repos from the AWS console.
  2. Add the following s3cmd cookbook to the Berksfile.
  3. site :opscode
    
    cookbook 'gitolite', git: 'git@github.com:julionc/chef-gitolite.git'
    cookbook 's3cmd', git: 'git@github.com:freerobby/s3cmd.git'
    metadata
    
  4. Also add depends 's3cmd' to metadata.rb.
  5. name             'git-server'
    maintainer       'YOUR_NAME'
    maintainer_email 'YOUR_EMAIL'
    license          'All rights reserved'
    description      'Installs/Configures git-server'
    long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
    version          '0.1.0'
    
    depends 'apt'
    depends 'git'
    depends 'gitolite'
    depends 's3cmd'
    
  6. Edit the recipes/default.rb as so:
  7. include_recipe 'apt'
    include_recipe 'git'
    include_recipe 'gitolite'
    
    gitolite_user node.gitolite.username do
      home node.gitolite.home
      version node.gitolite.version
      ssh_key node.misc.ssh_key
    end
    
    include_recipe 's3cmd'
    
    
  8. Update the Vagrantfile from Part 2 with the s3cmd attributes highlighted.
  9. Vagrant.configure("2") do |config|
      config.vm.hostname = "git-server"
      config.vm.box = "aws-basic"
      config.vm.boot_timeout   = 120
      config.omnibus.chef_version = :latest
      config.berkshelf.enabled = true
    
      config.vm.provision :chef_solo do |chef|
        chef.json = {
          misc: {
            ssh_key: File.read(ENV['MY_PUBLIC_SSH_KEY_PATH'])
          },
          s3cmd: {
            aws_access_key_id: ENV['AWS_ACCESS_KEY_ID'],
            aws_secret_access_key: ENV['AWS_SECRET_ACCESS_KEY']
          }
        }
        chef.run_list = [
          'recipe[git-server::default]'
        ]
      end
      config.vm.provider :aws do |aws, override|
        aws.access_key_id = ENV['AWS_ACCESS_KEY_ID']
        aws.secret_access_key = ENV['AWS_SECRET_ACCESS_KEY']
        aws.keypair_name = ENV['AWS_KEYPAIR_NAME']
        aws.security_groups = ['your-security-group-here']
        aws.instance_type = "t1.micro"
        aws.ami = "ami-d0f89fb9"
        override.ssh.username = "ubuntu"
        override.ssh.private_key_path = ENV['MY_PRIVATE_AWS_SSH_KEY_PATH']
      end
    end
    
  10. Now run vagrant provision. Below is the partial output for a successfully installed s3cmd.
  11. ....
    [2013-10-10T11:23:31+00:00] INFO: directory[/usr/local/share/s3cmd] created directory /usr/local/share/s3cmd
    [2013-10-10T11:23:31+00:00] INFO: git[/usr/local/share/s3cmd] cloning repo git://github.com/s3tools/s3cmd.git to /usr/local/share/s3cmd
    [2013-10-10T11:23:32+00:00] INFO: git[/usr/local/share/s3cmd] checked out branch: v1.1.0-beta3 reference: 7d047bd9d4df6cbfa1ea91c9171091cd9347cd9f
    [2013-10-10T11:23:32+00:00] INFO: execute[build_s3cmd] ran successfully
    [2013-10-10T11:23:32+00:00] INFO: link[/usr/local/bin/s3cmd] created
    [2013-10-10T11:23:32+00:00] INFO: template[s3cfg] backed up to /var/chef/backup/root/.s3cfg.chef-20131010112332
    [2013-10-10T11:23:32+00:00] INFO: template[s3cfg] updated file contents /root/.s3cfg
    [2013-10-10T11:23:32+00:00] INFO: template[s3cfg] mode changed to 600
    [2013-10-10T11:23:32+00:00] INFO: Chef Run complete in 8.671023313 seconds
    ....
    
  12. Test that s3cmd installed properly by using it to list the new bucket created in Step 1.
  13. $ vagrant ssh -c "sudo su - root -c 's3cmd ls'"
    # output
    # stdin: is not a tty
    # 2013-10-11 11:16  s3://git-server-repos
    
    IMPORTANT: If you don't get a bucket listing with the above command make sure that your AWS credentials are correct and then re-provision.
  14. Add a Cron job to backup your repositories daily. Start by adding depends 'cron' to metadata.rb.
  15. name             'git-server'
    maintainer       'YOUR_NAME'
    maintainer_email 'YOUR_EMAIL'
    license          'All rights reserved'
    description      'Installs/Configures git-server'
    long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
    version          '0.1.0'
    
    depends 'apt'
    depends 'git'
    depends 'gitolite'
    depends 's3cmd'
    depends 'cron'
    
  16. Include the cron recipe as well as the backup command to be run.
  17. include_recipe 'cron'
    
    cron_d 'backup_git_repos_to_s3' do
      minute  0
      hour    3
      command "s3cmd sync #{node.gitolite.home}/repositories s3://git-server-repos/"
    end
    
    DISCUSSION: A file called backup_git_repos_to_s3 will be created in the /etc/cron.d directory and be run by the root user.
  18. Run vagrant provision and then test that all went as planned.
  19. $ vagrant ssh -c 'sudo cat /etc/cron.d/backup_git_repos_to_s3'
    # Crontab for backup_git_repos_to_s3 managed by Chef. Changes will be overwritten.
    
    0 3 * * * root s3cmd sync /home/git/repositories s3://git-server-repos/
    
    IMPORTANT: By default Ubuntu Server is set for UTC time so for example, 3 AM UST would be 10 PM EDT.
    NOTE: I will leave the final testing of the repository backup to you. It is best to test the Cron job by vagrant ssh'ing in and manually editing the time within /etc/cron.d/backup_git_repos_to_s3. Then when you are done testing run vagrant provision again and /etc/cron.d/backup_git_repos_to_s3 will be overwritten with your preferred time and command.

Finished

This concludes a 3-part tutorial where we installed an automatically-backed-up Git server on EC2 with a minimal amount of code thanks to the combined power of Chef and the Vagrant AWS plugin. I think that you will find Vagrant is a fine tool for production environments. You can re-visit Part 1 to read the benefits of using Vagrant for production, staging, or QA environs. Lastly, you will find full source code of this tutorial on Github.

Comment on this article?