Backing Up Files Using rclone

Overview

It is highly recommended that you backup all your files to an external location so you do not lose access to your data, or incase of accidental removal, or an accidental change in your data.

A program called rclone can help you copy your files to a cloud storage service such as: box, google drive, and many other cloud storage services. This guide will teach you how to set up rclone to sync your account’s files to the cloud.

Copying your files to two services, such as Google Drive and Box, provides redundant backups in case of data loss at one of those services. This will reduce your chance of losing important data.

You will lose access to your CSIF files when your account is deactivated -- for instance, after you graduate or when a non-CS major is no longer taking CS classes. 

rclone is provided on all systems in the CSIF -- Google Drive and Box are provided by Campus IT.

Before We Begin

Make sure you have an account with a supported cloud storage service. This guide will go through the process of setting up rclone to work with Box and Google Drive as UC Davis provides free support for; however, the process is similar for all other cloud storage services.

The services that rclone can connect to are:

  • Amazon Drive
  • Amazon S3 Drive
  • Backblaze B2
  • Box
  • Dropbox
  • Google Cloud Storage
  • Google Drive
  • Hubic
  • JottaCloud
  • Koofr
  • Mega
  • Microsoft Azure Blob Storage
  • Microsoft Onedrive
  • OpenDrive
  • Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
  • pCloud
  • QingCloud Object Storage
  • Webdav
  • Yandex

rclone can also backup your data to an ftp, http, and sftp source if you have one set up.

For more information, please see: https://rclone.org/

Quick Setup

  1. Install rclone (if needed, see below)
  2. Setup rclone remote service provider with: $ rclone config
  3. Secure rclone with: $ rclone config (then option s, option a …)
  4. Browse for location to store backup
  5. Create backup

Installing rclone

Some of the destinations rclone supports, such as Box, requires you to have access to a local copy of rclone and a browser for authentication during the first time setup. If you choose to use such destinations, you can either go to the CSIF and do it in person or you can install rclone on your local machine just for the authentication portion. If you choose to set up rclone in person at the CSIF you can skip the installation steps for rclone as shown in this section.

To install rclone on Linux/Mac OS, simply open up a terminal and type in the following command:

$ curl https://rclone.org/install.sh | sudo bash

For Windows users just click the download button from the download page and extract the files in the .zip folder. The link to the download page is: https://rclone.org/downloads/

Set Up rclone remote storage provider

Login to the CSIF to perform these steps. If you are physically in the CSIF -- logged in locally at a desktop system -- you will have fewer steps to perform than if you are logged in remotely through SSH.

Start configuration of a remote storage provider with this command, follow the instructions below for hosting your files on Box or Google Drive:

$ rclone config

Hosting files on Box

  1. Select option n to create a new remote service
  2. Give the service a memorable name. You will be using this name to refer to this account.
  3. Select the option for Box (6)
  4. When it asks you to enter a client_id, and client_secret, leave them blank.
  5. It will then ask you if you want to edit advanced config, Go ahead and say no as the default options are good enough.
  6. It will then ask: Remote config -- Use auto config?
    • If you are logged in locally at a desktop system, enter y. This will open up a browser, follow the instructions
    • If you are logged in remotely via SSH, enter n.
      1. If you are on Windows, open up a command prompt and cd to the folder where you extracted rclone to. For Linux/Mac OS, open up a terminal
      2. In your terminal (Windows or Linux/Mac), enter the following command:
$ rclone authorize "box"
  1. In the browser that pops up log into Box.
    • UC Davis provides an unlimited Box that converts to a standard account when you leave UC Davis. If you wish to use this, then login with your UC Davis Email and kerberos password to login.
  2. Grant permissions for rclone to access the files on your Box drive.
  3. If you entered no on step 6 (logged in remotely), you should now see a long token in your local terminal. Copy that string into your SSH session.
    • It should look something like this:
result> {"access_token":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" ,"token_type":"bearer","refresh_token":"zzzzzzzzzzzz...zzzzzzzzzzzzzzzzzzz",
"expiry":"2019-07-23T15:45:36.0631615-07:00"}
  1. rclone will then show you the configuration and ask you if everything looks correct. If there are no issues, go ahead and enter y
  2. It will then ask if you want to secure your access token by encrypting the configuration file. This is not required; however, it is highly suggested.
    • If you wish to secure your config with a password you can move onto the Securing Configuration section below
    • If you do not care about it, you can now quit the setup with q, and move onto the Backing Up Files section below

Hosting files on Google Drive

  1. Select option n to create a new remote service
  2. Give the service a memorable name. You will be using this name to refer to this account.
  3. Select the option for Google Drive (12)
  4. It will then ask you to enter a client_idclient_secret.
    1. You can leave this blank for normal use cases, but your upload speed will be slower.
    2. If you have a lot of files or plan to use rclone a lot, then you will want to create your own. You can do this by following the instructions at: https://rclone.org/drive/#making-your-own-client-id
  5. It will then ask you how much permissions you wish to give rclone. We recommend the option drive or drive.file
  6. It will then ask you for a root_folder_id and a service_account_file. Leave these two blank.
  7. It will then ask if you want to edit advanced config, Go ahead and say no as the default options are good enough.
  8. It will then ask you if you are working remotely.
    1. If you are logged in locally at a desktop system, enter y. This will open up a browser
    2. If you are logged in remotely through SSH, enter n. They will give you a link. Open the link in a browser on your local computer
  9. Log into your google drive and grant rclone permissions to access your Google Drive.
  10. You should now receive a long verification code. Copy it into the prompt Enter verification code>
  11. rclone will then ask you if this is a team drive, choose n
  12. rclone will then show you the configuration and ask you if everything looks correct. If there are no issues, go ahead and enter y
  13. It will then ask if you want to secure your access token by encrypting the configuration file. This is not required; however, it is highly suggested.
    1. If you wish to secure your config with a password you can move onto the Securing Configuration Section below
    2. If you do not care about it, you can now quit the setup with q, and move onto the Backing Up Files section below

Securing Configuration

We recommend that you encrypt the configuration file as it contains your access token. Anyone with that token will be able to log into your cloud storage with full permissions. You can encrypt the configuration file by doing the following

  1. Enter the config setup if you haven’t already by typing the following command:
$ rclone config
  1. Choose option s to set the configuration password
  2. Choose option a to confirm that you want to add a password
  3. Type a password and confirm it
  4. You are done, choose q to quit the setup

From this point forward you will need to type that password whenever you run rclone.

Backing Up Files

TIP

List the CloudDrives you have successfully set up with this command:
$ rclone listremotes

To backup your files, first find the location you wish to backup to on your remote location. You can do this by using your browser with the cloud service, or from the command line using rclone’s browsing commands. 

The examples below assume that you named your cloud storage service CloudDrive in the setup (you probably named them differently).  Don’t forget the colon “:”.

To list the top level directories use this command:

$ rclone lsd CloudDrive:

To list the top level files use this command:

$ rclone ls CloudDrive:

You can browse other locations by adding a path to the end of the command:

$ rclone ls CloudDrive:MyFiles/Backups

This command will list all files in the MyFiles/Backups folder on your Google Drive if it exists

Once you find the location you wish to create your backup, you can use this command to create the backup:

$ rclone copy ~ CloudDrive:MyFiles/Backups/test

This command will copy your entire home directory into the folder MyFiles/Backups/test on your google drive. You can change the remote path to whatever you want by editing this path.

If you only want to backup a specific folder you can add a path to the folder after the ~

For example

$ rclone copy ~/Downloads CloudDrive:MyFiles/Backups/test

This command will only backup your Downloads folder

You can place this backup wherever you want by editing the path after the remote drive name

Updating backups

If it has been a while since a backup was created you will want to update it. Rclone will not do this automatically. You must run it manually. They can do this with the following command:

$ rclone sync -p ~/Downloads CloudDrive:MyFiles/Downloads

This will sync your downloads folder with the backup’s download folder located at MyFiles/Downloads. 

If you wish to see the changes that the program will make before you do the sync, add the
--dry-run flag and it won’t make any changes 

Further Information

https://rclone.org/