Using rclone to recover data from Backblaze buckets
I use Cloud Sync Tasks in TrueNAS to push data offsite into Backblaze, through TrueNAS you can push or pull data as needed but in a DR scenario you may not have your TrueNAS instance available. I wanted to know how to recover from just a laptop and these notes are what I put together to reference should that time ever come.
You need the rclone
binary installed, I get mine from my "scoop.sh" package manager.
Configure your remote
Run rclone config
to open our configuration menu.
- Press "n" to add a new remote
- Choose a name for your remote (this name is used later)
- A large list pops up of known storage solutions, choose "3" for Backblaze B2
- Enter your Account ID or Application Key ID (these are how you pushed data into B2)
- Enter your Application Key
- You are asked if you want to "hard delete" or "hide files" when they are deleted from this remote. We want to use "false" to prevent stupid accidents.
- We do not want to edit the advanced config, hit "n"
- You are presented with the completed config now, you can accept it with "y"
- Press "q" to end or "n" to make another remote
rclone config file
Remote with encryption
Setting up an encrypted remote will build upon an existing remote from the step above. You cannot setup an encrypted remote from scratch. To setup our encrypted remote we need to know both the remote name as well as the specific bucket we want. Run rclone lsd {remote}:
to get the list of available buckets and make not of it.
rclone config
again to setup our encrypted remote.- Press "n" to add a new remote
- Choose a name for this remote ("existingRemote-crypt" is what I use)
- Press "12" to start the encrypted remote setup
- Input the remote and bucket that is encrypted tin the format "remote:bucket"
- Choose to encrypt names, obfuscate them or do nothing.
- Choose to encrypt directory names or do nothing
- Press "Y" then enter the password used for your remote.
- Choose to enter the salt or not, in TrueNAS this seems to be required.
- We do not want to edit the advanced config, hit "n"
- You are presented with the completed config, you can accept it with "y"
- Press "q" to end or "n" to add another remote.
rclone lsd
you will only use rclone ls
to explore the remote
Viewing an encrypted bucket in plaintext and cipher-text
> rclone ls b2:backuptest384595
74 FooestOfRoos.txt.bin
> rclone ls b2-crypt:
26 FooestOfRoos.txt
Rclone Usage Basics
List remotes
rclone listremotes
List buckets on remote (the colon is required)
rclone lsd {remote}:
List files in a bucket
rclone ls {remote}:{bucket}
Explore the remote
To list your buckets in the remote:
> rclone lsd b2:
-1 2022-01-25 13:55:05 -1 backuptest384595
To list the contents of a bucket run
> rclone ls b2:backuptest384595
64083579 WindowsTH-KB2693643-x64.msu
.bin
extensions on file where you do not expect, this designates the file are encrypted. See how to setup an encrypted rclone mount above.
Pull from the remote
You should read over the page on how copy works to understand the basics such as --dry-run
and how rclone appends a trailing /
automatically. The basics are rclone copy --progress {remote}:{bucket}\file C:\Dest
when copying data out of Backblaze.
Copying from a plain text remote
> rclone copy --progress b2:backuptest384595\WindowsTH-KB2693643-x64.msu ./
Transferred: 61.115 MiB / 61.115 MiB, 100%, 4.145 MiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 15.8s
Copying from an encrypted remote
> rclone copy --progress b2-crypt:\FooestOfRoos.txt .\
Transferred: 26 B / 26 B, 100%, 0 B/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 1.5s
Backblaze recommends that you do lots of transfers simultaneously for maximum speed. In tests from [sic]an SSD equipped laptop the optimum setting is about --transfers 32
though higher numbers may be used for a slight speed improvement. The optimum number for you may vary depending on your hardware, how big the files are, how much you want to load your computer, etc. The default of --transfers 4
is definitely too low for Backblaze B2 though.
Note that uploading big files (bigger than 200 MiB by default) will use a 96 MiB RAM buffer by default. There can be at most --transfers
of these in use at any moment, so this sets the upper limit on the memory used.