Storage and backups

Not so much news... or... is it?

This seems a bit odd when realizing that computers have been around for the general public since decades now. But still a lot of people have not yet really organized this. If you have or if you think I am nagging about something that won't happen to you then feel free to spend your time in a better way. If you have not or you think you have taken care of this but you are not really sure or curious you are very welcome to read on.

Reason for transition

Recently I transitioned from a MacPro and an Intel Macbook Pro to a new Macbook Pro with an Apple Silicon M1 max. The main reason for this transition is Adobe who made very clear that they keep their applications running under Windows but that for the Mac you'd need a more recent operating system. So I upgraded my MacPro 2009 all the way up to Catalina. It worked quite well but started to show peculiarities so I took my time to design a one computer configuration that would serve my needs as I expect Adobe to make their way into the next generation operating system. And I am not the type to sit and wait for that.

My criteria were quite some...

I have big files and I combine them in several ways. I make panoramas by combining the 42Mpix RAW files as TIFFs in Photoshop which grows them to 300-500 MB each and I stack very many files (100-300) for astrophotography. So I measured what the MacPro was doing and taking. Now this requires my new RAM to go all the way up to 64GB and my direct storage to go up into 2 TB. The processing of all this requires diskspace, that is direct storage. It needs to be fast and it needs to be spacious. Processing astro shots can claim over a TB with ease. Adding to that the need for space to store the operating system and active data that is nice to have at hand I came to 2TB. But as heat causes thermal throtteling (slowing down) on the M1 I need a version that handles heat dissipation best and that is the 16" M1 max with 10 processor cores and 24 graphic cores. All tests proof this machine as the one to have the best thermal control.

Large capacity storage

So... where to store the 3 TB of files comprising everything related to my image processing and of course my photos? They would have to be stored outside the Macbook Pro which results in filetransfer by USB. And that is slow in comparison with the bus in the old MacPro. Well there is a solution to that. I measured the speed in the MacPro and that came to around 260MB/sec on my internal SSD disks. I already had two Samsung T5 disks used for extending the disk space of my old Macbook Pro and as travelbackup. The problem is that they max out at 2TB and so does the successor T7. Anyway they reached around 355MB/sec on the USB 3, gen 2 ports of the MacPro (518 MB/sec on my old laptop). The disk with normal data reached around 190MB/sec and the disk with my photos reached 210. To put things into perspective, the speed of the internal disk on my 2018 Macbook Pro is 2647 MB/sec and on the M1max it is 5924 MB/sec.

What it came down to

To make a longer story shorter I bought the exact same type of disk that i used for my photos: Western Digital Red Pro 6TB and put them in an OWC ThunderBay 4 mini (Thunderbolt 3) Incl SoftRAID in a RAID 0 configuration. This means that every file is split in 2 parts which are simultaneously written to the two disks, each of the merged disks getting the other half of the file. This speeds up the now merged to 12 TB disks to 480 MB/sec. Besides this the Gemini is a docking station so over that one cable connection on Thunderbolt 4 I get 40Gb/sec which is a bandwidth of over 11 GB/sec.

Backing up

Now how to back all this up? I could change to 2x 6TB in mirror by RAID 1. But a mirror is not a backup! A backup is a system of storing versions of the contents of a disk. Stacked by hours to a day. Stacked by days to a week or month. All that stacked to a year and so on. So you can go back in time to retrieve files that you have lost but also that you have deleted at some time in the past that turned out not to be so smart.

Backing up means you have to keep administration of which file was modified since the last backup. That is quite a challenge and this can't be done without dedicated backup software. For Mac there is an excellent backup system which is part of the system itself, you have a Mac you have an excellent backup system called Time Machine.

Backup procedure and vulnerabilities

Once you have your backup you have done part one of what is needed to secure your files. When burglars take your stuff you want your backups to be secured by a password. That is the second and easy step. The next one needs somewhat more organizing. Add an extra disk to your backup pool and when switched once per month or any other interval, store the previous disk at a place outside your home. When fire gets your house your backup burns with it but your previous disk will be safe anywhere else. Ask a family member or trustworthy friend to store the disk in their home.

But we are still not there.

There are still two threats to tackle. Number one is the solarwind. Huh? Yes... When the 11 year cycle of the sun grows to its top the sun throws larger and larger amounts of plasma into space. That plasma is the solar wind that, when it collides with our upper atmosphere, causes the Northern Lights. But it has a much less friendly second effect and that is the disturbance of wireless communication all the way up to failing electronic equipment and disruption and even destroying electronics. Known events are the Carrington event 1-2 september 1859, the explosion of sea mines located at the entrance of Vietnamese harbours 4 august 1972, 13 march 1989 Canadian power plant. The power of the solar wind can destroy the contents on your harddisks, conventional or SSD and even burn out the electronic circuits. And as we are on it... Also when the suns cycle winds down this can happen. Number two has recently become a renewed threat by Putins intention to re-open the option of nuclear weapons. The explosion of such a weapon causes an EMP or ElectroMagnetic Pulse having the same effect as the solar wind. Now this second option won't make much sense to save your personal pictures as problems will be of more existensial nature but you may have your reasons. If anyone dismisses your reasons they are still yours and no one is to judge that.

Anyway, if you want to go to the lenghts of preserving data for very long periods and whatever catastrophe you can do the folowing. Burn your data onto a Bluray disc and be sure to do that on an M-Disc quality disk. You do need a specific burner that is capable of BDXL M-Disc. I use the OWC Mercury Pro BDXL Bluray burner which is capable of burning 100GB discs. That is the final frontier. And preserve those discs in a fireproof safe. This is the way archives function. As the 100GB discs are expensive and as they only need to counteract a catastrophy they only contain files that I really do not want to loose so they contain my primary collection. Just be sure to run a verify action before closing the disc after the burn process! These disc are guaranteed for 200-1000 years. The one remaining concern is the same for every archive... are there bluray disc players in 2222 or 2422 to read our discs? That may sound funny and when you stopped laughing realize that tapedrives and drives for 1.44 MB floppydiscs don't exist anymore... as do many other media readers. That is the nightmare for big professional archives.

I hope that you now can make wise decisions on how to make a backup and which procedure works for you.