A really curious thing happened in the Photo Taco Facebook group. Listener Jim Ruse asked a question there about an experience he was having where some of the photos he copied from his internal hard drive over to an external hard drive would be unreadable a few months later.
Hey folks. I’ve got an issue I would like to share. It’s happened multiple times and I can’t seem to stop it from happening. After a big project – maybe a 2 or 3 week trip to, say, Europe, I spend hours and hours and hours processing maybe hundreds of photos. Those that are processed using my best techniques in Photoshop end up very large files. I then move ALL of those large files to an external hard drive, leaving only images processed in Lightroom on my iMac internal drive. Then, maybe months later, I decide I want to print one of those PS files. I go to my external, and many of my PS files are gone. The file names are still there on the hard drive, but there is nothing in them. They are gone forever. I conclude there is something happening during the Copy/Paste process when I move the files to external drive that destroys all my hard work. When it happens, I get so depressed I can’t stand it. Am I doing the transfer from internal to external all wrong? Should I be doing it a different way? Thanks!Jim Ruse – Photo Taco Facebook Group
A few Photo Taco listeners tried to help Jim out with his problem, but things weren’t adding up to me. Like me, Steve has a worked in IT for a long time and has had a bit of an emphasis in storage in particular so I reached out to him to see what he thought. It similarly bothered Steve and was very kind to reached out to Jim and worked through things a lot.
Let’s walk through what exactly was going on here and then let’s talk about things photographers can do to avoid having a similar challenge.
Like most photographers these days, Jim is augmenting the storage space available on the drive inside of his computer with external storage. A totally legitimate use case. After copying an image from his internal hard drive to his external drive the files seemed to “decay” over time. Immediately after the copy of a few hundred images to the external drive everything looked fine but when he went back a few months later to use one of those images several of them would be missing their thumbnails and fail to open in Photoshop.
Steve asked Jim to send him one of these files that was broken, and when he tried that didn’t work. Steve asked him what error message he was getting when he attempted to open one of these files in Photoshop and the answer was interesting – the error was File Not Found. When a file gets corrupted Photoshop will usually say something about being unable to open the file or that the file is unrecognized, not that the file is not found.
With the important clue that the error was the file was not found, Steve thought that the file must technically not be there any longer. Steve worked with Jim to get remote access to his computer and now as they looked at the same file in the screen shot above, the file now showed up as being zero bytes in size in Finder. Within 2 days the file had decayed from showing as being 879.1MB in size to now being zero bytes. To make things worse, it wasn’t just this one file, it was 8 to 15 of them in the folder.
Steve continued to investigate by seeing if he could copy the files from this folder to a temporary folder on the internal drive using Finder. The copy failed with an error message that the files couldn’t be found. This led Steve to believe that the drive had indexing issues.
Steve ran the Mac OS Disk Utility First Aid which repaired a damaged table of the drive. After the table for the drive was fixed, they went back to look at the files in Finder and these files that seemed to decay over time no longer showed up. The files likely never made it when copying from the internal to the external drive, or if they did the drive seems to be in poor enough health it can’t keep data like it should.
Even though the First Aid utility on MacOS seems to have fixed the corrupted table, the solution here is to replace the external hard drive. The drive has demonstrated that it can no longer be trusted and once a drive has violated that trust a single time photographers should never fully trust a drive again.
Let’s repeat that. This is one of the important take-aways for photographers. If a hard drive has any sort of abnormal behavior STOP TRUSTING IT! As soon as a drive starts acting up stop putting your photos on it and get the drive replaced.
Proper Archival Workflow For Photographers
Jim wasn’t doing anything wrong in using external storage for archiving his photos as his internal drive was filling up. This is a totally legitimate workflow for photographers. Internal hard drives are fast but small today and nearly every photographer has the need to “archive” their photos by moving them off the internal drive to a bigger and less expensive external drive.
However, Jim did make a mistake in not recognizing the drive was behaving abnormally and replacing it. In his question Jim said “…it’s happened multiple times and I can’t seem to stop it from happening.” Instead of losing all trust with this drive and replacing it as soon as possible, Jim altered his behavior working with the drive and started to copy only a handful of images to the drive at a time. He had seen problems when attempting to copy more than 10 or so files to the external drive, so he just started to copy 10 at a time.
Again, if something starts happening with a drive that is abnormal from what you have seen it do in the past, start down the path of replacing it as soon as possible. One of the things that can decrease the risk that your archival copy from the internal drive to the external drive didn’t work is using software capable of doing validated copies. More on this in a moment but the software does a little extra work as you copy and can help you notice a drive is not working correctly.
This brings us to another important take-away for photographers. Archiving photos is NOT the same as backing photos up.
Please don’t mistake archival of your photos with backup of your photos. They are two very different things. Every photographer should implement a 3-2-1 backup solution and you should check out this episode Jeff did with Brent on backup for photographers.
Episode Sponsor: CanvasDiscount
We know you have a lot of choices for your photo printing needs but if you are looking for good quality at the very lowest price you really need to check out our partner CanvasDiscount.com.
CanvasDiscount offers a guarantee of the lowest price in the USA yet they are rated as #1 by TrustPilot. Their service has fast turnaround shipping in 24 hours and Jeff recently trusted CanvasDiscount with a large canvas print of his most recent family portrait for his living room wall . He is very happy with the results and confident you will be too.
We have worked out a deal with CanvasDiscount for listeners of Master Photography to get an additional 15% off their already low pricing by using coupon code MASTERPHOTO15. That is MASTERPHOTO15 at CanvasDiscount.com.
Recommended Storage Tools For Photographers
Copy and paste in both Finder on MacOS and File Explorer in Windows do not validate the files they copy. Neither does Lightroom for that matter. If you really want to make sure that your images are being copied from one location to another without any corruption, you need to do a validated copy and probably shouldn’t use Finder/Explorer.
With a validated copy a check is done on the file after it has been copied to the destination and compared with the source. Software can create something called a checksum on a file and when that checksum of the file at the destination matches the checksum of the file at the source you can be sure that every bit and byte of that file successfully made it over to the destination.
There are multiple ways to accomplish this. You can do the copy using Finder/Explorer as you normally do and then use command line tools like a DOS prompt in Windows or a Terminal window on MacOS to run commands and validate things. I don’t imagine most photographers have any interest in doing this.
You can also do the copy using software that has checksum validated copies as a feature. These tools not only validate your images are making it fully in tact from source to destination, but they tend to do it more quickly than Finder/Explorer!
Recommended tools for validated copying:
- TeraCopy (Win & Mac, free with some limits, $25 for Pro). Similar tool Duplicate for Mac: FREE (Untested)
- Hedge (Mac & Win, $50 for 3 months or $130 for long-term). Bit of caution with this tool based on this article: https://www.newsshooter.com/2019/10/28/what-is-the-fastest-offload-software/
- Shotput Pro (Mac & Win, $149, yearly update subscription, 15 or 45 day rental available)
- True Check (Mac, $149, yearly update subscription, 15 or 45 day rental available)
- ChronoSync (Mac only, free trial for 15 days, $50)
As part of a good 3-2-1 backup solution, photographers usually have the need to replicate MOST of what happens on their primary drive to a secondary drive. Most is a critical word there because there is good reason to not want every action taken on the primary drive to be taken on the secondary drive. For example, if you accidentally delete an image from your primary drive you probably don’t want that action replicated over to the secondary drive.
Just like with the validated copy, this replication really needs to have checksum validation as part of the process. The same mishap can take place as the files are cloned/mirrored over to another drive as when you are doing one-off copies.
Tools for cloning/mirroring:
- Bvckup2 (Win only, 2 week trial, $30 for basic)
- Carbon Copy Cloner (Mac only, 30 day trial, $40)
- ChronoSync (Mac only, free trial for 15 days, $50)
- Super Duper $28
Your 3-2-1 backup solution needs to include an offsite copy and both Jeff and Steve have been using BackBlaze (Win & Mac, $6/mo for unlimited backups or $60/year) for years.
NOTE: Link gives you a free month and Jeff a free month! Check out the podcast episode Jeff did with Jim Goldestein from BackBlaze about Online Backup.
Jeff: Duramont Ergonomic Adjustable Office Chair with Lumbar Support ($330). I just replaced the office chair I have been using for the past three or four years with this one. I wore out the spring in the chair I was using and it was making me lean as I sat in it. Not good since I work from home (even before COVID-19) and edit in this chair which means I sit in it a good 8 to 10 hours a day. I decided I needed to get a good chair and that means spending more than the $100 or so I had been to this point. There may be better chairs than this one, there are certainly far more expensive chairs. This seemed like the best price to performance I could find and was recommended as one that competes well with task chairs like those from Herman Miller without the much larger price tag.
- Facebook group is Master Photography Podcast
- Instagram account for the show is @masterphotographypodcast
- Find Jeff’s work at https://www.jsharmonphotos.com. Check out his Photo Taco podcast over at https://phototacopodcast.com where you can search all kinds of topics and find shows discussing the details. He is on Facebook: https://www.facebook.com/harmon.jeff, Instagram: https://www.instagram.com/harmonjeff/ (@harmonjeff), and Twitter: https://twitter.com/harmon_jeff (@harmon_jeff)
- Find Steve’s work at: stevebrazill.com, his incredible podcast at behindtheshot.tv, and Twitter / Instagram: @SteveBrazill / @behindtheshottv