Workflow Hardware Upgrade: Wiebetech Solution
As technology changes, so do our needs as a photographer. This December, we've upgraded our boxes we use to store our images.
Here's our system:
When considering other solutions, we particularly do not like the Drobo boxes for several reasons, and want to caution you strongly before considering them. Problems abound, as reported all over the internet, however, I am sure that some people will sing their praises. Below are several problems - each in-and-of themselves would be a reason not to use the boxes. Together, they make a compelling argument to avoid Drobo. If you're not going to choose the system we're reviewing and reporting on today, then consider other solutions.
Here are some of the Drobo issues:
- Proprietary file format. Thus, if you need to pull a drive from their box, you cannot plug it into a Mac or PC and browse/access the files. Further, unless you re-insert the drives with the exact same configuration as when the files were written, you have no access to the files.
- The drives/boxes tend to be far too slow for drive-access intensive needs like opening, saving, and other file-releated needs.
- A mirrored drive array is not a backup, just reliability protection. You'd need to run two Drobos to do have two backups, as whatever you do (or is erroneously affected on one drive) is immediately replicated on the second. So, you're protected from "drive failure", but not an accidental deletion, or file corruption which then corrupts the file on the mirrored drive. A backup would not only protect you from drive failure, but also those accidental deletions, accidental "save" when you meant "save as" file changes, and other unwanted file changes. Further, the Drobo isn't a true "mirror", it's an odd-flavor RAID 5.
- An electrical fault that fries one drive likely will fry the other. You do truly need a dual drive system, with redundancy offline (and preferably off-site) in order to be properly protected. When we have both drives mounted (as explained in the video) it is for manual mirroring, then the backup copy of the primary drive goes offline (and, in a perfect world, off-site.) We try very hard to keep drives seperate to protect them, and thus, our images.
- The Drobo does not check the integrity of the data. This is a problem from a data-integrity standpoint.
(Comments, if any, after the Jump)
Please post your comments by clicking the link below. If you've got questions, please pose them in our Photo Business Forum Flickr Group Discussion Threads.
13 comments:
How do you share your files with others in your office? We really need some type of NAS server so multiple people can access the images and video files at once.
I believe the best is a JBOD -just a bunch of disks, with separate disks for original downloads and edited files. I have a nightly backup with a mirror of that offline and offsite, and the same for my archived drives. The drives are accessibkle over my network, and I mostly use Lightroom, so it is the catalog that is also backed up.
John,
I like the Wiebetech units as well - I have several.( I particularly like the little 2 bay 2.5 inch unit.) I also have some Drobos, and some of what you say here is simply not accurate.
1. Keep in mind that while drive mirroring is not proprietary, *all* hardware-based drive spanning is proprietary. And some people simply need to do this for performance reasons. High-end video editing nearly requires it. I wish this "proprietary" argument would not keep showing up.
2. Tell that to Rich Harrington (no relation, as you know) who runs a very busy high-end video post production house with Drobos providing the back-end storage. If it's fast enough for him, it's fast enough for you.
3.While Mirrored RAID does provide some level of fault-tolerance, I would suggest it's much safer to have that second drive in a separate box, most of the time. Also "Further, the Drobo isn't a true "mirror", it's an odd-flavor RAID 5." All RAID 5 is proprietary. I'd say, instead, that it's the next generation of RAID 5-like architecture (with some pretty significant advantages, when it's time to upgrade capacity).
4. Agree with the voltage spike hazard, but using those drives as mirrored RAID makes it really hard to protect against this. If you pull a drive out of the box, and then do any work at all, the Wiebetech will want to rebuild the mirror from scratch, which can take a while. It's workable, but probably less ideal for most people than simply having a second drive that can be more easily taken offline and brought back online easily.
5. What do you mean by this? "The Drobo does not check the integrity of the data. This is a problem from a data-integrity standpoint." If you mean that Drobo does not do data scrubbing in the background, well, neither does the wiebetech last time I asked their chief product engineer.
Also, now is not a great time to upgrade drive storage for many people. The floods in Thailand have spiked prices through the roof for spinning disks. I am only buying drives as they are needed right now, and I'm even repurposing some drives to forestall new purchases.
Peter Krogh
Great post, John. Simple, no-nonsense backup strategy. Thanks.
Peter - We'll have to agree to disagree here then on the Drobo units. To answer your points:
1) High end video on units where the choke point is the port (FW, USB, etc) benefit from it almost to the point of requiring it - however high end video production houses would be best served with a SCSI or eSATA/SATA/Multi-lane connection. The fact remains that Drobo units do write in a proprietary format. People need look no further than the efficacy of Sony BetaMAX, which was a proprietary video format that tried to compete with the open VHS format, and failed. If you had 100 BetaMAX movies back in the day that Sony stopped making those units, you'd be pretty upset. However, I'd avoid storing my images in a proprietary format like this. Further, while we are big proponents of Adobe's DNG open format, we use a "trust but verify" approach, still storing the original RAW camera files, but using our DNG's as our Master images.
2) I'm familiar with Rich Harrington, although we've never met. Further, I am not familiar with his system, but will highlight the point you made in your representation of his system - the Drobo's are used in the "back-end storage", which suggests to me that they are not fast enough to provide "front-end" or "active", and instead he uses a faster "flavor" of connectivity. This is an indicator to me that my point about speed on the front end (and most photographers I know don't have back-end AND front-end storage) is valid.
3) I appreciate your point about keeping drives in separate boxes, and while (as you can see in the video) we do have separate boxes to a degree) we usually keep the backup drive offline (i.e. not plugged in) except for when we are doing active archiving. We'll have to disagree about Drobo somehow being "the next generation of RAID-5 like architecture..." because they are in no position to be defining architectural standards for storage, in my opinion.
4) When I referred to a "mirrored RAID" I may have not been clear - in our redundancy drive the box handles the RAID mirror, but in our backup tower, we manually mirror it, using Synchronize PRO, and then take the backup offline.
5) I didn't suggest "data scrubbing", I was saying that Drobo isn't as "data fault" protective as other systems.
To your point about the floods in Thailand - I agree. That said, however, this was more of a post about the hardware the drives use and not advice to buy drives themselves. I also experienced drive costs spiking - in trying to purchase 6 drive units, I was limited by the website to a maximum of 3, which was at a price of $103, and then when I went back to buy the next 3, the site had jacked the price up to $139. Crazy. We buy drives because we need them, so while we shop for best prices, we understand that we buy what we have to when we have to. Upgrading our hardware for January 2012 was of value to us, and I know others.
Thank you for your comment. I (and I know many others) value your opinion, and the more discourse on this subject the better.
Regards,
John
John,
You need to do some more research.
1. "however high end video production houses would be best served with a SCSI or eSATA/SATA/Multi-lane connection."
Drobo offers an iSCSI option, which runs on standard Gigiabit cabling (a significant advantage for some - very long runs, standard switches, every computer has this port). It tops out at a theoretical 120 MB/s. While eSATA 3 and 6 GB provides higher theoretical throughput, you only get it with multi-drive striped arrays, or with very fast SSD/SSD arrays. None of the single drives or mirrors you outline are likely to be running anywhere near that fast, and most will be considerably slower in real throughput.
2." the Drobo's are used in the "back-end storage", which suggests to me that they are not fast enough to provide "front-end" or "active", and instead he uses a faster "flavor" of connectivity"
I believe that Rich has striped arrays as the working drives on each of the many workstations in the studio. But those working RAIDs are backed up each day to the Drobo servers.
(And, note that the configuration you outline would not be fast enough for their working drives, as it does not include any striped arrays. )
I know that Rich needs massive speed and capacity. They add 1-2 TB of data per week, so speed is of the essence, just in terms of basic data handling. They have a far larger need for speed then either you or I.
(BTW, I'm pretty sure I introduced you guys to each other at a private event in New York a few months ago.)
3. "We'll have to disagree about Drobo somehow being "the next generation of RAID-5 like architecture..." because they are in no position to be defining architectural standards for storage, in my opinion."
Again, you don't seem to understand the basics of ANY drive spanning. It's always done in a proprietary way. There is no official standard for how RAID 5 works, it's always managed by some device in some proprietary way. Drive spanning design is always up to the software/hardware engineers who decide how to split, manage and reconstruct the data.
If you don't believe me, pull your RAID 5, 6 or 10 drives from one manufacturer's box and try to make them work in another. Not going to happen.
There is this idea that won't die that RAID 5 is some kind of standard, rather than a rough description of how the drive spanning has been implemented. Stop it. Now. It makes you look ignorant.
5. " I was saying that Drobo isn't as "data fault" protective as other systems."
On what do you base that opinion?
Peter
Peter -
I suppose it would have been valuable for you to have mentioned that Richard Harrington appears in at least one video on the Drobo website, as well as arranging discounts on the devices with Drobo for readers of his blog. I'm glad he has a good relationship with them and seems happy. I also note that you've done at least one webinar on the Drobo Broadcast Network. Disclosing these facts (any any other relations you may have with them) when defending Drobo would have been valuable. I can state that I have not done any Wiebetech sponsored activities nor have I been paid by them.
To your points:
Since I was not aware of Rich's hardware setup, I relied on your characterization of it with Drobo's being used in his "back end", so you're now disclosing more information. That said, while I appreciate your promotion of him as a satisfied Drobo user, there (as noted above) needs to be an asterisk next to his name from the standpoint of someone who has some form of a relationship with them beyond that of a normal user.
As to our meeting - now that you mention it, we were introduced (by you) and spoke briefly during a private event in New York, so I stand corrected on that point.
Also, as I understand it, the Drobo offers no data security during the rebuild of a failed drive, or when adding a new one. A decent RAID system maintains data integrity/security during that process, because the drives are working much harder during that phase than at other times, and if a drive fails during the rebuild/expansion phase, you're SOL. Has Drobo upgraded their technology since I last checked?
Further, I want readers to know that Peter *IS* an expert on the subject of Digital Asset Management systems (I recommend his book high up in the Recommended Reading list), and I appreciate his taking time to post (and re-post) here on this important subject. As I said before, the more discourse on the subject the better. If, after people read the post and the comments, they then decide to buy a Drobo, they will have been well informed before making a purchasing decision.
Regards,
John
John,
Drobo has given me units to test and use, but they have never paid me anything. Wiebetech has also given me units to test and use, but has not paid me directly either. Wiebetech has also set up a series of seminars at some pro level shops such as Fotocare and E. P. Levine where I have been paid by attendee fees.
__________
I don't know about Rich's relationship with Drobo, but it's entirely clear when visiting his shop that they are a busy high-functioning shop, run by one of the most knowledgeable tech gurus around. There's simply no way that he would use Drobos if the units were sub-par. His investment in plant and equipment if many times larger than either of ours, and the cost of data storage is a much smaller percentage of the business.
>"Also, as I understand it, the Drobo offers no data security during the rebuild of a failed drive, or when adding a new one. A decent RAID system maintains data integrity/security during that process, because the drives are working much harder during that phase than at other times, and if a drive fails during the rebuild/expansion phase, you're SOL. Has Drobo upgraded their technology since I last checked?"
You are incorrect about this. All RAID 5 is exposed during a rebuild. If a drive goes down. there is no parity (data protection) for some chunk of data, and you are unprotected until it is rebuilt. RAID 6 adds to this protection by creating a second parity file, which can survive the loss of two drives.
(Also note that with RAID 5 there is *no* "expansion phase". The only way to expand RAID 5 is to transfer all data, pull all drives, buy and format new ones, and then transfer data back on.)
The 5 drive and larger Drobos will add a second parity copy of the data, allowing the unit to survive the loss of 2 drives, like a RAID 6, but without the huge problems RAID 6 encounters in upgrade. For these units, there is protection during both expansion and during rebuild.
Your information seems to be several years out-of-date. A quick visit to the Drobo product page will reveal that the company has extended their line of products way beyond the needs of photographers into the world of enterprise storage, medical data storage and more. As a matter of fact, their enterprise storage devices are pretty sophisticated. You can run an enterprise-class SAN off it.
It would be a lot like saying, "well, Nikon sucks because they only have 12 megapixel cameras."
________
Both Wiebetech and Drobo offer some very good storage solutions, including some that go way beyond the general needs of photographers. In both cases, a single box should be considered a single device which can fail totally, with no possibility of recovery (falling off a table while running could easily do this).
And, as Bill points out, for many people, a JBOD will be the best solution - cheap, easy to spec, implement, diagnose and upgrade. The primary copy of my still images continues to reside on a JBOD.
But many people simply need drive spanning to do their daily work. For these people, a choice needs to be made between the various options. And Drobo is a reasonable option. None of the criticisms you level at the company are particularly relevant to the current product line, except the cheapest units. (Again, it would be like saying, "Nikon sucks, that D3100 is a really flimsy camera").
Peter
Peter -
As I said, holding out Rich as an example of a consumer/prosumer/pro who uses equipment when he sure seems to have some relationship with them isn't holding out a clearly unbiased person. He may be unbiased, but as of now, it doesn't appear that way. Perhaps you can disclose your fiduciary relationship with him (if there is one) and to what degree he is someone you've used as a sub-contractor for work you're doing, as I understand that's the case. I'm not trying to pick on you Peter, but the lack of disclosure casts a shadow in what is often a good message you deliver.
We can both agree that JBOD is the best solution for most people.
Further, as I said before, there are countless examples online of people having horrible horrible experiences with Drobo units. These recounting of experiences go far and away beyond the errant nut job who has a grudge against the company - there is a clear discourse across many pro's (and prosumers) recounting that Drobo units have been problematic and in many cases catastrophic to their data.
I will concede that when I spent time looking at the Drobos, their proprietary format was (and remains) a non-starter for me, and I did look at them several times after that, however, not recently.
When Drobo goes the way of the Zip Drive and the Jazz Drive, I'll be sure to remind you of how bad Drobo was, and I'll hope that you migrated whatever data you had on your Drobos to a newer/better system. As someone who spent days copying data off Zips and Jazz's, onto standard drives, I know all too well the risks of a proprietary drive system as technology evolves. And, I'll point out, in a nod to you, that you do preach the importance of migrating to new technology. For me, I just wouldn't be caught in the Drobo dead-end because I'll never know if I can back out of that alley with my data intact, and that's not a risk I'm willing to take.
>Perhaps you can disclose your fiduciary relationship with him (if there is one) and to what degree he is someone you've used as a sub-contractor for work you're doing, as I understand that's the case.
You make it sound so nefarious. Rich is writing some copy that I am editing as the director of the dpBestflow.org project - a creative-commons-licensed project funded originally by the Library of Congress with the sole mission to help photographers understand how to preserve their images.
He (and I) are working at well-below market rates (and even the cost of doing business) to help photographers navigate the difficulties of digital workflow. I'm sure you can find some way to make that sound unethical.
>"there are countless examples online of people having horrible horrible experiences with Drobo units."
Yes, I've heard lots of these. And I've heard even more from people who have had entire conventional RAIDs go down. Any drive spanning should be considered somewhat dangerous, and people should only do it because there is a reason to, and have good backups.
>I will concede that when I spent time looking at the Drobos, their proprietary format was (and remains) a non-starter for me,
I assume that you always say the same thing about any RAID 5 or RAID 6 or RAID 10 - BECAUSE THEY ARE ALL PROPRIETARY!!!!!!! (Yes, I'm shouting).
>When Drobo goes the way of the Zip Drive and the Jazz Drive, I'll be sure to remind you of how bad Drobo was, and I'll hope that you migrated whatever data you had on your Drobos to a newer/better system. As someone who spent days copying data off Zips and Jazz's, onto standard drives, I know all too well the risks of a proprietary drive system as technology evolves.
Hmmm, I'm not sure if you really don't understand this, or if you just like to argue. EVERY drive or drive array is a temporary place to park your data. EVERY SINGLE ONE. I'm no more afraid of Drobos becoming obsolete than I am of any drive becoming unreadable. (BTW, I use mine for backup, not for primary storage, but not because I'm afraid of them suddenly becoming proprietarily unreadable)
And anyone who thinks that drives in a spanned array can be put into another box is fooling themselves. Drobos are probably marginally more likely to be able to moved to a new box than "RAID5" , due to the high number of compatible units out there.
This really should not be that hard to understand.
Peter
Peter -
Relax - there's no need to shout. You've gotten all up in arms here, and I have simply been asking questions because your disclosures about free hardware and financial relationships between you and Rich were not disclosed at a time when you were defending Drobo units and citing Rich as an example of why people should use Drobo units. It think it's fair and reasonable to question the message when there is money and freebies that *could* taint the message, and those facts were not disclosed.
Had you said "Rich Harrington* uses Drobos and continues to do so..." and added at the end with the asterisk "*Rich Harrington works for me as a sub-contractor and appears on the Drobo website in their customer testimonials page and also has arranged discounts for his blog readers with Drobo..." I think people would have taken away a very different message about the potential bias in the statements you were making. Had you said "I use Drobos*...." and then added in the end "...Drobo has provided me with free equipment to test and use and from which I provided them with feedback on their equipment..." or, more to the motive "...Drobo, in recognizing that I am a leader in the industry on the subject of workflow, provided me with free equipment to test and use, knowing that if I did, there was a high probability I would speak well of the equipment and recommend it...a recommendation that became an endorsement which they didn't have to pay for aside from some free equipment..." that would have been an even more insightful disclosure (and probably closer to the truth and motives on their part).
As to dpBestflow - I am very aware and supportive of the mission of the project (our blog post on it's launch with your photo here) - one which was - in part - funded by the LOC - specifically a funds-matching program where most of ASMP's "matching" was in the form of in-kind contributions. The LOC, which provided the matching funds following an application by ASMP for those funds, no longer is supporting the program, and has no plans to do so in the future. dpBestflow is an ASMP project, that matching funds by the LOC following an application for those funds, supported, and which they are no longer supporting. So, yes, you're the new Director of an ASMP project that the LOC granted matching funds to before you were director, and they no longer are supporting it financially. In 2009, you earned, according to ASMP records here over $36,000 - a substantial sum for part time work for many photographers, especially at that time. The then Director made over $50,000. I present all of this because you've put forth a statement that implies strongly that you have been given some form of a stamp of approval by the LOC, when it was the application for funds that was, and you were the recipient of that largess after the award - a program, I say again, that was not renewed. (It should have been, in my opinion.)
As I said before, we'll have to agree to disagree on the benefits and risks of Drobo units.
Regards,
John
I wanted to comment on the argument about proprietary file systems.
I have personally used ReadyNAS systems since before they were bought by Netgear and have been a satisfied customer for many years. Besides that I have no relationship with them.
http://www.readynas.com
Drives can be pulled from the ReadyNAS and the RAID5 filesystem mounted in a Linux machine for data recovery. There are several tutorials available for doing so and one forum member even created custom VMware images for readers to run in a virtual machine to recover data without having knowledge of how to install Linux.
Their developers have confirmed the following in the forums:
"There is no “proprietary” filesystem running on any ReadyNAS. It is “straight”, ordinary, unmodified EXT3. You can even run all the e2fsprogs (debugfs, e2fsck, etc) on a 16KB ReadyNAS filesystem on a standard Linux PC with no modifications. Mounting a 16KB ReadyNAS filesystem on an x86 PC requires about the same amount of work as mounting an NTFS filesystem read/write on Linux — using a FUSE driver to mount it."
I've never had the need to do it myself, but with the number of people who have and the tutorials available, I don't think I'd have any trouble.
This is one of the reasons that I keep ReadyNAS at the top of my list when I'm looking at storage options.
I have been wanting to start using a drobo system myself...
Post a Comment