I work with big files. Really big. Daily, I back up a file that is larger than 200 gigabytes. We have a Thecus N7700 network attached storage (NAS) box that holds about seven terabytes. Every day I copy the latest stuff there and delete some of the older files. It all works fine except for just one little problem: deleting a 200 gigabyte file takes a long time and interrupts other processing.
How long? More than a minute. Seriously. And during that time, any other process that is trying to access files on the NAS gets really slow. Sometimes, deleting the file causes the NAS to become unresponsive so long that other processes’ IO requests time out and the program crashes. That is not a happy state of affairs when I’m running a job that takes 36 hours.
It appears that the slowdown is due to indirect block pointer updates in the ext3 file system, as described in this post.
Is this a fundamental shortcoming of the ext3 file system? If it is, what are my options? The Thecus supports a file system called ZFS, but from what I’ve read about it online, I don’t want to go down that path. I wonder if a firmware upgrade would solve my problem.
I know nothing about ext3, being a Windows user, but after reading about it a bit it sure sounds like your problem is just the way ext3 is. I know just about as much about ext4, but the Wikipedia article on it says “Extents are introduced to replace the traditional block mapping scheme used by ext2/3 filesystems. An extent is a range of contiguous physical blocks, improving large file performance and reducing fragmentation.” And that sure sounds like it relates to your problem. Of course ext4 may not be supported, or might have its own set of problems.
Jim, what is it you don’t like about ZFS?
Roy: Yes, that’s pretty much what I determined in my research: “That’s the way ext3 is.” Unfortunately, ext4 is not an option.
Darrin: I have no direct experience with ZFS. Information I’ve read online indicates that it’s slower than ext3, doesn’t support ACL (not something I need right now) or NFS, and worst of all there are documented problems with the implementation on the Thecus N7700. It looks like there is upgraded firmware for the N7700 that supports XFS, which seems to be the preferred option.
Upgrading the firmware, backing up 7 TB of data, and rebuilding the RAID will be quite a job . . .
This thought comes from left field so it probably will not fit your situation. But there is always a chance…
If the files are only backup copies, and not referenced other than for (1) restore or (2) delete, then just maybe if they were compressed the smaller files would be quicker to delete. I looked at one tool a couple of years ago that looked promising for on-the-fly compression, HyperBac. Despite the fact that it is marketed for compressing backups it actually appeared to be far more flexible than that. Quoting a friend… “Strictly speaking it’s not even really a database backup product – it’s really a file system layer filter driver – the fact it’s compressing “database backups” is irrelevant really. They’re just checking, for licensing purposes, the name of the exe (e.g. bcp.exe, sqlserver.exe etc.) to see whether to compress a file it produces.”
The files are not “only backup copies”. The large files are archived for a week or two while we run reports against them. Periodically (end of month, for example), we’ll save one for longer term. But the files are already highly compressed. The uncompressed data is approximately 2.5 times as large.
I have no doubt that the files would be deleted faster if they were smaller. But even if compression knocked 50% off the size of the file, it’d still take more than 30 seconds to delete. And the file grows a bit every day . . .