Q:

How can I build a file server that we can incrementally grow, in Linux?

I work in an scientific lab with a limited budget and ever-growing need for storage. A year ago we needed ~2-3 TB storage, today we need 13+ TB which fills our current server (linux, raid 6 with 9 drives), and it will only keep growing.  The files are huge - 50GB+ each. 
I want to build a server that:
a) Can handle uneven disk sizes, so we can replace older drives with larger ones as they become available in the market.
b) Can handle additions of disks (that are potentially larger than any of other disk) after creation of the initial "volume".  Ideally, I just want to pop in a drive into a hotswap bay and make it part of the volume.
b) Has redundancy and can handle multiple disk failures.
d) Quick 'fscking'. Last time our current server did this it took forever to get back up.
Can I do (a) and (b) with RAID?  I know I can partition equal sized raid-partitions from unevenly sized disks, but I do not want to get into the business of micromanaging partitions with multiple raid arrays.  Is ZFS an option? (FreeBSD would be acceptable too.)
Performance, honestly, is not much of an issue.  It will only store and serve static content to a handful of researchers.  Our LAN is 1Gbit and our WAN is only 100Mbit.  
Any and all suggestions welcome.

A:

I would recommend ZFS-based solutions, but running a purpose-built OS (NexentaStor), versus trying to run additional applications on the system. This gives you the flexibility to treat your storage as an appliance and eliminates application dependencies. Export to your Linux systems via NFS or iSCSI.
The remainder of your requirements are well-covered by ZFS solutions. Do you have a budget established?
I recommend going with an integrator/partner who can help design a robust system and alleviate any scaling/longevity issues. Chances are that they've seen a situation like yours or have dealt with similar requirements. However, if you go it on your own, do your due diligence and avoid the mistakes others have made. 
Good places to start:
http://www.zfsbuild.com/
http://hardforum.com/showthread.php?t=1573272
http://www.nex7.com/readme1st

A:

Divorce your storage needs from your processing needs. You can purchase a san for relatively low prices (HP P2000's are cheap if you're not buying them fully populated, and you can expand them up to 6 shelves of 3TB disks, for 36x6 TB of storage RAW), and if you stick with iSCSI you can avoid costly HBAs and fibre. iSCSI would also let you continue utilizing the server you're currently using as your file server, it just lets you attach a bunch of disks. You can also look at older models like MSA 2000's. Dell also has some offerings which are decent. Then, connect it to your server with a few initial disks (2 for raid 1, 3 for raid 5 or raid 1e, 4 for raid 1+0 or raid 6 depending on your storage requirements). 
On the server, use LVM to make a physical volume out of these, and create a volume group/logical volume on top of that. In the future you can add additional disks to the SAN chassis, and add them as new physical volumes, and then expand the logical volume. Then when you max out capacity of your SAN's disk shelf, you just add another shelf with a couple disks, and grow the same way as the first shelf. We use this for many of our storage systems. Some careful planning can help you keep from having data for the same logical volume hosted on two different shelves (in case some idjit decides they want to trip over a cable connecting the two san shelves). 
For the filesystem, ext3, ZFS, ext4 all support increasing the size of the filesystem on the fly. However if you are ever concerned you'd need to shrink it, lean towards ext3. It's better documented and supported. This solution is simple, doesn't require a consultant, and allows you to grow your solution with your dataset.
You really want to remove Storage from your server though if you need to expand it so much.

