Codewise.org

Available at GitHub and Codewise.org.

BackFS is a FUSE read-only filesystem designed to provide a large local disk cache for remote network filesystems.

When mounting, you specify the remote filesystem's mount point, the path which it should use as its cache (probably a separate filesystem), and optionally the size to which that cache should grow. If you don't specify a size, the cache will grow to fill the device it is on (this is why you should use a separate FS for the cache ;). When the cache is full, adding new entries will remove the least recently accessed entries to make room. The cache is persistent across unmounts; all data structrures are stored on-disk alongside the cache. Cache data is automatically invalidated when the backing store's data is changed.

BackFS currently has basic functionality, but it is in active development, and I have many more features planned, so consider it in beta right now.

Planned features:

  • eventually, I'd like to make BackFS a write-through cache instead of read-only.

See the GitHub project page for a more detailed description, including all the gory implementation details.

My motivation: Codewise.org used to be primarily run from a Linode VPS server, which had much cheaper bandwidth than it did disk space. (It was actually cheaper to just get more Linodes than to expand the disk space of an existing one — their pricing structure was very bizarre.) On the other hand, I had a server in my apartment with terabytes of disk space, but a much slower Internet connection.

BackFS arose as a way to serve the huge amount of data on my home server with more reasonable speed. Frequently accessed data would be ready in the cache for immediate serving, and less-frequently accessed data would be pulled down on an as-needed basis. BackFS allows a good compromise between the limited fast local storage and the virtually unlimited slow backing store.