Memory-backed Filesystem for Temporary Storage of Whisper Data

Instead of buying and installing SSDs, storing Graphite’s whisper files in a memory-backed filesystem can be a good way to go if you have the RAM to spare. Depending on your environment, you may or may not care about losing a few minutes (or hours) of metric data. I know we certainly don’t care about losing 30 minutes, so there’s no reason for our carbon-cache instances to be scrawling to persistent storage 24/7.

Here’s the internal carbon Average Update Time metric showing the switch from writing to spinning disk and then memory. No shocker here:

Likewise, it’s no shocker that CPU wait time was greatly reduced. You can see me testing 5 rsyncs from memory-backed filesystem to the old spinning disk in the right of the graph:

You have 2 commonly found implementation choices for your memory-backed filesystem: tmpfs or a standard RAM disk with non-journaling filesystem applied

tmpfs

The Linux tmpfs implementation would look something like the following.

1. Stop your carbon-cache services, then:

   cd /opt/graphite/storage
   mv whisper whisper.permanent
   mkdir whisper.ephemeral
   ln -s ./whisper.ephemeral ./whisper
   
   echo 0 > /proc/sys/vm/swappiness
   
   mount -t tmpfs -o size=5g tmpfs /opt/graphite/storage/whisper.ephemeral
   

2. Add the following, modified to suit, to your /etc/rc.local or other preferred boot-time mechanism:

   mount -t tmpfs -o size=5g tmpfs /opt/graphite/storage/whisper.ephemeral
   

3. One-way synchronize the persistent storage with the state of the ephemeral storage every 30 minutes:

   */30 * * * * rsync --archive /opt/graphite/storage/whisper.ephemeral/ /opt/graphite/storage/whisper.permanent
   

Explicit RAM disk

To implement the more predictable RAM disk setup, you would:

1. Add ramdisk_size=N (where N is the size of the RAM disk you want specified in KB) to the kernel line in grub.conf for your distribution. Reboot.
2. Create a filesystem on the RAM disk device and mount it. You pointedly do not want to waste efficiency by using a journaling filesystem here, so use an ext2 filesystem and do not reserve any “minfree” space.

   cd /opt/graphite/storage
   mv whisper whisper.permanent
   mkdir whisper.ephemeral
   ln -s ./whisper.ephemeral ./whisper
   
   mkfs.ext2 -m 0 /dev/ram0
   mount /dev/ram0 /opt/graphite/storage/whisper.ephemeral
   

3. Add the following, modified to suit, to your /etc/rc.local or other preferred boot-time mechanism:

   mkfs.ext2 -m 0 /dev/ram0
   mount /dev/ram0 /opt/graphite/storage/whisper.ephemeral
   

4. One-way synchronize the persistent storage with the state of the ephemeral storage every 30 minutes:

   */30 * * * * rsync --archive /opt/graphite/storage/whisper.ephemeral/ /opt/graphite/storage/whisper.permanent
   

As always, I welcome your thoughts.