Memory-backed Filesystem for Temporary Storage of Whisper Data

Instead of buying and installing SSDs, storing Graphite’s whisper files in a memory-backed filesystem can be a good way to go if you have the RAM to spare. Depending on your environment, you may or may not care about losing a few minutes (or hours) of metric data. I know we certainly don’t care about losing 30 minutes, so there’s no reason for our carbon-cache instances to be scrawling to persistent storage 24/7.

Here’s the internal carbon Average Update Time metric showing the switch from writing to spinning disk and then memory. No shocker here:

tmpfs-graphite-avgUpdateTime

Likewise, it’s no shocker that CPU wait time was greatly reduced. You can see me testing 5 rsyncs from memory-backed filesystem to the old spinning disk in the right of the graph:

tmpfs-graphite-cpuWait

You have 2 commonly found implementation choices for your memory-backed filesystem: tmpfs or a standard RAM disk with non-journaling filesystem applied

  tmpfs RAM disk
Create w/o reboot yes no
Resize w/o reboot yes no
Dynamically allocated yes no
Can be swapped yes no

tmpfs

The Linux tmpfs implementation would look something like the following.

  1. Stop your carbon-cache services, then:
    cd /opt/graphite/storage
    mv whisper whisper.permanent
    mkdir whisper.ephemeral
    ln -s ./whisper.ephemeral ./whisper
    
    echo 0 > /proc/sys/vm/swappiness
    
    mount -t tmpfs -o size=5g tmpfs /opt/graphite/storage/whisper.ephemeral
    
  2. Add the following, modified to suit, to your /etc/rc.local or other preferred boot-time mechanism:
    mount -t tmpfs -o size=5g tmpfs /opt/graphite/storage/whisper.ephemeral
    
  3. One-way synchronize the persistent storage with the state of the ephemeral storage every 30 minutes:
    */30 * * * * rsync --archive /opt/graphite/storage/whisper.ephemeral/ /opt/graphite/storage/whisper.permanent
    

Explicit RAM disk

To implement the more predictable RAM disk setup, you would:

  1. Add ramdisk_size=N (where N is the size of the RAM disk you want specified in KB) to the kernel line in grub.conf for your distribution. Reboot.
  2. Create a filesystem on the RAM disk device and mount it. You pointedly do not want to waste efficiency by using a journaling filesystem here, so use an ext2 filesystem and do not reserve any “minfree” space.
    cd /opt/graphite/storage
    mv whisper whisper.permanent
    mkdir whisper.ephemeral
    ln -s ./whisper.ephemeral ./whisper
    
    mkfs.ext2 -m 0 /dev/ram0
    mount /dev/ram0 /opt/graphite/storage/whisper.ephemeral
    
  3. Add the following, modified to suit, to your /etc/rc.local or other preferred boot-time mechanism:
    mkfs.ext2 -m 0 /dev/ram0
    mount /dev/ram0 /opt/graphite/storage/whisper.ephemeral
    
  4. One-way synchronize the persistent storage with the state of the ephemeral storage every 30 minutes:
    */30 * * * * rsync --archive /opt/graphite/storage/whisper.ephemeral/ /opt/graphite/storage/whisper.permanent
    

As always, I welcome your thoughts.

Filed under DevOps, Linux, Sysadmin, UNIX/Linux