{"id":1748,"date":"2013-05-13T21:12:31","date_gmt":"2013-05-14T02:12:31","guid":{"rendered":"http:\/\/www.kickflop.net\/blog\/?p=1748"},"modified":"2013-05-13T21:12:31","modified_gmt":"2013-05-14T02:12:31","slug":"memory-backed-filesystem-for-temporary-storage-of-whisper-data","status":"publish","type":"post","link":"https:\/\/www.kickflop.net\/blog\/2013\/05\/13\/memory-backed-filesystem-for-temporary-storage-of-whisper-data\/","title":{"rendered":"Memory-backed Filesystem for Temporary Storage of Whisper Data"},"content":{"rendered":"

Instead of buying and installing SSDs, storing Graphite’s whisper files in a memory-backed filesystem can be a good way to go if you have the RAM to spare. Depending on your environment, you may or may not care about losing a few minutes (or hours) of metric data. I know we certainly don’t care about losing 30 minutes, so there’s no reason for our carbon-cache instances to be scrawling to persistent storage 24\/7.<\/p>\n

Here’s the internal carbon Average Update Time metric showing the switch from writing to spinning disk and then memory. No shocker here:<\/p>\n

\"tmpfs-graphite-avgUpdateTime\"<\/a><\/p>\n

Likewise, it’s no shocker that CPU wait time was greatly reduced. You can see me testing 5 rsyncs from memory-backed filesystem to the old spinning disk in the right of the graph:<\/p>\n

\"tmpfs-graphite-cpuWait\"<\/a><\/p>\n

You have 2 commonly found implementation choices for your memory-backed filesystem: tmpfs or a standard RAM disk with non-journaling filesystem applied<\/p>\n\n\n\n\n\n\n\n\n
 <\/th>\ntmpfs<\/th>\nRAM disk<\/th>\n<\/tr>\n<\/thead>\n
Create w\/o reboot<\/td>\nyes<\/td>\nno<\/td>\n<\/tr>\n
Resize w\/o reboot<\/td>\nyes<\/td>\nno<\/td>\n<\/tr>\n
Dynamically allocated<\/td>\nyes<\/td>\nno<\/td>\n<\/tr>\n
Can be swapped<\/td>\nyes<\/td>\nno<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

tmpfs<\/h2>\n

The Linux tmpfs<\/code> implementation would look something like the following.<\/p>\n

    \n
  1. Stop your carbon-cache services, then:\n
    \r\ncd \/opt\/graphite\/storage\r\nmv whisper whisper.permanent\r\nmkdir whisper.ephemeral\r\nln -s .\/whisper.ephemeral .\/whisper\r\n\r\necho 0 > \/proc\/sys\/vm\/swappiness\r\n\r\nmount -t tmpfs -o size=5g tmpfs \/opt\/graphite\/storage\/whisper.ephemeral\r\n<\/pre>\n<\/li>\n
  2. Add the following, modified to suit, to your \/etc\/rc.local<\/code> or other preferred boot-time mechanism:\n
    \r\nmount -t tmpfs -o size=5g tmpfs \/opt\/graphite\/storage\/whisper.ephemeral\r\n<\/pre>\n<\/li>\n
  3. One-way synchronize the persistent storage with the state of the ephemeral storage every 30 minutes:\n
    \r\n*\/30 * * * * rsync --archive \/opt\/graphite\/storage\/whisper.ephemeral\/ \/opt\/graphite\/storage\/whisper.permanent\r\n<\/pre>\n<\/li>\n<\/ol>\n
    Explicit RAM disk<\/h2>\n
    To implement the more predictable RAM disk setup, you would:<\/p>\n
      \n
    1. Add ramdisk_size=N<\/code> (where N is the size of the RAM disk you want specified in KB) to the kernel<\/code> line in grub.conf<\/code> for your distribution. Reboot.<\/li>\n
    2. Create a filesystem on the RAM disk device and mount it. You pointedly do not want to waste efficiency by using a journaling filesystem here, so use an ext2<\/code> filesystem and do not reserve any “minfree” space.\n
      \r\ncd \/opt\/graphite\/storage\r\nmv whisper whisper.permanent\r\nmkdir whisper.ephemeral\r\nln -s .\/whisper.ephemeral .\/whisper\r\n\r\nmkfs.ext2 -m 0 \/dev\/ram0\r\nmount \/dev\/ram0 \/opt\/graphite\/storage\/whisper.ephemeral\r\n<\/pre>\n<\/li>\n
    3. Add the following, modified to suit, to your \/etc\/rc.local<\/code> or other preferred boot-time mechanism:\n
      \r\nmkfs.ext2 -m 0 \/dev\/ram0\r\nmount \/dev\/ram0 \/opt\/graphite\/storage\/whisper.ephemeral\r\n<\/pre>\n<\/li>\n
    4. One-way synchronize the persistent storage with the state of the ephemeral storage every 30 minutes:\n
      \r\n*\/30 * * * * rsync --archive \/opt\/graphite\/storage\/whisper.ephemeral\/ \/opt\/graphite\/storage\/whisper.permanent\r\n<\/pre>\n<\/li>\n<\/ol>\n
      As always, I welcome your thoughts.<\/p>\n","protected":false},"excerpt":{"rendered":"
      Instead of buying and installing SSDs, storing Graphite’s whisper files in a memory-backed filesystem can be a good way to…<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[51,35,11,48],"tags":[],"class_list":["post-1748","post","type-post","status-publish","format-standard","hentry","category-devops","category-linux","category-sysadmin","category-unixlinux"],"_links":{"self":[{"href":"https:\/\/www.kickflop.net\/blog\/wp-json\/wp\/v2\/posts\/1748","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kickflop.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kickflop.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kickflop.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kickflop.net\/blog\/wp-json\/wp\/v2\/comments?post=1748"}],"version-history":[{"count":10,"href":"https:\/\/www.kickflop.net\/blog\/wp-json\/wp\/v2\/posts\/1748\/revisions"}],"predecessor-version":[{"id":1773,"href":"https:\/\/www.kickflop.net\/blog\/wp-json\/wp\/v2\/posts\/1748\/revisions\/1773"}],"wp:attachment":[{"href":"https:\/\/www.kickflop.net\/blog\/wp-json\/wp\/v2\/media?parent=1748"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kickflop.net\/blog\/wp-json\/wp\/v2\/categories?post=1748"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kickflop.net\/blog\/wp-json\/wp\/v2\/tags?post=1748"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}