A seemingly simple move of NFS-exported home directories resulted in Apache suEXEC freaking out when referencing the new space. Here are the details and solution, because sometimes a debugging story is fun.
\n<\/p>\n
Setting: R&D environment of ~800 physical and virtual hosts running Windows, Linux, Solaris and using DAS, NAS, and SAN with NFS and OpenAFS. Our direct customers are scientists, researchers, and software and hardware engineers.<\/em><\/p>\n A small portion of our users, say 20%, have NFS-based home directories. Over the last ~18 years these have been served from varied hosts running SunOS 4.1 – 5.10. As part of our department’s ongoing weening from Sun\/Oracle hardware, it was recently finally time to get this NFS data moved from the host (Sun-Fire V240). Our Isilon IQ 7200x 3-node cluster, which I’d personally had only limited interaction with (and no training with) to date, seemed the appropriate destination for the data.<\/p>\n After moving the data (twice[1<\/a>]), changing automounter maps in LDAP, and successfully performing basic usability testing, I called it a night and ended the Planned Outage event. Because what better time to make changes than Friday at 10PM, right?<\/p>\n Jimbo: “Our project’s CGI scripts in Perl at http:\/\/lab.our.org\/~jimbo\/cgi-bin, that have been working fine for 8 years, are all giving 500 Internal Server errors. We use these scripts as part of a high-value production process.”<\/p>\n Let’s ignore for now everything wrong with the sentence above. I know.<\/em><\/p>\n I SSHed into Then I noticed that Okay, that explains the 500 HTTP errors. However:<\/p>\n From here, scratching my head for 30 minutes or so, I mucked around in the Isilon OneFS ACL for According to the Monday<\/h2>\n
lab.mitre.org<\/code> and had a look through recent data in \/var\/log\/httpd\/lab-error_log<\/code>. It showed nothing interesting.<\/p>\nsuexec_log<\/code> had been written to recently. Having no idea what suexec_log even was, I looked through it to find that it’s the log for Apache httpd’s suEXEC<\/a> mechanism. While I’m not entirely Apache httpd clueless, I don’t exactly deal with every nook of it on even a yearly basis. I read up on suEXEC’s role and functionality then returned to suexec_log<\/code>:<\/p>\n\r\ncannot stat directory: (\/home\/jimbo\/public_html\/cgi-bin)\r\n<\/pre>\n
\r\nlab.our.org# cd \/home\/jimbo\/public_html\/cgi-bin\r\nlab.our.org# pwd\r\n\/home\/jimbo\/public_html\/cgi-bin\r\nlab.our.org#\r\n<\/pre>\n
\/home\/jimbo<\/code>, somewhat convinced I’d hit an ACL peculiarity like the previous one<\/a>. That led nowhere, so it was time to examine just what the heck Apache httpd and its suexec<\/code> helper were doing under the covers. Firing up strace -fq ... <\/code> on the httpd PIDs while hitting one of Jimbo’s troubled CGI scripts, I saw this:<\/p>\n\r\n...\r\nsetgroups32(2, [1130, 67]) = 0\r\nsetuid32(21389) = 0\r\ngetcwd(\"\/home\/jimbo\/public_html\/cgi-bin\"..., 4096) = 32\r\nchdir(\"\/home\/jimbo\") = 0\r\nchdir(\"public_html\") = 0\r\ngetcwd(\"\/home\/jimbo\/public_html\"..., 4096) = 24\r\nchdir(\"\/home\/jimbo\/public_html\/cgi-bin\") = 0\r\nlstat64(\"\/home\/jimbo\/public_html\/cgi-bin\", {st_mode=S_IFDIR|0755, st_size=1955, ...}) = 0\r\nwrite(2, \"suexec policy violation: see sue\"..., 57) = 57\r\n<\/pre>\nlstat()<\/code> man page, a return code of 0 is success. The line following the lstat()<\/code> call is writing to stderr that there was a suexec problem and to see our friend suexec_log<\/code> for information. We already know how that goes. For some reason, lstat()<\/code> was returning what we understand to be success and there was no further system call taking place before the code was logging an error line. Funky.<\/p>\n