rpc.mountd segfaults when trying to mount nfs share

[brandon.dacrib]

I have a pair of TS859U-RP devices that are experiencing the same failure. When I try to mount an nfs share there is a timeout on the client
[root@somehost ~]# mount nasb:/content /mnt/nasb/blah/
mount: mount to NFS server 'nasb' failed: timed out (retrying).
mount: mount to NFS server 'nasb' failed: timed out (retrying).

There is also an error written to /mnt/HDA_ROOT/.logs/kmsg on the nas like:
<6>rpc.mountd[522]: segfault at b73aa078 ip b76fdf56 sp bf8b03a4 error 7 in libuLinux_statistics.so.0.0[b76fd000+3000]
<6>rpc.mountd[6539]: segfault at b7418078 ip b776bf56 sp bfd69a94 error 7 in libuLinux_statistics.so.0.0[b776b000+3000]
... repeating

an error email is generated:
Server Name: nasb
IP Address: x.x.x.x
Date/Time: 2011/03/28 15:02:55
Level: Warning
Re-launch process [rpc.mountd].

and the device momentarily stops serving nfs requests to all hosts. This device has that very same nfs share mounted on 23 other hosts and functioning as confirmed by running "netstat -a | grep nfs | grep ESTAB | wc -l " on the nas. Is there some soft limit to the number of hosts that can mount a particular nfs share? Either way it shouldn't segfault but return some error.

This is a raid6 volume formatted with ext4, and using firmware 3.4.1 Build0315(latest). I tried contacting support a few times but have gotten nothing but the standard upgrade/reapply your firmware and reboot response. No satisfaction

[schumaku]

Wild guess as there is zero indication of the nfs client system in use. Asuming its Linux or other U**x type supporting nfsv4 already, try this:

# mount -o vers=3 ....

Answered at least a dozen times - searching the forum should help.

However: Of course this kind of activities should not crash rpc.mountd - please provide more details!

[brandon.dacrib]

The clients are running CentOS and RHEL 5. They default to using nfs version 3, as evinced by running "nfsstat -c" and seeing that only nfs version 3 is in use on any of my clients. If you or anyone could point me to specific metrics or data points that would be of assistance then I will gladly grab them and post here.

[brandon.dacrib]

I also tried getting an strace on the nas when the segfault happens: http://pastebin.com/GKvVV1tQ I don't see anything obvious right before the segfault but that could be due to my lack of interpretation skills. Much respect to anyone that can take a look at that and give some insight.

[schumaku]

The strace does not look that bad for me, too.

Does the libuLinux_statistics.so.0.0 roughly matching to one day before 0315 on your NAS?

[/] # ls -ls /usr/lib/libuLinux_statistics.so*
0 lrwxrwxrwx 1 admin administ 25 Mar 23 15:37 /usr/lib/libuLinux_statistics.so -> libuLinux_statistics.so.0
0 lrwxrwxrwx 1 admin administ 27 Mar 23 15:37 /usr/lib/libuLinux_statistics.so.0 -> libuLinux_statistics.so.0.0
12 -rw-r--r-- 1 admin administ 12072 Mar 14 17:17 /usr/lib/libuLinux_statistics.so.0.0

[brandon.dacrib]

actually no mine doesn't

Code: Select all

 
[~] # ll /usr/lib/libuLinux_statistics.so
lrwxrwxrwx    1 admin    administ       25 Feb 25 16:25 /usr/lib/libuLinux_statistics.so -> libuLinux_statistics.so.0
[~] # md5sum /usr/lib/libuLinux_statistics.so
b971406f9f7b0f181f236cc8ac6baee4  /usr/lib/libuLinux_statistics.so


same file with a different date on my the second nas

Code: Select all

[~] # ll /usr/lib/libuLinux_statistics.so
lrwxrwxrwx    1 admin    administ       25 Nov 16 06:10 /usr/lib/libuLinux_statistics.so -> libuLinux_statistics.so.0
[~] # md5sum /usr/lib/libuLinux_statistics.so
b971406f9f7b0f181f236cc8ac6baee4  /usr/lib/libuLinux_statistics.so

does your md5sum match what I have?

[brandon.dacrib]

I should note that these are all the same file:

Code: Select all

[~] # md5sum /usr/lib/libuLinux_statistics.so*
b971406f9f7b0f181f236cc8ac6baee4  /usr/lib/libuLinux_statistics.so
b971406f9f7b0f181f236cc8ac6baee4  /usr/lib/libuLinux_statistics.so.0
b971406f9f7b0f181f236cc8ac6baee4  /usr/lib/libuLinux_statistics.so.0.0


[schumaku]

Yes, two are just symlinks:

[/] # ls -ls /usr/lib/libuLinux_statistics.so*
0 lrwxrwxrwx 1 admin administ 25 Mar 23 15:37 /usr/lib/libuLinux_statistics.so -> libuLinux_statistics.so.0
0 lrwxrwxrwx 1 admin administ 27 Mar 23 15:37 /usr/lib/libuLinux_statistics.so.0 -> libuLinux_statistics.so.0.0
12 -rw-r--r-- 1 admin administ 12072 Mar 14 17:17 /usr/lib/libuLinux_statistics.so.0.0

While the md5sum on that file is the same, potentially it has not changed for a while ...

[/] # md5sum /usr/lib/libuLinux_statistics.so*
b971406f9f7b0f181f236cc8ac6baee4 /usr/lib/libuLinux_statistics.so
b971406f9f7b0f181f236cc8ac6baee4 /usr/lib/libuLinux_statistics.so.0
b971406f9f7b0f181f236cc8ac6baee4 /usr/lib/libuLinux_statistics.so.0.0

...however...

...and using firmware 3.4.1 Build0315(latest). ...

...that does not match to the two file dates of your symlinks (Feb 25 16:25 and Nov 16 06:10). Unless I'm fully out of my ** (watching Swiss ice hockey semi-final, last game from a best-of-seven series Kloten vs. Berne), I tend to state

1. The revision date of the effective file /usr/lib/libuLinux_statistics.so.0.0 must match the firmware build date.
2. The revision date of the symlinks must match the effective boot time.

At least the second point is not correct on your systems. Have rebooted after updating the firmware?

[brandon.dacrib]

Hmm... Veeery interesting... I can try another reboot and see if anything changes.. I will post back in the morning.

[brandon.dacrib]

I tried re applying the firmware update and rebooting... this time the date on that file actually did change
[~] # ll /usr/lib/libuLinux_statistics.so*
lrwxrwxrwx 1 admin administ 25 Mar 30 04:13 /usr/lib/libuLinux_statistics.so -> libuLinux_statistics.so.0
lrwxrwxrwx 1 admin administ 27 Mar 30 04:13 /usr/lib/libuLinux_statistics.so.0 -> libuLinux_statistics.so.0.0
-rw-r--r-- 1 admin administ 11.8k Mar 14 15:05 /usr/lib/libuLinux_statistics.so.0.0

I can try moving all of those mounts back over to this box tonight to see if all is well... but it definitely appears to have updated that file this time Thank you very much for your help in narrowing this down.