CIFS vs NFS

[acw]

anyone tried testing CIFS vs NFS? any noticeable performance differences?

I have tried the SAMBA mount, and have some difficulty in file/dir ownership and permission issues. Specifically, the ownership shown in remote mounts are different than what are actually on the actual file system.

Example :

On NAS, this is the dir :

drwxrwxrwx 9 root root 4096 May 17 06:45 .
drwxrwxrwx 4 root root 1024 May 14 13:44 ..
drwxr-xr-x 2 nobody nobody 4096 Apr 22 08:37 bin
-rw-r--r-- 1 root root 16598 May 21 12:55 cifs_log
drwxrwxrwx 3 nobody nobody 4096 May 21 12:28 download
-rw-r--r-- 1 root root 0 Apr 17 08:41 http_access_log
-rw-r--r-- 1 root root 1434 May 21 12:49 http_error_log
drwxr-xr-x 2 nobody nobody 4096 Apr 22 08:38 lighttpd
drwx------ 2 root root 16384 Feb 25 21:25 lost+found
drwxr-xr-x 2 nobody nobody 4096 Apr 22 08:37 usr



but on remote server that's mounting that dir, it's showing as :

drwxrwxrwx 1 root root 4096 May 21 12:55 .
drwxr-xr-x 7 root root 1024 May 14 12:56 ..
drwxrwxrwx 1 root root 0 Apr 22 08:37 bin
-rw-rw-rw- 1 root root 16598 May 21 12:55 cifs_log
drwxrwxrwx 1 root root 0 May 21 12:28 download
-rw-rw-rw- 1 root root 0 Apr 17 08:41 http_access_log
-rw-rw-rw- 1 root root 1434 May 21 12:49 http_error_log
drwxrwxrwx 1 root root 0 Apr 22 08:38 lighttpd
drwxrwxrwx 1 root root 0 Feb 25 20:25 lost+found
drwxrwxrwx 1 root root 0 Apr 22 08:37 usr

[PeterNic]

Yes, Unix-style permissions don't work well in samba (pretty much by design). In fact, we added nfs access to our NAS appliance in the next version of AppLogic and changed the WEB class to access the 'fs' terminal over nfs instead of cifs. The main reason for this was to be able to get permissions correctly; performance is not an issue -- nfs is somewhat better than samba.

I don't have the numbers off the top of my head, but I recall that nfs was about the same as samba on its read performance and somewhat faster than samba in writes. I'll dig up the numbers we saw and post them here.

The only case in which nfs is not well suited is when you have multiple clients modifying the same files... the nfs semantics differ from local fs, while cifs/smb semantics are generally closer to the local file system. If that is a concern, you may also want to check the newer CIFS fs (which is different from the samba client and I recently read is trying to provide full Unix permissions semantics when both the client and the server are Unix).

In all other cases, nfs is a simple solution that can work great. Just try to keep the number of different users to a minimum (root and nobody should be sufficient in most cases); if you need more different users -AND- to access the storage from multiple client appliances, you will need to ensure the user ID mapping is the same.

Regards,
-- Peter

[Pavel]

Here are the performance results:


Small files Big Files
NFS 3.31 18.2
CIFS 3.16 13.1

All values are in MB/s

Big Files test: copy of 820MB file
Small Files test: copy of 66K files with total size ~770MB

Both tests run with:
for i in `seq 1 5`; do rm -rf *; time cp -r /mnt .; done

[acw]

Thanks for the info. We tried NFS, and getting strange result. When we tested copying files from /usr to NFS and /usr to samba mount, we get :


NFS (file 17MB): 2.65 MB/s
CIFS (file 17MB): 2.60 MB/s

NFS (file 89KB): 4.37 MB/s
CIFS (file 89KB): 0.51 MB/s

When we change directory to be in the actual mounts, and then copy from within the mount to another directory within the mount, we get :

(Samba)
[atest]# pwd
/mnt/sambatest/atest
[atest]# time cp -r * ../z/

real 0m54.651s
user 0m0.010s
sys 0m6.710s




(NFS)
[atest]# pwd
/mnt/nfstest/atest
[atest]# time cp -r * ../z1/

real 6m5.941s
user 0m0.000s
sys 0m2.980s


when copying from a local dir to mount (i.e. /usr/localtest is the source), we get :

(SAMBA)
[localtest]# pwd
/usr/download
[localtest]# time cp -r * /mnt/sambatest/z/

real 0m29.424s
user 0m0.020s
sys 0m5.170s


(NFS)
[localtest]# pwd
/usr/download
[localtest]# time cp -r * /mnt/nfstest/z1/

real 5m42.824s
user 0m0.020s
sys 0m2.150s



This is in /etc/fstab for mounting nfs
apps:/mnt/data /mnt/nfstest nfs users,sync,rsize=8192,wsize=8192,nolock 0 0


and this is in /etc/exports :
/mnt/data 10.168.22.0/24(rw,sync,no_wdelay,no_root_squash)

We're getting similar when we remove rsize/wsize.
Can you try similar test and see if you can duplicate this result?

[PeterNic]

The only thing I can see on the surface is the 'sync' attribute in both the client side (mount) and the server side (export). Especially on the server side, this is very brutal: it requires the server to have FLUSHED all data to the disk before completing the write. Note that samba (cifs) doesn't use such mode.

BTW, have you tried copying from the NFS mount to the local dir? It should be much faster...

Also, is the 'sync' write behavior important for your case? Is writing to the fs important?

(Background to the two questions above: If you use the nfs to read files using the web server, you will be doing mostly reads, so the nfs write performance is not very important; if you are primarily using nfs for writing the WEB server logs, the sync behavior is probably not needed -- at worst you will lose the last few entries before a crash)

I will need a bit more info to reproduce the tests: what are the files being copied (1 file/many files; what size(s)). If none of the above suggestions helps, please post the test details here and Pavel will try to reproduce. Also, what class servers is your grid on; can you post the output of 'cat /proc/cpuinfo' here as well?

Regards,
-- Peter

[acw]

Just tried without 'sync' on both server and client sides, and seeing tremendous difference. Now seeing from local to nfs :


real 0m30.035s
user 0m0.010s
sys 0m0.300s


The planned use is to write posted files from web users to the storage, so, I was looking for more guaranteed writes. Maybe we'll address it from within our app. We'll also play around with rsize and wsize and see the impacts and will post here again.



cat /proc/cpuinfo

processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 8
model name : AMD Athlon(tm) Proswssor
stepping : 1
cpu MHz : 1825.976
cache size : 256 KB
fdiv_bug : no
hlt_bug : yes
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu tsc msr pae mce cx8 apic mca cmov pat pse36 mmx fxsr sse pni syscall mp mmxext 3dnowext 3dnow
bogomips : 1661.33