Linux FUSE Internals for developers

In this post, I will cover FUSE internals for FUSE 2.9.3.

• Install package fuse and fuse-devel on CentOS.
• getattr() is a must in a FUSE file-system. Any lame implementation is okay;
  • Just be careful of the file size in stat structure. If you forgot to compile user file system with 64-bit flags on. Otherwise the statst_size is signed int (32 – 1 bit field).
  • Your file size should not exceed > 2GB. Otherwise, it will be overflowed to zero.
• In the user application, be careful with file I/O operations. A read () immediately followed by a write() would fetch you nothing. You should first lseek() to beginning of the file in your application.
• FUSE has two modes of operation:
  • Single thread (very low performance, easy to debug)
  • Multi-thread (default operation)
• Multi-thread spawns multiple threads during read operation. I observed almost single thread like behavior for writes.
• Code to multi-thread I/O implementation is in lib/fuse_loop_mt.c.
• FUSE uses worker threads to handle I/O requests, using struct fuse_worker. A worker thread is created in fuse_start_thread().
• Each worker run fuse_do_work() function. This is an infinite loop and terminates only on session exit OR if number of active threads exceed than required.
• User implementation of file system APIs are populated in const struct fuse_operations. It has address of all implemented APIs. FUSE ultimately calls these APIs for file system operations.
• FUSE 2.7 reads 8K data by default, in two 4K chunks
  • Read happens in last 4K and the first 4K data block

• An example:

  I had set the file size as 4MB in getattr () implementation. If you forget to compile with 64-bit flags, you will get zero length files.

  int bb_getattr(const char *path, struct stat *statbuf)
  {
      int retstat = 0;
      memset(statbuf, 0, sizeof(struct stat));
      if (strcmp(path, "/") == 0) {
          statbuf->st_mode = S_IFDIR | 0755;
          statbuf->st_nlink = 2;
      } else {
          statbuf->st_mode = S_IFREG | 0444;
          statbuf->st_nlink = 1;
          statbuf->st_size = 4 * 1024* 1024;
      }   
      return retstat;
  }

  The sequence of calls and their arguments is as follows:

  bb_getattr(path="/abcd.txt", statbuf=0xc5387960)
      rootdir = "/tmp", path = "/abcd.txt"

  bb_open(path"/abcd.txt", fi=0xc5daaa50)
      rootdir = "/tmp", path = "/abcd.txt"
      fi:
      flags = 0x00008002
      fh_old = 0x00000000
      writepage = 0
      direct_io = 0
      keep_cache = 0
      fh = 0x0000000000000001
      lock_owner = 0x0000000000000000

  bb_write(path="/abcd.txt", buf=0xc4966050, size=10, offset=0, fi=0xc5387a50)
      fi:
      flags = 0x00000000
      fh_old = 0x00000001
      writepage = 0
      direct_io = 0
      keep_cache = 0
      fh = 0x0000000000000001
      lock_owner = 0x0000000000000000

  bb_read(path="/abcd.txt", buf=0x06ccbd90, size=12288, offset=4096, fi=0xc5daaa50)  <- Here
      fi:
      flags = 0x00000000
      fh_old = 0x00000001
      writepage = 0
      direct_io = 0
      keep_cache = 0
      fh = 0x0000000000000001
      lock_owner = 0x0000000000000000
  
  bb_read(path="/abcd.txt", buf=0x06ccbd90, size=4096, offset=0, fi=0xc5daaa50)
      fi:
      flags = 0x00000000
      fh_old = 0x00000001
      writepage = 0
      direct_io = 0
      keep_cache = 0
      fh = 0x0000000000000001
      lock_owner = 0x0000000000000000

WRITE stack trace
================
(gdb) bt
#0  bb_write (path=0x7ffc68000990 "/test_file.0", buf=0x7ffff6f42060 "", size=4096, offset=4096, fi=0x7ffff6f40550) at bbfs.c:136
#1  0x00007ffff7dc885f in fuse_fs_write_buf (fs=0x280f090, path=0x7ffc68000990 "/test_file.0", buf=0x7ffff6f40580, off=4096, fi=0x7ffff6f40550)
    at fuse.c:1878
#2  0x00007ffff7dccb37 in fuse_lib_write_buf (req=0x7ffc680008c0, ino=2, buf=0x7ffff6f40580, off=4096, fi=0x7ffff6f40550) at fuse.c:3278
#3  0x00007ffff7dd461b in do_write_buf (req=0x7ffc680008c0, nodeid=2, inarg=0x7ffff6f42038, ibuf=0x7ffff6f40800) at fuse_lowlevel.c:1300
#4  0x00007ffff7dd7369 in fuse_ll_process_buf (data=0x280f220, buf=0x7ffff6f40800, ch=0x280ece0) at fuse_lowlevel.c:2437
#5  0x00007ffff7dd9aa5 in fuse_session_process_buf (se=0x280ed30, buf=0x7ffff6f40800, ch=0x280ece0) at fuse_session.c:87
#6  0x00007ffff7dd0f6a in fuse_do_work (data=0x7ffff00008c0) at fuse_loop_mt.c:117
#7  0x00000037bc2079d1 in start_thread () from /lib64/libpthread.so.0
#8  0x00000037bbee8b6d in clone () from /lib64/libc.so.6

READ stack trace
=================
(gdb) bt
#0  bb_read (path=0x7ffff38c55b0 "/test_file.0", buf=0x7ffff38c56c0 "", size=4096, offset=8192, fi=0x7ffff79635d0) at bbfs.c:111
#1  0x00007ffff7dc841e in fuse_fs_read_buf (fs=0x280f090, path=0x7ffff38c55b0 "/test_file.0", bufp=0x7ffff7963578, size=4096, off=8192, 
    fi=0x7ffff79635d0) at fuse.c:1794
#2  0x00007ffff7dcca1d in fuse_lib_read (req=0x7ffff002a1e0, ino=2, size=4096, off=8192, fi=0x7ffff79635d0) at fuse.c:3252
#3  0x00007ffff7dd42c7 in do_read (req=0x7ffff002a1e0, nodeid=2, inarg=0x7ffff7965038) at fuse_lowlevel.c:1232
#4  0x00007ffff7dd73ce in fuse_ll_process_buf (data=0x280f220, buf=0x7ffff7963800, ch=0x280ece0) at fuse_lowlevel.c:2441
#5  0x00007ffff7dd9aa5 in fuse_session_process_buf (se=0x280ed30, buf=0x7ffff7963800, ch=0x280ece0) at fuse_session.c:87
#6  0x00007ffff7dd0f6a in fuse_do_work (data=0x280ee30) at fuse_loop_mt.c:117
#7  0x00000037bc2079d1 in start_thread () from /lib64/libpthread.so.0
#8  0x00000037bbee8b6d in clone () from /lib64/libc.so.6

Compiling FUSE based file system with your FUSE build

Suppose hello.c has implementation of file system APIs and your FUSE installation resides in /home/k/Desktop/my_fuse_2.9.3.

$gcc -g hello.c -o hi -D_FILE_OFFSET_BITS=64 -I/home/k/Desktop/my_fuse_2.9.3/include -lpthread
 -L/home/k/Desktop/my_fuse_2.9.3/lib -lfuse -LLIBDIR=/home/k/my_fuse_2.9.3/lib
 -Wl,-rpath -Wl,/home/k/my_fuse_2.9.3/lib