File systems and file size

Consider the following piece of Fortran code which produces a set of files output000000, output000001, ... each containing one element of the array c:

      real c(1000000)
      character*12 filename

      do i = 1,1000000
         write(filename,'("output",i6.6)') i-1
         open(unit=10,file=filename)
         write(11,*) c(i)
         close(11)
      enddo

Even though there is only 4Mbytes of data, it is likely to fill a 4Gbyte disk !!

The reason is that file systems allocate disk space in chunks. The size of the chunks is chosen when tuning the file system and maybe anywhere from 512bytes to 32Mbytes (as on the high performance file systems on the VPP300). On the Power Challenge the chunk is 8 blocks each of 512bytes hence the above statement about 4GBs. When one chunk is filled, a second (of possibly different and often larger size) is allocated.

So obviously,


large files are more space-efficient than small ones
(regardless of whether they are binary or not). The idea that 1,000,000 files containing 1 number each is roughly the same as 1 file containing 1,000,000 numbers is way off!

To monitor the disk capacity of the system use df -k. If a filesystem that effects you (your home filesystem or scratch areas) are getting full (> 95%), let us know. To monitor your own usage try running du -k in your home directory.