PGFIO/stdio: File too large

Accelerated Computing HPC Compilers Legacy PGI Compilers
1

Hi,

I’m developing a python module that calls Fortran executable to convert RPN (Environment Canada) files to r2c. I do not have the wisdom of the ancients : I’m not familiar with Fortran.

I got this error on my logfiles :

PGFIO/stdio: File too large
PGFIO-F-/formatted write/unit=10/error code returned by host stdio - 27.
File name = /data/local/tmpdirs/afqscli/26968/tmpIw37tm/tmprOmNG-/TT_0-9950sg_2004-06.r2c formatted, sequential access record = 1
In source file fst2r2c.f90, at line number 302 (line 307 sometimes too)

301 !           Write time stamp to R2C file
302             write(iunr2c,'(A,I8,I8,A,A,A,A,A)'), &
303                ':Frame ',ip2,ip2,' "',strymd,' ',strhms,'"'
304
305 !           Write data to R2C file
306             do j = 1, yCount
307                write(iunr2c,trim(r2cFormat))(fldr2c(i,j),i=1,xCount)
308             enddo

This error occurs when the r2c file is larger than 2GB : the fortran executable seems to be unable to deal with big files. After searching for a while I found here a topic indicating I had to recompile the fst2r2c executable using the -Mlfs option. I actually did it but the error still occurs with files larger than 2GB.

-Mlfs (32-bit Linux only) Link in the Large File Support routines
available on Linux versions later than > Red Hat 7.0 or SuSE 7.1> .
This will support files from Fortran I/O that are larger than
2GB. Equivalent to > -L$PGI/linux86/7.1-4/liblf> .

The program is run on this server :
Linux hawa 2.6.26-bpo.2-686-bigmem #1 SMP Fri Jul 3 21:38:05 UTC 2009 i686 GNU/Linux (Debian 32bits).

I wanted to know if the -Mlfs option (equivalent to -L$PGI/linux86/9.0/liblf in my case) was supported on my machine or not as I’m not on Fedora or SuSE. If not, should I consider moving my application to a 64bits server to fix that error ?

Regards,

kevindelorme

2

Hi Kevin,

I was able to construct the following program to successfully write out a > 2 GB file on a 32-bit system internally:

program iotest

  character :: a(1024,1024)
  integer :: i, j, f

  do i = 1, 1024
    do j = 1, 1024
      a(i,j) = 'x'
    end do
  end do

  open (f, file='myfile.txt', status='new')
  do i = 1, 2049
    write (f, *) a
  end do
  close (f)

end program iotest

I compiled it with PGI 9.0-5 as follows:

  pgf90 -o iotest -Mlfs iotest.f90

Running it on our 32-bit system here, I do get a large file:

cparrott /tmp $ ~/iotest
cparrott /tmp $ ls -l *.txt
-rw-rw-r--  1 cparrott sw 2203625736 Apr  4 07:56 myfile.txt
cparrott /tmp $

I just posted this to reassure both of us that this works, at least as of the PGI release you specified. :)

That said, there could be a number of reasons as to why your program is not working correctly. I notice you are running on a fairly old version of Debian - it is possible that there is an issue with that OS that is causing -Mlfs to not work correctly. We are not running that version of Debian here internally, and do not officially support it, so I cannot really reproduce the conditions you described here in-house. (However, we do support recent versions of Ubuntu, which is a fork of Debian.)

I do think moving to a 64-bit system would make this task a bit easier for you, though you might have to add the -mcmodel=medium flag if your application has a static data area that is larger than 2 GB. (This will just fail outright in 32-bit mode.) So, my initial suggestion would be to try moving over to a 64-bit system if you can. If your application still fails there, then let’s see if we can work on reproducing your problem here internally.

Sorry for the long explanation - hope this helps.

Best regards,

+chris