Hello guys,
so, is there a solution to the output redirection problem with PGI compilers?
I don’t experience this problem with any other compilers (pathscale, intel), so I am quite surprised that there seem to be no simple fix to this problem. At least I couldn’t find one in this forum or PGI docs.
I repeat: whenever one redirects the output to a file, there is a significant delay in updating its content. It seems that the output log is being updated only when it gains a certain “critical mass” (buffer size?) or when the program is finished.
I am using Fortran 90, but as far as I know the same problem exists with C++.
Thanks -
Denis.
Hi Denis -
Do you have any more information on this? I don’t see any issue when I redirect stdout to a file. A simple:
csh> make >& build.log
doesn’t seem to have any appreciable delay on my system.
What shell are you running, what OS and version, and what command are you using? Do you notice the delay when you don’t redirect the output?
-dave
Thanks for the reply, Dave.
It’s not about MAKE, it is about any executable, produced by PGI compilers.
I have a fortran code that outputs some messages on the screen like
print *,‘calculating big A matrix’
!.. actual calculation of A, takes some time
print *,‘calculating big B matrix’
!.. actulal calculation of B, takes some time
etc.
So, if I compile the code with pgf90 and run it
./executable
then it immediately produces the message “calculating big A matrix”, as expected. Once matrix A is calculated, it immediately produces message “calculating big B matrix”, etc.
Now, if I start
(./executable >& output.txt) &
(redirecting the output of the same executable to the file output.txt), then the file output.txt contains (initially) nothing, even though it should already have contained “calculating big A matrix”. The message will only appear later on, when (as I understand it) enough messages will be accumulated to be written to output.txt.
Of course, once the code is finished output.txt will contain all the messages (because, probably, the buffer is flushed at the end).
However, there is always a delay in writing the messages to the file (compared to the screen output). There is NO delay in the performance. Only in outputing the messages.
The system I have:
Suse Linux 9.2 64bit running on an dual-CPU Opteron 250
As I mentioned I am using PGF90.
If I am compiling the same code with other compilers, there is no delay in the output and the output.txt contains “up-to-date” info all the time.
If I look into output.txt file while the program is still running I may find something like
…
calculating big A matrix
calcula
I.e. only a PART of the message (‘calculating big B matrix’) is shown. This is (as I understand) due to the fact that the output buffer is flushed when it reaches a certain size. Once it “accumulates” enough data (or when the program is finished) the rest of the message will be written to output.txt:
…
calculating big A matrix
calculating big B matrix
…
My question is how to control that internal stdout output buffer of the PGI compilers? Can I change its size? Can I tell the compiler to produce a code without an stdout output buffer at all, so that the messages would be written to a file immediately - as they appear on the screen?
Hi Denis,
How are you printing your output? Are you using WRITE or PRINT? Do you open and close stdout?
Since it only happens with redirection, this suggests an OS or program problem. However, since it only occurs with PGI compiled code, this suggests a compiler problem. So far we been unable to recreate the issue so don’t have a good handle on why it would occur. Can you send the code in question to trs@pgroup.com?
Ohio State has a good web page about how to flush fortran output (
Here). Note the section about repeatedly open and closing files.
Hello,
I have exactly the same problem as Denis. Has a solution been found in the meantime?
In my case the redirected output is only written in the log file after the code is done.
My output statement is simply write(*,…). I use pgf90. I have Gentoo linux and it’s an intel. The shell is tcsh. I find the solution suggested by Ohio State, as recommended by Mat, too awkward for this problem. Isn’t there just a flag I can set when compiling, which would resolve this problem?
Just as in Denis’ case, the redirection works fine with a different compiler. Also, if I use pgf90 and I don’t redirect the output it is nicely written on the screen while the code is running.
I hope someone could help. I would be grateful. Many thanks in advance.
Jurgen
Hi Jurgen,
It’s most likely an issue with the way stdio works on Linux. When stdio writes to a “tty” it uses line buffering. For pipes and redirection, it uses block buffering. One solution is to explicitly set line buffering via the 3F runtime routine “setbuf3f”.
setvbuf3f(lu,type,size)
lu is the logical unit
type is 0 - full buffering, 1 - line buffering, 2 - no buffering
size is the size of the new buffer
For the logical unit, use 6 for standard out (stdout). So, for line
buffering you may use the following call at the beginning of your program:
call setvbuf3f(6,1,180)
Note that the size doesn’t actually set the buffer size as the Linux man pages indicate, but it does correctly set the buffering type.
Hope this helps,
Mat