running pgf95 on the gfortran testsuite

janus · January 25, 2018, 8:50pm

Dear PGI team,

I’d like to report here on my recent endeavor of running (parts of) the gfortran test suite with various compilers, among them PGI 17.10. For details see this c.l.f. thread: Redirecting to Google Groups

The result for PGI was the following:

pgf95 17.10:
CMP: 78% tests passed, 336 tests failed out of 1554
EXE: 63% tests passed, 576 tests failed out of 1554

… which means that PGI was able to compile about 78% of the considered tests successfully, while it could successfully execute roughly 63%. One thing that is striking about the PGI results is that the gap between compile-time success and run-time success is relatively large, compared to other compilers.

Among the compile-time failures, I count about 77 internal compiler errors (which is more than for the other tested compilers, but to be fair, often multiple ICEs stem from a single test case, so the the number of tests which fail by ICE is a bit smaller). Here is the full list:

$ grep 'Internal compiler error' Testing/Temporary/LastTest.log 
PGF90-S-0000-Internal compiler error. rewrite_block_forall: illegal statement in FORALL      46 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/aliasing_dummy_4.f90: 18)
PGF90-F-0000-Internal compiler error. rewrite_block_forall: someting is wrong      46 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/aliasing_dummy_4.f90: 18)
PGF90-F-0000-Internal compiler error. expected initialization assignment     432 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/alloc_comp_assign_2.f90: 45)
PGF90-W-0000-Internal compiler error. sub_i in constructf90 is not back       8 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/alloc_comp_assign_7.f90: 33)
PGF90-F-0000-Internal compiler error. Errors in Lowering       1 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/alloc_comp_constructor_3.f90: 16)
PGF90-F-0000-Internal compiler error. Errors in Lowering       2 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/allocate_deferred_char_scalar_2.f03: 21)
PGF90-F-0000-Internal compiler error. Errors in Lowering       3 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/allocate_with_source_7.f08: 77)
PGF90-S-0000-Internal compiler error. bad dtype      -1 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/array_constructor_type_14.f03: 19)
PGF90-S-0000-Internal compiler error. unexpected ast type in initialization expr     148 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/array_initializer_1.f90: 21)
PGF90-S-0000-Internal compiler error. mkexpr1: bad id      14 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/associate_25.f90: 21)
PGF90-S-0000-Internal compiler error. mkexpr1: 0 dtype       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/associate_25.f90: 22)
PGF90-F-0000-Internal compiler error. size_shape:  stride is not 1       3 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/bound_4.f90: 11)
PGF90-F-0000-Internal compiler error. size_shape:  stride is not 1      11 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/char_length_5.f90: 60)
PGF90-F-0000-Internal compiler error. mod_type/data type: bad data type:       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/char_result_14.f90: 52)
PGF90-F-0000-Internal compiler error. Errors in Lowering       1 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/char_result_13.f90: 14)
PGF90-F-0000-Internal compiler error. mod_type/data type: bad data type:       6 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/char_result_15.f90: 21)
PGF90-F-0000-Internal compiler error. rewrite_sub_args: can't find array     420 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/char_result_3.f90: 41)
PGF90-F-0000-Internal compiler error. Errors in Lowering       1 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/class_allocate_21.f90: 20)
PGF90-F-0000-Internal compiler error. unknown state file error       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/class_assign_1.f08: 30)
PGF90-S-0000-Internal compiler error. compute_size, AC_ACONST in AC_EXPR       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/class_to_type_1.f03: 21)
PGF90-W-0000-Internal compiler error. sub_i in constructf90 is not back       8 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/class_to_type_1.f03: 21)
PGF90-F-0000-Internal compiler error. insert_sym: bad hash     675 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/class_result_6.f90: 19)
PGF90-F-0000-Internal compiler error. insert_sym: bad hash     626 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/class_result_5.f90: 14)
PGF90-F-0000-Internal compiler error. Errors in ILM file       2 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/dynamic_dispatch_10.f03: 53)
PGF90-F-0000-Internal compiler error. rewrite_subroutine: unknown stmt found      95 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/dynamic_dispatch_5.f03: 156)
PGF90-S-0000-Internal compiler error. select_rtemp: bad ili       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/finalize_15.f90: 187)
PGF90-S-0000-Internal compiler error. exp_call: ili ret type not cased     136 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/finalize_15.f90: 187)
PGF90-S-0000-Internal compiler error. get_const_from_ast: can't get const value       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/ieee/ieee_7.f90: 10)
PGF90-S-0000-Internal compiler error. get_const_from_ast: can't get const value       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/ieee/ieee_7.f90: 10)
PGF90-S-0000-Internal compiler error. get_const_from_ast: can't get const value       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/ieee/ieee_7.f90: 10)
PGF90-F-0000-Internal compiler error. Subscript for array is out-of-bounds       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/ieee/ieee_7.f90: 10)
PGF90-S-0000-Internal compiler error. bad dtype       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/inline_sum_1.f90: 21)
PGF90-S-0000-Internal compiler error. size_of: bad dtype        0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/inline_sum_1.f90: 21)
PGF90-S-0000-Internal compiler error. bad dtype       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/inline_sum_1.f90: 21)
PGF90-S-0000-Internal compiler error. alignment: bad dtype        0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/inline_sum_1.f90: 21)
PGF90-S-0000-Internal compiler error. lower_sptr: bad sptr       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/mapping_1.f90: 69)
PGF90-F-0000-Internal compiler error. Errors in Lowering       4 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/pr39865.f90: 26)
PGF90-S-0000-Internal compiler error. get_const_from_ast: can't get const value       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/pr67140.f90: 7)
PGF90-S-0000-Internal compiler error. get_const_from_ast: can't get const value       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/pr67140.f90: 7)
PGF90-S-0000-Internal compiler error. get_const_from_ast: can't get const value       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/pr67140.f90: 7)
PGF90-F-0000-Internal compiler error. Subscript for array is out-of-bounds       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/pr67140.f90: 7)
PGF90-S-0000-Internal compiler error. size_of: attempt to get size of assumed size character       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/pr83874.f90: 14)
PGF90-F-0000-Internal compiler error. Errors in Lowering       5 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/proc_decl_17.f90: 50)
PGF90-F-0000-Internal compiler error. Errors in ILM file      12 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/proc_ptr_comp_17.f90: 21)
PGF90-F-0000-Internal compiler error. Errors in ILM file       4 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/proc_ptr_comp_16.f90: 27)
PGF90-F-0000-Internal compiler error. insert_sym: bad hash     652 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/proc_ptr_result_1.f90: 85)
PGF90-S-0000-Internal compiler error. get_const_from_ast: can't get const value       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/quad_3.f90: 10)
PGF90-S-0000-Internal compiler error. get_const_from_ast: can't get const value       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/quad_3.f90: 10)
PGF90-S-0000-Internal compiler error. get_const_from_ast: can't get const value       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/quad_3.f90: 10)
PGF90-F-0000-Internal compiler error. Subscript for array is out-of-bounds       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/quad_3.f90: 10)
PGF90-S-0000-Internal compiler error. get_const_from_ast: can't get const value       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/quad_2.f90: 14)
PGF90-S-0000-Internal compiler error. get_const_from_ast: can't get const value       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/quad_2.f90: 14)
PGF90-S-0000-Internal compiler error. get_const_from_ast: can't get const value       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/quad_2.f90: 14)
PGF90-F-0000-Internal compiler error. Subscript for array is out-of-bounds       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/quad_2.f90: 14)
PGF90-W-0000-Internal compiler error. sub_i in constructf90 is not back       8 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/realloc_on_assign_17.f90: 40)
PGF90-W-0000-Internal compiler error. sub_i in constructf90 is not back       8 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/realloc_on_assign_17.f90: 40)
PGF90-W-0000-Internal compiler error. sub_i in constructf90 is not back       9 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/realloc_on_assign_17.f90: 40)
PGF90-W-0000-Internal compiler error. sub_i in constructf90 is not back       8 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/realloc_on_assign_17.f90: 43)
PGF90-W-0000-Internal compiler error. sub_i in constructf90 is not back       8 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/realloc_on_assign_17.f90: 43)
PGF90-W-0000-Internal compiler error. sub_i in constructf90 is not back       8 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/realloc_on_assign_17.f90: 43)
PGF90-W-0000-Internal compiler error. sub_i in constructf90 is not back       9 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/realloc_on_assign_17.f90: 43)
PGF90-F-0000-Internal compiler error. Errors in Lowering       2 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/reshape_transpose_1.f90: 18)
PGF90-F-0000-Internal compiler error. Subscript for array is out-of-bounds       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/round_4.f90: 30)
PGF90-F-0000-Internal compiler error. insert_sym: bad hash     822 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/select_type_37.f03: 76)
PGF90-F-0000-Internal compiler error. unknown state file error       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/submodule_11.f08: 19)
PGF90-F-0000-Internal compiler error. unknown state file error       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/submodule_30.f08: 19)
PGF90-S-0000-Internal compiler error. compute_size, AC_ACONST in AC_EXPR       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/type_to_class_1.f03: 17)
PGF90-W-0000-Internal compiler error. sub_i in constructf90 is not back       8 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/type_to_class_1.f03: 17)
PGF90-S-0000-Internal compiler error. sym_of_ast: unexpected ast       5 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/typebound_call_10.f03: 36)
PGF90-S-0000-Internal compiler error. memsym_of_ast:unexp.ast       5 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/typebound_call_10.f03: 36)
PGF90-S-0000-Internal compiler error. sym_of_ast: unexpected ast       5 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/typebound_call_10.f03: 36)
PGF90-S-0000-Internal compiler error. memsym_of_ast:unexp.ast       5 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/typebound_call_10.f03: 36)
PGF90-F-0000-Internal compiler error. Errors in Lowering       1 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/typebound_call_20.f03: 40)
PGF90-F-0000-Internal compiler error. insert_sym: bad hash     860 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/typebound_operator_9.f03: 158)
PGF90-F-0000-Internal compiler error. Subscript for array is out-of-bounds       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/unlimited_polymorphic_13.f90: 14)
PGF90-S-0000-Internal compiler error. select_rtemp: bad ili       0 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/unlimited_polymorphic_14.f90: 13)
PGF90-S-0000-Internal compiler error. exp_call: ili ret type not cased      79 (/home/jweil/gcc/trunk/gcc/testsuite/gfortran.dg/unlimited_polymorphic_14.f90: 13)

I hope the PGI team will be able to look into some of these problems. If you see any issues with my methodology (or ways in which my testing procedure could be improved), please let me know.

Cheers,
Janus

tull · January 25, 2018, 9:27pm

Thank you for the report. We are aware of the gfortran tests.

dave

janus · January 25, 2018, 9:43pm

You’re welcome.

We are aware of the gfortran tests.

I’m sure you were aware of their existence. Whether you were aware of the failures is not so clear, given their relatively large amount.

I guess both the number of ICEs and runtime failures is worrysome, but if you plan to start looking into some of them, I guess the runtime stuff deserves somewhat higher priority. I would say it’s more critical from a user’s point of view, since for the ICEs it’s pretty easy to spot the problem, while for the runtime issues one may not even be aware that there is a problem (if a bad result is generated silently).

Cheers,
Janus

tull · January 26, 2018, 8:19pm

I did not say we were ignoring your report, I said we were aware of your report.

Your are test driving the PGI compilers for GPUs by pulling tests off the internet that test the boundaries and features of Fortran. This is similar to checking out a race car by looking for ash trays and automatic windshield wipers.

Are you testing other compilers as well, and are you testing their
performance on GPUs?

Performance tests usually involve many calculations that are checked for correctness and timed for speed. Those would be
of greater interest, to everyone.

dave

janus · January 29, 2018, 9:16pm

Yes, sir, absolutely. Although “pulling tests of the internet” sounds a bit snarky. I’d rather say I used the largest collection of Fortran test cases that is publicly available (and it’s pretty well maintained and basically assembled just for that purpose).

This is similar to checking out a race car by looking for ash trays and automatic windshield wipers.

Nah, here I disagree. I don’t really care about ash trays. It’s more like I’m checking your brakes, your tires and probably also your engine.

I will not deny that some amount of the failures reported may be false positives, and some may well be ashtrays and stereo and eye candy (see my lengthy disclaimer at c.l.f.).

But then there are more than 200 out of the 1500 tests that compile but fail at runtime. (“The engine stops, smoke comes out, the bumper falls off.”) That’s quite a bit and I hope you’re at least slightly worried about some of them.

Are you testing other compilers as well, and are you testing their
performance on GPUs?

I did test other compilers (see c.l.f.).

Note that I’m actually not doing this to show off how bad your compiler (or anyone’s compiler) is. I basically did it because it was so simple, and because it has lots of potential for uncovering compiler bugs and highlighting areas that need improvement. And I do hope it will lead to an improvement of compiler quality in the end.

Performance tests usually involve many calculations that are checked for correctness and timed for speed. Those would be
of greater interest, to everyone.

I agree that testing for runtime performance is a useful thing to do as well. But it’s not what this set of test cases is made for. They do actually involve lots of correctness checking, though.

Cheers,
Janus

tull · January 30, 2018, 12:20am

Thank you again for you report. We are aware of the GNU tests.

Do you have a program that could use the computational power
of GPUs?