ASAN test failure on FreeBSD #10714

AMDmi3 · 2024-10-29T14:55:07Z

Is there an existing issue for this?

I have searched the existing issues

Describe the bug

I've just added support for running tests in FreeBSD port of ES, and there was test failure, seemingly caused by ASAN. Not sure if test failed on earlier versions. I haven't tested the game itself.

UpdateCTestConfiguration  from :/work/usr/ports/games/endless-sky/work/.build/DartConfiguration.tcl
Parse Config file:/work/usr/ports/games/endless-sky/work/.build/DartConfiguration.tcl
UpdateCTestConfiguration  from :/work/usr/ports/games/endless-sky/work/.build/DartConfiguration.tcl
Parse Config file:/work/usr/ports/games/endless-sky/work/.build/DartConfiguration.tcl
Test project /work/usr/ports/games/endless-sky/work/.build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
    Start 1: unit

1: Test command: /work/usr/ports/games/endless-sky/work/.build/tests/endless-sky-tests
1: Working Directory: /work/usr/ports/games/endless-sky/work/endless-sky-0.10.10/tests
1: Test timeout computed to be: 1500
1: =================================================================
1: ==64627==ERROR: AddressSanitizer: container-overflow on address 0x5080000000a1 at pc 0x0000004388f2 bp 0x7fffffffc870 sp 0x7fffffffc020
1: WRITE of size 4 at 0x5080000000a1 thread T0
1:     #0 0x4388f1 in memmove /usr/src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors_memintrinsics.inc:98:3
1:     #1 0x80110d108 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::basic_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, unsigned long, unsigned long, std::__1::allocator<char> const&) (/lib/libc++.so.1+0xe2108)
1:     #2 0x6e8a2b in DataFile::LoadData(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) (/work/usr/ports/games/endless-sky/work/.build/tests/endless-sky-tests+0x6e8a2b)
1:     #3 0x6e7f6a in DataFile::Load(std::__1::basic_istream<char, std::__1::char_traits<char>>&) (/work/usr/ports/games/endless-sky/work/.build/tests/endless-sky-tests+0x6e7f6a)
1:     #4 0x6e7e01 in DataFile::DataFile(std::__1::basic_istream<char, std::__1::char_traits<char>>&) (/work/usr/ports/games/endless-sky/work/.build/tests/endless-sky-tests+0x6e7e01)
1:     #5 0x47edff in AsDataNodes(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) /work/usr/ports/games/endless-sky/work/endless-sky-0.10.10/tests/unit/src/helpers/datanode-factory.cpp:32:20
1:     #6 0x47fe08 in AsDataNode(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>) /work/usr/ports/games/endless-sky/work/endless-sky-0.10.10/tests/unit/src/helpers/datanode-factory.cpp:39:15
1:     #7 0x57e9e1 in __cxx_global_var_init.5 /work/usr/ports/games/endless-sky/work/endless-sky-0.10.10/tests/unit/src/test_distance_calculation_settings.cpp:75:30
1:     #8 0x57e9e1 in _GLOBAL__sub_I_test_distance_calculation_settings.cpp /work/usr/ports/games/endless-sky/work/endless-sky-0.10.10/tests/unit/src/test_distance_calculation_settings.cpp
1: 
1: 0x5080000000a1 is located 1 bytes inside of 96-byte region [0x5080000000a0,0x508000000100)
1: allocated by thread T0 here:
1:     #0 0x46ce3d in operator new(unsigned long) /usr/src/contrib/llvm-project/compiler-rt/lib/asan/asan_new_delete.cpp:86:3
1:     #1 0x4ba780 in void* std::__1::__libcpp_operator_new[abi:se180100]<unsigned long>(unsigned long) /usr/include/c++/v1/new:271:10
1:     #2 0x4ba780 in std::__1::__libcpp_allocate[abi:se180100](unsigned long, unsigned long) /usr/include/c++/v1/new:295:10
1:     #3 0x4ba780 in std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>::allocate[abi:se180100](unsigned long) /usr/include/c++/v1/__memory/allocator.h:125:32
1:     #4 0x4ba780 in std::__1::__allocation_result<std::__1::allocator_traits<std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>::pointer> std::__1::__allocate_at_least[abi:se180100]<std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>(std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>&, unsigned long) /usr/include/c++/v1/__memory/allocate_at_least.h:55:19
1:     #5 0x4ba780 in std::__1::__split_buffer<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>&) /usr/include/c++/v1/__split_buffer:343:25
1:     #6 0x5a1c4d in std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>::reserve(unsigned long) /usr/include/c++/v1/vector:1425:49
1:     #7 0x6e83a2 in DataFile::LoadData(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) (/work/usr/ports/games/endless-sky/work/.build/tests/endless-sky-tests+0x6e83a2)
1:     #8 0x6e7f6a in DataFile::Load(std::__1::basic_istream<char, std::__1::char_traits<char>>&) (/work/usr/ports/games/endless-sky/work/.build/tests/endless-sky-tests+0x6e7f6a)
1:     #9 0x6e7e01 in DataFile::DataFile(std::__1::basic_istream<char, std::__1::char_traits<char>>&) (/work/usr/ports/games/endless-sky/work/.build/tests/endless-sky-tests+0x6e7e01)
1:     #10 0x47edff in AsDataNodes(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) /work/usr/ports/games/endless-sky/work/endless-sky-0.10.10/tests/unit/src/helpers/datanode-factory.cpp:32:20
1:     #11 0x47fe08 in AsDataNode(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>) /work/usr/ports/games/endless-sky/work/endless-sky-0.10.10/tests/unit/src/helpers/datanode-factory.cpp:39:15
1:     #12 0x57e9e1 in __cxx_global_var_init.5 /work/usr/ports/games/endless-sky/work/endless-sky-0.10.10/tests/unit/src/test_distance_calculation_settings.cpp:75:30
1:     #13 0x57e9e1 in _GLOBAL__sub_I_test_distance_calculation_settings.cpp /work/usr/ports/games/endless-sky/work/endless-sky-0.10.10/tests/unit/src/test_distance_calculation_settings.cpp
1:     #14 0x800959f5c  (/libexec/ld-elf.so.1+0xaf5c)
1:     #15 0x800958b88  (/libexec/ld-elf.so.1+0x9b88)
1:     #16 0x800955df8  (/libexec/ld-elf.so.1+0x6df8)
1: 
1: HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_container_overflow=0.
1: If you suspect a false positive see also: https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow.
1: SUMMARY: AddressSanitizer: container-overflow (/lib/libc++.so.1+0xe2108) in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::basic_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, unsigned long, unsigned long, std::__1::allocator<char> const&)
1: Shadow bytes around the buggy address:
1:   0x507ffffffe00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1:   0x507ffffffe80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1:   0x507fffffff00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1:   0x507fffffff80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1:   0x508000000000: fa fa fa fa fc fc fc fc fc fc fc fc fc fc fc fc
1: =>0x508000000080: fa fa fa fa[fc]fc fc fc fc fc fc fc fc fc fc fc
1:   0x508000000100: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
1:   0x508000000180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
1:   0x508000000200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
1:   0x508000000280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
1:   0x508000000300: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
1: Shadow byte legend (one shadow byte represents 8 application bytes):
1:   Addressable:           00
1:   Partially addressable: 01 02 03 04 05 06 07 
1:   Heap left redzone:       fa
1:   Freed heap region:       fd
1:   Stack left redzone:      f1
1:   Stack mid redzone:       f2
1:   Stack right redzone:     f3
1:   Stack after return:      f5
1:   Stack use after scope:   f8
1:   Global redzone:          f9
1:   Global init order:       f6
1:   Poisoned by user:        f7
1:   Container overflow:      fc
1:   Array cookie:            ac
1:   Intra object redzone:    bb
1:   ASan internal:           fe
1:   Left alloca redzone:     ca
1:   Right alloca redzone:    cb
1: ==64627==ABORTING
1/2 Test #1: unit .............................***Failed    0.13 sec

Full log: https://gist.github.com/AMDmi3/e31a6af6d78956e90393cff580e459eb

Steps to Reproduce

Build on FreeBSD 14.1 and run tests.

Expected Behavior

Tests passing

Screenshots

No response

Link to save file

No response

Operating System

FreeBSD 14.1

Game Source

Built from source

Game Version

0.10.10

Additional Information

No response

The text was updated successfully, but these errors were encountered:

TomGoodIdea · 2024-10-29T16:02:09Z

Where does it say it's because of asan? Have you tried without it?

tibetiroka · 2024-10-29T17:24:00Z

==64627==ERROR: AddressSanitizer: container-overflow

This seems pretty clear it's asan.
I can't reproduce it on linux, though, and I don't think we have any BSD-specific code there, so I'm not sure why it's happening.

AMDmi3 · 2024-10-29T21:38:32Z

I don't see anything suspicious in the code either. But tests still fails without asan: https://gist.github.com/AMDmi3/f4e15d40834d30dee2c02a4c6dbd3671. Maybe it's something clang specific.

tibetiroka · 2024-10-30T07:28:30Z

Most of the failures look like this. For some reason, a couple of lines are prepended to the output of warnings.Flush().

At a glance, the extra text seems to be describing an object of sorts, perhaps something like a DataNode, which is weird, since we have no such code.

What commit are you on?

AMDmi3 · 2024-10-30T09:34:11Z

What commit are you on?

That's v0.10.10 tag

TomGoodIdea · 2024-10-30T18:01:32Z

The only BSD ifdef we have works also for Linux.

endless-sky/source/Files.cpp

Lines 116 to 126 in a7f23d4

    
           #if defined __linux__ || defined __FreeBSD__ || defined __DragonFly__ 
        
           	// Special case, for Linux: the resource files are not in the same place as 
        
           	// the executable, but are under the same prefix (/usr or /usr/local). 
        
           	static const string LOCAL_PATH = "/usr/local/"; 
        
           	static const string STANDARD_PATH = "/usr/"; 
        
           	static const string RESOURCE_PATH = "share/games/endless-sky/"; 
        
           	if(!resources.compare(0, LOCAL_PATH.length(), LOCAL_PATH)) 
        
           		resources = LOCAL_PATH + RESOURCE_PATH; 
        
           	else if(!resources.compare(0, STANDARD_PATH.length(), STANDARD_PATH)) 
        
           		resources = STANDARD_PATH + RESOURCE_PATH; 
        
           #endif

So it may be some library difference (I don't have FreeBSD installed, so I can't check).
Does just running the game return the same error?

AMDmi3 · 2024-10-30T18:12:50Z

Unfortunately I can't check right now as most OpenGL apps no longer work with my ancient nvidia.

thewierdnut · 2024-10-30T21:14:11Z

looks like its using libc++ instead of libstdc++.... let me see if that triggers the difference.

thewierdnut · 2024-10-30T21:18:38Z

@AMDmi3 do you know what version of libc++ you are running?

AMDmi3 · 2024-10-30T22:23:26Z

Should be the same as clang/llvm which is 18.1.5
_LIBCPP_VERSION is 180100

thewierdnut · 2024-10-30T22:30:36Z

I think I am reproducing this (or a similar bug) on linux using libc++. I'll try to track down.

thewierdnut · 2024-10-30T23:41:12Z

@AMDmi3 If you set the environment variable ASAN_OPTIONS=detect_container_overflow=0, does the test pass?

thewierdnut · 2024-10-31T00:18:25Z

The asan failure I am hitting is the container overflow check, which in theory is supposed to monitor for accesses in between the end of a container and its capacity. In practice, it is known to be broken in mixed asan/non asan builds. This only works for containers that have been correctly annotated.

I reproduced a container overflow failure in the very first string access from the unit test, specifically in a string::size() call that in theory should only access the very first byte of SSO-optimized string structure. It never would have accessed any heap allocated memory at all, given the string was only 6 bytes long.

I recommend that we disable container overflow checking, since it only works for annotated types anyways (I strongly suspect that only libc++ has these annotations), and is broken if the program links with any other libraries that happen to also use libc++ strings or vectors. This can be done by setting the ASAN_OPTIONS=detect_container_overflow=0 environment variable before running the test.

tibetiroka · 2024-10-31T06:28:08Z

That doesn't explain why the test fails without sanitizers, though.

AMDmi3 · 2024-10-31T12:00:53Z

I'm really sorry for misleading, the cause of test failure was that a patch I've used to debug asan failure accidentally got committed.

@AMDmi3 If you set the environment variable ASAN_OPTIONS=detect_container_overflow=0, does the test pass?

Yes, this fixes asan failure.

After these fixed there's a single failure left:

1: Test command: /work/usr/ports/games/endless-sky/work/.build/tests/endless-sky-tests
1: Working Directory: /work/usr/ports/games/endless-sky/work/endless-sky-0.10.10/tests
1: Test timeout computed to be: 1500
1: Randomness seeded to: 2747297804
1: 
1: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1: endless-sky-tests is a Catch2 v3.7.1 host application.
1: Run with -? for options
1: 
1: -------------------------------------------------------------------------------
1: Scenario: Obtaining a random value
1:       Given: a list with no content
1:        When: a random selection is performed
1:        Then: an informative runtime exception is thrown
1: -------------------------------------------------------------------------------
1: /work/usr/ports/games/endless-sky/work/endless-sky-0.10.10/tests/unit/src/test_weightedList.cpp:349
1: ...............................................................................
1: 
1: /work/usr/ports/games/endless-sky/work/endless-sky-0.10.10/tests/unit/src/test_weightedList.cpp:352: FAILED:
1:   CHECK_THROWS_WITH( list.Get(), Catch::Matchers::ContainsSubstring("empty weighted list") )
1: with expansion:
1:   "Unknown exception" contains: "empty weighted list"
1: 
1: ===============================================================================
1: test cases:    78 |    77 passed | 1 failed
1: assertions: 75959 | 75958 passed | 1 failed
1:

This check is already under #ifndef APPLE

endless-sky/tests/unit/src/test_weightedList.cpp

Lines 351 to 353 in e24ae69

    
           #ifndef __APPLE__ 
        
           				CHECK_THROWS_WITH( list.Get(), Catch::Matchers::ContainsSubstring("empty weighted list") ); 
        
           #endif

Why's that? A quick check shows that CHECK_THROWS_WITH + Catch::Matchers::ContainsSubstring + std::runtime_error combination work fine.

tibetiroka · 2024-10-31T12:21:46Z

Why's that? A quick check shows that CHECK_THROWS_WITH + Catch::Matchers::ContainsSubstring + std::runtime_error combination work fine.

It doesn't work on older Apple systems due to missing library support. The same may be true under *BSD.

thewierdnut · 2024-10-31T12:26:39Z

Heh... I had to remove test_weightedList.cpp from cmake to get it to compile on debian.

tibetiroka · 2024-10-31T12:28:49Z

I'm not using vcpkg and it works just fine. I'm not on debian, though.

AMDmi3 · 2024-10-31T12:56:44Z

No, it compiles fine, but as it can be seen in the log, for some reason it sees exception message as Unknown exception instead of Attempted to call Get on an empty weighted list.

tibetiroka · 2024-10-31T13:13:52Z

Yeah, it seems exception messages aren't supported in whatever standard library version you are using.

AMDmi3 · 2024-10-31T13:35:16Z

They are.

A quick check shows that CHECK_THROWS_WITH + Catch::Matchers::ContainsSubstring + std::runtime_error combination work fine.

This passes:

#include <catch2/catch_all.hpp>
void t() {
	throw std::runtime_error("Attempted to call Get on an empty weighted list.");
}
TEST_CASE("test") {
	CHECK_THROWS_WITH(t(), Catch::Matchers::ContainsSubstring("empty weighted list"));
}

AMDmi3 added bug Something in the game is not behaving as intended unconfirmed More information is needed to be sure this report is true labels Oct 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ASAN test failure on FreeBSD #10714

ASAN test failure on FreeBSD #10714

AMDmi3 commented Oct 29, 2024

TomGoodIdea commented Oct 29, 2024

tibetiroka commented Oct 29, 2024

AMDmi3 commented Oct 29, 2024

tibetiroka commented Oct 30, 2024

AMDmi3 commented Oct 30, 2024

TomGoodIdea commented Oct 30, 2024

AMDmi3 commented Oct 30, 2024

thewierdnut commented Oct 30, 2024

thewierdnut commented Oct 30, 2024

AMDmi3 commented Oct 30, 2024

thewierdnut commented Oct 30, 2024

thewierdnut commented Oct 30, 2024

thewierdnut commented Oct 31, 2024

tibetiroka commented Oct 31, 2024

AMDmi3 commented Oct 31, 2024

tibetiroka commented Oct 31, 2024

thewierdnut commented Oct 31, 2024 •

edited

Loading

tibetiroka commented Oct 31, 2024

AMDmi3 commented Oct 31, 2024

tibetiroka commented Oct 31, 2024

AMDmi3 commented Oct 31, 2024 •

edited

Loading

ASAN test failure on FreeBSD #10714

ASAN test failure on FreeBSD #10714

Comments

AMDmi3 commented Oct 29, 2024

Is there an existing issue for this?

Describe the bug

Steps to Reproduce

Expected Behavior

Screenshots

Link to save file

Operating System

Game Source

Game Version

Additional Information

TomGoodIdea commented Oct 29, 2024

tibetiroka commented Oct 29, 2024

AMDmi3 commented Oct 29, 2024

tibetiroka commented Oct 30, 2024

AMDmi3 commented Oct 30, 2024

TomGoodIdea commented Oct 30, 2024

AMDmi3 commented Oct 30, 2024

thewierdnut commented Oct 30, 2024

thewierdnut commented Oct 30, 2024

AMDmi3 commented Oct 30, 2024

thewierdnut commented Oct 30, 2024

thewierdnut commented Oct 30, 2024

thewierdnut commented Oct 31, 2024

tibetiroka commented Oct 31, 2024

AMDmi3 commented Oct 31, 2024

tibetiroka commented Oct 31, 2024

thewierdnut commented Oct 31, 2024 • edited Loading

tibetiroka commented Oct 31, 2024

AMDmi3 commented Oct 31, 2024

tibetiroka commented Oct 31, 2024

AMDmi3 commented Oct 31, 2024 • edited Loading

thewierdnut commented Oct 31, 2024 •

edited

Loading

AMDmi3 commented Oct 31, 2024 •

edited

Loading