[CI] Update macOS agent to 10.15 Catalina #6680

sharvilshah · 2020-09-29T11:56:13Z

Updating the Azure Pipelines agent to use 10.15 Catalina image. This will also help with the EndpointSecurity work.

Here is the agent info: https://github.com/microsoft/azure-pipelines-image-generation/blob/master/images/macos/macos-10.15-Readme.md

Closes #6707

sharvilshah · 2020-09-29T12:48:48Z

It correctly picked up the new version (https://dev.azure.com/trailofbits/osquery/_build/results?buildId=3828&view=logs&j=7087e851-80cb-52bb-c098-0ae67b77f687&t=ffe76d4b-c1d9-4998-b70a-742ec95e6afe&l=6) but couple of macOS tests are failing.

These are the same macOS tests that have failed on other older PRs.

sharvilshah · 2020-10-02T17:16:07Z

@alessandrogario @Smjert thoughts on this?

Smjert · 2020-10-02T19:15:45Z

@alessandrogario @Smjert thoughts on this?

For the wifi table failure, could this be related? #6235

For the startup_items one, seems a simple suffix / difference?

[----------] 1 test from StartupItemsTests
21: [ RUN      ] StartupItemsTests.test_parse_startup_items
21: /Users/runner/work/1/s/osquery/tables/system/tests/darwin/startup_items_tests.cpp:35: Failure
21: Expected equality of these values:
21:   "/Applications/iTunes.app/Contents/MacOS/iTunesHelper.app/"
21:   results[0]["path"]
21:     Which is: "/Applications/iTunes.app/Contents/MacOS/iTunesHelper.app"
21: [  FAILED  ] StartupItemsTests.test_parse_startup_items (1 ms)
21: [----------] 1 test from StartupItemsTests (1 ms total)

For the integration tests we have:

83: [----------] 1 test from apps
83: [ RUN      ] apps.test_sanity
83: /Users/runner/work/1/s/tests/integration/tables/apps.cpp:55: Failure
83: Expected equality of these values:
83:   data1.size()
83:     Which is: 0
83:   1ul
83:     Which is: 1
83: [  FAILED  ] apps.test_sanity (1888 ms)
83: [----------] 1 test from apps (1888 ms total)

and the failing check is

osquery/tests/integration/tables/apps.cpp

Lines 52 to 55 in ecc5403

    
           // Not totally sure what apps we expect on the VMs used by CI. 
        
           auto const data1 = execute_query( 
        
               "select * from apps where path = '/Applications/Preview.app'"); 
        
           ASSERT_EQ(data1.size(), 1ul);

So I guess that CI installations do not (always?) have Preview.app.

Then we have:

83: [----------] 1 test from browserPlugins
83: [ RUN      ] browserPlugins.test_sanity
83: /Users/runner/work/1/s/tests/integration/tables/browser_plugins.cpp:40: Failure
83: Value of: data.empty()
83:   Actual: true
83: Expected: false
83: [  FAILED  ] browserPlugins.test_sanity (4 ms)
83: [----------] 1 test from browserPlugins (4 ms total)

and the failing check is:

osquery/tests/integration/tables/browser_plugins.cpp

Lines 39 to 41 in ecc5403

    
           auto const data = execute_query("select * from browser_plugins"); 
        
           ASSERT_FALSE(data.empty()); 
        
           validate_rows(data, row_map);

So here too probably not all installation contains a browser with plugins?

I'm not sure what is considered normal (the presence or possible absence).

sharvilshah · 2020-10-14T13:12:08Z

Somehow, after upgrading Xcode, psutil cannot be found (81: ModuleNotFoundError: No module named 'psutil' ) even though it's listed as a brew argument.

Trying closing/reopening PR just in case.

sharvilshah · 2020-10-14T14:08:01Z

Weird: for the macOS Debug build it found psutil and those tests ran just fine, but for the Release one, it still errors out with No module named psutil.

sharvilshah · 2020-10-14T16:32:26Z

From the logs on the macOS Release build, it seems like psutil was downloaded/built successfully:

2020-10-14T13:13:59.8277960Z Requirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (50.3.0)
2020-10-14T13:13:59.9246930Z Collecting pexpect==3.3
2020-10-14T13:13:59.9522100Z   Downloading pexpect-3.3.tar.gz (132 kB)
2020-10-14T13:14:00.9819820Z Collecting psutil
2020-10-14T13:14:00.9898800Z   Downloading psutil-5.7.2.tar.gz (460 kB)
2020-10-14T13:14:01.5372550Z Collecting timeout_decorator
2020-10-14T13:14:01.5443640Z   Downloading timeout-decorator-0.4.1.tar.gz (4.8 kB)
2020-10-14T13:14:01.9548670Z Collecting six
2020-10-14T13:14:01.9638910Z   Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
2020-10-14T13:14:02.0008950Z Collecting thrift==0.11.0
2020-10-14T13:14:02.0079000Z   Downloading thrift-0.11.0.tar.gz (52 kB)
2020-10-14T13:14:02.3584700Z Collecting osquery
2020-10-14T13:14:02.3658200Z   Downloading osquery-3.0.6-py2.py3-none-any.whl (26 kB)
2020-10-14T13:14:02.4599490Z Collecting argparse>=1.1
2020-10-14T13:14:02.4669080Z   Downloading argparse-1.4.0-py2.py3-none-any.whl (23 kB)
2020-10-14T13:14:02.5094480Z Collecting future
2020-10-14T13:14:02.5173360Z   Downloading future-0.18.2.tar.gz (829 kB)
2020-10-14T13:14:03.1498290Z Building wheels for collected packages: pexpect, psutil, timeout-decorator, thrift, future
2020-10-14T13:14:03.1508440Z   Building wheel for pexpect (setup.py): started
2020-10-14T13:14:03.5415330Z   Building wheel for pexpect (setup.py): finished with status 'done'
2020-10-14T13:14:03.5424800Z   Created wheel for pexpect: filename=pexpect-3.3-py3-none-any.whl size=47715 sha256=e16a423718800c54ba6226569c01a41b8bd7d911c98df04a6f66644f0059837b
2020-10-14T13:14:03.5425980Z   Stored in directory: /Users/runner/Library/Caches/pip/wheels/9c/64/9b/c3b92f1cc7cad56fd0501605b9decb19b31bfad00343bec2ec
2020-10-14T13:14:03.5443130Z   Building wheel for psutil (setup.py): started
2020-10-14T13:14:06.6634560Z   Building wheel for psutil (setup.py): finished with status 'done'
2020-10-14T13:14:06.6649910Z   Created wheel for psutil: filename=psutil-5.7.2-cp39-cp39-macosx_10_15_x86_64.whl size=234171 sha256=e4414ba89a0e2a3ce368f29f860627ea34ab651977c181ca0471220687c8b418
2020-10-14T13:14:06.6651030Z   Stored in directory: /Users/runner/Library/Caches/pip/wheels/5d/fd/59/996857f9128ee706989567c761b9aa449f2d5d99435944d5be
2020-10-14T13:14:06.6667400Z   Building wheel for timeout-decorator (setup.py): started
2020-10-14T13:14:07.0141210Z   Building wheel for timeout-decorator (setup.py): finished with status 'done'
2020-10-14T13:14:07.0148890Z   Created wheel for timeout-decorator: filename=timeout_decorator-0.4.1-py3-none-any.whl size=5016 sha256=6e5c2f7d84384617105461111eb33b08a61851f6f60d6b7f4955ae2e308fa11f
2020-10-14T13:14:07.0149930Z   Stored in directory: /Users/runner/Library/Caches/pip/wheels/98/72/dd/72dccfb123aed230c3728fe5aa2c36773de829470f202d4613
2020-10-14T13:14:07.0165650Z   Building wheel for thrift (setup.py): started
2020-10-14T13:14:11.6806190Z   Building wheel for thrift (setup.py): finished with status 'done'
2020-10-14T13:14:11.6823380Z   Created wheel for thrift: filename=thrift-0.11.0-cp39-cp39-macosx_10_15_x86_64.whl size=157635 sha256=f587dd0696bb9b6c37a21dab8f115f78ed393983cde6aa3d9c6e5e1fde53c5c7
2020-10-14T13:14:11.6824530Z   Stored in directory: /Users/runner/Library/Caches/pip/wheels/ac/fd/c5/7426f46ac975f85f721bcc1596eefafef37a852237e644c350
2020-10-14T13:14:11.6841530Z   Building wheel for future (setup.py): started
2020-10-14T13:14:12.6027580Z   Building wheel for future (setup.py): finished with status 'done'
2020-10-14T13:14:12.6049550Z   Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491059 sha256=c90a7f135eb94f5a7c90b2c2f9f0cbf4cfde8c4350376a9f7e6efbdf03b5ea55
2020-10-14T13:14:12.6058920Z   Stored in directory: /Users/runner/Library/Caches/pip/wheels/2f/a0/d3/4030d9f80e6b3be787f19fc911b8e7aa462986a40ab1e4bb94
2020-10-14T13:14:12.6060840Z Successfully built pexpect psutil timeout-decorator thrift future
2020-10-14T13:14:12.6502220Z Installing collected packages: pexpect, psutil, timeout-decorator, six, thrift, argparse, future, osquery
2020-10-14T13:14:13.7134670Z Successfully installed argparse-1.4.0 future-0.18.2 osquery-3.0.6 pexpect-3.3 psutil-5.7.2 six-1.15.0 thrift-0.11.0 timeout-decorator-0.4.1
2020-10-14T13:14:14.3883110Z 
2020-10-14T13:14:14.3957270Z ##[section]Finishing: Install Homebrew and prerequisites

Does this seem like a transient issue given that the Debug build worked just fine?

theopolis · 2020-10-14T16:48:54Z

If you close and reopen the PR it will run all the tests again (this is a good trick for discovering flaky/transient issues).

sharvilshah · 2020-10-20T10:18:25Z

There is no way to ssh into the test runner/instance right?

The DEBUG target still can't find psutil:

2020-10-20T07:01:41.1869450Z       Start 81: tools_tests_testosqueryd
2020-10-20T07:01:41.1869910Z 
2020-10-20T07:01:41.1871780Z 81: Test command: /usr/local/Cellar/cmake/3.18.4/bin/cmake "-E" "env" "PYTHONPATH=/Users/runner/work/1/b/build/python_path" "/usr/local/Frameworks/Python.framework/Versions/3.8/bin/python3.8" "-u" "test_osqueryd.py" "--verbose" "--build" "/Users/runner/work/1/b/build" "--test-configs-dir" "/Users/runner/work/1/b/build/test_configs"
2020-10-20T07:01:41.1872960Z 81: Test timeout computed to be: 300
2020-10-20T07:01:41.5796140Z 81: Traceback (most recent call last):
2020-10-20T07:01:41.5796910Z 81:   File "test_osqueryd.py", line 18, in <module>
2020-10-20T07:01:41.5797250Z 81:     import test_base
2020-10-20T07:01:41.5797690Z 81:   File "/Users/runner/work/1/b/build/tools/tests/test_base.py", line 13, in <module>
2020-10-20T07:01:41.5798160Z 81:     import psutil
2020-10-20T07:01:41.5799270Z 81: ModuleNotFoundError: No module named 'psutil'
2020-10-20T07:01:41.5900990Z 81/83 Test #81: tools_tests_testosqueryd ..............................................***Failed    0.40 sec
2020-10-20T07:01:41.6310960Z test 82
2020-10-20T07:01:41.6311440Z       Start 82: tools_tests_testosqueryi
2020-10-20T07:01:41.6311750Z 
2020-10-20T07:01:41.6313930Z 82: Test command: /usr/local/Cellar/cmake/3.18.4/bin/cmake "-E" "env" "PYTHONPATH=/Users/runner/work/1/b/build/python_path" "/usr/local/Frameworks/Python.framework/Versions/3.8/bin/python3.8" "-u" "test_osqueryi.py" "--verbose" "--build" "/Users/runner/work/1/b/build" "--test-configs-dir" "/Users/runner/work/1/b/build/test_configs"
2020-10-20T07:01:41.6315140Z 82: Test timeout computed to be: 300
2020-10-20T07:01:41.7528000Z 82: Traceback (most recent call last):
2020-10-20T07:01:41.7528980Z 82:   File "test_osqueryi.py", line 17, in <module>
2020-10-20T07:01:41.7529780Z 82:     import test_base
2020-10-20T07:01:41.7530230Z 82:   File "/Users/runner/work/1/b/build/tools/tests/test_base.py", line 13, in <module>
2020-10-20T07:01:41.7530760Z 82:     import psutil
2020-10-20T07:01:41.7531950Z 82: ModuleNotFoundError: No module named 'psutil'
2020-10-20T07:01:41.7630710Z 82/83 Test #82: tools_tests_testosqueryi ..............................................***Failed    0.13 sec

But the RELEASE one is just fine

Smjert · 2020-10-20T13:53:02Z

The DEBUG target still can't find psutil:

2020-10-20T07:01:41.1869450Z       Start 81: tools_tests_testosqueryd
2020-10-20T07:01:41.1869910Z 
2020-10-20T07:01:41.1871780Z 81: Test command: /usr/local/Cellar/cmake/3.18.4/bin/cmake "-E" "env" "PYTHONPATH=/Users/runner/work/1/b/build/python_path" "/usr/local/Frameworks/Python.framework/Versions/3.8/bin/python3.8" "-u" "test_osqueryd.py" "--verbose" "--build" "/Users/runner/work/1/b/build" "--test-configs-dir" "/Users/runner/work/1/b/build/test_configs"
2020-10-20T07:01:41.1872960Z 81: Test timeout computed to be: 300
2020-10-20T07:01:41.5796140Z 81: Traceback (most recent call last):
2020-10-20T07:01:41.5796910Z 81:   File "test_osqueryd.py", line 18, in <module>
2020-10-20T07:01:41.5797250Z 81:     import test_base
2020-10-20T07:01:41.5797690Z 81:   File "/Users/runner/work/1/b/build/tools/tests/test_base.py", line 13, in <module>
2020-10-20T07:01:41.5798160Z 81:     import psutil
2020-10-20T07:01:41.5799270Z 81: ModuleNotFoundError: No module named 'psutil'
2020-10-20T07:01:41.5900990Z 81/83 Test #81: tools_tests_testosqueryd ..............................................***Failed    0.40 sec
2020-10-20T07:01:41.6310960Z test 82
2020-10-20T07:01:41.6311440Z       Start 82: tools_tests_testosqueryi
2020-10-20T07:01:41.6311750Z 
2020-10-20T07:01:41.6313930Z 82: Test command: /usr/local/Cellar/cmake/3.18.4/bin/cmake "-E" "env" "PYTHONPATH=/Users/runner/work/1/b/build/python_path" "/usr/local/Frameworks/Python.framework/Versions/3.8/bin/python3.8" "-u" "test_osqueryi.py" "--verbose" "--build" "/Users/runner/work/1/b/build" "--test-configs-dir" "/Users/runner/work/1/b/build/test_configs"
2020-10-20T07:01:41.6315140Z 82: Test timeout computed to be: 300
2020-10-20T07:01:41.7528000Z 82: Traceback (most recent call last):
2020-10-20T07:01:41.7528980Z 82:   File "test_osqueryi.py", line 17, in <module>
2020-10-20T07:01:41.7529780Z 82:     import test_base
2020-10-20T07:01:41.7530230Z 82:   File "/Users/runner/work/1/b/build/tools/tests/test_base.py", line 13, in <module>
2020-10-20T07:01:41.7530760Z 82:     import psutil
2020-10-20T07:01:41.7531950Z 82: ModuleNotFoundError: No module named 'psutil'
2020-10-20T07:01:41.7630710Z 82/83 Test #82: tools_tests_testosqueryi ..............................................***Failed    0.13 sec

But the RELEASE one is just fine

Looking on the CI, it seems that it's bound to a specific version of the image.
There are runs where also the release build errored out in the same manner.

Current image version: '20201003.1' these are the ones that do not fail
Current image version: '20201011.1' these are the ones that always fail

So there might be something in the env vars or install paths that's misconfigured, which impedes python to find system installed libraries.

There is no way to ssh into the test runner/instance right?

Not that I know of; I've searched in the past but there's no mention about it or options on the UI I could find.

sharvilshah · 2020-10-20T15:36:20Z

Thanks @Smjert, that's super helpful. It seems we are hitting actions/runner-images#1819 and actions/runner-images#1818

Smjert · 2020-11-26T14:24:54Z

@sharvilshah I think those two issues have been fixed, there are other test failures now.

About updating the xcode toolchain version, can we also update it in the docs?
Here still mentions version 10.2.1 (which I forgot to update when changing to 10.3): https://github.com/osquery/osquery/blob/master/docs/wiki/development/building.md

Thanks!

This will also help with the EndpointSecurity work

theopolis · 2020-12-18T15:16:28Z

Looks like one more fix to table integration tests (or actual table code) is needed: browser_plugins.

sharvilshah · 2020-12-18T15:21:22Z

Okay, taking a look.

sharvilshah · 2020-12-21T13:44:53Z

osquery> .mode line
osquery> select * from browser_plugins;
               uid = 0
              name = QuickTime Plug-in 7.7.3
        identifier = com.apple.QuickTime Plugin.plugin
           version = 7.7.3
               sdk = 
       description = The QuickTime Plugin allows you to view a wide variety of multimedia content in web pages. For more information, visit the <A HREF="http://www.apple.com/quicktime">QuickTime</A> Web site.
development_region = en_US
            native = 0
              path = /Library/Internet Plug-Ins/Disabled Plug-Ins/QuickTime Plugin.plugin/
          disabled = 1
osquery> select * from os_version;
         name = Mac OS X
      version = 10.14.6
        major = 10
        minor = 14
        patch = 6
        build = 18G103
     platform = darwin
platform_like = darwin
     codename = 
         arch = x86_64

Regarding the failing browser_plugins test, this is failing because on macOS 10.14 and prior, there is always a quicktime plug-in that's installed (although disabled). That's what the test expects.

Starting with macOS 10.15 Catalina, C/NPAPI plugins are deprecated, and QuickTime being one of them is removed. Hence there are no plugins installed on the system by default.

Is there still any value to having this test? We can of course branch... if running on < 10.15, validate rows, if not assume non exists. Thoughts?

theopolis · 2020-12-21T13:57:23Z

Good find, yeah I think we should branch and test the two distinct expectations.

sharvilshah · 2020-12-21T16:47:39Z

Thanks @theopolis! I think this should be good to go now.

terracatta · 2020-12-21T19:04:10Z

Not to derail this, but I want to point out building on Big Sur vs Catalina will impact the os_version table. See my failed PR #6824 (comment).

I realize it's late in the game to consider using 11.x vs 10.15 but I just wanted to throw this out there in case there isn't much new test breakage building on Big Sur vs Catalina.

Smjert · 2020-12-21T19:10:22Z

Not to derail this, but I want to point out building on Big Sur vs Catalina will impact the os_version table. See my failed PR #6824 (comment).

I realize it's late in the game to consider using 11.x vs 10.15 but I just wanted to throw this out there in case there isn't much new test breakage building on Big Sur vs Catalina.

There's no Big Sur support on Azure Pipelines for now unfortunately.

sharvilshah · 2020-12-21T19:15:21Z

Hey @terracatta, thanks! I am aware about the version differences, but as Stefano mentioned, Big Sur images aren't yet available on Azure Pipelines or Github Actions.

This just moves the needle from Mojave on the CI.
Whenever Big Sur images are baked and and we adopt them, we can temporarily use SYSTEM_VERSION_COMPAT=1 to deal with the breakages, if any, until we update our logic to deal with 10.16/11.0 naming.

mike-myers-tob added CI/CD Anything about the Continuous Integration or Continuous Deployment tool used by the repository macOS macOS Catalina For things pertaining to macOS 10.15 labels Sep 29, 2020

sharvilshah changed the title ~~[Draft] [CI] Update macOS agent to 10.15 Catalina~~ [CI] Update macOS agent to 10.15 Catalina Oct 2, 2020

sharvilshah marked this pull request as ready for review October 2, 2020 17:15

mike-myers-tob requested a review from alessandrogario October 2, 2020 17:22

sharvilshah closed this Oct 8, 2020

sharvilshah reopened this Oct 8, 2020

mike-myers-tob mentioned this pull request Oct 9, 2020

Update Xcode version in Azure pipeline ahead of Oct 20th deprecation #6707

Closed

sharvilshah closed this Oct 14, 2020

sharvilshah reopened this Oct 14, 2020

sharvilshah closed this Oct 20, 2020

sharvilshah reopened this Oct 20, 2020

Smjert closed this Nov 21, 2020

Smjert reopened this Nov 21, 2020

Smjert closed this Nov 21, 2020

Smjert reopened this Nov 21, 2020

sharvilshah added 2 commits December 17, 2020 22:23

[CI] Update macOS agent to 10.15 Catalina

249242a

This will also help with the EndpointSecurity work

update xcode

71d0944

Update docs for Xcode version

476f9ec

theopolis force-pushed the patch-1 branch from 75196b7 to 476f9ec Compare December 18, 2020 03:27

Update and fix browser_plugin test for macOS 10.15 Catalina

1f5f3b1

theopolis approved these changes Dec 22, 2020

View reviewed changes

theopolis merged commit fb2bb95 into osquery:master Dec 22, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] Update macOS agent to 10.15 Catalina #6680

[CI] Update macOS agent to 10.15 Catalina #6680

sharvilshah commented Sep 29, 2020 •

edited by mike-myers-tob

Loading

sharvilshah commented Sep 29, 2020

sharvilshah commented Oct 2, 2020

Smjert commented Oct 2, 2020 •

edited

Loading

sharvilshah commented Oct 14, 2020

sharvilshah commented Oct 14, 2020

sharvilshah commented Oct 14, 2020

theopolis commented Oct 14, 2020

sharvilshah commented Oct 20, 2020

Smjert commented Oct 20, 2020

sharvilshah commented Oct 20, 2020

Smjert commented Nov 26, 2020

theopolis commented Dec 18, 2020

sharvilshah commented Dec 18, 2020

sharvilshah commented Dec 21, 2020

theopolis commented Dec 21, 2020

sharvilshah commented Dec 21, 2020

terracatta commented Dec 21, 2020 •

edited

Loading

Smjert commented Dec 21, 2020 •

edited

Loading

sharvilshah commented Dec 21, 2020

[CI] Update macOS agent to 10.15 Catalina #6680

[CI] Update macOS agent to 10.15 Catalina #6680

Conversation

sharvilshah commented Sep 29, 2020 • edited by mike-myers-tob Loading

sharvilshah commented Sep 29, 2020

sharvilshah commented Oct 2, 2020

Smjert commented Oct 2, 2020 • edited Loading

sharvilshah commented Oct 14, 2020

sharvilshah commented Oct 14, 2020

sharvilshah commented Oct 14, 2020

theopolis commented Oct 14, 2020

sharvilshah commented Oct 20, 2020

Smjert commented Oct 20, 2020

sharvilshah commented Oct 20, 2020

Smjert commented Nov 26, 2020

theopolis commented Dec 18, 2020

sharvilshah commented Dec 18, 2020

sharvilshah commented Dec 21, 2020

theopolis commented Dec 21, 2020

sharvilshah commented Dec 21, 2020

terracatta commented Dec 21, 2020 • edited Loading

Smjert commented Dec 21, 2020 • edited Loading

sharvilshah commented Dec 21, 2020

sharvilshah commented Sep 29, 2020 •

edited by mike-myers-tob

Loading

Smjert commented Oct 2, 2020 •

edited

Loading

terracatta commented Dec 21, 2020 •

edited

Loading

Smjert commented Dec 21, 2020 •

edited

Loading