Refactor data node startup, add startup mechanism for new node types #20181

moesterheld · 2024-08-14T11:56:29Z

Description

Modifies the CmdLineTool to allow configurable startup of new node types.
A GraylogNodeConfiguration interface configures certain whether certain common startup features should be run.
For a node incorporating the features we deem as common, a CommonNodeConfiguration is provided which sets these flags as default.

Data node startup has been changed to use the refactored CmdLineTool directly, setting the necessary features as required by the data node. Additionally, bootstrapping of the data node has been revamped. This does not completely decouple the startup logic from Graylog startup, but uses a common codebase instead of the previously duplicated code.

An example node is currently included in the PR to showcase how new node types can be created (ExampleCommand). This should be removed before merging.

/nocl

Motivation and Context

solves #18908
solves #18011

How Has This Been Tested?

run data node, Graylog, certutil, journal commands manually

Screenshots (if appropriate):

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Refactoring (non-breaking change)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.

…ks for fresh installation of the Graylog cluster

…sn't do the fresh installation detection any more

todvora

Tested locally, works fine and the code seems good to me. But it's still a draft and we need somebody from architecture to look at this, so only commenting, not approving for now.

…Graylog node

bernd

A first quick review. Will do testing and a more detailed review next.

data-node/src/main/java/org/graylog/datanode/bootstrap/DatanodeBootstrap.java

data-node/src/main/java/org/graylog/datanode/commands/Datanode.java

data-node/src/main/java/org/graylog/datanode/bootstrap/DatanodeCmdLineTool.java

graylog2-server/src/main/java/org/graylog2/bootstrap/CmdLineTool.java

graylog2-server/src/main/java/org/graylog2/commands/ExampleCommand.java

graylog2-server/src/main/java/org/graylog2/commands/ExampleCommandConfiguration.java

graylog2-server/src/main/java/org/graylog2/commands/ServerCommandsProvider.java

Co-authored-by: Bernd Ahlers <bernd@users.noreply.github.com>

bernd

Thanks! A few comments.

graylog2-server/src/main/java/org/graylog2/CommonNodeConfiguration.java

data-node/src/main/java/org/graylog/datanode/commands/Datanode.java

data-node/src/main/java/org/graylog/datanode/bootstrap/DatanodeBootstrap.java

graylog2-server/src/main/java/org/graylog2/bindings/GraylogNodeModule.java

graylog2-server/src/main/java/org/graylog2/bindings/MongoDbConnectionModule.java

graylog2-server/src/main/java/org/graylog2/commands/AbstractNodeCommand.java

graylog2-server/src/main/java/org/graylog2/featureflag/ImmutableFeatureFlagsCollector.java

…Command

moesterheld · 2024-12-05T14:42:01Z

@bernd I addressed your comments.
I also cleaned up the redundant bindings and made all commands, including the cli commands use the AbstractNodeCommand. The only redundant bindings are now for preflight. In the future, we could think about making preflight bindings configurable from a GraylogNodePreflightModule and/or getPreflightCommandBindings methods in the AbstractNodeCommand.

bernd

I left two pings in comments that haven't been answered. (or I didn't see it)

data-node/src/main/java/org/graylog/datanode/bootstrap/DatanodeBootstrap.java

graylog2-server/src/main/java/org/graylog2/bindings/GraylogNodeModule.java

# Conflicts: # data-node/src/main/java/org/graylog/datanode/bindings/PreflightChecksBindings.java

bernd

Thank you! 🙏

thll · 2024-12-13T20:25:03Z

graylog2-server/src/main/java/org/graylog2/bindings/GraylogNodeModule.java

+        install(new ServerStatusBindings(configuration.withCapabilities()));
+
+        bind(EncryptedValueService.class).asEagerSingleton();
+        bind(InputConfigurationBeanDeserializerModifier.class).toInstance(InputConfigurationBeanDeserializerModifier.withoutConfig());


@moesterheld: For the server, this binding breaks the saving of inputs that contain encrypted values. You can reproduce it by creating an AWS CloudTrail where you only set the title (and node) and leave everything else empty. You can save it, but afterwards the inputs page is broken and you have to remove the input manually from MongoDB.

We should either go back to a just-in-time binding (for the server) or do a regular singleton scope binding. I didn't test the latter but it should be fine.

Thanks @thll , I will fix this and try to add an automated test that covers this somehow.

Thank you! Yeah, having some kind of integration test for this would be great as the cause is very subtle and hard to find.

moesterheld and others added 15 commits August 14, 2024 09:41

Initial reworked, simplified abstract node startup

8292e51

use Graylog CmdLineTool in data node

17e4b3a

remove unused pid file declaration

c2c7d71

remove freshInstallation check as it doesn't make sense since it chec…

271ea0c

…ks for fresh installation of the Graylog cluster

move mongo preflight check into regular preflight checks since it doe…

60db00d

…sn't do the fresh installation detection any more

remove migration feature for data node

a440a7c

code cleanup

77c361d

remove unused local flag, code cleaup

7440e65

rename server to datanode for better clarification

cd5eb96

fix feature flag test to use config for env/sys props

7e61307

remove Graylog path configuration from data node configuration

8420a3e

temporary fix for excluding zstd library fix from data node

9dc7534

add requireexplicitbindings for new node types

82a0a19

Merge branch 'master' into refactor/abstract-node-startup

13435d6

remove unneeded settings bean

0cbad48

moesterheld requested a review from todvora August 16, 2024 14:46

Merge branch 'master' into refactor/abstract-node-startup

9c1cfe3

todvora reviewed Sep 5, 2024

View reviewed changes

moesterheld and others added 4 commits September 23, 2024 16:33

Merge branch 'master' into refactor/abstract-node-startup

1f315bf

Merge branch 'master' into refactor/abstract-node-startup

ac8ff99

do not use pluginLoaderConfig for fixing zstd temp directory

881e1eb

do tls protocols configuration and setting of netty defaults only in …

c2c48e1

…Graylog node

moesterheld marked this pull request as ready for review October 18, 2024 09:58

moesterheld requested a review from bernd October 18, 2024 15:09

Merge branch 'master' into refactor/abstract-node-startup

5a7089c

bernd requested changes Nov 21, 2024

View reviewed changes

Merge branch 'master' into refactor/abstract-node-startup

fb678d3

bernd mentioned this pull request Nov 21, 2024

Experiment to introduce an abstract node command #19110

Closed

moesterheld added 2 commits November 22, 2024 09:26

remove ExampleCommand, replace with MinimalNode test

7087af4

change forbidden method invocation

4d22665

moesterheld and others added 4 commits November 22, 2024 09:51

Change cmdline option description

861f0e8

Co-authored-by: Bernd Ahlers <bernd@users.noreply.github.com>

make usage of NodeIdFile configurable

9ea12dd

Merge branch 'master' into refactor/abstract-node-startup

7910ccf

adjust ServerStatus and provide common node command test

2cc4aa8

bernd requested changes Nov 27, 2024

View reviewed changes

bernd mentioned this pull request Nov 28, 2024

Add new command to generate IntelliJ run configurations Graylog2/graylog-project-cli#48

Open

moesterheld and others added 11 commits November 28, 2024 17:12

code cleanup

dab9b56

make configuration a field, remove unused constructor

0dc9999

code cleanup

a6ccc9b

use parent field

7eb0b08

make Datanode and Server extend AbstractNodeCommand

7be3252

Merge branch 'master' into refactor/abstract-node-startup

1a4e19b

change field names to camel case

6fafa14

remove redundant bindings, make journal commands extends AbstractNode…

f702c6e

…Command

remove static field

d39ca69

Merge branch 'master' into refactor/abstract-node-startup

03b8c74

remove redundant bindings

fa2ecbf

moesterheld requested a review from bernd December 5, 2024 14:42

bernd requested changes Dec 10, 2024

View reviewed changes

data-node/src/main/java/org/graylog/datanode/bootstrap/DatanodeBootstrap.java Show resolved Hide resolved

graylog2-server/src/main/java/org/graylog2/bindings/GraylogNodeModule.java Outdated Show resolved Hide resolved

moesterheld added 2 commits December 10, 2024 15:09

always applySecuritySettings

c246d8b

add javadocs to GraylogNodeModule

a203f34

moesterheld requested a review from bernd December 10, 2024 14:26

moesterheld and others added 2 commits December 10, 2024 15:26

Merge branch 'master' into refactor/abstract-node-startup

f445c4c

Merge branch 'master' into refactor/abstract-node-startup

9351d89

# Conflicts: # data-node/src/main/java/org/graylog/datanode/bindings/PreflightChecksBindings.java

bernd approved these changes Dec 12, 2024

View reviewed changes

moesterheld merged commit b4e1e07 into master Dec 12, 2024
6 checks passed

moesterheld deleted the refactor/abstract-node-startup branch December 12, 2024 15:02

thll reviewed Dec 13, 2024

View reviewed changes

moesterheld mentioned this pull request Dec 16, 2024

Fix input configuration encryption error #21196

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor data node startup, add startup mechanism for new node types #20181

Refactor data node startup, add startup mechanism for new node types #20181

moesterheld commented Aug 14, 2024

todvora left a comment

bernd left a comment

bernd left a comment

moesterheld commented Dec 5, 2024

bernd left a comment •

edited

Loading

bernd left a comment

thll Dec 13, 2024

moesterheld Dec 16, 2024

thll Dec 16, 2024

Refactor data node startup, add startup mechanism for new node types #20181

Refactor data node startup, add startup mechanism for new node types #20181

Conversation

moesterheld commented Aug 14, 2024

Description

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

todvora left a comment

Choose a reason for hiding this comment

bernd left a comment

Choose a reason for hiding this comment

bernd left a comment

Choose a reason for hiding this comment

moesterheld commented Dec 5, 2024

bernd left a comment • edited Loading

Choose a reason for hiding this comment

bernd left a comment

Choose a reason for hiding this comment

thll Dec 13, 2024

Choose a reason for hiding this comment

moesterheld Dec 16, 2024

Choose a reason for hiding this comment

thll Dec 16, 2024

Choose a reason for hiding this comment

bernd left a comment •

edited

Loading