Skip to content

Commit

Permalink
[ISSUE apache#2576] Enhanced Configuration management (apache#2917)
Browse files Browse the repository at this point in the history 
* [ISSUE apache#2576]Enhancement the config manage for project

* [1] test code

* EventMeshServer config improve

* Extract the key prefix of the @configFiled field into @config

* Improve configuration management of SPI implementations
[1] config improve of RocketMQProducerImpl and RocketMQConsumerImpl
[2] config UT of RocketMQProducerImpl and RocketMQConsumerImpl

* Improve configuration management of SPI implementations
[1] extensionInstance config init process improve
[2] RocketMQProducerImpl and RocketMQConsumerImpl config UT improve

* Improve configuration management of Kafka SPI Impl
[1] KafkaConsumerImpl and KafkaProducerImpl config improve
[2] KafkaConsumerImpl and KafkaProducerImpl config UT

* Improve configuration management of Redis SPI Impl
[1] RedisProperties config improve

* Improve configuration management of Redis SPI Impl
[1] RedisProperties config and UT improve
[2] New parser ConvertProperties
[3] To fix a bug where parser convertEnum could not parse null values

* Fix config UT of Redis SPI Impl

* Tweak Convert's code framework, for ease of enhancement (apache#2784)

[1] Split the converter classes in Convert into separate files
[2] Add the converter field to ConfigFiled, used to specify the converter class of the field

* Improve configuration management of Knative SPI Impl (apache#2791)

[1] KnativeConsumerImpl and KnativeProducerImpl config improve
[2] KnativeConsumerImpl and KnativeProducerImpl config UT

* Improve configuration management of Pravega SPI Impl

[1] PravegaConsumerImpl and PravegaProducerImpl config improve
[2] PravegaConsumerImpl and PravegaProducerImpl config UT
[3] add new converter URIConverter

* Improve configuration management of Pulsar SPI Impl

[1] PulsarConsumerImpl and PulsarProducerImpl config improve
[2] PulsarConsumerImpl and PulsarProducerImpl config UT

* Improve configuration management of Rabbitmq SPI Impl

[1] RabbitmqConsumer and RabbitmqProducer config improve
[2] RabbitmqConsumer and RabbitmqProducer config UT

* Improve configuration management of Prometheus SPI Impl

[1] PrometheusMetricsRegistry config improve
[2] PrometheusMetricsRegistry config UT

* Improve configuration management of AuthHttpBasic SPI Impl

[1] AuthHttpBasicService config improve
[2] AuthHttpBasicService config UT

* Enhance configuration management capabilities - @configFiled

[1] Add two properties to @configFiled,findEnv / notNull, and the configuration file parsing capabilities that go with them
[2] Delete @NotNull and replace it with @ConfigFiled.notNull

* Improve configuration management of Pinpoint SPI Impl

[1] PinpointTraceService config improve
[2] PinpointTraceService config UT

* Improve configuration management of Zipkin SPI Impl

[1] ZipkinTraceService config improve
[2] ZipkinTraceService config UT

* Add repeatable functionality to @config

For tagging multiple configuration fields on a class

* Improve configuration management of TracePlugin

[1] ExporterConfiguration config improve
[2] ExporterConfiguration config UT

* Improve configuration management of WebHook

[1] WebHook config improve
[2] WebHook config UT

* Add field beNumber for @configFiled

[1] If String config field not a number, an exception is thrown

* Change the configuration management class CommonConfiguration to a new framework

[1] Replace the CommonConfiguration field modifier public with private
[2] Use getter/setter methods to access it

* Change CommonConfiguration subclass to new framework

[1] EventMeshHTTPConfiguration/EventMeshGrpcConfiguration/EventMeshTCPConfiguration

* Change CommonConfiguration and subclass to new framework

[1] EventMeshHTTPConfiguration/EventMeshGrpcConfiguration/EventMeshTCPConfiguration

* Add field notEmpty for @configFiled

[1] If it cannot be empty but is empty, an exception is thrown

* Change the SPI RocketMQ config to new framework

* Change the SPI kafka config to new framework

* Change the SPI Redis config to new framework

* Change the SPI knative config to new framework

* Change the SPI pravega config to new framework

* Change the SPI pulsar config to new framework

* Change the SPI rabbitmq config to new framework

* Change the SPI Prometheus config to new framework

* Change the SPI auth-http-basic config to new framework

* Change the SPI trace-api config to new framework

* Change the SPI pinpoint config to new framework

* Change the SPI zipkin config to new framework

* Use getter/setter methods to access EventMeshGrpcConfiguration

* Use getter/setter methods to access EventMeshHTTPConfiguration

* Enhanced Configuration management

* remove temporary blocking @ignore

* fix styleCheck

* fix UT

* rename ConfigService method

* Revision review comments

* Fixed code review comments
[1] save the last accessible status of reflection field and recovery it after custom operation

* Fixed code review comments
[1] CLASS_PATH_PREFIX/FILE_PATH_PREFIX constants optimization
[2] Some other code details optimization

* Fixed code review comments
[1] add UT for ConfigService#getConfig(ConfigInfo configInfo)

Co-authored-by: githublaohu <2372554140@qq.com>
Co-authored-by: mike_xwm <mike_xwm@126.com>
  • Loading branch information
@eight-nines @githublaohu @xwm1992
3 people authored Jan 14, 2023
1 parent 5ab4da9 commit fc7825e
Show file tree
Hide file tree
Showing 182 changed files with 4,378 additions and 3,104 deletions.
1 change: 1 addition & 0 deletions eventmesh-common/build.gradle
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ dependencies {
implementation "javax.annotation:javax.annotation-api:1.3.2"

implementation "com.github.stefanbirkner:system-rules"
implementation "org.yaml:snakeyaml"

compileOnly 'org.projectlombok:lombok'
annotationProcessor 'org.projectlombok:lombok'
Expand Down
198 changes: 58 additions & 140 deletions eventmesh-common/src/main/java/org/apache/eventmesh/common/config/CommonConfiguration.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,187 +17,105 @@

package org.apache.eventmesh.common.config;

import org.apache.eventmesh.common.utils.AssertUtils;
import org.apache.eventmesh.common.utils.ConfigurationContextUtil;
import org.apache.eventmesh.common.utils.IPUtils;

import org.apache.commons.lang3.StringUtils;
import org.apache.commons.collections4.CollectionUtils;

import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import java.util.function.Supplier;
import java.util.stream.Collectors;

import com.google.common.base.Preconditions;
import org.assertj.core.util.Strings;

import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@NoArgsConstructor
@Config(prefix = "eventMesh")
public class CommonConfiguration {
private transient String eventMeshEnv = "P";
private transient String eventMeshIDC = "FT";
private transient String eventMeshCluster = "LS";

private transient String eventMeshName = "";
private transient List<String> eventMeshProvideServerProtocols;
private transient String sysID = "5477";
private transient String eventMeshConnectorPluginType = "rocketmq";
private transient String eventMeshSecurityPluginType = "security";
private transient String eventMeshRegistryPluginType = "namesrv";
private transient List<String> eventMeshMetricsPluginType;
private transient String eventMeshTracePluginType;

private transient String namesrvAddr = "";
private transient String eventMeshRegistryPluginUsername = "";
private transient String eventMeshRegistryPluginPassword = "";
private transient Integer eventMeshRegisterIntervalInMills = 10 * 1000;
private transient Integer eventMeshFetchRegistryAddrInterval = 10 * 1000;
private transient String eventMeshServerIp = null;
private transient boolean eventMeshServerSecurityEnable = false;

private transient boolean eventMeshServerRegistryEnable = false;

private transient boolean eventMeshServerTraceEnable = false;

protected transient ConfigurationWrapper configurationWrapper;

private transient String eventMeshWebhookOrigin = "eventmesh." + eventMeshIDC;

public CommonConfiguration(ConfigurationWrapper configurationWrapper) {
this.configurationWrapper = configurationWrapper;
}

public String getMeshGroup() {
return String.join("-", this.eventMeshEnv, this.eventMeshCluster, this.sysID);
}

@ConfigFiled(field = "sysid", beNumber = true, notEmpty = true)
private String sysID = "5477";

public void init() {
@ConfigFiled(field = "server.env", notEmpty = true)
private String eventMeshEnv = "P";

if (configurationWrapper != null) {
eventMeshEnv = checkNotEmpty(ConfKeys.KEYS_EVENTMESH_ENV);
@ConfigFiled(field = "server.idc", notEmpty = true)
private String eventMeshIDC = "FT";

sysID = checkNumeric(ConfKeys.KEYS_EVENTMESH_SYSID);
@ConfigFiled(field = "server.name", notEmpty = true)
private String eventMeshName = "";

eventMeshCluster = checkNotEmpty(ConfKeys.KEYS_EVENTMESH_SERVER_CLUSTER);
@ConfigFiled(field = "server.cluster", notEmpty = true)
private String eventMeshCluster = "LS";

eventMeshName = checkNotEmpty(ConfKeys.KEYS_EVENTMESH_SERVER_NAME);
@ConfigFiled(field = "server.hostIp", reload = true)
private String eventMeshServerIp = null;

eventMeshIDC = checkNotEmpty(ConfKeys.KEYS_EVENTMESH_IDC);
@ConfigFiled(field = "registry.plugin.server-addr", notEmpty = true)
private String namesrvAddr = "";

eventMeshServerIp = get(ConfKeys.KEYS_EVENTMESH_SERVER_HOST_IP, IPUtils::getLocalAddress);

eventMeshConnectorPluginType = checkNotEmpty(ConfKeys.KEYS_ENENTMESH_CONNECTOR_PLUGIN_TYPE);
@ConfigFiled(field = "trace.plugin", notEmpty = true)
private String eventMeshTracePluginType;

eventMeshServerSecurityEnable = Boolean.parseBoolean(get(ConfKeys.KEYS_EVENTMESH_SECURITY_ENABLED, () -> "false"));
@ConfigFiled(field = "metrics.plugin", notEmpty = true)
private List<String> eventMeshMetricsPluginType;

eventMeshSecurityPluginType = checkNotEmpty(ConfKeys.KEYS_ENENTMESH_SECURITY_PLUGIN_TYPE);
@ConfigFiled(field = "registry.plugin.type", notEmpty = true)
private String eventMeshRegistryPluginType = "namesrv";

eventMeshServerRegistryEnable = Boolean.parseBoolean(get(ConfKeys.KEYS_EVENTMESH_REGISTRY_ENABLED, () -> "false"));
@ConfigFiled(field = "security.plugin.type", notEmpty = true)
private String eventMeshSecurityPluginType = "security";

eventMeshRegistryPluginType = checkNotEmpty(ConfKeys.KEYS_ENENTMESH_REGISTRY_PLUGIN_TYPE);
@ConfigFiled(field = "connector.plugin.type", notEmpty = true)
private String eventMeshConnectorPluginType = "rocketmq";

namesrvAddr = checkNotEmpty(ConfKeys.KEYS_EVENTMESH_REGISTRY_PULGIN_SERVER_ADDR);

eventMeshRegistryPluginUsername =
configurationWrapper.getProp(ConfKeys.KEYS_EVENTMESH_REGISTRY_PULGIN_USERNAME, "");
@ConfigFiled(field = "registry.plugin.username")
private String eventMeshRegistryPluginUsername = "";

eventMeshRegistryPluginPassword =
configurationWrapper.getProp(ConfKeys.KEYS_EVENTMESH_REGISTRY_PULGIN_PASSWORD, "");
@ConfigFiled(field = "registry.plugin.password")
private String eventMeshRegistryPluginPassword = "";

String metricsPluginType = configurationWrapper.getProp(ConfKeys.KEYS_EVENTMESH_METRICS_PLUGIN_TYPE);
if (StringUtils.isNotEmpty(metricsPluginType)) {
eventMeshMetricsPluginType = Arrays
.stream(metricsPluginType.split(","))
.filter(StringUtils::isNotBlank)
.map(String::trim)
.collect(Collectors.toList());
}
@ConfigFiled(field = "server.registry.registerIntervalInMills")
private Integer eventMeshRegisterIntervalInMills = 10 * 1000;

eventMeshProvideServerProtocols = getProvideServerProtocols();
@ConfigFiled(field = "server.registry.fetchRegistryAddrIntervalInMills")
private Integer eventMeshFetchRegistryAddrInterval = 10 * 1000;

eventMeshServerTraceEnable = Boolean.parseBoolean(get(ConfKeys.KEYS_EVENTMESH_TRACE_ENABLED, () -> "false"));
if (eventMeshServerTraceEnable) {
eventMeshTracePluginType = checkNotEmpty(ConfKeys.KEYS_EVENTMESH_TRACE_PLUGIN_TYPE);
}
}
}

private String checkNotEmpty(String key) {
String value = configurationWrapper.getProp(key);
if (value != null) {
value = StringUtils.deleteWhitespace(value);
}
AssertUtils.notBlack(value, key + " is invalidated");
return value;
}

private String checkNumeric(String key) {
String value = configurationWrapper.getProp(key);
if (value != null) {
value = StringUtils.deleteWhitespace(value);
}
Preconditions.checkState(StringUtils.isNotEmpty(value) && StringUtils.isNumeric(value), key + " is invalidated");
return value;
}
@ConfigFiled(field = "server.trace.enabled")
private boolean eventMeshServerTraceEnable = false;

private String get(String key, Supplier<String> defaultValueSupplier) {
String value = configurationWrapper.getProp(key);
if (value != null) {
value = StringUtils.deleteWhitespace(value);
}
return StringUtils.isEmpty(value) ? defaultValueSupplier.get() : value;
}
@ConfigFiled(field = "server.security.enabled")
private boolean eventMeshServerSecurityEnable = false;

public List<String> getProvideServerProtocols() {
String provideProtocols = configurationWrapper.getProp(ConfKeys.KEYS_EVENTMESH_SERVER_PROVIDE_PROTOCOLS);
if (StringUtils.isNotEmpty(provideProtocols)) {
return Arrays.stream(provideProtocols.split(",")).filter(StringUtils::isNotBlank).map(String::trim).collect(Collectors.toList());
} else {
return Collections.singletonList(ConfigurationContextUtil.HTTP);
}
}
@ConfigFiled(field = "server.registry.enabled")
private boolean eventMeshServerRegistryEnable = false;

static class ConfKeys {
public static final String KEYS_EVENTMESH_ENV = "eventMesh.server.env";

public static final String KEYS_EVENTMESH_IDC = "eventMesh.server.idc";
@ConfigFiled(field = "server.provide.protocols", reload = true)
private List<String> eventMeshProvideServerProtocols;

public static final String KEYS_EVENTMESH_SYSID = "eventMesh.sysid";

public static final String KEYS_EVENTMESH_SERVER_CLUSTER = "eventMesh.server.cluster";
@ConfigFiled(reload = true)
private String eventMeshWebhookOrigin;

public static final String KEYS_EVENTMESH_SERVER_NAME = "eventMesh.server.name";
@ConfigFiled(reload = true)
private String meshGroup;

public static final String KEYS_EVENTMESH_SERVER_PROVIDE_PROTOCOLS = "eventMesh.server.provide.protocols";
public void reload() {
this.eventMeshWebhookOrigin = "eventmesh." + eventMeshIDC;

public static final String KEYS_EVENTMESH_SERVER_HOST_IP = "eventMesh.server.hostIp";

public static final String KEYS_EVENTMESH_SERVER_REGISTER_INTERVAL = "eventMesh.server.registry.registerIntervalInMills";

public static final String KEYS_EVENTMESH_SERVER_FETCH_REGISTRY_ADDR_INTERVAL = "eventMesh.server.registry.fetchRegistryAddrIntervalInMills";

public static final String KEYS_ENENTMESH_CONNECTOR_PLUGIN_TYPE = "eventMesh.connector.plugin.type";

public static final String KEYS_EVENTMESH_SECURITY_ENABLED = "eventMesh.server.security.enabled";

public static final String KEYS_ENENTMESH_SECURITY_PLUGIN_TYPE = "eventMesh.security.plugin.type";

public static final String KEYS_EVENTMESH_REGISTRY_ENABLED = "eventMesh.registry.plugin.enabled";

public static final String KEYS_ENENTMESH_REGISTRY_PLUGIN_TYPE = "eventMesh.registry.plugin.type";

public static final String KEYS_EVENTMESH_REGISTRY_PULGIN_SERVER_ADDR = "eventMesh.registry.plugin.server-addr";

public static final String KEYS_EVENTMESH_REGISTRY_PULGIN_USERNAME = "eventMesh.registry.plugin.username";

public static final String KEYS_EVENTMESH_REGISTRY_PULGIN_PASSWORD = "eventMesh.registry.plugin.password";

public static final String KEYS_EVENTMESH_METRICS_PLUGIN_TYPE = "eventMesh.metrics.plugin";
if (Strings.isNullOrEmpty(this.eventMeshServerIp)) {
this.eventMeshServerIp = IPUtils.getLocalAddress();
}

public static final String KEYS_EVENTMESH_TRACE_ENABLED = "eventMesh.server.trace.enabled";
if (CollectionUtils.isEmpty(eventMeshProvideServerProtocols)) {
this.eventMeshProvideServerProtocols = Collections.singletonList(ConfigurationContextUtil.HTTP);
}

public static final String KEYS_EVENTMESH_TRACE_PLUGIN_TYPE = "eventMesh.trace.plugin";
meshGroup = String.join("-", this.eventMeshEnv, this.eventMeshCluster, this.sysID);
}
}
54 changes: 54 additions & 0 deletions eventmesh-common/src/main/java/org/apache/eventmesh/common/config/Config.java
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.eventmesh.common.config;

import java.lang.annotation.ElementType;
import java.lang.annotation.Repeatable;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

/**
* Record information about the configuration class to be converted
*/
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
@Repeatable(Config.Configs.class)
public @interface Config {

String field() default "";

String path() default "";

String prefix() default "";

String hump() default ".";

boolean removePrefix() default true;

boolean monitor() default false;

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@interface Configs {
Config[] value();
}
}



80 changes: 80 additions & 0 deletions eventmesh-common/src/main/java/org/apache/eventmesh/common/config/ConfigFiled.java
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.eventmesh.common.config;

import org.apache.eventmesh.common.config.convert.ConvertValue.DefaultConverter;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

/**
* Record information about the field in the configuration class to be converted
*/
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.FIELD})
public @interface ConfigFiled {

/**
* @return The key name of the configuration file
*/
String field() default "";

/**
* Note : When reload is true, the class must have a reload method
*
* @return Whether to reload. This parameter is used when other fields are associated
*/
boolean reload() default false;

/**
* In some special cases, used to specify the converter class of the field
*
* @return field converter
*/
Class<?> converter() default DefaultConverter.class;

/**
* if the configuration filed is empty, try to read from env, by field
*
* @return Whether to try to read from env if the configuration filed is empty
*/
boolean findEnv() default false;

/**
* If it cannot be null but is null, an exception is thrown
*
* @return Whether the field can be null
*/
boolean notNull() default false;

/**
* If it cannot be empty but is empty, an exception is thrown
*
* @return Whether the field can be empty
*/
boolean notEmpty() default false;

/**
* If it's not a number, an exception is thrown
*
* @return Whether the field must be number
*/
boolean beNumber() default false;
}
Loading

0 comments on commit fc7825e

Please sign in to comment.