Merge pull request #6 from opensrp/populate-existing-location

Populate existing location
opensrp · Sep 24, 2021 · e8281ef · e8281ef
2 parents 4ef9fa8 + d4442a1
commit e8281ef
Show file tree

Hide file tree

Showing 14 changed files with 440 additions and 232 deletions.
diff --git a/README.MD b/README.MD
@@ -30,9 +30,9 @@ This tool can read data from 2 sources:
 
 1. Download the latest version of the `*-fat.jar` from [Releases](https://github.com/ellykits/opensrp-data-import/releases). Alternatively you can build the project by running the command `./gradlew shadowJar` then locate the uber jar inside `build/lib` directory.
 
-2. Copy the content of the file `application_sample.properties` into your configs file. Replace the placeholders  with the correct values. It is okay to retain the defaults like `request.limit` config but you can alter the figures like required. (Optimal number of batch record that can be processed by OpenSRP in one request is 60 - this is of course dependent on the server specs)
+2. Copy the content of the file `application_sample.properties` into your configs file. Replace the placeholders  with the correct values. It is okay to retain the defaults like `request.limit` config, but you can alter the figures like required. (Optimal number of batch record that can be processed by OpenSRP in one request is 60 - this is of course dependent on the server specs)
 
-3. Execute the commands as desired. When importing data from OpenMRS it is adviced to follow the order defined in the documentation.
+3. Execute the commands as desired. When importing data from OpenMRS it is advised to follow the order defined in the documentation.
 
 > NOTE: Some actions are dependent on others for instance you cannot import teams from OpenMRS without adding the locations first.
 
@@ -68,7 +68,7 @@ For example run the following command to list available options (`app.properties
 
 ```shell script
 
-java -jar opensrp-data-import-3.0.5-SNAPSHOT-fat.jar --configs-file app.properties  --help
+java -jar opensrp-data-import-3.1.0-SNAPSHOT-fat.jar --configs-file app.properties  --help
 
 ```
 
@@ -126,21 +126,21 @@ Country Id,Country,Region Id,Region,County Id,County,Health Facility Id,Health F
 ,Kenya,,Nairobi,,Nairob,,Nairobi Hospital
 ```
 
-The ID columns are mandatory as they are used to map locations to their parents. EMPTY ID column means new location, so the system will generate a UUID for the specified location. Existing locations MUST have their IDs pre-populated to avoid duplications. Run the locations with EMPTY ID column once; you can retrieve the generated location ids from this file `/home/<your_username>/opensrp-data/locations.csv` for future use.
+The ID columns are mandatory as they are used to mapping locations to their parents. EMPTY ID column means new location, so the system will generate a UUID for the specified location. Existing locations MUST have their IDs pre-populated to avoid duplications. Run the locations with EMPTY ID column once; you can retrieve the generated location ids from this file `/home/<your_username>/opensrp-data/locations.csv` for future use.
 
 Also NOTE that the columns should be ordered according to the location hierarchy. It is logical to group all the locations belonging to one level.
 
-The tool will validate the csv columns and fail if the correct format is not provided. The only thing that is assumed is you have ordered them corectly.
+The tool will validate the csv columns and fail if the correct format is not provided. The only thing that is assumed is you have ordered them correctly.
 
 #### Users file CSV file format
-The CSV tempalte for importing users to Keycloak and adding as practitioners in OpenSRP is as follows:
+The CSV template for importing users to Keycloak and adding as practitioners in OpenSRP is as follows:
 
 ```csv
 Parent Location,Location,First Name,Last Name,Username,Password
 ```
 
 The names of the column headers is not important as the CSV reader uses the position of the column to read the file, the content however MUST be in that order.
-The **Parent Location** is necessary because you can have locations sharing a name at different level. Also not that the names MUST be exactly what was provided in the source file.
+The **Parent Location** is necessary because you can have locations sharing a name at different level. Also, not that the names MUST be exactly what was provided in the source file.
 
 
 #### Accessing Generated CSV files
@@ -168,6 +168,7 @@ This tool will generate CSV files with the content of the data that has been pos
 | keycloak.rest.users.count.url | Endpoint for counting Keycloak Users. Format *{{keycloak-host}}/auth/admin/realms/{{realm}}/users/count* |
 | keycloak.rest.groups.url | Endpoint for fetching Keycloak groups. Format *{{keycloak-host}}/auth/admin/realms/{{realm}}/groups* |
 | opensrp.rest.location.url | OpenSRP endpoint for posting locations |
+| opensrp.rest.location.fetch.url | Endpoint for fetching all opensrp locations |
 | opensrp.rest.location.tag.url |OpenSRP endpoint for posting location tags |
 | opensrp.rest.organization.url | OpenSRP endpoint for posting organizations|
 | opensrp.rest.organization.location.url | OpenSRP endpoint for mapping organizations to locations |
@@ -179,7 +180,7 @@ This tool will generate CSV files with the content of the data that has been pos
 | request.interval | Set time interval in milliseconds between each request. Default `10000` |
 | request.timeout | Sets the timeout in milliseconds. If an action is not completed before this timeout, the action is considered as a failure default `-1` (no timeout)|
 | reset.timeout |  Sets the time in ms before it attempts to re-close the circuit (by going to the half-open state). If the circuit is closed when the timeout is reached, nothing happens. `-1` disables this feature. Default `10000` |
-
+| team.name.prefix | Prefix for the name of team e.g. if the prefix is 'Team', the organizations will be created as Team Jenga, Team Ona |
 
 ### Building Project
 

diff --git a/build.gradle.kts b/build.gradle.kts
@@ -10,7 +10,7 @@ plugins {
 }
 
 group = "org.smartregister"
-version = "3.0.5-SNAPSHOT"
+version = "3.1.0-SNAPSHOT"
 
 repositories {
   mavenCentral()

diff --git a/conf/application_sample.properties b/conf/application_sample.properties
@@ -16,6 +16,7 @@ keycloak.rest.groups.url={{keycloak-host}}/auth/admin/realms/{{realm}}/groups
 single.request.interval=500
 keycloak.request.delay=50000
 opensrp.rest.location.url={{opensrp-host}}/opensrp/rest/location/add?is_jurisdiction=true
+opensrp.rest.location.fetch.url={{opensrp-host}}/opensrp/rest/location/getAll
 opensrp.rest.location.tag.url={{opensrp-host}}/opensrp/rest/location-tag
 opensrp.rest.organization.url={{opensrp-host}}/opensrp/rest/organization/add
 opensrp.rest.organization.location.url={{opensrp-host}}/opensrp/rest/organization/assignLocationsAndPlans
@@ -27,3 +28,4 @@ request.interval=10000
 request.timeout=-1
 reset.timeout=2000
 circuit.breaker.max.retries=10
+team.name.prefix=""
diff --git a/src/main/kotlin/org/smartregister/dataimport/Debug.kt b/src/main/kotlin/org/smartregister/dataimport/Debug.kt
@@ -16,12 +16,15 @@ import org.smartregister.dataimport.shared.*
 fun main() {
   val configs = JsonObject().apply {
     put(IMPORT_OPTION, "locations")
-//    put(SOURCE_FILE, "assets/locations.csv")
-//    put(USERS_FILE, "assets/liberia-users.csv")
-//    put(SKIP_USER_GROUP, true)
+    put(SOURCE_FILE, "assets/locations.csv")
+    put(USERS_FILE, "assets/users.csv")
+    put(LOAD_EXISTING_LOCATIONS, true)
+//    put(ORGANIZATION_LOCATIONS_FILE, "assets/organization_locations.csv")
+    put(CREATE_NEW_TEAMS, "yes")
+    put(SKIP_USER_GROUP, true)
     put(SKIP_LOCATION_TAGS, true)
-//    put(SKIP_LOCATIONS, true)
-//    put(GENERATE_TEAMS, "Health Facility")
+    put(SKIP_LOCATIONS, true)
+    put(GENERATE_TEAMS, "Health Facility")
   }
   Vertx.vertx().deployVerticle(MainVerticle(), deploymentOptionsOf(config = configs))
 }
diff --git a/src/main/kotlin/org/smartregister/dataimport/Main.kt b/src/main/kotlin/org/smartregister/dataimport/Main.kt
@@ -28,19 +28,32 @@ class Application : CliktCommand(name = "opensrp-data-import") {
     names = arrayOf("--source-file", "-s")
   )
 
-  private val overwrite: Boolean by option(
-    help = "Overwrite existing locations", names = arrayOf("--overwrite", "-oW")
+  private val loadExistingLocations: Boolean by option(
+    help = "Load existing locations from OpenSRP Server", names = arrayOf("--load-existing-locations", "-lE")
   ).flag(default = false)
 
+  private val createNewTeams: String? by option(
+    help = "Indicate whether to create new teams for existing locations. Valid options 'yes' or 'no'",
+    names = arrayOf("--create-new-teams", "-cT")
+  ).choice(
+    YES,
+    NO
+  )
+
   private val generateTeams: String? by option(
     help = "Indicate the location level for team assignment", names = arrayOf("--assign-team", "-aT")
   )
 
   private val usersFile: String? by option(
-    help = "File containing users details. This is used to export to ",
+    help = "CSV file containing user details. CSV Header(Parent Location, Location, First Name, Last Name, Username, Password) ",
     names = arrayOf("--users-file", "-u")
   )
 
+  private val organizationLocationsFile: String? by option(
+    help = "CSV file containing organization locations. CSV Header (Organization, Jurisdiction)",
+    names = arrayOf("--organization-locations-file", "-oL")
+  )
+
   private val skipLocationTags: Boolean by option(
     help = "Skip importing location tags", names = arrayOf("--skip-location-tags", "-sT")
   ).flag(default = false)
@@ -87,21 +100,30 @@ class Application : CliktCommand(name = "opensrp-data-import") {
 
     if (configsFile != null && importOption != null) {
       mainVerticle.setConfigsFile(configsFile!!)
-
       val configs = JsonObject().apply {
         put(IMPORT_OPTION, importOption)
         put(SOURCE_FILE, sourceFile)
         put(USERS_FILE, usersFile)
+        put(ORGANIZATION_LOCATIONS_FILE, organizationLocationsFile)
         put(SKIP_LOCATION_TAGS, skipLocationTags)
         put(SKIP_LOCATIONS, skipLocations)
         put(SKIP_USER_GROUP, skipUserGroup)
         put(GENERATE_TEAMS, generateTeams)
-        put(OVERWRITE, overwrite)
+        put(LOAD_EXISTING_LOCATIONS, loadExistingLocations)
+        put(CREATE_NEW_TEAMS, createNewTeams)
       }
-
       vertx.deployVerticle(mainVerticle, deploymentOptionsOf(config = configs))
     } else {
-      echo("--configs-file and --import options are required --source-file optional. Use --help for more information")
+      echo(
+        """
+          Error: Missing required command option
+          Description:
+          --configs-file: Provide configurations file (Required *)
+          --import: Indicate the resource to import e.g. locations (Required *)
+          Solution:
+          Run --help for more information
+        """.trimMargin()
+      )
     }
   }
 }
diff --git a/src/main/kotlin/org/smartregister/dataimport/openmrs/BaseOpenMRSVerticle.kt b/src/main/kotlin/org/smartregister/dataimport/openmrs/BaseOpenMRSVerticle.kt
@@ -42,7 +42,6 @@ abstract class BaseOpenMRSVerticle : BaseVerticle() {
       vertx.exceptionHandler().handle(sqlException)
     }
 
-
     limit = config.getInteger("data.limit", 50)
   }
 

diff --git a/src/main/kotlin/org/smartregister/dataimport/opensrp/OpenSRPPractitionerVerticle.kt b/src/main/kotlin/org/smartregister/dataimport/opensrp/OpenSRPPractitionerVerticle.kt
@@ -27,7 +27,6 @@ class OpenSRPPractitionerVerticle : BaseOpenSRPVerticle() {
 
     if (config.getString(SOURCE_FILE, "").isNullOrBlank()) {
       val baseUrl = config.getString("keycloak.rest.users.url")
-      val queryParameters = mutableMapOf(FIRST to "0", MAX to limit.toString())
       val countResponse = awaitResult<HttpResponse<Buffer>?> {
         webRequest(
           method = HttpMethod.GET,
@@ -36,6 +35,7 @@ class OpenSRPPractitionerVerticle : BaseOpenSRPVerticle() {
         )
       }
 
+      val queryParameters = mutableMapOf(FIRST to "0", MAX to limit.toString())
       if (countResponse != null) {
         val userCount = countResponse.bodyAsString().toLong()
         var offset = 0

diff --git a/src/main/kotlin/org/smartregister/dataimport/opensrp/location/OpenSRPLocationTagVerticle.kt b/src/main/kotlin/org/smartregister/dataimport/opensrp/location/OpenSRPLocationTagVerticle.kt
@@ -24,7 +24,7 @@ class OpenSRPLocationTagVerticle : BaseOpenSRPVerticle() {
   override suspend fun start() {
     super.start()
     val sourceFile = config.getString(SOURCE_FILE)
-    val skipLocationsTags = config.getBoolean(SKIP_LOCATION_TAGS)
+    val skipLocationsTags = config.getBoolean(SKIP_LOCATION_TAGS, false)
     if (!skipLocationsTags) {
       if (sourceFile.isNullOrBlank()) {
         deployVerticle(OpenMRSLocationTagVerticle(), OPENMRS_LOCATION_TAGS)
-Original file line number
+Diff line change
@@ Expand Up / @@ -42,7 +42,6 @@ abstract class BaseOpenMRSVerticle : BaseVerticle() { @@
           vertx.exceptionHandler().handle(sqlException)
         }
         limit = config.getInteger("data.limit", 50)
       }
@@ Expand Down @@