Rewards and validators improvements

Stop indexing on error Fix error when validator is not presented in Reward list events Set https endpoint for Kusama staging Update Kusama staging api url Set batch size 30 Switch to https Set batchSize 15 Set batchSize 10 Improve rewards handling Set default batch size Fix undefined slashDeferDuration for older specVersions Create LICENSE Fix polkadot sync error (#37) Adding Docker waiting and changing naming Update project.yaml to 0.2.0 (#38) * update project.yaml to 0.2.0 * delete yarn.lock * add yarn.lock to gitignore NOVA_3665: switch <action>.multi to multiQuery (#40) ISSUE-39: Fee calculation with withdraw (#42) Set extrinsicHash as optional ISSUE-39: Fee calculation with withdraw (#42) Update polkadot api Check erasStakersClipped exists Update readme and package name Set extrinsic hash for stacke change as optional Fix failed reward destribution handling Adding GitHub Actions for deploy to staging (#63) * add statemine Signed-off-by: Stepan Lavrentev <lawrentievsv@gmail.com> * fix constraint * ISSUE-51: Assets transfer history * NOVA_3665: switch <action>.multi to multiQuery (#40) * ISSUE-39: Fee calculation with withdraw (#42) * Update polkadot api * Check erasStakersClipped exists * Update readme and package name * Set extrinsic hash as optional * Custom asset transfer * first iteration * fix scripts * fix * fix deploy branch * fix * Transform assetTransfer to jsonfield * upgrade polkadot/api to 7 * fix schema * rename * update workflows * Fix for assettransfer checking * add more networks in list * add tests for promote to prod * add new networks * Fix transfer problem * fix bifrost and karura * Add failed transfer all handlers * add promotion to prod with checks * add more networks * add networks and fix * Don't create receiver history element for transferAll * temporal --force * remove --force * add fix for moonbeam * fix version for moonriver * fix shiden * change acala url * add new networks * Merge pull request #77 from nova-wallet/add-evm-hash-support Add evm hash support * update moonbeam types * kusama rewards fix * Merge pull request #80 from nova-wallet/fix/change-kintsugi-account-type Fix: change kintsugi account type * change some parameters for GHA * Delete robonomics * Improving Usability of Development * Add manual deploy workflow Co-authored-by: asmadek <alma2610@ya.ru> ISSUE-85: fix extrinsic creation for evm transactions Add configs for networks Revert "Add configs for networks" This reverts commit 7cb93e2. Fuxing problems after first iteration (#87) * update trigger for workflow * add script to update projects description * Fix readme * subsocial types version * turn off auto deployment Reolve conflicts problem in GH actions (#91) fix fix fix fix fix fix fix fix fix update try fix fix fix Adding merge master adding crab-solochain (#94) Remove hardcode parameters from GH workflow (#95) * add more workflow parameters * fixing typos Fix for nodle-solochain and subsocial-solochain (#93) * fix nodle solochain * fix for subsocial-solochain * fixing type format Adding new chains. Phala/Equilibrium/Turing (#99) * adding phala network * fix for nodle solochain * Adding equilibrium * Adding turing network CU-2d8auyg_MoonriverMoonbeam-problems (#101) * set version of docker image * Use polkadot util for etherium address encode Co-authored-by: asmadek <alma2610@ya.ru> fix: fee calculation for evm transactions Adding restrictions for ipfs-unixfs (#104) Adding Aleph Zero network (#107) Change endpoint to OnFinality (#110) * Change endpoint on OnFinality * Change all available projects to onfinality node Adding composable network (#111) Add parachain stacking support fix era index with payment delay Changing start block for moonbeam/moonriver (#117) Adding polkadex project (#121) Fixed moonbeam for custom assets (#123) * Fixed problem with moonbeam custom_assets * Adding handlers to all networks Adding condition about fee calculation (#125) GitHub Actions. Improve process. (#115) * change manual deploy logic * adding project status workflow * add network status generation * set schedule cron * add manual run Integritee and nodle solochains deleting (#128) Adding parachain staking to Turing (#130) Update description for all projects (#119) * draft initialisation * add some logic * update logic and testing * fix description * add westmint * update description * fix for assets projects * one of the variant of descriptins * Rework implementation * improve highlighting * fix descriptions * update some formats * add some format fix Adding OriginTrail, Litentry, UNIQUE, Dora (#132) * Adding Dora * Adding Unique network * Adding litentry * Adding bifrost-polkadot * Adding origin-trail Adding Pichiu (#135) * Add Mangata X * Add Pichiu * add types for pichiu * Revert "Add Mangata X" This reverts commit 42031b9. Co-authored-by: Stepan Lavrentev <lawrentievsv@gmail.com> Remove currencies handler Acala/Karura (#136) Co-authored-by: Stepan Lavrentev <lawrentievsv@gmail.com> Adding parachain staking to Zeitgest (#141) * Adding parachain staking to Zeitgest * remove useless handlers Adding features column for the project list (#138) * adding features colomn * make feature shorter * make orml name shorter * fix name * fix comments * improve orml\assets handling * change 'None' case handling * fix some comments Adding Bajun, Imbue, Kabocha, Tanganika projects (#140) * Adding kabocha * Adding tanganika * Adding Bajun * Adding Imbue * remove relaychain's handlers Deleting solochain projects (#147) * delete crab-solochain * Deleting Equilibrium Fixing indexer's problems (Bajun, Crust Shadow) (#148) * Fix bajun problem * fixing crust shadow Adding legend and fix size (#151) * Adding agenda and fix length * fix line breaker * fix some comments Deleting Westmint (#153) Fix for generate project statues script (#155) * Fix statuses script * add fix for stage too Adding Bifrost parachain staking (#157) Adding parachainStaking for Bifrost (#159) Update Turing node Adding GM network (#162) * Increase image versions * Adding GM network * fix naming Adding tinkernet and change name for GM (#164) Fixing some errors (#165) * Change default node for Bajun * Adding tinkernet and change name for GM * fix endpoint for basilisk * Update encointer endpoint * Update calamari endpoint * Update efinity endpoint * Delete subsocial solochain Fix chains version first iteration Acala in separate directory Adding possibility to run locally Adding acala from root folder Temporary fix for types problem Adding aleph-zero project Adding altair fix handlers for aleph-zero fix acala handlers Change enpoints for acala, altair adding astar adding bajun Adding basilisk Adding bifrost Adding bifrost-polkadot Adding bit-country rename project files adding calamari Adding centrifuge Adding clover Adding composable Adding dora Adding efinity Adding encointer Adding gm adding hydra Adding imbue Adding integritee Adding Interlay Adding kabocha Adding karura Adding khala Adding kico Adding kilt Adding kintsugi Adding kusama fix kusama Adding litentry Adding litmus Adding moonbeam Adding moonriver Adding nodle, origin-trail, parallel Adding parallel-heiko Adding some networks Adding action for deploy to IPFS Adding permission for shell Fix GH action fix array fix picasso project.file fix typo change to matrix deployment Change GH action logic Adding yarn cache fix github actions Adding brach creation in setup step Fix push command fix push action fix GH action fix push step Update CIDs for IFPS Fix chains version (#168) Migrate to IPFS (#170) * Fix chains version * first iteration * Acala in separate directory * Adding possibility to run locally * Adding acala from root folder * Temporary fix for types problem * Adding aleph-zero project * Adding altair * fix handlers for aleph-zero * fix acala handlers * Change enpoints for acala, altair * adding astar * adding bajun * Adding basilisk * Adding bifrost * Adding bifrost-polkadot * Adding bit-country * rename project files * adding calamari * Adding centrifuge * Adding clover * Adding composable * Adding dora * Adding efinity * Adding encointer * Adding gm * adding hydra * Adding imbue * Adding integritee * Adding Interlay * Adding kabocha * Adding karura * Adding khala * Adding kico * Adding kilt * Adding kintsugi * Adding kusama * fix kusama * Adding litentry * Adding litmus * Adding moonbeam * Adding moonriver * Adding nodle, origin-trail, parallel * Adding parallel-heiko * Adding some networks * Adding action for deploy to IPFS * Adding permission for shell * Fix GH action * fix array * fix picasso project.file * fix typo * change to matrix deployment * Change GH action logic * Adding yarn cache * fix github actions * Adding brach creation in setup step * Fix push command * fix push action * fix GH action * fix push step * Adding ternoa * Delete transfer_keep_alive * Delete test runner * Adding typess for Ternoa * Upgrade to polkadot/api v8 * Remove useless lines remove ignore
novasamatech · Sep 19, 2022 · 54503cf · 54503cf
1 parent d83b300
commit 54503cf
Show file tree

Hide file tree

Showing 297 changed files with 53,271 additions and 2,343 deletions.
diff --git a/.github/get_all_project_files.sh b/.github/get_all_project_files.sh
@@ -0,0 +1,30 @@
+#!/bin/bash
+
+MAIN_DIRECTORY='.'
+
+JSON="{\"include\":["
+
+for item in "$MAIN_DIRECTORY"/*.yaml
+do
+    # Skip base project.yaml which are using for initialization
+    if [[ "$item" == "./project.yaml" ]]; then
+        continue
+    fi
+
+    arrIN=(${item//\// })
+
+    JSONline="{\"project_file\": \"${arrIN[1]}\"},"
+
+    if [[ "$JSON" != *"$JSONline"* ]]; then
+        JSON="$JSON$JSONline"
+    fi
+
+done <<< "$DIFF"
+
+# Remove last "," and add closing brackets
+if [[ $JSON == *, ]]; then
+    JSON="${JSON%?}"
+fi
+JSON="$JSON]}"
+
+echo $JSON
diff --git a/.github/workflows/deploy_to_ipfs.yaml b/.github/workflows/deploy_to_ipfs.yaml
@@ -0,0 +1,89 @@
+name: Manual deploy to IPFS
+
+on:
+  workflow_dispatch:
+
+env:
+  SUBQUERY_ORG: nova-wallet
+  SUBQL_ACCESS_TOKEN: ${{ secrets.SUBQUERY_TOKEN }}
+
+jobs:
+
+  setup:
+    runs-on: ubuntu-latest
+    outputs:
+      project_files: ${{ steps.project_files.outputs.files }}
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: 🧾 Get list of projects
+        id: project_files
+        run: |
+          chmod +x .github/get_all_project_files.sh
+          echo "::set-output name=files::$(.github/get_all_project_files.sh)"
+
+      - name: 🔄 Install dependencies
+        run: yarn install
+
+      - name: 🧬 Types generation
+        run: yarn codegen
+
+      - name: 🏗 Build from sources
+        run: yarn build
+
+
+  deploy-to-IPFS:
+    needs: setup
+    name: 📬 Deploy to subquery IPFS
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      max-parallel: 5
+      matrix: ${{fromJson(needs.setup.outputs.project_files)}}
+
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: 🔄 Install dependencies
+        if: ${{ steps.yarn-cache.outputs.cache-hit != 'true' }}
+        run: yarn install
+
+      - name: 🧬 Types generation
+        run: yarn codegen
+
+      - name: 📬 Deploy to IPFS
+        run:  |
+          ./node_modules/.bin/subql publish -f ${{ matrix.project_file }}  -o
+
+      - uses: actions/upload-artifact@v2
+        with:
+          name: cids
+          path: .*cid
+
+
+  push-changed-cids:
+      needs: deploy-to-IPFS
+      name: 🏹 Push changes to branch
+      runs-on: ubuntu-latest
+
+      steps:
+        - uses: actions/checkout@v2
+
+        - name: Download built artifact
+          uses: actions/download-artifact@v2
+          with:
+            name: cids
+            path: ./ipfs-cids
+
+        - name: ➡️ Commit files
+          run: |
+            git config --local user.email "41898282+github-actions[bot]@users.noreply.github.com"
+            git config --local user.name "github-actions[bot]"
+            git add -f ipfs-cids
+            git commit -m "Update CIDs for IFPS"
+
+        - name: 🏹 Push changes
+          uses: ad-m/github-push-action@master
+          with:
+            github_token: ${{ secrets.GITHUB_TOKEN }}
+            branch: refs/heads/update-cids-${{github.run_number}}
diff --git a/.github/workflows/get_projects_status.yaml b/.github/workflows/get_projects_status.yaml
@@ -0,0 +1,30 @@
+name: Projects healthcheck
+
+on:
+  schedule:
+    - cron: '0 0/4 * * *'
+  workflow_dispatch:
+
+env:
+ SUBQUERY_TOKEN: ${{ secrets.SUBQUERY_TOKEN }}
+
+jobs:
+  generate_project_list:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Install python dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r ./scripts/python_scripts/requirements.txt
+
+      - name: Generate table
+        run: python ./scripts/python_scripts/generate_network_status.py
+
+      - name: Deploy report to Github Pages
+        uses: peaceiris/actions-gh-pages@v2
+        env:
+          PERSONAL_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          PUBLISH_BRANCH: gh-pages
+          PUBLISH_DIR: gh-pages
diff --git a/.github/workflows/manual_deploy_to_stg.yaml b/.github/workflows/manual_deploy_to_stg.yaml
@@ -0,0 +1,114 @@
+name: Manual deploy to staging
+
+on:
+  workflow_dispatch:
+    inputs:
+      networks:
+        description: 'Which networks will deploy?'
+        required: true
+        type: string
+      batch_size:
+        description: 'Set batch size [Documentation](https://doc.subquery.network/academy/tutorials_examples/batch-size.html#why-change-the-batch-size)'
+        required: true
+        default: "15"
+      indexer_image_version:
+        description: 'Version of docker image for indexer'
+        required: true
+        default: v0.33.0
+      query_image_version:
+        description: 'Version of docker image for query'
+        required: true
+        default: v0.14.1
+      subquery_cli_version:
+        description: '[CLI](https://github.com/fewensa/subquery-cli/releases) version'
+        required: true
+        default: 0.2.6
+      reindex:
+        description: 'Re-index these networks? ⚠️Attention this will clear all data on stage env⚠️'
+        required: true
+        type: boolean
+        default: false
+
+env:
+  SUBQUERY_ORG: nova-wallet
+
+jobs:
+
+  setup:
+    runs-on: ubuntu-latest
+    outputs:
+      matrix: ${{ steps.set-matrix.outputs.matrix }}
+    steps:
+    - id: set-matrix
+      run: |
+        list_elements=(${{ github.event.inputs.networks }})
+        JSON="{\"include\":["
+
+        for item in ${list_elements[*]}
+        do
+            JSONline="{\"chain\": \"$item\"},"
+
+            if [[ "$JSON" != *"$JSONline"* ]]; then
+                JSON="$JSON$JSONline"
+            fi
+            echo $item
+        done <<< "$DIFF"
+
+        # Remove last "," and add closing brackets
+        if [[ $JSON == *, ]]; then
+          JSON="${JSON%?}"
+        fi
+        JSON="$JSON]}"
+        echo $JSON
+
+        echo "::set-output name=matrix::$( echo "$JSON" )"
+
+
+  deploy-subquery:
+    needs: setup
+    name: Deploy subquery
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      max-parallel: 5
+      matrix: ${{fromJson(needs.setup.outputs.matrix)}}
+
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Install dependencies
+        run: |
+          mkdir -p $HOME/.local/bin
+          curl -LO https://github.com/fewensa/subquery-cli/releases/download/v${{ github.event.inputs.subquery_cli_version }}/subquery-linux-x86_64.zip
+          unzip subquery-linux-x86_64.zip -d $HOME/.local/bin/
+          rm -rf subquery-linux-x86_64.zip
+      - name: Get branch name
+        id: branch-name
+        uses: tj-actions/branch-names@v5.1
+
+      - name: Deploy
+        if: ${{ github.event.inputs.reindex == 'false' }}
+        run: |
+          subquery --token ${{ secrets.SUBQUERY_TOKEN }} deployment deploy \
+            --org ${{ env.SUBQUERY_ORG }} \
+            --key nova-wallet-${{ matrix.chain }} \
+            --branch deploy \
+            --sub-folder ./networks/${{ matrix.chain }} \
+            --type stage \
+            --indexer-image-version ${{ github.event.inputs.indexer_image_version }} \
+            --query-image-version ${{ github.event.inputs.query_image_version }} \
+            --batch-size ${{ github.event.inputs.batch_size }}
+
+      - name: Redeploy
+        if: ${{ github.event.inputs.reindex == 'true' }}
+        run: |
+          subquery --token ${{ secrets.SUBQUERY_TOKEN }} deployment deploy \
+            --org ${{ env.SUBQUERY_ORG }} \
+            --key nova-wallet-${{ matrix.chain }} \
+            --branch deploy \
+            --sub-folder ./networks/${{ matrix.chain }} \
+            --type stage \
+            --indexer-image-version ${{ github.event.inputs.indexer_image_version }} \
+            --query-image-version ${{ github.event.inputs.query_image_version }} \
+            --batch-size ${{ github.event.inputs.batch_size }} \
+            --force
diff --git a/.github/workflows/pr_by_push_to_master.yaml b/.github/workflows/pr_by_push_to_master.yaml
@@ -0,0 +1,65 @@
+name: Make pull request to deploy branch after push to master
+
+on:
+  push:
+    branches:
+      - 'master'
+
+jobs:
+  create_new_branch:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout to current branch
+        uses: actions/checkout@master
+        with:
+          ref: ${{ github.ref }}
+
+      - name: Prepare brach to commit
+        run: bash ./scripts/prepare_folders.sh
+
+      - name: Commit files
+        run: |
+          git config --local user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          git config --local user.name "github-actions[bot]"
+          git add -f networks/**/
+          git commit -m "Prepare to pull request to deploy branch"
+
+      - name: Push changes
+        uses: ad-m/github-push-action@master
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          branch: refs/heads/update-deploy-${{github.run_number}}
+
+
+  create_new_pullrequest:
+    needs: create_new_branch
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout to deploy branch
+        uses: actions/checkout@master
+        with:
+          ref: refs/heads/deploy
+
+      - name: Debug
+        run: ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/   /' -e 's/-/|/'
+
+      - name: Merge master to deploy
+        run: |
+          git config --local user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          git config --local user.name "github-actions[bot]"
+          git fetch --all
+          git merge origin/master
+          git push
+        continue-on-error: true
+
+      - uses: repo-sync/pull-request@v2
+        name: Creating pull-request
+        with:
+          source_branch: "update-deploy-${{github.run_number}}"
+          destination_branch: "deploy"
+          pr_title: "Update deploy branch"
+          pr_body: |
+              Update deploy branch
+          pr_reviewer: "stepanLav"
+          pr_draft: false
+          github_token: ${{ secrets.GITHUB_TOKEN }}