checklist for doing a release:

1. Check weewx.conf for local changes. In particular, check to make sure:

1. debug=0 in weewx.conf

2. Make sure the version is correct

1. modify pyproject.toml

2. make version

3. Make sure all changes have been logged

1. doc_src/changes.md

2. doc_src/upgrade.md

3. make deb-changelog

4. make redhat-changelog

5. make suse-changelog

4. Build the documentation

1. make build-docs

5. Create the packages

1. make pypi-package

2. make debian-package

3. make redhat-package

4. make suse-package

6. Check of each installation method

1. do a git install + upgrade

2. do a pip install + upgrade

3. do a debian install + upgrade

4. do a redhat install + upgrade

7. Commit and tag

1. git commit -m "Version X.Y.Z"

2. git push

3. git tag vX.Y.Z

4. git push --tags

8. Upload wheel to pypi.org

1. make upload-pypi

9. Upload to weewx.com

1. make upload-docs

2. make upload-src

3. make upload-debian

4. make upload-redhat

5. make upload-suse

10. Update the deb repository

1. make pull-apt-repo

2. make update-apt-repo

3. make push-apt-repo

11. Update the redhat repository

1. make pull-yum-repo

2. make update-yum-repo

3. make push-yum-repo

12. Update the suse repository

1. make pull-suse-repo

2. make update-suse-repo

3. make push-suse-repo

13. Reshuffle files on the server

1. make release

2. make release-apt-repo

3. make release-yum-repo

4. make release-suse-repo

14. Announce the release to the weewx user's group.

pre-requisites ----------------------------------------------------------------

Building the documentation requires mkdocs, which is installed using pip.

Signing packages requires gpg and local copy of authorized private+public keys.

Verifying the packages requires rpmlint/lintian.

The debian repo management requires aptly.

The redhat repo management requires createrepo.

The suse repo management requires createrepo.

To build the docs:

pip3 install --user mkdocs

pip3 install --user mkdocs-material

To build pypi package install the following (see the pypi section):

pip3 install --user poetry

To build debian package install the following (use 'apt-get install'):

git

rsync

gpg

debhelper

lintian

To build redhat package install the following (use 'yum install'):

git

rsync

gnupg

rpm-build

rpm-sign

rpmlint

To build suse package install the following (use 'zypper install'):

git

rsync

rpm-build

rpmlint

createrepo_c

howto -------------------------------------------------------------------------

how to update the version number:

Change the version number in pyproject.toml

make version # this propagates the version number everywhere

git commit -a -m "bump to version x.y.z"

how to build wheel and source tarball:

make pypi-package

how to build debian package:

make deb-changelog

emacs pkg/debian/changelog # add package-specific changes, if any

git commit -m "update deb changelog" pkg/debian/changelog

make debian-package

how to build redhat package:

make redhat-changelog

emacs pkg/changelog.el # add package-specific changes, if any

git commit -m "update redhat changelog" pkg/changelog.el

make redhat-package

how to build redhat package:

make suse-changelog

emacs pkg/changelog.suse # add any package-specific changes, if any

git commit -m "update suse changelog" pkg/changelog.suse

make suse-package

to build redhat packages with custom rpm revision:

make redhat-changelog RPMREVISION=2

make redhat-package RPMREVISION=2

make pull-yum-repo

make update-yum-repo RPMREVISION=2

make push-yum-repo

ssh weewx.com rsync -arv /var/www/html/yum-test/ /var/www/html/yum

suse also uses RPMREVISION, but debian uses DEBREVISION. this is useful when

there is a change to asset(s) in the packaging, but not part of weewx itself.

to display debconf variables:

sudo debconf-show weewx

to manually purge debconf variables:

echo PURGE | sudo debconf-communicate weewx

to sign rpm packages you need .rpmmacros in your home directory:

~/.rpmmacros

%_gpg_name YOUR_NAME_HERE

to sign the apt Release using key 'XXX':

gpg -abs -u XXX -o Release.gpg Release

to sign the RPM repository metadata using key 'XXX':

gpg -abs -u XXX -o repomd.xml.asc repomd.xml

to generate gpg key used for signing packages:

gpg --gen-key

gpg --list-keys

gpg --list-secret-keys

to export the text version of a public key:

gpg --export --armor > username.gpg.key

list keys known by rpm:

rpm -q --gpg-pubkey

rpm -q --gpg-pubkey --qf '%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n'

delete keys known by rpm:

rpm -e gpg-pubkey-XXXXX

Install using pip

make pypi-package

pip install dist/weewx-x.y.z-py3-none-any.whl

weectl station create

debian install/remove:

apt-get install weewx # install with apt

apt-get remove weewx # remove with apt

apt-get purge weewx # purge removes /etc/weewx and debconf settings

dpkg -i weewx_x.y.z-r.deb # install

(apt-get install weewx) # finish install if dependencies failed

dpkg -r weewx # remove

dpkg -P weewx # purge

redhat install/remove:

yum install weewx-x.y.z-r.rpm [--nogpgcheck] # install with yum

yum remove weewx # remove with yum

rpm -i weewx-x.y.z-r.rpm # install with rpm directly

rpm -e weewx # remove with rpm

suse install/remove:

zypper install weewx-x.y.z-r.rpm [--nogpgcheck] # install with zypper

zypper remove weewx # remove with zypper

rpm -i weewx-x.y.z-r.rpm # install with rpm directly

rpm -e weewx # remove with rpm

howto: build the documents --------------------------------------------------

Prerequisites:

- Python 3.7 or greater, with pip installed

- Install mkdocs:

python3 -m pip install mkdocs

- Install the theme "material" for mkdocs:

python3 -m pip install mkdocs-material

Steps:

- make build-docs

howto: build and publish wheels to pypi.org -----------------------------------

Prerequisites:

- Python 3.7 or greater, with pip installed

- Install poetry:

curl -sSL https://install.python-poetry.org | python3 -

- Get an API token from pypi.org

See https://pypi.org/manage/account/token/

- Tell poetry to use it:

poetry config pypi-token.pypi pypi-substitute-your-pypi-key

Steps:

- Build the wheel

make pypi-package

- Publish to pypi.org

make upload-pypi

howto: deb repository ---------------------------------------------------------

aptly has two different mechanisms for doing a 'publish': switch or update.

we use snapshots, and publish using 'publish switch', rather than publishing

using a simple 'publish update'.

There are two apt repositories: python2 and python3

to do apt repo updates you must first install aptly:

https://www.aptly.info/download/

for example, on debian:

echo "deb http://repo.aptly.info/ squeeze main" | sudo tee /etc/apt/sources.list.d/aptly.list

wget -qO - https://www.aptly.info/pubkey.txt | sudo apt-key add -

sudo apt-get update

sudo apt-get install aptly

create local debian repo using aptly:

aptly repo create -distribution=squeeze -component=main -architectures=all python2-weewx

aptly repo create -distribution=buster -component=main -architectures=all python3-weewx

put a bunch of deb files into an empty apt repo:

for f in `ls distdir`; do aptly repo add python2-weewx distdir/$f; done

create a snapshot:

aptly snapshot create python-weewx-x.y.z-n from repo python2-weewx

aptly snapshot create python3-weewx-x.y.z-n from repo python3-weewx

publish using snapshot:

aptly publish -architectures=all snapshot python-weewx-x.y.z-n python2

aptly publish -architectures=all snapshot python3-weewx-x.y.z-n python3

update using 'publish switch':

aptly repo add python2-weewx dist/python-weewx_x.y.z-n_all.deb

aptly snapshot create python-weewx-x.y.z-n from repo python2-weewx

aptly publish switch squeeze python2 python-weewx-x.y.z-n

aptly repo add python3-weewx dist/python3-weewx_x.y.z-n_all.deb

aptly snapshot create python3-weewx-x.y.z-n from repo python3-weewx

aptly publish switch buster python3 python3-weewx-x.y.z-n

update using 'publish update':

aptly publish repo -architectures=all python2-weewx squeeze

aptly repo add python2-weewx dist/squeeze/python-weewx_x.y.z-n_all.deb

aptly publish update squeeze python2

aptly publish repo -architectures=all python3-weewx buster

aptly repo add python3-weewx dist/buster/python3-weewx_x.y.z-n_all.deb

aptly publish update buster python3

clone the published apt repo to local space:

mkdir -p ~/.aptly

rsync -arv USER@weewx.com:/var/www/html/aptly-test/ ~/.aptly

synchronize local aptly changes with the published apt repo:

rsync -arv ~/.aptly/ USER@weewx.com:/var/www/html/aptly-test

switch from testing to production (this is done at weewx.com):

rsync -arv /var/www/html/aptly-test/ /var/www/html/aptly

for clients to use an apt repo at weewx.com:

curl -s http://weewx.com/keys.html | sudo apt-key add -

echo "deb [arch=all] http://weewx.com/apt/ squeeze main" | sudo tee /etc/apt/sources.list.d/weewx.list

echo "deb [arch=all] http://weewx.com/apt/ buster main" | sudo tee /etc/apt/sources.list.d/python3-weewx.list

howto: yum repository ---------------------------------------------------------

create yum repo:

mkdir -p ~/.yum/weewx/{el7,el8,el9}/RPMS

update local yum repo with latest rpm:

cp *.el7.rpm ~/.yum/weewx/el7/RPMS

createrepo -o ~/.yum/weewx/el7 ~/.yum/weewx/el7

cp *.el8.rpm ~/yum/weewx/el8/RPMS

createrepo -o ~/.yum/weewx/el8 ~/.yum/weewx/el8

cp *.el9.rpm ~/yum/weewx/el9/RPMS

createrepo -o ~/.yum/weewx/el9 ~/.yum/weewx/el9

clone the published yum repo to local space:

mkdir -p ~/.yum

rsync -arv USER@weewx.com:/var/www/html/yum-test/ ~/.yum

synchronize local yum changes with published yum repo:

rsync -arv ~/.yum/ USER@weewx.com:/var/www/html/yum-test

switch from testing to production (this is done at weewx.com):

rsync -arv /var/www/html/yum-test/ /var/www/html/yum

notes -------------------------------------------------------------------------

there are multiple changelogs:

docs/changes.md - definitive changelog for the application

pkg/debian/changelog - changes to the debian packaging

pkg/changelog.el - changes to the redhat packaging

pkg/changelog.suse - changes to the suse packaging

when signing RPMs, gpg info must match the name and email in the latest package

changelog entry.

when signing apt Release using aptly, beware that aptly uses the first gpg key

that it finds. that might not be what you want.

the debian changelog *must* have a version number that matches the app version.

the redhat package will build if the version numbers do not match. use the

rpm-changelog and deb-changelog targets to ensure that changelog versions match

the application version for a release.

there are many ways to build a debian package. first tried dpkg (uses DEBIAN

dir and is fairly low-level) but that does not create changes and source diffs.

then dried dpkg-buildpackage (uses debian dir and is higher level) but misses

the config and templates. ended up using dpkg-buildpackage with some manual

(scripted) file manipulation.

both debian and redhat deprecate the use of '/usr/bin/env python' as the

shebang - they want a very tight binding to the operating system python as the

shebang. in fact, since late 2019 redhat build tools see any shebang other

than a tight coupling to the operating system's python as an error and refuse

to accept it. however, the bsd platforms prefer this as the shebang, and the

other approach fails on bsd. macos uses something else.

since weewx5 supports any python 3.6+, older weewx supports python 2.7 and any

python 3.5+, weewx does not have to be tightly coupled to any specific python

installation on the system.

so the source code should use the '/usr/bin/env python' shebang. on platforms

that refuse to accept this, the package builder will replace this with

whatever that platform will accept. for pip installs, pip does the shebang

mangling and sets entry points that are appropriate for its configuration.

for everything else, using the env in shebang and making the entry points

executable enables either 'python foo.py' or 'foo.py' invocation.

the /etc/default/weewx plus shell stubs in /usr/bin/wee* is used in deb/rpm

and non-linux installations to provide python flexibility, so that users can

use a single weewx installation to experiment with different python versions.

this is particularly helpful when running weewx directly from source in a git

clone, it also works in the deb/rpm installs where the python is managed

separately from the system's python, as well as the non-linux, non-pip

installations.

unfortunately, this can cause problems during an installation. if the pre/post

install scripts invoke /usr/bin/weectl instead of the python code directly,

they can end up getting python2 or a python3 that does not have the right

modules installed. so maintainer scripts and scriptlets must ensure that they

use a known-working python.

prefer to use expressive but no overly verbose output in scriptlets and

maintainer scripts. many redhat scriptlets seem to be silent, perhaps only

emitting messages when there is a failure. unfortunately, most of our failures

have been due to things that we never even predicted, let alone created error

traps for. so having output from each step helps us figure out where a

failure came from. apparently redhat will add better logging/capture from

scriptlets in redhat 10 - the anaconda installer already does a pretty good

job of capturing and isolating output from scriptlets, but redhat plans to add

similar functionality to dnf in redhat 10.

fedora docs say that a service may be enabled if it does not alter other

services, does not fail, does not require configuration before starting, and

does not listen for outside connections. so we can enable the weewx daemon,

but that does *not* mean we can run it.

for a new install, we only enable and start the primary unit. for an upgrade,

on a system with systemd we remove any sysv bits and migrate them to systemd.

signing packages --------------------------------------------------------------

gpg is used to sign the deb repository and rpm packages. SHA1 is no longer

acceptable for signing, so be sure that your gpg keys and the signing command

use SHA256 instead. this should be the default when building on redhat9 and

later. if it is not the default, then it can be forced with a change to the

signing macro in .rpmmacros - add the line --digest-algo sha256

%__gpg_sign_cmd %{__gpg} \

gpg --no-verbose --no-armor \

%{?_gpg_digest_algo:--digest-algo %{_gpg_digest_algo}} \

--no-secmem-warning \

--digest-algo sha256 \

%{?_gpg_sign_cmd_extra_args:%{_gpg_sign_cmd_extra_args}} \

-u "%{_gpg_name}" -sbo %{__signature_filename} %{__plaintext_filename}

In the debian world, you sign the repository (specifically the file 'Release'),

not the individual .deb files. So if you need to re-sign, you re-build and

re-sign the repository; there is no need touch the individual .deb files.

Signing is controlled by the -us and -uc options to dpkg-build. If you do not

specify those options, then dpkg-build will try to sign the .dsc, .buildinfo,

and .changes files. The .deb itself is not signed.

SUSE wants you to sign the RPMs as well as the repository metadata. The meta

data are in repomd.xml, and a fully signed repository must include the files

repomd.xml.asc and repomd.xml.key. So although it is possible to use one key

for the meta data and different keys for the RPMs, it is probably best to sign

with a single, shared key.

On SUSE, zypper keeps the repo information in a local cache /var/cache/zypp/raw

unit tests --------------------------------------------------------------------

prerequisites:

python 3.7

python-usb

pyephem

to set up mysql server with user and permissions for testing:

make test-setup

to run all unit tests:

make test

(note: do not run this as root)

to clean up after running tests:

make test-clean

guidelines:

unit tests should put transient files in /var/tmp/weewx_test

testing -----------------------------------------------------------------------

what to test when creating debian and redhat packages:

install, upgrade, remove, purge

install, modify files, remove

install previous release, modify files, upgrade, remove

Using pip:

- new install to user space

make pypi-package

pip install dist/weewx-x.y.z-py3-none-any.whl --user

weectl station create

- upgrade user data

modify ~/weewx-data/weewx.conf

weectl station upgrade

- new install using pip to /opt/weewx

make pypi-package

sudo pip install dist/weewx-x.y.z-py3-none-any.whl

sudo weectl station create /opt/weewx/weewx.conf

- upgrade using setup.py to /opt/weewx

setup.py install home=/opt/weewx

modify /opt/weewx/weewx.conf

setup.py install home=/opt/weewx

on centos and suse:

- new install using rpm

rpm -i weewx_x.y.z.rpm

- upgrade using rpm

rpm -i weewx_r.s.t.rpm

rpm -U weewx_x.y.z.rpm

- upgrade using rpm with extensions installed

rpm -i weewx_r.s.t.rpm

wee_extension --install cmon

rpm -U weewx_x.y.z.rpm

debian:

- new install usinb dpkg

dpkg -i weewx_x.y.z.deb

- upgrade using dpkg take maintainer's version of weewx.conf

dpkg -i weewx_r.s.t.deb

modify /etc/weewx/weewx.conf

dpkg -i weewx_x.y.z.deb

- upgrade using dpkg use old version of weewx.conf

dpkg -i weewx_r.s.t.deb

modify /etc/weewx/weewx.conf

dpkg -i weewx_x.y.z.deb

- reconfigure using dpkg

dpkg-reconfigure weewx

all platforms:

- installation and removal of extensions

weectl extension install https://github.com/matthewwall/weewx-cmon/archive/master.zip

weectl extension install ~/weewx-data/examples/pmon

weectl extension uninstall cmon

weectl extension uninstall pmon

- reconfigure

weectl station reconfigure

weectl station reconfigure --driver=weewx.drivers.vantage --no-prompt