Issues for Automatic Updates

Update project page to reflect current state of the module

tedbow — Mon, 27 Jan 2025 14:22:20 +0000

Problem/Motivation

The project page is out of date again.

At least the portion about experimental features. Automatic Updates Extensions is not longer experimental. Probably other parts are out of date too.

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Don't try to automatically register Composer Stager services

phenaproxima — Mon, 13 Jan 2025 18:37:10 +0000

Problem/Motivation

We need to fix #3498636: AJAX HTTP error occurred during installation due to undefined service PreconditionInterface if we want to ship Automatic Updates 3.x in Drupal CMS.

It might not be wise for us to ship Automatic Updates 4.x, since Package Manager is alpha stability in core and therefore can (and will) make API breaks.

#3498636: AJAX HTTP error occurred during installation due to undefined service PreconditionInterface is caused by the contrib version of Package Manager trying to automatically scan Composer Stager for services. Core removed that in #3479205: Wire Composer Stager into Package Manager's services.yml. Let's do what it did and list the services in our services.yml file, relying on autowiring to take care of inter-class dependencies.

This blocks a stable release of Drupal CMS, so I will have to tag 3.1.7 when I commit this, even though CI is failing for reasons that are not related to this.

Create 4.x branch that uses Package Manager from core

phenaproxima — Thu, 09 Jan 2025 18:45:14 +0000

Problem/Motivation

Package Manager is in core 11.1, and this presents a problem for anyone using Automatic Updates 3.1.x -- which includes Drupal CMS itself. The presence of two divergent versions of Package Manager can lead to weird errors if Drupal loads the wrong one. We're already seeing evidence of this in issues like #3499257: TypeError: Drupal\package_manager\Validator\DiskSpaceValidator::temporaryDirectory(): Return value must be of type string, false returned in Drupal\package_manager\Validator\DiskSpaceValidator->temporaryDirectory() and #3498636: AJAX HTTP error occurred during installation due to undefined service PreconditionInterface. This kind of uncertainty is dangerous and it means Drupal CMS cannot ship with Automatic Updates 3.x.

However, we must include Automatic Updates -- it's a table-stakes feature for Drupal CMS, and it has long been promised as one of our out-of-the-box features. To make sure it works, we need to be able to ship a version that doesn't have Package Manager, and relies fully on the core version.

Proposed resolution

Branch Automatic Updates 4.x and remove Package Manager from it. Everything else will be identical, but it will make the appropriate adjustments (almost entirely in the tests) to work with the core version of Package Manager. Support for any version of Drupal before 11.1 will be dropped.

Due to the urgency of this for Drupal CMS's stable release on January 15th, this will need to:

Be merged even if not all tests are currently passing. The work I've already done proves that test failures are the fault of the tests making bad, outdated, or fragile assumptions; it's not because of problems with the module itself or any particular incompatibility with core's version of Package Manager. I'll fix everything I can in the time I have, but any outstanding failures I'm not able to squash before the deadline will need to be fixed in follow-ups.
Immediately become a tagged release of Automatic Updates 4.x. It would be acceptable for it to be an alpha tag (4.0.0-alpha1).

Drupal CMS's freeze deadline is the end of the day on Monday, January 13th. I will merge and tag this MR by this deadline, and I will do it by unilateral executive decision if I need to. I wish we'd had some time to properly prepare for this before now, but @tedbow is allocated full-time to the Experience Builder project, and I was allocated to Drupal CMS and related projects, and there was always way more than a day's worth of work for me to do on all that other stuff.

Update Extensions page error if Composer not found

juxelle — Mon, 20 Jan 2025 22:19:47 +0000

Problem/Motivation

Installed Drupal on a shared hosting server where it couldn't find the Composer executable.
On the Extend page most tabs show an error message that Composer can't be found.

The Extend->Update Extensions tab has an uncaught exception instead.
The exception is "PhpTuf\ComposerStager\API\Exception\LogicException: The composer executable cannot be found. Make sure it's installed and in the $PATH in PhpTuf\ComposerStager\Internal\Finder\Service\ExecutableFinder->find() (line 34 of [path-to]/vendor/php-tuf/composer-stager/src/Internal/Finder/Service/ExecutableFinder.php)."

Composer is installed and was used to get the Drupal files. I don't know why it couldn't find the Composer executable, my guess is it's a permissions issue?

Steps to reproduce

Install Drupal
Make sure Composer isn't in the $PATH
Check Extend->Update for message that the Composer executable can't be found
Go to Extend->Update Extensions tab - this will give an exception

Proposed resolution

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

admin/modules/automatic-update-extensions do not sohw up

joachim namyslo — Wed, 15 Jan 2025 19:30:43 +0000

Hello everyone,

I couldn't keep my fingers off Drupal CMS, of course, and installed the thing directly after the release was made.

on /admin/modules/automatic-update-extensions there is a classic wsod with the default sentence

The website encountered an unexpected error. Try again later.

in /var/log/apache2/error.log i got the same message you can see here:

[Wed Jan 15 20:26:24.418176 2025] [proxy_fcgi:error] [pid 2471] [client ::1:50228] AH01071: Got error 'PHP message: Uncaught PHP Exception Drupal\\package_manager\\Exception\\ComposerNotReadyException: "Failed to run process: <em class="placeholder">The command "'/usr/bin/composer' 'validate' '--check-lock' '--no-check-publish' '--with-dependencies' '--no-ansi' '--working-dir=/var/www/cms'" failed.\n\nExit Code: 2(Misuse of shell builtins)\n\nWorking directory: /var/www/cms/web\n\nOutput:\n================\nasm89/stack-cors is valid\ncarbonphp/carbon-doctrine-types is valid\nchi-teck/drupal-code-generator is valid\nclue/stream-filter is valid\ncommerceguys/addressing is valid\ncomposer/installers is valid\ncomposer/semver is valid\nconsolidation/annotated-command is valid\nconsolidation/config is valid\nconsolidation/filter-via-dot-access-data is valid\nconsolidation/log is valid\nconsolidation/output-formatters is valid\nconsolidation/robo is valid\nconsolidation/site-alias is valid\nconsolidation/site-process is valid\ndavedevelopment/stiphle ...', referer: http://localhost/admin/modules

I used

OS: Ubuntu noble 24.04 x86_64
Host: Windows Subsystem for Linux - Ubuntu (2.3.26)
;looool;. 'oooooooooo, Kernel: Linux 5.15.167.4-microsoft-standard-WSL2
Server version: Apache/2.4.58 (Ubuntu)
Server built: 2024-10-02T12:40:51
PHP 8.3.6 (cli) (built: Dec 2 2024 12:36:18) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.3.6, Copyright (c) Zend Technologies
with Zend OPcache v8.3.6, Copyright (c), by Zend Technologies
loaded via php-fpm

mariadb Ver 15.1 Distrib 10.11.8-MariaDB, for debian-linux-gnu (x86_64) using EditLine wrapper

#[PHP Modules]
calendar
Core
ctype
curl
date
dom
exif
FFI
fileinfo
filter
ftp
gd
gettext
hash
iconv
json
libxml
mbstring
memcache
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
Phar
posix
random
readline
Reflection
session
shmop
SimpleXML
sockets
sodium
SPL
standard
sysvmsg
sysvsem
sysvshm
tokenizer
uploadprogress
xml
xmlreader
xmlwriter
xsl
yaml
Zend OPcache
zlib

[Zend Modules]
Zend OPcache

 whereis composer
composer: /usr/bin/composer

Hopefully I did not miss any critical information.

How to reproduce the issue

Install the latest Drupal CMS Release as usual
login as admin
go to: admin/modules/automatic-update-extensions

that should be enough to reproduce it.

"Update" and "Update extensions" displayed under Appearance

matthieuscarset — Thu, 16 Jan 2025 20:04:58 +0000

Problem/Motivation

Some links are displayed on the Appareance page while they should not be there.

The "Update" and "Update extensions" local tasks are displayed (see attached screenshot).

Steps to reproduce

Install Drupal CMS (i used DDEV).
Nothing specific, just go through the installer.
Go to Appareance
See the links.

Proposed resolution

TBD

Add support for patches.json

darren oh — Tue, 07 Jan 2025 21:01:42 +0000

Problem/Motivation

The Composer Patches plugin reads from patches.json as well as from composer.json. Automatic Updates only copies composer.json.

Steps to reproduce

Move patches to patches.json.
Run composer install.
Run composer update to set Drupal to an outdated version.
Use Automatic Updates to update.
Patches are not applied.

Proposed resolution

Copy patches.json as well as composer.json.

Remaining tasks

Create branch to fix issue.
Review code.
Test.
Merge branch.

User interface changes

None.

API changes

Patches in patches.json are applied in automatic updates.

Data model changes

None.

Automatic Updates Initiative meeting on Jan 7, 2024

hestenet — Tue, 07 Jan 2025 18:13:31 +0000

This meeting:
➤ Is for core developers, initiative contributors, the Drupal Association and anyone interested in the initiative.
➤ Usually happens every other Tuesday at 1700 UTC.
➤ Is done over chat.
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to
➤ *Transcript will be exported and posted* to the agenda issue. For anonymous comments, start with a :bust_in_silhouette: emoji. To take a comment or thread off the record, start with a :no_entry_sign: emoji.

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

:one: Do you have any topics to propose for the meeting today? Feel free to propose them in this thread, and then I will give them their own unique threads for discussion. Conversation moving slow? Go ahead and open your own thread in the next numeric order.

Automatic Updates Initiative meeting on Dec 10, 2024

hestenet — Tue, 10 Dec 2024 17:28:42 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

Consider displaying the core and extensions update forms on one page

pameeela — Thu, 31 Oct 2024 05:32:53 +0000

Problem/Motivation

Discussed with @phenaproxima that the current UX is a bit confusing, with two separate update pages, one for core and one for contrib. I created a couple of issues to try to clarify this, but the ideal solution would be to have the updates display on a single page, if possible.

Just creating an issue to capture this for discussion for now, not expecting anything to happen on it.

Change message to see status report to match core

quietone — Mon, 16 Sep 2024 05:45:52 +0000

Problem/Motivation

I was playing around with this and got an error message,

Error message

    Your site does not pass some readiness checks for automatic updates. It cannot be automatically updated until further action is performed.
    See status report for more details.
    Your version of Drupal is no longer supported. Upgrading is strongly recommended! See the available updates page for more information and to update your software.

I think the string "See status report for more details." should use the same text as the existing in \Drupal\system\Controller\SystemController::overview

Steps to reproduce

Fresh install of Drupal 11.x
Install automatic updates

Proposed resolution

In \Drupal\automatic_updates\Validation\AdminStatusCheckMessages::displayResultSummary change this

      $this->t('<a href=":url">See status report for more details.</a>', [

      $this->t('Check the <a href=":url">status report</a> for more information.', [

Remaining tasks

User interface changes

API changes

Data model changes

Automatic updates for distributions

user654 — Sat, 21 Nov 2020 09:06:08 +0000

It would be great if automatic updates worked also for Drupal Distributions.
Thanks

Prompt for temporary access when file system is not writable

darren oh — Thu, 16 Jul 2020 21:40:34 +0000

Motivation

Automatic Updates currently asks admins to configure core files to be writable by the PHP user. While the update process itself is trustworthy, this enables vulnerabilities in Drupal, contributed projects, or dependencies to be exploited to modify core files. This is a serious problem in the real world that could negate the advantage of automatic updates.

Users who are comfortable with the command line can set up cron to run automatic updates as a user who has permission to modify core files. Yet if the command line is the only way to run updates securely, many vulnerable sites will never be fixed.

Proposed resolution

This problem was previously solved by the Update module. When updating contrib projects, if PHP did not have permission to change a contributed module, the Update module checked for file transfer backends. If one existed, authorize.php prompted for the user name and password of a user who had permission to write to the file system. No credentials were stored in Drupal, so an authorized user had to be present to modify files.

File transfer base class

Remaining tasks

Add prompt for file system user credentials if file system is not writable and file transfer backend is available.
~~Add script to run automatic updates from the command line as a user with permission to modify core files.~~ DONE:
- #3351895: Add Drush command to allow running cron updates via console and by a separate user, for defense-in-depth
- #3360485: Add Symfony Console command to allow running cron updates via console and by a separate user, for defense-in-depth
Update messages and documentation to explain how to configure automatic updates without giving PHP write access to core files.

User interface changes

Changes notices and adds a prompt to the admin UI when user initiates automatic update.

When it is installed, Package Manager should try to detect the paths of Composer and rsync

phenaproxima — Thu, 25 Jul 2024 03:39:18 +0000

Problem/Motivation

OK, look...it just sucks when you install Project Browser or Automatic Updates, only to get a nasty error because the paths to Composer and/or rsync are not in the web server's PATH.

There's no reason it has to be this way. We can't prevent it in under all circumstances, but we can make it a little less likely if Package Manager is installed at the command line (as is the case in the Starshot prototype, or any time Drush installs Package Manager).

Proposed resolution

During hook_install(), Package Manager should use Symfony's executable finder to try and locate Composer and rsync. If either is found, they should be written to the package_manager.settings config.

Automatic Updates Initiative meeting on Nov 12, 2024

hestenet — Tue, 12 Nov 2024 17:52:50 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

Improve messaging to users to backup database

Theresa.Grannum — Thu, 09 Jun 2022 18:42:15 +0000

Problem/Motivation

Move forward with feedback received from @rkoller, please find original comment below:

"I have applied the latest patch and have taken a look at the micro copy in the context of admin/reports/updates/automatic-update and admin/automatic-update-ready/. I like and agree that you have added texts on both pages" I would have a few additional points and thoughts to add.

1. I am not sure if starting off by providing a gently and friendly reminder saying, it is a good idea to back up your database, and then in the next step state in bold letters, that it is strongly recommended to back up the database and site, isn't a bit of mixed messaging. I would rather go with an explicit and clear statement on the automatic-update page that it is highly recommended to back up the database and site and then phrase the ready to update text more as the friendly reminder by stating that the downloaded assets will be applied next and that this step cannot be undone, the reason why an update is a good idea. Just provide clear and actionable steps what is recommended in the first place. Currently it sounds more like an update might be a good idea but isn't necessary and then you read on the ready to update page it is recommended after all. So a more explicit variant for the automatic-update page could be:
It is strongly recommended to back up your database and site before you begin.
(*maybe that sentence in bold on the automatic-update page?)

2. On the automatic-update page you have "back up your database" on the ready to update page you have "back up your database and site". It is advisable to be consistent with the actionable recommendation imho. Currently if someone is following the recommendation on the first page and just backs up the database could be puzzled when retrying and then reading on the ready to update page that database AND site should be updated now. Might be even considered annoying for some people. I would suggest to go with the link text "backup your database and site" on both pages.

3. One question that is probably still valid from #4. If an update is triggered manually there is the recommendation to back up the database and site while when the update is triggered and run automatically via cron there is no recommendation. Manually you get the strong recommendation to back up in the micro copy while when run automatically there is no notification for that? That "might" cause also uncertainty once someone triggers a manual update?

4. On the ready to update page you have the first line "Drupal core will be updated to 9.3.15" in a regular weight while the next line is bold. The sole focus is on the bold typeface (at least for me the "Drupal core will be updated part..." was more or less "invisible" or at least unnoticed that way) . I would perhaps make the whole block in a regular typeface and phrase the back up your database part a little bit different. Not that strongly but explaining the reasoning a little bit. I've added the word step to "this cannot be undone. At the end i would provide a brief note about what happens when you hit the cancel update button, stating that the downloaded assets get removed (aka they have to be re-downloaded if the user triggers the update process again):

Drupal core will be updated to 9.3.15. This step cannot be undone. The reason why to back up your database and site before the update was recommended in the first place. If you cancel the update process the downloaded assets get removed.

5. I could offer to put the issue on fridays ux meeting agenda (there is no entry yet - https://www.drupal.org/project/drupal/issues/3284204). There would be a few more pair of eyes to take a look at the issue and the word smithing part in particular. But i don't know how time sensitive the issue is since it is a stable blocker. But i think it would be a good choice.
And if i remember correctly Aaron also suggested a few weeks back in one automatic update issue to bring the automatic updates module in general to one of the UX meetings so that we could take a look as a group."

Tweaks to the core update page

pameeela — Thu, 31 Oct 2024 05:29:51 +0000

Problem/Motivation

Related to #3484802: Update extensions page title to be more clear the 'Update' and 'Update extensions' tabs are a bit confusing, it's not obvious that one is for core only. There are some other slightly odd things about this page that we could try to fix too.

Admittedly the entire 'Extend' section needs a UX overhaul; I don't want to try to solve this here, so am proposing a minor tweaks only.

Current page:

Proposed resolution

TBC

Remaining tasks

TBC

User interface changes

TBC

Update extensions page title to be more clear

pameeela — Thu, 31 Oct 2024 04:45:38 +0000

Problem/Motivation

With the extensions module enabled, we get two new tabs in the extend section: 'Update' and 'Update Extensions'. It's not clear from this that one is for core and one is for contrib, so I think we can try to clarify this slightly by renaming them. I don't want to try to solve the UX of this section here, so am proposing a minor tweak only.

Also, the 'Update Extensions' page currently has the title 'Automatic Updates Form'. Guessing this is just an oversight.

Before:

Proposed resolution

Rename the tabs and the page. Also remove the colons from the table headers, and use sentence case, to conform with the standard style.

After:

Remaining tasks

Create an MR with the changes
Update tests if needed?
Review
Merge

User interface changes

As noted.

Installed unsupported modules breaks Update extensions page

arunkumark — Tue, 13 Aug 2024 06:09:26 +0000

Problem/Motivation

If the website has been installed with a module and it becomes unsupported we are getting page break errors.

The website encountered an unexpected error. Try again later.

RuntimeException: The project 'pdf_serialization' can not be updated because its status is unsupported in Drupal\package_manager\ProjectInfo->getInstallableReleases() (line 103 of modules/contrib/automatic_updates/package_manager/src/ProjectInfo.php).
Drupal\automatic_updates_extensions\Form\UpdaterForm->getRecommendedModuleUpdates() (Line: 97)
Drupal\automatic_updates_extensions\Form\UpdaterForm->buildForm(Array, Object)
call_user_func_array(Array, Array) (Line: 536)

Steps to reproduce

1. Installed any modules that are supported earlier
2. Example pdf_serialization install using the command composer require drupal/pdf_serialization and install.
3. Install and enable the Automatic Updates
4. Visit the /admin/modules/automatic-update-extensions
5. Getting the page break error.

Proposed resolution

Instead of error we can able to show the message on the Update page.

Remaining tasks

Nil

User interface changes

API changes

Data model changes

[PP-1] Support any package to scaffold files

omkar.podey — Fri, 24 Mar 2023 12:03:18 +0000

Problem/Motivation

We also might want to consider at some supporting any package if drupal/core-composer-scaffold provided a UI(PROBABLY MEANT API) like getAllScaffoldfiles() that would return all files across all packages that are allowed. Without this I don't think we can support it.

Link to the comment => https://www.drupal.org/project/automatic_updates/issues/3338346#comment-...

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Document results of security audit

catch — Wed, 02 Oct 2024 13:45:51 +0000

Problem/Motivation

The Drupal Association commissioned a security audit of php-tuf and related packages, as well as rugged. I don't think this audit explicitly included package manager and automatic updates.

It's my understanding that the audit found some PHP issues all of which were minor and could be fixed in public. And some rugged code and workflow issues which could nearly all (or perhaps all) be fixed in public.

However, I am not aware of a way to find out:

1. Which issues were reported against which projects without reading the full audit.
2. Which issue/MR these issues were worked on.
3. Whether those issues/MRs were already resolved.

If there are still issues being worked on in private that also aren't fixed yet, we can't disclose those publicly, but we could maybe put 'private issue 1, project A, in progress' or alternatively just state when there are no open private issues.

Steps to reproduce

Proposed resolution

Document in this issue or somewhere else the above information.

Remaining tasks

User interface changes

API changes

Data model changes

Automatic Updates Initiative meeting on Oct 15, 2024

hestenet — Tue, 01 Oct 2024 17:13:13 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

StageBase does not need to use a NullLogger

phenaproxima — Wed, 03 Apr 2024 19:12:41 +0000

Problem/Motivation

StageBase's constructor calls $this->setLogger(new NullLogger()), but this isn't necessary. Logging calls can just use the nullsafe operator instead.

Remove constructor doc comments

phenaproxima — Tue, 10 Sep 2024 18:45:52 +0000

Problem/Motivation

Core's coding standards no longer require long-winded doc comments on class constructors that merely list parameters. We should remove these, just to lessen the number of lines in our code base and make our code easier to read.

We don't need to remove the docs for every single constructor; just the ones that have no additional information that you can't glean from the type hints.

Possible random failure in build tests for cron updates

tedbow — Fri, 27 Oct 2023 13:34:40 +0000

Problem/Motivation

https://www.drupal.org/pift-ci-job/2795252

There was just a doc block change that triggered the tests. On retest it passed

There error comes from the first line

$this->assertExpectedStageEventsFired(ConsoleUpdateStage::class, wait: 360);
$this->assertCronUpdateSuccessful();

My guess is the 360 wait not long enough because by the events logged we have not gotten to post-apply.
the other possibility is that randomly there is something that goes wrong with the apply.

but I am leaning towards the wait not being long enough because I have only seen this with cron related build tests we just have to wait and don't get intermittent UI checks.

Steps to reproduce

Proposed resolution

Run a test run here of just testAutomatedCron but run say 50 times. See if we get random fails
Try increase wait time and see if it goes away, We could also decrease the wait time and see if it happens more
If the longer wait time makes the problem go away just increase the wait time. \Drupal\Tests\package_manager\Build\TemplateProjectTestBase::assertExpectedStageEventsFired checks every 5 seconds if all the event have fired so increasing the wait time should not actually make the test run longer in most cases(as we very rarely see this error)

Remaining tasks

User interface changes

API changes

Data model changes

Use composer scripts to automate schema update

tyler36 — Thu, 06 Jul 2023 02:57:22 +0000

Recently, I encounter a login issue after updating Drupal from 10.0 to 10.1. https://www.drupal.org/project/drupal/issues/3370483
The solution was to manually run update after running a composer update command.

Although I have been using Drupal for a while, I have not been running the update script and, luckily, not encountered an issue until now.

For new developers (and forgetful ones), it may not be immediately obvious this is required.

The current experience:
- composer update
- Login to site
- Visit '/admin/reports'
- Click "status report"
- Scroll down to check for Database updates
- If exists, click `update`

Proposed resolution

Suggested experience:
- composer update

Composer supports various command event scripts that act similar to hooks for common commands.

Using this, we can automate scripts _after_ developers run composer update.
Something similar to:

    "scripts": {
        "post-update-cmd": [
            "drush updb -y"
        ]
    },

The above _example_ illustrators a working implementation when the project has drush installed. Not ever project has Drush so we would need to implment our own script which does the functional equavilent.

This is a repost of issue which I was closed as a duplicate and refered to Automatic Updates initiative. This request is a subset however, and not about checking for updates, verifiying compatability, download and installing. This request is automatically running the database updates, after a developer has specifically asked composer to update a package.

Add Configure link to info file

prashant.c — Tue, 05 Dec 2023 05:04:54 +0000

Problem/Motivation

The configure link missing for the module on the "/admin/modules" page.

Steps to reproduce

Enable the "Automatic Updates" module.

Proposed resolution

Add configure link in the .info.yml file.

Remaining tasks

Adding code to the module's info.yml file. Either it should take to the "/admin/reports/updates/settings" page or "/admin/reports/updates" page.

Along with this the "Automatic Updates Extensions" should also have the configure link to its settings page which is available at "/admin/reports/updates/automatic-update-extensions"

User interface changes

API changes

Data model changes

Automatic Updates Initiative meeting on Oct 1, 2024

hestenet — Tue, 17 Sep 2024 15:46:14 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

Automatic Updates Initiative meeting on Sep 17, 2024

hestenet — Tue, 17 Sep 2024 15:45:49 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

Automated Drupal 11 compatibility fixes for automatic_updates

project update bot — Thu, 04 Apr 2024 12:58:04 +0000

Problem/Motivation

Hello project maintainers,

This is an automated issue to help make this module compatible with Drupal 11.

Changes will periodically be added to this issue that remove deprecated API uses. To stop further changes from being posted, change the status to anything other than Active, Needs review, Needs work or Reviewed and tested by the community. Alternatively, you can remove the "ProjectUpdateBotD11" tag from the issue to stop the bot from posting updates.

The changes will be posted by the Project Update Bot official user account. This account will not receive any issue credit contributions for itself or any company.

Proposed resolution

You have a few options for how to use this issue:

Accept automated changes until this issue is closed
If this issue is left open (status of Active, Needs review, Needs work or Reviewed and tested by the community) and the "ProjectUpdateBotD11" tag is left on this issue, new changes will be posted periodically if new deprecation fixes are needed.

As the Drupal Rector project improves and is able to fix more deprecated API uses, the changes posted here will cover more of the deprecated API uses in the module.

Patches and/or merge requests posted by others are ignored by the bot, and general human interactions in the issue do not stop the bot from posting updates, so feel free to use this issue to refine bot changes. The bot will still post new changes then if there is a change in the new generated patch compared to the changes that the bot posted last. Those changes are then up to humans to integrate.
Leave open but stop new automated changes.
If you want to use this issue as a starting point to remove deprecated API uses but then don't want new automated changes, remove the "ProjectUpdateBotD11" tag from the issue and use it like any other issue (the status does not matter then). If you want to receive automated changes again, add back the "ProjectUpdateBotD11" tag.
Close it and don't use it
If the maintainers of this project don't find this issue useful, they can close this issue (any status besides Active, Needs review, Needs work and Reviewed and tested by the community) and no more automated changes will be posted here.

If the issue is reopened, then new automated changes will be posted.

If you are using another issue(s) to work on Drupal 11 compatibility it would be very useful to other contributors to add those issues as "Related issues" when closing this issue.

Remaining tasks

Using the patches

Apply the latest patch in the comments by Project Update Bot or human contributors that made it better.
Thoroughly test the patch. These patches are automatically generated so they haven't been tested manually or automatically.
Provide feedback about how the testing went. If you can improve the patch, post an updated patch here.

Using the merge request

Review the merge request and test it.
Thoroughly test the changes. These changes are automatically generated so they haven't been tested manually or automatically.
Provide feedback about how the testing went. If you can improve the merge request, create a new branch and merge request and work from there.

Warning: The 'project-update-bot-only' branch will always be overwritten. Do not work in that branch!

Providing feedback

If there are problems with one of the changes posted by the Project Update Bot, such as it does not correctly replace a deprecation, you can file an issue in the Drupal Rector issue queue. For other issues with the bot, for instance if the issue summary created by the bot is unclear, use the Project analysis issue queue.

Automated Drupal 11 compatibility fixes for automatic_updates

project update bot — Sat, 06 Apr 2024 15:37:28 +0000

Problem/Motivation

Hello project maintainers,

This is an automated issue to help make this module compatible with Drupal 11.

The changes will be posted by the Project Update Bot official user account. This account will not receive any issue credit contributions for itself or any company.

Proposed resolution

You have a few options for how to use this issue:

Accept automated changes until this issue is closed
If this issue is left open (status of Active, Needs review, Needs work or Reviewed and tested by the community) and the "ProjectUpdateBotD11" tag is left on this issue, new changes will be posted periodically if new deprecation fixes are needed.

As the Drupal Rector project improves and is able to fix more deprecated API uses, the changes posted here will cover more of the deprecated API uses in the module.

Patches and/or merge requests posted by others are ignored by the bot, and general human interactions in the issue do not stop the bot from posting updates, so feel free to use this issue to refine bot changes. The bot will still post new changes then if there is a change in the new generated patch compared to the changes that the bot posted last. Those changes are then up to humans to integrate.
Leave open but stop new automated changes.
If you want to use this issue as a starting point to remove deprecated API uses but then don't want new automated changes, remove the "ProjectUpdateBotD11" tag from the issue and use it like any other issue (the status does not matter then). If you want to receive automated changes again, add back the "ProjectUpdateBotD11" tag.
Close it and don't use it
If the maintainers of this project don't find this issue useful, they can close this issue (any status besides Active, Needs review, Needs work and Reviewed and tested by the community) and no more automated changes will be posted here.

If the issue is reopened, then new automated changes will be posted.

If you are using another issue(s) to work on Drupal 11 compatibility it would be very useful to other contributors to add those issues as "Related issues" when closing this issue.

Remaining tasks

Using the patches

Apply the latest patch in the comments by Project Update Bot or human contributors that made it better.
Thoroughly test the patch. These patches are automatically generated so they haven't been tested manually or automatically.
Provide feedback about how the testing went. If you can improve the patch, post an updated patch here.

Using the merge request

Review the merge request and test it.
Thoroughly test the changes. These changes are automatically generated so they haven't been tested manually or automatically.
Provide feedback about how the testing went. If you can improve the merge request, create a new branch and merge request and work from there.

Warning: The 'project-update-bot-only' branch will always be overwritten. Do not work in that branch!

Providing feedback

Change default setting to `allow_core_minor_updates` to true

tedbow — Tue, 29 Nov 2022 19:02:36 +0000

Problem/Motivation

Currently we have a hidden config to enable minor updates

Now that we are on track to be Starshot we should support this out of the box.

We finished #3314143: Add documentation for testing minor updates

Proposed resolution

Change the default of allow_core_minor_updates to true
update tests and add to update path test

Remaining tasks

User interface changes

API changes

Data model changes

SiteFilesExcluder.php should also take into account assets:// files

tedbow — Thu, 18 Apr 2024 20:39:24 +0000

Problem/Motivation

SiteFilesExcluder.php should also take into account assets:// files

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Automatic Updates Initiative meeting on Aug 6, 2024

hestenet — Tue, 06 Aug 2024 16:47:32 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

0️⃣ Who is here today? Share the most random fact you can think of!

hestenet (he/him)	Tim from the DA :wave::skin-tone-3:My fact is that Alligators and Crocodiles don't have a traditional immune system with white blood cells and such - instead their blood is actually chemically hostile to invasive pathogens.
ergonlogic	Christopher from ConsensusRandom fact: The plastic tip at the end of a shoelace is called an "aglet"
dts	Hi folks. Just checking in on things because @hestenet (he/him) reached out to me on the root keys management challenge.

1️⃣ Do you have any topics to propose for the meeting today? Feel free to propose them in this thread, and then I will give them their own unique threads for discussion. Conversation moving slow? Go ahead and open your own thread in the next numeric order.

2️⃣ The DA and the Consensus Enterprises team met to regroup on some follow up issues for Key Rotation with Rugged, performance/quality of life etc.

3️⃣ We did regroup a bit on binning strategy - and the team is going to try and add a command for changing the number of bins without a highly disruptive resigning of everything. https://gitlab.com/rugged/rugged/-/issues/194However @phenaproxima - folks did ask if you could provide more specific details about the client performance issues you've been encountering in terms of the distinction between # of http requests as performance issues, size of bins, etc... If you have any key metrics you are looking at/can share examples that would help.

4️⃣ Are there remaining 'stable' blockers for AutoUpdates?

INCOMPLETE

5️⃣ HSM-based root keys

hestenet (he/him)

DA has to complete rotation process to satisfaction

INCOMPLETE

Participants:

hestenet, ergonlogic, dts, phenaproxima, drumm

Meeting link: https://drupal.slack.com/archives/C7QJNEY3E/p1722962751256029

ergonlogic

@dts Previously, you'd suggested that https://github.com/sigstore/root-signing might be useful. Could you elaborate on what you had in mind?

Automatic Updates Initiative meeting on July 23, 2024

hestenet — Tue, 23 Jul 2024 16:38:35 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

0️⃣ Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

hestenet (he/him)	Tim from the DA, kicking off these threads :wave::skin-tone-3:
tedbow	Ted from Acquia
lamech	Dan from Consensus.
ergonlogic	Christopher, from Consensus, recently back from leave

1️⃣ Do you have any topics to propose for the meeting today? Feel free to propose them in this thread, and then I will give them their own unique threads for discussion. Conversation moving slow? Go ahead and open your own thread in the next numeric order.

2️⃣ The AutoUpdates endpoint is in production for both core and contrib :tada: - but there are follow-ups that will need some threads below.

3️⃣ Follow-up: Exercising a key rotation.

hestenet (he/him)

@drumm and I need to schedule this soon - but we're doing the SSO window this week, so maybe next week.

4️⃣ Follow-up: Rugged issues

INCOMPLETE

5️⃣ Follow-up - Binning strategy - see this thread: https://drupal.slack.com/archives/C7QJNEY3E/p1721741665691099

hestenet (he/him)

These are issues we've uncovered since taking this to production under the full scale of core and contrib:https://gitlab.com/rugged/rugged/-/issues/192 securesystemslib includes non-compliant `keyid_hash_algorithms` property when generating key IDshttps://gitlab.com/rugged/rugged/-/issues/191 Reset processing targets batch on bootNice to have - https://gitlab.com/rugged/rugged/-/issues/149 Clean up more completely when targets containing empty directories are processed.

@drumm Is there any update on popularity-based binning for TUF? (edited)

INCOMPLETE

Participants:

hestenet, tedbow, lamech, ergonlogic, phenaproxima, drumm

Meeting link: https://drupal.slack.com/archives/C7QJNEY3E/p1721752763467489

phenaproxima	It occurs to me that we could use both popularity binning and hash binning. Example — the top 50 modules have one bin. The next 50 are in another. Same with the next two groups of 50. After that, everything is just in hash bins.
drumm	I thought you found some good speedups in the client and hadn’t prioritized it. And I’ve been short on time with Drupal.org upgrades.Fewer hashed bins - straightforwardDelegations per-project - kinda hard, need to build support for that into Rugged.Delegations by popularity - kinda hard, and more complex integration since rugged is decoupled from Drupal.org’s DB, would be another integration point to communicate the binsAnd on top of that, any change either needs a week outage to re-sign everything, or an update to Rugged to allow TUF repo structure rearranging
drumm	How is the client side speed going in general?
phenaproxima	It’s…not too bad, but would be better to optimize more.
drumm	One thing I’d like to see is calculating the number of http requests there would be with each strategy to see how effective each would be, for the starshot demo and any other good test cases we have. As in get the list of URLs that will be requested and calculate where they’d fall with a few different numbers of fewer bins and the delegation strategies
hestenet (he/him)	What's 'not too bad' look like in the real world?
phenaproxima	@hestenet (he/him) That’s a good idea, finding some metrics. If I had an ordered list of the most popular modules, I could maybe (using the Starshot prototype as a model) tell you how many HTTP requests we were going to do.
drumm	You have one, but Slack is where thoughts go to be forgotten https://drupal.slack.com/archives/C02CRC4BZ0V/p1720800512896799?thread_t...
phenaproxima	Yeah, I remember that you sent one. Just needed to dig it up. Thanks @drumm!
ergonlogic	Re. metrics, please take into account the size of the various metadata file downloads. Hashed bins are meant to minimize overall bandwidth usage.

Automatic Updates Initiative meeting on July 9, 2024

hestenet — Tue, 25 Jun 2024 16:54:01 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

0️⃣ Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

hestenet (he/him)	Tim from the DA getting things organized.
tedbow	Ted from Acquia
drumm	:wave:
xjm	:wave: Getting caught up
Kristen Pol (she/her)	Kristen, just crossed the Oregon => California border, seeing what ya'll have been up to

1️⃣ Do you have any topics to propose for the meeting today? Feel free to propose them in this thread, and then I will give them their own unique threads for discussion. Conversation moving slow? Go ahead and open your own thread in the next numeric order.

2️⃣ Rugged: Status and Follow-upsContrib endpoint: https://packages.drupal.org/8/Core endpoint: https://packagist-signed.drupalcode.org/https://drupal.slack.com/archive... (edited)

2️⃣ 1️⃣ The core endpoint above is a Satis mirror of all of:https://packagist.org/packages/drupal/We have a few things to address:Satis performance/completing the backfilling/signing of this dataBin sizes - this is also going to include recipes and other general projects.(edited)

hestenet (he/him)	@drumm Can you maybe speak a bit more to the state of the satis situation?
hestenet (he/him)	Part of the satis issues seem like they may be related to GitHub rate limiting and timeouts - that's being investigated by @Max Whitehead
drumm	And/or DNS resolving issues, or memory/etc constraints
xjm	Off-topic: I read "Satan's mirror" instead of "Satis mirror".
hestenet (he/him)	:rolling_on_the_floor_laughing: :devil:
Kristen Pol (she/her)	I just drove by Satan's lake today so that's on topic (a bit late)

2️⃣ 2️⃣ Rugged follow up issues:https://gitlab.com/rugged/rugged/-/issues/192 securesystemslib includes non-compliant `keyid_hash_algorithms` property when generating key IDshttps://gitlab.com/rugged/rugged/-/issues/191 Reset processing targets batch on bootNice to have - https://gitlab.com/rugged/rugged/-/issues/149 Clean up more completely when targets containing empty directories are processedPlus: Would like to exercise a key rotation. (edited)

3️⃣ Updating PHP-TUF / AutoUpdates contrib to use the new signed endpointsIt worked with the contrib endpoint. In a holding pattern on the core endpoint because of the satis issues noted in 2️⃣ 1️⃣ (edited)

INCOMPLETE

Participants:

hestenet, tedbow, drumm, xjm, Kristen Pol, catch

Meeting link: https://drupal.slack.com/archives/C7QJNEY3E/p1720546367108319

tedbow

We still to make sure this is very solid before turn this on by default and require TUF.BecauseIf there are problems with the drupal.org TUF endpoints that stops an AutoUpdate of a critical security update from happening then if could be argued the site is less secure than if they didn’t have TUF at all(and they got the update)TUF validation happens on the Composer level so if there was problem with the drupal.org endpoint then the site could not do any Composer operations at all not just Package Manager operationsSo far we have really only had 1 day where we haven’t a problem using both endpoints so we need to test more. Maybe a scheduled GitHub workflow on https://github.com/php-tuf/drupal-project could help prove it is working consistently

Automatic Updates Initiative meeting on June 11, 2024

hestenet — Tue, 28 May 2024 17:14:31 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

0️⃣ Who is here today? Comment in the thread below to introduce yourself and maybe let us know if there's a cool restaurant in your area or something like that that you want to try soon 🙂

hestenet (he/him)	Tim L from the DA - there is a new vegan mexican street food place nearby called Chilango PDX I'd like to try.hestenet
TravisCarden	Travis Carden from Acquia. I'd like to try your vegan Mexican street food place, too. :stuck_out_tongue:
tekNorah	tekNorah lurking from Mokena, IL I’ve been thinking a lot about an all-you-can-eat Sushi place up the road lately (edited)

1️⃣ Do you have any topics to propose for the meeting today? Feel free to propose them in this thread, and then I will give them their own unique threads for discussion. Conversation moving slow? Go ahead and open your own thread in the next numeric order.

TravisCarden

Symfony Process broke Composer Stager builds--and my heart: https://github.com/symfony/symfony/pull/57317. I might have been able to work around it with Composer conflicts and constraints, but drupal/core-recommended requires exactly the first version that contained the regression.

Honestly, we're approaching the finish line of having the production Rugged endpoint up and running, but we're very much in the 'deploy changes, find problems, troubleshoot, repeat' phase.

2️⃣ We may have some issues testing on staging - https://drupal.slack.com/archives/C7QJNEY3E/p1718039588124509One possibly related to the htaccessOne maybe related to the caching mirror for core packages.@drumm to look when back tomorrow.

3️⃣ Staging and production signing keysI'm still collecting the keys from all the keyholders for staging and production so we can get real keys installed.

4️⃣ Per @TravisCardenSymfony Process broke Composer Stager builds--and my heart: https://github.com/symfony/symfony/pull/57317. I might have been able to work around it with Composer conflicts and constraints, but drupal/core-recommended requires exactly the first version that contained the regression.

INCOMPLETE

Participants:

hestenet, TravisCarden, tekNorah, Brad Jones, larowlan

Meeting link: https://drupal.slack.com/archives/C7QJNEY3E/p1718124850737159

hestenet (he/him)	@TravisCarden do we need to tag in any other folks to be made aware? (edited)
TravisCarden	Adam H and Ted B are already aware on our side. If we have any contacts on the Symfony side, that would be nice.
hestenet (he/him)	Not that I personally know, but maybe someone will come along..
hestenet (he/him)	:question-spin: :symfony: Does anyone have symfony contacts? Please let us know here.
Brad Jones	Don't we have a formal contact with them via the security team?
hestenet (he/him)	Good point, I think we might. I'll repost over there.
larowlan	alexpott, xjm and I have access to their security team
TravisCarden	Would it be appropriate to ask one of them to take a peek at this issue? Or to ping someone more fitting?
Brad Jones	I've filed issues on Symfony before when there's an issue affecting Drupal and have gotten feedback from maintainers pretty quickly. Sometimes it's not the answer I wanted but if you have a proposed PR that helps too (as you do.)

Automatic Updates Initiative meeting on June 25, 2024

hestenet — Tue, 11 Jun 2024 17:30:52 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

0️⃣ Who is here today? Comment in the thread below to introduce yourself and share a random factoid/weird wikipedia article

INCOMPLETE

1️⃣ Do you have any topics to propose for the meeting today? Feel free to propose them in this thread, and then I will give them their own unique threads for discussion. Conversation moving slow? Go ahead and open your own thread in the next numeric order.

tedbow	Ted from Acquia
pooja_sharma	pooja_sharma , India
Kristen Pol (she/her)	Kristen, checking in on progress during road trip :mount_fuji:

INCOMPLETE

2️⃣ Status update: Official auto-updates endpoints on Drupal.org - using the Rugged server - per @drummThe top 150 contrib projects should be available on the actual production endpoint with proper signing tomorrow. Packaging pipeline integration and then backfilling the rest comes soon.The Fastly config to create the proper url structure for accessing the prod endpoint is in progress. The Core packages may be available by end of week.

Kristen Pol (she/her)

Are there Starshot “tracks” for this initiative? I know Ted shared some issues… I’ll need to find that starting point

INCOMPLETE

3️⃣ Are there Starshot “tracks” for this initiative?See this issue[#3454620]

hestenet (he/him)	@drumm Will @tedbow need to commit a new initial root?
Kristen Pol (she/her)	Exciting!
drumm	I believe so, and getting any workarounds for staging cleaned out
drumm	The contrib TUF metadata is now available https://packages.drupal.org/8/metadata/1.root.json. Right now this just has the automatic_updates project. Once connected to packaging, it will have any project that has had a commit to a release branch or release updated. Then I’ll backfill starting with the 150 most used modules & themes
drumm	Its now connected to packaging and the 200 most used projects are being processed
drumm	1,200 projects have been processed, so we’ll either finish or find scalability issues with contrib this week
tedbow	@drumm so in our test composer we have 2 TUF URL’s that we sethttps://drupal:drupal@packages.staging.devdrupal.org/8 https://signed-packagist.staging.devdrupal.orgThis is replacing staging.devdrupal.org/8 correct?what about the other one?
drumm	Right. Core is still being worked on. Don’t have an ETA right now, but it might be this week
tedbow	@drumm ok thanks. They will be 2 separate URLs though, correct?
drumm	Right
tedbow	@drumm sorry I see both urls above have staging.devdrupal.org/8 so to be clear this is replacing https://drupal:drupal@packages.staging.devdrupal.org/8 , correct
drumm	http://packages.drupal.org/8/metadata/ is the TUF repo for the https://packages.drupal.org/8/ composer metadata. So that set of things can all be replaced.

hestenet (he/him)

per @Kristen Pol (she/her)

Participants:

hestenet

Participants:

tedbow, pooja_sharma, Kristen Pol, hestenet, guptahemant

Meeting link: https://drupal.slack.com/archives/C7QJNEY3E/p1719334474163109

Automatic Updates Initiative meeting on May 14, 2024

hestenet — Tue, 30 Apr 2024 16:54:41 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

0️⃣ Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

hestenet (he/him)	Tim from the DA, recovering from DrupalCon
xjm	:wave: xjm, also recovering
tedbow	Ted from Acquia, same, and also from delayed flight getting me home at 3am:pensive:
xjm	And I thought my midnight arrivals were bad :disappointed:

1️⃣ Do you have any topics to propose for the meeting today? Go ahead and open your own thread in the next numeric order.

2️⃣ Core signing mirror - what is the state of testing this out?

INCOMPLETE

3️⃣ Additional Rugged fixes/enhancements related to HSM root key management are making lovely progress here: https://gitlab.com/rugged/rugged/-/issues/159

tedbow

Last time I tested at Drupalcon(can’t remember which day) there was problem with the hash validation. I think @drumm said this was known cache problem. Trying again now…

INCOMPLETE

4️⃣ Any implications of Starshot we need to talk about here?

hestenet (he/him)

@ergonlogic - anything to summarize here?

ergonlogic

Sure. Over the past couple weeks, we've:Added checks to ensure that rugged generate-keys won't overwrite existing keys (along w/ a --force option) (See: #162 for details)Fixed a bug to allow removal of targets cleanly (see #179 for details)Added signature validation whenever we're operating on root metadata (#172)Added a signature threshold validity check (#182), so that we know when partial root metadata is ready to be deployed.Added a command to generate new partial root metadata based on the current one (#163)

INCOMPLETE

Participants:

hestenet, xjm, tedbow, ergonlogic

Meeting link: https://drupal.slack.com/archives/C7QJNEY3E/p1715714019551819

xjm	I think the main thing is that we consider Autoupdates (along with Project Browser and Recipes) prerequisites for the initiative

Automatic Updates Initiative meeting on May 28, 2024

hestenet — Tue, 28 May 2024 16:56:09 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

0️⃣ Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

hestenet (he/him)	Hi! I'm Tim from the DA, in Portland, OR
Aziz	Hi! I'm Aziz from Sousse, in Tunisia. I am aiming to contribute more to Automatic Updates module.
tedbow	Hi:wave: Ted from Acquia in Ithaca, NY

1️⃣ Do you have any topics to propose for the meeting today? Feel free to propose them in this thread, and then I will give them their own unique threads for discussion. Conversation moving slow? Go ahead and open your own thread in the next numeric order.

2️⃣ Production deployment:The production infrastructure is actually now up and running, with temporary keys.

INCOMPLETE

2️⃣ 1️⃣ We are still doing some troubleshooting of things that cause stuck queues in the signing process

hestenet (he/him)	@drumm feel free to go deeper into the details where relevant.
drumm	We’re still working out various things:

2️⃣ 2️⃣ We need to start putting in real, HSM-based keys (hopefully this week)

3️⃣ At some point soon here, @tedbow, we'll be able to have you try pointing things to the new prod endpoint.

tedbow

:tada:Great let me know I will give it a try

4️⃣ When the prod endpoint is available and has the real keys, what's the remaining checklist for AutoUpdates?

INCOMPLETE

Participants:

hestenet, Aziz, tedbow, drumm

Meeting link: https://drupal.slack.com/archives/C7QJNEY3E/p1716915388100799

tedbow

We would need to switch on TUF requirement. This would be very easy.But we would need some idea of the caching lag will be. I know at Drupalcon at least for the staging TUF data there was some lag for Drupal core because the files are on github and the TUF metadata is on d.o.So I think we need to make sure this lag is very minimal. Because the way the Composer Plugin works(and they way Composer works) you want not be able to do any Composer operations if some of the packages had invalid TUF metadata(maybe because a Drupal core release just happened)

Automatic Updates Initiative meeting on Apr 30, 2024

hestenet — Tue, 30 Apr 2024 16:54:07 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

0️⃣ Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

INCOMPLETE

1️⃣ Do you have any topics to propose for the meeting today? Go ahead and open your own thread in the next numeric order. (edited)

hestenet (he/him)	Tim from the DA :wave::skin-tone-3:
tedbow	Ted from Acquia :wave:
hestenet (he/him)	How's Ithaca?
tedbow	Getting warm!
hestenet (he/him)	Nice! :sunglasses:

2️⃣ Ongoing conversation about the core signing mirror:Rugged queuing issues which had caused some instability/unexpected expiration seem to be resolved:https://drupal.slack.com/archives/C7QJNEY3E/p1713290697392609

3️⃣ Additional Rugged fixes/enhancements related to HSM root key management are making lovely progress here: https://gitlab.com/rugged/rugged/-/issues/159

hestenet (he/him)

Feel free to add any color commentary @ergonlogic

ergonlogic

Sure, here are recent accomplishments:Last week, we had been focused on a stability issue w/ refreshing expiring metadata that was affecting production (#173). The underlying issue was identified, fixed, and tested. The fix has been deployed, and it appears to have stabilized the production system. So this has been resolved.Along similar lines, we found a configuration setting for Celery that we believe will help with RabbitMQ dropping messages, especially around service restarts (#176). This was one of the root causes identified in #173.We also got Rugged signed-up for the Gitlab FLOSS program. So we now get all the features and lots of CI minutes.We've continued to refine the Runbooks for HSM-based root key management. this includes splitting them out of the main docs, into a template that can then be forked by the DA (and others) to host ceremony artifacts, etc.Next steps:For the next few days, we're focused on testing and streamlining the process of updating root metadata (ie. adding, removing and rotating keys, as well as re-signing any metadata whose key was changed)

4️⃣ Testing TUF Dev metadata

INCOMPLETE

@drumm I think I found problem with the TUF staging data which relates to this problem I am seeing try to test it[304] https://drupal:@packages.staging.devdrupal.org/8/metadata/bin_198-199...
[TUF] Target 'files/packages/8/p2/drupal/automatic_updates.json' limited to 26600 bytes.
Downloading https://drupal:@packages.staging.devdrupal.org/files/packages/8/p2/dr...
> pre-file-download: Tuf\ComposerIntegration\Plugin_composer_tmp2->preFileDownload
[TUF] Loading https://drupal:drupal@packages.staging.devdrupal.org/8/metadata/bins.json from static cache.
Downloading https://drupal:***@packages.staging.devdrupal.org/8/metadata/bin_236-237... if modified
https://drupal:drupal@packages.staging.devdrupal.org/8 could not be fully loaded (Maximum allowed download size reached. Content-length header indicates 27969 bytes. Allowed 26600 bytes), package information was loaded from the local cache and may be out of datecc @phenaproximawill explain…

tedbow

@drumm I am running into a new problem Invalid credentials (HTTP 403) for 'https://drupal:drupal@packages.staging.devdrupal.org/8/packages.json', aborting.

Participants:

hestenet, tedbow, TravisCarden, ergonlogic, drumm

Meeting link: https://drupal.slack.com/archives/C7QJNEY3E/p1714495523963969

phenaproxima	If I see stuff like this, I immediately suspect CRLF tomfoolery.
tedbow	when I request this directly https://packages.staging.devdrupal.org/8/metadata/bin_198-199.json"files/packages/8/p2/drupal/automatic_updates.json": { "hashes": { "sha256": "10c2001a0f0a8ac76580700d6eba02f943ee70a19104482234de5b4104ee244c" }, "length": 26599 }

Automatic Updates Initiative meeting on Apr 16, 2024

hestenet — Tue, 16 Apr 2024 18:05:48 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

0️⃣ Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

tedbow

Ted from Acquia

1️⃣ Do you have any topics to propose for the meeting today? Feel free to propose them in this thread, and then I will give them their own unique threads for discussion. Conversation moving slow? Go ahead and open your own thread in the next numeric order.

2️⃣ The Core Signing mirror is available for testing: https://drupal.slack.com/archives/C7QJNEY3E/p1712868963832679Would appreciate any feedback/follow up on that when possible.

INCOMPLETE

0️⃣ Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

tedbow	Will try to test tomorrow
hestenet (he/him)	Just touching base @tedbow - have you had any chance to give this a look?
tedbow	@hestenet (he/him) sorry testing this out now https://github.com/php-tuf/drupal-project/pull/4We have pipeline that create a project so should fail if there is problem
hestenet (he/him)	Alrighty, @drumm and I can keep an eye on it.
tedbow	hmm pipeline not running:confused:
hestenet (he/him)	Not running at all, or failing early?
tedbow	not being kicked off

tedbow

Ted from Acquia

1️⃣ Do you have any topics to propose for the meeting today? Feel free to propose them in this thread, and then I will give them their own unique threads for discussion. Conversation moving slow? Go ahead and open your own thread in the next numeric order.

2️⃣ The Core Signing mirror is available for testing: https://drupal.slack.com/archives/C7QJNEY3E/p1712868963832679Would appreciate any feedback/follow up on that when possible.

INCOMPLETE

Participants:

tedbow, hestenet, phenaproxima, drumm, ergonlogic, catch

Meeting link: https://drupal.slack.com/archives/C7QJNEY3E/p1713290662163639

tedbow	Will try to test tomorrow
hestenet (he/him)	Just touching base @tedbow - have you had any chance to give this a look?
tedbow	@hestenet (he/him) sorry testing this out now https://github.com/php-tuf/drupal-project/pull/4We have pipeline that create a project so should fail if there is problem
hestenet (he/him)	Alrighty, @drumm and I can keep an eye on it.
tedbow	hmm pipeline not running:confused:
hestenet (he/him)	Not running at all, or failing early?
tedbow	not being kicked off

Automatic Updates Initiative meeting on Mar 19, 2024

hestenet — Tue, 19 Mar 2024 17:24:56 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

0️⃣ Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

TravisCarden	Travis from Acquia, of Composer Stager fame. :wink:
hestenet (he/him)	:tada:

1️⃣ Do you have any topics to propose for the meeting today? Feel free to propose them in this thread, and then I will give them their own unique threads for discussion. Conversation moving slow? Go ahead and open your own thread in the next numeric order.

2️⃣ Server Side Audit Follow-ups

hestenet (he/him)

Root Key Signing processNext up:I am running a second run through of our root-key handling ritual with our HSMshttps://rugged.works/how-to/hsm/The final version of our personal ritual will likely be a bit more lightweightWe need still need to resolve a binary format mismatch (just a different option flag) We need to solve the rotation/invalidation ritual We need to go back throught he audit report to see if anything else is outstanding, but there wasn't much on the Rugged side.

3️⃣ Client Side Audit Follow-ups

4️⃣ What are our general next steps and blockers to official commit as 'experimental' to core - from the Drupal code/client side?What is the timeline like for client components? (edited)

hestenet (he/him)

Not audit specific, but also client side - @TravisCarden has recently released: https://github.com/php-tuf/composer-stager/releases/tag/v2.0.0-beta4

hestenet (he/him)	@phenaproxima and @tedbow and @effulgentsiaI see a lot of activity on the php-tuf repo (and @TravisCarden’s work on composer-stager too).Separately from the audit threads: How much of that is now in 'we're doing nice to haves but things are actually ready to go' vs, 'we're still clearing blockers?'
phenaproxima	PHP-TUF’s biggest advancement recently is that we’ve been able to ditch the Python dependency.

5️⃣ What are our general next steps and blockers to official commit as 'experimental' to core - from the DA infra/server side? (edited)

hestenet (he/him)	From the DA/Infra side:We are finalizing the root key base ritual We need to test the root key rotation/invalidation ritualI need to distribute the ritual process/materials to key holdersWe would then need to redeploy our container stack as the production endpoint with the real root key signatures installed. (and whatever I'm missing @drumm?)
drumm	Core signing - get packagist clone hosting container setup, deploy everything to staging, iterate, production
hestenet (he/him)	Of course! Thanks.

Participants:

hestenet, phenaproxima, drumm, TravisCarden

Meeting link: https://drupal.slack.com/archives/C7QJNEY3E/p1710869015059659

Automatic Updates Initiative meeting on Apr 2, 2024

hestenet — Tue, 02 Apr 2024 16:48:46 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

0️⃣ Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

hestenet (he/him)	Tim from the DA.
TravisCarden	Travis Carden from Acquia.
tedbow	Ted from Acquia

1️⃣ Do you have any topics to propose for the meeting today? Feel free to propose them in this thread, and then I will give them their own unique threads for discussion. Conversation moving slow? Go ahead and open your own thread in the next numeric order.

2️⃣ At some point (hopefully this week?) I plan to catch up the meeting credits for this initiative from Q1 of this year.

3️⃣ AutoUpdates infrastructureCore package mirror for signing. Full redeploy of staging with core package signing + root keys(edited)

4️⃣ Root key signing management

hestenet (he/him)

@nnewton has been getting help from @Max Whitehead on the provisioning of the core packaging mirror infra for @drumm.

INCOMPLETE

5️⃣ Symfony 7 updates

hestenet (he/him)

One of the things I'm currently looking at is the comparison between the HSM runbook:https://rugged.works/how-to/hsm/init_tuf_repo/And the process we call generating 'intermediate keys' in the old signify runbook:https://git.drupalcode.org/project/infrastructure/-/tree/main/signing#dr...

5️⃣ 1️⃣ Composer Stager Symfony 7 https://drupal.slack.com/archives/C7QJNEY3E/p1712010554181099

5️⃣ 2️⃣ Package Managerhttps://drupal.slack.com/archives/C7QJNEY3E/p1712010629789249

5️⃣ 3️⃣ Any other aspects of the client side code?

hestenet (he/him)	I think @effulgentsia you have summarized some in-flight issues here:#3346707: Add Alpha level Experimental Package Manager module#comment-15525901
effulgentsia	Yes, from my perspective, the two key next steps are:Get Composer Stager committed (or at least RTBC'd if we want to postpone the actual commit until Package Manager is also ready) to core. See[#3346707]) once @tedbow updates it.
tedbow	@effulgentsia do you think it is useful to update even if a bunch of tests might fail? thinking of this https://github.com/php-tuf/composer-stager/pull/351

6️⃣ Are there any other blocking issues (server, client or otherwise) that we should be tracking?The roadmap issue for review is:#3319030: Drupal Core Roadmap for Automatic Updates (edited)

7️⃣ I created an issue to make Automatic Update Extensions stabl[#3436741]This allows updating modules and themes.It is only in the contrib version but I think entice more people totry contrib and therefore the code that is in the Core merge requestsThere are few child issues to resolve but nothing major

hestenet (he/him)

Nice! That's really cool

@TravisCarden I think we will need new beta of Composer Stager with the Symfony 7 changes to be able to make the core merge requests work

Participants:

hestenet, TravisCarden, tedbow, ergonlogic

Meeting link: https://drupal.slack.com/archives/C7QJNEY3E/p1712076557638149

TravisCarden	When would you like that? And would you like it to contain that change and nothing else?
tedbow	@TravisCarden I guess as soon possilbe.I guess would like just that change or at least not other BC breaking changes.Just quickly looking at the commit log this jumps out at me https://github.com/php-tuf/composer-stager/commit/6f1051ccffa4f312f71eb0...

Automatic Updates Initiative meeting on Feb 6, 2024

hestenet — Tue, 06 Feb 2024 17:40:38 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

0️⃣ Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

hestenet (he/him)	Tim from the DA, getting things started.
tedbow	Ted from Acquia

1️⃣ Do you have any topics to propose for the meeting today? Feel free to propose them in this thread, and then I will give them their own unique threads for discussion. Conversation moving slow? Go ahead and open your own thread in the next numeric order.

2️⃣ :beetle: Bug - the automatic re-signing in staging to keep the meta data from expiring has stopped working. Most likely because our RabbitMQ instance is falling on its face. We are investigating and figuring out a fix.

3️⃣ As mentioned yesterday - Root Key HSM handling:https://gitlab.com/rugged/rugged/-/issues/160#note_1748431038works - and @ergonlogic is buttoning things up, documented the proposed ritual, and will let us know when we can deploy and test this out.

4️⃣ In yesterday's chat with the auditors @tedbow generously helped to make sure they were on the same page about scope and attack surface, and talked through some general thoughts around all that.

Participants:

hestenet, tedbow

Meeting link: https://drupal.slack.com/archives/C7QJNEY3E/p1707241300321479

Automatic Updates Initiative meeting on Feb 20, 2024

hestenet — Tue, 06 Feb 2024 17:40:52 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

0️⃣ Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

hestenet (he/him)	Tim from the DA :wave::skin-tone-3:
tedbow	Ted from Acquia

2️⃣ On Monday we held the Security Audit Report meeting for the client side work.TLDR; Good work everyone! It was tough to find anything serious. There are a few issues that need to be addressed confidentially, and a few that can be addressed as hardening steps in public.Who is comfortable keeping track of that? P.S: If you missed that call and are a member of that private channel, both the report and the recording are in there.

tedbow	I can make spreadsheet or doc(private) of the issues and corresponding public or private issues
hestenet (he/him)	That would be huge, thanks, Ted.

3️⃣ Tomorrow @ergonlogic, @drumm and I are meeting on the root key handling features and ritual that were added to Rugged. Will report back following that.

Participants:

hestenet, tedbow

Meeting link: https://drupal.slack.com/archives/C7QJNEY3E/p1708450870656059

Automatic Updates Initiative meeting on Mar 5, 2024

hestenet — Tue, 05 Mar 2024 16:53:50 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

0️⃣ Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

hestenet (he/him)	Tim from the DA kicking off early, because the HSM test process overlaps the officially scheduled time.
TravisCarden	Travis Carden from Acquia
tedbow	Ted from Acquia, catching up

1️⃣ Do you have any topics to propose for the meeting today? Feel free to propose them in this thread, and then I will give them their own unique threads for discussion. Conversation moving slow? Go ahead and open your own thread in the next numeric order.

2️⃣ HSM signing ritual end-to-end test - @drumm and @ergonlogic and I are doing this later today - and can report back to this thread afterwards.Rough draft of the ritual runbook in this thread (edited)

INCOMPLETE

3️⃣ Server-side security audit remediationsMost of the recommendations will be resolved when we verify completion of the root key management/HSM ritual - but we'll do a pass through the report to verify anything else.

hestenet (he/him)

Runbook draft: https://rugged.works/how-to/hsm/

4️⃣ Client-side security audit remediationsWithout naming any non-public issues - how is this going?

5️⃣ Planning for deploymentWill be based on outcome of HSM test, and status of audit remediations

Participants:

hestenet, TravisCarden, tedbow, dts

Meeting link: https://drupal.slack.com/archives/C7QJNEY3E/p1709657658540349

Automatic Updates Initiative meeting on Jan 23, 2024

hestenet — Tue, 23 Jan 2024 18:04:48 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

0️⃣ Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

tedbow	Ted from Acquia, AutoUpdates tech lead
hestenet (he/him)	Welcome back from PTO! :wave::skin-tone-3:
dts	Hi all. David from SF here. 🙂
Warped	Chief resident lurker, and occasional Contributor who likes to learn and keep up to date.

1️⃣ Do you have any topics to propose for the meeting today? Feel free to propose them in this thread, and then I will give them their own unique threads for discussion. Conversation moving slow? Go ahead and open your own thread in the next numeric order.

2️⃣ @ergonlogic Would you care to give us an update about how the HSM integration into Rugged is coming along?

3️⃣ As mentioned before - working with Cure53 on the Drupal code side of the audit.

ergonlogic	Sure. I've completed #161 (Initialize repo w/ existing root metadata), and I've started on #160 (Generate root JSON w/ HSM-based root key(s))
hestenet (he/him)	Anyway we can help with that? Run some tests using our HSMs or anything?

4️⃣ Module/Drupal code side updates here please!

5️⃣ Decide if we should make Rsync a hard requirement[#3416542]

tedbow	Thanks to @effulgentsia for writing that upTLDR, rsync is very good, trying to redo it in PHP is very hard
Warped	Comments added to issue. Hopefully helpful comments. :thinking_face:
tedbow	@Warped thanks. replied

6️⃣ Reminder we have stable 3.0.0 version of the Contrib module https://www.drupal.org/project/automatic_updatesPlease use it!If don’t plan to leave it on actual updates we could also use people testing it on different systemsWe have ~100 sites reporting using the 3.x version and ~170 using the 2.x verisonSmall numbers but also nobody is failing bugs/issues so hopefully that is good sign

7️⃣ Probably going to mark the 8.x-2.x branch of the module as supported[#3416554]

Incomplete transcript

Participants:

tedbow, hestenet, dts, Warped, ergonlogic

Meeting link: https://drupal.slack.com/archives/C7QJNEY3E/p1706033100270769

tedbow

We had problems with drupalci so switched testing of 3.0.x to gitlabciWe don’t know of any reason to stay on 8.x-2.x except that it supports Drupal 9Drupal 9 has reached EOL

Automatic Updates Initiative meeting on Jan 9, 2024

hestenet — Tue, 12 Dec 2023 17:53:16 +0000

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

0️⃣ Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

hestenet (he/him)	Tim from the DA
TravisCarden	Travis from Acquia

1️⃣ Met with Cure53 who are doing the Drupal code side of the audit work today, and set up a private Slack-connect channel with them and invited some folks to it. They get started in about a week.

2️⃣ Working with consensus to get the essential root key handling stuff together by end of Jan hopefully (with a 2nd phase for more robust key handling features).Some relevant issues in this thread from yesterday: https://drupal.slack.com/archives/C7QJNEY3E/p1704734820127769

3️⃣ General updates from the Module side?

TravisCarden	Well, if nobody else is here to say it... v3.0.0 has been released. 🙂 (edited)
hestenet (he/him)	Oh yes! :tada:
wimleers (he/him)	Ted is on some much-deserved vacation 🙂
hestenet (he/him)	Excellent - good for him

Participants:

hestenet, TravisCarden, Wim Leers

Meeting link: https://drupal.slack.com/archives/C7QJNEY3E/p1704823476632409

automatic_updates_modules_installed() significantly slows down module install

ressa — Thu, 01 Aug 2024 08:39:35 +0000

Problem/Motivation

Usually, Starshot is installed very quickly. For some reason, the installation now takes much longer, just short of three minutes. It looks like the installation of each modules takes about 7 seconds.

Perhaps there is a setting in Automatic Updates which can be added in Starshot to not check for status, assuming this is the problem?

Originally reported in Installation stalls for a minute, or more #157 in Starshot Github repo.