Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci/cron: move concurrency from Bash to Haskell #7696

Merged
merged 1 commit into from
Oct 15, 2020

Conversation

garyverhaegen-da
Copy link
Contributor

One small step further in removing the Bash.

CHANGELOG_BEGIN
CHANGELOG_END

One small step further in removing the Bash.

CHANGELOG_BEGIN
CHANGELOG_END
Copy link
Contributor

@cocreature cocreature left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thank you!

"wait $pid",
"done",
"'"]
Control.Concurrent.Async.forConcurrently_ (map uri $ assets release) (\url -> do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

forConcurrently_ will spawn an unbounded number of threads. I guess we don’t care since the number of assets is small? If we do care, the easiest solution is usually just to throw a semaphore around the map function you pass to forConcurrently_.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Bash code this replaces was also unbounded, so at least this is not getting worse. 🙂

I'm not concerned; we're not going to have a very big number of files anyway. If you are concerned, I can add a semaphore with, say, 30 slots, so it won't change anything for the happy path (16 assets atm) but will shield against future issues. I'd prefer to do that in a future PR, though.

Comment on lines +275 to +278
"set -euo pipefail",
"eval \"$(dev-env/bin/dade assist)\"",
"cd \"" <> tmp <> "\"",
"wget --quiet \"" <> show url <> "\"",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like we could just call wget directly and -o to avoid the cd but I’ll leave that up to you.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ì'm planning to remove wget in my next PR (perhaps next-next now, if the next one is the semaphore), so I don't see much point in trying to make the Bash code nicer. I would point though that we are not in the tmpdir in terms of Haskell process cwd as far as I can tell, so the cd there is not superfluous as it stands.

@garyverhaegen-da
Copy link
Contributor Author

/azp run

@azure-pipelines
Copy link
Contributor

Azure Pipelines successfully started running 1 pipeline(s).

@garyverhaegen-da garyverhaegen-da merged commit 305a097 into master Oct 15, 2020
@garyverhaegen-da garyverhaegen-da deleted the ci-cron-concurrency-in-haskell branch October 15, 2020 12:00
garyverhaegen-da added a commit that referenced this pull request Oct 15, 2020
As requested in review of #7696.

CHANGELOG_BEGIN
CHANGELOG_END
garyverhaegen-da added a commit that referenced this pull request Oct 16, 2020
As requested in review of #7696.

CHANGELOG_BEGIN
CHANGELOG_END
garyverhaegen-da added a commit that referenced this pull request Oct 19, 2020
As promised in #7696.

CHANGELOG_BEGIN
CHANGELOG_END
garyverhaegen-da added a commit that referenced this pull request Oct 19, 2020
As promised in #7696.

CHANGELOG_BEGIN
CHANGELOG_END
garyverhaegen-da added a commit that referenced this pull request Oct 19, 2020
As promised in #7696.

CHANGELOG_BEGIN
CHANGELOG_END
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants