Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve async runtime scaling #946

Merged
merged 18 commits into from
Aug 23, 2023
Merged

Improve async runtime scaling #946

Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
d5f4aa7
Stub: async scheduler
jaspervdj Aug 3, 2022
912e36b
Stub: async scheduler
jaspervdj Aug 3, 2022
4c86f66
Stub: async scheduler
jaspervdj Aug 3, 2022
ecafce9
Stub: async scheduler
jaspervdj Aug 3, 2022
ef9fcc8
Stub: async scheduler
jaspervdj Aug 3, 2022
7cc5e9d
Stub: async scheduler
jaspervdj Aug 3, 2022
d022b27
Stub: async scheduler
jaspervdj Aug 4, 2022
7f411cb
Stub: async scheduler
jaspervdj Aug 4, 2022
1ea25de
Stub: async scheduler
jaspervdj Aug 4, 2022
d3274cc
Stub: async scheduler
jaspervdj Aug 4, 2022
1f9ba8b
Stub: async scheduler
jaspervdj Aug 4, 2022
866202c
Stub: async scheduler
jaspervdj Aug 4, 2022
04e9f87
Stub: async scheduler
jaspervdj Aug 4, 2022
8266a73
Stub: async scheduler
jaspervdj Aug 4, 2022
0e1dcf2
Stub: async scheduler
jaspervdj Aug 4, 2022
5c70f28
Make logger more testable
jaspervdj Aug 12, 2022
3a54b21
Add better error for cycles and test for this
jaspervdj Aug 12, 2022
8f8f578
Merge branch 'master' into async-scheduler
jaspervdj Aug 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Stub: async scheduler
  • Loading branch information
@jaspervdj
jaspervdj committed Aug 3, 2022
commit 7cc5e9dd11309a599de8fe176f7ad1b60e6f84b0
15 changes: 10 additions & 5 deletions lib/Hakyll/Core/Runtime.hs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,9 @@ schedulerBlock
-> Scheduler
-> (Scheduler, Block)
schedulerBlock identifier deps0 compiler scheduler@Scheduler {..}
| all done deps1 = (scheduler, BlockContinue)
| null deps1 =
trace ("done for identifier " <> show identifier <> ", " <> show deps1 <> ", " <> show deps0) $
(scheduler, BlockContinue)
| otherwise =
( scheduler
{ schedulerQueue =
Expand All @@ -273,7 +275,6 @@ schedulerBlock identifier deps0 compiler scheduler@Scheduler {..}
Map.insertWith Set.union depId (Set.singleton identifier) acc)
schedulerTriggers
deps1

}
, BlockBlocked
)
Expand Down Expand Up @@ -313,7 +314,7 @@ schedulerSnapshot identifier snapshot compiler scheduler@Scheduler {..} =
, schedulerWorking = Set.delete identifier schedulerWorking
, schedulerSnapshots = Set.insert (identifier, snapshot) schedulerSnapshots
, schedulerTodo =
trace ("insert for identifier " <> show identifier) $
trace ("snapshot for identifier " <> show identifier <> " " <> snapshot) $
Map.insert identifier compiler schedulerTodo
}

Expand All @@ -328,7 +329,9 @@ schedulerWrite identifier depFacts scheduler@Scheduler {..} =
schedulerUnblock identifier $ scheduler
{ schedulerWorking = Set.delete identifier schedulerWorking
, schedulerFacts = Map.insert identifier depFacts schedulerFacts
, schedulerDone = Set.insert identifier schedulerDone
, schedulerDone =
trace ("write for identifier " <> show identifier) $
Set.insert identifier schedulerDone
, schedulerTodo =
trace ("delete for identifier " <> show identifier) $
Map.delete identifier schedulerTodo
Expand Down Expand Up @@ -529,14 +532,16 @@ chase2 = do
liftIO $ write path item
Logger.debug logger $ "Routed to " ++ path

Logger.message logger $ "Saved _final for " <> show id'
liftIO $ save store item
threads <- liftIO . IORef.atomicModifyIORef' scheduler $
schedulerWrite id' facts
pure (True, threads)

CompilerRequire reqs c -> do
block <- liftIO . IORef.atomicModifyIORef' scheduler $
schedulerBlock id' reqs c
schedulerBlock id' reqs $ Compiler $
\_ -> pure $ CompilerRequire reqs c
case block of
BlockContinue -> pure (True, 0)
BlockBlocked -> pure (True, 0)
Expand Down