-
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add push! implementation for AbstractArray depending only on resize! #55470
Add push! implementation for AbstractArray depending only on resize! #55470
Conversation
# multi-item push!, pushfirst! (built on top of type-specific 1-item version) | ||
# (note: must not cause a dispatch loop when 1-item case is not defined) | ||
push!(A, a, b) = push!(push!(A, a), b) | ||
push!(A, a, b, c...) = push!(push!(A, a, b), c...) | ||
pushfirst!(A, a, b) = pushfirst!(pushfirst!(A, b), a) | ||
pushfirst!(A, a, b, c...) = pushfirst!(pushfirst!(A, c...), a, b) | ||
|
||
# sizehint! does not nothing by default | ||
sizehint!(a::AbstractVector, _) = a |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unrelated change? Also no tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#51903 introduced a dependency on sizehint!
for append!
that did not previously exist.
I will add tests for append.
itemT = item isa T ? item : convert(T, item)::T | ||
new_length = length(a) + 1 | ||
resize!(a, new_length) | ||
a[new_length] = itemT |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
julia> o = OffsetVector([1, 2, 3], -1);
julia> @invoke push!(o::AbstractVector, 4::Any)
ERROR: BoundsError: attempt to access 4-element OffsetArray(::Vector{Int64}, 0:3) with eltype Int64 with indices 0:3 at index [4]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this re-establishes the state before #51903 but I agree this is an issue.
Would end
work better here or should we require or dispatch on one-based indexing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
on the specific case of OffsetArrays
, resize!
works on the the parent array (so length(a) + 1
is fine)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure I understand your point.
…55470) Fix #55459 In Julia 1.10, `push!` and `append!` would be functional for `AbstractVector` implementations if `resize!` and `setindex!` were defined. As of #51903 by @vtjnash as in Julia 1.11.0-rc2, `append!` now depends on an implementation of `sizehint!` and `push!`. Since `push!` also depends on `append!`, a stack overflow situation can easily be created. To avoid this, this pull request defines the following * Add generic versions of `push!(a::AbstractVector, x)` which do not depend on `append!` * Add default implementation of `sizehint!` that is a no-op The implementation of `push!(a::AbstractVector, x)` is a generic version based on the implementation of `push!(a::Vector, x)` without depending on internals. # Example for SimpleArray Consider the `SimpleArray` example from test/abstractarray.jl: ```julia mutable struct SimpleArray{T} <: AbstractVector{T} els::Vector{T} end Base.size(sa::SimpleArray) = size(sa.els) Base.getindex(sa::SimpleArray, idx...) = getindex(sa.els, idx...) Base.setindex!(sa::SimpleArray, v, idx...) = setindex!(sa.els, v, idx...) Base.resize!(sa::SimpleArray, n) = resize!(sa.els, n) Base.copy(sa::SimpleArray) = SimpleArray(copy(sa.els)) ``` Note that `setindex!` and `resize!` are implemented for `SimpleArray`. ## Julia 1.10.4: push! is functional On Julia 1.10.4, `push!` has a functional implementation for `SimpleArray` ```julia-repl julia> push!(SimpleArray{Int}(zeros(Int,5)), 6) 6-element SimpleArray{Int64}: 0 0 0 0 0 6 ``` ## Julia 1.11.0-rc2 and nightly: push! requires sizehint! and is prone to stack overflow Before this pull request, on Julia 1.11.0-rc2 and nightly, `push!` fails for want of `sizehint!`. ```julia-repl julia> push!(SimpleArray{Int}(zeros(Int,5)), 6) ERROR: MethodError: no method matching sizehint!(::SimpleArray{Int64}, ::Int64) The function `sizehint!` exists, but no method is defined for this combination of argument types. ... ``` After implementing `sizehint!`, `push!` still fails with a stack overflow. ```julia-repl julia> Base.sizehint!(a::SimpleArray, x) = a julia> push!(SimpleArray{Int}(zeros(Int, 5)), 6) Warning: detected a stack overflow; program state may be corrupted, so further execution might be unreliable. ERROR: StackOverflowError: Stacktrace: [1] _append! @ ./array.jl:1344 [inlined] [2] append! @ ./array.jl:1335 [inlined] [3] push!(a::SimpleArray{Int64}, iter::Int64) @ Base ./array.jl:1336 --- the above 3 lines are repeated 79982 more times --- [239950] _append! @ ./array.jl:1344 [inlined] [239951] append! @ ./array.jl:1335 [inlined] ``` This is because the new implementation of `append!` depends on `push!`. ## After this pull request, push! is functional. After this pull request, there is a functional `push!` for `SimpleArray` again as in Julia 1.10.4: ```julia-repl julia> push!(SimpleArray{Int}(zeros(Int, 5), 6) 6-element SimpleArray{Int64}: 0 0 0 0 0 6 ``` (cherry picked from commit cf4c30a)
…actVector (#55480) Per #55470 (comment), the `push!(::AbstractArray, ...)` array implementation assumed one-based indexing and did not account for an `OffsetVector` scenario. Here we add tests for `push!(::AbstractArray, ...)` and `append(::AbstractArray, ...)` including using `@invoke` to test the effect on `OffsetVector`. cc: @fredrikekre
…uliaLang#55470) Fix JuliaLang#55459 In Julia 1.10, `push!` and `append!` would be functional for `AbstractVector` implementations if `resize!` and `setindex!` were defined. As of JuliaLang#51903 by @vtjnash as in Julia 1.11.0-rc2, `append!` now depends on an implementation of `sizehint!` and `push!`. Since `push!` also depends on `append!`, a stack overflow situation can easily be created. To avoid this, this pull request defines the following * Add generic versions of `push!(a::AbstractVector, x)` which do not depend on `append!` * Add default implementation of `sizehint!` that is a no-op The implementation of `push!(a::AbstractVector, x)` is a generic version based on the implementation of `push!(a::Vector, x)` without depending on internals. # Example for SimpleArray Consider the `SimpleArray` example from test/abstractarray.jl: ```julia mutable struct SimpleArray{T} <: AbstractVector{T} els::Vector{T} end Base.size(sa::SimpleArray) = size(sa.els) Base.getindex(sa::SimpleArray, idx...) = getindex(sa.els, idx...) Base.setindex!(sa::SimpleArray, v, idx...) = setindex!(sa.els, v, idx...) Base.resize!(sa::SimpleArray, n) = resize!(sa.els, n) Base.copy(sa::SimpleArray) = SimpleArray(copy(sa.els)) ``` Note that `setindex!` and `resize!` are implemented for `SimpleArray`. ## Julia 1.10.4: push! is functional On Julia 1.10.4, `push!` has a functional implementation for `SimpleArray` ```julia-repl julia> push!(SimpleArray{Int}(zeros(Int,5)), 6) 6-element SimpleArray{Int64}: 0 0 0 0 0 6 ``` ## Julia 1.11.0-rc2 and nightly: push! requires sizehint! and is prone to stack overflow Before this pull request, on Julia 1.11.0-rc2 and nightly, `push!` fails for want of `sizehint!`. ```julia-repl julia> push!(SimpleArray{Int}(zeros(Int,5)), 6) ERROR: MethodError: no method matching sizehint!(::SimpleArray{Int64}, ::Int64) The function `sizehint!` exists, but no method is defined for this combination of argument types. ... ``` After implementing `sizehint!`, `push!` still fails with a stack overflow. ```julia-repl julia> Base.sizehint!(a::SimpleArray, x) = a julia> push!(SimpleArray{Int}(zeros(Int, 5)), 6) Warning: detected a stack overflow; program state may be corrupted, so further execution might be unreliable. ERROR: StackOverflowError: Stacktrace: [1] _append! @ ./array.jl:1344 [inlined] [2] append! @ ./array.jl:1335 [inlined] [3] push!(a::SimpleArray{Int64}, iter::Int64) @ Base ./array.jl:1336 --- the above 3 lines are repeated 79982 more times --- [239950] _append! @ ./array.jl:1344 [inlined] [239951] append! @ ./array.jl:1335 [inlined] ``` This is because the new implementation of `append!` depends on `push!`. ## After this pull request, push! is functional. After this pull request, there is a functional `push!` for `SimpleArray` again as in Julia 1.10.4: ```julia-repl julia> push!(SimpleArray{Int}(zeros(Int, 5), 6) 6-element SimpleArray{Int64}: 0 0 0 0 0 6 ```
…actVector (JuliaLang#55480) Per JuliaLang#55470 (comment), the `push!(::AbstractArray, ...)` array implementation assumed one-based indexing and did not account for an `OffsetVector` scenario. Here we add tests for `push!(::AbstractArray, ...)` and `append(::AbstractArray, ...)` including using `@invoke` to test the effect on `OffsetVector`. cc: @fredrikekre
Are these related to this PR? |
The second one with
|
Backported PRs: - [x] #54962 <!-- Add timing to precompile trace compile --> - [x] #55180 <!-- compress jit debuginfo for easy memory savings --> - [x] #54919 <!-- Fix annotated join with non-concrete eltype iters --> - [x] #55013 <!-- [docs] change docstring to match code --> - [x] #55017 <!-- TOML: Make `Dates` a type parameter --> - [x] #54033 <!-- Fix a bug in `stack`'s DimensionMismatch error message --> - [x] #55242 <!-- fix at-main docstring to not code quote a compat box --> - [x] #55261 <!-- Make `jl_*affinity` tests more portable --> - [x] #54736 <!-- specificity: ensure fast-path in `sub/eq_msp` handle missing `UnionAll` wrapper correctly. --> - [x] #55299 <!-- typeintersect: fix bounds merging during inner `intersect_all`. --> - [x] #55302 <!-- Add `lbt_forwarded_funcs()` to debug LBT forwarding issues --> - [x] #55148 <!-- Random: Mark unexported public symbols as public --> - [x] #55303 <!-- avoid overflowing show for OffsetArrays around typemax --> - [x] #55317 <!-- Restrict argument to `isleapyear(::Integer)` --> - [x] #55327 <!-- Profile: Fix stdlib paths --> - [x] #55330 <!-- [libblastrampoline] Bump to v5.11.0 --> - [x] #55310 <!-- Preserve structure in scaling triangular matrices by NaN --> - [x] #55329 <!-- mapreduce: don't inbounds unknown functions --> - [x] #55356 <!-- Profile: close files when assembling heap snapshot --> - [x] #55371 <!-- Fix tr for block SymTridiagonal --> - [x] #55307 <!-- Make REPL.TerminalMenus public --> - [x] #55362 <!-- inference: fix missing LimitedAccuracy markers --> - [x] #55306 <!-- AllocOpt: Fix stack lowering where alloca continas boxed and unboxed data --> - [x] #55395 <!-- fix #55389: type-unstable `join` --> - [x] #55226 <!-- re-add `unsafe_convert` for Reinterpret and Reshaped array --> - [x] #55405 <!-- handle unbound vars in NTuple fields --> - [x] #55365 <!-- ml-matches: ensure all methods are included --> - [x] #55428 <!-- codegen: move undef freeze before promotion point --> - [x] #55419 <!-- `stale_cachefile`: handle if the expected cache file is missing --> - [x] #55470 <!-- Add push! implementation for AbstractArray depending only on resize! --> - [x] #55483 <!-- fix hierarchy level of "API reference" in `Dates` documentation --> - [x] #55268 <!-- simplify complex atanh and remove singularity perturbation --> - [x] #55441 <!-- fix Event to use normal Condition variable --> - [x] #55413 <!-- subtyping: fast path for lhs union and rhs typevar --> - [x] #55492 <!-- build: add missing dependencies for expmap --> - [x] #55507 <!-- Fix fast getptls ccall lowering. --> - [x] #55424 <!-- add missing clamp function for IOBuffer --> - [x] #55504 <!-- Update symmetric docstring to reflect the type of uplo --> - [x] #55107 <!-- Make the memory GEP an inbounds GEP since the bounds check has happened somewhere else --> - [x] #55411 <!-- Vendor the terminfo database for use with base/terminfo.jl --> - [x] #55452 <!-- Do not load `ScopedValues` with `using` --> - [x] #55407 <!-- Remove deprecated non string API for LLVM pass pipeline and parse all options --> - [x] #55461 <!-- 🤖 [master] Bump the StyledStrings stdlib from d7496d2 to f6035eb --> - [x] #55433 <!-- Backport #55407 to 1.11 --> - [x] #55225 <!-- [1.11 backport] trace-compile: don't generate `precompile` statements for OpaqueClosure methods (#55072) --> - [x] #55212 <!-- Make `Base.depwarn()` public --> - [x] #552 - [x] #55052 <!-- Fix `(l/r)mul!` with `Diagonal`/`Bidiagonal` --> - [x] #55251 <!-- Restrict binary ops for Diagonal and Symmetric to Number eltypes -->95 <!-- LAPACK: Aggressive constprop to concretely infer syev!/syevd! --> - [x] #55522 <!-- Fix tr for Symmetric/Hermitian block matrices --> Need manual backport: - [x] #55342 <!-- Ensure bidiagonal setindex! does not read indices in error message --> Contains multiple commits, manual intervention needed: - [ ] #55336 <!-- codegen: take gc roots (and alloca alignment) more seriously --> Non-merged PRs with backport label: - [ ] #55506 <!-- Fix indexing in _mapreducedim for OffsetArrays --> - [ ] #55500 <!-- make jl_thread_suspend_and_get_state safe --> - [ ] #55499 <!-- propagate the terminal's `displaysize` to the `IOContext` used by the REPL --> - [ ] #55458 <!-- Allow for generically extracting unannotated string --> - [ ] #55457 <!-- Make AnnotateChar equality consider annotations --> - [ ] #55453 <!-- Privatise the annotations API, for StyledStrings --> - [ ] #55443 <!-- Add test for upper/lower/titlecase and fix call --> - [ ] #55355 <!-- relocation: account for trailing path separator in depot paths --> - [ ] #55220 <!-- `isfile_casesensitive` fixes on Windows --> - [ ] #55169 <!-- `propertynames` for SVD respects private argument --> - [ ] #54457 <!-- Make `String(::Memory)` copy --> - [ ] #53957 <!-- tweak how filtering is done for what packages should be precompiled --> - [ ] #51479 <!-- prevent code loading from lookin in the versioned environment when building Julia --> - [ ] #50813 <!-- More doctests for Sockets and capitalization fix --> - [ ] #50157 <!-- improve docs for `@inbounds` and `Base.@propagate_inbounds` --> - [ ] #41244 <!-- Fix shell `cd` error when working dir has been deleted -->
…actVector (#55480) Per #55470 (comment), the `push!(::AbstractArray, ...)` array implementation assumed one-based indexing and did not account for an `OffsetVector` scenario. Here we add tests for `push!(::AbstractArray, ...)` and `append(::AbstractArray, ...)` including using `@invoke` to test the effect on `OffsetVector`. cc: @fredrikekre (cherry picked from commit 5230d27)
…actVector (#55480) Per #55470 (comment), the `push!(::AbstractArray, ...)` array implementation assumed one-based indexing and did not account for an `OffsetVector` scenario. Here we add tests for `push!(::AbstractArray, ...)` and `append(::AbstractArray, ...)` including using `@invoke` to test the effect on `OffsetVector`. cc: @fredrikekre
Fix #55459
In Julia 1.10,
push!
andappend!
would be functional forAbstractVector
implementationsif
resize!
andsetindex!
were defined.As of #51903 by @vtjnash as in Julia 1.11.0-rc2,
append!
now depends on an implementation ofsizehint!
andpush!
. Sincepush!
also depends onappend!
, a stackoverflow situation can easily be created.
To avoid this, this pull request defines the following
push!(a::AbstractVector, x)
which do not depend onappend!
sizehint!
that is a no-opThe implementation of
push!(a::AbstractVector, x)
is a generic version based on the implementationof
push!(a::Vector, x)
without depending on internals.Example for SimpleArray
Consider the
SimpleArray
example from test/abstractarray.jl:Note that
setindex!
andresize!
are implemented forSimpleArray
.Julia 1.10.4: push! is functional
On Julia 1.10.4,
push!
has a functional implementation forSimpleArray
Julia 1.11.0-rc2 and nightly: push! requires sizehint! and is prone to stack overflow
Before this pull request, on Julia 1.11.0-rc2 and nightly,
push!
fails for want ofsizehint!
.After implementing
sizehint!
,push!
still fails with a stack overflow.This is because the new implementation of
append!
depends onpush!
.After this pull request, push! is functional.
After this pull request, there is a functional
push!
forSimpleArray
again as in Julia 1.10.4: