Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add push! implementation for AbstractArray depending only on resize! #55470

Merged
merged 1 commit into from
Aug 12, 2024

Conversation

mkitti
Copy link
Contributor

@mkitti mkitti commented Aug 12, 2024

Fix #55459

In Julia 1.10, push! and append! would be functional for AbstractVector implementations
if resize! and setindex! were defined.

As of #51903 by @vtjnash as in Julia 1.11.0-rc2, append! now depends on an implementation of
sizehint! and push!. Since push! also depends on append!, a stack
overflow situation can easily be created.

To avoid this, this pull request defines the following

  • Add generic versions of push!(a::AbstractVector, x) which do not depend on append!
  • Add default implementation of sizehint! that is a no-op

The implementation of push!(a::AbstractVector, x) is a generic version based on the implementation
of push!(a::Vector, x) without depending on internals.

Example for SimpleArray

Consider the SimpleArray example from test/abstractarray.jl:

mutable struct SimpleArray{T} <: AbstractVector{T}
    els::Vector{T}
end
Base.size(sa::SimpleArray) = size(sa.els)
Base.getindex(sa::SimpleArray, idx...) = getindex(sa.els, idx...)
Base.setindex!(sa::SimpleArray, v, idx...) = setindex!(sa.els, v, idx...)
Base.resize!(sa::SimpleArray, n) = resize!(sa.els, n)
Base.copy(sa::SimpleArray) = SimpleArray(copy(sa.els))

Note that setindex! and resize! are implemented for SimpleArray.

Julia 1.10.4: push! is functional

On Julia 1.10.4, push! has a functional implementation for SimpleArray

julia> push!(SimpleArray{Int}(zeros(Int,5)), 6)
6-element SimpleArray{Int64}:
 0
 0
 0
 0
 0
 6

Julia 1.11.0-rc2 and nightly: push! requires sizehint! and is prone to stack overflow

Before this pull request, on Julia 1.11.0-rc2 and nightly, push! fails for want of sizehint!.

julia> push!(SimpleArray{Int}(zeros(Int,5)), 6)
ERROR: MethodError: no method matching sizehint!(::SimpleArray{Int64}, ::Int64)
The function `sizehint!` exists, but no method is defined for this combination of argument types.
...

After implementing sizehint!, push! still fails with a stack overflow.

julia> Base.sizehint!(a::SimpleArray, x) = a

julia> push!(SimpleArray{Int}(zeros(Int, 5)), 6)
Warning: detected a stack overflow; program state may be corrupted, so further execution might be unreliable.
ERROR: StackOverflowError:
Stacktrace:
      [1] _append!
        @ ./array.jl:1344 [inlined]
      [2] append!
        @ ./array.jl:1335 [inlined]
      [3] push!(a::SimpleArray{Int64}, iter::Int64)
        @ Base ./array.jl:1336
--- the above 3 lines are repeated 79982 more times ---
 [239950] _append!
        @ ./array.jl:1344 [inlined]
 [239951] append!
        @ ./array.jl:1335 [inlined]

This is because the new implementation of append! depends on push!.

After this pull request, push! is functional.

After this pull request, there is a functional push! for SimpleArray again as in Julia 1.10.4:

julia> push!(SimpleArray{Int}(zeros(Int, 5), 6)
6-element SimpleArray{Int64}:
 0
 0
 0
 0
 0
 6

@nsajko nsajko added arrays [a, r, r, a, y, s] backport 1.11 Change should be backported to release-1.11 labels Aug 12, 2024
@vtjnash vtjnash added the merge me PR is reviewed. Merge when all tests are passing label Aug 12, 2024
@oscardssmith oscardssmith merged commit cf4c30a into JuliaLang:master Aug 12, 2024
10 checks passed
@oscardssmith oscardssmith removed the merge me PR is reviewed. Merge when all tests are passing label Aug 12, 2024
# multi-item push!, pushfirst! (built on top of type-specific 1-item version)
# (note: must not cause a dispatch loop when 1-item case is not defined)
push!(A, a, b) = push!(push!(A, a), b)
push!(A, a, b, c...) = push!(push!(A, a, b), c...)
pushfirst!(A, a, b) = pushfirst!(pushfirst!(A, b), a)
pushfirst!(A, a, b, c...) = pushfirst!(pushfirst!(A, c...), a, b)

# sizehint! does not nothing by default
sizehint!(a::AbstractVector, _) = a
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated change? Also no tests.

Copy link
Contributor Author

@mkitti mkitti Aug 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#51903 introduced a dependency on sizehint! for append! that did not previously exist.

I will add tests for append.

itemT = item isa T ? item : convert(T, item)::T
new_length = length(a) + 1
resize!(a, new_length)
a[new_length] = itemT
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

julia> o = OffsetVector([1, 2, 3], -1);

julia> @invoke push!(o::AbstractVector, 4::Any)
ERROR: BoundsError: attempt to access 4-element OffsetArray(::Vector{Int64}, 0:3) with eltype Int64 with indices 0:3 at index [4]

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this re-establishes the state before #51903 but I agree this is an issue.

Would end work better here or should we require or dispatch on one-based indexing?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on the specific case of OffsetArrays, resize! works on the the parent array (so length(a) + 1 is fine)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand your point.

KristofferC pushed a commit that referenced this pull request Aug 13, 2024
…55470)

Fix #55459

In Julia 1.10, `push!` and `append!` would be functional for
`AbstractVector` implementations
if `resize!` and `setindex!` were defined.

As of #51903 by @vtjnash as in Julia 1.11.0-rc2, `append!` now depends
on an implementation of
`sizehint!` and `push!`. Since `push!` also depends on `append!`, a
stack
overflow situation can easily be created.

To avoid this, this pull request defines the following

* Add generic versions of `push!(a::AbstractVector, x)` which do not
depend on `append!`
* Add default implementation of `sizehint!` that is a no-op

The implementation of `push!(a::AbstractVector, x)` is a generic version
based on the implementation
of `push!(a::Vector, x)` without depending on internals.

# Example for SimpleArray

Consider the `SimpleArray` example from test/abstractarray.jl:

```julia
mutable struct SimpleArray{T} <: AbstractVector{T}
    els::Vector{T}
end
Base.size(sa::SimpleArray) = size(sa.els)
Base.getindex(sa::SimpleArray, idx...) = getindex(sa.els, idx...)
Base.setindex!(sa::SimpleArray, v, idx...) = setindex!(sa.els, v, idx...)
Base.resize!(sa::SimpleArray, n) = resize!(sa.els, n)
Base.copy(sa::SimpleArray) = SimpleArray(copy(sa.els))
```

Note that `setindex!` and `resize!` are implemented for `SimpleArray`.

## Julia 1.10.4: push! is functional

On Julia 1.10.4, `push!` has a functional implementation for
`SimpleArray`

```julia-repl
julia> push!(SimpleArray{Int}(zeros(Int,5)), 6)
6-element SimpleArray{Int64}:
 0
 0
 0
 0
 0
 6
```

## Julia 1.11.0-rc2 and nightly: push! requires sizehint! and is prone
to stack overflow

Before this pull request, on Julia 1.11.0-rc2 and nightly, `push!` fails
for want of `sizehint!`.

```julia-repl
julia> push!(SimpleArray{Int}(zeros(Int,5)), 6)
ERROR: MethodError: no method matching sizehint!(::SimpleArray{Int64}, ::Int64)
The function `sizehint!` exists, but no method is defined for this combination of argument types.
...
```

After implementing `sizehint!`, `push!` still fails with a stack
overflow.

```julia-repl
julia> Base.sizehint!(a::SimpleArray, x) = a

julia> push!(SimpleArray{Int}(zeros(Int, 5)), 6)
Warning: detected a stack overflow; program state may be corrupted, so further execution might be unreliable.
ERROR: StackOverflowError:
Stacktrace:
      [1] _append!
        @ ./array.jl:1344 [inlined]
      [2] append!
        @ ./array.jl:1335 [inlined]
      [3] push!(a::SimpleArray{Int64}, iter::Int64)
        @ Base ./array.jl:1336
--- the above 3 lines are repeated 79982 more times ---
 [239950] _append!
        @ ./array.jl:1344 [inlined]
 [239951] append!
        @ ./array.jl:1335 [inlined]
```

This is because the new implementation of `append!` depends on `push!`.

## After this pull request, push! is functional.

After this pull request, there is a functional `push!` for `SimpleArray`
again as in Julia 1.10.4:

```julia-repl
julia> push!(SimpleArray{Int}(zeros(Int, 5), 6)
6-element SimpleArray{Int64}:
 0
 0
 0
 0
 0
 6
```

(cherry picked from commit cf4c30a)
@KristofferC KristofferC mentioned this pull request Aug 13, 2024
68 tasks
jishnub pushed a commit that referenced this pull request Aug 16, 2024
…actVector (#55480)

Per
#55470 (comment),
the `push!(::AbstractArray, ...)` array implementation assumed one-based
indexing and did not account for an `OffsetVector`
scenario.

Here we add tests for `push!(::AbstractArray, ...)` and
`append(::AbstractArray, ...)` including using `@invoke` to test the
effect on `OffsetVector`.

cc: @fredrikekre
lazarusA pushed a commit to lazarusA/julia that referenced this pull request Aug 17, 2024
…uliaLang#55470)

Fix JuliaLang#55459

In Julia 1.10, `push!` and `append!` would be functional for
`AbstractVector` implementations
if `resize!` and `setindex!` were defined.

As of JuliaLang#51903 by @vtjnash as in Julia 1.11.0-rc2, `append!` now depends
on an implementation of
`sizehint!` and `push!`. Since `push!` also depends on `append!`, a
stack
overflow situation can easily be created.

To avoid this, this pull request defines the following

* Add generic versions of `push!(a::AbstractVector, x)` which do not
depend on `append!`
* Add default implementation of `sizehint!` that is a no-op

The implementation of `push!(a::AbstractVector, x)` is a generic version
based on the implementation
of `push!(a::Vector, x)` without depending on internals.

# Example for SimpleArray

Consider the `SimpleArray` example from test/abstractarray.jl:

```julia
mutable struct SimpleArray{T} <: AbstractVector{T}
    els::Vector{T}
end
Base.size(sa::SimpleArray) = size(sa.els)
Base.getindex(sa::SimpleArray, idx...) = getindex(sa.els, idx...)
Base.setindex!(sa::SimpleArray, v, idx...) = setindex!(sa.els, v, idx...)
Base.resize!(sa::SimpleArray, n) = resize!(sa.els, n)
Base.copy(sa::SimpleArray) = SimpleArray(copy(sa.els))
```

Note that `setindex!` and `resize!` are implemented for `SimpleArray`.

## Julia 1.10.4: push! is functional

On Julia 1.10.4, `push!` has a functional implementation for
`SimpleArray`

```julia-repl
julia> push!(SimpleArray{Int}(zeros(Int,5)), 6)
6-element SimpleArray{Int64}:
 0
 0
 0
 0
 0
 6
```

## Julia 1.11.0-rc2 and nightly: push! requires sizehint! and is prone
to stack overflow

Before this pull request, on Julia 1.11.0-rc2 and nightly, `push!` fails
for want of `sizehint!`.

```julia-repl
julia> push!(SimpleArray{Int}(zeros(Int,5)), 6)
ERROR: MethodError: no method matching sizehint!(::SimpleArray{Int64}, ::Int64)
The function `sizehint!` exists, but no method is defined for this combination of argument types.
...
```

After implementing `sizehint!`, `push!` still fails with a stack
overflow.

```julia-repl
julia> Base.sizehint!(a::SimpleArray, x) = a

julia> push!(SimpleArray{Int}(zeros(Int, 5)), 6)
Warning: detected a stack overflow; program state may be corrupted, so further execution might be unreliable.
ERROR: StackOverflowError:
Stacktrace:
      [1] _append!
        @ ./array.jl:1344 [inlined]
      [2] append!
        @ ./array.jl:1335 [inlined]
      [3] push!(a::SimpleArray{Int64}, iter::Int64)
        @ Base ./array.jl:1336
--- the above 3 lines are repeated 79982 more times ---
 [239950] _append!
        @ ./array.jl:1344 [inlined]
 [239951] append!
        @ ./array.jl:1335 [inlined]
```

This is because the new implementation of `append!` depends on `push!`.

## After this pull request, push! is functional.

After this pull request, there is a functional `push!` for `SimpleArray`
again as in Julia 1.10.4:

```julia-repl
julia> push!(SimpleArray{Int}(zeros(Int, 5), 6)
6-element SimpleArray{Int64}:
 0
 0
 0
 0
 0
 6
```
lazarusA pushed a commit to lazarusA/julia that referenced this pull request Aug 17, 2024
…actVector (JuliaLang#55480)

Per
JuliaLang#55470 (comment),
the `push!(::AbstractArray, ...)` array implementation assumed one-based
indexing and did not account for an `OffsetVector`
scenario.

Here we add tests for `push!(::AbstractArray, ...)` and
`append(::AbstractArray, ...)` including using `@invoke` to test the
effect on `OffsetVector`.

cc: @fredrikekre
@KristofferC
Copy link
Member

https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_hash/13d440d_vs_34c3a63/FlexiMaps.primary.log

ERROR: LoadError: MethodError: no method matching resize!(::FlexiMaps.MappedArray{Int64, 1, Base.Fix2{typeof(+), Int64}, Vector{Int64}}, ::Int64)
The function `resize!` exists, but no method is defined for this combination of argument types.

Closest candidates are:
  resize!(!Matched::JLArrays.JLArray{T, 1}, ::Integer) where T
   @ JLArrays ~/.julia/packages/JLArrays/hD5YX/src/JLArrays.jl:375
  resize!(!Matched::StructArrays.StructArray, ::Integer)
   @ StructArrays ~/.julia/packages/StructArrays/CjQ4L/src/structarray.jl:429
  resize!(!Matched::OffsetArrays.OffsetVector{T} where T, ::Integer)
   @ OffsetArrays ~/.julia/packages/OffsetArrays/hwmnB/src/OffsetArrays.jl:606
  ...

Stacktrace:
  [1] push!(a::FlexiMaps.MappedArray{Int64, 1, Base.Fix2{typeof(+), Int64}, Vector{Int64}}, item::Int64)
    @ Base ./abstractarray.jl:3506
  [2] top-level scope
    @ ~/.julia/packages/FlexiMaps/n1CYU/test/runtests.jl:226
  [3] eval
    @ ./boot.jl:430 [inlined]

https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_hash/13d440d_vs_34c3a63/ZippedArrays.primary.log

zipped arrays: Test Failed at /home/pkgeval/.julia/packages/ZippedArrays/p7ZPn/test/runtests.jl:311
  Expression: sizehint!(ZippedVector(a, b, c, 1:n), n + 50)
    Expected: Exception
  No exception thrown

Are these related to this PR?

@mkitti
Copy link
Contributor Author

mkitti commented Aug 22, 2024

FlexiMaps defines Base.append!(A::MappedArray, iter). push!(::MappedArray, ...) used to forward to append!(::MappedArray, ...). After this PR, push! no longer forwards to append!. Perhaps FlexiMaps should implement push! directly?

The second one with ZippedVector used to throw an exception to the MethodError below for UnitRange. I'm not sure if this should be fixed or not. This pull request provided a default implementation which just ignores the hint.

sizehint!(ZippedVector(a,b,c,1:n), n + 50)
ERROR: MethodError: no method matching sizehint!(::UnitRange{Int64}, ::Int64)

KristofferC added a commit that referenced this pull request Aug 26, 2024
Backported PRs:
- [x] #54962 <!-- Add timing to precompile trace compile -->
- [x] #55180 <!-- compress jit debuginfo for easy memory savings -->
- [x] #54919 <!-- Fix annotated join with non-concrete eltype iters -->
- [x] #55013 <!-- [docs] change docstring to match code -->
- [x] #55017 <!-- TOML: Make `Dates` a type parameter -->
- [x] #54033 <!-- Fix a bug in `stack`'s DimensionMismatch error message
-->
- [x] #55242 <!-- fix at-main docstring to not code quote a compat box
-->
- [x] #55261 <!-- Make `jl_*affinity` tests more portable -->
- [x] #54736 <!-- specificity: ensure fast-path in `sub/eq_msp` handle
missing `UnionAll` wrapper correctly. -->
- [x] #55299 <!-- typeintersect: fix bounds merging during inner
`intersect_all`. -->
- [x] #55302 <!-- Add `lbt_forwarded_funcs()` to debug LBT forwarding
issues -->
- [x] #55148 <!-- Random: Mark unexported public symbols as public -->
- [x] #55303 <!-- avoid overflowing show for OffsetArrays around typemax
-->
- [x] #55317 <!-- Restrict argument to `isleapyear(::Integer)` -->
- [x] #55327 <!-- Profile: Fix stdlib paths -->
- [x] #55330 <!-- [libblastrampoline] Bump to v5.11.0 -->
- [x] #55310 <!-- Preserve structure in scaling triangular matrices by
NaN -->
- [x] #55329 <!-- mapreduce: don't inbounds unknown functions -->
- [x] #55356 <!-- Profile: close files when assembling heap snapshot -->
- [x] #55371 <!-- Fix tr for block SymTridiagonal -->
- [x] #55307 <!-- Make REPL.TerminalMenus public -->
- [x] #55362 <!-- inference: fix missing LimitedAccuracy markers -->
- [x] #55306 <!-- AllocOpt: Fix stack lowering where alloca continas
boxed and unboxed data -->
- [x] #55395 <!-- fix #55389: type-unstable `join` -->
- [x] #55226 <!-- re-add `unsafe_convert` for Reinterpret and Reshaped
array -->
- [x] #55405 <!-- handle unbound vars in NTuple fields -->
- [x] #55365 <!-- ml-matches: ensure all methods are included -->
- [x] #55428 <!-- codegen: move undef freeze before promotion point -->
- [x] #55419 <!-- `stale_cachefile`: handle if the expected cache file
is missing -->
- [x] #55470 <!-- Add push! implementation for AbstractArray depending
only on resize! -->
- [x] #55483 <!-- fix hierarchy level of "API reference" in `Dates`
documentation -->
- [x] #55268 <!-- simplify complex atanh and remove singularity
perturbation -->
- [x] #55441 <!-- fix Event to use normal Condition variable -->
- [x] #55413 <!-- subtyping: fast path for lhs union and rhs typevar -->
- [x] #55492 <!-- build: add missing dependencies for expmap -->
- [x] #55507 <!-- Fix fast getptls ccall lowering. -->
- [x] #55424 <!-- add missing clamp function for IOBuffer -->
- [x] #55504 <!-- Update symmetric docstring to reflect the type of uplo
-->
- [x] #55107 <!-- Make the memory GEP an inbounds GEP since the bounds
check has happened somewhere else -->
- [x] #55411 <!-- Vendor the terminfo database for use with
base/terminfo.jl -->
- [x] #55452 <!-- Do not load `ScopedValues` with `using` -->
- [x] #55407 <!-- Remove deprecated non string API for LLVM pass
pipeline and parse all options -->
- [x] #55461 <!-- 🤖 [master] Bump the StyledStrings stdlib from d7496d2
to f6035eb -->
- [x] #55433 <!-- Backport #55407
to 1.11 -->
- [x] #55225 <!-- [1.11 backport] trace-compile: don't generate
`precompile` statements for OpaqueClosure methods (#55072) -->
- [x] #55212 <!-- Make `Base.depwarn()` public -->
- [x] #552
- [x] #55052 <!-- Fix `(l/r)mul!` with `Diagonal`/`Bidiagonal` -->
- [x] #55251 <!-- Restrict binary ops for Diagonal and Symmetric to
Number eltypes -->95 <!-- LAPACK: Aggressive constprop to concretely
infer syev!/syevd! -->
- [x] #55522 <!-- Fix tr for Symmetric/Hermitian block matrices -->

Need manual backport:
- [x] #55342 <!-- Ensure bidiagonal setindex! does not read indices in
error message -->

Contains multiple commits, manual intervention needed:

- [ ] #55336 <!-- codegen: take gc roots (and alloca alignment) more
seriously -->


Non-merged PRs with backport label:
- [ ] #55506 <!-- Fix indexing in _mapreducedim for OffsetArrays -->
- [ ] #55500 <!-- make jl_thread_suspend_and_get_state safe -->
- [ ] #55499 <!-- propagate the terminal's `displaysize` to the
`IOContext` used by the REPL -->
- [ ] #55458 <!-- Allow for generically extracting unannotated string
-->
- [ ] #55457 <!-- Make AnnotateChar equality consider annotations -->
- [ ] #55453 <!-- Privatise the annotations API, for StyledStrings -->
- [ ] #55443 <!-- Add test for upper/lower/titlecase and fix call -->
- [ ] #55355 <!-- relocation: account for trailing path separator in
depot paths -->
- [ ] #55220 <!-- `isfile_casesensitive` fixes on Windows -->
- [ ] #55169 <!-- `propertynames` for SVD respects private argument -->
- [ ] #54457 <!-- Make `String(::Memory)` copy -->
- [ ] #53957 <!-- tweak how filtering is done for what packages should
be precompiled -->
- [ ] #51479 <!-- prevent code loading from lookin in the versioned
environment when building Julia -->
- [ ] #50813 <!-- More doctests for Sockets and capitalization fix -->
- [ ] #50157 <!-- improve docs for `@inbounds` and
`Base.@propagate_inbounds` -->
- [ ] #41244 <!-- Fix shell `cd` error when working dir has been deleted
-->
@KristofferC KristofferC removed the backport 1.11 Change should be backported to release-1.11 label Aug 26, 2024
KristofferC pushed a commit that referenced this pull request Sep 9, 2024
…actVector (#55480)

Per
#55470 (comment),
the `push!(::AbstractArray, ...)` array implementation assumed one-based
indexing and did not account for an `OffsetVector`
scenario.

Here we add tests for `push!(::AbstractArray, ...)` and
`append(::AbstractArray, ...)` including using `@invoke` to test the
effect on `OffsetVector`.

cc: @fredrikekre
(cherry picked from commit 5230d27)
KristofferC pushed a commit that referenced this pull request Sep 12, 2024
…actVector (#55480)

Per
#55470 (comment),
the `push!(::AbstractArray, ...)` array implementation assumed one-based
indexing and did not account for an `OffsetVector`
scenario.

Here we add tests for `push!(::AbstractArray, ...)` and
`append(::AbstractArray, ...)` including using `@invoke` to test the
effect on `OffsetVector`.

cc: @fredrikekre
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrays [a, r, r, a, y, s]
Projects
None yet
Development

Successfully merging this pull request may close these issues.

push!, append! AbstractVector implementations no longer work on Julia 1.11+
7 participants