Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Got an error while training on GPU with Mish activation function #1235

Closed
bienpierre opened this issue Jun 18, 2020 · 10 comments
Closed

Got an error while training on GPU with Mish activation function #1235

bienpierre opened this issue Jun 18, 2020 · 10 comments

Comments

@bienpierre
Copy link

Hello,
I would like to define a NN model with Mish activation function defined within NNlib.jl and train it on GPU.

using Flux
using Flux: @epochs
using CuArrays

xs = [rand(2,50)] |> gpu
ys = [rand(2,50)] |> gpu

data_gpu = zip(xs, ys) |> gpu

m = Chain( Dense(2, 2, mish), Dense(2, 2, mish)) |> gpu

loss_gpu(x, y) = Flux.mse(m(x), y)

ps = Flux.params(m)
opt = RADAM()

@epochs 150 Flux.train!(loss_gpu, Flux.params(m), data_gpu, opt) #gpu compute

I have the following error:

Warning: calls to Base intrinsics might be GPU incompatible
│ exception =
│ You called exp(x::T) where T<:Union{Float32, Float64} in Base.Math at special/exp.jl:75, maybe you intended to call exp(x::Float32) in CUDAnative at C:\Users\userfolder.julia\packages\CUDAnative\hfulr\src\device\cuda\math.jl:101 instead?
│ Stacktrace:
│ [1] exp at special/exp.jl:75
│ [2] softplus at C:\Users\userfolder.julia\packages\NNlib\FAI3o\src\activation.jl:189
│ [3] #25 at C:\Users\userfolder.julia\packages\GPUArrays\1wgPO\src\broadcast.jl:49
└ @ CUDAnative C:\Users\userfolder.julia\packages\CUDAnative\hfulr\src\compiler\irgen.jl:111
┌ Warning: calls to Base intrinsics might be GPU incompatible
│ exception =
│ You called log1p(x::Float32) in Base.Math at special/log.jl:361, maybe you intended to call log1p(x::Float32) in CUDAnative at C:\Users\userfolder.julia\packages\CUDAnative\hfulr\src\device\cuda\math.jl:85 instead?
│ Stacktrace:
│ [1] log1p at special/log.jl:361
│ [2] softplus at C:\Users\userfolder.julia\packages\NNlib\FAI3o\src\activation.jl:189
│ [3] #25 at C:\Users\userfolder.julia\packages\GPUArrays\1wgPO\src\broadcast.jl:49
└ @ CUDAnative C:\Users\userfolder.julia\packages\CUDAnative\hfulr\src\compiler\irgen.jl:111
┌ Warning: calls to Base intrinsics might be GPU incompatible
│ exception =
│ You called exp(x::T) where T<:Union{Float32, Float64} in Base.Math at special/exp.jl:75, maybe you intended to call exp(x::Float32) in CUDAnative at C:\Users\userfolder.julia\packages\CUDAnative\hfulr\src\device\cuda\math.jl:101 instead?
│ Stacktrace:
│ [1] exp at special/exp.jl:75
│ [2] softplus at C:\Users\userfolder.julia\packages\NNlib\FAI3o\src\activation.jl:189
│ [3] #25 at C:\Users\userfolder.julia\packages\GPUArrays\1wgPO\src\broadcast.jl:49
└ @ CUDAnative C:\Users\userfolder.julia\packages\CUDAnative\hfulr\src\compiler\irgen.jl:111
┌ Warning: calls to Base intrinsics might be GPU incompatible
│ exception =
│ You called log1p(x::Float32) in Base.Math at special/log.jl:361, maybe you intended to call log1p(x::Float32) in CUDAnative at C:\Users\userfolder.julia\packages\CUDAnative\hfulr\src\device\cuda\math.jl:85 instead?
│ Stacktrace:
│ [1] log1p at special/log.jl:361
│ [2] softplus at C:\Users\userfolder.julia\packages\NNlib\FAI3o\src\activation.jl:189
│ [3] #25 at C:\Users\userfolder.julia\packages\GPUArrays\1wgPO\src\broadcast.jl:49
└ @ CUDAnative C:\Users\userfolder.julia\packages\CUDAnative\hfulr\src\compiler\irgen.jl:111
┌ Warning: calls to Base intrinsics might be GPU incompatible
│ exception =
│ You called tanh(x::T) where T<:Union{Float32, Float64} in Base.Math at special/hyperbolic.jl:151, maybe you intended to call tanh(x::Float32) in CUDAnative at C:\Users\userfolder.julia\packages\CUDAnative\hfulr\src\device\cuda\math.jl:55 instead?
│ Stacktrace:
│ [1] tanh at special/hyperbolic.jl:151
│ [2] #25 at C:\Users\userfolder.julia\packages\GPUArrays\1wgPO\src\broadcast.jl:49
└ @ CUDAnative C:\Users\userfolder.julia\packages\CUDAnative\hfulr\src\compiler\irgen.jl:111
ERROR: InvalidIRError: compiling #25(CuArrays.CuKernelState, CUDAnative.CuDeviceArray{ForwardDiff.Dual{Nothing,Float32,1},2,CUDAnative.AS.Global}, Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}},Zygote.var"#1620#1623"{typeof(mish)},Tuple{Base.Broadcast.Extruded{CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global},Tuple{Bool,Bool},Tuple{Int64,Int64}}}}) resulted in invalid LLVM IR
Reason: unsupported dynamic function invocation (call to print)
Stacktrace:
[1] print_to_string at strings/io.jl:138
[2] string at strings/io.jl:177
[3] throw_complex_domainerror at math.jl:32
[4] log1p at special/log.jl:389
[5] log1p at C:\Users\userfolder.julia\packages\ForwardDiff\cXTw0\src\dual.jl:203
[6] softplus at C:\Users\userfolder.julia\packages\NNlib\FAI3o\src\activation.jl:189
[7] mish at C:\Users\userfolder.julia\packages\NNlib\FAI3o\src\activation.jl:206
[8] #1620 at C:\Users\userfolder.julia\packages\Zygote\tJj2w\src\lib\broadcast.jl:162
[9] _broadcast_getindex_evalf at broadcast.jl:630
[10] _broadcast_getindex at broadcast.jl:603
[11] getindex at broadcast.jl:563
[12] #25 at C:\Users\userfolder.julia\packages\GPUArrays\1wgPO\src\broadcast.jl:50
Reason: unsupported call through a literal pointer (call to jl_array_grow_end)
Stacktrace:
[1] _growend! at array.jl:827
[2] resize! at array.jl:1019
[3] print_to_string at strings/io.jl:140
[4] string at strings/io.jl:177
[5] throw_complex_domainerror at math.jl:32
[6] log1p at special/log.jl:389
[7] log1p at C:\Users\userfolder.julia\packages\ForwardDiff\cXTw0\src\dual.jl:203
[8] softplus at C:\Users\userfolder.julia\packages\NNlib\FAI3o\src\activation.jl:189
[9] mish at C:\Users\userfolder.julia\packages\NNlib\FAI3o\src\activation.jl:206
[10] #1620 at C:\Users\userfolder.julia\packages\Zygote\tJj2w\src\lib\broadcast.jl:162
[11] _broadcast_getindex_evalf at broadcast.jl:630
[12] _broadcast_getindex at broadcast.jl:603
[13] getindex at broadcast.jl:563
[14] #25 at C:\Users\userfolder.julia\packages\GPUArrays\1wgPO\src\broadcast.jl:50
Reason: unsupported call through a literal pointer (call to jl_array_del_end)
Stacktrace:
[1] _deleteend! at array.jl:836
[2] resize! at array.jl:1024
[3] print_to_string at strings/io.jl:140
[4] string at strings/io.jl:177
[5] throw_complex_domainerror at math.jl:32
[6] log1p at special/log.jl:389
[7] log1p at C:\Users\userfolder.julia\packages\ForwardDiff\cXTw0\src\dual.jl:203
[8] softplus at C:\Users\userfolder.julia\packages\NNlib\FAI3o\src\activation.jl:189
[9] mish at C:\Users\userfolder.julia\packages\NNlib\FAI3o\src\activation.jl:206
[10] #1620 at C:\Users\userfolder.julia\packages\Zygote\tJj2w\src\lib\broadcast.jl:162
[11] _broadcast_getindex_evalf at broadcast.jl:630
[12] _broadcast_getindex at broadcast.jl:603
[13] getindex at broadcast.jl:563
[14] #25 at C:\Users\userfolder.julia\packages\GPUArrays\1wgPO\src\broadcast.jl:50
Reason: unsupported call through a literal pointer (call to jl_array_to_string)
Stacktrace:
[1] String at strings/string.jl:39
[2] print_to_string at strings/io.jl:140
[3] string at strings/io.jl:177
[4] throw_complex_domainerror at math.jl:32
[5] log1p at special/log.jl:389
[6] log1p at C:\Users\userfolder.julia\packages\ForwardDiff\cXTw0\src\dual.jl:203
[7] softplus at C:\Users\userfolder.julia\packages\NNlib\FAI3o\src\activation.jl:189
[8] mish at C:\Users\userfolder.julia\packages\NNlib\FAI3o\src\activation.jl:206
[9] #1620 at C:\Users\userfolder.julia\packages\Zygote\tJj2w\src\lib\broadcast.jl:162
[10] _broadcast_getindex_evalf at broadcast.jl:630
[11] _broadcast_getindex at broadcast.jl:603
[12] getindex at broadcast.jl:563
[13] #25 at C:\Users\userfolder.julia\packages\GPUArrays\1wgPO\src\broadcast.jl:50
Reason: unsupported call to the Julia runtime (call to jl_f_tuple)
Stacktrace:
[1] expm1 at math.jl:286
[2] tanh at special/hyperbolic.jl:169
[3] tanh at C:\Users\userfolder.julia\packages\ForwardDiff\cXTw0\src\dual.jl:203
[4] mish at C:\Users\userfolder.julia\packages\NNlib\FAI3o\src\activation.jl:206
[5] #1620 at C:\Users\userfolder.julia\packages\Zygote\tJj2w\src\lib\broadcast.jl:162
[6] _broadcast_getindex_evalf at broadcast.jl:630
[7] _broadcast_getindex at broadcast.jl:603
[8] getindex at broadcast.jl:563
[9] #25 at C:\Users\userfolder.julia\packages\GPUArrays\1wgPO\src\broadcast.jl:50
Reason: unsupported call to the Julia runtime (call to jl_type_error)
Stacktrace:
[1] expm1 at math.jl:286
[2] tanh at special/hyperbolic.jl:165
[3] tanh at C:\Users\userfolder.julia\packages\ForwardDiff\cXTw0\src\dual.jl:203
[4] mish at C:\Users\userfolder.julia\packages\NNlib\FAI3o\src\activation.jl:206
[5] #1620 at C:\Users\userfolder.julia\packages\Zygote\tJj2w\src\lib\broadcast.jl:162
[6] _broadcast_getindex_evalf at broadcast.jl:630
[7] _broadcast_getindex at broadcast.jl:603
[8] getindex at broadcast.jl:563
[9] #25 at C:\Users\userfolder.julia\packages\GPUArrays\1wgPO\src\broadcast.jl:50
Reason: unsupported call to the Julia runtime (call to jl_type_error)
Stacktrace:
[1] expm1 at math.jl:286
[2] tanh at special/hyperbolic.jl:169
[3] tanh at C:\Users\userfolder.julia\packages\ForwardDiff\cXTw0\src\dual.jl:203
[4] mish at C:\Users\userfolder.julia\packages\NNlib\FAI3o\src\activation.jl:206
[5] #1620 at C:\Users\userfolder.julia\packages\Zygote\tJj2w\src\lib\broadcast.jl:162
[6] _broadcast_getindex_evalf at broadcast.jl:630
[7] _broadcast_getindex at broadcast.jl:603
[8] getindex at broadcast.jl:563
[9] #25 at C:\Users\userfolder.julia\packages\GPUArrays\1wgPO\src\broadcast.jl:50
Reason: unsupported call through a literal pointer (call to jl_alloc_string)
Stacktrace:
[1] _string_n at strings/string.jl:60
[2] StringVector at iobuffer.jl:31
[3] #IOBuffer#315 at iobuffer.jl:114
[4] multiple call sites at unknown:0
Reason: unsupported call through a literal pointer (call to jl_string_to_array)
Stacktrace:
[1] unsafe_wrap at strings/string.jl:71
[2] StringVector at iobuffer.jl:31
[3] #IOBuffer#315 at iobuffer.jl:114
[4] multiple call sites at unknown:0
Reason: unsupported call through a literal pointer (call to memset)
Stacktrace:
[1] fill! at array.jl:382
[2] #IOBuffer#315 at iobuffer.jl:121
[3] multiple call sites at unknown:0
Reason: unsupported call through a literal pointer (call to jl_alloc_string)
Stacktrace:
[1] _string_n at strings/string.jl:60
[2] string at strings/substring.jl:191
[3] throw_complex_domainerror at math.jl:32
[4] log1p at special/log.jl:389
[5] log1p at C:\Users\userfolder.julia\packages\ForwardDiff\cXTw0\src\dual.jl:203
[6] softplus at C:\Users\userfolder.julia\packages\NNlib\FAI3o\src\activation.jl:189
[7] mish at C:\Users\userfolder.julia\packages\NNlib\FAI3o\src\activation.jl:206
[8] #1620 at C:\Users\userfolder.julia\packages\Zygote\tJj2w\src\lib\broadcast.jl:162
[9] _broadcast_getindex_evalf at broadcast.jl:630
[10] _broadcast_getindex at broadcast.jl:603
[11] getindex at broadcast.jl:563
[12] #25 at C:\Users\userfolder.julia\packages\GPUArrays\1wgPO\src\broadcast.jl:50
Reason: unsupported call through a literal pointer (call to RtlMoveMemory)
Stacktrace:
[1] unsafe_copyto! at array.jl:242
[2] macro expansion at gcutils.jl:91
[3] __unsafe_string! at strings/substring.jl:178
[4] string at strings/substring.jl:194
[5] throw_complex_domainerror at math.jl:32
[6] log1p at special/log.jl:389
[7] log1p at C:\Users\userfolder.julia\packages\ForwardDiff\cXTw0\src\dual.jl:203
[8] softplus at C:\Users\userfolder.julia\packages\NNlib\FAI3o\src\activation.jl:189
[9] mish at C:\Users\userfolder.julia\packages\NNlib\FAI3o\src\activation.jl:206
[10] #1620 at C:\Users\userfolder.julia\packages\Zygote\tJj2w\src\lib\broadcast.jl:162
[11] _broadcast_getindex_evalf at broadcast.jl:630
[12] _broadcast_getindex at broadcast.jl:603
[13] getindex at broadcast.jl:563
[14] #25 at C:\Users\userfolder.julia\packages\GPUArrays\1wgPO\src\broadcast.jl:50
Stacktrace:
[1] check_ir(::CUDAnative.CompilerJob, ::LLVM.Module) at C:\Users\userfolder.julia\packages\CUDAnative\hfulr\src\compiler\validation.jl:116
[2] macro expansion at C:\Users\userfolder.julia\packages\CUDAnative\hfulr\src\compiler\driver.jl:193 [inlined]
[3] macro expansion at C:\Users\userfolder.julia\packages\TimerOutputs\7Id5J\src\TimerOutput.jl:228 [inlined]
[4] #codegen#156(::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::typeof(CUDAnative.codegen), ::Symbol, ::CUDAnative.CompilerJob) at C:\Users\userfolder.julia\packages\CUDAnative\hfulr\src\compiler\driver.jl:191
[5] #codegen at .\none:0 [inlined]
[6] #compile#155(::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::typeof(CUDAnative.compile), ::Symbol, ::CUDAnative.CompilerJob) at C:\Users\userfolder.julia\packages\CUDAnative\hfulr\src\compiler\driver.jl:52
[7] #compile#154 at .\none:0 [inlined]
[8] #compile at .\none:0 [inlined] (repeats 2 times)
[9] macro expansion at C:\Users\userfolder.julia\packages\CUDAnative\hfulr\src\execution.jl:393 [inlined]
[10] #cufunction#200(::Nothing, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(CUDAnative.cufunction), ::GPUArrays.var"#25#26", ::Type{Tuple{CuArrays.CuKernelState,CUDAnative.CuDeviceArray{ForwardDiff.Dual{Nothing,Float32,1},2,CUDAnative.AS.Global},Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}},Zygote.var"#1620#1623"{typeof(mish)},Tuple{Base.Broadcast.Extruded{CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global},Tuple{Bool,Bool},Tuple{Int64,Int64}}}}}}) at C:\Users\userfolder.julia\packages\CUDAnative\hfulr\src\execution.jl:360
[11] cufunction(::Function, ::Type) at C:\Users\userfolder.julia\packages\CUDAnative\hfulr\src\execution.jl:360
[12] macro expansion at C:\Users\userfolder.julia\packages\CUDAnative\hfulr\src\execution.jl:179 [inlined]
[13] macro expansion at .\gcutils.jl:91 [inlined]
[14] macro expansion at C:\Users\userfolder.julia\packages\CUDAnative\hfulr\src\execution.jl:176 [inlined]
[15] _gpu_call(::CuArrays.CuArrayBackend, ::Function, ::CuArray{ForwardDiff.Dual{Nothing,Float32,1},2,Nothing}, ::Tuple{CuArray{ForwardDiff.Dual{Nothing,Float32,1},2,Nothing},Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}},Zygote.var"#1620#1623"{typeof(mish)},Tuple{Base.Broadcast.Extruded{CuArray{Float32,2,Nothing},Tuple{Bool,Bool},Tuple{Int64,Int64}}}}}, ::Tuple{Tuple{Int64},Tuple{Int64}}) at C:\Users\userfolder.julia\packages\CuArrays\HE8G6\src\gpuarray_interface.jl:62
[16] gpu_call(::Function, ::CuArray{ForwardDiff.Dual{Nothing,Float32,1},2,Nothing}, ::Tuple{CuArray{ForwardDiff.Dual{Nothing,Float32,1},2,Nothing},Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}},Zygote.var"#1620#1623"{typeof(mish)},Tuple{Base.Broadcast.Extruded{CuArray{Float32,2,Nothing},Tuple{Bool,Bool},Tuple{Int64,Int64}}}}}, ::Int64) at C:\Users\userfolder.julia\packages\GPUArrays\1wgPO\src\abstract_gpu_interface.jl:151
[17] gpu_call at C:\Users\userfolder.julia\packages\GPUArrays\1wgPO\src\abstract_gpu_interface.jl:128 [inlined]
[18] copyto! at C:\Users\userfolder.julia\packages\GPUArrays\1wgPO\src\broadcast.jl:48 [inlined]
[19] copyto! at .\broadcast.jl:863 [inlined]
[20] copy(::Base.Broadcast.Broadcasted{Base.Broadcast.ArrayStyle{CuArray},Tuple{Base.OneTo{Int64},Base.OneTo{Int64}},Zygote.var"#1620#1623"{typeof(mish)},Tuple{CuArray{Float32,2,Nothing}}}) at .\broadcast.jl:839
[21] materialize at .\broadcast.jl:819 [inlined]
[22] broadcast_forward at C:\Users\userfolder.julia\packages\Zygote\tJj2w\src\lib\broadcast.jl:168 [inlined]
[23] adjoint at C:\Users\userfolder.julia\packages\Zygote\tJj2w\src\lib\broadcast.jl:178 [inlined]
[24] _pullback at C:\Users\userfolder.julia\packages\ZygoteRules\6nssF\src\adjoint.jl:47 [inlined]
[25] adjoint at C:\Users\userfolder.julia\packages\Zygote\tJj2w\src\lib\lib.jl:153 [inlined]
[26] _pullback at C:\Users\userfolder.julia\packages\ZygoteRules\6nssF\src\adjoint.jl:47 [inlined]
[27] broadcasted at .\broadcast.jl:1231 [inlined]
[28] Dense at C:\Users\userfolder.julia\packages\Flux\NpkMm\src\layers\basic.jl:115 [inlined]
[29] Dense at C:\Users\userfolder.julia\packages\Flux\NpkMm\src\layers\basic.jl:126 [inlined]
[30] applychain at C:\Users\userfolder.julia\packages\Flux\NpkMm\src\layers\basic.jl:30 [inlined]
[31] Chain at C:\Users\userfolder.julia\packages\Flux\NpkMm\src\layers\basic.jl:32 [inlined]
[32] _pullback(::Zygote.Context, ::Chain{Tuple{Dense{typeof(mish),CuArray{Float32,2,Nothing},CuArray{Float32,1,Nothing}},Dense{typeof(mish),CuArray{Float32,2,Nothing},CuArray{Float32,1,Nothing}}}}, ::CuArray{Float32,2,Nothing}) at C:\Users\userfolder.julia\packages\Zygote\tJj2w\src\compiler\interface2.jl:0
[33] loss_gpu at .\none:1 [inlined]
[34] _pullback(::Zygote.Context, ::typeof(loss_gpu), ::CuArray{Float32,2,Nothing}, ::CuArray{Float32,2,Nothing}) at C:\Users\userfolder.julia\packages\Zygote\tJj2w\src\compiler\interface2.jl:0
[35] adjoint at C:\Users\userfolder.julia\packages\Zygote\tJj2w\src\lib\lib.jl:153 [inlined]
[36] _pullback at C:\Users\userfolder.julia\packages\ZygoteRules\6nssF\src\adjoint.jl:47 [inlined]
[37] #15 at C:\Users\userfolder.julia\packages\Flux\NpkMm\src\optimise\train.jl:88 [inlined]
[38] _pullback(::Zygote.Context, ::Flux.Optimise.var"#15#23"{typeof(loss_gpu),Tuple{CuArray{Float32,2,Nothing},CuArray{Float32,2,Nothing}}}) at C:\Users\userfolder.julia\packages\Zygote\tJj2w\src\compiler\interface2.jl:0
[39] pullback(::Function, ::Zygote.Params) at C:\Users\userfolder.julia\packages\Zygote\tJj2w\src\compiler\interface.jl:96
[40] gradient(::Function, ::Zygote.Params) at C:\Users\userfolder.julia\packages\Zygote\tJj2w\src\compiler\interface.jl:46
[41] macro expansion at C:\Users\userfolder.julia\packages\Flux\NpkMm\src\optimise\train.jl:87 [inlined]
[42] macro expansion at C:\Users\userfolder.julia\packages\Juno\oLB1d\src\progress.jl:119 [inlined]
[43] #train!#12(::Flux.Optimise.var"#18#26", ::typeof(Flux.Optimise.train!), ::typeof(loss_gpu), ::Zygote.Params, ::Base.Iterators.Zip{Tuple{Array{CuArray{Float32,2,Nothing},1},Array{CuArray{Float32,2,Nothing},1}}}, ::RADAM) at C:\Users\userfolder.julia\packages\Flux\NpkMm\src\optimise\train.jl:80
[44] train!(::Function, ::Zygote.Params, ::Base.Iterators.Zip{Tuple{Array{CuArray{Float32,2,Nothing},1},Array{CuArray{Float32,2,Nothing},1}}}, ::RADAM) at C:\Users\userfolder.julia\packages\Flux\NpkMm\src\optimise\train.jl:78
[45] top-level scope at C:\Users\userfolder.julia\packages\Flux\NpkMm\src\optimise\train.jl:120
[46] top-level scope at C:\Users\userfolder.julia\packages\Juno\oLB1d\src\progress.jl:119

Version Pkg Status:

[7c4d4715] AmplNLWriter v0.5.0
[c52e3926] Atom v0.12.2
[fbb218c0] BSON v0.2.5
[6e4b80f9] BenchmarkTools v0.4.3
[a134a8b2] BlackBoxOptim v0.5.0
[a076750e] CPLEX v0.6.3
[9961bab8] Cbc v0.6.6
[e2554f3b] Clp v0.7.0
[35d6a980] ColorSchemes v3.5.0
[a6e380b2] ControlSystems v0.5.9
[3a865a2d] CuArrays v1.7.2
[587475ba] Flux v0.10.3
[f6369f11] ForwardDiff v0.10.10
[b6b21f68] Ipopt v0.6.1
[4076af6c] JuMP v0.20.1
[e5e0dc1b] Juno v0.7.2
[b964fa9f] LaTeXStrings v1.1.0
[23992714] MAT v0.7.0
[10e44e05] MATLAB v0.7.3
[cc2ba9b6] MLDataUtils v0.5.0
[ab2f91bb] OSQP v0.6.0
[429524aa] Optim v0.20.1
[ccf2f8ad] PlotThemes v1.0.1
[91a5bcdd] Plots v0.29.1
[92933f4c] ProgressMeter v1.2.0
[37e2e3b7] ReverseDiff v1.2.0
[2913bbd2] StatsBase v0.32.1
[a759f4b9] TimerOutputs v0.5.3
[e88e6eb3] Zygote v0.4.7
[ade2ca70] Dates
[37e2e46d] LinearAlgebra

Regards
@ocissor
Copy link

ocissor commented Jun 21, 2020

So the problem that you are facing is because of your data_gpu. You cannot zip your data and pass it in your training, you have to create a loader using DataLoader(xs, ys, batchsize=1). The reason will be clear to you with this example:
xs=rand(2,2)
0.596748 0.91317
0.834779 0.209982
ys=rand(2,2)
0.0395694 0.445609
0.182906 0.123037
data=zip(xs,ys)
for i in data
print(i)
end
the result that you will get is:
(0.5967477946961266, 0.03956942972990518)(0.8347788110081911, 0.1829063889455127)(0.9131699265086033, 0.4456093452723533)(0.20998161632583257, 0.12303724269965355)
as you can see the first tuple has (xs[1,1],ys[1,1]) that means the input to your dense layer is xs[1,1] and label is ys[1,1] .which is not right according to your model as it needs a vector of shape (2,1),But if I use DataLoader,
data=DataLoader(xs,ys,batchsize=1)
for i in data
print(i)
end
You get an output like this :
([0.5967477946961266; 0.8347788110081911], [0.03956942972990518; 0.1829063889455127])([0.9131699265086033; 0.20998161632583257], [0.4456093452723533; 0.12303724269965355])
so the input to your model is [0.5967477946961266; 0.8347788110081911] and label is [0.03956942972990518; 0.1829063889455127] which is correct according to your chain.

Hope This Helps.

@ocissor
Copy link

ocissor commented Jun 21, 2020

One more thing is that initialize your xs=rand(2,50) rather than xs=[rand(2,50)] and your ys also as rand(2,50) rather than [rand(2,50)].

@bienpierre
Copy link
Author

Thanks for answering,
I did the suggested modifications.

using Flux
using Flux: @epochs
using CuArrays

xs = rand(2,50) |> gpu
ys = rand(2,50) |> gpu

data = Flux.Data.DataLoader(xs,ys,batchsize=128, shuffle=false)

m = Chain(Dense(2,2,mish), Dense(2,2,mish)) |> gpu

loss_gpu(x,y) = Flux.mse(m(x),y)

ps = Flux.params(m)

opt = RADAM()

@epochs 150 Flux.train!(loss_gpu, ps, data, opt)

However, it does not work, with same previous errors.

I modify the activation function, and it works:

using Flux
using Flux: @epochs
using CuArrays

xs = rand(2,50) |> gpu
ys = rand(2,50) |> gpu

data = Flux.Data.DataLoader(xs,ys,batchsize=128, shuffle=false)

m = Chain(Dense(2,2,swish), Dense(2,2,swish)) |> gpu

loss_gpu(x,y) = Flux.mse(m(x),y)

ps = Flux.params(m)

opt = RADAM()

@epochs 150 Flux.train!(loss_gpu, ps, data, opt)

image

@ocissor
Copy link

ocissor commented Jun 22, 2020

can you run the code without using GPU and see whether you get an error or not, because I don't have a GPU and The code runs perfectly on my system.

@bienpierre
Copy link
Author

I run the code without GPU , it works.

using Flux
using Flux: @epochs
using CuArrays

xs = rand(2,50)
ys = rand(2,50)

data = Flux.Data.DataLoader(xs,ys,batchsize=128, shuffle=false)

m = Chain(Dense(2,2,mish), Dense(2,2,mish))

loss_gpu(x,y) = Flux.mse(m(x),y)

ps = Flux.params(m)

opt = RADAM()

@epochs 150 Flux.train!(loss_gpu, ps, data, opt)

@bienpierre
Copy link
Author

I use CUDANative.jl (I got information from discourse julia forum) and I rewrite the activation function :

using Flux
using Flux: @epochs
using CuArrays
using CUDAnative

xs = rand(2,50) |> gpu
ys = rand(2,50) |> gpu

data = Flux.Data.DataLoader(xs,ys,batchsize=128, shuffle=false)

my_mish(x::Real) = x * CUDAnative.tanh(CUDAnative.log(CUDAnative.exp(x)+1)) 

#m = Chain(Dense(2,2,mish), Dense(2,2,mish)) |> gpu
m = Chain(Dense(2,2,my_mish), Dense(2,2,my_mish)) |> gpu

loss_gpu(x,y) = Flux.mse(m(x),y)

ps = Flux.params(m)

opt = RADAM()

@epochs 150 Flux.train!(loss_gpu, ps, data, opt)

And it works:

image

@ocissor
Copy link

ocissor commented Jun 22, 2020

great!

@ocissor
Copy link

ocissor commented Jun 22, 2020

Lets Close this issue now

@CarloLucibello
Copy link
Member

mish should work out-of-the-box with cuarrays: https://github.com/JuliaGPU/CuArrays.jl/blob/aeba8c975447cf5387fcc77f29ffdb7dbddf713f/src/nnlib.jl#L20
If that's not the case, we should leave this issue open and investigate why

@bienpierre
Copy link
Author

Hello,

I have upgraded Julia to 1.4.2 and packages too:
image

Trianing is working with Mish:
image

Regards

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@CarloLucibello @bienpierre @ocissor