- Sponsor
-
Notifications
You must be signed in to change notification settings - Fork 126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
An attempt to combine dense, depthwise and groupwise conv through DenseConvDims
#146
Open
arhik
wants to merge
2
commits into
FluxML:master
Choose a base branch
from
arhik:common_convdims
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+702
−664
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,120 @@ | ||
| export AbstractDims | ||
|
|
||
| """ | ||
| AbstractDims | ||
| Type system-level information about convolution dimensions. Critical for things like | ||
| `im2col!()` to generate efficient code, and helpful to reduce the number of kwargs | ||
| getting passed around. | ||
| We don't want to specialize on things like image size/channel count, so we generally | ||
| store those as fields, just for convenience, and to allow for non-breaking changes when | ||
| we decide we _do_ want to specialize on those values. We always want to specialize on | ||
| things like stride, padding, dilation, and kernel flipping though. | ||
| """ | ||
| abstract type AbstractDims{N, S, P, D, F} end | ||
|
|
||
| # Hack to get rid of type parameters | ||
| function basetype(::Type{C}) where {C <: AbstractDims} | ||
| if C <: ConvDims | ||
| return ConvDims | ||
| elseif C <: PoolDims | ||
| return PoolDims | ||
| else | ||
| return nothing | ||
| end | ||
| end | ||
|
|
||
| # Obvious getter definitions for the type system-level definitions | ||
| spatial_dims(c::AbstractDims{N,S,P,D,F}) where {N, S, P, D, F} = N | ||
| stride(c::AbstractDims{N,S,P,D,F}) where {N, S, P, D, F} = S | ||
| padding(c::AbstractDims{N,S,P,D,F}) where {N, S, P, D, F} = P | ||
| dilation(c::AbstractDims{N,S,P,D,F}) where {N, S, P, D, F} = D | ||
| flipkernel(c::AbstractDims{N,S,P,D,F}) where {N, S, P, D, F} = F | ||
|
|
||
| """ | ||
| im2col_dims(c::AbstractDims) | ||
| im2col calculates, for each output pixel, the "convolution" of N kernels where N is the | ||
| number of output channels, by doing a matrix multiply. The dimensions of that matrix | ||
| are given by this function. | ||
| """ | ||
| im2col_dims(c::AbstractDims) = (prod(output_size(c)), prod(kernel_size(c))*channels_in(c)) | ||
|
|
||
| # Protect your skin, kids. Also do common validation of stride, padding, etc... | ||
| function check_spdf(x_size::NTuple{N}, w_size::NTuple{N}, stride, padding, dilation) where {N} | ||
| # Number of spatial dimensions in `x` and `w`. | ||
| nd = N - 2 | ||
|
|
||
| # Given a number, duplicate it out to have `nd` length. If it's already a collection, | ||
| # just splat it out into a tuple so it's always a tuple. We'll lint length later. | ||
| expand_size(p::Number) = ntuple(_ -> Int(p), nd) | ||
| expand_size(p) = tuple(p...) | ||
|
|
||
| # Convert stride, padding, dilation, etc.. to fully-specified tuples | ||
| pstride = expand_size(stride) | ||
| pdilation = expand_size(dilation) | ||
| ppadding = expand_size(padding) | ||
|
|
||
| if length(pstride) != nd | ||
| throw(DimensionMismatch("Stride $(length(stride))d, should be $(nd)d!")) | ||
| end | ||
| if length(pdilation) != nd | ||
| throw(DimensionMismatch("Dilation $(length(pdilation))d, should be $(nd)d!")) | ||
| end | ||
|
|
||
| # padding is kind of a special case; we allow it to be either 2-length or 4-length, | ||
| # since we support asymmetrical padding | ||
| if length(ppadding) != 2*nd | ||
| if length(ppadding) == nd | ||
| # Do this repeat dance so that we get lo/hi symmetrical padding | ||
| ppadding = tuple(repeat(collect(ppadding), inner=2)...) | ||
| else | ||
| throw(DimensionMismatch("Padding $(length(ppadding))d, should be either $(nd)d or $(2*nd)d!")) | ||
| end | ||
| end | ||
|
|
||
| # Assert that kernel size * dilation is <= padded input size | ||
| for idx in 1:nd | ||
| Is = x_size[idx] | ||
| Pl = ppadding[(idx - 1)*2 + 1] | ||
| Ph = ppadding[(idx - 1)*2 + 2] | ||
| Ks = w_size[idx] | ||
| Ds = pdilation[idx] | ||
| if Is + Pl + Ph < (Ks - 1)*Ds + 1 | ||
| throw(DimensionMismatch("Kernel * dilation (($Ks - 1) * $Ds + 1) cannot be larger than input + padding ($Is + $Pl + $Ph)!")) | ||
| end | ||
| end | ||
|
|
||
| return pstride, ppadding, pdilation | ||
| end | ||
|
|
||
| """ | ||
| output_size(c::AbstractDims) | ||
| Calculate the output (spatial) dimensions of the convolution. Get channel count via | ||
| `channels_out(c)`, and batch count is unknowable. | ||
| """ | ||
| function output_size(c::AbstractDims) | ||
| I = input_size(c) | ||
| K = kernel_size(c) | ||
| S = stride(c) | ||
| P = padding(c) | ||
| D = dilation(c) | ||
|
|
||
| return ntuple(spatial_dims(c)) do i | ||
| return div(I[i] + P[(i-1)*2 + 1] + P[(i-1)*2 + 2] - (K[i] - 1) * D[i] - 1, S[i]) + 1 | ||
| end | ||
| end | ||
|
|
||
| # Override show() for these beauties | ||
| function Base.show(io::IO, cdims::C) where {C <: AbstractDims} | ||
| I = (input_size(cdims)..., channels_in(cdims)) | ||
| O = (output_size(cdims)..., channels_out(cdims)) | ||
| K = kernel_size(cdims) | ||
| S = stride(cdims) | ||
| P = padding(cdims) | ||
| D = dilation(cdims) | ||
| F = flipkernel(cdims) | ||
| print(io, "$(basetype(C)): $I * $K -> $O, stride: $S pad: $P, dil: $D, flip: $F") | ||
| end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,77 +1,79 @@ | ||
| export DenseConvDims | ||
|
|
||
| """ | ||
| DenseConvDims | ||
| Concrete subclass of `ConvDims` for a normal, dense, conv2d/conv3d. | ||
| """ | ||
| struct DenseConvDims{N,K,C_in,C_out,S,P,D,F} <: ConvDims{N,S,P,D,F} | ||
| I::NTuple{N,Int} | ||
| end | ||
|
|
||
| # Getters for the fields | ||
| input_size(c::DenseConvDims) = c.I | ||
| kernel_size(c::DenseConvDims{N,K,C_in,C_out,S,P,D,F}) where {N,K,C_in,C_out,S,P,D,F} = K | ||
| channels_in(c::DenseConvDims{N,K,C_in,C_out,S,P,D,F}) where {N,K,C_in,C_out,S,P,D,F} = C_in | ||
| channels_out(c::DenseConvDims{N,K,C_in,C_out,S,P,D,F}) where {N,K,C_in,C_out,S,P,D,F} = C_out | ||
|
|
||
| # Convenience wrapper to create DenseConvDims objects | ||
| function DenseConvDims(x_size::NTuple{M}, w_size::NTuple{M}; | ||
| stride=1, padding=0, dilation=1, flipkernel::Bool=false) where M | ||
| # Do common parameter validation | ||
| stride, padding, dilation = check_spdf(x_size, w_size, stride, padding, dilation) | ||
|
|
||
| # Ensure channels are equal | ||
| if x_size[end-1] != w_size[end-1] | ||
| xs = x_size[end-1] | ||
| ws = w_size[end-1] | ||
| throw(DimensionMismatch("Input channels must match! ($xs vs. $ws)")) | ||
| end | ||
|
|
||
| # The type parameters are what | ||
| return DenseConvDims{ | ||
| M - 2, | ||
| w_size[1:end-2], | ||
| x_size[end-1], | ||
| w_size[end], | ||
| stride, | ||
| padding, | ||
| dilation, | ||
| flipkernel | ||
| }( | ||
| # Input spatial size | ||
| x_size[1:end-2], | ||
| ) | ||
| end | ||
|
|
||
| # Auto-extract sizes and sub out to big brother above | ||
| function DenseConvDims(x::AbstractArray, w::AbstractArray; kwargs...) | ||
| if ndims(x) != ndims(w) | ||
| throw(DimensionMismatch("Rank of x and w must match! ($(ndims(x)) vs. $(ndims(w)))")) | ||
| end | ||
| return DenseConvDims(size(x), size(w); kwargs...) | ||
| end | ||
|
|
||
| # Useful for constructing a new DenseConvDims that has only a few elements different | ||
| # from the original progenitor object that it inherits shapes from. | ||
| function DenseConvDims(c::ConvDims; N=spatial_dims(c), I=input_size(c), K=kernel_size(c), | ||
| C_in=channels_in(c), C_out=channels_out(c), S=stride(c), | ||
| P=padding(c), D=dilation(c), F=flipkernel(c)) | ||
| return DenseConvDims{N, K, C_in, C_out, S, P, D, F}(I) | ||
| end | ||
|
|
||
| function check_dims(x::NTuple{M}, w::NTuple{M}, y::NTuple{M}, cdims::DenseConvDims) where {M} | ||
| # First, check that channel counts are all correct: | ||
| @assert x[M-1] == channels_in(cdims) DimensionMismatch("Data input channel count ($(x[M-1]) vs. $(channels_in(cdims)))") | ||
| @assert y[M-1] == channels_out(cdims) DimensionMismatch("Data output channel count ($(y[M-1]) vs. $(channels_out(cdims)))") | ||
| @assert w[M-1] == channels_in(cdims) DimensionMismatch("Kernel input channel count ($(w[M-1]) vs. $(channels_in(cdims)))") | ||
| @assert w[M] == channels_out(cdims) DimensionMismatch("Kernel output channel count ($(w[M]) vs. $(channels_out(cdims)))") | ||
|
|
||
| # Next, check that the spatial dimensions match up | ||
| @assert x[1:M-2] == input_size(cdims) DimensionMismatch("Data input spatial size ($(x[1:M-2]) vs. $(input_size(cdims)))") | ||
| @assert y[1:M-2] == output_size(cdims) DimensionMismatch("Data output spatial size ($(y[1:M-2]) vs. $(output_size(cdims)))") | ||
| @assert w[1:M-2] == kernel_size(cdims) DimensionMismatch("Kernel spatial size ($(w[1:M-2]) vs. $(kernel_size(cdims)))") | ||
|
|
||
| # Finally, check that the batch size matches | ||
| @assert x[M] == y[M] DimensionMismatch("Batch size ($(x[M]) vs. $(y[M]))") | ||
| end | ||
| # export DenseConvDims | ||
| # | ||
| # """ | ||
| # DenseConvDims | ||
| # | ||
| # Concrete subclass of `ConvDims` for a normal, dense, conv2d/conv3d. | ||
| # """ | ||
| # struct DenseConvDims{N,K,C_in,C_out,S,P,D,F,G} <: ConvDims{N,S,P,D,F} | ||
| # I::NTuple{N,Int} | ||
| # end | ||
| # | ||
| # # Getters for the fields | ||
| # input_size(c::DenseConvDims) = c.I | ||
| # kernel_size(c::DenseConvDims{N,K,C_in,C_out,S,P,D,F,G}) where {N,K,C_in,C_out,S,P,D,F,G} = K | ||
| # channels_in(c::DenseConvDims{N,K,C_in,C_out,S,P,D,F,G}) where {N,K,C_in,C_out,S,P,D,F,G} = C_in | ||
| # channels_out(c::DenseConvDims{N,K,C_in,C_out,S,P,D,F,G}) where {N,K,C_in,C_out,S,P,D,F,G} = C_out | ||
| # group_count(c::DenseConvDims{N,K,C_in,C_out,S,P,D,F,G}) where {N,K,C_in,C_out,S,P,D,F,G} = G | ||
| # | ||
| # # Convenience wrapper to create DenseConvDims objects | ||
| # function DenseConvDims(x_size::NTuple{M}, w_size::NTuple{M}; | ||
| # stride=1, padding=0, dilation=1, flipkernel::Bool=false, groupcount=1) where M | ||
| # # Do common parameter validation | ||
| # stride, padding, dilation = check_spdf(x_size, w_size, stride, padding, dilation) | ||
| # | ||
| # # Ensure channels are equal | ||
| # if x_size[end-1] != w_size[end-1]*groupcount | ||
| # xs = x_size[end-1] | ||
| # ws = w_size[end-1]*groupcount | ||
| # throw(DimensionMismatch("Input channels must match! ($xs vs. $ws)")) | ||
| # end | ||
| # | ||
| # # The type parameters are what | ||
| # return DenseConvDims{ | ||
| # M - 2, | ||
| # w_size[1:end-2], | ||
| # x_size[end-1], | ||
| # w_size[end], | ||
| # stride, | ||
| # padding, | ||
| # dilation, | ||
| # flipkernel, | ||
| # groupcount | ||
| # }( | ||
| # # Input spatial size | ||
| # x_size[1:end-2], | ||
| # ) | ||
| # end | ||
| # | ||
| # # Auto-extract sizes and sub out to big brother above | ||
| # function DenseConvDims(x::AbstractArray, w::AbstractArray; kwargs...) | ||
| # if ndims(x) != ndims(w) | ||
| # throw(DimensionMismatch("Rank of x and w must match! ($(ndims(x)) vs. $(ndims(w)))")) | ||
| # end | ||
| # return DenseConvDims(size(x), size(w); kwargs...) | ||
| # end | ||
| # | ||
| # # Useful for constructing a new DenseConvDims that has only a few elements different | ||
| # # from the original progenitor object that it inherits shapes from. | ||
| # function DenseConvDims(c::ConvDims; N=spatial_dims(c), I=input_size(c), K=kernel_size(c), | ||
| # C_in=channels_in(c), C_out=channels_out(c), S=stride(c), | ||
| # P=padding(c), D=dilation(c), F=flipkernel(c), G=group_count(c)) | ||
| # return DenseConvDims{N, K, C_in, C_out, S, P, D, F, G}(I) | ||
| # end | ||
| # | ||
| # function check_dims(x::NTuple{M}, w::NTuple{M}, y::NTuple{M}, cdims::DenseConvDims) where {M} | ||
| # # First, check that channel counts are all correct: | ||
| # @assert x[M-1] == channels_in(cdims) DimensionMismatch("Data input channel count ($(x[M-1]) vs. $(channels_in(cdims)))") | ||
| # @assert y[M-1] == channels_out(cdims) DimensionMismatch("Data output channel count ($(y[M-1]) vs. $(channels_out(cdims)))") | ||
| # @assert w[M-1] == channels_in(cdims)/group_count(cdims) DimensionMismatch("Kernel input channel count ($(w[M-1]) vs. $(channels_in(cdims)/group_count(cdims)))") | ||
| # @assert w[M] == channels_out(cdims) DimensionMismatch("Kernel output channel count ($(w[M]) vs. $(channels_out(cdims)))") | ||
| # | ||
| # # Next, check that the spatial dimensions match up | ||
| # @assert x[1:M-2] == input_size(cdims) DimensionMismatch("Data input spatial size ($(x[1:M-2]) vs. $(input_size(cdims)))") | ||
| # @assert y[1:M-2] == output_size(cdims) DimensionMismatch("Data output spatial size ($(y[1:M-2]) vs. $(output_size(cdims)))") | ||
| # @assert w[1:M-2] == kernel_size(cdims) DimensionMismatch("Kernel spatial size ($(w[1:M-2]) vs. $(kernel_size(cdims)))") | ||
| # | ||
| # # Finally, check that the batch size matches | ||
| # @assert x[M] == y[M] DimensionMismatch("Batch size ($(x[M]) vs. $(y[M]))") | ||
| # end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,84 +1,90 @@ | ||
| export DepthwiseConvDims | ||
|
|
||
| """ | ||
| DepthwiseConvDims | ||
| Concrete subclass of `ConvDims` for a depthwise convolution. Differs primarily due to | ||
| characterization by C_in, C_mult, rather than C_in, C_out. Useful to be separate from | ||
| DenseConvDims primarily for channel calculation differences. | ||
| """ | ||
| struct DepthwiseConvDims{N,K,C_in,C_mult,S,P,D,F} <: ConvDims{N,S,P,D,F} | ||
| I::NTuple{N, Int} | ||
| end | ||
|
|
||
| # Getters for the fields | ||
| input_size(c::DepthwiseConvDims) = c.I | ||
| kernel_size(c::DepthwiseConvDims{N,K,C_in,C_mult,S,P,D,F}) where {N,K,C_in,C_mult,S,P,D,F} = K | ||
| channels_in(c::DepthwiseConvDims{N,K,C_in,C_mult,S,P,D,F}) where {N,K,C_in,C_mult,S,P,D,F} = C_in | ||
| channels_out(c::DepthwiseConvDims{N,K,C_in,C_mult,S,P,D,F}) where {N,K,C_in,C_mult,S,P,D,F} = C_in * C_mult | ||
| channel_multiplier(c::DepthwiseConvDims{N,K,C_in,C_mult,S,P,D,F}) where {N,K,C_in,C_mult,S,P,D,F} = C_mult | ||
|
|
||
|
|
||
| # Convenience wrapper to create DepthwiseConvDims objects | ||
| function DepthwiseConvDims(x_size::NTuple{M}, w_size::NTuple{M}; | ||
| stride=1, padding=0, dilation=1, flipkernel::Bool=false) where M | ||
| # Do common parameter validation | ||
| stride, padding, dilation = check_spdf(x_size, w_size, stride, padding, dilation) | ||
|
|
||
| # Ensure channels are equal | ||
| if x_size[end-1] != w_size[end] | ||
| xs = x_size[end-1] | ||
| ws = w_size[end] | ||
| throw(DimensionMismatch("Input channels must match! ($xs vs. $ws)")) | ||
| end | ||
|
|
||
| return DepthwiseConvDims{ | ||
| M - 2, | ||
| # Kernel spatial size | ||
| w_size[1:end-2], | ||
| # Input channels | ||
| x_size[end-1], | ||
| # Channel multiplier | ||
| w_size[end-1], | ||
| stride, | ||
| padding, | ||
| dilation, | ||
| flipkernel | ||
| }( | ||
| # Image spatial size | ||
| x_size[1:end-2], | ||
| ) | ||
| end | ||
|
|
||
| # Auto-extract sizes and just pass those directly in | ||
| function DepthwiseConvDims(x::AbstractArray, w::AbstractArray; kwargs...) | ||
| if ndims(x) != ndims(w) | ||
| throw(DimensionMismatch("Rank of x and w must match! ($(ndims(x)) vs. $(ndims(w)))")) | ||
| end | ||
| return DepthwiseConvDims(size(x), size(w); kwargs...) | ||
| end | ||
|
|
||
| # Useful for constructing a new DepthwiseConvDims that has only a few elements different | ||
| # from the original progenitor object. | ||
| function DepthwiseConvDims(c::DepthwiseConvDims; N=spatial_dims(c), I=input_size(c), K=kernel_size(c), | ||
| C_in=channels_in(c), C_m=channel_multiplier(c), S=stride(c), | ||
| P=padding(c), D=dilation(c), F=flipkernel(c)) | ||
| return DepthwiseConvDims{N, K, C_in, C_m, S, P, D, F}(I) | ||
| end | ||
|
|
||
| # This one is basically the same as for DenseConvDims, we only change a few lines for kernel channel count | ||
| function check_dims(x::NTuple{M}, w::NTuple{M}, y::NTuple{M}, cdims::DepthwiseConvDims) where {M} | ||
| # First, check that channel counts are all correct: | ||
| @assert x[M-1] == channels_in(cdims) DimensionMismatch("Data input channel count ($(x[M-1]) vs. $(channels_in(cdims)))") | ||
| @assert y[M-1] == channels_out(cdims) DimensionMismatch("Data output channel count ($(y[M-1]) vs. $(channels_out(cdims)))") | ||
| @assert w[M-1] == channel_multiplier(cdims) DimensionMismatch("Kernel multiplier channel count ($(w[M-1]) vs. $(channel_multiplier(cdims))") | ||
| @assert w[M] == channels_in(cdims) DimensionMismatch("Kernel input channel count ($(w[M]) vs. $(channels_in(cdims)))") | ||
|
|
||
| # Next, check that the spatial dimensions match up | ||
| @assert x[1:M-2] == input_size(cdims) DimensionMismatch("Data input spatial size ($(x[1:M-2]) vs. $(input_size(cdims)))") | ||
| @assert y[1:M-2] == output_size(cdims) DimensionMismatch("Data output spatial size ($(y[1:M-2]) vs. $(output_size(cdims)))") | ||
| @assert w[1:M-2] == kernel_size(cdims) DimensionMismatch("Kernel spatial size ($(w[1:M-2]) vs. $(kernel_size(cdims)))") | ||
|
|
||
| # Finally, check that the batch size matches | ||
| @assert x[M] == y[M] DimensionMismatch("Batch size ($(x[M]) vs. $(y[M]))") | ||
| end | ||
| # export DepthwiseConvDims | ||
| # | ||
| # """ | ||
| # DepthwiseConvDims | ||
| # | ||
| # Concrete subclass of `ConvDims` for a depthwise convolution. Differs primarily due to | ||
| # characterization by C_in, C_mult, rather than C_in, C_out. Useful to be separate from | ||
| # DenseConvDims primarily for channel calculation differences. | ||
| # """ | ||
| # struct DepthwiseConvDims{N,S,P,D,F} <: ConvDims{N,S,P,D,F} | ||
| # I::NTuple{N, Int} | ||
| # K::NTuple{N, Int} | ||
| # C_in::Int | ||
| # C_mult::Int | ||
| # end | ||
| # | ||
| # # Getters for the fields | ||
| # input_size(c::DepthwiseConvDims) = c.I | ||
| # kernel_size(c::DepthwiseConvDims) = c.K | ||
| # channels_in(c::DepthwiseConvDims) = c.C_in | ||
| # channels_out(c::DepthwiseConvDims) = c.C_in * channel_multiplier(c) | ||
| # channel_multiplier(c::DepthwiseConvDims) = c.C_mult | ||
| # | ||
| # | ||
| # # Convenience wrapper to create DepthwiseConvDims objects | ||
| # function DepthwiseConvDims(x_size::NTuple{M}, w_size::NTuple{M}; | ||
| # stride=1, padding=0, dilation=1, flipkernel::Bool=false) where M | ||
| # # Do common parameter validation | ||
| # stride, padding, dilation = check_spdf(x_size, w_size, stride, padding, dilation) | ||
| # | ||
| # # Ensure channels are equal | ||
| # if x_size[end-1] != w_size[end] | ||
| # xs = x_size[end-1] | ||
| # ws = w_size[end] | ||
| # throw(DimensionMismatch("Input channels must match! ($xs vs. $ws)")) | ||
| # end | ||
| # | ||
| # return DepthwiseConvDims{ | ||
| # M - 2, | ||
| # stride, | ||
| # padding, | ||
| # dilation, | ||
| # flipkernel | ||
| # }( | ||
| # # Image spatial size | ||
| # x_size[1:end-2], | ||
| # | ||
| # # Kernel spatial size | ||
| # w_size[1:end-2], | ||
| # | ||
| # # Input channels | ||
| # x_size[end-1], | ||
| # | ||
| # # Channel multiplier | ||
| # w_size[end-1], | ||
| # ) | ||
| # end | ||
| # | ||
| # # Auto-extract sizes and just pass those directly in | ||
| # function DepthwiseConvDims(x::AbstractArray, w::AbstractArray; kwargs...) | ||
| # if ndims(x) != ndims(w) | ||
| # throw(DimensionMismatch("Rank of x and w must match! ($(ndims(x)) vs. $(ndims(w)))")) | ||
| # end | ||
| # return DepthwiseConvDims(size(x), size(w); kwargs...) | ||
| # end | ||
| # | ||
| # # Useful for constructing a new DepthwiseConvDims that has only a few elements different | ||
| # # from the original progenitor object. | ||
| # function DepthwiseConvDims(c::DepthwiseConvDims; N=spatial_dims(c), I=input_size(c), K=kernel_size(c), | ||
| # C_in=channels_in(c), C_m=channel_multiplier(c), S=stride(c), | ||
| # P=padding(c), D=dilation(c), F=flipkernel(c)) | ||
| # return DepthwiseConvDims{N, S, P, D, F}(I, K, C_in, C_m) | ||
| # end | ||
| # | ||
| # # This one is basically the same as for DenseConvDims, we only change a few lines for kernel channel count | ||
| # function check_dims(x::NTuple{M}, w::NTuple{M}, y::NTuple{M}, cdims::DepthwiseConvDims) where {M} | ||
| # # First, check that channel counts are all correct: | ||
| # @assert x[end-1] == channels_in(cdims) DimensionMismatch("Data input channel count ($(x[end-1]) vs. $(channels_in(cdims)))") | ||
| # @assert y[end-1] == channels_out(cdims) DimensionMismatch("Data output channel count ($(y[end-1]) vs. $(channels_out(cdims)))") | ||
| # @assert w[end-1] == channel_multiplier(cdims) DimensionMismatch("Kernel multiplier channel count ($(w[end-1]) vs. $(channel_multiplier(cdims))") | ||
| # @assert w[end] == channels_in(cdims) DimensionMismatch("Kernel input channel count ($(w[end]) vs. $(channels_in(cdims)))") | ||
| # | ||
| # # Next, check that the spatial dimensions match up | ||
| # @assert x[1:end-2] == input_size(cdims) DimensionMismatch("Data input spatial size ($(x[1:end-2]) vs. $(input_size(cdims)))") | ||
| # @assert y[1:end-2] == output_size(cdims) DimensionMismatch("Data output spatial size ($(y[1:end-2]) vs. $(output_size(cdims)))") | ||
| # @assert w[1:end-2] == kernel_size(cdims) DimensionMismatch("Kernel spatial size ($(w[1:end-2]) vs. $(kernel_size(cdims)))") | ||
| # | ||
| # # Finally, check that the batch size matches | ||
| # @assert x[end] == y[end] DimensionMismatch("Batch size ($(x[end]) vs. $(y[end]))") | ||
| # end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,120 +1,120 @@ | ||
| ## This file contains adapter code for doing depthwise convolutions with im2col. | ||
|
|
||
|
|
||
| """ | ||
| depthwiseconv_im2col!(y, x, w, cdims, col=similar(x); alpha=1, beta=0) | ||
| Perform a depthwise convolution using im2col and GEMM, store the result in `y`. | ||
| See `conv_im2col!()` for an explanation of optional parameters. | ||
| """ | ||
| depthwiseconv_im2col! | ||
|
|
||
| function depthwiseconv_im2col!( | ||
| y::AbstractArray{T,5}, x::AbstractArray{T,5}, | ||
| w::AbstractArray{T,5}, cdims::DepthwiseConvDims; | ||
| col::AbstractArray{T,2} = similar(x, im2col_dims(cdims)), | ||
| alpha=T(1), beta=T(0)) where T | ||
| check_dims(size(x), size(w), size(y), cdims) | ||
| # This functions exactly the same as conv_im2col!(), except that we shard the | ||
| # incoming data into slices of single channels. This means that we need to walk | ||
| # each pointer forward individually, as done below, taking a single input channel | ||
| # and combining it with each kernel individually, before walking forward and doing | ||
| # the next input channel. | ||
| M = prod(output_size(cdims)) | ||
| N = channel_multiplier(cdims) | ||
| K = prod(kernel_size(cdims)) | ||
|
|
||
| dcdims = DenseConvDims(cdims) | ||
| @inbounds for batch_idx in 1:size(x)[end] | ||
| im2col!(col, view(x, :, :, :, :, batch_idx), dcdims) | ||
|
|
||
| # We do a separate convolution for each channel in x, as we must | ||
| for c_in in 1:channels_in(cdims) | ||
| # Walk each pointer forward as we process each input channel | ||
| GC.@preserve col, w, y, begin | ||
| col_ptr = pointer(col, (c_in-1)*M*K+1) | ||
| w_ptr = pointer(w, (c_in-1)*K*N+1) | ||
| y_ptr = pointer(y, ((batch_idx - 1)*channels_in(cdims) + c_in - 1)*M*N + 1) | ||
| gemm!(Val(false), Val(false), M, N, K, alpha, col_ptr, w_ptr, beta, y_ptr) | ||
| end | ||
| end | ||
| end | ||
| return y | ||
| end | ||
|
|
||
| """ | ||
| ∇depthwiseconv_filter_im2col!(dw, w, dy, cdims, col=similar(dw); alpha=1, beta) | ||
| Depthwise conv2d backward pass onto the weights using im2col and GEMM. | ||
| See the documentation for `conv_im2col!()` for explanation of optional parameters. | ||
| """ | ||
| ∇depthwiseconv_filter_im2col! | ||
|
|
||
| function ∇depthwiseconv_filter_im2col!( | ||
| dw::AbstractArray{T,5}, x::AbstractArray{T,5}, | ||
| dy::AbstractArray{T,5}, cdims::DepthwiseConvDims; | ||
| col::AbstractArray{T,2} = similar(dw, im2col_dims(cdims)), | ||
| alpha=T(1), beta=T(0)) where T | ||
| check_dims(size(x), size(dw), size(dy), cdims) | ||
|
|
||
| M = prod(kernel_size(cdims)) | ||
| N = channel_multiplier(cdims) | ||
| K = prod(output_size(cdims)) | ||
|
|
||
| @inbounds for batch_idx in 1:size(x)[end] | ||
| im2col!(col, view(x, :, :, :, :, batch_idx), cdims) | ||
|
|
||
| # We do a separate convolution for each channel in x, as we must | ||
| for c_in in 1:channels_in(cdims) | ||
| # Walk each pointer forward as we process each input channel | ||
| GC.@preserve col, dw, dy, begin | ||
| col_ptr = pointer(col, (c_in - 1)*M*K + 1) | ||
| dy_ptr = pointer(dy, (batch_idx - 1)*N*K*channels_in(cdims) + (c_in - 1)*K*N + 1) | ||
| dw_ptr = pointer(dw, (c_in - 1)*M*N + 1) | ||
| gemm!(Val(true), Val(false), M, N, K, alpha, col_ptr, dy_ptr, beta, dw_ptr) | ||
| end | ||
| end | ||
|
|
||
| # Because we accumulate over batches in this loop, we must set `beta` equal | ||
| # to `1.0` from this point on. | ||
| beta = T(1) | ||
| end | ||
| return dw | ||
| end | ||
|
|
||
| """ | ||
| depthwiseconv2d_Δx_im2col!(dx, w, dy, cdims, col=similar(dx); alpha=1, beta=0) | ||
| Depwthwise conv2d backward pass onto the input using im2col and GEMM. | ||
| See the documentation for `conv_im2col!()` for explanation of optional parameters. | ||
| """ | ||
| ∇depthwiseconv_data_im2col! | ||
|
|
||
| function ∇depthwiseconv_data_im2col!( | ||
| dx::AbstractArray{T,5}, dy::AbstractArray{T,5}, | ||
| w::AbstractArray{T,5}, cdims::DepthwiseConvDims; | ||
| col::AbstractArray{T,2} = similar(dx, im2col_dims(cdims)), | ||
| alpha=T(1), beta=T(0)) where T | ||
| check_dims(size(dx), size(w), size(dy), cdims) | ||
|
|
||
| M = prod(output_size(cdims)) | ||
| N = prod(kernel_size(cdims)) | ||
| K = channel_multiplier(cdims) | ||
|
|
||
| @inbounds for batch_idx in 1:size(dx)[end] | ||
| # We do a separate convolution for each channel in x, as we must | ||
| for cidx in 1:channels_in(cdims) | ||
| GC.@preserve col, w, dy, begin | ||
| # Walk each pointer forward as we process each input channel | ||
| dy_ptr = pointer(dy, (batch_idx - 1)*M*K*channels_in(cdims)+(cidx - 1)*K*M + 1) | ||
| w_ptr = pointer(w, (cidx - 1)*K*N + 1) | ||
| col_ptr = pointer(col, (cidx - 1)*M*N + 1) | ||
| gemm!(Val(false), Val(true), M, N, K, alpha, dy_ptr, w_ptr, T(0), col_ptr) | ||
| end | ||
| end | ||
| col2im!(view(dx, :, :, :, :, batch_idx), col, cdims) | ||
| end | ||
| return dx | ||
| end | ||
| # ## This file contains adapter code for doing depthwise convolutions with im2col. | ||
| # | ||
| # | ||
| # """ | ||
| # depthwiseconv_im2col!(y, x, w, cdims, col=similar(x); alpha=1, beta=0) | ||
| # | ||
| # Perform a depthwise convolution using im2col and GEMM, store the result in `y`. | ||
| # | ||
| # See `conv_im2col!()` for an explanation of optional parameters. | ||
| # """ | ||
| # depthwiseconv_im2col! | ||
| # | ||
| # function depthwiseconv_im2col!( | ||
| # y::AbstractArray{T,5}, x::AbstractArray{T,5}, | ||
| # w::AbstractArray{T,5}, cdims::DepthwiseConvDims; | ||
| # col::AbstractArray{T,2} = similar(x, im2col_dims(cdims)), | ||
| # alpha=T(1), beta=T(0)) where T | ||
| # check_dims(size(x), size(w), size(y), cdims) | ||
| # | ||
| # # This functions exactly the same as conv_im2col!(), except that we shard the | ||
| # # incoming data into slices of single channels. This means that we need to walk | ||
| # # each pointer forward individually, as done below, taking a single input channel | ||
| # # and combining it with each kernel individually, before walking forward and doing | ||
| # # the next input channel. | ||
| # M = prod(output_size(cdims)) | ||
| # N = channel_multiplier(cdims) | ||
| # K = prod(kernel_size(cdims)) | ||
| # | ||
| # dcdims = DenseConvDims(cdims) | ||
| # @inbounds for batch_idx in 1:size(x)[end] | ||
| # im2col!(col, view(x, :, :, :, :, batch_idx), dcdims) | ||
| # | ||
| # # We do a separate convolution for each channel in x, as we must | ||
| # for c_in in 1:channels_in(cdims) | ||
| # # Walk each pointer forward as we process each input channel | ||
| # GC.@preserve col, w, y, begin | ||
| # col_ptr = pointer(col, (c_in-1)*M*K+1) | ||
| # w_ptr = pointer(w, (c_in-1)*K*N+1) | ||
| # y_ptr = pointer(y, ((batch_idx - 1)*channels_in(cdims) + c_in - 1)*M*N + 1) | ||
| # gemm!(Val(false), Val(false), M, N, K, alpha, col_ptr, w_ptr, beta, y_ptr) | ||
| # end | ||
| # end | ||
| # end | ||
| # return y | ||
| # end | ||
| # | ||
| # """ | ||
| # ∇depthwiseconv_filter_im2col!(dw, w, dy, cdims, col=similar(dw); alpha=1, beta) | ||
| # | ||
| # Depthwise conv2d backward pass onto the weights using im2col and GEMM. | ||
| # See the documentation for `conv_im2col!()` for explanation of optional parameters. | ||
| # """ | ||
| # ∇depthwiseconv_filter_im2col! | ||
| # | ||
| # function ∇depthwiseconv_filter_im2col!( | ||
| # dw::AbstractArray{T,5}, x::AbstractArray{T,5}, | ||
| # dy::AbstractArray{T,5}, cdims::DepthwiseConvDims; | ||
| # col::AbstractArray{T,2} = similar(dw, im2col_dims(cdims)), | ||
| # alpha=T(1), beta=T(0)) where T | ||
| # check_dims(size(x), size(dw), size(dy), cdims) | ||
| # | ||
| # M = prod(kernel_size(cdims)) | ||
| # N = channel_multiplier(cdims) | ||
| # K = prod(output_size(cdims)) | ||
| # | ||
| # @inbounds for batch_idx in 1:size(x)[end] | ||
| # im2col!(col, view(x, :, :, :, :, batch_idx), cdims) | ||
| # | ||
| # # We do a separate convolution for each channel in x, as we must | ||
| # for c_in in 1:channels_in(cdims) | ||
| # # Walk each pointer forward as we process each input channel | ||
| # GC.@preserve col, dw, dy, begin | ||
| # col_ptr = pointer(col, (c_in - 1)*M*K + 1) | ||
| # dy_ptr = pointer(dy, (batch_idx - 1)*N*K*channels_in(cdims) + (c_in - 1)*K*N + 1) | ||
| # dw_ptr = pointer(dw, (c_in - 1)*M*N + 1) | ||
| # gemm!(Val(true), Val(false), M, N, K, alpha, col_ptr, dy_ptr, beta, dw_ptr) | ||
| # end | ||
| # end | ||
| # | ||
| # # Because we accumulate over batches in this loop, we must set `beta` equal | ||
| # # to `1.0` from this point on. | ||
| # beta = T(1) | ||
| # end | ||
| # return dw | ||
| # end | ||
| # | ||
| # """ | ||
| # depthwiseconv2d_Δx_im2col!(dx, w, dy, cdims, col=similar(dx); alpha=1, beta=0) | ||
| # | ||
| # Depwthwise conv2d backward pass onto the input using im2col and GEMM. | ||
| # See the documentation for `conv_im2col!()` for explanation of optional parameters. | ||
| # """ | ||
| # ∇depthwiseconv_data_im2col! | ||
| # | ||
| # function ∇depthwiseconv_data_im2col!( | ||
| # dx::AbstractArray{T,5}, dy::AbstractArray{T,5}, | ||
| # w::AbstractArray{T,5}, cdims::DepthwiseConvDims; | ||
| # col::AbstractArray{T,2} = similar(dx, im2col_dims(cdims)), | ||
| # alpha=T(1), beta=T(0)) where T | ||
| # check_dims(size(dx), size(w), size(dy), cdims) | ||
| # | ||
| # M = prod(output_size(cdims)) | ||
| # N = prod(kernel_size(cdims)) | ||
| # K = channel_multiplier(cdims) | ||
| # | ||
| # @inbounds for batch_idx in 1:size(dx)[end] | ||
| # # We do a separate convolution for each channel in x, as we must | ||
| # for cidx in 1:channels_in(cdims) | ||
| # GC.@preserve col, w, dy, begin | ||
| # # Walk each pointer forward as we process each input channel | ||
| # dy_ptr = pointer(dy, (batch_idx - 1)*M*K*channels_in(cdims)+(cidx - 1)*K*M + 1) | ||
| # w_ptr = pointer(w, (cidx - 1)*K*N + 1) | ||
| # col_ptr = pointer(col, (cidx - 1)*M*N + 1) | ||
| # gemm!(Val(false), Val(true), M, N, K, alpha, dy_ptr, w_ptr, T(0), col_ptr) | ||
| # end | ||
| # end | ||
| # col2im!(view(dx, :, :, :, :, batch_idx), col, cdims) | ||
| # end | ||
| # return dx | ||
| # end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@staticfloat. This is only change. Only weights dimensions shrink as in here by groupcount value in third axis. if groupcount == 1 Nothing changes. When groupcount == 2 (lets say), then one group of weights operate only on the half of the input channels. and produce only one output channel. These weights groups have to be occupied across all the channels and we will have to use new group of weights (occupying all input channels in blocks) until their output matches output number of channels.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When groupcount == 7; we should make sure input channels can be divided exactly into 7 groups. Then we should check if output channels are multiple of groupcount(Since one group can only produce one output).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should then distribute these 7 groups to operate on different input channels in blocks of div(input_channels(), 7)