Does Intel hardware support Ta != Tb? I'm surprised if they would ever be different types?

Also in the CUDA + HIP AMD backends it is not necessary to provide the matrix_layout (doc should be updated to "layout" at some point) template parameter to joint_matrix_mad: it looks like it is always necessary for intel hardware? if so I can add it to the CUDA implementation so that the interfaces match.

Ah I forgot that Intel hardware supports store for non accumulator matrices, I'll update #6657 adding the use template parameter and throw a runtime error if it is not use::accumulator in the CUDA backend.

We remove L from the template arguments of store and replace it with "unused" in joint matrix.

I guess if you are removing packed_b and packed_a this sentence will change like this?

There is a potential issue here: Rows and Cols have default values, but there isn't a default value for the use parameter. Assuming the Rows/Cols default value is necessary long term (it isn't in the CUDA backend), should we switch the order of the parameters here?

@yubingex007-a11y, "use" implementation seems to override the default values for rows and cols. Can you please check that?
I think use should come second in order so we don't have to set a default value for it

Can you clarify the distinction between the matrix_layout L template parameter and matrix_layout memL parameter here?

Specifically do you plan on keeping the matrix_layout L template parameter in joint_matrix_load and joint_matrix_store?
Perhaps you have only kept this in because you think it is necessary for compatibility with the CUDA backend, since I notice that in the latest implementation, #5835, L is set as layout::unused by default?
In the CUDA implementation I took the approach of overloading joint_matrix_load: the first overload,

In the Intel backend can the layouts of all matrices be specified at runtime? Or is it the case that, as with the CUDA backend, some of them require that the layout is known at compile time?

Same here, we remove L from the template arguments of store and replace it with "unused" in joint matrix.

Is it acceptable to make the above change to fully support the CUDA functionality as we discussed and as I implemented here: 4949464

Also I think that the A, B, C matrices should also be passed by reference.

What happens if we just add a new type for D matrix but keep it as a returned argument of joint_matrix_mad?

The issue is that the data type held by the D matrix can differ from the data type held by the C matrix. For example if type C is fp16 and type D is fp32; does the intel backend not have such cases? The only way to deal with this is using information provided by the user via arguments to joint_matrix_mad. The cleanest way to do this is the above change where the data type of D can be inferred from the joint_matrix D argument. This change allows us to implement two missing mma operations in the CUDA backend that have been added in this commit: e55e5f0.

Another option is if there is an optional output "precision" fourth argument to joint_matrix_mad that will specify the type of D: in any case I don't see how a fourth argument to joint_matrix_mad can be avoided if we want to enable the missing cases introduced in e55e5f0

The only other option I see is to not allow the type of D to differ from the type of C. I don't see how adding a new type for D matrix and keeping it as a returned argument can affect this in any way.

We agreed the first resolution here is to add a new template type Td.
Also, replace Lc, La, Lb by unused

@JackAKirk, I thought for the CUDA backend, LA and LB are needed so we should replace them by unused here, right?

@JackAKirk, I thought for the CUDA backend, LA and LB are needed so we should replace them by unused here, right?

Yes that's right.

I meant we should NOT replace them, sorry.

Yeah I meant to keep the template parameters but set them as default unused since I think that in the Intel backend you don't use them, and always use joint_matrix with layout::unused? In the CUDA backend they can be inferred from the joint_matrix so also don't have to be explicitly specified.

That's right, good

Also (C) is rendered as the copyright sign so this needs to be corrected somehow.

It isn't clear here as a reader how "symmetric" or "tiled" layouts relate to the values of the enum class "layout" below.

I will remove them

I personally think it is better to avoid using "tensor" apart from where absolutely necessary :i.e for Tensor Cores (c), because the word "tensor" means something different from "matrix": a matrix is a tensor if there exists transformational rules for the matrix elements with respect to a coordinate system, and this property that distinguishes tensor is actually not relevant at all for the hardware operations and this extension more generally. A (any-dimensional) matrix is not necessarily a tensor although a matrix can also be a tensor: In situations where all matrices operated on are either all tensors or all not tensors it doesn't make much difference whether we refer to "matrices" as "tensors", but for applications where both (non-tensor)"matrices" and "tensors" are used, proper use of language becomes important.

I'm not sure where the trend of replacing (any-dimensional)"matrix" with "tensor" originated, perhaps it was when "tensorflow" sounded better than "matrixflow", but I don't think there is really a reason for adopting this trend in this extension, particularly since we have adopted the naming conventions of joint_matrix etc rather than e.g. joint_tensor.

A matrix is technically a 2d tensor and they are both mathematical objects.
tensorflow supports more than 2 dimensions. That's why I believe it is not called matrixflow.

I tend to refer to the hardware as tensor hardware because even though they are matrix hardware (supports only 2 dimensions), they enable ndimensional tensor computing (if the code is inside loops and such). Also, they are called TPUs in general in the literature. That's why the Nvidia tensor cores are called as such for instance. Habana uses tensor cores terminology as well.

Of course, we should refer to this extension as matrix because it enables only 2d computations.

A matrix is only a (representation of a) 2-tensor (in a given basis) if it has a basis independent meaning, likewise for a 3d matrix (3-array) etc. I personally think that Tensor Processing Unit is a terrible name for the literature to have adopted for Processing Units that perform basis dependent computations on 2-dimensional arrays (matrices) that may be part of n-dimensional arrays (n-dimensional matrices) for n>=3, that may or may not be part of a program that grants the n-matrix a tensorial nature. Here is an article that mentions the question of over-use of tensor in ML, https://www.linkedin.com/pulse/tensorsand-applications-ml-demystified-andriy-strepetilov/, with a mostly reasonably description of the meaning of tensor. It's not worth worrying about too much but I guess... for a ML algorithm you can probably define a conservation law (conserving the output) and then define a geometry (set of transformations that satisfy the conservation law) for the inputs such that the inputs are invariant with respect to those transformations (then they are tensors). So maybe "tensor"flow is justified that way.. But for the TPUs I don't see how using "tensor" leads to anything but confusion personally. If we adopt "tensor" to mean n-matrix (n-array) then what will we call something that has tensorial properties...
Probably it doesn't matter at this point because as you say TPU appears to have become accepted as the general term in the literature. But when a word is misappropriated in society, meaning tends to be lost, and society can become poorer for it.

Actually there are some aspects of the Tensor Cores backend that are not AOT, since there is JIT compilation from the ptx level to SASS level when the code is run. I think this would be more accurate and simpler personally:

AOT is used here to reflect the SYCL compilation flow: If it does not use SPIRV and target is specified at compile time so this is basically AOT compilation for SYCL. But I will change the sentence to the one that does not talk about the AOT detail.

Should this refer to joint_matrix_accumulator?

This is a good question. What do you think @gmlueck ?

You have this inconsistency throughout the spec. In many places, you refer to the three matrices as a, b, and c. However, the use enum refers to them as a, b, and accumulator. You should decide on one or the other and then use it consistently throughout. Since you call the first two matrices a and b, it seems natural to me to call the third matrix c. Is there a good reason to call the third matrix accumulator?

etc

We need typename S as an implementation detail so we can support const T cases via: std::enable_if_t<std::is_same<S, std::remove_const_t>::value

Can you remind me of the rules for using Layout if you want portable code that works on both Intel and CUDA? I thought the "A" and "B" matrices couldn't be dynamic if you wanted to run on CUDA, is that right?

We want the default template parameters to result in portable code, and I see that the template parameter here defaults to dynamic. Is that portable?

Don't we also want a description somewhere in this spec explaining how to use "Layout" if you want portable code? I don't see anything like that now.

Can you remind me of the rules for using Layout if you want portable code that works on both Intel and CUDA? I thought the "A" and "B" matrices couldn't be dynamic if you wanted to run on CUDA, is that right?

Yes the A/B matrices cannot be dynamic for cuda: we use partial specializations for all valid joint_matrix declarations in the cuda backend, so it isn't possible to construct such a use::a/use::b joint_matrix with layout::dynamic in the cuda backend.

We want the default template parameters to result in portable code, and I see that the template parameter here defaults to dynamic. Is that portable?

Don't we also want a description somewhere in this spec explaining how to use "Layout" if you want portable code? I don't see anything like that now.

FYI for the cuda backend it is also useful for layout::dynamic to be set as default because it means users don't even need to provide a layout template argument when constructing an accumulator joint_matrix: As here in the test for the portable interfaces:

https://github.com/intel/llvm-test-suite/blob/b8a6c135726072ab3e8ab07cb73b2c1d81fbafb0/SYCL/Matrix/joint_matrix_tensorcores.cpp#L151

Yes the A/B matrices cannot be dynamic for cuda: we use partial specializations for all valid joint_matrix declarations in the cuda backend, so it isn't possible to construct such a use::a/use::b joint_matrix with layout::dynamic in the cuda backend.

This restriction isn't conveyed by this spec. Do we expect this specification to describe a common API that includes CUDA?

FYI for the cuda backend it is also useful for layout::dynamic to be set as default because it means users don't even need to provide a layout template argument when constructing an accumulator joint_matrix

I agree that this is useful. This could be achieved by defaulting the Layout parameter conditionally, depending on the value of Use.

#6662 (comment) is relevant here, but I'm sure @dkhaldi will explain the details.

Please see under "Layout" subsection:
"Important | In both AMX and DPAS support, the layout template parameter can be dynamic for Matrix A and B as well."

I only talk about what is allowed for AMX and DPAS. This is not conveying the restriction about the CUDA backend because I expect Jack to update this file after it is merged to add the CUDA backend restrictions in joint matrix type (dynamic is the default for only matrix C) and in the joint_matrix_load (load that takes layout as argument is only for C matrix).

So the restrictions that apply for the CUDA backend will be explained in the specification text by Jack and not imposed in the syntax. If imposed in the syntax, this means we are restricting Intel backend as well.

@JackAKirk, @gmlueck do you want to proceed differently?

You are right about prioritizing the portable case. But one of the outcomes of that meeting as well is that the restrictions will still be made in the text of the spec, not in the syntax. Not just yet.

If we make the restriction in the syntax, we are excluding the Intel flexibility.
In the text, we should add that for the CUDA backend, dynamic as default applied to only matrix accumulator.

One of the reasons of that outcome is that I did not want to make more changes at a time.
The change you are suggesting (in the syntax) should be made in a separate PR after we discuss that specific issue more.

Can you please add this as an open issue in the list at the bottom of this spec, then? I just want to make sure we do not forget about issues like this.

Here is a full implementation of the agreed interfaces: #7077 with a working cuda implementation. See that we still use the layout::dynamic default parameter here:

I will add it to open issues.
Note that this is not the only non portable code that is included in this document. We still need to sort things like "packed" layout.
Once we have the full support of this new API (in a month or so) mentioned by Jack (#7077). I will open up discussion for the non portable scenarios that are still in this document.

Please see under "Layout" subsection: "Important | In both AMX and DPAS support, the layout template parameter can be dynamic for Matrix A and B as well."

I only talk about what is allowed for AMX and DPAS. This is not conveying the restriction about the CUDA backend because I expect Jack to update this file after it is merged to add the CUDA backend restrictions in joint matrix type (dynamic is the default for only matrix C) and in the joint_matrix_load (load that takes layout as argument is only for C matrix).

So the restrictions that apply for the CUDA backend will be explained in the specification text by Jack and not imposed in the syntax. If imposed in the syntax, this means we are restricting Intel backend as well.

@JackAKirk, @gmlueck do you want to proceed differently?

I think this is fine as you describe.
If you think that some information is more appropriate in a cuda only extension or intel only extension or the main matrix extension, then I don't mind moving it, especially since these extensions are all experimental.

I imagine that it is more traditional that extensions are additive rather that subtractive, but I don't think this is a big issue at the moment: so long as the interfaces (note that I don't say "code" here since DPAS and Tensor Cores don't ever have cases where parameters provided to the interfaces are portable) written by a user are fully portable across cuda and intel for the cases where the hardware allows them to (all the practically useful cases) then this is the main thing.

Why aren't there two versions of joint_matrix_store, as you have for joint_matrix_load?

No need for two versions. In both Intel and CUDA backends, store must take layout as a dynamic argument.
@JackAKirk please confirm

What if the template parameter of the matrix is something other than dynamic, though? For example, consider the case when Layout is row_major. Does it still make sense to require the user to specify a layout parameter at runtime to joint_matrix_store? What if the layout they specify at runtime is different from the Layout template parameter? Is that OK?

In the CUDA case, you can only store to matrix accumulator.
For the Intel backend, Besides C, you can store to matrix A or B. In this case, if users decide to do that, they assume it is not portable code, so they would use the version of joint matrix where A and B type layouts are dynamic.