@ichasepucks : Thanks for reporting, I'll try to reproduce your issue via some GitHub action mac workflow

FWIW, I'm trying to have some colleagues reproduce this. So far, one with an Intel Mac and an old version of Docker, 20.10.12 and Maven 3.8.2 has this working.

If you look in your build listing you see buildx build --progress=plain --builder multiarch --platform linux/arm64
then later in the push listing you have buildx build --progress=plain --builder multiarch --platform linux/amd64,linux/arm64

I think plugin builds it for a single platform "linux/arm64" and later on it try to push as a multi platform image.

When I look in the plugin sources BuildXService:134 buildAndLoadSinglePlatform() there is only support for single platform at a time, while BuildXService:147 pushMultiPlatform() performs multiplatform push.

I actualy tryied to do a multiplatform build with the maven-plugin with a number of different configuration all accoding to the documentation but never suceeded and I connsider this a bug or a not yet supported fetaure.

What is needed here i to rewrite the BuildXService:134 buildAndLoadSinglePlatform() to :

    protected void buildAndLoadPlatforms(
            List<String> buildX, String builderName, BuildDirs buildDirs,
            ImageConfiguration imageConfig, String configuredRegistry, File buildArchive
    )  throws MojoExecutionException {
        List<String> platforms = imageConfig.getBuildConfiguration().getBuildX().getPlatforms();

        // "load" of multiplatform builds into the local image repo is not supported by docker v24.0.6
        // single non native platform can be loaded and runed in dockers QEMU emulator
        buildX(buildX, builderName, buildDirs, imageConfig, configuredRegistry,
                    platforms, buildArchive, platforms.size()==1 ? "--load" : null);
    }

This was the easy part, then there are a number of test-cases that needs to be modified and adopted to the fact that multiplatform builds can be done and single platform builds don't need to be native.

After some iterations I finally understood how the plugin works and have åcome to a conclusion that that is works fine as it is, and reasons for it. The basic reason is that multiplatform images cant be loaded in your local docker repo you have on your machine. It only have support for single platform but it can actualy be a foreign platform. Later on when you run deploy it will rerun buildx ... -push that builds the multiplatform image and pushes it upstreams.

This all works fine for me, so disregards all my previous comments.

Hi @rohanKanojia this was really bugging me again so I built the plugin locally and started trying some things. I've narrowed this down to using the configuration. https://github.com/fabric8io/docker-maven-plugin/blob/master/src/main/java/io/fabric8/maven/docker/service/BuildXService.java#L76-L79

However, I don't know why this is causing the problem. I'm not sure what you'd recommend to resolve this? An option to ignore config could work but probably should understand why it's causing the failure more?

Edit:
I was reading more on https://docs.docker.com/engine/reference/commandline/cli/#configuration-files. This makes sense why it's not able to authenticate correctly since it's losing all of my auth settings in my config.json in my home dir. This works on my remote agents due to an older docker version. Why is this being set during the buildx push command anyway?

@rohanKanojia I would be willing to submit a PR for this if you could provide the necessary background info and some guidance on a potential approach.

some guidance on a potential approach.

@ichasepucks : Do you have something in mind that you would like to propose to fix this? I will try my best to help you. Problem is that I don't have mac machine, I'm not sure I would be able to reproduce the issue. Do you have some minimal reproducer project that I can build to reproduce the problem?

@rohanKanojia thanks for responding. I don't currently have a simple generic repro but I may be able to create one. I think my first question is why is the config added by default on new versions of Docker, here? One solution is to add a configuration option that only adds this when present. By default, I would expect most users to have their config directory already set up and configured how they need? Just trying to make sure I'm not missing an obvious reason for this bit of code.

Actually looks like this is a duplicate of #1701. I verified this works with version 0.43.3. This was broken in https://github.com/fabric8io/docker-maven-plugin/pull/1703/files.

@ichasepucks : I see. let me look into this.

@ichasepucks : Could you please share the location where you've installed docker-buildx? Is it in $HOME/.docker/cli-plugins ? What happens if you install the plugin in some other folder? I'm suspecting whether this is related to docker/for-mac#6928

docker --config path/to/config buildx ... seems to work okay on Linux and Windows but it fails on Mac OS.

@ichasepucks : I think this comment is very accurate description of the issue you're facing docker/for-mac#6928 (comment)

I believe --config defaults to ~/.docker, so that buildx is loaded from ~/.docker/cli-plugins.

If you set --config=/tmp/docker, it will stop looking in ~/.docker/cli-plugins, and fall back to looking in the system directories for buildx.

From your docker info, I'm guessing you only have buildx installed in ~/.docker/cli-plugins, so setting this flag breaks your env.

I think we can add a workaround in DMP code to copy docker-buildx binary to config directory so that Mac OS docker CLI is able to resolve buildx when --config is overridden. I tested this approach with GitHub Actions Mac workflow and I was able to build and push to a ghcr.io registry.

I have pushed code related to above mentioned approach in this branch https://github.com/rohanKanojia/docker-maven-plugin/tree/pr/buildx-push-fails-mac .

Could you please try out this branch and confirm if it is working for your environment?

Hi @rohanKanojia this still doesn't work. The problem is not resolving buildx. The problem is that we should not be setting a docker --config option by default. This is only a problem when you need to access images that require authentication and we've changed from a default docker config that has been previously authenticated to a blank one that has not. There is no reason to send a --config option by default to docker. Why are we doing this?

@ichasepucks :

this still doesn't work.

Could you please elaborate what error you faced while running it? I had tested it on MacOS via GitHub action macos-13 runner.

Docker Maven Plugin supports reading registry credentials from maven settings.xml, properties apart from default docker config. We need to set --config flag to send config information for these scenarios.

I think we can try to set --config only for push scenarios.

@rohanKanojia the error is the same as originally reported. I have server entries for the repositories I need authentication to but I still get a 403 Forbidden from the docker registry. Let me see if I can put some extra logging in to see if this is actually trying to use the auth configuration passed. It appears not to be in this context.

Edit: I see [INFO] DOCKER> Credentials helper reply for "docker-credential-osxkeychain" is docker-credential-osxkeychain (github.com/docker/docker-credential-helpers) v0.7.0. Does this indicate it's not using the server settings in settings.xml? I don't see any AuthConfig logs at all in my output.

@ichasepucks : Could you please set a breakpoint on this line and see what value is being picked inside AuthConfig?

Run mvnDebug docker:push from terminal and connect via Remote JVM Debug from your IDE.

@rohanKanojia debugged this a bit more and I think I see what's going on now. The AuthConfig object is non null and contains the configuredRepository which is the same as I defined in the pushRegistry configuration. The config.json is being written with these auth credentials. The problem however, is that the base FROM image is hosted on a registry that needs READ auth also. This other registry is actually what is failing on the FROM attempt to pull the base image. So this isn't actually about pushing at all.

Is there a way I can get more than 1 registry in the AuthConfig object?

@ichasepucks : I think we already handle case of fetching auth config from base images. Could you please set a breakpoint here and see what's happening here?

@rohanKanojia BuildMojo doesn't appear to be used during the buildx push operation. When I added logging to the BuildMojo code I can see the source registry is created as expected in the config.json but since this code path doesn't execute in the buildx push path the authconfig doesn't contain the source registry credentials. I think this is the source of the issue. Does this make sense?

@ichasepucks : Are you able to get issue fixed if you copy paste this code in RegistryService :

@rohanKanojia sorry I didn't follow the request. Can you tell me where you want that line copied to or clarify what you want me to try?

sorry I didn't follow the request. Can you tell me where you want that line copied to or clarify what you want me to try?

@ichasepucks : I just tried making modifications in RegistryService. I want you to try out this branch (I haven't tested it myself though since I don't have same environment as you) .

@rohanKanojia you got it! This is working now!

@ichasepucks : Would you like to submit a PR to fix this issue?

I'm not sure what I would do different from your code changes? Happy to submit your changes from my own fork if that is helpful to you.

@ichasepucks : I just did a spike. I think we would also need to adapt existing tests + write new tests

@rohanKanojia I see. I could possibly do that in the next week or 2 if you don't have time to add them. Thanks.

@ichasepucks : oh okay, I will try to complete it in coming weekend.

@ichasepucks : I've merged #1751

Could you please give a try to Docker Maven Plugin 0.44-SNASHOT and provide feedback whether it's working as expected for your use case?

@rohanKanojia I can confirm that 0.44-SNAPSHOT is working as expected for me. Thanks!

@ichasepucks : Thanks a lot, I'll ask some other users as well and will cut a release next week.

@ichasepucks : I've released 0.44.0 that contains fix for your issue. Would appreciate if you could try it out and provide feedback.

@rohanKanojia 0.44.0 works for me. Thanks.