And I thought I was the only one crazy enough to do alpine updates. 😆
Actually, they used to be much harder. I did a lot of cleanup on it.

This PR looks good.

The big shortcoming is that you need to build and push out alpine, and then all of the other dependent packages, which is not too bad, it just takes forever. And is why the tests are failing.

I don't want to hold you up, so I can consider building and pushing it out, but if you can think of a sane way to get this into CI, so that it happens automatically, that would be great. During PR, it should be built and saved as an artifact to be used solely inside the PR; once merged to master, it should be published. Of course, you don't want to build all of them twice, once in PR and once after merge to master, since those are long slow builds.

Thoughts?

@ChrisIgel some of the commits were messed up. Not your fault, just has to do with the stages of stuff in alpine. When you do the build, it updates something in versions.<arch>, which, if missed when pushed, misses an update and everything downstream is affected.

I rebuilt them and force-pushed to the branch. Before trying any changes, be sure to pull down with a --rebase.

I pushed out the alpine updates, and am pushing out all of the other packages. Will update when done.

In the meantime, if you have ideas about how to make the alpine update better, all for it.

So if I understand it correctly the order of necessary steps would be this:

1. Build the new alpine image
2. Bump all alpine image references
3. Build the new tools images
4. Bump all tool image references
5. Build all package images
6. Bump all package image references

Of course if no changes to alpine or tools were made these steps can be omitted. They should probably be inserted into the ci.yml after the Cache Packages.

So the main problem then is the multi-arch builds with buildx won't work locally without pushing first, right? So all other Dockerfiles referencing the new unpushed alpine tag won't work.

I can think of three solutions currently:

1. Separate the whole build process by architecture
2. Run a service container during the build that hosts a docker registry, that is also part of the current github action cache
3. Enable the containerd storage backend for docker, so local multi-arch images are supported (https://docs.docker.com/engine/storage/containerd/)

I think number 3 is probably the most promising, albeit being experimental.

I did the following locally and it worked:

cd tools/alpine

# build the alpine images
docker buildx build --platform linux/amd64,linux/arm64,linux/riscv64 --no-cache --load -t linuxkit/alpine:a5085830bac643ce1e2230d1675195807c9a55f6 .

# check the local image cache
docker image ls --tree -a
# should show something like this:
# IMAGE                                                      ID             DISK USAGE   CONTENT SIZE
# linuxkit/alpine:a5085830bac643ce1e2230d1675195807c9a55f6   9f652e7faaa7       3.91GB         2.93GB
# ├─ linux/amd64                                             2bee1befda0e       1.92GB          952MB
# ├─ linux/arm64                                             6e2487c7c90d        901MB          901MB
# └─ linux/riscv64                                           50e84056eb26       1.08GB         1.08GB

After that building Dockerfiles locally referencing linuxkit/alpine:a5085830bac643ce1e2230d1675195807c9a55f6 worked for me for all 3 platforms. I did it with a dummy Dockerfile only containing FROM ... and the following command:

docker buildx build --platform linux/amd64,linux/arm64,linux/riscv64 -f Dockerfile -t test .

# result:
# IMAGE                                                      ID             DISK USAGE   CONTENT SIZE
# test:latest                                                0d7df5a151ab       3.91GB         2.93GB
# ├─ linux/amd64                                             2bee1befda0e       1.92GB          952MB
# ├─ linux/arm64                                             6e2487c7c90d        901MB          901MB
# └─ linux/riscv64                                           50e84056eb26       1.08GB         1.08GB

This showed the same correct hashes as the base image which is the expected behavior.

Let me know what you think or if I even got the main problem.

There are a few issues that kind of mix together here. I will try and separate them out, and then address what you wrote.

• building takes time, especially with emulation.
• the build step for each alpine image actually affects the digest tag of the image; for all other images, you can get the tag without building

Let's start with the second. You wrote this:

1. Build the new alpine image
2. Bump all alpine image references
3. Build the new tools images
4. Bump all tool image references
5. Build all package images
6. Bump all package image references

It is slightly different.

1. Make your changes to the new alpine image and commit. This changes the linuxkit/alpine:<tag> tag, but we are not done.
2. Build the image for each architecture. This changes the files versions.<arch> for each arch, including the arch-specific tag (e.g. linuxkit/alpine:<tag>-arm64). Unlike other packages, the arch-specific tag for alpine is different per arch. Then you commit those.
3. Now you have the final linuxkit/alpine:<tag>. This opens up the next steps.
4. Bump all alpine image references. Commit.
5. Bump all tool image references. Commit.
6. Bump all package image references. Commit.

There are some test packages and such, but those are details.

Notice that you actually don't need to build anything in order to get the downstream info. If I update tools/containerd-dev without building, as long as I make the changes and commit, I can run lkt pkg show-tag tools/containerd-dev and I will know what downstream should have. The only exception is linuxkit/alpine, for which I need to complete the versions.<arch> in order to get the final tag, and that is only updated based on the build.

If we could find a way to calculate all of that without the full build of all arch, then the process becomes easier: make your changes in alpine, and everywhere downstream, commit, push out, then worry about builds.

The issue with "building takes time", is the reason we mostly push packages out by hand. If we didn't, the following would be the process:

1. I make a change to a package
2. I git push and open a PR
3. CI picks it up and builds the package (it already does this for most)
4. It might take a long time, eventually builds, tests pass
5. merge to master
6. New CI kicks off, builds same packages all over again, pushes them out

That is slow and quite wasteful, but not awful. The only time you really run into problems is when building the kernel, or when you update alpine, which means you changed a lot of downstream packages and rebuild them all.

Enable the containerd storage backend for docker, so local multi-arch images are supported

That is the reason that linuxkit has its own cache. When we first created it, docker didn't have the containerd storage backend at all. Then it was designed, planned, and slowly implemented. If we could rely on it always being there, and we could access the images to feed them into buildkit the way we do now with lkt cache images, the lkt cache could go away (and I would be quite pleased).

But I don't see how using docker containerd cache vs lkt cache solves the issues?

So the main problem then is the multi-arch builds with buildx won't work locally without pushing first, right? So all other Dockerfiles referencing the new unpushed alpine tag won't work.

multi-arch builds with buildx do work. lkt pkg build just invokes buildkit in a docker-container driver, and saves the output directly to linuxkit's own cache (oci v1 layout). We save that cache as artifacts between jobs in the same workflow.

Actually, I don't think our CI builds tools (including alpine) at all. It builds packages here, but no tools. That might be useful.

BTW CI is almost green, we can merge it in after.

Merged in. We should continue the alpine+packages discussion though.