Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update install_protoc.sh script to support apple architectures #732

Merged

Conversation

Crazyglue
Copy link
Contributor

Description

Upon pulling a fresh copy of the repo and executing make image/build the install_protoc.sh step fails with the following error/output:

docker build . -f Dockerfile -t kubeflow/model-registry:main
[+] Building 54.2s (23/26)                                                     docker:desktop-linux
 => [internal] load build definition from Dockerfile                                           0.0s
 => => transferring dockerfile: 1.86kB                                                         0.0s
 => [internal] load metadata for registry.access.redhat.com/ubi8/ubi-minimal:latest            2.3s
 => [internal] load metadata for registry.access.redhat.com/ubi8/go-toolset:1.22               2.8s
 => [internal] load .dockerignore                                                              0.0s
 => => transferring context: 89B                                                               0.0s
 => [builder  1/19] FROM registry.access.redhat.com/ubi8/go-toolset:1.22@sha256:9b7b1a47d31d  11.5s
 => => resolve registry.access.redhat.com/ubi8/go-toolset:1.22@sha256:9b7b1a47d31de7c15966a1c  0.0s
 => => sha256:fca967375d30048d40b3bcce430254d923336ce40617c1e782320c7caf 109.28MB / 109.28MB  10.5s
 => => sha256:201ab9feeb208eea05bc5ab3cdbb3b6fbd6b81ce58f540105286a023cc7 147.38MB / 147.38MB  7.5s
 => => sha256:43c5f145da5695cc536035c1f66d893b88261e4dbac8aceb94c3b4692812f 18.30MB / 18.30MB  5.8s
 => => sha256:bf4c3401d54802c092d22292f1fe5a29cbca5cf11f35cf559898cbf01cb69 76.90MB / 76.90MB  5.1s
 => => extracting sha256:bf4c3401d54802c092d22292f1fe5a29cbca5cf11f35cf559898cbf01cb69350      0.9s
 => => extracting sha256:43c5f145da5695cc536035c1f66d893b88261e4dbac8aceb94c3b4692812f5b1      0.2s
 => => extracting sha256:201ab9feeb208eea05bc5ab3cdbb3b6fbd6b81ce58f540105286a023cc756989      1.7s
 => => extracting sha256:fca967375d30048d40b3bcce430254d923336ce40617c1e782320c7caf976250      1.0s
 => [stage-1 1/3] FROM registry.access.redhat.com/ubi8/ubi-minimal:latest@sha256:cf095e566891  2.0s
 => => resolve registry.access.redhat.com/ubi8/ubi-minimal:latest@sha256:cf095e5668919ba1b4ac  0.0s
 => => sha256:2f831c9d38e67cb7ea03b615d2158a73e5018371f79337ecc92342aa32ab2 37.71MB / 37.71MB  1.6s
 => => extracting sha256:2f831c9d38e67cb7ea03b615d2158a73e5018371f79337ecc92342aa32ab2901      0.4s
 => [internal] load build context                                                              0.1s
 => => transferring context: 2.36MB                                                            0.1s
 => [builder  2/19] WORKDIR /workspace                                                         0.3s
 => [builder  3/19] COPY [go.mod, go.sum, ./]                                                  0.0s
 => [builder  4/19] RUN go mod download                                                       15.0s
 => [builder  5/19] RUN yum remove -y nodejs npm                                               1.0s
 => [builder  6/19] RUN yum module -y reset nodejs                                             4.8s
 => [builder  7/19] RUN yum module -y enable nodejs:18                                         0.4s 
 => [builder  8/19] RUN yum install -y nodejs npm java-11 python3                             17.7s 
 => [builder  9/19] COPY [Makefile, main.go, .openapi-generator-ignore, openapitools.json, ./  0.1s 
 => [builder 10/19] COPY cmd/ cmd/                                                             0.0s 
 => [builder 11/19] COPY api/ api/                                                             0.0s 
 => [builder 12/19] COPY internal/ internal/                                                   0.0s 
 => [builder 13/19] COPY scripts/ scripts/                                                     0.0s 
 => [builder 14/19] COPY pkg/ pkg/                                                             0.0s 
 => [builder 15/19] COPY patches/ patches/                                                     0.0s
 => [builder 16/19] COPY templates/ templates/                                                 0.0s
 => ERROR [builder 17/19] RUN make deps                                                        0.4s
------                                                                                              
 > [builder 17/19] RUN make deps:                                                                   
0.079 ./scripts/install_protoc.sh                                                                   
0.081 ++ uname -s                                                                                   
0.081 + OSTYPE=Linux                                                                                
0.081 + OS=linux                                                                                    
0.081 + [[ linux =~ darwin ]]
0.081 ++ uname -m
0.082 + ARCH=aarch64
0.082 + [[ aarch64 == \a\r\m* ]]
0.082 + [[ aarch64 == \s\3\9\0\x ]]
0.082 + [[ aarch64 == \p\p\c\6\4\l\e ]]
0.082 +++ dirname ./scripts/install_protoc.sh
0.083 ++ realpath ./scripts/..
0.083 + PROJECT_ROOT=/workspace
0.083 + VERSION=24.3
0.083 + URL=https://github.com/protocolbuffers/protobuf/releases/download/v24.3/protoc-24.3-linux-aarch64.zip
0.083 + wget -qx https://github.com/protocolbuffers/protobuf/releases/download/v24.3/protoc-24.3-linux-aarch64.zip -O /workspace/protoc.zip
0.352 make: *** [Makefile:133: bin/protoc] Error 8
------
Dockerfile:34
--------------------
  32 |     
  33 |     # Download tools
  34 | >>> RUN make deps
  35 |     
  36 |     # Build
--------------------
ERROR: failed to solve: process "/bin/sh -c make deps" did not complete successfully: exit code: 2
make: *** [image/build] Error 1

After quickly glancing at the output and the URL being constructed to download protoc, this is the URL generated

https://github.com/protocolbuffers/protobuf/releases/download/v24.3/protoc-24.3-linux-aarch64.zip

For Apple M-series laptops, aarch64 should be aarch_64 for the URL to resolve correctly.

This PR addresses this by simply piggybacking off of the $ARCH check to set that variable correctly. Now it will check for arm* OR aarch64 and set the $ARCH variable to aarch_64.

How Has This Been Tested?

I tested this locally by execute the command as described above, making the change, then running the make image/build command again. The docker image builds with the following output:

docker build . -f Dockerfile -t kubeflow/model-registry:main
[+] Building 68.6s (27/27) FINISHED                                            docker:desktop-linux
 => [internal] load build definition from Dockerfile                                           0.0s
 => => transferring dockerfile: 1.86kB                                                         0.0s
 => [internal] load metadata for registry.access.redhat.com/ubi8/ubi-minimal:latest            0.2s
 => [internal] load metadata for registry.access.redhat.com/ubi8/go-toolset:1.22               1.7s
 => [internal] load .dockerignore                                                              0.0s
 => => transferring context: 89B                                                               0.0s
 => [builder  1/19] FROM registry.access.redhat.com/ubi8/go-toolset:1.22@sha256:9b7b1a47d31de  0.0s
 => => resolve registry.access.redhat.com/ubi8/go-toolset:1.22@sha256:9b7b1a47d31de7c15966a1c  0.0s
 => [internal] load build context                                                              0.0s
 => => transferring context: 17.79kB                                                           0.0s
 => CACHED [stage-1 1/3] FROM registry.access.redhat.com/ubi8/ubi-minimal:latest@sha256:cf095  0.0s
 => => resolve registry.access.redhat.com/ubi8/ubi-minimal:latest@sha256:cf095e5668919ba1b4ac  0.0s
 => CACHED [builder  2/19] WORKDIR /workspace                                                  0.0s
 => CACHED [builder  3/19] COPY [go.mod, go.sum, ./]                                           0.0s
 => CACHED [builder  4/19] RUN go mod download                                                 0.0s
 => CACHED [builder  5/19] RUN yum remove -y nodejs npm                                        0.0s
 => CACHED [builder  6/19] RUN yum module -y reset nodejs                                      0.0s
 => CACHED [builder  7/19] RUN yum module -y enable nodejs:18                                  0.0s
 => CACHED [builder  8/19] RUN yum install -y nodejs npm java-11 python3                       0.0s
 => CACHED [builder  9/19] COPY [Makefile, main.go, .openapi-generator-ignore, openapitools.j  0.0s
 => CACHED [builder 10/19] COPY cmd/ cmd/                                                      0.0s
 => CACHED [builder 11/19] COPY api/ api/                                                      0.0s
 => CACHED [builder 12/19] COPY internal/ internal/                                            0.0s
 => [builder 13/19] COPY scripts/ scripts/                                                     0.0s
 => [builder 14/19] COPY pkg/ pkg/                                                             0.0s
 => [builder 15/19] COPY patches/ patches/                                                     0.0s
 => [builder 16/19] COPY templates/ templates/                                                 0.0s
 => [builder 17/19] RUN make deps                                                             33.7s
 => [builder 18/19] RUN make clean build/prepare                                              29.2s 
 => [builder 19/19] RUN CGO_ENABLED=0 GOOS=linux GOARCH=arm64 make build/compile               3.2s 
 => [stage-1 2/3] COPY --from=builder /workspace/model-registry .                              0.0s 
 => exporting to image                                                                         0.5s 
 => => exporting layers                                                                        0.4s 
 => => exporting manifest sha256:e68df7cce414392a3cc65a56e6d1362d4e15dcfe30bf7e689ca824810936  0.0s 
 => => exporting config sha256:3a15c4aad5ce63170373a1c8583aa3d6b8c3d3250a5362d2f40f083d977cee  0.0s 
 => => exporting attestation manifest sha256:0032a30024e47dd216d352e903920f2516c22532d7b8d43a  0.0s
 => => exporting manifest list sha256:dc07617434293488db32b3cdc80cfb30ebf9e71ec3caa583f5a7213  0.0s
 => => naming to docker.io/kubeflow/model-registry:main                                        0.0s
 => => unpacking to docker.io/kubeflow/model-registry:main                                     0.1s

Merge criteria:

  • All the commits have been signed-off (To pass the DCO check)
  • [ x] The commits have meaningful messages; the author will squash them after approval or in case of manual merges will ask to merge with squash.
  • Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
  • [x ] The developer has manually tested the changes and verified that the changes work.
  • [ x] Code changes follow the kubeflow contribution guidelines.

If you have UI changes

  • The developer has added tests or explained why testing cannot be added.
  • Included any necessary screenshots or gifs if it was a UI change.
  • Verify that UI/UX changes conform the UX guidelines for Kubeflow.

@Crazyglue Crazyglue changed the title fix: update install_protoc.sh script to support apple architectures Update install_protoc.sh script to support apple architectures Jan 23, 2025
@Crazyglue Crazyglue force-pushed the fix/apple-m-install-protoc-fix branch from 3d2ec76 to 6c041a6 Compare January 23, 2025 18:06
@Crazyglue Crazyglue force-pushed the fix/apple-m-install-protoc-fix branch from 6c041a6 to 134a87f Compare January 23, 2025 18:10
Copy link
Member

@tarilabs tarilabs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you @Crazyglue !

we possibly lost this during some more recent work, and the Mac local development "path" is not tested via GHA, so great catch

/lgtm
/approve

Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tarilabs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot merged commit 62cfc06 into kubeflow:main Jan 23, 2025
14 checks passed
@Crazyglue Crazyglue deleted the fix/apple-m-install-protoc-fix branch January 23, 2025 18:21
@rareddy
Copy link
Contributor

rareddy commented Jan 23, 2025

Congratulations @Crazyglue on your first PR to Kubeflow!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants