Skip to content

GH-743: Exclude jni/ from the source-release zip #1383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ump45nose wants to merge 1 commit into
base: main
Choose a base branch
from
Open

GH-743: Exclude jni/ from the source-release zip #1383

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 105 additions & 0 deletions dev/release/source-release.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

<!--
Assembly descriptor for the arrow-java-root-*-source-release.zip artifact
built by the apache-release profile.

This is the same as the "source-release" descriptor shipped in
org.apache.apache.resources:apache-source-release-assembly-descriptor:1.8
(used by the Apache parent POM), except that it also excludes jni/.
-->
<assembly xmlns="http://maven.apache.org/ASSEMBLY/2.2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.2.0 https://maven.apache.org/xsd/assembly-2.2.0.xsd">
<id>source-release</id>
<formats>
<format>zip</format>
</formats>
<fileSets>
<!-- main project directory structure -->
<fileSet>
<directory>.</directory>
<outputDirectory></outputDirectory>
<useDefaultExcludes>true</useDefaultExcludes>
<excludes>

@jbonofre jbonofre Nov 6, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same job leaves a few other non source files in the working directory. They are also in the 19.0.0 zip and are not covered here:

  • apache-arrow-java-19.0.0.tar.gz: the git archive source release, nested inside this one
  • artifacts/release-source/*.sha256 and *.sha512, plus empty artifacts/jni-*/ directories
  • an empty binaries/ directory: the dist dir jni_full_build.sh creates before running Maven

We could add an exclude for each of these, but then the list has to track whatever the CI job happens to leave behind.

<!-- Pre-built JNI libraries that release CI puts in jni/ before
building the release artifacts. They are binaries, not source,
and are several hundred MB. See GH-743. -->
<exclude>jni/**</exclude>

@jbonofre jbonofre Nov 6, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This covers the extracted jni/ directory, but the binaries job in rc.yml also leaves the compressed tarballs at the repo root and never deletes them:

mv artifacts/*/*.tar.gz .
tar -xf apache-arrow-java-*.tar.gz --strip-components=1
tar -xvzf jni-linux-x86_64.tar.gz
...

So they are still packaged. From the published 19.0.0 zip:

61.6 MB  jni-linux-aarch_64.tar.gz
66.0 MB  jni-linux-x86_64.tar.gz
36.5 MB  jni-macos-aarch_64.tar.gz
44.8 MB  jni-macos-x86_64.tar.gz
14.6 MB  jni-windows-x86_64.tar.gz

With the current matrix, the next RC zip would still carry about 100MB of the same native libraries, just tarred. The local test with a fake jni/ directory doesn't reproduce the CI layout, which is why this didn't show up.

NB: I don't think we should consider that for 20.0.0, we can do the cleanup after this release.


<!-- build output -->
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?${project.build.directory}(/.*)?]</exclude>

<!-- NOTE: Most of the following excludes should not be required
if the standard release process is followed. This is because the
release plugin checks out project sources into a location like
target/checkout, then runs the build from there. The result is
a source-release archive that comes from a pretty clean directory
structure.

HOWEVER, if the release plugin is configured to run extra goals
or generate a project website, it's definitely possible that some
of these files will be present. So, it's safer to exclude them.
-->

<!-- IDEs -->
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?maven-eclipse\.xml]</exclude>
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.project]</exclude>
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.classpath]</exclude>
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.iws]</exclude>
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.idea(/.*)?]</exclude>
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?out(/.*)?]</exclude>
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.ipr]</exclude>
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.iml]</exclude>
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.settings(/.*)?]</exclude>
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.externalToolBuilders(/.*)?]</exclude>
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.deployables(/.*)?]</exclude>
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.wtpmodules(/.*)?]</exclude>

<!-- CI -->
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.github(/.*)?]</exclude>
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.Jenkinsfile]</exclude>
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.asf\.yaml]</exclude>
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.travis\.yml]</exclude>
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?deploySite\.sh]</exclude>

<!-- misc -->
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?cobertura\.ser]</exclude>

<!-- develocity-maven-extension working files -->
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.mvn/\.develocity(/.*)?]</exclude>

<!-- release-plugin temp files -->
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?pom\.xml\.releaseBackup]</exclude>
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?release\.properties]</exclude>

<!-- maven-shade-plugin -->
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?dependency-reduced-pom\.xml]</exclude>

<!-- flatten-maven-plugin -->
<exclude>%regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.flattened-pom\.xml]</exclude>
</excludes>
</fileSet>
<!-- license, readme, etc. calculated at build time -->
<fileSet>
<directory>${project.build.directory}/maven-shared-archive-resources/META-INF</directory>
<outputDirectory></outputDirectory>
</fileSet>
</fileSets>
</assembly>
26 changes: 26 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -786,6 +786,32 @@ under the License.
</reporting>

<profiles>
<profile>
<id>apache-release</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<id>source-release-assembly</id>
<configuration>
<!-- Use our own descriptor instead of the Apache parent's
"source-release" one so that pre-built JNI libraries
in jni/ aren't included in the source release. -->
<descriptorRefs combine.self="override"/>

@jbonofre jbonofre Nov 6, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than forking the descriptor, I would fix this where the files come from. The source-release descriptor zips the whole working directory, and the binaries job unpacks its downloads into that same directory. Two options:

  1. Build from a clean subdirectory in rc.yml, so downloads and outputs sit next to the source tree instead of inside it:
mv artifacts/*/*.tar.gz .
mkdir arrow-java
tar -xf apache-arrow-java-*.tar.gz -C arrow-java --strip-components=1
tar -xvzf jni-linux-x86_64.tar.gz
...

then run ci/scripts/jni_full_build.sh arrow-java jni binaries, checkout apache/arrow-testing to arrow-java/testing, and point "Prepare docs" at arrow-java/target/site/apidocs. That handles the tarballs, artifacts/, binaries/ and the nested source tarball in one go.

  1. Set <skipAssembly>true</skipAssembly> on this execution, as bom/pom.xml already does. The source release we vote on and publish to dist.apache.org is the git archive tarball.

Either way we drop dev/release/source-release.xml. Otherwise we would have to keep it in sync with the upstream descriptor on every org.apache:apache parent bump. 19.0.0 already leaked .mvn/.develocity/ because of that kind of drift.

<descriptors>
<descriptor>${maven.multiModuleProjectDirectory}/dev/release/source-release.xml</descriptor>
</descriptors>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>

<profile>
<!-- Certain flags are required on newer JDK but not recognized on earlier JDK -->
<id>jdk24</id>
Expand Down
Loading