Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider dropping a couple of packages from the git-sdk-* repositories #5395

Open
dscho opened this issue Feb 3, 2025 · 6 comments
Open

Comments

@dscho
Copy link
Member

dscho commented Feb 3, 2025

The git-sdk-* repositories hold fully-populated MSYS2 setups (with a couple of packages drawn from Git for Windows' rather than MSYS2's Pacman repositories). They are kept up to date by the sync workflows that update the Pacman packages and commit the result.

So what decided which packages are installed in those MSYS2 setups? The principal idea was that a regular Git for Windows release process should not need to install any additional packages; If it does, that's a bug. It was designed this way so that it is easy to "time-travel" to the point in time when a specific Git for Windows version was built, to be able to build security fix releases from that very point; This ensures that users who had the preceding version installed can upgrade to the security fix release with minimal risk of regressions.

And what about packages that are not required for building mingw-w64-git nor for any other part of releasing Git for Windows' installers, portable Gits, archives, NuGet packages and MinGits?

Today, I had the opportunity to have a closer look. Specifically, I looked at the difference between the packages recorded in /etc/package-versions.txt and the packages actually installed in git-sdk-64. Concretely, I looked at the output of this lengthy command-line:

$ diff -u \
  <(git -C / ls-tree github/main:var/lib/pacman/local/ |
    grep -ve ' blob ' |
    sed 's/.*\t\(.*\)-[0-9].*-[0-9]*$/\1/') \
 <(sed -n '/^mingw-w64-/d;s/\(.*\) [0-9].*-[0-9]*$/\1/p' \
    </usr/src/build-extra/versions/package-versions-2.47.1.2.txt |
    sort |
    uniq) |
  sed -n '1,3d;s/^-//p'
Full output
asciidoc
autoconf-wrapper
autoconf2.13
autoconf2.69
autoconf2.71
autoconf2.72
autogen
automake-wrapper
automake1.10
automake1.11
automake1.12
automake1.13
automake1.14
automake1.15
automake1.16
automake1.17
automake1.6
automake1.7
automake1.8
automake1.9
autotools
bash-completion
binutils
bison
bsdcpio
bsdtar
crypt
curl
diffstat
docbook-xml
docbook-xsl
docbook-xsl-ns
flex
gcc
gdb
gdbm
gettext
gettext-devel
git
git-for-windows-keyring
glib2
gnutls
groff
heimdal
help2man
icu
info
isl
libarchive
libasprintf
libatomic_ops
libgc
libgdbm
libgettextpo
libgpgme
libguile
libiconv-devel
libltdl
liblzo2
libpcre16
libpcre2_16
libpcre2_32
libpcre32
libpcrecpp
libpcreposix
libpipeline
libtool
libunrar
libxml2
libxslt
libxxhash
lndir
m4
make
man-db
markdown
mingw-w64-i686-asciidoctor
mingw-w64-i686-binutils
mingw-w64-i686-brotli
mingw-w64-i686-bzip2
mingw-w64-i686-c-ares
mingw-w64-i686-ca-certificates
mingw-w64-i686-crt-git
mingw-w64-i686-curl-openssl-alternate
mingw-w64-i686-curl-winssl
mingw-w64-i686-cv2pdb
mingw-w64-i686-expat
mingw-w64-i686-gcc
mingw-w64-i686-gcc-fortran
mingw-w64-i686-gcc-libgfortran
mingw-w64-i686-gcc-libs
mingw-w64-i686-gdb
mingw-w64-i686-gdbm
mingw-w64-i686-gettext-libtextstyle
mingw-w64-i686-gettext-runtime
mingw-w64-i686-gettext-tools
mingw-w64-i686-gmp
mingw-w64-i686-gnutls
mingw-w64-i686-headers-git
mingw-w64-i686-isl
mingw-w64-i686-jansson
mingw-w64-i686-jemalloc
mingw-w64-i686-libffi
mingw-w64-i686-libiconv
mingw-w64-i686-libidn2
mingw-w64-i686-libmangle-git
mingw-w64-i686-libpsl
mingw-w64-i686-libssh2-wincng
mingw-w64-i686-libsystre
mingw-w64-i686-libtasn1
mingw-w64-i686-libtre
mingw-w64-i686-libunistring
mingw-w64-i686-libwinpthread-git
mingw-w64-i686-libxml2
mingw-w64-i686-libyaml
mingw-w64-i686-make
mingw-w64-i686-mpc
mingw-w64-i686-mpdecimal
mingw-w64-i686-mpfr
mingw-w64-i686-ncurses
mingw-w64-i686-nettle
mingw-w64-i686-nghttp2
mingw-w64-i686-openssl
mingw-w64-i686-p11-kit
mingw-w64-i686-pcre
mingw-w64-i686-pcre2
mingw-w64-i686-pdcurses
mingw-w64-i686-pkgconf
mingw-w64-i686-python
mingw-w64-i686-readline
mingw-w64-i686-rtmpdump
mingw-w64-i686-ruby
mingw-w64-i686-spdylay
mingw-w64-i686-sqlite3
mingw-w64-i686-tcl
mingw-w64-i686-termcap
mingw-w64-i686-tk
mingw-w64-i686-tools-git
mingw-w64-i686-tzdata
mingw-w64-i686-windows-default-manifest
mingw-w64-i686-wineditline
mingw-w64-i686-winpthreads-git
mingw-w64-i686-xxhash
mingw-w64-i686-xz
mingw-w64-i686-zlib
mingw-w64-i686-zstd
mingw-w64-x86_64-7zip
mingw-w64-x86_64-antiword
mingw-w64-x86_64-asciidoctor
mingw-w64-x86_64-binutils
mingw-w64-x86_64-brotli
mingw-w64-x86_64-busybox
mingw-w64-x86_64-bzip2
mingw-w64-x86_64-c-ares
mingw-w64-x86_64-ca-certificates
mingw-w64-x86_64-connect
mingw-w64-x86_64-crt-git
mingw-w64-x86_64-curl-openssl-alternate
mingw-w64-x86_64-curl-winssl
mingw-w64-x86_64-cv2pdb
mingw-w64-x86_64-expat
mingw-w64-x86_64-gcc
mingw-w64-x86_64-gcc-libs
mingw-w64-x86_64-gdb
mingw-w64-x86_64-gdbm
mingw-w64-x86_64-gettext-libtextstyle
mingw-w64-x86_64-gettext-runtime
mingw-w64-x86_64-gettext-tools
mingw-w64-x86_64-git
mingw-w64-x86_64-git-credential-manager
mingw-w64-x86_64-git-doc-html
mingw-w64-x86_64-git-extra
mingw-w64-x86_64-git-lfs
mingw-w64-x86_64-gmp
mingw-w64-x86_64-gnutls
mingw-w64-x86_64-headers-git
mingw-w64-x86_64-isl
mingw-w64-x86_64-jansson
mingw-w64-x86_64-jemalloc
mingw-w64-x86_64-libdeflate
mingw-w64-x86_64-libffi
mingw-w64-x86_64-libiconv
mingw-w64-x86_64-libidn2
mingw-w64-x86_64-libmangle-git
mingw-w64-x86_64-libpsl
mingw-w64-x86_64-libssh2-wincng
mingw-w64-x86_64-libsystre
mingw-w64-x86_64-libtasn1
mingw-w64-x86_64-libtre
mingw-w64-x86_64-libunistring
mingw-w64-x86_64-libwinpthread-git
mingw-w64-x86_64-libxml2
mingw-w64-x86_64-libyaml
mingw-w64-x86_64-libzip
mingw-w64-x86_64-lzo2
mingw-w64-x86_64-mpc
mingw-w64-x86_64-mpdecimal
mingw-w64-x86_64-mpfr
mingw-w64-x86_64-ncurses
mingw-w64-x86_64-nettle
mingw-w64-x86_64-nghttp2
mingw-w64-x86_64-odt2txt
mingw-w64-x86_64-openssl
mingw-w64-x86_64-osslsigncode
mingw-w64-x86_64-p11-kit
mingw-w64-x86_64-pcre
mingw-w64-x86_64-pcre2
mingw-w64-x86_64-pdcurses
mingw-w64-x86_64-pkgconf
mingw-w64-x86_64-python
mingw-w64-x86_64-readline
mingw-w64-x86_64-rtmpdump
mingw-w64-x86_64-ruby
mingw-w64-x86_64-spdylay
mingw-w64-x86_64-sqlite3
mingw-w64-x86_64-tcl
mingw-w64-x86_64-termcap
mingw-w64-x86_64-tk
mingw-w64-x86_64-tools-git
mingw-w64-x86_64-tzdata
mingw-w64-x86_64-windows-default-manifest
mingw-w64-x86_64-wineditline
mingw-w64-x86_64-winpthreads-git
mingw-w64-x86_64-wintoast
mingw-w64-x86_64-xpdf-tools
mingw-w64-x86_64-xxhash
mingw-w64-x86_64-xz
mingw-w64-x86_64-zlib
mingw-w64-x86_64-zstd
mpc
mpdecimal
msys2-keyring
msys2-launcher
msys2-runtime-devel
msys2-w32api-headers
msys2-w32api-runtime
p7zip
pacman
pacman-contrib
pacman-mirrors
patchutils
pcre
pcre2
perl-Locale-Gettext
perl-Module-Build
perl-Test-Pod
perl-XML-Parser
perl-YAML-Syck
perl-inc-latest
pkgconf
pkgfile
python
rsync
swig
tcl
time
tzcode
unrar
windows-default-manifest
xmlto
xxhash
xz
zstd

I analyzed these manually and sorted them into the following categories:

Can probably be dropped

asciidoc
autoconf2.13
autoconf2.69
autoconf2.71
automake1.10
automake1.11
automake1.12
automake1.13
automake1.14
automake1.15
automake1.16
automake1.6
automake1.7
automake1.8
automake1.9
libatomic_ops
libgpgme
liblzo2
libpcre16
libpcre32
libpcrecpp
libpcreposix
libunrar
mingw-w64-i686-gcc-fortran
mingw-w64-i686-gcc-libgfortran
mingw-w64-i686-gnutls
mingw-w64-i686-jansson
mingw-w64-i686-jemalloc
mingw-w64-i686-libxml2
mingw-w64-i686-pcre
mingw-w64-i686-pdcurses
mingw-w64-i686-rtmpdump
mingw-w64-i686-spdylay
mingw-w64-i686-tools-git
mingw-w64-x86_64-jansson
mingw-w64-x86_64-jemalloc
mingw-w64-x86_64-libdeflate
mingw-w64-x86_64-libmangle-git
mingw-w64-x86_64-pdcurses
mingw-w64-x86_64-rtmpdump
mingw-w64-x86_64-spdylay
mingw-w64-x86_64-tools-git
p7zip
pcre
pkgfile
unrar

Are not strictly needed, but would be good to keep

In particular the autoconf and gcc stuff is required to build other packages than mingw-w64-git, i.e. for builds that happen more frequently than releasing Git for Windows versions. It would be good to avoid having to install them every time.

Also includes Python and the GNU debugger, but not Ruby (because that is needed to run AsciiDoctor, which is needed to build mingw-w64-git-doc-*).

Full list
autoconf-wrapper
autoconf2.72
autogen
automake-wrapper
automake1.17
autotools
binutils
bash-completion
bison
bsdcpio
bsdtar
crypt
curl
diffstat
flex
gcc
gdb
gdbm
gettext
gettext-devel
git
glib2
gnutls
heimdal
help2man
icu
info
isl
libarchive
libasprintf
libgc
libgdbm
libgettextpo
libguile
libiconv-devel
libltdl
libpcre2_16
libpcre2_32
libpipeline
libtool
libxxhash
lndir
m4
mingw-w64-i686-gdb
mingw-w64-i686-libsystre
mingw-w64-i686-libtre
mingw-w64-i686-mpdecimal
mingw-w64-i686-ncurses
mingw-w64-i686-pkgconf
mingw-w64-i686-python
mingw-w64-i686-sqlite3
mingw-w64-i686-tzdata
mingw-w64-i686-xxhash
mingw-w64-i686-xz
mingw-w64-x86_64-gdb
mingw-w64-x86_64-mpdecimal
mingw-w64-x86_64-ncurses
mingw-w64-x86_64-pkgconf
mingw-w64-x86_64-python
mingw-w64-x86_64-sqlite3
mingw-w64-x86_64-tzdata
mingw-w64-x86_64-xxhash
mpdecimal
msys2-runtime-devel
pkgconf
python
rsync
swig
tcl
time
tzcode
windows-default-manifest
xxhash

Must keep

These are either needed to build mingw-w64-git or to keep the Git SDK Bash running smoothly.

Full list
docbook-xml
docbook-xsl
docbook-xsl-ns
git-for-windows-keyring
groff
libxml2
libxslt
make
man-db
markdown
mingw-w64-i686-asciidoctor
mingw-w64-i686-binutils
mingw-w64-i686-brotli
mingw-w64-i686-bzip2
mingw-w64-i686-c-ares
mingw-w64-i686-ca-certificates
mingw-w64-i686-crt-git
mingw-w64-i686-curl-openssl-alternate
mingw-w64-i686-curl-winssl
mingw-w64-i686-cv2pdb
mingw-w64-i686-expat
mingw-w64-i686-gcc
mingw-w64-i686-gcc-libs
mingw-w64-i686-gdbm
mingw-w64-i686-gettext-libtextstyle
mingw-w64-i686-gettext-runtime
mingw-w64-i686-gettext-tools
mingw-w64-i686-gmp
mingw-w64-i686-headers-git
mingw-w64-i686-isl
mingw-w64-i686-libffi
mingw-w64-i686-libiconv
mingw-w64-i686-libidn2
mingw-w64-i686-libpsl
mingw-w64-i686-libssh2-wincng
mingw-w64-i686-libtasn1
mingw-w64-i686-libunistring
mingw-w64-i686-libwinpthread-git
mingw-w64-i686-libyaml
mingw-w64-i686-make
mingw-w64-i686-mpc
mingw-w64-i686-mpfr
mingw-w64-i686-nettle
mingw-w64-i686-nghttp2
mingw-w64-i686-openssl
mingw-w64-i686-p11-kit
mingw-w64-i686-pcre2
mingw-w64-i686-readline
mingw-w64-i686-ruby
mingw-w64-i686-tcl
mingw-w64-i686-termcap
mingw-w64-i686-tk
mingw-w64-i686-windows-default-manifest
mingw-w64-i686-wineditline
mingw-w64-i686-winpthreads-git
mingw-w64-i686-zlib
mingw-w64-i686-zstd
mingw-w64-x86_64-7zip
mingw-w64-x86_64-antiword
mingw-w64-x86_64-asciidoctor
mingw-w64-x86_64-binutils
mingw-w64-x86_64-brotli
mingw-w64-x86_64-busybox
mingw-w64-x86_64-bzip2
mingw-w64-x86_64-c-ares
mingw-w64-x86_64-ca-certificates
mingw-w64-x86_64-connect
mingw-w64-x86_64-crt-git
mingw-w64-x86_64-curl-openssl-alternate
mingw-w64-x86_64-curl-winssl
mingw-w64-x86_64-cv2pdb
mingw-w64-x86_64-expat
mingw-w64-x86_64-gcc
mingw-w64-x86_64-gcc-libs
mingw-w64-x86_64-gdbm
mingw-w64-x86_64-gettext-libtextstyle
mingw-w64-x86_64-gettext-runtime
mingw-w64-x86_64-gettext-tools
mingw-w64-x86_64-git
mingw-w64-x86_64-git-credential-manager
mingw-w64-x86_64-git-doc-html
mingw-w64-x86_64-git-extra
mingw-w64-x86_64-git-lfs
mingw-w64-x86_64-gmp
mingw-w64-x86_64-gnutls
mingw-w64-x86_64-headers-git
mingw-w64-x86_64-isl
mingw-w64-x86_64-libffi
mingw-w64-x86_64-libiconv
mingw-w64-x86_64-libidn2
mingw-w64-x86_64-libpsl
mingw-w64-x86_64-libssh2-wincng
mingw-w64-x86_64-libsystre
mingw-w64-x86_64-libtasn1
mingw-w64-x86_64-libtre
mingw-w64-x86_64-libunistring
mingw-w64-x86_64-libwinpthread-git
mingw-w64-x86_64-libxml2
mingw-w64-x86_64-libyaml
mingw-w64-x86_64-libzip
mingw-w64-x86_64-lzo2
mingw-w64-x86_64-mpc
mingw-w64-x86_64-mpfr
mingw-w64-x86_64-nettle
mingw-w64-x86_64-nghttp2
mingw-w64-x86_64-odt2txt
mingw-w64-x86_64-openssl
mingw-w64-x86_64-osslsigncode
mingw-w64-x86_64-p11-kit
mingw-w64-x86_64-pcre
mingw-w64-x86_64-pcre2
mingw-w64-x86_64-readline
mingw-w64-x86_64-ruby
mingw-w64-x86_64-tcl
mingw-w64-x86_64-termcap
mingw-w64-x86_64-tk
mingw-w64-x86_64-windows-default-manifest
mingw-w64-x86_64-wineditline
mingw-w64-x86_64-winpthreads-git
mingw-w64-x86_64-wintoast
mingw-w64-x86_64-xpdf-tools
mingw-w64-x86_64-xz
mingw-w64-x86_64-zlib
mingw-w64-x86_64-zstd
mpc
msys2-keyring
msys2-launcher
msys2-w32api-headers
msys2-w32api-runtime
pacman
pacman-contrib
pacman-mirrors
patchutils
pcre2
perl-Locale-Gettext
perl-XML-Parser
perl-YAML-Syck
perl-inc-latest
perl-Module-Build
perl-Test-Pod
xmlto
xz
zstd

I likely made a couple of mistakes, so take that categorization with a grain of salt and ensure to test a git-artifacts build!

Speaking of git-artifacts build: We probably want to make sure that the makepkg-mingw call does not pass the -s flag, which would also make for a fine opportunity to migrate the build-mingw-w64-git code from build-extra's please.sh to git-for-windows-automation, where it would find a more natural habitat.

@rimrul
Copy link
Member

rimrul commented Feb 3, 2025

asciidoc

I think we're using asciidoctor everywhere nowadays, so this should be fine.

autoconf2.13 autoconf2.69 autoconf2.71 automake1.10 automake1.11 automake1.12 automake1.13 automake1.14 automake1.15 automake1.16 automake1.6 automake1.7 automake1.8 automake1.9

Getting rid of old autoconf/automake versions should be fine.

libpcre16 libpcre32 libpcrecpp libpcreposix
mingw-w64-i686-pcre
pcre

PCRE1. We really don't need to keep this around.

mingw-w64-i686-gcc-fortran mingw-w64-i686-gcc-libgfortran

We've got rid of amd64 fortran years ago and don't need i686 fortran either.

git-for-windows/git-sdk-64@3764e6c

libgpgme

doesn't pacman depend on this?

@dscho
Copy link
Member Author

dscho commented Feb 4, 2025

asciidoc

I think we're using asciidoctor everywhere nowadays, so this should be fine.

Right, we did use it at some point early on, I think, but then never dropped it from the git-sdk-* repositories. The early Windows/ARM64 builds of mingw-w64-git used mingw-w64-asciidoc instead (see git-for-windows/git-sdk-arm64@5d9adfd) and even there we use asciidoctor now (see git-for-windows/git-sdk-arm64@b47a1fa). We'll need to remember to drop mingw-w64-clang-aarch64-asciidoc in git-sdk-arm64...

libpcre16 libpcre32 libpcrecpp libpcreposix
mingw-w64-i686-pcre
pcre

PCRE1. We really don't need to keep this around.

Note, though:

$ pactree -r mingw-w64-x86_64-pcre
mingw-w64-x86_64-pcre
└─mingw-w64-x86_64-odt2txt 

odt2txt is needed to make the .odt-related /etc/gitattributes functional (see the corresponding parts of /etc/gitconfig and of astextplain). Therefore we need mingw-w64-i686-pcre at least in git-sdk-32's build-installers sparse-checkout definition.

mingw-w64-i686-gcc-fortran mingw-w64-i686-gcc-libgfortran

We've got rid of amd64 fortran years ago and don't need i686 fortran either.

git-for-windows/git-sdk-64@3764e6c

Great find!

libgpgme

doesn't pacman depend on this?

No, it does not:

$ pactree -r libgpgme
libgpgme

pacman.exe does link to libgpgme (see here and here), but statically.

@rimrul
Copy link
Member

rimrul commented Feb 4, 2025

$ pactree -r mingw-w64-x86_64-pcre mingw-w64-x86_64-pcre └─mingw-w64-x86_64-odt2txt

We could update that patch to use PCRE2.

That reminds me, docx2txt is implemented as a perl script.

@dscho
Copy link
Member Author

dscho commented Feb 4, 2025

We could update that patch to use PCRE2.

I guess that should work!

That reminds me, docx2txt is implemented as a perl script.

Ouch, you're right. I guess we'd have to find an alternative (or ask Devin to transpile it to C or Rust 😉)

@rimrul
Copy link
Member

rimrul commented Feb 5, 2025

or ask Devin to transpile it to C or Rust 😉

I have no clue who Devin is. If you're talking about the odt2txt developer, he's called Dennis, and we haven't seen any activity since ~2018. Also he's not involved with docx2txt.

If you're talking about the docx2txt developer, he's called Sandeep and we haven't seen any activity since ~2016.

@dscho
Copy link
Member Author

dscho commented Feb 5, 2025

or ask Devin to transpile it to C or Rust 😉

I have no clue who Devin is.

Sorry, I had misspelled the name. It was a joke, I was referring to https://devin.ai/ :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants