View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0009438ardourbugspublic2023-08-25 04:312024-12-06 22:19
Reporterergen Assigned To 
PrioritylowSeverityminorReproducibilitysometimes
Status newResolutionopen 
PlatformApple MacintoshOSMacOSOS Version10.12 or later
Product Version7.5 
Summary0009438: update ardour-build-tools/build-stack
Description- change zlib source to github
- run curl without -k if it fails initially
- build harfbuzz before freetype to prevent undefined HB_SCRIPT_ADLAM/HB_SCRIPT_OSAGE error
TagsNo tags attached.

Activities

ergen

2023-08-25 04:31

reporter  

0001-change-zlib-source-to-github-try-curl-without-k-buil.patch (4,111 bytes)    
From 147c5fccc1b0090d792150a6e00d50ab9c920de0 Mon Sep 17 00:00:00 2001
From: perry <pbtnguyen@gmail.com>
Date: Fri, 25 Aug 2023 00:28:03 -0400
Subject: [PATCH] change zlib source to github;try curl without -k;build
 harfbuzz before freetype

---
 build-stack | 78 ++++++++++++++++++++++++++---------------------------
 1 file changed, 39 insertions(+), 39 deletions(-)

diff --git a/build-stack b/build-stack
index 2fb1354..7cd0fe1 100755
--- a/build-stack
+++ b/build-stack
@@ -69,7 +69,7 @@ function fetch_as () {
     fi
     echo "--- Downloading.. $2"
     if uname -a | grep --silent arwin ; then
-        curl -L -k -o $1 $2
+        curl -L -k -o $1 $2 || curl -L -o $1 $2
     else
         wget --no-check-certificate -O $1 $2
     fi
@@ -346,7 +346,7 @@ defmod 'glib' '2.56.4' 'http://ftp.gnome.org/pub/gnome/sources/glib/2.56/' 'xz'
 defmod 'pkgconfig' 'pkg-config' '0.29.2' 'http://pkgconfig.freedesktop.org/releases/' 'gz'
 defmod 'termcap' '1.3.1' 'http://ftpmirror.gnu.org/termcap/' 'gz'
 defmod 'readline' '8.2' 'http://ftpmirror.gnu.org/readline/' 'gz'
-defmod 'zlib' '1.2.13' 'https://zlib.net/' 'gz'
+defmod 'zlib' '1.2.13' 'https://github.com/madler/zlib/releases/download/v1.2.13/' 'gz'
 defmod 'libiconv' '1.16' 'https://ftp.gnu.org/pub/gnu/libiconv/' 'gz'
 defmod 'gettext' '0.21' 'http://ftpmirror.gnu.org/gettext/' 'gz'
 defmod 'expat' '2.5.0' 'https://downloads.sourceforge.net/project/expat/expat/2.5.0/' 'gz'
@@ -905,6 +905,43 @@ else
   basic_builder pixman
 fi
 
+tar xf $MOD_harfbuzz_local_tarball && \
+(
+  set -e
+  cd $MOD_harfbuzz_unpacked
+  # work around clang Werror,Wunused-but-set-variable
+  patch -p1 << EOF
+diff --git a/src/hb-subset-cff1.cc b/src/hb-subset-cff1.cc
+index f3ed518..029ccf6 100644
+--- a/src/hb-subset-cff1.cc
++++ b/src/hb-subset-cff1.cc
+@@ -402,12 +402,11 @@ struct cff_subset_plan {
+   void plan_subset_encoding (const OT::cff1::accelerator_subset_t &acc, hb_subset_plan_t *plan)
+   {
+     const Encoding *encoding = acc.encoding;
+-    unsigned int  size0, size1, supp_size;
++    unsigned int  size0, size1;
+     hb_codepoint_t  code, last_code = CFF_UNDEF_CODE;
+     hb_vector_t<hb_codepoint_t> supp_codes;
+ 
+     subset_enc_code_ranges.resize (0);
+-    supp_size = 0;
+     supp_codes.init ();
+ 
+     subset_enc_num_codes = plan->num_output_glyphs () - 1;
+@@ -443,7 +442,6 @@ struct cff_subset_plan {
+ 	  code_pair_t pair = { supp_codes[i], sid };
+ 	  subset_enc_supp_codes.push (pair);
+ 	}
+-	supp_size += SuppEncoding::static_size * supp_codes.length;
+       }
+     }
+     supp_codes.fini ();
+EOF
+
+  basic_buildonly harfbuzz
+) || die "harfbuzz build failed"
+
 if [ x$OSX = x ] ; then
   echo "=== freetype (w/bytecode patch)"
   tar xf $MOD_freetype_local_tarball && \
@@ -974,43 +1011,6 @@ else
   ) || die "cairo build failed"
 fi
 
-tar xf $MOD_harfbuzz_local_tarball && \
-(
-  set -e
-  cd $MOD_harfbuzz_unpacked
-  # work around clang Werror,Wunused-but-set-variable
-  patch -p1 << EOF
-diff --git a/src/hb-subset-cff1.cc b/src/hb-subset-cff1.cc
-index f3ed518..029ccf6 100644
---- a/src/hb-subset-cff1.cc
-+++ b/src/hb-subset-cff1.cc
-@@ -402,12 +402,11 @@ struct cff_subset_plan {
-   void plan_subset_encoding (const OT::cff1::accelerator_subset_t &acc, hb_subset_plan_t *plan)
-   {
-     const Encoding *encoding = acc.encoding;
--    unsigned int  size0, size1, supp_size;
-+    unsigned int  size0, size1;
-     hb_codepoint_t  code, last_code = CFF_UNDEF_CODE;
-     hb_vector_t<hb_codepoint_t> supp_codes;
- 
-     subset_enc_code_ranges.resize (0);
--    supp_size = 0;
-     supp_codes.init ();
- 
-     subset_enc_num_codes = plan->num_output_glyphs () - 1;
-@@ -443,7 +442,6 @@ struct cff_subset_plan {
- 	  code_pair_t pair = { supp_codes[i], sid };
- 	  subset_enc_supp_codes.push (pair);
- 	}
--	supp_size += SuppEncoding::static_size * supp_codes.length;
-       }
-     }
-     supp_codes.fini ();
-EOF
-
-  basic_buildonly harfbuzz
-) || die "harfbuzz build failed"
-
 basic_builder fribidi
 
 GLOBAL_CONF="$GLOBAL_CONF $PANGO_CONF --with-included-modules=yes" \
-- 
2.39.2 (Apple Git-143)
0001-change-zlib-source-to-github-try-curl-without-k-buil.patch (4,111 bytes)   

x42

2023-08-25 04:53

administrator   ~0027995

can you elaborate on why you swapped harfbuzz and freetype?

Also did you mean
curl -L -o $1 $2 || curl -L -k -o $1 $2


IIRC insecure was for old OSX builders, which did not have updated certs. So we can probably get rid of that, too.

ergen

2023-09-22 20:01

reporter   ~0028099

I was getting error 'undefined "HB_SCRIPT_ADLAM"/"HB_SCRIPT_OSAGE"' when building freetype. I believe the issue is that freetype is expecting those symbols, which are defined by harfbuzz, which in the original build script is built after freetype. When I built harfbuzz before freetype the error went away.

after I did this i read that freetype is not actually even used on macos https://ardour.org/current_dependencies.html -- if this is true should freetype just be removed from the build stack for macos? maybe harfbuzz too?

your suggestion regarding curl definitely seems better practice than mine, i was mainly just trying to introduce the least amount of change in behavior. i'll try building fresh without inseucre and update the patch accordingly.

x42

2023-09-23 12:21

administrator   ~0028101

I have applied the curl change. I'm still curious about the freetype issue. There is circular dependency: freetype can use harfbuzz and harfbuzz can use freetype. In our case this is a non-issue since freetype is not directly used and text rendering happens via pango/cairo -> harfbuzz -> freetype/CoreFont.

The build script runs regularly and on different systems for nightly and local builds on High Siearra (Intel), Mojave (Intel) , BigSur (Intel and M1) and Venuta (M2).

So it's curious that you ran into those issues, while it works otherwis

ergen

2024-12-06 05:40

reporter   ~0029141

please update expat to 2.6.4 -- 2.6.3 has been marked vulnerable on sourceforge and has been removed.

ergen

2024-12-06 08:37

reporter   ~0029142

sources relying on http://ftp.acc.umu.se/pub/gnome/sources need to be updated as well (403 forbidden): gnomecommon, gnomedocutils and pangomm.

I caught these during the build since they failed to be extracted--I think they ought to be checked before building at least cursorily, it saves time. I patched `fetch_as` to check if downloaded packages are HTML documents and delete&fail the build if so.
0001-update-expat-to-2.6.4-and-gnomecommon-gnomedocutils-.patch (2,703 bytes)    
From f29c32f60bfd8e4c773d9265a86dcbe404cd093b Mon Sep 17 00:00:00 2001
From: perry <pbtnguyen@gmail.com>
Date: Fri, 6 Dec 2024 03:26:17 -0500
Subject: [PATCH 1/2] update expat to 2.6.4 and gnomecommon, gnomedocutils and
 pangomm sources

---
 build-stack | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/build-stack b/build-stack
index 224ebe2..bd58672 100755
--- a/build-stack
+++ b/build-stack
@@ -360,15 +360,15 @@ defmod 'readline' '8.2' 'http://ftpmirror.gnu.org/readline/' 'gz'
 defmod 'zlib' '1.3.1' 'https://zlib.net/' 'gz'
 defmod 'libiconv' '1.16' 'https://ftp.gnu.org/pub/gnu/libiconv/' 'gz'
 defmod 'gettext' '0.22' 'http://ftpmirror.gnu.org/gettext/' 'gz'
-defmod 'expat' '2.6.3' 'https://downloads.sourceforge.net/project/expat/expat/2.6.3/' 'xz'
+defmod 'expat' '2.6.4' 'https://downloads.sourceforge.net/project/expat/expat/2.6.4/' 'xz'
 defmod 'libxml2' '2.9.12' 'http://xmlsoft.org/download/' 'gz'
 defmod 'libxslt' '1.1.34' 'http://xmlsoft.org/download/' 'gz'
 defmod 'tiff' '4.0.3' 'http://download.osgeo.org/libtiff/old/' 'gz'
 defmod 'libpng' '1.6.37' 'https://downloads.sourceforge.net/project/libpng/libpng16/1.6.37/' 'xz'
 defmod 'jpeg' 'jpegsrc.v9a' '' 'http://www.ijg.org/files/' 'gz' 'jpeg-9a'
 defmod 'itstool' '2.0.7' 'http://files.itstool.org/itstool/' bz2
-defmod 'gnomecommon' 'gnome-common' '2.34.0' 'http://ftp.acc.umu.se/pub/gnome/sources/gnome-common/2.34/' 'bz2'
-defmod 'gnomedocutils' 'gnome-doc-utils' '0.20.10' 'http://ftp.acc.umu.se/pub/gnome/sources/gnome-doc-utils/0.20/' 'xz'
+defmod 'gnomecommon' 'gnome-common' '2.34.0' 'https://download.gnome.org/sources/gnome-common/2.34/' 'bz2'
+defmod 'gnomedocutils' 'gnome-doc-utils' '0.20.10' 'https://download.gnome.org/sources/gnome-doc-utils/0.20/' 'xz'
 defmod 'pixman' '0.42.2' 'http://www.cairographics.org/releases/' 'gz'
 defmod 'libffi' '3.4.2' 'https://github.com/libffi/libffi/releases/download/v3.4.2/' 'gz'
 defmod 'freetype' '2.9' 'https://downloads.sourceforge.net/project/freetype/freetype2/2.9/' 'bz2'
@@ -384,7 +384,7 @@ defmod boost boost_1_68_0 '' http://sourceforge.net/projects/boost/files/boost/1
 defmod fftw 3.3.8 http://fftw.org/  gz
 defmod glibmm 2.56.1 https://download.gnome.org/sources/glibmm/2.56/ xz
 defmod cairomm 1.12.2 http://cairographics.org/releases/ gz
-defmod pangomm 2.42.0 http://ftp.acc.umu.se/pub/gnome/sources/pangomm/2.42/ xz
+defmod pangomm 2.42.0 https://download.gnome.org/sources/pangomm/2.42/ xz
 defmod libart_lgpl 2.3.21 http://ftp.gnome.org/pub/gnome/sources/libart_lgpl/2.3/ bz2
 defmod liblo 0.28 http://downloads.sourceforge.net/liblo/ gz
 defmod raptor2 2.0.14 http://download.librdf.org/source/ gz
-- 
2.24.3 (Apple Git-128)
0001-update-expat-to-2.6.4-and-gnomecommon-gnomedocutils-.patch (2,703 bytes)   
0002-fail-build-if-downloaded-or-cached-file-is-an-HTML-d.patch (1,074 bytes)    
From f577c268b00f6cc55b162b852ee687663a2f1568 Mon Sep 17 00:00:00 2001
From: perry <pbtnguyen@gmail.com>
Date: Fri, 6 Dec 2024 03:27:04 -0500
Subject: [PATCH 2/2] fail build if downloaded or cached file is an HTML
 document

---
 build-stack | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)

diff --git a/build-stack b/build-stack
index bd58672..b9a92e6 100755
--- a/build-stack
+++ b/build-stack
@@ -63,14 +63,17 @@ function die () {
 }
 
 function fetch_as () {
-    if [ -f $1 ] ; then
-        return
+    if ! [ -f $1 ] ; then
+        echo "--- Downloading.. $2"
+        if uname -a | grep --silent arwin ; then
+            curl -L -o $1 $2 || curl -L -k -o $1 $2
+        else
+            wget --no-check-certificate -O $1 $2
+        fi
     fi
-    echo "--- Downloading.. $2"
-    if uname -a | grep --silent arwin ; then
-        curl -L -o $1 $2 || curl -L -k -o $1 $2
-    else
-        wget --no-check-certificate -O $1 $2
+    if file $1 | grep -q HTML ; then
+       rm $1
+       return 1
     fi
 }
 
-- 
2.24.3 (Apple Git-128)
0002-fail-build-if-downloaded-or-cached-file-is-an-HTML-d.patch (1,074 bytes)   

ergen

2024-12-06 15:56

reporter   ~0029145

(I meant delete the package, not the build)

x42

2024-12-06 18:13

administrator   ~0029146

Thanks. I've applied the first patch,

but we cannot rely on "file" -- that is not available on our nightly build systems, and adding it to our build-stack seems unreasonable.
Maybe check "$?" (curl's error code)?

x42

2024-12-06 18:14

administrator   ~0029147

Here it's not usually an issue since sources, once downloaded, are cached

ergen

2024-12-06 22:19

reporter   ~0029149

$? appears to return 0 even on 403, but I found you can pass -f to curl to fail on 403 and other server errors, patch attached. I don't currently have access to test wget though.

With this curl flag, invalid html docs shouldn't be cached anymore; if necessary I think it is fine to just manually run " find /var/tmp/src_cache -exec sh -c 'file {} | grep -q HTML && rm {}' \; ".
fail-curl-on-403.patch (401 bytes)    
diff --git a/build-stack b/build-stack
index bd58672..f53ed81 100755
--- a/build-stack
+++ b/build-stack
@@ -68,7 +68,7 @@ function fetch_as () {
     fi
     echo "--- Downloading.. $2"
     if uname -a | grep --silent arwin ; then
-        curl -L -o $1 $2 || curl -L -k -o $1 $2
+        curl -f -L -o $1 $2 || curl -f -L -k -o $1 $2
     else
         wget --no-check-certificate -O $1 $2
     fi
fail-curl-on-403.patch (401 bytes)   

Issue History

Date Modified Username Field Change
2023-08-25 04:31 ergen New Issue
2023-08-25 04:31 ergen File Added: 0001-change-zlib-source-to-github-try-curl-without-k-buil.patch
2023-08-25 04:53 x42 Note Added: 0027995
2023-09-22 20:01 ergen Note Added: 0028099
2023-09-23 12:21 x42 Note Added: 0028101
2024-12-06 05:40 ergen Note Added: 0029141
2024-12-06 08:37 ergen Note Added: 0029142
2024-12-06 08:37 ergen File Added: 0001-update-expat-to-2.6.4-and-gnomecommon-gnomedocutils-.patch
2024-12-06 08:37 ergen File Added: 0002-fail-build-if-downloaded-or-cached-file-is-an-HTML-d.patch
2024-12-06 15:56 ergen Note Added: 0029145
2024-12-06 18:13 x42 Note Added: 0029146
2024-12-06 18:14 x42 Note Added: 0029147
2024-12-06 22:19 ergen Note Added: 0029149
2024-12-06 22:19 ergen File Added: fail-curl-on-403.patch