Merge jemalloc 5.3.0

RELNOTES

@@ -10,6 +10,10 @@ newline.  Entries should be separated by a newline.
 
 Changes to this file should not be MFCed.
 
+c43cad871720:
+	jemalloc 5.3.0 has landed. See contrib/jemalloc/ChangeLog
+	for the long list of changes.
+
 cce64f2e6851:
 	Add support for the NFSv4.2 Clone operation, which uses
 	block cloning to "copy on write" files on an NFS server.

UPDATING

@@ -27,6 +27,9 @@ NOTE TO PEOPLE WHO THINK THAT FreeBSD 15.x IS SLOW:
 	world, or to merely disable the most expensive debugging functionality
 	at runtime, run "ln -s 'abort:false,junk:false' /etc/malloc.conf".)
 
+20250815:
+	jemalloc 5.3.0 has been committed to the tree.
+
 20250815:
 	The removal of Secure RPC DES authentication notced in 20250810
 	has been reverted.  (However, it is still non-functional.)

contrib/jemalloc/ChangeLog

@@ -4,6 +4,106 @@ brevity.  Much more detail can be found in the git revision history:
 
     https://github.com/jemalloc/jemalloc
 
+* 5.3.0 (May 6, 2022)
+
+  This release contains many speed and space optimizations, from micro
+  optimizations on common paths to rework of internal data structures and
+  locking schemes, and many more too detailed to list below.  Multiple percent
+  of system level metric improvements were measured in tested production
+  workloads.  The release has gone through large-scale production testing.
+
+  New features:
+  - Add the thread.idle mallctl which hints that the calling thread will be
+    idle for a nontrivial period of time.  (@davidtgoldblatt)
+  - Allow small size classes to be the maximum size class to cache in the
+    thread-specific cache, through the opt.[lg_]tcache_max option.  (@interwq,
+    @jordalgo)
+  - Make the behavior of realloc(ptr, 0) configurable with opt.zero_realloc.
+    (@davidtgoldblatt)
+  - Add 'make uninstall' support.  (@sangshuduo, @Lapenkov)
+  - Support C++17 over-aligned allocation.  (@marksantaniello)
+  - Add the thread.peak mallctl for approximate per-thread peak memory tracking.
+    (@davidtgoldblatt)
+  - Add interval-based stats output opt.stats_interval.  (@interwq)
+  - Add prof.prefix to override filename prefixes for dumps.  (@zhxchen17)
+  - Add high resolution timestamp support for profiling.  (@tyroguru)
+  - Add the --collapsed flag to jeprof for flamegraph generation.
+    (@igorwwwwwwwwwwwwwwwwwwww)
+  - Add the --debug-syms-by-id option to jeprof for debug symbols discovery.
+    (@DeannaGelbart)
+  - Add the opt.prof_leak_error option to exit with error code when leak is
+    detected using opt.prof_final.  (@yunxuo)
+  - Add opt.cache_oblivious as an runtime alternative to config.cache_oblivious.
+    (@interwq)
+  - Add mallctl interfaces:
+    + opt.zero_realloc  (@davidtgoldblatt)
+    + opt.cache_oblivious  (@interwq)
+    + opt.prof_leak_error  (@yunxuo)
+    + opt.stats_interval  (@interwq)
+    + opt.stats_interval_opts  (@interwq)
+    + opt.tcache_max  (@interwq)
+    + opt.trust_madvise  (@azat)
+    + prof.prefix  (@zhxchen17)
+    + stats.zero_reallocs  (@davidtgoldblatt)
+    + thread.idle  (@davidtgoldblatt)
+    + thread.peak.{read,reset}  (@davidtgoldblatt)
+
+  Bug fixes:
+  - Fix the synchronization around explicit tcache creation which could cause
+    invalid tcache identifiers.  This regression was first released in 5.0.0.
+    (@yoshinorim, @davidtgoldblatt)
+  - Fix a profiling biasing issue which could cause incorrect heap usage and
+    object counts.  This issue existed in all previous releases with the heap
+    profiling feature.  (@davidtgoldblatt)
+  - Fix the order of stats counter updating on large realloc which could cause
+    failed assertions.  This regression was first released in 5.0.0.  (@azat)
+  - Fix the locking on the arena destroy mallctl, which could cause concurrent
+    arena creations to fail.  This functionality was first introduced in 5.0.0.
+    (@interwq)
+
+  Portability improvements:
+  - Remove nothrow from system function declarations on macOS and FreeBSD.
+    (@davidtgoldblatt, @fredemmott, @leres)
+  - Improve overcommit and page alignment settings on NetBSD.  (@zoulasc)
+  - Improve CPU affinity support on BSD platforms.  (@devnexen)
+  - Improve utrace detection and support.  (@devnexen)
+  - Improve QEMU support with MADV_DONTNEED zeroed pages detection.  (@azat)
+  - Add memcntl support on Solaris / illumos.  (@devnexen)
+  - Improve CPU_SPINWAIT on ARM.  (@AWSjswinney)
+  - Improve TSD cleanup on FreeBSD.  (@Lapenkov)
+  - Disable percpu_arena if the CPU count cannot be reliably detected.  (@azat)
+  - Add malloc_size(3) override support.  (@devnexen)
+  - Add mmap VM_MAKE_TAG support.  (@devnexen)
+  - Add support for MADV_[NO]CORE.  (@devnexen)
+  - Add support for DragonFlyBSD.  (@devnexen)
+  - Fix the QUANTUM setting on MIPS64.  (@brooksdavis)
+  - Add the QUANTUM setting for ARC.  (@vineetgarc)
+  - Add the QUANTUM setting for LoongArch.  (@wangjl-uos)
+  - Add QNX support.  (@jqian-aurora)
+  - Avoid atexit(3) calls unless the relevant profiling features are enabled.
+    (@BusyJay, @laiwei-rice, @interwq)
+  - Fix unknown option detection when using Clang.  (@Lapenkov)
+  - Fix symbol conflict with musl libc.  (@georgthegreat)
+  - Add -Wimplicit-fallthrough checks.  (@nickdesaulniers)
+  - Add __forceinline support on MSVC.  (@santagada)
+  - Improve FreeBSD and Windows CI support.  (@Lapenkov)
+  - Add CI support for PPC64LE architecture.  (@ezeeyahoo)
+
+  Incompatible changes:
+  - Maximum size class allowed in tcache (opt.[lg_]tcache_max) now has an upper
+    bound of 8MiB.  (@interwq)
+
+  Optimizations and refactors (@davidtgoldblatt, @Lapenkov, @interwq):
+  - Optimize the common cases of the thread cache operations.
+  - Optimize internal data structures, including RB tree and pairing heap.
+  - Optimize the internal locking on extent management.
+  - Extract and refactor the internal page allocator and interface modules.
+
+  Documentation:
+  - Fix doc build with --with-install-suffix.  (@lawmurray, @interwq)
+  - Add PROFILING_INTERNALS.md.  (@davidtgoldblatt)
+  - Ensure the proper order of doc building and installation.  (@Mingli-Yu)
+
 * 5.2.1 (August 5, 2019)
 
   This release is primarily about Windows.  A critical virtual memory leak is

contrib/jemalloc/FREEBSD-upgrade

@@ -1,189 +1,91 @@
 #!/bin/sh
-#
-# Usage: cd /usr/src/contrib/jemalloc
-#        ./FREEBSD-upgrade <command> [args]
-#
-# At least the following ports are required when importing jemalloc:
-# - devel/autoconf
-# - devel/git
-# - devel/gmake
-# - textproc/docbook-xsl
-# - textproc/libxslt
-#
-# The normal workflow for importing a new release is:
-#
-#   cd /usr/src/contrib/jemalloc
-#
-# Merge local changes that were made since the previous import:
-#
-#   ./FREEBSD-upgrade merge-changes
-#   ./FREEBSD-upgrade rediff
-#
-# Extract latest jemalloc release.
-#
-#   ./FREEBSD-upgrade extract <rev>
-#
-# Fix patch conflicts as necessary, then regenerate diffs to update line
-# offsets:
-#
-#   ./FREEBSD-upgrade rediff
-#   ./FREEBSD-upgrade extract <rev>
-#
-# Do multiple buildworld/installworld rounds.  If problems arise and patches
-# are needed, edit the code in ${work} as necessary, then:
-#
-#   ./FREEBSD-upgrade rediff
-#   ./FREEBSD-upgrade extract <rev>
-#
-# The rediff/extract order is important because rediff saves the local
-# changes, then extract blows away the work tree and re-creates it with the
-# diffs applied.
-#
-# Finally, to clean up:
-#
-#  ./FREEBSD-upgrade clean
 
-set -e
-set -x
+# Note: you need docbook installed, as well as gmake (we need it to
+# make private_namespace.h)
 
-if [ ! -x "FREEBSD-upgrade" ] ; then
-  echo "Run from within src/contrib/jemalloc/" >&2
-  exit 1
-fi
+# git subtree merge -- not committed at this time.
+ git subtree merge -P contrib/jemalloc vendor/jemalloc
+cd contrib/jemalloc
 
-if [ "x${JEMALLOC_REPO}" = "x" ] ; then
-  JEMALLOC_REPO=https://github.com/jemalloc/jemalloc.git
-fi
+# Gut the tests, since they take up too much space.
+# Everything else can stay, but if not, add more to trim (there's
+# always a trade off between time and saved size.
+git rm -rf test msvc
+git commit --amend
 
-src=`pwd`
+# kill the tests with empty files so we don't have to hack configure.ac
+mkdir -p test/include/test
+touch test/include/test/jemalloc_test_defs.h.in
+touch test/include/test/jemalloc_test.h.in
+echo 'exit 0' > test/test.sh.in
 
-jemalloc_tmp="jemalloc.tmp"
-tmpdir="${src}/../${jemalloc_tmp}"
-bare_repo="${tmpdir}/jemalloc_bare.git"
-work="jemalloc_work.git"
-work_repo="${tmpdir}/${work}"
-namespace_repo="${tmpdir}/jemalloc_namespace.git"
-changes="${src}/FREEBSD-changes"
-
-do_fetch() {
-  local rev=$1
-  if [ ! -d "${bare_repo}" ] ; then
-    mkdir -p "${bare_repo}"
-    git clone --bare ${JEMALLOC_REPO} ${bare_repo}
-  fi
-  (
-    cd ${bare_repo}
-    git fetch origin ${rev}
-  )
-}
+# Reconfigure -- needed only to regenerate the .h files... We don't
+# use all the files generated.
+#
+# Also note: 5.2 lacks --with-lg-page-sizes, but 5.3 has it.
+# Also, there's got to be a way to not hard-wire version / hash.
+./autogen.sh --enable-xmalloc --enable-fill --enable-lazy-lock --enable-stats \
+	     --enable-utrace --with-malloc-conf=abort_conf:false \
+	     --with-xslroot=/usr/local/share/xsl/docbook --with-private-namespace=__ \
+	     --with-lg-page-sizes=12,13,14,15,16 \
+	     --with-version=5.3.0-0-g54eaed1d8b56b1aa528be3bdd1877e59c56fa90c
 
-do_extract_helper() {
-  local rev=$1
-  local repo=$2
-  do_fetch ${rev}
-  rm -rf ${repo}
-  git clone ${bare_repo} ${repo}
-  (
-    cd ${repo}
-    if [ "x${rev}" != "x" ] ; then
-      # Use optional rev argument to check out a revision other than HEAD on
-      # master.
-      git checkout ${rev}
-    fi
-  )
-}
+# Copy over the important generated .h files in configure
+cp ./include/jemalloc/jemalloc.h ../../lib/libc/stdlib/malloc/jemalloc/include/jemalloc
+git add ../../lib/libc/stdlib/malloc/jemalloc/include/jemalloc/jemalloc.h
+cp ./include/jemalloc/jemalloc_defs.h ../../lib/libc/stdlib/malloc/jemalloc/include/jemalloc
+git add ../../lib/libc/stdlib/malloc/jemalloc/include/jemalloc/jemalloc_defs.h
 
-do_autogen() {
-  ./autogen.sh --enable-xmalloc --enable-utrace \
-    --with-malloc-conf=abort_conf:false \
-    --with-xslroot=/usr/local/share/xsl/docbook --with-private-namespace=__ \
-    --with-lg-page-sizes=12,13,14,16
-}
+# need to make the namespace .h files, and copy a small subset into the tree
+# These are super-awkward to generate at buildworld time. Also, we assume we
+# only have to make one of these (currently true due to current unlikely to
+# change dependencies.
+gmake include/jemalloc/internal/private_namespace.h
+for i in private_namespace.h jemalloc_internal_defs.h public_namespace.h jemalloc_preamble.h; do
+    cp include/jemalloc/internal/$i ../../lib/libc/stdlib/malloc/jemalloc/include/jemalloc/internal/
+    git add ../../lib/libc/stdlib/malloc/jemalloc/include/jemalloc/internal/$i
+    rm include/jemalloc/internal/$i
+done
+# OK, commit all the generated files
+git add VERSION
+git commit --amend
 
-do_extract_diff() {
-  local rev=$1
-  local repo=$2
-  do_extract_helper ${rev} ${repo}
-  (
-    cd ${repo}
-    # Apply diffs before generating files.
-    patch -p1 < "${src}/FREEBSD-diffs"
-    find . -name '*.orig' -delete
-    # Generate files.
-    do_autogen
-    gmake dist
-  )
-}
+# Clean up the mess
+git clean -f .
 
-do_extract_namespace() {
-  local rev=$1
-  local repo=$2
-  do_extract_helper ${rev} ${repo}
-  (
-    cd ${repo}
-    # Generate files.
-    do_autogen
-    gmake include/jemalloc/internal/private_namespace.h
-  )
-}
+# Save the cheat sheet
+cp ~/jemalloc-upd FREEBSD-upgrade
+git add FREEBSD-upgrade
+git commit --amend
 
-do_extract() {
-  local rev=$1
-  do_fetch ${rev}
-  do_extract_diff ${rev} ${work_repo}
-  do_extract_namespace ${rev} ${namespace_repo}
-}
+# Remove hash.c from lib/libc/stdlib/malloc/jemalloc/Makefile.inc
+# mutex_pool.c prng.c
+# Add
+# bin_info.c san.c san_bump.c counter.c prof_data.c prof_log.c prof_recent.c prof_stats.c prof_sys.c
+# emap.c edata.c edata_cache.c pa.c pa_extra.c pac.c decay.c hpa.c hpa_hooks.c fxp.c hpdata.c pai.c
+# ecache.c ehooks.c eset.c sec.c cache_bin.c peak_event.c psset.c inspect.c exp_grow.c thread_event.c
+#
 
-do_diff() {
-  (
-    cd ${work_repo}
-    find . -name '*.orig' -delete
-    find . -name '*.rej' -delete
-    git add -A
-    git diff --cached
-  ) > FREEBSD-diffs
-}
+# Manually comment out the following in lib/libc/stdlib/malloc/jemalloc/include/jemalloc/jemalloc.h
+# /* #define JEMALLOC_HAVE_ATTR_FORMAT_GNU_PRINTF */
+# Add
+# #define JEMALLOC_OVERRIDE_VALLOC
+# and #include "jemalloc_FreeBSD.h"
+# With some adjustments to the old jemalloc_FreeBSD.h, but git can help
 
-command=$1
-shift
-case "${command}" in
-  merge-changes) # Merge local changes that were made since the previous import.
-    rev=`cat VERSION |tr 'g' ' ' |awk '{print $2}'`
-    # Extract code corresponding to most recent import.
-    do_extract ${rev}
-    # Compute local differences to the upstream+patches and apply them.
-    (
-      cd ${tmpdir}
-      diff -ru -X ${src}/FREEBSD-Xlist ${work} ../jemalloc > ${changes} || true
-    )
-    (
-      cd ${work_repo}
-      patch -p1 < ${changes} || true
-      find . -name '*.orig' -delete
-    )
-    # Update diff.
-    do_diff
-    ;;
-  extract) # Extract upstream sources, apply patches, copy to contrib/jemalloc.
-    rev=$1
-    do_extract ${rev}
-    # Delete existing files so that cruft doesn't silently remain.
-    rm -rf ChangeLog COPYING VERSION doc include src
-    # Copy files over.
-    tar cf - -C ${work_repo} -X FREEBSD-Xlist . |tar xvf -
-    internal_dir="include/jemalloc/internal"
-    grep -v ' isthreaded ' \
-      "${namespace_repo}/${internal_dir}/private_namespace.h" \
-      > "${internal_dir}/private_namespace.h"
-    ;;
-  rediff) # Regenerate diffs based on working tree.
-    do_diff
-    ;;
-  clean) # Remove working tree and temporary files.
-    rm -rf ${tmpdir} ${changes}
-    ;;
-  *)
-    echo "Unsupported command: \"${command}\"" >&2
-    exit 1
-    ;;
-esac
+# Had to manually remove
+# -#define __malloc_options_1_0 JEMALLOC_N(__malloc_options_1_0)
+# -#define _malloc_first_thread JEMALLOC_N(_malloc_first_thread)
+# -#define __malloc_message_1_0 JEMALLOC_N(__malloc_message_1_0)
+# -#define isthreaded JEMALLOC_N(isthreaded)
+#
+# Also had to remove the following to fix jemalloc 3 ABI compat
+# -#define je_allocm JEMALLOC_N(je_allocm)
+# -#define je_dallocm JEMALLOC_N(je_dallocm)
+# -#define je_nallocm JEMALLOC_N(je_nallocm)
+# -#define je_rallocm JEMALLOC_N(je_rallocm)
+# -#define je_sallocm JEMALLOC_N(je_sallocm)
+# Without the diff you end up with non-exported _je_je*allocm symbols. With you get symbols of the form:
+#  365: 000000000018e2a0   406 FUNC    WEAK   DEFAULT   14 rallocm@FBSD_1.3 (5)
+#  657: 000000000018e2a0   406 FUNC    GLOBAL DEFAULT   14 __rallocm@FBSD_1.3 (5)
+#