Consolidate trace output into a single recorder.dat

.gitignore

@@ -106,5 +106,8 @@ install
 .idea/
 /cmake-build-debug/
 
+# Claude Code
+.claude/
+
 # clangd database
 compile_commands.json

CLAUDE.md

@@ -0,0 +1,118 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What Recorder Is
+
+Recorder is a parallel I/O tracing library that intercepts I/O calls at multiple levels of the stack (POSIX, MPI-IO, HDF5, PnetCDF, NetCDF) without requiring application modification. It works via `LD_PRELOAD` to inject `librecorder.so` at runtime.
+
+## Build Commands
+
+```bash
+# Configure (out-of-source build required)
+cmake -B build -S . -DCMAKE_INSTALL_PREFIX=$PREFIX
+
+# Optional configure flags
+-DRECORDER_WITH_HDF5=/path/to/hdf5
+-DRECORDER_WITH_PNETCDF=/path/to/pnetcdf
+-DRECORDER_WITH_NETCDF=/path/to/netcdf
+-DRECORDER_ENABLE_CUDA_TRACE=ON
+-DRECORDER_ENABLE_FCNTL_TRACE=ON     # default ON
+-DRECORDER_ENABLE_PARQUET=ON
+
+# Build
+cmake --build build
+
+# Run all tests (requires MPI)
+cd build && ctest
+
+# Run a single test
+cd build && ctest -R test_mpi -V
+
+# Install
+cmake --build build --target install
+```
+
+HDF5 is detected automatically. PnetCDF and NetCDF require explicit paths via `-DRECORDER_WITH_*`.
+
+## Repository Structure
+
+- `lib/` — core library implementation
+- `include/` — public headers; `recorder-gotcha.h` (~260 KB) is auto-generated from function lists
+- `test/` — CTest-integrated test suite
+- `tools/` — post-processing binaries (recorder2text, recorder2timeline, conflict-detector, etc.)
+- `deps/GOTCHA/` — bundled function-hooking library (submodule)
+- `tools/reporter/` — Python-based trace analysis
+- `tools/generator/generator.py` — regenerates `recorder-gotcha.h` from function lists
+
+All build outputs (libraries and binaries) go to `build/bin/`.
+
+## Architecture
+
+### Interception Flow
+
+1. At program load, `__attribute__((constructor))` calls `recorder_init()` for non-MPI programs. For MPI programs, init is deferred to the `MPI_Init` wrapper.
+2. `recorder_init()` calls `gotcha_init()` → registers ~300 function wrappers; `logger_init()` → opens per-rank trace files; `utils_init()` → sets up helpers.
+3. Every wrapped function follows the same pattern using two macros in `recorder.h`:
+   - `RECORDER_INTERCEPTOR_PROLOGUE` — saves timestamps, allocates a `Record`, calls the real function via GOTCHA
+   - `RECORDER_INTERCEPTOR_EPILOGUE` — serializes arguments to strings, calls `logger_record_exit()`, returns to caller
+4. At `MPI_Finalize` (or program exit), `recorder_finalize()` flushes and compresses trace data.
+
+### Key Data Structures
+
+- **`Record`** (`recorder-logger.h`) — one per intercepted call; holds `tstart`/`tend`, `func_id`, `arg_count`, `args[]`, `call_depth`, and a per-thread stack pointer for nested-call tracking.
+- **`RecorderLogger`** — per-process singleton holding rank, record count, CST/CFG grammar state, and compression flags.
+- **`RecorderMetadata`** — trace file header written once per rank; encodes version, feature flags, and MPI info.
+
+### Trace Compression Pipeline
+
+Recorder applies multiple compression passes:
+1. **Sequitur** (`recorder-sequitur.c`) — detects repeated call sequences and encodes them as grammar rules (intra-process).
+2. **CST/CFG** (`recorder-cst-cfg.c`) — context-sensitive and context-free grammar compression.
+3. **Inter-process pattern recognition** — rank 0 coordinates cross-rank deduplication.
+4. **Zlib** — applied to timestamps and structured data.
+
+All compression is controlled at runtime via environment variables (see below).
+
+### GOTCHA Wrappers
+
+`recorder-gotcha.h` declares all wrapped functions using three macros:
+- `GOTCHA_WRAP(func, ret_type, args)` — declares the wrapper and a handle
+- `GOTCHA_WRAP_ACTION(func)` — creates the `gotcha_binding_t` entry
+- `GOTCHA_REAL_CALL(func)` — invokes the original function through GOTCHA
+
+`recorder-gotcha.c` populates `gotcha_binding_t` arrays per layer and calls `gotcha_wrap()` for each enabled layer during init. Layers can be disabled at runtime via `RECORDER_*_TRACING=0`.
+
+## Runtime Configuration (Environment Variables)
+
+```
+RECORDER_TRACES_DIR             output directory for trace files
+RECORDER_POSIX_TRACING          0/1 (default 1)
+RECORDER_MPIIO_TRACING          0/1 (default 1)
+RECORDER_MPI_TRACING            0/1 (default 0)
+RECORDER_HDF5_TRACING           0/1 (default 1, no-op if built without HDF5)
+RECORDER_PNETCDF_TRACING        0/1 (default 1)
+RECORDER_NETCDF_TRACING         0/1 (default 1)
+RECORDER_INTERPROCESS_COMPRESSION              0/1
+RECORDER_INTERPROCESS_PATTERN_RECOGNITION      0/1
+RECORDER_INTRAPROCESS_PATTERN_RECOGNITION      0/1
+RECORDER_STORE_TID              record thread IDs
+RECORDER_STORE_CALL_DEPTH       record call nesting depth
+RECORDER_EXCLUSION_FILE         file of path prefixes to skip
+RECORDER_INCLUSION_FILE         file of path prefixes to trace exclusively
+RECORDER_WITH_NON_MPI=1         enable tracing for non-MPI programs
+```
+
+## Tests
+
+CTest tests run each binary with `LD_PRELOAD` pointing to `librecorder.so` and isolated per-test `RECORDER_TRACES_DIR` directories under `build/test/traces/`.
+
+| Test | Processes | Notes |
+|---|---|---|
+| `test_posix` | 1 | POSIX I/O |
+| `test_iopr` | 2 | lseek + pwrite patterns |
+| `test_mpi` | 4 | MPI collectives, point-to-point, MPI-IO |
+| `test_hdf5` | 1 | HDF5 (only when HDF5_FOUND) |
+| `test_phdf5` | 2 | Parallel HDF5 (only when HDF5_FOUND) |
+
+`test_signal` and `test_hybrid` are compiled but excluded from CTest: `test_signal` is designed to receive a kill signal mid-run and crashes during normal exit; `test_hybrid` relies on a hardcoded `mpirun` path.

CMakeLists.txt

@@ -16,11 +16,6 @@ set(RECORDER_PACKAGE_STRING "${RECORDER_PACKAGE_NAME} ${RECORDER_PACKAGE_VERSION
 set(RECORDER_PACKAGE_TARNAME "${RECORDER_PACKAGE}")
 
 
-# ignore warning message
-if(POLICY CMP0074)
-    cmake_policy(SET CMP0074 OLD)
-endif()
-
 
 #------------------------------------------------------------------------------
 # Setup install and output Directories
@@ -231,7 +226,7 @@ if(CMAKE_PROJECT_NAME STREQUAL RECORDER)
 endif()
 
 if(CMAKE_PROJECT_NAME STREQUAL RECORDER AND BUILD_TESTING)
-    # include test sub directory
+    add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/test)
 endif()
 
 #-----------------------------------------------------------------------------

include/recorder-gotcha.h

@@ -7,8 +7,10 @@
 #include <stdbool.h>
 #include "mpi.h"
 
+#ifdef RECORDER_WITH_HDF5
 #include "hdf5.h"
 #include "hdf5_hl.h"
+#endif
 
 #ifdef RECORDER_WITH_PNETCDF
 #include "pnetcdf.h"
@@ -29,6 +31,7 @@
 #define CONST
 #endif
 
+#ifdef RECORDER_WITH_HDF5
 #if H5_VERS_MAJOR <= 1
 
 #if H5_VERS_MINOR < 10
@@ -77,6 +80,7 @@
 #endif
 
 #endif  /* H5_VERS_MAJOR <= 1 */
+#endif  /* RECORDER_WITH_HDF5 */
 
 #ifdef NC_VERSION_MAJOR
 #ifdef NC_VERSION_MINOR
@@ -373,6 +377,7 @@ GOTCHA_WRAP(MPI_Comm_split_type, int, (MPI_Comm comm, int split_type, int key, M
 
 
 // HDF5
+#ifdef RECORDER_WITH_HDF5
 GOTCHA_WRAP(H5Dread, herr_t, (hid_t dset_id, hid_t mem_type_id, hid_t mem_space_id, hid_t file_space_id, hid_t dxpl_id, void *buf));
 GOTCHA_WRAP(H5Adelete_by_idx, herr_t, (hid_t loc_id, const char *obj_name, H5_index_t idx_type, H5_iter_order_t order, hsize_t n, hid_t lapl_id));
 GOTCHA_WRAP(H5Fflush, herr_t, (hid_t object_id, H5F_scope_t scope));
@@ -1122,6 +1127,7 @@ GOTCHA_WRAP(H5ESregister_insert_func, herr_t, (hid_t es_id, H5ES_event_insert_fu
 GOTCHA_WRAP(H5Iregister_future, hid_t, (H5I_type_t type, const void *object, H5I_future_realize_func_t realize_cb, H5I_future_discard_func_t discard_cb));
 GOTCHA_WRAP(H5ESfree_err_info, herr_t, (size_t num_err_info, H5ES_err_info_t err_info[]));
 GOTCHA_WRAP(H5ESregister_complete_func, herr_t, (hid_t es_id, H5ES_event_complete_func_t func, void *ctx));
+#endif /* RECORDER_WITH_HDF5 */
 
 // PnetCDF functions
 #ifdef RECORDER_WITH_PNETCDF
@@ -2042,6 +2048,7 @@ GOTCHA_WRAP(ncmpi_mget_varm_longlong_all, int, (int ncid, int num, int varids[],
 GOTCHA_WRAP(ncmpi_mget_varm_ulonglong_all, int, (int ncid, int num, int varids[], MPI_Offset* const starts[], MPI_Offset* const counts[], MPI_Offset* const strides[], MPI_Offset* const imaps[], unsigned long long *bufs[]));
 #endif /* RECORDER_WITH_PNETCDF */
 
+// NETCDF
 #ifdef RECORDER_WITH_NETCDF
 GOTCHA_WRAP(nc__create, int, (const char *path, int cmode, size_t initialsz, size_t *chunksizehintp, int *ncidp));
 GOTCHA_WRAP(nc__enddef, int, (int ncid, size_t h_minfree, size_t v_align, size_t v_minfree, size_t r_align));

include/recorder-logger.h

@@ -65,8 +65,46 @@ typedef struct CallSignature_t {
 } CallSignature;
 
 
+/* Maximum length (including null terminator) of a traced function name stored in recorder.dat */
+#define RECORDER_FUNC_NAME_LEN  64
+
+/*
+ * Section type identifiers used in the combined recorder.dat file.
+ *
+ * recorder.dat layout:
+ *   [RecorderFileHeader]                 16 bytes
+ *   [RecorderSectionEntry * num_sections] 24 bytes each
+ *   [section data ...]
+ */
+typedef enum {
+    RECORDER_SECTION_METADATA   = 0,  /* RecorderMetadata struct + function name table      */
+    RECORDER_SECTION_TIMESTAMPS = 1,  /* merged timestamp data (same layout as recorder.ts) */
+    RECORDER_SECTION_GLOBAL_CST = 2,  /* merged CST  (interprocess_compression = true)      */
+    RECORDER_SECTION_GLOBAL_CFG = 3,  /* unique grammars (interprocess_compression = true)  */
+    RECORDER_SECTION_CFG_META   = 4,  /* grammar metadata (interprocess_compression = true) */
+    RECORDER_SECTION_RANK_CST   = 5,  /* per-rank CST (interprocess_compression = false)    */
+    RECORDER_SECTION_RANK_CFG   = 6,  /* per-rank CFG (interprocess_compression = false)    */
+} RecorderSectionType;
+
+typedef struct RecorderFileHeader_t {
+    char     magic[8];        /* "RECORDER" */
+    uint32_t format_version;  /* 1          */
+    uint32_t num_sections;
+} RecorderFileHeader;
+
+typedef struct RecorderSectionEntry_t {
+    uint32_t type;    /* RecorderSectionType                              */
+    int32_t  rank;    /* -1 for global sections, rank index for per-rank  */
+    uint64_t offset;  /* byte offset of section data from start of file   */
+    uint64_t size;    /* byte size of section data                        */
+} RecorderSectionEntry;
+
 typedef struct RecorderMetadata_t {
+    int    version_major;
+    int    version_minor;
+    int    version_patch;
     int    total_ranks;
+    int    num_funcs;                           // number of function names stored after the struct
     bool   posix_tracing;
     bool   mpi_tracing;
     bool   mpiio_tracing;

lib/CMakeLists.txt

@@ -12,7 +12,6 @@ include_directories(${CMAKE_SOURCE_DIR}/deps/GOTCHA/include)
 set(RECORDER_SRCS
         ${CMAKE_SOURCE_DIR}/include/recorder.h
         ${CMAKE_SOURCE_DIR}/include/recorder-sequitur.h
-        ${CMAKE_CURRENT_SOURCE_DIR}/recorder-hdf5.c
         ${CMAKE_CURRENT_SOURCE_DIR}/recorder-cst-cfg.c
         ${CMAKE_CURRENT_SOURCE_DIR}/recorder-mpi.c
         ${CMAKE_CURRENT_SOURCE_DIR}/recorder-init-finalize.c
@@ -34,15 +33,31 @@ set(RECORDER_SRCS
 #------------------------------------------------------------------------------
 # External dependencies
 #------------------------------------------------------------------------------
-find_package(HDF5 REQUIRED)
+# HDF5: use -DRECORDER_WITH_HDF5=/path/to/hdf5/install (consistent with PnetCDF/NetCDF)
+# Falls back to find_package(HDF5) if RECORDER_WITH_HDF5 is not set.
+if(RECORDER_WITH_HDF5)
+    find_path(HDF5_INCLUDE_DIRS hdf5.h ${RECORDER_WITH_HDF5}/include)
+    find_path(HDF5_LIBRARY_FOUND libhdf5.so ${RECORDER_WITH_HDF5}/lib)
+    if(HDF5_INCLUDE_DIRS AND HDF5_LIBRARY_FOUND)
+        set(HDF5_FOUND TRUE)
+        set(HDF5_LIBRARIES ${RECORDER_WITH_HDF5}/lib/libhdf5.so)
+    endif()
+else()
+    find_package(HDF5)
+endif()
 if(HDF5_FOUND)
+    message("-- " "Found HDF5:" ${HDF5_INCLUDE_DIRS})
+    set(RECORDER_SRCS
+        ${CMAKE_CURRENT_SOURCE_DIR}/recorder-hdf5.c
+        ${RECORDER_SRCS}
+    )
     include_directories(${HDF5_INCLUDE_DIRS})
     set(RECORDER_EXT_INCLUDE_DEPENDENCIES ${HDF5_INCLUDE_DIRS}
             ${RECORDER_EXT_INCLUDE_DEPENDENCIES})
     set(RECORDER_EXT_LIB_DEPENDENCIES
             ${HDF5_LIBRARIES} ${RECORDER_EXT_LIB_DEPENDENCIES})
 else()
-    message(STATUS, "HDF5 not found")
+    message("-- " "HDF5 not found. Will build Recorder without it. To use a local installation, set -DRECORDER_WITH_HDF5=/path/to/hdf5")
 endif()
 
 # PnetCDF is not a CMAKE project, we use find_path() to locate it
@@ -62,7 +77,7 @@ if(PNETCDF_INCLUDE_DIRS AND PNETCDF_LIBRARY_FOUND)
     set(RECORDER_EXT_LIB_DEPENDENCIES
             ${PNETCDF_LIBRARIES} ${RECORDER_EXT_LIB_DEPENDENCIES})
 else()
-    message("-- " "PnetCDF not found. Will build Recorder without it.")
+    message("-- " "PnetCDF not found. Will build Recorder without it. To use a local installation, set -DRECORDER_WITH_PNETCDF=/path/to/pnetcdf")
 endif()
 
 # Look for NetCDF
@@ -82,7 +97,7 @@ if(NETCDF_INCLUDE_DIRS AND NETCDF_LIBRARY_FOUND)
     set(RECORDER_EXT_LIB_DEPENDENCIES
             ${NETCDF_LIBRARIES} ${RECORDER_EXT_LIB_DEPENDENCIES})
 else()
-    message("-- " "NetCDF not found. Will build Recorder without it.")
+    message("-- " "NetCDF not found. Will build Recorder without it. To use a local installation, set -DRECORDER_WITH_NETCDF=/path/to/netcdf")
 endif()
 
 
@@ -170,6 +185,7 @@ target_compile_definitions(recorder
         PRIVATE $<$<BOOL:${HAVE___LXSTAT64}>:HAVE___LXSTAT64>
         PRIVATE $<$<BOOL:${HAVE___FXSTAT}>:HAVE___FXSTAT>
         PRIVATE $<$<BOOL:${HAVE___FXSTAT64}>:HAVE___FXSTAT64>
+        PRIVATE $<$<BOOL:${HDF5_FOUND}>:RECORDER_WITH_HDF5>
         PRIVATE $<$<AND:$<BOOL:${PNETCDF_INCLUDE_DIRS}>,$<BOOL:${PNETCDF_LIBRARY_FOUND}>>:RECORDER_WITH_PNETCDF>
         PRIVATE $<$<AND:$<BOOL:${NETCDF_INCLUDE_DIRS}>,$<BOOL:${NETCDF_LIBRARY_FOUND}>>:RECORDER_WITH_NETCDF>
         PRIVATE $<$<BOOL:${RECORDER_ENABLE_FCNTL_TRACE}>:RECORDER_ENABLE_FCNTL_TRACE>
@@ -254,3 +270,9 @@ set(RECORDER_INCLUDES_INSTALL_TIME
         ${RECORDER_EXT_INCLUDE_DEPENDENCIES}
         PARENT_SCOPE
         )
+
+# Propagate optional library detection results so test/CMakeLists.txt can
+# conditionally register HDF5 tests.
+set(HDF5_FOUND    ${HDF5_FOUND}    PARENT_SCOPE)
+set(HDF5_INCLUDE_DIRS ${HDF5_INCLUDE_DIRS} PARENT_SCOPE)
+set(HDF5_LIBRARIES    ${HDF5_LIBRARIES}    PARENT_SCOPE)

lib/recorder-gotcha.c

@@ -4,7 +4,11 @@
 static bool posix_tracing    = true;
 static bool mpi_tracing      = false;
 static bool mpiio_tracing    = true;
+#ifdef RECORDER_WITH_HDF5
 static bool hdf5_tracing     = true;
+#else
+static bool hdf5_tracing     = false;
+#endif
 static bool pnetcdf_tracing  = true;
 static bool netcdf_tracing   = true;
 
@@ -187,6 +191,7 @@ struct gotcha_binding_t mpi_wrap_actions [] = {
 };
 
 struct gotcha_binding_t hdf5_wrap_actions [] = {
+#ifdef RECORDER_WITH_HDF5
     GOTCHA_WRAP_ACTION(H5Dread),
     GOTCHA_WRAP_ACTION(H5Adelete_by_idx),
     GOTCHA_WRAP_ACTION(H5Fflush),
@@ -936,6 +941,7 @@ struct gotcha_binding_t hdf5_wrap_actions [] = {
     GOTCHA_WRAP_ACTION(H5Iregister_future),
     GOTCHA_WRAP_ACTION(H5ESfree_err_info),
     GOTCHA_WRAP_ACTION(H5ESregister_complete_func),
+#endif /* RECORDER_WITH_HDF5 */
 };
 
 
@@ -2171,13 +2177,11 @@ void gotcha_register_functions() {
     char* posix_tracing_env   = getenv(RECORDER_POSIX_TRACING);
     char* mpi_tracing_env     = getenv(RECORDER_MPI_TRACING);
     char* mpiio_tracing_env   = getenv(RECORDER_MPIIO_TRACING);
-    char* hdf5_tracing_env    = getenv(RECORDER_HDF5_TRACING);
     char* pnetcdf_tracing_env = getenv(RECORDER_PNETCDF_TRACING);
     char* netcdf_tracing_env  = getenv(RECORDER_NETCDF_TRACING);
     if (posix_tracing_env) posix_tracing = atoi(posix_tracing_env);
     if (mpi_tracing_env) mpi_tracing = atoi(mpi_tracing_env);
     if (mpiio_tracing_env) mpiio_tracing = atoi(mpiio_tracing_env);
-    if (hdf5_tracing_env) hdf5_tracing = atoi(hdf5_tracing_env);
     if (pnetcdf_tracing_env) pnetcdf_tracing = atoi(pnetcdf_tracing_env);
     if (netcdf_tracing_env) netcdf_tracing = atoi(netcdf_tracing_env);
 
@@ -2193,10 +2197,14 @@ void gotcha_register_functions() {
         gotcha_wrap(mpiio_wrap_actions,
                     sizeof(mpiio_wrap_actions)/sizeof(struct gotcha_binding_t),
                     "recorder_mpiio_actions");
+#ifdef RECORDER_WITH_HDF5
+    char* hdf5_tracing_env = getenv(RECORDER_HDF5_TRACING);
+    if (hdf5_tracing_env) hdf5_tracing = atoi(hdf5_tracing_env);
     if (hdf5_tracing)
         gotcha_wrap(hdf5_wrap_actions,
                     sizeof(hdf5_wrap_actions)/sizeof(struct gotcha_binding_t),
                     "recorder_hdf5_actions");
+#endif /* RECORDER_WITH_HDF5 */
     if (pnetcdf_tracing)
         gotcha_wrap(pnetcdf_wrap_actions,
                     sizeof(pnetcdf_wrap_actions)/sizeof(struct gotcha_binding_t),