Skip to content

Multi-GPU support: P4estMesh3D - #3364

Open
benegee wants to merge 60 commits into
mainfrom
bg/multi-gpu-3d
Open

Multi-GPU support: P4estMesh3D#3364
benegee wants to merge 60 commits into
mainfrom
bg/multi-gpu-3d

Conversation

@benegee

@benegee benegee commented Oct 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

3D counterpart of, and based on, #3240

benegee and others added 30 commits June 25, 2026 11:51
@benegee
adapt_structure
3d24fb8
@benegee
Merge branch 'main' into bg/adapt-mpicache
5463667
@benegee
prolong2mpiinterfaces
072b754
@benegee
integrate_via_indices 2D
ed8fc47
@benegee
Merge branch 'main' into bg/adapt-mpicache
69cdbb8
@benegee
move to CPU as in non-parallel case
78ccbda
@benegee
Merge branch 'main' into bg/adapt-mpicache
cb69596
@benegee
P4estMeshParallel{2} GPU support
a737166 
- prolong2mpiinterfaces
- calc_mpi_interface_flux
- start_mpi_send
- finish_mpi_receive
@benegee
rhs_hyperbolic for parallel GPU
b935ced
@benegee
remove reset du
1f9d981
@benegee
fmt
7f1ba04
@benegee
fix bug
a48f151
@benegee
use trixi_timeit_ext
5fcbd10
@benegee
avoid kernel launches with zero size
18df67e
@benegee
missed line
cc61eba
@benegee
removed parallel meshes
84abfff 
should already be included
@benegee
try to set up test with mpi and kernelabstractions
8ea605b
@benegee
add kernelabstractions_mpi to SPECIAL_PROCESS_SUITES
d317322
@benegee
backend arguments
0d31cc5
@benegee
typo
429de3d
@benegee
Merge branch 'main' into bg/adapt-mpicache
909db1b
@benegee
another backend argument
cbccef7
@benegee
another one
daaa898
@benegee
added NEWS.md
36b9235
@benegee
unpack
5200c40
@benegee
Merge branch 'bg/adapt-mpicache' of github.com:trixi-framework/Trixi.…
439db9f 
…jl into bg/adapt-mpicache
@benegee
Merge branch 'main' into bg/adapt-mpicache
40c8f43
@benegee
Merge branch 'main' into bg/adapt-mpicache
cfeef65
@benegee
fmt
596ad55
@benegee
Merge branch 'main' into bg/adapt-mpicache
15d9fb5
@benegee
benegee marked this pull request as ready for review October 19, 2026 11:33
@benegee benegee mentioned this pull request Oct 19, 2026
Open
26 of 51 tasks
@benegee

benegee commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

Ready for review.

@ranocha ranocha added the parallelization Related to MPI, threading, tasks etc. label Oct 22, 2026
ranocha
ranocha requested changes Oct 22, 2026
View reviewed changes

@ranocha ranocha left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot! Did you test it on your local system? Do you have some rough first performance results you can share with us?

Comment thread src/solvers/dgsem_p4est/dg_parallel_gpu.jl Resolved
Comment thread src/solvers/dgsem_p4est/dg_parallel_gpu.jl Resolved
Comment thread src/solvers/dgsem_p4est/dg_parallel_gpu.jl Resolved
Comment thread test/test_kernelabstractions_mpi.jl Resolved
@benegee

benegee commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

Thanks a lot! Did you test it on your local system? Do you have some rough first performance results you can share with us?

Yes, tested on our workstation. For the advection elixir (tiny, but used in the tests) I got

                                            Trixi.jl
─────────────────────────────────────────────────────────────────────────────────────────────────
                                                       Time                   Allocations
                                              ──────────────────────    ────────────────────────
              Tot / % measured:                   5.98s / 85.4%              657MiB / 95.0%
 ───────────────────────────────────────────  ──────────────────────    ────────────────────────
 Section                              ncalls    time    %tot     avg      alloc    %tot      avg
─────────────────────────────────────────────────────────────────────────────────────────────────
 rhs_hyperbolic!                          96   3.74s   73.2%  39.0ms     417MiB   66.8%  4.34MiB
 ├─ ~rhs_hyperbolic!~                     96   1.32s   25.9%  13.8ms     118MiB   19.0%  1.23MiB
 ├─ volume integral                       96   1.13s   22.1%  11.7ms     140MiB   22.5%  1.46MiB
 │  ├─ ~volume integral~                  96   1.05s   20.6%  11.0ms     123MiB   19.6%  1.28MiB
 │  └─ reset ∂u/∂t                        96  74.3ms    1.5%   774μs    17.5MiB    2.8%   186KiB
 ├─ prolong2interfaces + flux             96   347ms    6.8%  3.62ms    39.6MiB    6.3%   422KiB
 ├─ MPI interface flux                    96   325ms    6.4%  3.38ms    35.5MiB    5.7%   378KiB
 ├─ prolong2mpiinterfaces                 96   296ms    5.8%  3.08ms    41.5MiB    6.7%   443KiB
 ├─ surface, Jacobian + source terms      96   265ms    5.2%  2.76ms    41.9MiB    6.7%   447KiB
 ├─ finish MPI receive                    96  56.4ms    1.1%   588μs    6.00KiB    0.0%    64.0B
 ├─ start MPI receive                     96   359μs    0.0%  3.74μs    13.5KiB    0.0%     144B
 ├─ start MPI send                        96  71.0μs    0.0%   740ns    13.5KiB    0.0%     144B
 ├─ prolong2boundaries                    96  41.4μs    0.0%   431ns    7.50KiB    0.0%    80.0B
 ├─ prolong2mpimortars                    96  39.4μs    0.0%   410ns    7.50KiB    0.0%    80.0B
 ├─ MPI mortar flux                       96  39.2μs    0.0%   408ns    7.50KiB    0.0%    80.0B
 ├─ boundary flux                         96  36.5μs    0.0%   380ns    7.50KiB    0.0%    80.0B
 ├─ synchronize                           96  36.4μs    0.0%   379ns    7.50KiB    0.0%    80.0B
 ├─ prolong2mortars                       96  35.9μs    0.0%   374ns    7.50KiB    0.0%    80.0B
 ├─ mortar flux                           96  34.2μs    0.0%   356ns    7.50KiB    0.0%    80.0B
 └─ finish MPI send                       96  27.0μs    0.0%   282ns    13.5KiB    0.0%     144B
 calculate dt                             20   1.37s   26.8%  68.3ms     207MiB   33.2%  10.3MiB
─────────────────────────────────────────────────────────────────────────────────────────────────

@benegee
benegee requested a review from ranocha November 4, 2026 10:27
@ranocha

ranocha commented Nov 5, 2026

Copy link
Copy Markdown
Member

The part ~rhs_hyperbolic!~ is quite large - do you have an idea whether that's real and what's going on there?

ranocha
ranocha requested changes Nov 5, 2026
View reviewed changes
Comment thread src/callbacks_step/analysis_dg3d_parallel.jl Resolved
Comment thread src/solvers/dgsem_p4est/dg_3d_parallel_gpu.jl Resolved
benegee and others added 3 commits November 5, 2026 12:44
@benegee

benegee commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

The part ~rhs_hyperbolic!~ is quite large - do you have an idea whether that's real and what's going on there?

Right! Thanks for the hint!

It's compilation. In my MPI launch the elixir just ran once. When running a second time in the same process, it now looks like this:

                                           Trixi.jl
─────────────────────────────────────────────────────────────────────────────────────────────────
                                                       Time                   Allocations
                                              ──────────────────────    ────────────────────────
              Tot / % measured:                   878ms / 99.6%             2.49MiB / 86.1%
 ───────────────────────────────────────────  ──────────────────────    ────────────────────────
 Section                              ncalls    time    %tot     avg      alloc    %tot      avg
─────────────────────────────────────────────────────────────────────────────────────────────────
 rhs_hyperbolic!                          96   728ms   83.2%  7.58ms    2.06MiB   96.2%  21.9KiB
 ├─ volume integral                       96   641ms   73.3%  6.68ms     432KiB   19.7%  4.50KiB
 │  ├─ ~volume integral~                  96   641ms   73.2%  6.67ms     357KiB   16.3%  3.72KiB
 │  └─ reset ∂u/∂t                        96   841μs    0.1%  8.76μs    75.0KiB    3.4%     800B
 ├─ finish MPI receive                    96  72.0ms    8.2%   750μs    6.00KiB    0.3%    64.0B
 ├─ prolong2interfaces + flux             96  8.77ms    1.0%  91.4μs     359KiB   16.4%  3.74KiB
 ├─ MPI interface flux                    96  2.16ms    0.2%  22.5μs     402KiB   18.4%  4.19KiB
 ├─ surface, Jacobian + source terms      96  1.45ms    0.2%  15.1μs     423KiB   19.3%  4.41KiB
 ├─ prolong2mpiinterfaces                 96  1.38ms    0.2%  14.4μs     348KiB   15.9%  3.62KiB
 ├─ ~rhs_hyperbolic!~                     96   168μs    0.0%  1.75μs    43.8KiB    2.0%     467B
 ├─ start MPI send                        96  52.8μs    0.0%   550ns    13.5KiB    0.6%     144B
 ├─ start MPI receive                     96  44.0μs    0.0%   459ns    13.5KiB    0.6%     144B
 ├─ prolong2boundaries                    96  39.6μs    0.0%   412ns    7.50KiB    0.3%    80.0B
 ├─ prolong2mpimortars                    96  39.3μs    0.0%   409ns    7.50KiB    0.3%    80.0B
 ├─ MPI mortar flux                       96  38.7μs    0.0%   403ns    7.50KiB    0.3%    80.0B
 ├─ synchronize                           96  37.2μs    0.0%   387ns    7.50KiB    0.3%    80.0B
 ├─ boundary flux                         96  36.6μs    0.0%   381ns    7.50KiB    0.3%    80.0B
 ├─ prolong2mortars                       96  36.4μs    0.0%   379ns    7.50KiB    0.3%    80.0B
 ├─ mortar flux                           96  35.4μs    0.0%   369ns    7.50KiB    0.3%    80.0B
 └─ finish MPI send                       96  18.1μs    0.0%   189ns    13.5KiB    0.6%     144B
 calculate dt                             20   147ms   16.8%  7.34ms    83.7KiB    3.8%  4.19KiB
─────────────────────────────────────────────────────────────────────────────────────────────────

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ranocha ranocha ranocha requested changes

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

gpu parallelization Related to MPI, threading, tasks etc.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@benegee @ranocha @DanielDoehring