Description

I thought it would be useful to have a section in the documentation on commodity graphs, as this is an important part of how MUSE works, and a fundamental improvement over MUSE1 where users had to manually define hierarchies. Rather than writing this manually, I tried to get copilot to do it mostly for me. This is based off #1569, where region-level graphs have been combined.

My process was as follows:

• Used a smaller model (Claude Sonnet) with a short prompt for a first draft.
• The result was pretty poor, but reading this was helpful for me to clarify in my mind exactly what I wanted and come up with a more detailed plan
• I then prompted a larger model (Claude Opus) with the following:

I want a section in the user-facing docs describing commodity graphs in the context of the model. It should mention the fact that graphs are built and why:

• to establish the potential flow relationships between commodities and regions implied by the processes
• to perform validation, to make sure that service demands can be met by the available processes, and that commodities abide by the rules for their type
• to ensure system integrity
• to determine the order of investments and price calculation: MUSE invests downstream to upstream, and calculates prices upstream to downstream
• to identify circularities
• to help visualise the model, via the save-graphs command
• for performance reasons, e.g. identifying independent commodity groups that can be run in parallel (not technically what we do currently, but this is a longer-term justification so worth including anyway)

The docs should focus on user facing implications, rather than algorithmic details. The goal is to help users to build models and diagnose problems

A few things to mention (allows users to understand visualised graphs and diagnose problems)

• nodes are commodities in a region (i.e. a market)
• edges are processes
• primary vs secondary edges
• a graph is created for each year. This is then validated for each timeslice to take into account process availabilities and demand
• information about visualising the graph, helping users to understand the visualisation
• the investment order is determined by performing a topological sort after collapsing strongly connected components. The price calculation order is the reverse of the investment order
• within cycles, markets are ordered with a MILP. Do not discuss the algorithm, but give a link to the in-code documentation. Do not discuss the investment process for circularities, only that the order is determined here
• independent market sets are grouped into layers, for performance reasons

There should be at least one diagrammatic example, maybe more. E.g. we should give an example of a set of commodities and processes, and how the nodes in the graph represent commodities and the edges are processes. For simplicity, limit this to one region (don't include regions on the graph). Colour code by SVD/SED/OTH. Include a circularity. Then from the graph, write down/show what the investment order would be, walking it through. It might be easier to give dot code and I go away and create a png/svg - not sure. Use the example models for inspiration.

This documentation should be easy to read for a MUSE newcomer. The aim is to educate users to help them use MUSE effectively, not to confuse them. Aim for conciseness and do not include unnecessary details that aren't important for MUSE users to know.

Come up with a detailed plan

• It then came back with a detailed plan which I was mostly happy with, so I set it off to implement this plan
• The result was remarkably(/scarily) good! It even designed commodity graphs and generated svgs. I had a few comments so sent a follow up prompt and made a few manual changes (mostly to make it more concise), but on the whole I didn't intervene much.

Notes:

• The example model doesn't exactly match one of the built in example models. I wanted to include a circularity, but the circularity example is a bit too complex for this.
• I think one further improvement we could make is to make the graphs from the save-graphs command more visually appealing. The first graph in the doc is good inspiration. If we did this then maybe we could combine these sections. Beyond the scope of this PR though.
• I've committed the svgs, and the corresponding dot files include the command that copilot used to generate them. I wonder if it would be worth including this in the doc build workflow? Maybe not as they're not likely to change often...
• I hope that this will be helpful both for users and for AI itself. That was part of my justification for using a big AI model (Opus) to put this together, so hopefully smaller models can read this and understand everything they need to know about commodity graphs.

Fixes # (issue)

Type of change

• Bug fix (non-breaking change to fix an issue)
• New feature (non-breaking change to add functionality)
• Refactoring (non-breaking, non-functional change to improve maintainability)
• Optimization (non-breaking change to speed up the code)
• Breaking change (whatever its nature)
• Documentation (improve or add documentation)

Key checklist

• All tests pass: $ cargo test
• The documentation builds and looks OK: $ cargo doc
• Update release notes for the latest release if this PR adds a new feature or fixes a bug
  present in the previous release

Further checks

• Code is commented, particularly in hard-to-understand areas
• Tests added that prove fix is effective or that feature works