Multi-scale mean field learning for adaptive decision making in multi-agent systems

Multi-agent reinforcement learning

MARL methods have been widely applied in various scenarios, giving rise to diverse algorithms tackling different challenges (Zhang et al., 2021; Hegde & Bouroche, 2024; Oroojlooy & Hajinezhad, 2023). Independent Proximal Policy Optimization (IPPO) (De Witt et al., 2020) trains each agent separately using Proximal Policy Optimization (PPO), offering stable policy optimization via effective exploration-exploitation trade-offs. Multi-Agent Proximal Policy Optimization (MAPPO) (Yu et al., 2022), in contrast, adopts centralized optimization to improve overall coordination. Heterogeneous-Agent Proximal Policy Optimization (HAPPO) (Kuba et al., 2022) further enhances communication efficiency and accelerates convergence under the centralized training and execution paradigm.

DGN (Jiang et al., 2018) promotes cooperation by learning latent features through graph convolution and applying relational regularization. Multi-Agent Deep Deterministic Policy Gradient (MADDPG) (Lowe et al., 2017) enables agents to explore latent action spaces more effectively using random perturbations, improving adaptability in complex environments. Finally, Nash Q-learning (Yang et al., 2018) introduces game-theoretic equilibrium optimization to MARL, and serves as a foundational component in many mean-field-based algorithms.

Although the aforementioned MARL algorithms show high performance in practical applications, their interaction logic limits their applicability. These algorithms are typically more suited for scenarios with fewer or simpler interactions and struggle to handle the challenges of large-scale, complex interaction environments.

Mean field multi-agent reinforcement learning

To simplify interaction modeling in multi-agent systems, the mean field (MF) method was introduced (Jusup et al., 2023; Li et al., 2024; Wu et al., 2024), where the influence of other agents is approximated by a virtual average agent. Building on this idea, MF-Q (Yang et al., 2018) applied Nash Q-learning to mean field settings and demonstrated superior performance over value-based approaches like MF-AC. Subsequent works introduced parameterized distributions to capture population-level behavior more accurately (Perrin et al., 2021, 2022), though these methods generally assume homogeneous agents and struggle with complex heterogeneous interactions.

To address such limitations, approaches like multi-typed mean fields (Ganapathi Subramanian et al., 2020) have been proposed for handling heterogeneous agent groups. General Mean Field Q-Learning (GMF-Q) (Guo et al., 2019) improves policy stability by integrating action distributions and stochastic policies using a Boltzmann exploration strategy. In the context of density regulation, Cui & Li (2024) apply robust mean field learning with stochastic disturbance modeling to enhance control over large-scale homogeneous robotic swarms.

Although these mean field based MARL methods effectively address the training complexity in systems with large numbers of agents, they still rely on global mean field information. Even though GMF-Q avoids complete dependence on selecting the optimal action by introducing action distributions, it remains heavily influenced by global information and lacks the ability to finely model the local environment.

Attention-based mean field multi-agent reinforcement learning

To address the limitations of traditional mean field methods—namely the over-reliance on global averages and the lack of local interaction modeling—recent studies have introduced attention mechanisms into mean field frameworks. Graph Attention Mean Field (GAT-MF) (Hao et al., 2023) integrates Graph attention networks with MADDPG, transforming the unweighted mean field into a weighted one that captures varying interaction strengths among agents. This enables more precise modeling in complex environments by dynamically assigning importance to different neighbors.

Similarly, Adaptive Mean Field Q-Learning (AMF-Q) (Wang et al., 2024b) incorporates attention into Q-learning to construct an adaptive mean field, allowing agents to flexibly weigh information from nearby agents. By modeling localized influence more accurately, AMF-Q enhances adaptability and interaction fidelity, especially in scenarios where local neighbor effects dominate agent behavior.

Although these methods can assign different weights to neighboring agents based on the local context, they have two main limitations: First, they lack a global perspective of the environment, relying too heavily on local information and neglecting the influence of the global context. Second, they are constrained by a single-scale modeling approach, which cannot effectively capture the multi-level information in the environment.

To address the above challenges, we propose a multi-scale mean field representation method. This approach introduces both near-field and far-field mean fields to comprehensively capture multi-level interaction information between agents in the environment. Specifically, we obtain information from neighboring agents and apply an attention mechanism to dynamically weigh the neighbor information, forming the near-field information. This process allows for a more accurate reflection of the direct influence neighbors have on the agent, ensuring that the agent focuses on the most relevant neighbors, thereby enhancing the modeling ability of local interactions. At the same time, we construct far-field information using the global action distribution, enabling the agent to gain a richer global perspective. By combining both local and global information, this multi-scale method allows the agent to understand and process interactions at multiple levels, leading to more comprehensive decision making. Through learning multi-scale mean field information, agents can obtain more sufficient and precise environmental information, thereby improving decision accuracy and overall performance.

Markov decision processes

Due to the problem of path planning in partially observable environments, we adopt the framework of decentralized partially observable Markov Decision Process (Dec-POMDP), whose basic form is expressed as: $⟨S,A_i,P,{\mathrm{\Omega }}_i,O,R,\gamma ⟩$, where S represents the global set of states. $A_i$ denotes the action set of agent $i$, $A={\prod }_{i=1}^M{A}_i$ represents the joint action space. ${\mathrm{\Omega }}_i$ is the observation space of agent $i$, and likewise, $\mathrm{\Omega }$ denotes the joint observation space. $O:A\times S\to \mathrm{\Omega }$ serves as the observation function of the agents, which can be expressed as the probability $P(o|(a,s))$, where $o\in \mathrm{\Omega }$. Given the current state set S and the joint action set A, the state transition function and the reward function are defined as follows: $P:S\times A\to S$, $R:S\times A\to R$. Here, the state transition function represents the probability $P(s^{\mathrm{\prime }}|(a,s))$ of transitioning from the current state $s\in S$ to the next state $s^{\mathrm{\prime }}\in S$ upon taking action $a\in A$. The long-term cumulative reward starting from time $t_0$ is given by:

(1) $$R_{t_0}=\sum _{t=t_0}^T{\gamma }^{t-t_0}R(s^t,a^t),$$where $\gamma$ in $[0,1]$ is the discount factor and T is the maximum length in each episodes.

Multi-agent Q-learning

MARL, based on RL, assigns a policy to each agent $i$. At each time step, the agent selects an action according to its policy. The future discounted return is used to evaluate the quality of the chosen policy. For a given policy $\mathit{\pi }=X_{i\in \mathcal{N}}{\pi }_i$ and initial state $s_0\sim b^0$, the future discounted return for agent $i$ can be expressed as follows:

(2) $$V^i(s;\mathit{\pi })={\mathbb{E}}_{\mathit{a}\sim \mathit{\pi }}\left\{\sum _{t=0}^{\mathrm{\infty }}{\gamma }^t{r}^i(s_t,{\mathit{a}}_t)|s_0=s\right\}.$$

The Q-value function for the multi-agent version can be given by the Bellman equation. During the decision making process, each agent aims to maximize its cumulative reward and will choose the optimal strategy for action. Therefore, for agent ii, the corresponding Q-value function $Q^i:S\times \mathit{A}\to \mathrm{\Re }$ can be written as follows:

(3) $$Q^i(s,a)=(1-\alpha )Q^i(s,a)+\alpha \left[r^i(s,a)+\gamma V^i(s^{\mathrm{\prime }})\right].$$

In addition to this, the value function of each agent is related to the joint strategy, and the Nash equilibrium is often seen as the goal of learning.

Nash Q-learning

In MARL, the goal of each agent is to learn an optimal strategy that maximizes its value function. Optimizing the strategy ${\nu }_{\mathit{\pi }}^i$ for agent $i$ requires the joint strategy $\mathit{\pi }$ of all agents. To achieve this, we can borrow the concept of Nash equilibrium from stochastic games to optimize the strategies of the agents. Specifically, a Nash equilibrium is represented by a set of joint strategies ${\mathit{\pi }}_{\ast }\mathrm{\triangleq }[{\mathit{\pi }}_{\ast }^1,\dots ,{\mathit{\pi }}_{\ast }^N]$, and for all states $s\in \mathcal{S},i\in \{1,\dots ,N\}$ and all valid ${\pi }^i$, the following condition must hold:

(4) $$v^i(s;{\pi }_{\ast })=v^i(s;{\pi }_{\ast }^i,{\pi }_{\ast }^{-i})\ge v^i(s;{\pi }^i,{\pi }_{\ast }^{-i}),$$where ${\pi }_{\ast }^{-i}\mathrm{\triangleq }[{\pi }_{\ast }^1,\dots ,{\pi }_{\ast }^{i-1},{\pi }_{\ast }^{i+1},\dots ,{\pi }_{\ast }^N]$ represents the joint strategy of all agents except agent $i$.

As a MARL algorithm, Nash Q-Learning defines a method for computing Nash strategies, consisting of two main steps: 1. Use the Lemke–Howson algorithm to compute the Nash equilibrium for the current stage of the game, and obtain an estimate of the Q-function. 2. Improve the estimate of the Q-function based on the updated Nash equilibrium values, thus optimizing the agent’s strategy. Under certain assumptions, the Nash operator ${\mathcal{H}}^{\mathrm{N}\mathrm{a}\mathrm{s}\mathrm{h}}$ can be represented by the following formula:

(5) $${\mathcal{H}}^{\mathrm{N}\mathrm{a}\mathrm{s}\mathrm{h}}Q(s,a)={\mathbb{E}}_{s^{\mathrm{\prime }}\sim p}\left[r(s,a)+\gamma v^{\mathrm{N}\mathrm{a}\mathrm{s}\mathrm{h}}(s^{\mathrm{\prime }})\right],$$where $Q\mathrm{\triangleq }[Q^1,\dots ,Q^N]$, $r(s,a)\mathrm{\triangleq }[r^1(s,a),\dots ,r^N(s,a)]$. Eventually, the Q-function will converge to the value corresponding to the Nash equilibrium, forming the so-called Nash Q-value.

Mean field approximation

The mean field approach refers to approximating the interactions between many agents as the interaction between an agent and a virtual agent that integrates other agents. If we assume that the agents in the environment only interact weakly with other agents through the mean field, then the mean field ${\mathcal{L}}^{\mathcal{i}}\in \delta a^{|\mathcal{O}||\mathcal{A}\mathcal{|}}$ can be written as:

(6) $${\mathcal{L}}^{\mathcal{i}}=\left({\mu }^{-i}(\cdot ),{\alpha }^{-i}(\cdot )\right)=\underset{N\to \mathrm{\infty }}{lim}\left(\frac{{\sum }_{j\ne i}\mathbf{1}(o^j=\cdot )}{N-1},\frac{{\sum }_{j\ne i}\mathbf{1}(a^j=\cdot )}{N-1}\right),$$where $o_j$ and $a_j$ represent the local state and action taken by agent $j$, and $-i$ denotes the set of all agents except for agent $i$.

Framework

For the proposed MSMF-Q method, the encoder consists of two convolutional layers followed by two separate fully connected (FC) layers: one for processing observation maps and one for agent-specific feature vectors. The decoder is composed of two FC layers. And the activation functions are ReLu. All Q-networks used in MSMF-Q and baseline methods follow this architecture unless otherwise specified.

In order to efficiently capture the interaction information of agents at multiple levels, we implemented the framework for training the multi-scale mean field representation method, as shown in Fig. 1. For a given initial environment, we first input each agent’s views and features into an encoder for embedding representation. Next, using the blue agent as an example, we extract the action vectors of all agents in the environment at the current time step and average them. This step is consistent with the traditional mean field approach, used to extract global macro-level information from the environment, i.e., far-field information. Then, based on the parameter k (where k is the number of neighbors selected to limit the range of the near field, with $k=3$ as shown in the figure), we choose the neighbors of the agent. We use distance as a reference, since, generally, the closer an agent is, the stronger its direct influence on the agent. We select the k nearest neighbors of the blue agent to extract its near-field information. The near-field information is dynamically weighted using the attention mechanism, assigning different weights to different agents. Since far-field information has only one value, it is directly passed through the attention mechanism without modification. Finally, the two vectors (near-field and far-field) are concatenated and output as the result of the attention module. This output is concatenated with the embedding representation from the first part and processed through a decoder. The final Q-value is then output as a reference for action selection. Finally, the Q-value is used to guide the agent in updating its state.

Multi-scale information extraction and mean field representation

In our work, we adopt a two-scale mean field representation, where the far-field represents the action distribution of all other agents. This can be expressed by the following formula:

(7) $$m_{\mathrm{f}\mathrm{a}\mathrm{r}}=\frac{1}{N}\sum _{i=1}^N{\pi }_i(a_i|s_i),$$where N represents the number of agents in the environment, and ${\pi }_i(a_i|s_i)$ denotes the probability distribution of agent i selecting action $a_i$ given state $s_i$.

The near-field represents the weighted sum of the action distributions of agent $i^{\mathrm{\prime }}sk$ neighbors, with weights computed based on the interaction strength between agents using an attention mechanism. The formula is as follows:

(8) $$m_{\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}}^{(i)}=\sum _{i\in {\mathcal{N}}_k(i)}{w}_{ij}\cdot {\pi }_i(a_i|s_i).$$

Finally, the two-scale mean field distributions are fused to obtain the final multi-scale mean field representation:

(9) $$m_{\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}}^{(i)}=m_{\mathrm{f}\mathrm{a}\mathrm{r}}+m_{\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}}^{(i)}.$$

Multi-head attention module

We use the attention mechanism to weight the action distributions of neighboring agents and leverage far-field information to learn global context, compensating for the limitations of local neighborhood information.

For the $i$-th query vector ${\mathbf{q}}_i\in {\mathbb{R}}^{d_h}$ and the $j$-th key vector ${\mathbf{k}}_{\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r},j}\in {\mathbb{R}}^{d_h}$ of the near field, the attention weight for the $k$ neighbors in the neighborhood is given by:

(10) $$w_{ij}^{\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}}=\frac{\exp \left(\frac{{\mathbf{q}}_i\cdot {\mathbf{k}}_{\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r},j}^{\mathrm{\top }}}{\sqrt{d_h}}\right)}{{\sum }_{l=1}^k\exp \left(\frac{{\mathbf{q}}_i\cdot {\mathbf{k}}_{\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r},l}^{\mathrm{\top }}}{\sqrt{d_h}}\right)}.$$

The final output of the near-field is then obtained as:

(11) $${\mathbf{y}}_{\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r},i}=\sum _{j=1}^k{w}_{ij}^{\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}}\cdot {\mathbf{v}}_{\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r},j},$$where ${\mathbf{v}}_{\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r},j}\in {\mathbb{R}}^{d_h}$ is the value vector for the $j$-th neighbor.

For the far-field, since there is only one global value, the attention weight for the far-field is computed based on the $i$-th query vector ${\mathbf{q}}_i\in {\mathbb{R}}^{d_h}$ and the global key vector ${\mathbf{k}}_{\mathrm{f}\mathrm{a}\mathrm{r}}\in {\mathbb{R}}^{d_h}$, as follows:

(12) $$w_i^{\mathrm{f}\mathrm{a}\mathrm{r}}=\frac{\exp \left(\frac{{\mathbf{q}}_i\cdot {\mathbf{k}}_{\mathrm{f}\mathrm{a}\mathrm{r}}^{\mathrm{\top }}}{\sqrt{d_h}}\right)}{\exp \left(\frac{{\mathbf{q}}_i\cdot {\mathbf{k}}_{\mathrm{f}\mathrm{a}\mathrm{r}}^{\mathrm{\top }}}{\sqrt{d_h}}\right)}=1.$$

The output of the far-field is denoted as:

(13) $${\mathbf{y}}_{\mathrm{f}\mathrm{a}\mathrm{r},i}=w_i^{\mathrm{f}\mathrm{a}\mathrm{r}}\cdot {\mathbf{v}}_{\mathrm{f}\mathrm{a}\mathrm{r}},$$where ${\mathbf{v}}_{\mathrm{f}\mathrm{a}\mathrm{r}}\in {\mathbb{R}}^{d_h}$ is the global value vector.

Finally, by fusing the mean field representations from both scales, we obtain the output vector of the attention module as:

(14) $${\mathbf{y}}_i={\mathbf{y}}_{\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r},i}+{\mathbf{y}}_{\mathrm{f}\mathrm{a}\mathrm{r},i}.$$

The information output $y_i$ by the multi-head attention module is processed through a decoder, which generates the model’s output, namely the Q-value. This Q-value reflects the action probability of the agent and will guide the agent in determining its next action.

The algorithm flow is shown in Algorithm 1.

Mathematical proof

1. Proof of boundedness for multi-scale mean field error. Yang et al.’s (2018) error bound relies on the Lipschitz continuity of single-scale mean fields. Our multi-scale field decomposes into the global field $m_{\mathrm{f}\mathrm{a}\mathrm{r}}$ and local field $m_{\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}}^{(i)}$ according to Eq. (9).

Error boundedness of global field. Eq. (7) is identical to Yang et al.’s (2018) classical mean field. According to Yang et al.’s (2018) proof, its approximation error satisfies:

(15) $$\Vert Q_j(s,a_j,m_{\mathrm{f}\mathrm{a}\mathrm{r}})-Q_j(s,\mathbf{a})\Vert \le {\epsilon }_{\mathrm{f}\mathrm{a}\mathrm{r}},$$where ${\epsilon }_{\mathrm{f}\mathrm{a}\mathrm{r}}$ is the Lipschitz constant related to the sparsity of inter-agent interaction intensity.

Error boundedness of local field. According to Eqs. (8) and (10), the local field is obtained by weighting neighbors through attention weights $w_{ij}$. Since the attention weights satisfy normalization and Lipschitz continuity, the approximation error of $m_{\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}}^{(i)}$ satisfies:

(16) $$\Vert Q_i(s,a_i,m_{\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}}^{(i)})-Q_i(s,\mathbf{a})\Vert \le {\epsilon }_{\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}},$$where ${\epsilon }_{\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}}$ depends on the neighbor count $k$ and the Lipschitz constant of the attention mechanism.

Composite error bound for multi-scale field. According to Eq. (9), since $m_{\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}}^{(i)}$ is a linear combination, its total error satisfies the triangle inequality:

(17) $$||Q_i(s,a_i,m_{\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}}^{(i)})-Q_i(s,\mathbf{a})||\le {\epsilon }_{\mathrm{f}\mathrm{a}\mathrm{r}}+{\epsilon }_{\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}}.$$

Thus, the approximation error of the multi-scale field remains bounded by a constant $\epsilon ={\epsilon }_{\mathrm{f}\mathrm{a}\mathrm{r}}+{\epsilon }_{\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}}$, and ${\epsilon }_{\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}}$ can be controlled by adjusting neighbor count k (validated in the hyperparameter analysis results of battle game task).

2. Argument for convergence to nash equilibrium. According to Eqs. (4) and (5), MSMF-Q’s update rules are based on Nash Q-learning. Yang et al. (2018) have proven that the operator in Eq. (5) converges to the Nash equilibrium Q-value. In MSMF-Q, the Q-function update can be expressed as:

(18) $$Q^i(s,a)\leftarrow (1-\alpha )Q^i(s,a)+\alpha \left[r^i+\gamma {max}_{a_i}{Q}^i(s^{\mathrm{\prime }},a_i,m_{\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}}^{(i)})\right].$$

Since $m_{\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}}^{(i)}$’s error is bounded ( $\epsilon$) and the Nash operator is robust to approximation errors, MSMF-Q’s updates converge to a neighborhood of the Nash equilibrium:

(19) $$\underset{t\to \mathrm{\infty }}{lim}{Q}_t=Q_{\mathrm{N}\mathrm{a}\mathrm{s}\mathrm{h}}^{\ast }+\mathcal{O}\mathcal{(}\epsilon \mathcal{)}\mathcal{,}$$where $\epsilon$ is a negligibly small constant (The reward curves verify policy convergence to a high-performance equilibrium).

Settings

The following hyperparameters were used throughout all experiments: a learning rate of 0.001 with the Adam optimizer, batch size of 64, and a target Q-network update frequency of every 10 steps. The discount factor $\gamma$ was set to 0.95. We adopted an $\epsilon$-greedy exploration strategy, with $\epsilon$ linearly annealed from 1.0 to 0.1 over the course of training. A replay buffer was used to store transitions, with a capacity of $2^{10}$ tuples.

For computing near-field interactions, we used the Euclidean distance between agents’ current coordinates to select the k nearest neighbors. The real-world grid task is based on internal layout data from railway station. The original station floor plan was abstracted into a $120\times 80$ 2D grid map, where each cell represents a navigable space or an obstacle.

Scenario 1: Battle game task. The first experiment was conducted in the MAgent system under the Pettingzoo package, a widely used environment for MARL. In this scenario, two teams of agents, each employing different MARL methods, compete against each other in a grid world. The objective of the game is to eliminate the opposing team, with agents capable of moving in four directions (up, down, left, or right) or attacking adjacent grids. The game dynamics are designed to simulate competitive decision making and strategy optimization, which are common challenges in distributed decision making systems such as resource allocation and competitive market simulations. In this experiment, we set the number of agents to 128 vs. 128, with rewards defined for various actions: moving (−0.005), attacking an enemy (0.200), killing an enemy (5.000), attacking an empty grid (−0.100), and being attacked or killed (−0.100). Specific values are shown in Table 1. These settings mimic real-time decision making in high-stakes environments where agents must learn to optimize both offensive and defensive strategies in the presence of competitors. In addition, in this scenario, we set the number of neighbors that need to be calculated for attention to be $k=16$, and the reason is given by the hyperparameter analysis experiment.

Scenario 2: Real grid world task. The second experiment applied the MSMF-Q method to a robotic path planning scenario, simulating a crowd evacuation in a real-world environment. Using station data from railway station, we constructed a two-dimensional grid model of the station and mapped it to a 2D grid world for agent navigation. This experiment is highly relevant to robotic scheduling and smart city management, where path planning plays a critical role in optimizing the movement of robots or automated crowd control in large-scale spaces. In this simulation, 512 agents were randomly assigned positions on the grid, and their objective was to reach one of the exits to successfully evacuate the station. The map size for each layer was set to $120\times 80$, representing the station layout. Reward values were set as follows: moving or waiting (up, down, left, right, or staying) (−0.3), collisions incurred (−2.0), and successfully reaching an exit (5.0). Specific values are shown in Table 2. This scenario evaluates the method’s ability to manage large-scale agent coordination and path optimization in real-world robotic applications, focusing on efficient route planning and dynamic decision making under various environmental conditions. Furthermore, hyperparameter configuration aligns with spatial agent density. For the evacuation task, neighborhood size k scales with population size N—set to $k=8(N=256)$, $k=12(N=512)$, and $k=16(N=1024)$—based on the density-to-k ratio.

All results presented are averaged over five independent runs using distinct random seeds. The reported standard deviations and success/win rates are computed based on these runs.

Baseline

We trained multiple mean field methods on the MAgent platform and our constructed real-world grid environment. On the MAgent platform, different MARL algorithms were used to control teams for competitive battles. For the evacuation task, the success rate of agents after 512 time steps was recorded. The algorithms involved are as follows: MF-Q (Yang et al., 2018): MF-Q was one of the first methods to apply mean field theory to MARL. It abstracts the action distribution of all agents into a virtual agent, transforming interactions between agents into interactions between an agent and the virtual agent. This significantly reduces the computational complexity caused by the large number of agents in MARL. GMF-Q (Guo et al., 2019): This method combines the Boltzmann strategy with Q-learning. Agents’ action selection not only aims to maximize the current Q-value but also considers the probability distribution of other actions. This strategy helps avoid local optima and enhances exploration. AMF-Q (Wang et al., 2024b): Building on traditional mean field methods, AMF-Q introduces an attention mechanism. This allows agents to dynamically consider the influence of neighboring agents on their decisions. The dynamic weighting mechanism effectively handles the impact of heterogeneous environments on agents. MSMF-Q: We proposed the method that uses an attention mechanism to dynamically weight neighbors and integrates global mean field information to account for global action distributions. This approach, combining near-field and far-field information, effectively captures multi-level environmental details.