Rate dependent neural responses of interaural-time-difference cues in fine-structure and envelope

Experiment 1

The EEG responses were measured as a function of carrier frequency, using SAM tones generated according to Bernstein & Trahiotis (2012), Hu, Klug & Dietz (2022). The modulation frequency was $f_m=$ 40 Hz, and $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{F}\mathrm{S}}$ was introduced by applying interaural phase differences (IPD) to the carrier of SAM tones. Four carrier frequencies, $f_c$ = [400, 800, 1,200, 1,600] Hz, were tested. The EEG stimuli lasted for 8 s (s) and followed a sequence: 2 s of the diotic stimulus (IPD = 0 in time window T1), 2 s of the dichotic stimulus (IPD = π/2 in time window T2; T1→T2 referred to as outward switching), 2 s of the standard stimulus (IPD = 0 in time window T3; T2→T3 referred to as inward switching), and 2 s of silence (in time window T4). An example of a stimulus used in EEG experiment 1 is illustrated in Fig. 3B, column 1 (40 Hz SAM tones with $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{F}\mathrm{S}}$, $f_c$ = 400 Hz and $f_m$ = 40 Hz).

Experiment 2

SAM tone at carrier frequency of $f_c$ = 4,000 Hz and four modulation frequencies $f_m$ = [40, 80, 160, 320] Hz were tested, with the ITD applied to the envelope instead of the carrier. Note that, as in Kohlrausch, Fassel & Dau (2000), no precautions were taken to mask possible distortion products in experiment 2. Since the modulation frequencies were below 320 Hz, we believe that the main results summarized below are not affected by detection cues based on distortion products.

Similar to the EEG paradigm in experiment 1, the EEG stimuli sequence consisted of 2 s of the diotic stimulus ( $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{E}\mathrm{N}\mathrm{V}}$ = 0 in the time window T1), 2 s of the dichotic stimulus ( $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{E}\mathrm{N}\mathrm{V}}$ = 500 µs in the time window T2; with an outward switching T1→T2), 2 s of the standard stimulus ( $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{E}\mathrm{N}\mathrm{V}}$ = 0 in the time window T3; with an inward switching T2→T3), and 2 s of silence (in the time window T4). An example of a stimulus used in EEG experiment 2 is presented in Fig. 3B, column 2 (SAM tones $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{E}\mathrm{N}\mathrm{V}}$, $f_c$ = 4,000 Hz and $f_m$ = 40 Hz).

Experiment 3

To further simulate the CI performance in NH listeners, filtered click trains were generated as described in Hu et al. (2017), Hu, Klug & Dietz (2022), with the bandwidth limited to 3,000–5,000 Hz, and a center frequency of $f_c$ = 4,000 Hz. A low-pass filtered noise, uncorrelated between the ears was added to the filtered click trains to mask potential distortion products. The low-pass noise was generated by creating broadband noise in the time domain, converting it to the frequency domain, and setting the power of all components above 1,000 Hz to zero. The noise was further manipulated to have a flat spectrum up to 200 Hz, with a decreasing spectral density of 3 dB/octave above 200 Hz. It was then filtered with a 5th-order, lowpass filter having a cut-off frequency of 1,000 Hz (Hu et al., 2017), and gated with 50-ms raised cosine ramps. The test stimulus was centered within the noise presentation in time, which was presented at 40 dB SPL.

Four fixed pulse rates of [40, 80, 160, 320] pps were employed. Each EEG stimuli sequence had a duration of 6 s, consisting 2 s of the diotic stimulus, followed by 2 s of the dichotic stimulus (with a transition from T1→T2, referred as outward switching), and 2 s of silence (in time window T4, with a T2→T3 inward switching). Fig. 3B, column 3, provides an example of the stimuli used in the EEG experiment 3 (filtered clicks IPTD, pulse rate = 160 pps and $f_m$= 10 Hz, IPTD = 0 or IPTD = 500 µs).

Time domain (CAEPs)

To validate our assumptions listed in the introduction (assumptions 1 to 4), we first examine the CAEPs in the time domain here. Figs. 4A–4C shows the individual (gray) and average EEG responses for experiments 1–3 in the time domain. The same processing procedure and automatic peak-detection method as described in “EEG data analysis” section were applied to all conditions. Each figure presents the four test conditions using different colors. The P1, N1, and P2 peaks of the average responses are marked with triangle symbols in each of the test conditions in panels 1–4. For easy comparison, the averaged data from panels 1–4 are overlaid in panel 5.

Figure 5 replots some results from Fig. 3. The upper panel shows the average responses evoked by 40 Hz SAM tones with $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{F}\mathrm{S}}$ changes ( $f_c=$ 400 Hz, named, fc400ITDfs, red) and by 40, 80, 160, and 320 Hz SAM tones with $f_c=$ 4,000 Hz and $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{E}\mathrm{N}\mathrm{V}}$ changes. The lower panel shows the responses evoked by 40, 80, 160, and 320 pps filtered click trains with interaural pulse time difference (IPTD) changes, in addition to the fc400ITDfs (red). By overlaying these curves, some ACC-like responses were able to be identified within the same ACC time duration as fc400ITDfs (after 2 and 4 s, red curve). However, the ACC responses evoked by envelope ITD changes were much smaller than those evoked by $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{F}\mathrm{S}}$ and close to the noise floor for both 4,000 Hz SAM tones (upper panel) and filtered clicks (lower panel). The automatically marked ACC peaks mostly fell within the same range as the noise within the detection windows in experiments 2 and 3, and the 1,600 Hz condition of experiment 1. Therefore, caution should be taken when interpreting the ACC peaks in these cases.

By observing the waveform morphology of the EEG responses shown in Figs. 4–5, and considering the information redundancy among P1, N1, and P2, in the statistical analysis, mainly the results of N1P2 amplitude (the amplitude difference between P2 and N1, i.e., P2-N1, in µV) and the N1 latency were reported. Figs. 6A–6C shows the violin plots (Hintze & Nelson, 1998) of the automatically detected N1P2 amplitude and N1 latency of experiment 1–3. The pair-wise Bonferroni corrected t-tests with p < 0.05 were marked with ‘*’ symbols.

In general, all test conditions in experiment 1 elicited both onset and offset CAEPs, confirming Assumption 1. Please refer to the following three subsections for more details.

• Assumptions 2: CAEPs in experiment 1

Experiment 1 was designed to validate the test setup and replicate results reported in the literature. Figs. 4 and 6A shows the amplitude and latency in experiment 1 ( $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{F}\mathrm{S}}$).

The offset responses (aqua) were relatively consistent across different carrier frequencies and the N1 latency of the offset responses was generally shorter compared to the onset and ACC responses. In general, most results from experiment 1 were consistent with Ross, Tremblay & Picton (2007), where CAEPS to IPD changes of SAM tones were recorded from 12 NH listeners using MEG. For example, the mean P1, N1, and P2 amplitudes of ACC were smaller than those of the onset response: P1, 1.684/1.405/1.005/0.248 µV; N1, 3.324/-1.299/-1.019/-2.146 µV; P2, 3.946/2.128/1.862/2.379 µV for onset/ACC1/ACC2/offset. The ACC latencies were delayed compared with the corresponding onset and offset ones: P1, 42/46/57/27 ms; N1, 114/132/137/95 ms; P2, 211/227/240/213 ms for the onset/ACC1/ACC2/offset. The mean latencies of both P1 and N1 fell within a similar range, albeit slightly smaller than those reported by Ross, Tremblay & Picton (2007). The latencies of $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{F}\mathrm{S}}$ change evoked ACC1 and ACC2 were longer than the onset ones, however, the differences between ACC and onset responses were smaller than (Ross, Tremblay & Picton, 2007). In line with (Ross, 2018), a tendency for larger responses to outward IPD changes (ACC1) compared to inward changes (ACC2) was observed for the lower carrier frequencies. Nevertheless, this tendency did not reach statistical significance here (p > 0.5).

The N1P2 amplitude was significantly affected by carrier frequency ( $f_c$), response type, and their interaction according to GLMrm (p < 0.005). There were no significant differences between 400, 800, and 1,200 Hz, but the N1P2 amplitude for the 1,600 Hz was significantly smaller than the other carrier frequencies. Pairwise comparisons showed no significant differences between carrier frequencies in most cases for both onset and offset CAEPs, except that the onset N1P2 amplitude of 1,200 Hz was slightly larger than that of 1,600 Hz (p = 0.048). For both ACC1 (outward) and ACC2 (inward) responses, there were no significant differences between 400 and 800 Hz, while the N1P2 amplitudes of 400 Hz and 800 Hz were significantly larger than those of 1,200 and 1,600 Hz. The onset CAEPs were significantly larger than the ACC1, ACC2, and offset CAEPs, but not significantly different amongst the other three types. Pairwise comparisons showed that the onset CAEPs were significantly larger than the offset CAEPs for 400 and 1,200 Hz, and no significant differences between ACC1 and ACC2 for all carrier frequencies. Significant correlations were observed between the onset N1P2 amplitudes of most carrier frequencies, except for 400 vs 1,200 Hz, and 400 vs 1,600 Hz.

The GLMrm analysis showed a significant effect of response type (p < 0.001) on the N1P2 latency, but no significant effect of $f_c$ and their interaction. The N1 latency of ACC responses was significantly larger than the onset response, while the offset response had the shortest latency and was significantly smaller than the other response types.

In summary: 1) Clear ACC responses were evident in experiment 1 for conditions with $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{F}\mathrm{S}}$ changes in carrier frequencies up to 1,200 Hz, supporting Assumption 2. 2) The N1 latency of ACC responses was the longest, while offset responses had shorter N1 latency and smaller amplitude than both onset and ACC responses. 3) the N1P2 amplitudes of 400 Hz and 800 Hz were significantly larger than those of 1,200 and 1,600 Hz.

• Assumptions 3: CAEPs of experiment 2

In experiment 2 ( $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{E}\mathrm{N}\mathrm{V}}$, Fig. 6B, also see Fig. 4), similar to experiment 1, there were clear onset and offset responses in all four test conditions. The onset N1P2 amplitude was comparatively larger than the offset responses, but the difference between the onset and offset CAEPs was smaller compared to those shown in Fig. 6A. Consistent with the findings of Ross (2018), the N1P2 amplitudes of both onset and offset CAEPs were larger than the ACC responses, due to the tiny (close to the noise floor) or absence of ACC responses.

Regarding the N1P2 amplitude in experiment 2, a GLMrm analysis revealed significant effects of $f_m$, response type, and their interaction. The mean amplitude was 4.249/4.228/5.258/5.665 µV for $f_m$ of 40/80/160/320 Hz, respectively. A significant difference between $f_m$ was only observed for 80 Hz vs 160 Hz. Pairwise comparisons within each response type also only showed a just significant smaller onset N1P2 amplitude in 80 Hz condition compared to the 160 Hz condition (p = 0.048). The mean amplitude was 8.547/2.543/1.853/6.458 µV for onset/ACC1/ACC2/offset, respectively. Both onset and offset CAEPs were larger than the ACC responses. There were no significant differences between ACC1 and ACC2, and between onset and offset. Pairwise comparisons also showed no significant differences between them within each $f_m$. The onset and offset CAEPs were significantly larger than ACC responses, except for some cases with $f_m$ = 80 Hz, and $f_m$ = 160 Hz. Significant correlations were observed among modulation frequencies for all onset N1P2 amplitudes and for most offset N1P2 amplitudes, except for 320 vs 40, and 320 vs 160 Hz.

For N1 latency, the GLMrm analysis showed no significant effect of either $f_m$ or response type. The mean latency was 107/109/107/101 ms for onset/ACC1/ACC2/offset, and 107/105/104/109 ms for 40/80/160/320 Hz.

In summary, the ACC responses evoked by $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{E}\mathrm{N}\mathrm{V}}$ changes were much smaller than those evoked by $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{F}\mathrm{S}}$ observed in experiment 1, supporting Assumption 3. However, categorizing ACC responses to $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{E}\mathrm{N}\mathrm{V}}$ changes as absent based solely on the signal- to- noise ratio might be misleading. For example, by overlaying multiple test conditions as shown in Fig. 5, some ACC-like responses became identifiable.

• Assumption 4: CAEPs of experiment 3

For the filtered click trains (Fig. 6C), no inward IPTD change (ACC2 responses) conditions were tested in experiment 3. Overall, the N1P2 amplitude of both onset and offset responses increased with increasing pulse rates. Similar to experiment 2, the ACC1 responses were either small (near the noise floor) or absent.

For N1P2 amplitude, GLMrm showed a significant effect of pulse rate, response type, and their interactions (p < 0.01). The mean amplitude was 2.31/3.43/5.36/5.35 µV for 40/80/160/320 pps, respectively. There were no significant differences between pulse rates of 40 and 80 pps, and between 160 and 320 pps. Within each response type, pairwise comparisons showed no significant differences between pulse rates for both ACC1 and offset responses. For the onset CAEPs, there were significant differences between most pulse rates (p < 0.01), except for conditions of 40 vs 80 pps, and 160 vs 320 pps. The mean amplitude was 5.48/2.52/4.34 µV for onset/ACC1/offset, and only the difference between onset and ACC1 responses was significant. Within each pulse rate, pairwise comparisons showed no significant differences between response types for most pulse rates, except that for 160 pps and 320 pps, there was a significantly larger onset N1P2 amplitude than the offset one (p = 0.009, and p = 0.014). There were no correlations between N1P2 amplitudes of different pulse rates for both onset and offset responses.

For N1 latency, GLMrm revealed a significant effect of pulse rate, but not of response types or their interactions. The mean latency was 141/134/119/114 ms for 40/80/160/320 pps, respectively. The N1 latency was significantly shorter for 320 pps compared to 40 pps and 80 pps, and for 160 pps compared to 80 pps. The mean latency was 131/129/120 ms for onset/ACC1/offset responses with no significant differences between them.

In summary, we were unable to confirm Assumption 4, as ACC responses evoked by both $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{E}\mathrm{N}\mathrm{V}}$ changes in experiment 2 and IPTD changes in experiment 3 were very small and closely approached to the noise floor, suggesting their potential absent in the classical EEG response interpretation.

Assumption 5: Frequency domain (ASSRs)

Figure 7 shows the average ASSRs across participants within the analysis window of T1234 or T124. Within each panel, the colored curves in the shaded area are the average ASSRs for each individual. The red, blue, black, and pink spectra are the overall average ASSRs across participants for different test conditions: (A) $f_c$ = [400, 800, 1,200, 1,600] Hz; (B) $f_m$ = [40, 80, 160, 320] Hz; (C) pulse rate = [40, 80, 160, 320] pps. The numbers with the same colors in the figure represent the ASSR amplitudes at the modulation frequency. The right panel displays the violin plots of the ASSR amplitude at the modulation frequency, within a duration of 8s (T1234) for the SAM tones or 6 s (T124) for the filtered clicks.

For experiment 1 (Fig. 7A), the group mean 40-Hz ASSR amplitudes were 0.187/0.173/0.152/0.142 µV for 400/800/1,200/1,600 Hz, respectively. The amplitude of the 40-Hz ASSR decrease gradually with increasing carrier frequency for the SAM-type stimuli, as previously reported by Ross (2018). However, these differences are not statistically significant. The 40-Hz ASSR of SAM tones with different carrier frequencies were correlated, except for the comparison between 800 and 1,600 Hz.

Figures 7B and 7C show the ASSRs at different modulation rates (40, 80, 160, and 320 Hz). The numbers shown in different colors are the average ASSR values for each modulation rate. In general, the ASSRs for high carrier frequency stimuli (Fig. 7B, with a smaller y-axis scale) were smaller than those for low-frequency (<= 1,600 Hz) SAM tones (Fig. 7A), and the ASSRs for filtered clicks (Fig. 7C) were larger than those for high-frequency SAM tones. In both types of high-frequency stimuli, the order was 40 Hz ASSR > 160 Hz ASSR > 80 Hz ASSR > 320 Hz ASSR. The filtered clicks elicited larger ASSRs, as suggested by Hu, Klug & Dietz (2022).

The mean overall amplitudes of the ASSRs in experiment 2 (Fig. 7B) were 0.093, 0.058, 0.071, and 0.022 µV for 40, 80, 160, and 320 Hz, respectively. The 320 Hz ASSR was significantly smaller than the others. There were no significant correlations between the ASSRs of different modulation frequencies, except for 160 Hz and 320 Hz.

The overall mean amplitudes of the ASSRs in experiment 3 (Fig. 7C) were 0.148/0.075/0.123/0.048 µV for 40/80/160/320 pps, respectively. The 160 pps showed significantly larger ASSRs compared to 80 and 320 pps. Significant correlations were also observed between the ASSR amplitudes of 160 pps and 80 pps, 160 pps and 320 pps, and 320 pps and 80 pps.

Some studies suggested that the ASSRs for modulation frequencies up to 50 Hz are most likely generated from the auditory cortex (Mäkelä & Hari, 1987; Pantev et al., 1994; Herdman et al., 2002). To compare the 40-Hz ASSRs evoked by the 40-Hz modulated SAM tones and 40-pps filtered clicks within different analysis windows (see “Fig. S3”), GLMrm (with factors: stimuli type [400/800/1200/1600/4000SAM/40-pps-clicks], and analysis window [T1, T2, and T4]) showed a significant effect of stimuli type, analysis window (p < 0.001), and their interaction. The mean amplitudes for 400, 800, 1,200, 1,600, 4,000 Hz SAM tones, and 40-pps filtered clicks were 0.168, 0.461, 0.142, 0.136, 0.091, and 0.154 µV, respectively. Pairwise comparison revealed significantly smaller amplitude of the 40-Hz ASSR evoked by the 4,000 Hz SAM tones compared to the four low-frequency SAM tones, but no significant differences between the other stimulus types, including 4,000 Hz SAM tones vs 40-pps filtered clicks. No significant differences were found between analysis windows T1 and T2, but as expected, T4 (silence) was significantly smaller than T1 and T2.

In summary, ASSRs were detectable in all three experiments, particularly for conditions with modulation rates below 320 Hz. ASSRs amplitudes are influenced by the modulation frequency, carrier frequency, and the type of stimuli used. Under the same modulation frequency, ASSRs were larger for conditions with carrier frequencies up to 1,600 Hz in experiment 1 compared to those with a carrier frequency of 4,000 Hz in experiments 2 and 3. This confirms the assumption 5. In addition, the ASSRs for filtered clicks were larger than those for high-frequency SAM tones. In both types of high-frequency stimuli, the order was 40 Hz ASSR > 160 Hz ASSR > 80 Hz ASSR > 320 Hz ASSR.

Assumption 6: time-frequency domain

As shown in Fig. 5, it is more challenging to determine the presence of detectable ACC responses evoked by $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{E}\mathrm{N}\mathrm{V}}$changes in high-frequency SAM tones and filtered clicks compared to those elicited by $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{F}\mathrm{S}}$changes in low-carrier frequencies. This requires more experience and possibly additional references, such as the overlying method demonstrated in Fig. 5. To gain a better understanding of the smaller ACC responses evoked by $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{E}\mathrm{N}\mathrm{V}}$changes and to enhance data visualization, Fig. 8 displays the responses (with a higher cutoff frequency of 1,000 Hz instead of 30 Hz) in the time-frequency domain: SAM $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{F}\mathrm{S}}$ (A), SAM $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{E}\mathrm{N}\mathrm{V}}$ (B), filtered clicks IPTD (C). The average response plotted in each subplot as the black solid line around ~10 Hz. In general, the time-frequency amplitudes are related to stimulus parameters, such as $f_m$, and $f_c$.

The onset and offset responses in the time-frequency domain (Fig. 8) displayed clear clusters of higher energy in the lower frequency range (<30 Hz) whenever there were detectable responses. In general, all three stimuli types evoked noticeable onset and offset CAEPs, except for the 40 pps filtered clicks.

The ACC responses evoked by $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{F}\mathrm{S}}$ changes were easily recognizable in both response waveforms (Fig. 4A) and the time-frequency domain (Fig. 8A) for the 400, 800, and 1,200 Hz SAM. However, it was smaller in 1,200 Hz SAM and not distinguishable in the 1,600 Hz in both the time and time-frequency domains.

In general, the ACC responses elicited by $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{E}\mathrm{N}\mathrm{V}}$changes were smaller compared to those evoked by $\mathrm{I}\mathrm{T}{\mathrm{D}}_{\mathrm{F}\mathrm{S}}$changes. It appears there is greater presence of low-frequency brain activity (<10 Hz) for both types of high-carrier frequency stimuli in some cases. The ACC responses are roughly similar in scale to neighboring brain activities, making it challenging to distinguish an ACC response even in the time-frequency domain (Figs. 8B and 8C).

As expected, Fig. 8 reveals 40-Hz ASSRs (between the two parallel red dashed lines around the 40 Hz) except during the 2s-silence period. Time-frequency visualization shows possible interactions between the transient CAEPs and the ASSRs. Whenever the P1-N1-P2 complex was detectable and prominent, there was a noticeable reset (represented by the blue gaps in the 40 Hz regions) in the ASSRs. This suggests the steady-state activity was desynchronized by the prominent transient CAEPs, which might similar to the findings reported as ‘interrupt’ responses by some other groups (e.g., Bianco et al., 2020). For example, for $f_c$ of 400, 800, and 1,200 Hz, the ASSRs were suppressed or reset around 0, 2, and 4 s, respectively. There were notable energy differences in the ASSRs in these time windows (T1, T2, T3), as shown in Fig. S2 within the frequency range of 2–50 Hz.

In summary, the wavelet time-frequency analysis indeed provided us with additional insights into the data. For example, it suggests potential interactions between the transient CAEPs and the ASSRs, confirming our assumption 6.