# Inter-valley spiral order in the Mott insulating state of trilayer graphene-boron nitride heterostructure

###### Abstract

Recent experiment has shown that the ABC-stacked trilayer graphene-boron nitride Moire super-lattice at half-filling is a Mott insulator. Based on symmetry analyses and effective band structure calculation, we propose a valley-contrasting chiral tight-binding model with local Coulomb interactions to describe this Moire super-lattice system. When the valence band is half-filled and the valley-contrasting staggered flux of per triangle acquires a value of , the Fermi surfaces are found to be perfectly nested between the two valleys. Such an effect can induce an inter-valley spiral order with a gap in the charge excitations, indicating that the Mott insulating behavior observed in the trilayer graphene-boron nitride Moire super-lattice results predominantly from the inter-valley scattering.

The Moire super-lattice in the van der Waals heterostructure composed of multi-layer graphenes and hexagonal boron nitrides (hBN) has recently attracted great interest Yankowitz ; Dean ; Hunt ; Ponomarenko ; YangZhang ; ShiWang . Both graphenes and hBN have hexagonal lattice structures, but the original lattice periodicity is ruined due to the mismatch between their lattice constants. Nevertheless, the periodicity can be restored on a much larger length scale, i.e., the Moire wave length ( nm), upon which a triangular Moire super-lattice emerges Dean ; Hunt ; Ponomarenko . On the other hand, bilayer graphene with a small twisted angle can also form the Moire band structure MacDonald2011 ; Magaud2012 ; Cao2016PRL . In the magic-angle twisted bilayer graphene, the Moire bandwidth is reduced dramatically and the local Coulomb repulsion becomes significant, leading to the observation of the Mott insulating state as well as the unconventional superconductivity around the half-filling Cao2018Corr ; Cao2018SC . Meanwhile, it has reported that a Mott insulating state also exists in the ABC-stacked trilayer graphene-hBN heterostructure FengWang . It seems that the observed Mott insulating behavior also results from the suppression of the kinetic energy. However, the real physical mechanism behind the experimental observation may be different.

In this paper, we will investigate the physical origin of the Mott insulating behavior observed in the trilayer-graphene-hBN heterostructure. Based on the symmetry analyses and effective band structure calculation, we propose a minimal tight-binding model with local Coulomb interactions. This model defined on a triangular lattice characterizes an interacting electron system in a staggered fictitious magnetic field for each of the two degenerate valley degrees of freedom. At half-filling, the two valley Fermi surfaces are found to be perfectly nested if the staggered flux of each triangle is close to . Such an effect leads to a novel correlated insulating state with an inter-valley spiral order and a charge excitation gap, giving a natural explanation to the experimental observation.

Moire band structure. - The ABC-stacked trilayer graphene (TLG) has the same Bravais lattice as in the monolayer graphene. But the electron and hole touching at zero energy support chiral quasiparticles with Berry phase, generalizing the low-energy band structure of the monolayer and bilayer graphene KoshinoMcCann . The hBN also forms a honeycomb lattice but has a lattice constant about larger than that of the graphene. Thus the heterostructure of TLG and hBN can form a triangular Moire super-lattice shown in Fig.1a, which contains three interlaced regions. The region shaded by blue circles shows the maximal alignment between the TLG and hBN, denoted as the zone; and the regions shaded by yellow or green triangles have a larger misalignment between the TLG and hBN, denoted as and zone, respectively. The zone differs from the zone by a sub-lattice exchange, defined by the rotation along the z-axis or the mirror reflection with respect to the x-z plane. Each unit cell of the Moire super-lattice includes the , and zone. The TLG-hBN heterostructure possesses the three-fold rotational symmetry along the -axis , the mirror reflection symmetry with respect to the y-z plane , and the time reversal symmetry .

For both TLG and hBN, the honeycomb lattice can be bipartitioned into two triangular sub-lattices. A Dirac cone is generated in the electronic structure at the charge neutral point (CNP). The Dirac fermions become massive when the symmetry between the two sub-lattices is broken Miller12PRB . In the hBN, boron and nitrogen atoms each form one of the sub-lattices, which breaks the symmetry between these two sub-lattices. This leads to a large energy gap (about eV) in the low-lying excitations Miller12PRB . Thus the electrons in the hBN have no contributions to the low-energy electronic states. However, the interplay between the TLG and hBN can modify the charge potential and break the sub-lattice symmetry in the TLG. Such a Moire potential modulation folds the original Brillouin zone of the graphene layers into many mini-Brillouin zones (mBZ), as displayed in Fig.1b. So the constructed mini-bands near the CNP originate from the low-energy valleys ( and shown in Fig.1b) in the original TLG WallbankFalko .

Unlike in the hBN, the sub-lattice symmetry of the TLG is preserved and the massless Dirac fermions survive at both and points in the original BZ. Then the low-energy excitations with valley degrees of freedom are characterized by the triple Dirac fermions with a higher order momentum dispersion MacDonald10ABC . The triple Dirac fermions on each valley are further split by the effective tunneling between the top A-layer and the bottom C-layer, i.e., the trigonal warping KoshinoMcCann . The splitting distance is considerably small with respect to the original BZ, but quite comparable to the Moire wave vector. This is the crucial feature of the Moire band structure near the CNP. After the BZ is folded by the Moire potential, the higher order dispersion becomes even flatter, leading to a flat mini-band around the Fermi level. Moreover, the Dirac point is gapped out by the interplay between the hBN and TLG, which breaks the sub-lattice symmetry. In the process of band folding, the two inequivalent valley points and remain decoupled, because the valley distance in the original BZ is significantly longer than the characteristic wave vector of the Moire potential. As a result, both the conductance and valence bands near the CNP in the mBZ are approximately four-fold degenerate, associated with the spin and valley degrees of freedom. Electrons around the valleys and are related to each other by one of the transformations: the time-reversal symmetry , mirror reflection , and rotation.

Using the effective two-component Hamiltonian for the TLG KoshinoMcCann , we have calculated the band structures with the Moire scattering potential assumed to act only on the bottom graphene layer FengWang . The details are given in Supplementary Material. Since the two valley bands are connected through the mirror transformation , we can focus on one of them. In Fig.2a, the electronic structure for the bands of valley is displayed. The contour plot of the corresponding valence band near the CNP is shown in Fig.2b. The triple Dirac points originally at are separated to locate along the boundary of the mBZ towards , reducing the energy dispersion and inducing the triple van Hove singularity points near . When the Dirac points are further gapped out, the remaining triple van Hove points are the most remarkable fingerprint of the Moire band structure. More precisely, three van Hove points actually line along the mBZ boundary and center around the zone corner . Increasing the value of pushes the three van Hove points towards . Because the valence band is separated from the other bands, we are able to write a one-band minimal tight-binding model with valley and spin degeneracy.

Model Hamiltonian. - The minimal model should satisfy all the symmetries mentioned above, and reproduce the key feature of mini-valence-band: the triple van Hove points and ultra-flat dispersion. In the triangular Moire lattice sites labelled by (Fig.3a), one can effectively treat the two valleys as a pair of pseudo-spin. When we use and () to denote the Pauli matrices on the valley pseudo-spin and the physical spins, respectively, the symmetry transformations can be expressed as , , . Actually the valley index is not preserved by the six-fold rotational symmetry , but the symmetry instead. Since the valley degrees are decoupled in the band folding, the nearest neighbor hopping should conserve the valley degrees of freedom.

Since the massless triple Dirac points have a well-defined nonzero winding number , the hopping term is allowed to carry a valley-contrasting flux, because the massive Dirac fermions carry half of the topological charge in the presence of a weak sub-lattice asymmetry Martin ; Volovik . As each valley contains three Dirac points, the total topological charge cannot be zero. Moreover, the time reversal symmetry requires the two valleys to have opposite flux phases, and the symmetry swaps both the hopping directions and valley degrees, which fixes the phases in the hopping integrals (Fig.3a). Thus the minimal tight-binding model for the valence band of the TLG-hBN heterostructure is given by the Hamiltonian

(1) |

where and are the vectors of the primitive unit cell, denote the valley indices, the fluxes alternate between the and triangles, and the hopping parameter measures effectively the valence bandwidth. The flux penetrating each triangle is given by , so this is a valley-contrasting chiral tight-binding model.

In the momentum space, the band dispersion becomes

(2) |

where the electronic band dispersion generally varies with the flux phase . By observing the band structure with varying , we noticed that the flux essentially tunes those three -related van Hove points. For the valley , when varies from 0 to , these van Hove points approach to along the mBZ boundary. Right at , they merge into one triple van Hove point, as shown in the expansion around :

where , and . When further increases, the triple van Hove point splits into three points along the line and its equivalents. The mini-band structure calculated from the low-energy effective band exhibits that three van Hove points are located right on the zone boundary in the vicinity of , and hence the flux of the minimal model may be slightly smaller than . In principle, the exact value of can be determined by matching the position of the van Hove points. In the following, we will focus on the ideal limit , which reveals the essential correlated physics in the TLG-hBN heterostructure.

When at half-filling, the Fermi surface becomes a perfect triangle that touches the mBZ corners, as shown in Fig.3b. In this case, the two Fermi sheets are perfectly nested, linked by the vectors and . These three nesting vectors are equivalent to each other, because they are simply related by the reciprocal vector. More explicitly, Fig.4a and Fig.4b display the relation .

In such a circumstance, the on-site Coulomb interactions become important. The full interactions of the model Hamiltonian should contain the local inter- and intra-valley interaction, the Hund’s rule coupling, and the pairing hopping terms

(3) | |||||

where is the local electron density operator.

Inter-valley spiral order. - At half-filling, the interaction contribution is apparently dominated by the first term of Eq. (3) due to the perfect nesting between the valley Fermi surfaces. This motivates us to introduce the following inter-valley order parameter

(4) |

to decouple the -term in Eq. (3) as

(5) |

where is spin independent. If we ignore the other interaction terms in Eq. (3), the above Hamiltonian under the mean-field approximation can be diagonalized, and the order parameter is determined by the self-consistent equation

(6) |

which is similar to the BCS gap equation. If we further assume that the overall mBZ contribution is dominated by a narrow shell of width around the Fermi energy, the solution to the above equation is then given

(7) |

where is the density of states at the Fermi level. At half-filling, diverges, and an infinitesimal interaction can induce a finite inter-valley long-range order and gap out the Fermi surfaces completely. This has been confirmed by the numerical solution to the self-consistent equation, as shown in Fig. 4c. Actually this is a very peculiar insulating state induced by the inter-valley scattering . In real space, describes an inter-valley spiral long-range order of the valley pseudo-spin:

(8) |

with . Therefore, it is this inter-valley spiral phase that describes the Mott insulating phase observed by the experiment in the TLG-hBN heterostructure FengWang .

Discussion and Conclusion. - Compared with the magic-angle twisted bilayer graphene Cao2018Corr , the kinetic energy of both systems is suppressed by the band folding, resulting in a similar Moire super-lattice and the mBZ. The Dirac cones of the twisted bilayer graphene are separated and hybridized, yielding the van Hove singularity at point and flat dispersion in between. Similarly, in the TLG-hBN, the triple Dirac cones from the ABC stacked trilayer are separated by the trigonal warping with a strong hybridization, yielding the triple van Hove points and flat dispersion in between. The drastic distinction between these two systems is reflected in their symmetries and Fermi surface structures of the Moire bands. More precisely, with respect to the same Moire triangular super-lattice, the TLG-hBN is invariant under the symmetry while the twisted bilayer preserves the symmetry instead. Consequently, the -symmetric Fermi surfaces in the latter are distinct from that of the former by degree rotation Cao2016PRL . If the Fermi surfaces have a nesting effect, the three nesting vectors are no longer connected by the reciprocal lattice vector. Then the twisted bilayer graphene will be subjected to an inter-valley triple- nesting, and the inter-valley Coulomb repulsion induces a drastically distinct Mott insulating phase.

Moreover, in the magic-angle twisted bilayer graphene, unconventional superconductivity was also observed slightly away from the half-filling Cao2018SC . Naturally, one would ask whether this TLG-hBN heterostructure could also become a superconductor by doping away from the half-filling. If yes, what is the most probable pairing symmetry. From our previous analysis, the inter-valley scattering should still be the most important channel of pairing interactions, because the intra-valley pairing is not energetically favored due to the peculiar Fermi surface structures. If only the inter-valley Coulomb repulsion is considered, there is no privilege between spin singlet and spin triplet pairing. However, the inter-valley Hund’s rule coupling favors a spin-triplet pairing state. Therefore, the superconducting state in the TLG-hBN is most likely to have spin-triplet and valley-singlet symmetry. A detailed discussion on this will be given in a separate paper.

In conclusion, we have proposed a minimal tight-binding model to describe the low-energy states of the TLG-hBN super-lattice. At half-filling, the Fermi surfaces are perfectly nested between the two valleys when the valley-contrasting staggered flux of each Moire triangle equals . This leads to a strong inter-valley scattering and the system becomes unstable against an inter-valley spiral order. We believe that this inter-valley spiral ordered phase is just the Mott insulating phase observed in the experiments FengWang .

Acknowledgment: This work was supported by the National Key Research and Development Program of MOST of China (No.2017YFA0302900) and by National Natural Science Foundation of China (No. 11474331).

Note added: While in the preparation of this work, we noticed that two preprints Senthil ; FuLiang on the model for magic-angle twisted bilayer graphene appear. One of them Senthil proposed a similar tight-binding model for the TLG-hBN heterostructure.

## I Supplementary Material

The low-energy band structure of the TLG is composed of A-sublattice on the bottom layer () and B-sublattice on the top layer (), while the other sublattices are bonded by the on-site interlayer coupling and belong to the high-energy sector KoshinoMcCann ; MacDonald10ABC . Therefore, a low-energy effective Hamiltonian on the two-layer triangular lattice that accounts for the TLG:

(9) |

where is a two-component spinor. The warped triple Dirac Hamiltonian matrix is given by

(10) | |||||

where , (). The triple Dirac dispersion is inherited from the three layers of graphene. On the other hand, the hBN has a large atomic energy eV, and can therefore be integrated out, leaving a Moire potential contribution to the TLG Miller12PRB :

(11) |

where

(12) | |||||

where denotes the reciprocal lattice site. By exact diagonalizing , we obtain the low-energy mini-band structure of the TLG-hBN heterostructure. The calculation result in this paper is performed by truncating and . Besides, we perform the Fourier transform to change the Bloch wave function to the real space for each momentum :

(13) |

Then from the probability distribution , the summation over the valence band gives rise to the local density of states (LDOS):

(14) |

Since we are mainly concerned with the valence band, which is well separated from the conductance band, we do not distinguish the two layer degrees of freedom. Otherwise there would be a two band model instead. As the two valley degrees of freedom are related via the mirror symmetry , they have the same LDOS distribution, which matches a triangular lattice. Although the maximum of the LDOS is shifted to zones, the LDOS is not at all depleted in zones. The Wannier function is therefore located on either or zone. The tight-binding model of the low-energy valence band should therefore be defined on a triangular lattice with two valleys on each site. Without the sublattice degrees of freedom, it leads to an equivalent minimal model whether the Wannier center is located on the or zone, as both of them share the same symmetries and . For the symmetric reason, we have assumed that the Wannier center is located on the zones.

## References

- (1) M. Yankowitz, J. Xue, D. Cormode, J. D. Sanchez-Yamagishi, K. Watanabe, T. Taniguchi, P. Jarillo-Herrero, P. Jacquod, B. J. LeRoy, Emergence of super-lattice Dirac points in graphene on hexagonal boron nitride, Nature Physics, 8, 382-386 (2012).
- (2) C. R. Dean, L. Wang, P. Maher, C. Forsythe, F. Ghahari, Y. Gao, J. Katoch, M. Ishigami, P. Moon, M. Koshino, T. Taniguchi, K. Watanabe, K. L. Shepard, J. Hone, P. Kim, Hofstadter’s butterfly and the fractal quantum Hall effect in moire super-lattices, Nature, 497, 598-602 (2013).
- (3) B. Hunt, J. D. Sanchez-Yamagishi, A. F. Young, M. Yankowitz, B. J. LeRoy, K. Watanabe, T. Taniguchi, P. Moon, M. Koshino, P. Jarillo-Herrero, R. C. Ashoori, Massive Dirac fermions and Hofstadter butterfly in a van der Waals heterostructure, Science, 340 (2013) 1427-1430.
- (4) L. A. Ponomarenko, R. V. Gorbachev, G. L. Yu, D. C. Elias, R. Jalil, A. A. Patel, A. Mishchenko, A. S. Mayorov, C. R. Woods, J. R. Wallbank, M. Mucha-Kruczynski, B. A. Piot, M. Potemski, I. V. Grigorieva, K. S. Novoselov, F. Guinea, V. I. Falko, A. K. Geim, Cloning of Dirac fermions in graphene superlattices, Nature, 497, 594-597 (2013).
- (5) W. Yang, G. Chen, Z. Shi, C. C. Liu, L. Zhang, G. Xie, M. Cheng, D. Wang, R. Yang, D. Shi, K. Watanabe, T. Taniguchi, Y. Yao, Y. Zhang, G. Zhang, Epitaxial growth of single-domain graphene on hexagonal boron nitride, Nature Mater, 12, 792-797 (2013).
- (6) Z. Shi, C. Jin, W. Yang, L. Ju, J. Horng, X. Lu, H. A. Bechtel, M. C. Martin, D. Fu, J. Wu, K. Watanabe, T. Taniguchi, Y. Zhang, X. Bai, E. Wang, G. Zhang, F. Wang, Gate-dependent pseudospin mixing in graphene/boron nitride moire superlattices, Nature Physics, 10, 743-747 (2014).
- (7) R. Bistritzer and A. H. MacDonald, Moire bands in twisted double-layer graphene, Proc. Natl. Acad. Sci. USA, 108, 1223-12237 (2011).
- (8) G. Trambly de Laissardiere, D. Mayou, and L. Magaud, Numerical studies of confined states in rotated bilayers of graphene, Phys. Rev. B 86, 125413 (2012).
- (9) Y. Cao, J. Y. Luo, V. Fatemi, S. Fang, J. D. Sanchez-Yamagishi, K. Watanabe, T. Taniguchi, E. Kaxiras, and P. Jarillo-Herrero, Superlattice-induced insulating states and valley-protected orbits in twisted bilayer graphene, Phys. Rev. Lett. 117, 116804 (2016).
- (10) Y. Cao, V. Fatemi, A. Demir, S. Fang, S. L. Tomarken, J. Y. Luo, J. Sanchez-Yamagishi, K. Watanabe, T. Taniguchi, E. Kaxiras, R. C. Ashoori, and P. Jarillo-Herror, Correlated insulator behaviour at half-filling in magic-angle graphene superlattices, Nature 556, 80-84 (2018).
- (11) Y. Cao, V. Fatemi, S. Fang, K. Watanabe, T. Taniguchi, E. Kaxiras, and P. Jarillo-Herrero, Magic-angle graphene superlattices: a new platform for unconventional superconductivity, Nature 556, 40-50 (2018).
- (12) G. Chen, L. Jiang, S. Wu, B. Lv, H. Li, K. Watanabe, T. Taniguchi, Z. Shi, Y. Zhang, and F. Wang, Gate-tunable Mott insulator in trilayer graphene-boron nitride Moire superlattice, arXiv:1803.01985.
- (13) M. Koshino and E. McCann, Trigonal warping and Berry’s phase in ABC-stacked multilayer graphene, Phys. Rev. B 80, 165409 (2009).
- (14) M. Kindermann, B. Uchoa, D. L. Miller, Zero-energy modes and gate-tunable gap in graphene on hexagonal boron nitride, Zero-energy modes and gate-tunable gap in graphene on hexagonal boron nitride, Phys. Rev. B, 86, 115415 (2012).
- (15) J. R. Wallbank, A. A. Patel, M. Mucha-Kruczynski, A. K. Geim, V. I. Falko, Generic miniband structure of graphene on a hexagonal substrate, Phys. Rev. B, 87, 245408 (2013).
- (16) F. Zhang, B. Sahu, H. K. Min, A. H. MacDonald, Band structure of ABC-stacked graphene trilayers, Physical Review B, 82, 035409 (2010).
- (17) J. Li, A. F. Morpurgo, Markus Butiker, and I. Martin, Marginality of bulk-edge correspondence for single-valley Hamiltonians, Phys. Rev. B 82, 245404 (2010).
- (18) G. E. Volovik, The Universe in a Helium Droplet, Clarendon Press, Oxford, 2003.
- (19) H. C. Po, L. Zou, A. Vishwanath, and T. Senthil, Origin of Mott insulating behavior and superconductivity in twisted bilayer graphene, arXiv:1803.09742.
- (20) N. F. Q. Yuan and Liang Fu, A model for metal-insulator transition in graphene superlattices and beyond, arXiv:1803.09699.