Given a topological manifold $X$, its mapping class group $MCG(X)$ is the group of isotopy classes of orientation preserving homeomorphisms $X\to X$.
Often this is considered specifically for $X$ a Riemann surface with punctures in which case a central role is played by Dehn twists?.
The mapping class group is of importance in many areas of geometry including study of Teichmüller spaces, of moduli spaces of surfaces, of automorphisms of free groups and in geometric and combinatorial group theory, hyperbolic geometry and so on. Some of the key contributors were Max Dehn, Jakob Nielsen, William Thurston, David Mumford. Recent proof of the related Mumford conjecture has been accomplished by Madsen and Weiss.
For $\mathbf{Aut}(X)$ the automorphism group of the manifold formed in Euclidean topological geometry, hence equipped with its canonical structure of a topological group. Let furthermore $\mathbf{Aut}_0(X) \hookrightarrow \mathbf{Aut}(X)$ be the inclusion of the connected component of the identity.
Then
is the corresponding coset space/quotient group. In other words, the mapping class group is the group of diffeomorphisms of $X$ onto itself, modulo isotopy.
If $X$ is a manifold with boundary $\partial X$, then it is usual to consider automorphisms which restrict to the identity on the boundary.
The classifying spaces of mapping class groups for 2-dimensional manifolds are encoded combinatorially in the geometric realization of a category of ribbon graphs. See there for details.
One of the classical results is that the mapping class group of the torus $\mathbb{R}^2/\mathbb{Z}^2 \cong (S^1)^2$ is isomorphic to $SL_2(\mathbb{Z}$ (more generally, $MCG(\mathbb{R}^n/\mathbb{Z}^n) \cong SL_n(\mathbb{Z})$). Certain generators called Dehn twists may be visualized as cutting a torus along a circle $\{a\} \times S^1$ (or $S^1 \times \{b\}$), thus producing a cylinder, then twisting one of the ends of the cylinder through $2\pi$ and reattaching the two ends.
Another example is a 2-disk with $n$ punctures. The group of diffeomorphisms (fixing the boundary pointwise) modulo isotopy is the braid group $B_n$.
Surveys include
Massyuyeau, A short introduction to mapping class groups (pdf)
Benson Farb, Dan Margalit, A primer on mapping class groups, draft, web
Nikolai V. Ivanov. Mapping class groups. In Handbook of geometric topology, pages 523–633. North-Holland, Amsterdam, 2002; 1998 draft: ps
See also
wikipedia: mapping class group
Ib Madsen, Michael Weiss, The stable moduli space of Riemann surfaces: Mumford's conjecture, Ann. of Math. (2) 165 (2007), no. 3, 843–941, MR2009b:14051, doi, math.AT/0212321
John Harer, The second homology group of the mapping class group of an orientable surface, Invent. Math., 72(2):221–239, 1983; The cohomology of the moduli space of curves in: Theory of moduli (Montecatini Terme, 1985), Lecture Notes in Math. 1337, p. 138–221. Springer, Berlin, 1988.
Robert C. Penner, The decorated Teichmüller space of punctured surfaces, Commun. Math. Phys. 113 (2) (1987) 299–339. MR89h:32044; A construction of pseudo-Anosov homeomorphisms, Trans. Amer. Math. Soc. 310 (1):179–197, 1988.
Alan Hatcher, William Thurston, A presentation for the mapping class group of a closed orientable surface, Topology, 19(3):221–237, 1980.
Max Dehn, Papers on group theory and topology. Springer-Verlag, New York, 1987. Transl. from German with intro. and appendix by John Stillwell, and appendix by Otto Schreier.
Jakob Nielsen, Untersuchungen zur Topologie der geschlossenen zweiseitigen Flächen. I-III, Acta Math. 50 (1927), no. 1, 189–358, MR1555256, doi; Acta Math. 53 (1929), no. 1, 1–76, MR1555290, doi; Acta Math. 58 (1932), no. 1, 87–167, MR1555345, doi
David Mumford, Abelian quotients of the Teichmüller modular group, J. Analyse Math., 18:227–244, 1967.
David Mumford, Towards an enumerative geometry of the moduli space of curves, Arithmetic and geometry, Vol. II, Birkhäuser Boston, Boston, MA, 1983, pp. 271–328, MR85j:14046
Ulrike Tillmann, On the homotopy of the stable mapping class group, Invent. Math. 130 (1997), no. 2, 257–275, MR99k:57036, doi
S. Morita, Introduction to mapping class groups of surfaces and related groups, in: Handbook of Teichmüller theory (A. Papadopoulos, editor), vol. I, EMS Publishing House, Zürich, 2007, 353–386.
E. Y. Miller, The homology of the mapping class group, J. Differential Geom. 24 (1986), no. 1, 1–14, MR88b:32051, euclid