A formal group is a group object internal to infinitesimal spaces. More general than Lie algebras, which are group objects in first order infinitesimal spaces, formal groups may be of arbitrary infinitesimal order. They sit between Lie algebras and finite Lie groups or algebraic groups.
Since infinitesimal spaces are typically modeled as formal duals to algebras, formal groups are typically conceived as group objects in formal duals to power series algebras.
One of the oldest formalisms is the formalism of formal group laws (early study by Bochner and Lazard), which are a version of representing a group operation in terms of coefficients of the formal power series rings. A formal group law of dimension is given by a set of power series of variables such that (in notation , , )
Formal group laws of dimension proved to be important in algebraic topology, especially in the study of cobordism, starting with the works of Novikov, Buchstaber and Quillen; among the generalized cohomology theories the complex cobordism is characterized by the so-called universal group law; moreover the usage is recently paralleled in the theory of algebraic cobordism of Morel and Levine in algebraic geometry. Formal groups are also useful in local class field theory; they can be used to explicitly construct the local Artin map according to Lubin and Tate.
Much more general are formal group schemes from (Grothendieck)
Formal group schemes are simply the group objects in a category of formal schemes; however usually only the case of the formal spectra of complete -algebras is considered; this category is equivalent to the category of complete cocommutative -Hopf algebras.
For a generalization over operads see (Fresse).
Formal geometry is closely related also to the rigid analytic geometry.
(nlab remark: we should explain connections to the Witt rings, Cartier/Dieudonné modules).