The notion of Calabi-Yau category is a horizontal categorification of that of Frobenius algebra – a Frobenius algebroid . Their name derives from the fact that the definition of Calabi-Yau categories have been originally studied as an abstract version of the derived category of coherent sheaves on a Calabi-Yau manifold.
A Calabi-Yau category is a Vect-enriched category $C$ equipped for each object $c \in C$ with a trace-like map
to the ground field, such that for all objects $d \in C$ the induced pairing
given by
is symmetric and non-degenerate.
A Calabi-Yau category with a single object is the same (or rather the delooping) of a Frobenius algebra.
A Calabi-Yau $A_\infty$-category of dimension $d \in \mathbb{N}$ is an A-∞ category $C$ equipped for each pair $a,b$ of objects a morphism of chain complexes
such that
this is symmetric in that
for $\sigma_{a,b} : C(a,b)\otimes C(b,a) \to C(b,a) \otimes C(a,b)$ the symmetry isomorphism of the symmetric monoidal category of chain complexes;
this is cyclically invariant in that for all elements $(\alpha_i)$ is the respective hom-complexes we have
Let $X$ be a smooth projective Calabi-Yau variety of dimension $d$. Write $D^b(X)$ for the bounded derived category of that of coherent sheaves on $X$.
Then $D^b(X)$ is a CY $A_\infty$-category in a naive way:
the non-binary composition maps are all trivial;
the pairing is given by Serre duality? (one needs also a choice of trivialization of the canonical bundle of $X$)
This is however not the morally correct CY $A_\infty$-structure associated with a Calabi-Yau. A correct choice is, for example, the Dolbeault dg enhancement of the derived category; see section 2.2 of Cos05.
The Fukaya category associated with a symplectic manifold $X$. But see mathoverflow for more discussion: http://mathoverflow.net/questions/13114/are-fukaya-categories-calabi-yau-categories
string topology: for $X$ a compact simply connected oriented manifold, its cohomology $H^{\bullet}(X)$ is naturally a Calabi-Yau $A_\infty$-category with a single object. The $A_\infty$ structure comes from the homological perturbation lemma. One could also use the dg algebra of cochains $C^\bullet(X)$.
Calabi-Yau $A_\infty$-categories classify TCFTs. This remarkable result is what actually one should expect. Indeed, TCFTs originally arose as an abstract version of the CFTs constructed from sigma-models whose targets are Calabi-Yau spaces.
Kevin Costello, Topological conformal field theories and Calabi-Yau categories (arXiv:math/0509264)
Kontsevich, Soibelman?. Notes on A-infinity algebras, A-infinity categories and non-commutative geometry http://arxiv.org/abs/math/0606241
Cho, Lee. Notes on Kontsevich-Soibelman’s theorem about cyclic A-infinity algebras http://arxiv.org/abs/1002.3653