The notion of Calabi-Yau category is a horizontal categorification of that of Frobenius algebra – a Frobenius algebroid . Their name derives from the fact that the definition of Calabi-Yau categories have been originally studied as an abstract version of the derived category of coherent sheaves on a Calabi-Yau manifold.
A Calabi-Yau category is a Vect-enriched category $C$ equipped for each object $c \in C$ with a trace-like map
to the ground field, such that for all objects $d \in C$ the induced pairing
given by
is symmetric and non-degenerate.
A Calabi-Yau category with a single object is the same (or rather: is equivalently the pointed monoid delooping) of a Frobenius algebra.
A Calabi-Yau $A_\infty$-category of dimension $d \in \mathbb{N}$ is an A-∞ category $C$ equipped for each pair $a,b$ of objects a morphism of chain complexes
such that
this is non-degenerate and is symmetric in that
for $\sigma_{a,b} : C(a,b)\otimes C(b,a) \to C(b,a) \otimes C(a,b)$ the symmetry isomorphism of the symmetric monoidal category of chain complexes;
this is cyclically invariant in that for all elements $(\alpha_i)$ is the respective hom-complexes we have
Let $X$ be a smooth projective Calabi-Yau variety of dimension $d$. Write $D^b(X)$ for the bounded derived category of that of coherent sheaves on $X$.
Then $D^b(X)$ is a CY $A_\infty$-category in a naive way:
the non-binary composition maps are all trivial;
the pairing is given by Serre duality (one needs also a choice of trivialization of the canonical bundle of $X$)
This is however not the morally correct CY $A_\infty$-structure associated with a Calabi-Yau. A correct choice is, for example, the Dolbeault dg-enhancement of the derived category
(Costello 04, 7.2, Costello 05, 2.2)
The Fukaya category associated with a symplectic manifold $X$. But see this MO discussion for more.
string topology: for $X$ a compact simply connected oriented manifold, its cohomology $H^{\bullet}(X)$ is naturally a Calabi-Yau $A_\infty$-category with a single object. The $A_\infty$ structure comes from the homological perturbation lemma. One could also use the dg algebra of cochains $C^\bullet(X)$.
Calabi-Yau $A_\infty$-categories classify TCFTs. This remarkable result is what actually one should expect. Indeed, TCFTs originally arose as an abstract version of the CFTs constructed from sigma-models whose targets are Calabi-Yau spaces.
Kevin Costello, Topological conformal field theories and Calabi-Yau categories (arXiv:math/0412149)
{Costello05} Kevin Costello, The Gromov-Witten potential associated to a TCFT (arXiv:0509264)
Maxim Kontsevich, Yan Soibelman. Notes on A-infinity algebras, A-infinity categories and non-commutative geometry http://arxiv.org/abs/math/0606241
Lee Cho, Notes on Kontsevich-Soibelman’s theorem about cyclic A-infinity algebras (arXiv:1002.3653)
Jacob Lurie, section 4.2 of On the Classification of Topological Field Theories (arXiv:0905.0465)