A coherent sheaf of modules is a geometric globalization of the notion of coherent module.
Let $(X,\mathcal{O})$ be a ringed space or, more generally, a ringed site.
A sheaf $\mathcal{E}$ on $X$ of $\mathcal{O}$-modules is
finitely generated, or of finite type , if every point $x \in X$ has an open neighbourhood such that there is a surjective morphism
from a free module to $\mathcal{E}|_{U}$, where $n$ is finite.
coherent if it is
finitely generated
for every open $U$ in the base space (resp. every object $U$ in the base site), every finite $p \in \mathbb{N}$ and every morphism
of $\mathcal{O}|_U$-modules has a finitely generated kernel.
finitely presented if there is an exact sequence of the form
with $p$ and $n$ finite.
Every finitely presented $\mathcal{O}$-module is finitely generated.
quasi coherent if it is locally – on a cover $\{U_i\}$ – presentable, i.e. for each $i$ there is an exact sequences
where $I_i$ and $J_i$ may be infinite, i.e. of $\mathcal{E}$ is locally the cokernel of free modules. For more see quasicoherent sheaf.
For a coherent sheaf $\mathcal{E}$ over a ringed space, for every point $y$ in the base space $X$ there is a neighborhood $V$ such that the $\mathcal{O}_X(V)$-module $\mathcal{E}(V)$ of sections of $\mathcal{E}$ over $V$ is finitely presented. On a noetherian scheme the notions of finitely presented and coherent sheaves of $\mathcal{O}$-modules agree, but this is not true on a general scheme or general analytic space; sometimes even the structure sheaf $\mathcal{O}$ itself is a counterexample (not coherent while finitely presented).
The notion of coherent sheaf behaves well on the category of noetherian schemes. On a general topological space, by a basic result of Serre, if two of the sheaves of $\mathcal{O}$-modules in a short exact sequence
are coherent then so is the third. All this holds even if $\mathcal{O}$ is a sheaf of noncommutative rings. For commutative $\mathcal{O}$, the inner hom $Hom_{\mathcal{O}}(\mathcal{E},\mathcal{F})$ in the category of sheaves of $\mathcal{O}$-modules is coherent if $\mathcal{E},\mathcal{F}$ are coherent.
A theorem of Serre says that the category of coherent sheaves over a projective variety of the form $Proj R$ where $R$ is a graded commutative Noetherian ring is equivalent to the localization of the category of finitely generated graded $R$-modules modulo its (“torsion”) subcategory of (finitely generated graded) $R$-modules of finite length.
First works on coherent sheaves in complex analytic geometry. Serre adapted their work to algebraic framework in his famous article FAC. Hartshorne’s definitions are changed/adapted to the special setup of Noetherian schemes with the excuse that the coherence does not behave that well otherwise; thus they differ from the definitions in EGA and FAC.
A. Vistoli commented at MathOverflow here that for some categorical purposes
one should interpret “coherent” as meaning “quasi-coherent of finite presentation”. The notion of coherent sheaf, as defined in EGA, is not functorial, that is, pullbacks of coherent sheaves are not necessarily coherent. Hartshorne’s book defines “coherent” as “quasi-coherent and finitely generated”, but this is a useless notion when working with non-noetherian schemes.
J-P. Serre, Faisceaux algébriques cohérents, Ann. of Math. (2) 61, (1955) 197–278, doi.
H. Grauert, R. Reinhold, Coherent analytic sheaves, Grundlehren der Math. Wissenschaften 265, Springer 1984. xviii+249 pp.
M. M. Kapranov, On the derived categories of coherent sheaves on some homogeneous spaces, Invent. Math. 92 (1988), no. 3, 479–508, doi.
D. O. Orlov, Производные категории когерентных пучков и эквивалентности между ними (pdf, Russian) Uspekhi Mat. Nauk 58 (2003), no. 3(351), 89–172; Engl. transl. Derived categories of coherent sheaves and equivalences between them, Russian Math. Surveys 58 (2003), no. 3, 511–591.
V. D. Golovin, Homology of analytic sheaves and duality theorems, Contemporary Soviet Mathematics (1989) viii+210 pp. transl. from Russian original Гомологии аналитических пучков и теоремы двойственности, Moskva, Nauka 1986. (192 pp.)
EGA 0.5.3.1
Qing Liu, Algebraic geometry and arithmetic curves, 5.1.3
Categories of ind-coherent sheaves on schemes and stacks are studied in Dennis Gaitsgory, Notes on Geometric Langlands: ind-coherent sheaves, arxiv/1105.4857