We may say that finite mathematics is mathematics done internal to the category FinSet of finite sets or the mathematics of itself. The latter (but not the former) includes the basic arithmetic of natural numbers, since these are the cardinalities of finite sets; we can go as far as rational numbers this way, but not real numbers. This also includes a great deal of combinatorics, basic algebra, and elementary logic, although not many advanced topics.
In the foundations of mathematics, finitism is the philosophy that one should do only finite mathematics. In a weak sense, one should not assume the axiom of infinity; in a strong sense, one should even deny it by an axiom of finiteness. This make it impossible to do analysis as we normally understand it.
Finitism (in the weak sense of not accepting an axiom of infinity) is essentially the mathematics that can be done internal to an arbitrary boolean topos (at least if one is not also being predicative or constructive). For constructive mathematics as usually practised, one goes beyond finitism by positing a natural numbers object.
Although often considered an extreme form of constructivism, finitism in the strong sense (actually denying the axiom of infinity) can make excluded middle and even the axiom of choice constructively acceptable (and similarly make power sets predicatively acceptable). This is because even constructivists agree that these are true in ; it's the extension of them to infinite sets that the first constructivists objected to.
Ultrafinitism is an even more extreme form of finitism, in which one doubts the existence even of very large numbers, numbers which in some sense it is not physically possible to write down. The theory of ultrafinite mathematics is most well developed by Edward Nelson.