An internal set is a basic object of internal set theory, which is a model of nonstandard analysis. The idea for analysis is that in addition to usual numbers one may have also infinitesimals with which the rules of manipulating are somewhat restricted, so that one does not get into inconsistencies.
(Note that this usage of “internal” is very different from that of internalization. In that usage, an internal set would be simply an object — or perhaps some kind of setoid, depending on the sort of category one is internalizing in.)
Among internal sets there is a subclass of standard sets; ZFC axiom schemes apply to the standard sets; every internal set has its standard part which is a standard set. This operation is called standardization. Using theory of internal sets one can implement transfer principle? which is one of the basic principles of nonstandard analysis.
There exists a model with a further extension, where even larger class is introduced, so-called external sets containing internal sets containing standard sets. Internal sets are introduced by Nelson.
Nelson’s internal set approach is used in these references
Edward Nelson, Internal set theory: a new approach to nonstandard analysis. Bull. Amer. Math. Soc. 83 (1977), no. 6, 1165–1198.
Nelson is writing a book on nonstandard analysis in this approach: draft first chapter
A. G. Kusraev, S. S. Kutateladze, Nonstandard Methods of Analysis. Mathematics and Its Applications 291, Kluwer 1994 (has also short excursion into external sets)
Sergio Albeverio, Jens Erik Fenstad, Raphael Hoegh-Krohn, Nonstandard Methods in Stochastic Analysis and Mathematical Physics, Academic Press 1986 (there are also Dover 2009 edition and a Russian translation)