VTOC (Volume Table of Contents) is the native filesystem of the IBM mainframe operating systems z/OS (previously called OS/390 and before that MVS) and z/VSE (formerly known as DOS/VSE). It originates with the OS/360 mainframe operating system released in 1965 but has evolved over the decades since.
Note that on z/OS and z/VSE, the term "dataset" is used instead of "file", but the meaning is fundamentally the same. On z/OS, the VTOC filesystem stores "datasets" whereas "files" (or to use their full name, "Unix files") are stored inside the Unix-compatible HFS and zFS filesystems. HFS and zFS filesystems themselves are stored as datasets inside the VTOC filesystem.
Note that other IBM mainframe operating systems, such as z/VM (formerly known as VM/CMS) use different filesystems instead.
The mainframe VTOC filesystem is supported by z/Linux as a partition table format, in which a Linux filesystem partition is stored in a single contiguous dataset.
Note that the term "VTOC" is also used on some other platforms (e.g. Solaris SPARC) to mean the partition table format, but that is a very different on-disk structure despite the similar name.