MP4 is the popular name for the MPEG-4 standard, defined by the Moving Picture Experts Group, codified as ISO/IEC 14496. Saying just what the "MP4 file format" is, is a somewhat complicated question. Parts 12, 14, and 15 specifically refer to "file formats.".
Part 12 is known as ISO Base Media File Format. It is derived largely from Apple's QuickTime format and is described as a "general format for timed sequences of media data."
Part 14 is an extension of Part 12. The recommended extension for it is ".mp4", though ".m4a" is often used for audio-only files.
Part 15 is "MPEG-4 File Format for Advanced Video Coding." This specifies the use of Part 10 encoding with the Part 12 file format. the extension ".avc" is often used with it.
"MP4 audio players" don't always support the MPEG-4 file formats, sometimes supporting only the AAC audio encoding from MPEG-4, Part 3 (aka MPEG-4 Audio). These would more properly be called AAC files, with the extension ".aac".