Tokenized BASIC
Tokenized BASIC is a method of storing programs in the BASIC programming language by encoding the various keywords of the language as "tokens" instead of as plain text. Since the tokens are shorter byte sequences than the full text of the keywords, such programs take up less storage space in memory and in external storage such as disks or tapes, which was a significant concern in an era when computers were much more limited in memory and disk space than they are at present. It can also take less processing time for the interpreters to parse the code when it is in the form of tokens, which is another important concern for slower computers. Since computers are much faster and have much more memory and disk space now, tokenized languages are rarely used for source code storage, though compilers may generate intermediate data that is tokenized in some way in the course of producing executable code from text-based sources.
In its heyday of the 1960s through 1980s, BASIC existed in many dialects, designed for specific machine platforms, and the format of tokenized programs was different in each. On systems where file types were commonly identified using extensions, .BAS was usually used for BASIC programs, while other systems had their own ways of identifying file types and often had a type code specific to their own platform's BASIC interpreter (or multiple codes for different versions of BASIC, such as Apple II DOS's 'I' for Integer BASIC and 'A' for Applesoft floating-point BASIC).
Some types of tokenized BASIC that were used:
- Apple Integer BASIC
- Applesoft BASIC
- Atari BASIC
- Commodore BASIC
- GW-BASIC / BASICA (IBM PC and compatibles)
- TRS-80 BASIC
