I have already written about old-school Traveller several years ago when I wrote of the "Three Creeps" - Modifier Creep, Complexity Creep, and Scale Creep - that is, gradual and (at least partially) unintentional processes of change in Traveller which made later Traveller products significantly different from the earlier game. But a recent discussion on the Citizens of the Imperium Forums brought forth again the subject of Proto-Traveller, that is the Traveller "Old-School" movement. For your reading comfort, pleasure, and ease, I will provide here a consolidated and clarified version of what I posted in that thread.
Proto Traveller is not necessarily a time period - as High Guard came in 1979, before the accepted 1980 "cut-off" point - and not necessarily a "tight" set of rules. It is an attitude, a style if you will. It is Traveller in the spirit of the first three books of the original boxed set, as well as early adventures, JTAS articles, and supplements. That is - small ship, small setting, simple rules. I'd also argue that it has a focus on civilian or quasi-civilian (ex-military) play rather than strictly military settings and plots as suggested by the weapon list in Book 1 and as opposed to the mercenary unit play of Book 4, the huge-combatant naval play of Book 5, and the strict military fleet-building play of Trillion Credits Squadron.
This Proto-Traveller attitude consciously and intentionally resists the Three Creeps - Complexity Creep, Modifier Creep, and Scale Creep.
Complexity Creep was introduced by Book 4 with its Advanced Character Generation system. Book 1 character generation was very simple and straightforward - and incredibly quick. Book 4 introduced a much more complex system for generating characters, and books 5-7 continued this trend. Book 5 put forth a ship design system which was far more complex than the one in Book 2. Book 8 suggested a highly complex robot design system similar to the similarly complex Striker wargame rules. The Proto-Traveller attitude resists this complexity and desires a return to the simplicity of Book 1/Supplement 4 character generation and of Book 2 ship design.
Modifier Creep began with Book 4 as well. The 2D6 curve used by Traveller is highly sensitive to modifiers, and hence the strictly limited skill acquisition in Book 1 character generation, as well as the moderate to-hit modifiers of most (though not all) Book 1 weapons. Book 4 introduced characters with much more skills and much higher skills. It also introduced advanced weaponry bearing massive to-hit modifiers guaranteeing auto-hits - and usually also auto-kills - on almost all targets. Books 5-7 continued this trend with their Advanced Character Generation systems. The Proto-Traveller attitude prefers smaller modifiers and more limited skills.
Scale Creep began with Book 5. Suddenly, instead of small, relatively affordable ships - you have massive dreadnoughts. Such vessels are monstrously expensive and thus require a similarly massive polity to support them. They are also far beyond the scale of player-centered starships, unless, of course, the players are Big Navy captains or admirals. Book 5 focuses on major naval engagements, not the affairs of merchants, scouts, and corsairs - or at most a mercenary company or a local small-scale naval patrol, as in Book 2. This soon escalated into a huge Imperium many Sectors across, supporting similarly huge navies. The Proto-Traveller attitude prefers smaller ships, smaller empires - and a greater focus on a small group of characters rather than on wider affairs of state and navy.
To expand my explanation above, another potential (fourth) "creep" in later Traveller was (possibly) "Military Creep". Note that the first book has ex-military characters who have clearly mustered out; they might have combat skills, but they are no longer soldiers. Also, note the Book 1 weapon list - all small arms, with the auto-rifle being the toughest gun around. Further, note the ship list - there are some paramilitary ships, even two "cruisers", but they are all small ships that a small PC-owned ship has a certain chance of defeating in combat, especially with missiles.
This is very similar to the Alien, Firefly, or The Expanse premise - characters might be ex-military, but they now work on civilian or at most paramilitary jobs, have a civilian starship (in The Expanse they do acquire a small military ship, but nevertheless not a big warship), and tot small arms. A single Xenomorph is a huge threat to the trader crew who at best have flamethrowers. Gangsters and outlaws with revolvers and shotguns are common foes. A real battleship is something you try to avoid or outsmart, not fire guns at. At most in Book 2 ship terms you can have a mercenary platoon, again armed with small arms for the most part.
Book 4 introduced military-grade weaponry and a big discussion of field artillery and armored warfare. Granted, it is a specific boo for mercenaries, but it did start the military "creep". Add to that huge battleships with all sorts of Naval guns in Book 5, and soon things start escalating into a very militarized outlook which, IMHO, MegaTraveller was the peak of it. It went from Firefly to Babylon 5 (both magnificent sci-fi, IMHO), from Dumarest to Honor Harrington. Especially the later seasons of Babylon 5 where the action often involved (wonderful) space battles between major combatants. There is of course nothing wrong with "Big Military" games - which are enormous fun - but it's very different than the more low-key picaresque adventures of a tiny starship's crew or very small mercenary detachment as depicted by the three first books.
Also, note that in Proto-Traveller you might deal with military themes such as in Kinunir or Chamax/Horde. But you always deal with them as civilians or ex-soldiers unwittingly caught in military affairs - PCs are never generals or fleet admirals or even the commanders of warships.
Finally, you should keep in mind the inspirational "source material" for Proto-Traveller, that is the science fiction stories it was inspired by. They usually had a picaresque element to them - usually, the protagonist, such as Dumarest, travelled the stars and ended up in all sorts of weird and cool situations. The focus was far less on large-scale setting construction and more about exotic and interesting locales to explore.