|Basic D&D - a successful commercial RPG.|
So why do some commercial RPGs succeed?
1) Usability out of the box. You take the book, read it cover to cover, and can start an actual game with people around an actual table. Good products, especially good "big" products (i.e. rulesets and settings), should always include a starter adventure with pre-generated PCs so that the book would be of immediate use to any group. This is the reason old-school D&D boxed sets always came with a pre-packaged adventure.
2) Editing and proofreading. Significant errata kills games. The game should be perfectly playable at its first printing, with no need for looking for errata online due to missing tables and similar screw-ups. Both old-school D&D and Classic Traveller could be played out of the box without needing much errata.
3) Focus on actual play. An RPG is a *GAME* first, anything else second. Mechanics should facilitate play. Fluff should be limited to what is useful in play. Setting histories of over 2-3 pages rarely figure in play, especially if they come in the first product in the line. Old-school D&D and Traveller, for example, came with VERY little fluff, yet were highly successful, as they were focused on play. In my Outer Veil setting for Traveller, for example, there are about 2 pages of setting history; and most other setting material is game-relevant (a few paragraphs on things such as culture, politics, etc with some hooks); flavor text is usually a paragraph or two at most at the beginning of each chapter.
4) Art and layout. Too much text with no art is painful on the eye. Tasteful stock art will do. Custom art conveying the feel of the setting is better. D&D always had art, especially in later editions. An index can help, too, and if you're publishing a PDF, make sure that it is indexed, bookmarked and fully searchable for maximum usability.
5) Aiming at the casual player. Most people who play RPGs are not the big enthusiasts, but rather people who play every 1-2 weeks for 3-5 hours each time over beer and pretzels (or Mountain Dew and pizza). They want a game which is easy to understand and use, requires little commitment, and which is fun from the get-go. Generally speaking, for the casual gamer, simple rules and quick chargen are preferable. The casual GM/DM/Referee/Judge, i.e. one with limited time to devote to the hobby, needs a game for which it is VERY easy to prep adventures. Old-school boxed-set D&D, or Classic Traveller for that matter, fit this to a T.
And, the most important -
6) Aim towards subjects and themes with a broad appeal. A lot of people dig faux-Tolkien fantasy, for example, and thus D&D appeals to them thematically. Traveller appealed very much to all Battlestar Galactica, Alien and Star Wars fans. Being overly unique and "original" can be bad for your game - it is generally better to take a common, even cliche, theme and build a good game around it than re-invent the wheel around something too focused.