About two years, on and off. I only started to study "seriously" recently. Most of the difficulty comes at the beginning when you're starting to memorize a lot of shit at once (kana, grammar rules, kanji, etc). Once you hit an intermediate level, you begin to understand and gain more confidence. It's just overcoming the initial hurdle which turns off most people.
If you're looking to read manga, you can pretty much start once you've learned the kana and a few hundred kanji. A lot of manga has the readings above each kanji, so you can just look that shit up without too much of a hassle.
However, when you first start out, you have to be prepared to read at the glacial speed of one line per minute before you finally get somewhere.
Learning Japanese is a huge investment of your effort and time, but I think it's worth it if you feel you're missing out on a lot of material that translators haven't gotten to yet.
Feels good when you don't have to rely on other people to translate for you, and you don't have to worry whether a translation is accurate or not. Shit's worth it, man.