Its definitely doable. The 80 was available in various parts of the world from the factory with a 5-speed manual behind a couple different motors, including the 1FZ and 3F(E), which came in the North American spec 80s. You could easily do it all factory if you wanted. It won't be cheap though. I would recommend checking over on the www.IH8MUD.com forums for more info--there's tons of info on it over there and its way more active than this site.
I lived in Saudi Arabia for 6 years and had a 1FZ-F with the H151f manual 5 speed, part time 4wd, etc. It was so far my favorite vehicle. I had stock diff gears and was running 285 75 r16 tires. Even though the carby version of the 1FZ was a little lower on power than its injected sibling, I never really felt the need for more power unless it was time for a tune-up. I loved driving that thing through the dunes. I used to lead multi-day desert trips so we were often loaded down. I haven't actually driven an auto 80 series, but I feel like with the manual I could make really good use of the available power and torque. It wasn't a rock crawler, though, and there were a few times I wished for lower gears crawling through rocky washes and up rock steps. I'm a little biased, though; my "car" in the U.S. is a 1st gen 4runner with dual cases so about 240:1 with the turbo transmission and 5.29 gears. Even so, I would have a manual in another 80 series in a hearbeat. If it hadn't been COVID times when we left last spring, I may have tried to bring it back to the U.S.