For simplicity, I would recommend a .36 for small game and a .50 for the rest.
Smaller the caliber, the faster it fouls. On a hot and humid day, I may have to run a wet patch down the barrel almost every shot with my .36! The .32's are even more persnickity. Plus, availability of ramrod stock for this caliber is limited. And at that skinny of a ramrod, you are gonna need replacements far more often. You get about 104 balls to the pound with a .36, so you are looking at something just under 70 grains of weight for the ball. NOT a long range shooter. Even with my hottest loads, I was unable to print on 16" diameter paper targets at 100 yds with a 15 mph crosswind. But to be honest, off bench on a great day and no wind, I could barely make my groupings under 8 inches. I am nobodies sniper.
For a big game gun, I say go with the .50 because it is so easy to find accessories, precast roundballs, patches, etc, etc. It is enough with a higher powder charge to ethically take elk at REASONABLE range. And if you are dealing with smaller deer at close ranges, you can scrimp on powder. I shoot a light load of 70 grains in my .50 and have only had one ball recovered! All the others were pass through.
Getting a .40 to split the difference is not a great choice in my opinion because even the hottest loads are on the puny side and isn't even legal in many states. So if you have to get only one, get the larger caliber and learn how it shoots at "blooper" loads, midrange loads, and hot loads. If you get lucky, the only thing that changes is the vertical point of impact. My .36 shoots the same at 15 grains all the way to 65 grains of powder, each one just hits a little higher on paper at a given range.