I have two small boats, a 9ft and a 12ft, My 9ft has a 4hp and my 12ft has a 15hp,
I use both on the walls, The boat with the 4hp really takes a long time too get out there, once your out there, Its pretty cool, you can get super close (you should know your structure before you get close with any size boat)
Fishing the inside is never a problem (except with other boaters going too fast and too close)
The outside is great on a calm day,
but can be sketchy when the weather is not so good,
Another problem with the small moter is not having the power the get out of harms way quickly i.e.(Boat wakes, swells, wildlife and weather). That being said, a small boat is a great way to get on the water without spending a ton of money. IMO everyone should start with a small boat, it teaches you ho to handle a vessel on a moving body of water (ocean with tides and currents), Its like learning to drive in a Honda instead of a Hummer, Most importantly, operating a small vessel around many larger boats, teaches you to have a lot more respect and concern for others when you upgrade to a bigger boat.
Here is my small boat set up in live bait mode
The small boat in Bug mode
A small boat is deadly for fishing the walls, harbors and bays, shut your motor off and use your oars and the fish dont have a chance. Good luck with your boating, whatever size it is.