Pole barn homes, container homes, geodesic domes, Quonset hut homes, I've had calls about all kinds of odd building types. Unfortunately you can't use the FHA/VA Construction to Perm loan for any of these. The program exists to help build homes that are mainstream, not unique or special. Sorry, not my rules.
It's next to impossible to appraise these homes due to few, if any, sale comps.
Lenders also have to think about what happens if they have to foreclose on any of these homes. They want to resell them as quickly as possible to minimize their loss. It's much easier and quicker to sell a home that appeals to the majority of buyers, not a small group.
If you want to build one of these homes you are going to need a large down payment, 30-40% or more. Even then you're going to need to shop around for a lender. Most lenders, even conventional lenders, have rules in place about not doing these loans.
Don't even think about building one of these homes if you have any past credit issues. If a lender is going to fund one of these homes they want to see not only a large down payment but outstanding credit, low debt to income and significant cash reserves. In short, they want to be very certain that they are not going to have to foreclose on one of these homes.
You can't finance a log home with the FHA/VA program. If you have a 20% down payment, good credit and sale comps you should be able to get financing with a conventional lender. Make sure you ask upfront about funding a log home. The producer of the log package is going to want to be paid for the logs before they are shipped. Many lenders will not want to pay until the logs are part of the home.