The less you put down the more important it is to build a home that is "common" for the area that you are building in. Why? Sooner or later an appraiser will have to set a value for your home. This value is NOT based on the dollars you have in your build but rather the sale price of similar homes in the area, in the last year.
As an example, if the highest 3 sales in the area in the last year are in the mid to high $100's, you can't build a $300,000 home with a minimum down payment. It does mean you are being over charged, or that you may not be able to sell your home for that much. It's just impossible for the appraiser to start with those sale prices and make adjustments to validate your build price. Appraisers are given guidelines by Fannie, Freddie and HUD that limit these adjustments.
You can't build duplexes, triplexes or quads with this program. You can not build a Pole Barn home, geodesic dome home or Octagon home (just to name a few), with the government programs.
You can not build on "excessive" land. Once again, look at your sale comps. If all your sale comps are on no more than 5 acres of land, you can't build on 20 acres or more, as an example. All this changes if you are putting down 20% or more on a conventional loan. Then the home doesn't have to appraise for the full building price for your project to be approved.
So take a close look at the area you are building in. Find out what the highest sale prices are and stay at or below those numbers. Give me a call and I can do the research for you.