Will I get called for play overuse if I only use one Quarter/Dime defense against the pass only sets (you know the ones) and only one GL defense against the goal line sets?

Basically, I want to cut all of the fat out of my playbook so I can concentrate on what plays are going to get called and be effective.

Except for overusing blitzes, play overuse only gets kicked in against base offensive formations (so an offense can't force you into play overuse by staying in a typically less-used formation that you don't have enough plays to call against).

Does the play count against these non-standard formations count towards the overall count for that play?

For example: If I use play A against the 5 wide set 15 times, and then play A is used against a base formation, does that count as the 16th time, or the first?

So as long as I use say a Quarter 4 Deep Man against the passing sets and a GL 3 Deep Zone (the only one that plays Cover 5) against the GL sets, I should be golden, right?

It's counting the times you use the play, but the penalty is only applied against the base offense. So when it's used against a base formation, it counts as #16 in your example.

Yes, unless those are also set to go against a standard offensive set.

Never. I'm almost completely rules based on defensive game planning.

FYI goal line sets do pull overuse penalties:

https://private75.myfootballnow.com/watch/1444#269427

I think that falls into the overusing blitzes logic that JDB describes above.