It's "affection for". We'd put "to" after "affection" only if something else in the sentence called for it: eg He gave a lot of affection to his wife and kids.

In the sense of "feeling of love or strong attachment", it is often in the plural; formerly followed by "to", but now more generally by "for" or "toward (s)", for example filial, social, or conjugal affections; to have an affection for or towards children