Can a set of rules decide what counts as a line of iambic pentameter?

Not quite, and the attempt is a good story. In the years between 1966 and 1977, linguists proposed three theories of verse structure, together called generative metrics, aimed at English iambic pentameter. Each tried to write explicit linguistic rules that would generate all well-formed lines of the metre and exclude any that were not.

The earliest and most discussed was by Morris Halle and Samuel Jay Keyser. They saw the line as a series of nominally ten Weak and Strong positions, and the test for a bad line rested on one concept, the Stress Maximum: a stressed syllable located between two unstressed syllables in the same syntactic constituent within a line. In their theory, if a Stress Maximum lands in a Weak position, the line is unmetrical.

The rules did capture things poets seem to know. Later generative metrists noted that poets treat polysyllabic words differently from monosyllables: a normally weak syllable may be stressed as a variation if it is a monosyllable, but not if it is part of a polysyllable. Shakespeare wrote "For the four winds blow in from every coast", where the stressed weak syllable is the one-syllable "four", but vanishingly few lines like "As gazelles leap a never-resting brook", where it is part of the two-syllable "gazelles". Poets differ, too: Pope followed this strictly, Shakespeare fairly strictly, Milton much less and Donne not at all, which may be why Ben Jonson said Donne deserved hanging for not keeping accent.

But the rules also let some prose through, because the Halle-Keyser system can identify passages of prose as iambic pentameter. And Derek Attridge argued the generative approach has not brought us closer to understanding why particular metrical forms are common in English, or why metre works so powerfully as a literary device.