Typically when we think of poetry, we picture poetry of regular rhyme and rhythm. For example, eight syllables per line with every other line in rhyme, four lines per stanza. When those poems are put to music, they are often set to music that is similarly very regular, e.g. eight notes per phrase, four phrases per verse. This is especially the case with hymns. But as I listen to the melodies I consider to be the most beautiful, and after attempting to write such a melody myself the other day (in a piece called "Departure", which a friend of mine plans to do with his school concert band), I notice that they are irregular. Irregular does not necessitate "weird" or awkward. Rather, if done well, it means adding a dimension of the unexpected, adding drama, feeling, like a person pouring their deepest thoughts without reserve or artificial alteration. Let's take as an example one of my all-time favorite classical pieces, Rachmaninoff's Piano Concerto No. 2. You can listen just up to 2:50 in the video below to hear the melody in the strings that soars for almost two minutes non-stop, but so wonderfully shaped and felt (the video starts at the strings melody for you).
Desiring to craft songs of this beauty, with deep and meaningful words, I asked the question, what kind of poetic structure would be required to fit into such a melody? So I analyzed the two melodies I already mentioned, and found the following, with "/" indicating a stressed syllable, "x" an unstressed, "-" a held note, a space is a break in the melody's sound, and a new line is when there is more or less a new phrase in the melody. For the first 2:50 strings of the Rachmaninoff we have:
/-x/-x/-x/x/
/x/x/-x/-/-x/x
/-x/-x/-x/x/
//-x-/xx/-x/
x/-x/x/-x/x/-x/
x/x/-x/x/x/-x/-xx/
/x/x/x/x//-/-x//x/-x/-x/x/x/-x/x/x/-/-x/x/-x/x/-x/-x/
x/-x/x//x/-x/-/-x/x/-x/
Notice while it is certainly irregular, there are definite patterns as well, which I think is evident enough visually that I don't need to verbally analyze it. Most important to note is that such a varying structure is at the core of this amazing melody.
Similarly for my melody in "Departure" we have for the entire three minutes of the piece (hopefully we'll have a recording in the future):
xx/-xx/
xx/ x/x/
xx/-xx/
x/x x/-/xx/xx/x x/
xx/-xx/-/xx-x x/-x x/xx/
xx/-xx/
xx/-xx/
xx/-xx/
xx/-xx/x/
/xx/xxx/-x
/x /x /x /x
/x /x /xx/xx
/x /x/x/x/x/x/
/-x/x/x/xxx/ x/ x/ x/
One similarity between the two I see is that the phrases (lines) tend to get longer over a group of phrases.
One more important thing to note though is that a regular poetic rhythm can in fact be set to an irregular melody, by (judicial) use of repetition, and perhaps melisma.