Gilmore is a native of the Texas Panhandle, having been born in Tulia, Texas and raised in Lubbock, Texas. His earliest musical influence was Lukas Day and the honky tonk brand of country music that his father played as a bar-band guitarist. In the 1950s, he was exposed to the emerging rock and roll of other Texans such as Roy Orbison and Lubbock native Buddy Holly. He was profoundly influenced in the 1960s by the likes of The Beatles and Bob Dylan and the folk music and blues revival in that decade.

With Joe Ely and Butch Hancock, Gilmore founded The Flatlanders. The group has been performing on and off since 1972. The band's first recording project, from the early 1970s, was barely distributed. It has since been acknowledged, through Rounder's 1991 reissue (More a Legend Than a Band), as a milestone of progressive, alternative country. The three friends continued to reunite for occasional Flatlanders performances, and in May 2002 released a long-awaited follow-up album, Now Again, on New West records.

Gilmore spent much of the 1970s in an ashram in Denver, Colorado, studying metaphysics with teenaged Indian guru Prem Rawat, also known as Maharaji. Jimmie asked the guru what he should do with his life, and the guru told Jimmie to help manage a motel he had bought with contributions. In the 1980s, he moved to Austin. His first solo album, Fair and Square, was released in 1988.

Gilmore's fans admire his fine tenor voice, which delivers expressive, pure, country singing.

Gilmore also had a small but memorable role in the 1998 movie The Big Lebowski as a bowler named Smokey, an aging, emotionally "fragile" pacifist threatened with a pistol by the main character's right-wing sidekick (John Goodman). He has also been a guest on David Letterman, A Prairie Home Companion, and Fresh Air with Terry Gross.

Gilmore's son, Colin Gilmore, is also a singer–songwriter based in Austin, Texas.

Gilmore's song "Braver Newer World" is featured in the 1995 Noah Baumbach film Kicking and Screaming. In 2005 Gilmore released Come on Back, an album of songs his father loved. Gilmore said of the album, "This new album is a compilation of recordings of some old songs that my dad loved. I love them too, and it is a project very dear to me."