A native of Ann Arbor, Michigan (born January 10, 1867), Duffy enrolled at the University of Michigan in 1885.
While attending the school as both an undergraduate and law student, he played halfback for the Michigan football team for seven years from 1885 to 1891.
 He was the team captain in 1888, and his brother John L. Duffy was captain of the 1887 team.
 Near the end of Duffy's career at Michigan, the Chicago Daily Tribune wrote: "Duffy is an old U. of M. player.
Although a swift runner and clever dodger his great strength lies in kicking ability, punting and drop-kicking."

As a rusher, Duffy had great speed.
At Michigan's May 1888 field day, Duffy won both sprint events, with times of 10.4 seconds in the 100-yard dash and 22.4 seconds in the 220-yard dash.
 Prior to the 1888 football game between Michigan and Notre Dame, the players competed in a 100-yard foot race, which Duffy won.
 Duffy also ran for two touchdowns in the 1888 Notre Dame game which Michigan won by a score of 26–6.

As a kicker, Duffy set or tied distance records for his era.
On May 22, 1886, at a field day in Ann Arbor, Duffy drop kicked a football 168 feet, 7½ inches.
Outing magazine reported that Duffy's kick broke "the best record of the world by 11 feet and 5 inches."
 A few days before his record-breaking kick at the May 1886 field day, Duffy had kicked the ball 194 feet in a practice session.
On the day of the event, he reportedly broke the record five times.
In 1888, the Detroit Free Press wrote:
Included in the team is J.E. Duffy, who holds the world's record for the drop kick, he having propelled the ball 168 feet seven and one half inches, at Ann Arbor May 22, 1886.
It may be explained that a drop kick is made by letting the ball fall from the hands and kicking it at the very instant it rises.
Duffy is a tall, well formed muscular player who gets over the ground like a wraith and fights for every point.

On November 22, 1891, Duffy also kicked a field goal from the 55-yard line against Cornell—a kick that either tied or fell one yard short of the American football record for the longest field goal to that time.

Michigan was undefeated in Duffy's first three seasons with the team, outscoring opponents 222 to 0 in the 1885, 1886 and 1887 seasons.
 With Duffy as captain in 1888, the team won its first four games, including two wins over Notre Dame, by a combined score of 126 to 20.

For the final game of the 1888 season, Michigan traveled to Chicago to play a game to raise money for charity against the Chicago University Club, a group described as a "picked team" made up of "eastern college experts."
 One of Michigan's stars, Horace Greely Prettyman, was ejected from the game for slugging.
Though Michigan lost the game 26–4, newspaper accounts credited Duffy's play.
In one instance, Chicago halfback Crawford (described as "a famous Yale man") came out of a heap of players, "dodging between the legs of the big fellows and knocking down the little fellows till Duffy got a full Nelson lock on him and hurled him neck over heels."
 Duffy accounted for Michigan's only points against Chicago, running for a touchdown in the second half "by good running and dodging."

In Duffy's first six years as a varsity football player at Michigan, the team compiled an overall record of 17 wins and 4 losses.
 In his seventh season, Duffy reportedly served as a player and coach for the 1891 team.
While official sources list only Mike Murphy and Frank Crawford as the coaches of the 1891 team, the Chicago Daily Tribune reported in November 1891 that the Michigan team was "coached systematically" by Murphy, Crawford, Horace Greely Prettyman and Duffy.
 The 1891 team finished with a 4–5 record, losing more games that year than in the prior six years combined.

Duffy received his undergraduate degree from the University of Michigan in 1890 and a law degree in 1892.
 While at Michigan, Duffy was a member of the Chi Psi (ΧΨ) fraternity.
A native of Ann Arbor, Michigan (born January 10, 1867), Duffy enrolled at the University of Michigan in 1885. While attending the school as both an undergraduate and law student, he played halfback for the Michigan football team for seven years from 1885 to 1891.  He was the team captain in 1888, and his brother John L. Duffy was captain of the 1887 team.  Near the end of Duffy's career at Michigan, the Chicago Daily Tribune wrote: "Duffy is an old U. of M. player. Although a swift runner and clever dodger his great strength lies in kicking ability, punting and drop-kicking."
As a rusher, Duffy had great speed. At Michigan's May 1888 field day, Duffy won both sprint events, with times of 10.4 seconds in the 100-yard dash and 22.4 seconds in the 220-yard dash.  Prior to the 1888 football game between Michigan and Notre Dame, the players competed in a 100-yard foot race, which Duffy won.  Duffy also ran for two touchdowns in the 1888 Notre Dame game which Michigan won by a score of 26–6.
As a kicker, Duffy set or tied distance records for his era. On May 22, 1886, at a field day in Ann Arbor, Duffy drop kicked a football 168 feet, 7½ inches. Outing magazine reported that Duffy's kick broke "the best record of the world by 11 feet and 5 inches."  A few days before his record-breaking kick at the May 1886 field day, Duffy had kicked the ball 194 feet in a practice session. On the day of the event, he reportedly broke the record five times. In 1888, the Detroit Free Press wrote:
Included in the team is J.E. Duffy, who holds the world's record for the drop kick, he having propelled the ball 168 feet seven and one half inches, at Ann Arbor May 22, 1886. It may be explained that a drop kick is made by letting the ball fall from the hands and kicking it at the very instant it rises. Duffy is a tall, well formed muscular player who gets over the ground like a wraith and fights for every point.
On November 22, 1891, Duffy also kicked a field goal from the 55-yard line against Cornell—a kick that either tied or fell one yard short of the American football record for the longest field goal to that time.
Michigan was undefeated in Duffy's first three seasons with the team, outscoring opponents 222 to 0 in the 1885, 1886 and 1887 seasons.  With Duffy as captain in 1888, the team won its first four games, including two wins over Notre Dame, by a combined score of 126 to 20.
For the final game of the 1888 season, Michigan traveled to Chicago to play a game to raise money for charity against the Chicago University Club, a group described as a "picked team" made up of "eastern college experts."  One of Michigan's stars, Horace Greely Prettyman, was ejected from the game for slugging. Though Michigan lost the game 26–4, newspaper accounts credited Duffy's play. In one instance, Chicago halfback Crawford (described as "a famous Yale man") came out of a heap of players, "dodging between the legs of the big fellows and knocking down the little fellows till Duffy got a full Nelson lock on him and hurled him neck over heels."  Duffy accounted for Michigan's only points against Chicago, running for a touchdown in the second half "by good running and dodging."
In Duffy's first six years as a varsity football player at Michigan, the team compiled an overall record of 17 wins and 4 losses.  In his seventh season, Duffy reportedly served as a player and coach for the 1891 team. While official sources list only Mike Murphy and Frank Crawford as the coaches of the 1891 team, the Chicago Daily Tribune reported in November 1891 that the Michigan team was "coached systematically" by Murphy, Crawford, Horace Greely Prettyman and Duffy.  The 1891 team finished with a 4–5 record, losing more games that year than in the prior six years combined.
Duffy received his undergraduate degree from the University of Michigan in 1890 and a law degree in 1892.  While at Michigan, Duffy was a member of the Chi Psi (ΧΨ) fraternity.
James E. Duffy (American football)