James Pinckney Pittman, son of Elias Pitman, was twelve years younger than his brother Henry Flowers Pitman. At Elias' death, James inherited the land that included Isham Pitman's grist mill and the surrounding area, including property that became downtown Fairmont proper. J. P., in 1870, constructed a rice mill as an adjunct to the existing grist mill. Apparently some enterprising farmers had figured out how to grow rice in the areas adjoining Ashpole Swamp and needed a place to process it.

Downtown was divided between Pittman land and Stephens land, with the Lumberton Highway (Main Street) serving as the dividing line. The area around the grist mill became Fisher Park, the land being inherited by Claudia Pittman Fisher and developed as Fisher Park after World War II by her and sons Willis, Pittman and Earl Fisher.

J. P.'s daughters married into prominent families of the area. Mollie, the oldest, married Andrew Justin Floyd, one of Ashpole's earliest merchants having opened for business in 1893. Claudia married Elijah Fisher, the railroad's depot agent who arrived shortly after the railroad began operating in 1898. Eula married Alex Davis of McDonald. Rose married J. Frank Davis, also of McDonald, later moving to Fairmont. Margaret Anne, the youngest, married Patrick R. Floyd, a son of Gus Floyd.