A 3" ordnance rifle had a range of about one mile at 5 degrees elevation so the 5,000 feet you mentioned is within the capability of the gun. However, at this range only fused shot or shell would have been effective. Canister would have been used at 400 yards or less, as Stewart's men got closer to the federal positions. In all probability, Capt. Marshal was in error when he said "opened with canister".
His report of ammunition expenditure indicates a long engagement of many hours, perhaps most of the daylight hours. It would take a long time to fire 700 rounds. Experience would indicate that you are correct in that Capt. Marshal opened before Stewart's charge at long range with shot and shell and continued with canister. Marshal simply made a error in stating he opened with canister. The Official Records of the Civil War are filled with too many writing and typo errors and cross referencing with other reports would clarify this error.
In my opinion, a correct civil war firing mission would be as outlined above. Canister is a special type of ammunition for special use and Capt. Marshal's mission that day was of a typical firing mission.
Hope this hepls
Ron