In fact, it took until the 16th minute of the game for the game's first chance. From a free kick on the right the ball dropped kindly for Chesterfield striker Evans on the volley but his powerful effort was caught well by Sam Walker in the Colchester goal.
The first half started slowly for both sides as they struggled to create any chances going forward.
However, parity lasted just five minutes as Evans' far post effort proved to be the difference.
A fairly quiet first half saw the Spireites take in a half time lead to the interval thanks to Jay O'Shea's opener, but Tarique Fosu drew the U's level straight after half time with his first touch after being introduced as substitute.
Colchester United bowed out of the Emirates FA Cup at the first time of asking on Saturday, as a Ched Evans goal separated the two sides come the final whistle.
Just under 10 minutes later came the game's second chance and this time Chesterfield took it.
Kristian Dennis found himself with time on the ball as he dribbled forward, he then slipped the ball through to O'Shea to slot the ball into the net to make it 1-0.
One nearly became two not long after as Chesterfield somehow didn’t double their advantage.
O'Shea received the ball on the left before cutting inside and shooting towards the far corner.
His shot was well saved by Walker who had seen the shot late, but it fell straight to Evans from the angle and his shot went straight across goal and away to safety.
Colchester's first chance came in the 34th minute, good work down the right by Drey Wright meant he was able to get a cross into the box. Kurtis Guthrie was the man arriving but his diving header fell just wide.
The U's best chance of the half fell to Wright after a clever build up involving Tom Eastman and Denny Johnstone, the latter finding Wright with a ball round the corner.
Wright then took on his first man and managed to beat a second, but his shot was closed down and saved well by Ryan Fulton in the Chesterfield goal.
Despite a slow start, Colchester looked to have been working their way into the game and they were rewarded in the opening minute of the second half.
Tarique Fosu replaced Kane Vincent-Young and the on loan winger scored with his first touch of the game.
A free kick came in from the right and after a scramble involving Guthrie and Johnstone, the ball fell to Fosu from a tight angle who finished well with a powerful shot across goal.
This seemed to swing the momentum of the match for a short period as Colchester then hit the bar after from a corner as Luke Prosser met the cross.
But Colchester were then punished minutes later, as Chesterfield scored against the run of play. The visitors retook the lead as Evans scored a superb volley as he met a loose ball from a cross to bury past Walker.
The U's then responded once again and could consider themselves unlucky not to be rewarded a penalty as Ian Evatt bundled Guthrie to the floor but the referee turned down any appeal from both the Colchester players and fans.
With just minutes left of the added on time at the end of the game, Macauley Bonne had another chance as he climbed highest to meet a cross from the right but his header was well saved.
Despite this chance, Chesterfield were able to see out the rest of the game and the win means they progress through to the Second Round of FA Cup.
Colchester will now turn their attention back to the league, as they look to get back to winning ways at home to Leyton Orient next Saturday, who sit second bottom in League Two.