"We will down the system for ..." is always the sentence that no one in the online business want to hear from their IT department; however, system upgrade always require downtime. Today, we delivered new functionalities to our system, mainly in the AGENT system.

To implement these auto/manual position taking functionalities, our DBAs had done lots of changes to our database. System down time were unavoidable and management allotted us two hours to deliver the software. 2 hours was supposed to be a lots of time for us. In the end, we just barely made it. All the replication servers had to be resynchronized, we had 7 replication servers. Of course, synchronization for all of these servers took some of our precious time.

My supervisor, my colleage, and I started our schedule earlier than usual to make it to the time allotted for the system downtime. My supervisor took care of resynchronization of all the data and all the replications. My colleague was to update the database code and to prepare new database structure scripts. I was in charged of the application code. Eventhough we had tested thoroughly with test scripts and passed the user acceptance tests, I was still a bit nervous. The funny thing was that I had done these things so many times, and still feel a little sweaty.

11:45 AM Vietnam time, the system was up again. It took us an hour to finish.

BUGS ... ERRORS ... we found those words again for this update. After new realeases were deployed, one of our DBAs - my colleague junior - figured one critical bug. And this bugs will only appear when agents and masters interact together on the same functions. This one was the "must" fix one ... I dealt with the bugs myself, even though those codes were developed by juniors and they were all present. While fixing bugs, it was a chance for me to check back their codes.

Even at the time of deploying, I also let my juniors to implement a small new request, I forgot this request somehow before the deploying. Last minutes implementation always still happens. Something called "PROCESS" isn't functioning well. I do know why and still observing. Precious experience.

I was really impressed with my junior; she is really improved. Definitely she could replace me, I will train her in depth everything I know.

I haven't finished improving odds display yet, so many "must" requests intervened. Anyways, I've gotta conclude it in the next 2 days...