Tonight we had encountered the database problem again. Main database CPU went up high again. This time, my supervisor intervened timely to kill off the physical network connection and prevented the system jammed. Everything happened transparently to the users before they even noticed. Closely monitoring the system performance at peak hours shows its true importance.

We narrowed down the problem that might cause the system to be jammed up. Our supervisor checked the user actions logs from database. At the time that windows performance monitor showed the signal of the jamming to be about to occur, the operators were doing settlement ( a process of calculating the win/lose of each tickets after a game is done ). We will monitor this problem again on Sunday night.

Around 3AM in the morning, another problem arised. This time, it was the network. Agent website, and a .NET member site could not be accessed. Ping plotter showed red color at the our ends. SingTel line was having problem. It wasn't the problem from the ISP, it was our internal server issues. There is a problem at our firewall.

In the ping plot we could see the route which a information packet is going around on the internet. It is like the tracert command of window XP. We could see all hops visually. By tracking the ping ploter, we could know where the problem of the network to occur; international gateway or in our DNS.