Programmer's Log

Monday, August 21, 2006

The first weekend of a new soccer season had passed ... All servers ran smoothly yesterday ... There were 2 big games yesterday Chelsea -vs- Manchester City and Manchester United -vs- Fulham.

Agent server reached 320 requests/sec yesterday with at peak 2,010 members. On top of these members, External Admin and Internal Agent were also run on this server. A dual Xeon 3.2GHz has been served us well.

With 8 applications is running on a single server using default setting of IIS6 will have an effect on the performance and response time. In Kaohsiung, we have 2 dedicated servers for Agent System ... now back in Taipei, 1 single server handles all and on top of it External Admin, Internal Agent are now also deployed on that agent server. I guess my supervisor created challenge for me to improve the performance instead just issuing new servers for me. And his decision turned out to be good.

After we moved back from Kaohsiung, I quickly noticed the executing time of a request increased dramatically from a few hundress milliseconds to 3-4 seconds on average. I discovered that running 6 or more web applications on a server with one IIS worker process only caused the executing time of a request to be longer. I must increase the number worker process for IIS but in which way?

Our applications must write the their own logs into harddrive in order for us to trace errors and trace customer activities. Increasing number of worker process will increase the chances of I/O queues to occur since 2 or more threads will fight each other to grab the privileges to write the logs. When this happens, requests into the system will be failed. This problem will definitely occur, unavoidable when using 2 or more worker processes if we still want to keep the application logging. Logging has been trememdously helpful for us. I don't want to remove it.

Compensating solution ... Yes, have one. I found out just using 2 worker processes; each worker process will handle specific applications. Why 2, not 4, or 8 ? tried ... After a series of trial and errors, with our 2 CPUs server, 2 worker process works the best; increasing to 4 worker processes, number of request failed increased much more than the gain of executing time of a request.

This method works well for us in the past weekend. No major problem or any issues. Executing time of a request is being kept below 800 milliseconds on average. Number of requests failed is below 4000 requests/per busy night like this weekend. The statistical performance numbers are really convincing to me that this method works well.

I could claim that our 2 dual Xeon 3.2Ghz server could support 2,500 users - 3000 users, but still need to try in practice.

I improved External Admin Odds Controller. Lots of new functionalities were put in.
- Hide odds, Open Odds were implemented.
- Implementation of odds selections for the users to choose

I think recently I overuse Web 2.0 technology. Really need documentation on security issues of this technology

Time to go to back sleep now ...


Post a Comment

<< Home