Programmer's Log

2006-12-01T20:22:00.000+07:00

2006-10-24T23:18:00.000+07:00

It has been 14 days since I last updated my blogs. Time sure flies. My schedule was filling with huge amount works and lots of problems on the performance sides of the system. I stopped implementing new requests, passed my tasks to my juniors, and concentrated more on improving the performance of our previous implementation.

We launched 3 sites of our .NET system. For the first week, everything went fine except for the odds information display trouble. The second week of launching, we started running into problems. Everything also started with the odds information display. Our odds cache wasn't stable. In order to relieve the load to the web server and the system itself we implemented a middle layer with .NET Web Services to obtain the odds information. We implemented it in the way that there was a infinite polling mechanism to get the information from the database and kept that information always fresh in the cache of the Web Services. For our problems, our polling mechanism always stopped unexpectedly. Before launching the new 3 sites, I worked around this problem by stopping the IIS to recycle its worker process on IIS schedule. This idea worked fine when we had less sites and the more sites we had problem started showing. The complains that we got from the customer more and more as my 3 sites stayed in the production. I listed down 3 things, I had to solve immediately.

Odds Cache -> use .NET Cache Object because it has the automatically expiration parameters for each objects stored in there.

Improve the display speed of odds information.

Check up our server. It starts to respond really slow on some requests

I tried to improve the odds display first. The solution for this was so much easier than I thought, I just changed the way of calling a function which is used to create our betting slip. Something could surprise people that it had an impact on the rendering engine of Internet Explorer 6.

<span onclick="a();">0.98</span>
<script type="text/javascript">
function a() { parent.frameA.buildSlip();}
</script>

It was not just out of no where I made this changes. I went to read some articles about IE having some memory leak issues in its javascript engine. And the part of circular references of register onclick event caught my attention.

I changed the odds cache to the new way using the Cache Object, and did more tunning on our display scripts for my last chance before the management's orders to switch 2 of our sites back to the good old ASP version. My tests showed positive results, deployed new changes to the production was the last thing I did before leaving the office.

On our critical night, our web server drive was filled up with logs files from IIS. I forgot to turn off the logs file. Next morning, my email was full of complains from the operators. :( They should've called me ... As promised with the management, we switched back 2 sites to our old versions.

... and to be continued ...

2006-10-10T11:59:00.000+07:00

Another week started with a lots of goals to achieve. Our first iteration of the new implementation of our Admin 1 system will conclude at the end of this week. My part hasn't done yet. I'm behind my own schedule.

Last week, our external consultant brought in a specialist who was from Taiwan to help us on improving our system. The specialist was introduced as Microsoft Product expert. However, the first impression we had for him was totally opposite. He introduced COM+ to us, a technology which Microsoft is replacing with .NET. The whole week, we spent our time in those COM+ meetings. The really good thing he tried to implement for us a COM+ server to try instead of telling us to implement what we didn't want to do.

We tried the COM+ servers over the weekend. Coincidently, the last weekend all the major leagues were off for the EURO 2008 qualification rounds. The load of the system won't be so high. My supervisor took one of our largest sites to test how the COM+ performed. The site was slower than one without COM+. It was obvious that things's slow down because now a request from a web server had to go through COM+ server to get the objects and after than came back to the COM+ server to execute the request. COM+ server introduced an intermediate step to connect to the database. Personally, I think this is one of the reasons for Microsoft to drop the COM and COM+ technology by .NET. ADO.NET layers provides the same thing what the COM+ does in term of reducing the number of connections from the web servers to the database servers with a pool of SQL connections. I also think our specialist knows this, and I hope that too.

Advanced Betting System from London came to our company in the middle of the week to introduce their betting software for bookies. First time, we got to attend these demo like this. It was a really well-organized system, and using service oriented software platform. They didn't use XML as the data transportation, but another Servlet thin layer instead to reduce the amount of data to transfer over the network. This is exactly what we want our software to be like in the future. My colleague was talking about this couple months ago and we are trying to design our Admin 1 system in the way that later on we could switch our system into services oriented software platform in the easiest way. My colleague is really good at visioning and look for the future development of our software. His Master degree in Computing Science shows its true values.

Saturday night was really peaceful for us. The load to the database wasn't high. There were no major problems except some little bugs on the interface of our .NET member sites. We had launched 3 more small sites and the .NET system is increasing its portion in our system.

... Sunday, our off day ...

2006-10-01T23:54:00.000+07:00

Being too confident has become a bad characteristic for us. Our headaches - our DB performance problems - came back. Our database was jammed again this Saturday.

Last week, we worked hard to get most the load of main database out to replication. Going to this Saturday, we were pretty confident that there would be a smooth night. 9PM peak time came with more than 40 live matches ranged from all the soccer leagues around the world. As usual, all the performance graphs and SQL Server profilers were turned on. This time I have learnt how to set up a profiler for myself. I also need to monitor the Admin 2, yesterday I continued to move out some pages to access the replication instead of the main Database. Everything was running smoothly. The main DB CPU usage was only around 10%. We started to relax a little.

There were some couple problems with our ASP sites. My supervisor changed the way we stored our odds change time. His display odds javascript didn't run correctly anymore. Problems were soon gone away, since he is my supervisor ... hehe i'm good he must be better ... jeese starting to be cocky ... (I should look at my first sentence again *winks*)

Since all the servers were running nicely. I went to fix bugs in our .NET member sites and ready for the launch of 9 more sites in the futures. All the sites must pass the acceptance tests. We failed 3 times already; however, the errors were not from our codes, it was just that we didn't have the correct data and enough data for the testing. The data we copied from the data warehouse into our development environment wasn't enough. I decided to deploy the rest of 9 sites into real production environment for a final testing.

11PM, the peak time almost passed and the load became less. Everything was running smoothly. My colleague left work. I still had some deployment on the way, and I decided to stay a little late. At the time I looked at the performance graph for my usual check. Something unexpected happened. The performance graph showed unusual behaviours. Database was jammed again. I looked over to my supervisor, and we started receiving complains.

... SETTLEMENT again ...

The operation did the calculation for the win/loss after the matches were over. This is called settlement process. And the database was jammed when the settlement code was executing. The problem was why it didn't happen when my supervisor executed the settlement himself. How to solve this? What could we do now? We are running out of options.

... I was off on Sunday ... but still kept monitoring the system from my house ... next week, we will have to continue to look into this issue again.

Database is not my specialty ... the only thing I know is to listen to my colleague and my supervisor ... well I will learn ...

This Saturday, our external consultant also invited a SQL specialist who was from Taiwan to help us to improve the database performance. He introduced us to COM+ ... a technology which Microsoft is replacing with .NET Remoting and Web services. The first impression from us wasn't too good for him. Let's see what he will have for us.

2006-09-28T23:33:00.000+07:00

Adapt to "constant" changes

Technology is changing every day, we have to update ourselves constantly to keep up with the pace. In 2002, our software was implemented with ASP and SQL 2000 and now we are replacing the old software with ASP.NET 2.0 and SQL 2005. Next year, we may change to something else ... who knows ... It is not only just the technology changes but also customers' requirements also change day by day. Today the requirement will be a square; but tomorrow, the square may turn into an oval. The businesses are changing, the requirements must be changed as well. To cope with new demands, new technology will be employed, and new ways of doing things will be used. Putting everything together creates a cirle. That cirle creates an indefinitely flow for those who choose to be in this technology industry.

Last night, I brought up new features to our Admin 2 website as well. I also found couple of bugs, bugs in the new features and also bugs in one of the old features.

Special character in the team name, my javascript will return an error when this team name occurs

Back functions of the live delete trace is not working

These were rare errors that's why our operators could not pick them up.

Today we start implementing the Company Admin website after a series of delay due to something called "PROCESS". The management wanted "PROCESS" and "PROCESS" reduced the productivities. "PROCESS" is necessary for any development firm; however, implementing it could not be done in a sort period of time. I have learnt a lots out of this process improvement period. For our company, heavy-weight process like CMMI is too much because our development team is pretty small. Light-weight process like Agile Development is more suitable to start with.

... first iteration is on the way ...

2006-09-25T15:11:00.000+07:00

Weekends has become our days for tuning and monitoring for the performance of the whole system. Our database performance issues is always on top our our lists. Last weekends wasn't an exception. We had done through a series of changes to improve our system. Costs and expensive lessons always accompanied with those changes. Expensive lessons were learnt, costs was paid under the expense of 3 hours downtime of our system on one of the most busy day of our business.

Saturday

Last week we implemented hardware load balancing. The load balancing worked like a charm. The hardware sent additional cookies in the URL and the response sent back to the firewall, based on the cookies it will send the requests to the correct server. Couple months back, we tried this method and we failed. Now with new hardware, it works so good. I won't have to do the software load balancing anymore for our Agent system.

Our night started with unusual behaviours of our servers which were detected from the graph of Window Performance Tools. At the moment, we have 2 servers for the Agent System. We deployed our Admin 2 on our Agent Server #1. This enabled us to have an idea about how our main database performed.

Explanation: The top window in the picture was the monitoring graph of our Agent Server #1, the underneath window was the the monitoring graph of Agent Server #2. The second graph showed the normal activities of a server. Red Line: CPU usage of the server; Blue Line: request executing in the CPU; Yellow Line: requests comming to the server per second. The first graph indicated that the database was being locked, and our website became slow on the customers' sides. The request executing ran high that meant there were request comming and some of them had to wait for the database to timeout. Time to do something to the database to release all the locks.

Explanation: This picture showed the normal activities of both Agent Servers.

The main CPU database remained at around 30%, but we still received complains from our operation that our web admin site was slow, and the Admin 2 as well. Database was jammed again? My supervisor then killed the physical connection which led to our main database. The situation became better. We learnt one more thing. When the database CPU is low, it doesn't mean that the system will perform normally. Even though the CPU is low, the database is still locked at somewhere.

My colleague added more counters into the Window Performance Tools to monitor the "lock requests" and "lock waits" for the database. After a while of monitoring, we found what we wanted. Before the database was jammed, the "lock waits" counter jumped higher, and so did the "lock request". At the same time, the performance tools on our Agent Web Servers also showed "Request Executing" Counter getting higher. My supervisor had to kill the physical network connection again in order for our database solved all the current locks.

We continued to monitor some more again. There were still lots of reading in our main database

balance of our agency system

bet list of nearly 40,000 customers

reports of our admin 2

total bets, and forecast in our company admin

"We will move all those read out of our main database." That was the conclusion for that we had for the night. I was in charged of changing the code of the agent system, and we encountered a big issue. Our current source code inside the repository was the ongoing development code. I could not fix on that source code. I had to rip out the production source, created a new project and made my amendments in there. Big room for errors was awaiting me.

Hardware issues sometimes also had affects on the performance? Our external consultant thought the hyper thread functionalities might create the performance issue as well. The external consultant also decided to turn the hyper thread feature off on the next reboot time of the system.

Sunday

Another weekend I had to come to work, and my supervisor has worked like this for the past 2 years to maintain the system up and running. I guess I'm starting following him. Well, it is the nature of our jobs as programmers.

I had 2 jobs to finish

remove all report store procedures call from our admin 2 to replication database

bring master agent list to replication and put some delays for the customers to see the new updates from replication

I finished the first task weekly. I grabbed my colleague junior to test my new changes while I proceeded the next task. I guessed he was being unlucky when he let me see him online on his off day. The test was fast, and everything was working fine. I deployed new version for our admin 2 quickly. Everything proceeded so smooth, and I should be out of the office pretty soon I thought.

Yesterday, before I left, I already set up the new project for our agent code in order to implement new codes without bringing our ongoing development into our development servers. I already tested the codes in our development environment. Everything rans smoothly. After getting signal from my supervisor, another version of agent systems were deployed again in our servers. I made a test site to test our new codes. I got errors, totally could not log in. There were strange errors. Parameter could not be null. Parameter value: value! What was the h*** going on? In the development system, everything was working fine. I had to revert on our old code back, and tried debugging the codes. After a while, I found out our new codes for new functionalities was causing problem. I passed in a null value into the String.IndexOf function of .NET . It was totally going off my mind when I saw the error message like that for that kind of mistake which was thrown out by the .NET Framework.

Database was restarted. Anxiously to wait for the main DB to be up again, strange things happened. In the performance windows, the CPU of the main DB quickly raised up to 100%. Our supervisors killed the LAN connection to the main DB again. After he killed the connection, the power in his house went out and it was time for him to come to the airport to catch the plane to Manila. No one turned the LAN connection back on. I had to call our external consultant to turn the connection back on. There were no remote connection for me to go in anymore. When the DB was back on, the CPU raised up to 100% again. We thought that it'd be the hyper thread function turned off causing the problem. Main DB was shut down again and restarted.

Our external consultant monitored the SQL profiler and told me everything single SP call from the agent system was going to the main DB instead of going replication servers. I moved the old codes back immediately and I grabbed my colleague to monitor for me and I went to figure out what went wrong. There were 2 things I picked up.

My junior changed my default setting inside one of the main components of our agent software packages.

The default connection to the main DB wasn't changed, it must be changed.

This was totally my mistake. I should've checked more carefully. Expensive lessons for me, I should've been more careful, and replaced the code one by one from the least access sites to the more access sites and always monitored the SQL profiler along the way. I underestimated the changes I did. The system I built and I still made mistakes. After fixing all the bugs, I learnt this time, and replaced the agents website one by one and monitored closely the profiler which my colleagues had set up for me.

After 3 hours downtime and unstable performance, our system was back on again. At the same time, my supervisor was doing the peformance improving implementation as well, he moved all the report from our member sites to replication. I decided to stay at work through the peak time to monitor a bit more closely, and to make sure everything running normally. Our system performed extremely good, no more jams for the night. Our agent servers didn't show anything unsual anymore. The main DB CPU was under 15% for the whole time. Next week, we will have to monitor again. Hopefully everything solved.

It was a long weekend ... going home and looking for my bed ... that was the only things left in my brain when I left work ...

2006-09-19T14:21:00.001+07:00

We have solved a lots of requests last week. Today, I have free time again to update this blog with the actions of last weekend. It was totally a busy weekend.

I finished implementing the reducing bandwidth for our member sites. The implementation wasn't to the point that I wanted it to be but it was alright, simple version first and more complicated version will come later. In comparison to the ASP version, our version reduces probably 15% more of the data transfer to the clients' browsers.

Last Saturday was a really chaotic day. We encountered with so many unknown problems. We kept receiving complains from our operations about the site being slows, and we experienced the lagness ourselves too. The network was really smooth, the ping plotter program didn't show any disruption. The CPU usage of the database was at 30%. Everything seemed normal except the sites were slow. The situation continued getting worse, the trading general dispute director also came to our IT department to check what went wrong; our big bosses were all present. Eventually we pinpointed the problem which probably was the upgrade of 2 RAMSAN units. We did upgraded them to double the units capacity on Thursday. Our supervisor tried to change the log of SQL server to different RAMSAN unit. The situation seemed better a little bit; however, there were still complains.

In technical term, our business is database-centric. Like an MMORPG game, the business relies extensively on the database performance with concurrently log in of more than 38000 users at peak time. Our database has problems; as a result, our business also suffers.

A half an hour later, we used the window performance monitoring system to monitor the write and read disk queue of those RAMSAN units. Everything was normal again. Strange, there was no big changes. At the time of the problem, we did a serial of changes to the disk; it didn't help. Then how everything became normal ?. The load to the database remained the same, even a bit more than half an hour ago. STRANGE !!! We had so many factors to consider in order to pinpoint the problem exactly.

Our Agent web server got problem as well. It reached the its capacity limits. Its CPU usuage was always at 97-99% at peak time. "On Sunday, we will try hardware load balancing from the F5 for the Agent system" my supervisor said to me. I went to set up the server for him to prepare for Sunday.

Saturday night became a long night for us. Database performance still remains our biggest issues.

Sunday night, with just a bit less load than Saturday, the system performed smoothly. Was it our external consultant's assumption correct? We didn't know, and could not claim what he found to be correct problems that we had on Saturday, too many factors to concern. He told me in the afternoon that he found the problems; the problems maybe came from those unused indecies which we left in the main table. He removed it out.

When it came to the settlement part, we still have big problems even though we were using our new codes for the settlement; we did test it thoroughly and it was running much faster than the older versions.

We succeeded with the load balancing from our F5 firewall with one of our site. That was the only good news for us for the whole weekend.

... definitely have to monitor again on the coming weekend ...

2006-09-13T23:24:00.000+07:00

"We will down the system for ..." is always the sentence that no one in the online business want to hear from their IT department; however, system upgrade always require downtime. Today, we delivered new functionalities to our system, mainly in the AGENT system.

To implement these auto/manual position taking functionalities, our DBAs had done lots of changes to our database. System down time were unavoidable and management allotted us two hours to deliver the software. 2 hours was supposed to be a lots of time for us. In the end, we just barely made it. All the replication servers had to be resynchronized, we had 7 replication servers. Of course, synchronization for all of these servers took some of our precious time.

My supervisor, my colleage, and I started our schedule earlier than usual to make it to the time allotted for the system downtime. My supervisor took care of resynchronization of all the data and all the replications. My colleague was to update the database code and to prepare new database structure scripts. I was in charged of the application code. Eventhough we had tested thoroughly with test scripts and passed the user acceptance tests, I was still a bit nervous. The funny thing was that I had done these things so many times, and still feel a little sweaty.

11:45 AM Vietnam time, the system was up again. It took us an hour to finish.

BUGS ... ERRORS ... we found those words again for this update. After new realeases were deployed, one of our DBAs - my colleague junior - figured one critical bug. And this bugs will only appear when agents and masters interact together on the same functions. This one was the "must" fix one ... I dealt with the bugs myself, even though those codes were developed by juniors and they were all present. While fixing bugs, it was a chance for me to check back their codes.

Even at the time of deploying, I also let my juniors to implement a small new request, I forgot this request somehow before the deploying. Last minutes implementation always still happens. Something called "PROCESS" isn't functioning well. I do know why and still observing. Precious experience.

I was really impressed with my junior; she is really improved. Definitely she could replace me, I will train her in depth everything I know.

I haven't finished improving odds display yet, so many "must" requests intervened. Anyways, I've gotta conclude it in the next 2 days...

2006-09-12T07:36:00.000+07:00

Another week passed by so fast. I had not been so productive last week. Things must be changed this week. Everyday, we received more and more requests. I just cleaned my desk at work; but definitely, it will quickly be filled again with papers. There are so many things on our hands: implementing new requests, improving and optimizing the current code, and training our juniors. Well, I guess nothing is for free, then just live with it.

Starting day of this week wasn't so bad. I decided to stay a bit late to finish all the pending requests that the management marked as URGENT on the ASP.NET member

Delay of 2 seconds between mix parlay bet was implemented. This one DBAs finished it. We don't show to the user that they are restricted to bet a new parlay within 2 seconds, we just tell them that they can't bet because of "Odds changed"

Provide Help section for the mix parlay

Implemented the count number of current games on the menu.

Preparing all the codes for the splitting of the servers tomorrow

Today, Boss comes back ... definitely more things to come ... *sigh* . We will have actions soon.

2006-09-09T08:27:00.000+07:00

Backlash

The past few days, I have been struggling with new odds display and reducing the bandwidth problems with my own code. Couple months ago, I implemented our Member website with only one purpose in mind "make it better than the ASP version, as much as possible". I did succeed, our ASP.NET version is better, faster, going back to the database much less on betting procedure. However, I want to improve it and face so many obstacles; they drive me to the point to reimplement everything. The inexperience in designing and coding now show its true shadow.

In order to improve the bandwidth, I must detect all possible odds changes.

In the ASP versions, store procedure returns all the bet type by rows, each odds will be on one row, detecting odds changes will be on one row only which is the odds change time

In ASP.NET version, for my convenience of coding, I requested to put all the odds type of a match in one rows in the record sets. Bettype 1,3,7,8 will be on 1 rows with 4 different change times. To detect odds changes, I will have to check all 4 values. It gains in performance because the cost of moving down a row in the dataset more than we check for 4 values. But the source code will be definitely more complicated.

After detecting all odds changes, I must send out all these changes into a compact form; then populate the those datas into correct odds position which will be identified by the odds id. Everything sounds to be easy. ..BUT.. things are not easy as it should be; I will have to deal with my improvement of bet engine which I used to be proud of.

In ASP version, each odds is just a link. Once the user click on the link, they will go to the database to get all the information about the choice that they want to bet; one more trip to the database to create the bet slip.

My improvement in ASP.NET version was to store all the possible information when I built the display into something called Javascript "betstring". This "betstring" contained all the necessary information about the choice of the users. There is no need to go back to the database to collect all the information any more.

Because of building the bet slip by "betstring", I need all the information about that user choice including the team names, league names and score. Those information usually contains most of the data sent to browser. If I continue to include these information into refresh data, then I won't save much the bandwidth. Previous improvement now has impacts

Everything has a way to solve except my code will look UGLY.

2006-09-04T17:51:00.000+07:00

Monday, another working week starts again. We were supposed to have long weekends this week because our national holiday was on Saturday. I didn't think that I'd have that luxury, and as usual I was right.

Problems arised. I got messages from our Customer Support Department at 1:30 AM in the morning. I was unlucky that being online at that time to check my email, and got caught. Anyways, problems had been detected; they had to be fixed.

777goal correct score, total goal didn't display --> I pointed the cache server into different database, and the database store procedure on there were not up to date.

s33bet members could not bet parlay bets --> Data conversion problem from the database. We just did some modifications on performance improvement for the database, some columns had been converted into different datatype. Bugs spotted.

s33bet minimum bet displayed incorrectly --> new minimum bet value had been introduced to our ASP version, we hadn't updated it into your .NET version

SOLUTION: come to work on holidays. quick, simple but not dirty as it sounds ...

In the evening, my mailbox received 2 more new requests. Just display requests for s33bet; geared on, I marked DONE status for those in 10 minutes. Anyways, I must update our requirement collector about these requests. He has the responsibility to document everything we did. That's called PROCESS. I'm being thinking on making a blog on PROCESS in our company soon. Hopefully it won't be called CHAOS ... j.k ...

Continue bring new method of reducing the bandwidth into our .NET member now ...

2006-09-01T12:00:00.000+07:00

It is time to implement new odds display functionality into our .NET member site. I created a different set of javascript and new set of code-behind files. I will start from scratch. Let's see what we have to do.

Users navigate the odds display page for the first time, need to build the tables.

When users refresh, the code-behind must send just enough informations to the browser, and the javascript will spot which odds are changed in order to update the display.

Deal with situations such as new live games, odds closed, live game ends, new games offered e.t.c

I broke everything down into small problems, and start sovling one by one.

1. Remove all the duplicate data in our old odds display engine first.
Team information of multiple handicap odds and the show time information will be removed first. Since we introduced double handicap for each match , this will reduce our bandwidth by quite a bit.

2. Implement code to detect all changes from the last refresh
This one will be a challenge. So many possible cases have to be considered. The hardest cases would be a match or an odds being removed, or added. I will detect just odds value changes first.

2006-08-30T22:01:00.000+07:00

Last weekend, our ASP Member were already updated new versions of odds display. The new version was implemented similar to what I did with the Forecast of Agent system. The javascript behind just updated new information from the database. The purpose of doing this is to reduce the bandwidth usage. Bandwidth issues are becoming a headache issue for us as our business keeps increasing. Our bandwidth usage is reaching our capacity day by day.

The display odds of ASP version is changed, our .NET member site display odds will have to be changed as well. New odds display engine must be reimplemented. This will be another challenge since I won't have any assistance from the database. The process of analyzing the odds data will be in the web servers. Everything must be planned out and designed thoroughly. I'm taking care of this functionality. Hopefully, tomorrow new ideas will come to me.

Today our big boss came up to us and asked us to find a way to detect bot accounts which are programs. Those programs log into our website automatically and grab our prices from our sites, then update into their sites sothat they don't need to control the odds. It is extremely hard to check for these programs. We have user login logs. We could start from there to check the login patterns of all the accounts and spot those with irregular patterns. Bots usually log in and log out all the time. We could start from there. However, some smarter programmers will randomize their login patterns or mimic the browser behaviours; to trace down those accounts will be really difficult. We will give more thought on this.

2006-08-27T06:26:00.001+07:00

Tonight we had encountered the database problem again. Main database CPU went up high again. This time, my supervisor intervened timely to kill off the physical network connection and prevented the system jammed. Everything happened transparently to the users before they even noticed. Closely monitoring the system performance at peak hours shows its true importance.

We narrowed down the problem that might cause the system to be jammed up. Our supervisor checked the user actions logs from database. At the time that windows performance monitor showed the signal of the jamming to be about to occur, the operators were doing settlement ( a process of calculating the win/lose of each tickets after a game is done ). We will monitor this problem again on Sunday night.

Around 3AM in the morning, another problem arised. This time, it was the network. Agent website, and a .NET member site could not be accessed. Ping plotter showed red color at the our ends. SingTel line was having problem. It wasn't the problem from the ISP, it was our internal server issues. There is a problem at our firewall.

In the ping plot we could see the route which a information packet is going around on the internet. It is like the tracert command of window XP. We could see all hops visually. By tracking the ping ploter, we could know where the problem of the network to occur; international gateway or in our DNS.

2006-08-25T23:48:00.000+07:00

This weekend, La Liga will start the 2006 - 2007 season joining England Premier League and Bundesliga. There major leagues have started; Series A will be the last one. A brand new soccer seasons is waiting for us. It is almost a year since we started with IBC - a sport betting system. We have owned more than 40% of the system which was developed by a Malaysian couple years back.

The system has grown quickly since last year from one database server, and around 20 web servers to 6 database servers and more than 50 web servers. The number of concurrent users has rised from approximately 10,000 to 37,000 users. Numbers of accounts has rised from 100,000 to 300,000 accounts. Within one year, the company business has been increased more than double.

We call our business as a type of gaming. This gaming industry is getting bigger and bigger everyday with online casino, online community games, lottery and sport betting. We mainly operate gaming with sport events. You guys could call this kind of business Gambling; doesn't matter to us, we still consider it as gaming with money.

Those are just a little brief introduction about our company and a little bit background of our software platform.

...

The 60% of our system is still in ASP. We are going to convert into .NET as well. Our architecture is already ready. My colleague and I have been designed it to contain lots of spaces for improvement later on. We must make everything planned out first as our system is constantly changing with the grow of the business. In our design, N-tier model is still in use, but we planned a way to quickly change from N-tier to services-oriented model. I will post a design eventually; and the design won't be in details, it will just be a rough idea design.

In the past 2 day, new functionalities have been implemented to our system

Mini betlist for member sites

Improvements to the Odds Controller of the External Admin

Next week, we will have a session on implementing Extreme Programming into our team. We start to follow Agile Development Methodology. Our work finally get paid off; we have convinced the management to apply Agile Development. My colleage and I will have to complete the process template as a proposal plan to them.

Another task that must put in to-do list is to back up our source code database. I want to back up the whole database frequently, but lots of suggestion were to just keep the release source not to keep the whole database. To me, keeping the whole development database will give us the benefits of getting all the metric of team development. We could figure out the team behaviours, real ownership of source code. These things are really important for team development improvement purposes.

Second weekends of the season, my colleage had applied new indecies into our database. How good it is? We will have the answer Saturday and Sunday night.

... No more weekends for me ... :'(

2006-08-24T05:50:00.000+07:00

Another day passed by. We are closing on our schedule. All the software have been prepared and waiting for approval to be in production; however, there is no signal from our trading department yet.

Testing more and more will give us benefit on the coming launching day and will reduce the chances of bugs which are always nemesis to programmers and businesses.

We did lots of testing today, and we found more bugs on our position taking functionalities. Instead of giving the tasks to my programmers, I went to fix it.

Our External Admin has been run for the second day now. No major problem were reported. Except the admin 2 people kept complaining that the program was slow. 1AM this morning, I checked with them on live games. The program was slow and half an hour later everything was normal. The problem must come from the network; maybe we have experienced some lagness with the internet connection into KBT.

Tomorrow, first thing to check is the ping plot of KBT network from 10 PM to 3AM to see whether we have some disruption with network. Hopefully the problem will appear on at the international gateway, or KBT gateway; or else we must ping each single Admin 2 people to see where the lagness occurs. We could recommend them to change the ISP or we have to find a better route for them to go into our server in KBT. The problem belongs to the network engineer. Maybe I jump a bit far now.

My to-do list gets longer and longer everyday. *sigh*

2006-08-23T04:20:00.000+07:00

Yesterday I got an email from my supervisor about patching our Member Website. I totally didn't understand what he meant. After arriving at work, I asked him on what was going on; then series of mixparlay bet were shown to me and my colleague.

"Our System has holes"

That was the conclusion that we arrived. Some smart a**es figured out the way, we sent information to our server to process mixparlay bets and exploited that.

A mix parlay bet is a combination of choices over many matches. In each match, a punter is only allowed to choose one product (one of handicap, over/under, 1x2, and total goal); at the moment we only offer that many products for mixparlay (Parlay betting explanation). For our company, a parlay must consist of 3 soccer matches or over.

In our ASP version, mix parlay information will be passed in the query string of an url; then send back to our servers for processing the bets (something thing like http://linkA?a=betinfoA_betinfoB_betinfoC).

In our ASP.NET version, mix parlay information is posted through a form then send back to the servers.

For ASP version, it is pretty easy just to change the query string link. For ASP.NET version, the punter could interfere with the posting action by creating a dummy post form. If exploiting this, the punter will pass through all our checking on the interface and creating sure win parlay combinations. For example, they could choose 3 same choices for a parlay, and the next one choose 3 choices of the opposite; dispite the results of the matches, the punter will win at least 1 combinations; and that win is enough to cover all the expenses and with profit.

... when it comes to money ... people are getting so smart at digging and finding holes ...

We immediately patched our systems for this vulnerability both in ASP and ASP.NET version. We implemented 2 more levels of checking the duplicate match in processing bet level and in database level, and of course, update immediately into production servers.

Testing servers were set up for our CSD to test our new position taking functionality on our agent system as well. 2 web servers were deployed with newest code from the development, and database server with newest data from production servers. I wonder how CSD were doing at the moment. Once the test is done, 1h downtime of the system will follow in order for us to run an update on the database and on the interface code.

External Admin was also updated; new functionalities launched. I'm checking how it runs at the moment. Better than the old version, there is no complain yet so far. From feedback of our operations, external people liked the new version. I cross my finger and wait for things to land on our to-do lists; when the customers like it, they will definitely throw at us a series of IMPROVEMENT requests ... "you can do that, now please do this for us" blah blah blah ... and I got 3 new requests already on my table for tomorrow.

On our schedule, I have to continue launching 2 more member websites in the set of 10 sites that we are operating now. This will be to be determined tomorrow.

2006-08-21T10:33:00.000+07:00

The first weekend of a new soccer season had passed ... All servers ran smoothly yesterday ... There were 2 big games yesterday Chelsea -vs- Manchester City and Manchester United -vs- Fulham.

Agent server reached 320 requests/sec yesterday with at peak 2,010 members. On top of these members, External Admin and Internal Agent were also run on this server. A dual Xeon 3.2GHz has been served us well.

With 8 applications is running on a single server using default setting of IIS6 will have an effect on the performance and response time. In Kaohsiung, we have 2 dedicated servers for Agent System ... now back in Taipei, 1 single server handles all and on top of it External Admin, Internal Agent are now also deployed on that agent server. I guess my supervisor created challenge for me to improve the performance instead just issuing new servers for me. And his decision turned out to be good.

After we moved back from Kaohsiung, I quickly noticed the executing time of a request increased dramatically from a few hundress milliseconds to 3-4 seconds on average. I discovered that running 6 or more web applications on a server with one IIS worker process only caused the executing time of a request to be longer. I must increase the number worker process for IIS but in which way?

Our applications must write the their own logs into harddrive in order for us to trace errors and trace customer activities. Increasing number of worker process will increase the chances of I/O queues to occur since 2 or more threads will fight each other to grab the privileges to write the logs. When this happens, requests into the system will be failed. This problem will definitely occur, unavoidable when using 2 or more worker processes if we still want to keep the application logging. Logging has been trememdously helpful for us. I don't want to remove it.

Compensating solution ... Yes, have one. I found out just using 2 worker processes; each worker process will handle specific applications. Why 2, not 4, or 8 ? tried ... After a series of trial and errors, with our 2 CPUs server, 2 worker process works the best; increasing to 4 worker processes, number of request failed increased much more than the gain of executing time of a request.

This method works well for us in the past weekend. No major problem or any issues. Executing time of a request is being kept below 800 milliseconds on average. Number of requests failed is below 4000 requests/per busy night like this weekend. The statistical performance numbers are really convincing to me that this method works well.

I could claim that our 2 dual Xeon 3.2Ghz server could support 2,500 users - 3000 users, but still need to try in practice.

I improved External Admin Odds Controller. Lots of new functionalities were put in.
- Hide odds, Open Odds were implemented.
- Implementation of odds selections for the users to choose

I think recently I overuse Web 2.0 technology. Really need documentation on security issues of this technology

Time to go to back sleep now ...

2006-08-20T05:58:00.000+07:00

6AM, checking the system log for the last time before leaving the office ...

Tonight we resumed our night shift to check on the system and to be stand by. The number of users was around 34,000 at peak time (10PM - 1 AM, Malaysia Time). System was running stable.

My colleague discovered bugs in Member Program which I wrote mainly. The number of online users was counted. Problems discovered was that I made a typo in my login page. Correct ServerID wasn't passed into my on-user count functions. I had been passing the dummy variable which was from the testing phases.

Our messaging system contained bugs as well; but no one noticed and we did passed users acceptance tests proudly. My collegue took care of it, mainly patches to the database.

Problems fixed immediately, but I couldn't update to production servers ... There were many live games. Decision was made swiftly and update'd be postponed until the next morning when there were the least live games.

The match between Arsenal -vs- Aston Villa was on the way.

DataBase CPU rised to 90%, our agent server CPU hit 100% and stayed there for a while . Operations experienced slowness and they couldn't change prices for the match, out there more than 34,000 users were effected ...

Unfortunately, we could not do much. From Profiler, live forecasting of the match Arsernal and Aston Villa ran excessively from main DB which is for just writing; forecasting of a game is to predicting the company win/loss of that game. Forecasting store procedures are always killers to the database, each SPs run at least 1 second on a 4 dual core 3.6Mhz Intel servers 16 Gigs of RAM with RAMSAN. We need to work on this if we want to reach 50,000 concurrent members by the end of this year.

10 more minutes, DB CPU dropped. My supervisor must have done something. We guessed he killed off the connection from web server to CPU; however, number of online users never dropped. It was not likely that he had done that.

For the next 2 hours, everything seemed to normal again.

While monitoring our servers, I reviewed and did couple testing for double handicap functionality for our External Admin again; made the javascript behind it to run smarter; the auto refresh would stop automatically after 1 minute if the script detects that the operations already closed all the available odds.

The launch of .NET member site for our IBC system considered successfully, not many major problems have occured for today - England Premier League started ...

The database performance still remains a big concern. We will have to monitor more closely in the next coming days.

2006-08-20T00:23:00.000+07:00

Today, I have started reading One Jump Ahead again - a great book about a life of a real programmer who was my respectful professor at U ... The idea of start blogging my daily technical activities popped out in my head, it will definitely be good for me to start to review this after a while ...

...