Monday, July 27, 2009

Why Netbeans rules Glassfish... racing got me to this conclusion

Hi,

Last weekend I ran into this race-condition on the server side of our game IFF (plug: http://www.sticktailgames.com:8080/IFFMobile/). I was stress testing the server and what do you know, all crumbled into a pile of rubbish as soon as, dare I say, more than one person at a time registered for the game. Now I do have made some short-cuts in the server code to circumvent some code because I want to test a lot, and don't want to have to wait if I don't have to. So entering an activation code on your mobile was taken out of the equation. Anyway, the problem is there and since I do expect some time in the future that two or more people register at the same time, I wanted this to be solved.
As it turned out, there was a race-condition; two processes were fighting for the same record in the database and that turned into an inconsistent state of the data. And since inconsistency in the database is worse than a crashing system that at least is consistent so I can start it up again... It turned out that this little piece of concurrency could be turned into a little piece of sequency and all would work out really nice. So how do you control this? I thought about using a queue to list the requests for a unique record in the database. Since the requests would be handled sequentially I would be out of the woods. A message queue is easily created and NB created all code, including the creation of the server resources for my MDB, queue etc.

Proper code generation is sooooo convenient.

But I needed to revert a lot of code this weekend to a state prior to the weekend, and that included my messaging stuff. Turns out, the resources NB generated in Glassfish v2.1 didn't want to move. Every attempt to delete them turned into an error message, stating that the resource could not be deleted as it was used in server server.

From the GF output in NB: Validation error: [ADMVAL1052: Element connector resource 'jms/iffBaseGenerationTopicFactory' can not be deleted because it is referenced from server 'server']

Not from NB nor from Glassfish' own admin console I couldn't delete the created ConnectionFactory. I had to open domains/domain1/config/domain.xml and delete the lines referring to my ConnectionFactory.

Once I found out how to remove the JMS resources I could get back on track. Now I'm back at solving my race-condition. Still thinkering on how to handle this. Probably I need to ensure that there'll be only one connection possible to the queue. So only one MDB can access the queue and handle the message. Any other approaches are appreciated in the comments of this post.

Iwan

6 comments:

Anonymous said...

Hoi Iwan,

Just read your blog and am wondering why you're running against this problem. Aren't you using an advanced relational database? I ask this because any decent database locks the table/row when a query requests a record. I would guess that would eliminate your "concurrency record fetch"-problem, would it not?

Vriendelijke groet

Philip said...

Even not-so-advanced databases provide concurrency handling. Check out Java DB (aka Apache Derby). It's free. Why reinvent the wheel?

Iwan said...

Yup, I'm using JavaDB, and in fact the problem is also faced in MySQL. It's a problem with optimistic locking, which I use since in 90% of all cases of retrieving a record is not to update it.

But I'll look into this from this angle. Guess I was too busy venturing in one direction and missing the obvious in the process.

Thanks,

Iwan

Iwan said...

Yup, I'm using JavaDB, and in fact the problem is also faced in MySQL. It's a problem with optimistic locking, which I use since in 90% of all cases of retrieving a record is not to update it.

But I'll look into this from this angle. Guess I was too busy venturing in one direction and missing the obvious in the process.

Thanks,

Iwan

Grzegorz said...

I read for a first time about concurrency in simple words on a blog.
Maybe Your problem is problem of locking resource?
If You do all methods of classes which is using jdbc synchronized?
It does not look like complicated producer-consument problem.
Wish You many players;)

Andries Spies said...

Thank you for the posting. Really helped me to solve this one problem of how to delete a queue from GlassFish after a `ADMVAL1052` message.