2 replies to this topic

[Cris D.]

I've heard several times over the years that turning on etomite's logging can cause bloat and speed issues as it grows. However, in this ip board forum, I can see the number of views and numbers of downloads that is getting into the 6 figures and the site seems to be going fine. I have not looked too hard into the etomite logging part of the code base, however, I have a requirement to show page hits on the page and using etomite's logging seems the simplest solution, but I don't want to use it if it is not sustainable.

I could easily write code and a table to just store page hits, but seeing that logging features are already running, it would seem silly to start from scratch. So...

1) Has anyone else modified the visitor logging code to track just page hits and not all of the other guff? I'm thinking that modification in this area may cause other issues with how the reporting tables (and other manager pages) display if I'm not careful.

2) Why is the etomite logging so bad that it causes performance issues when bloated? I was always under the impression that sql should be able to handle 1000000's of rows, so does anyone know where the performance bottleneck is? and can it be "fixed"?

[Dean]

Cris D., on 23 December 2009 - 09:27 PM, said:

However, in this ip board forum, I can see the number of views and numbers of downloads that is getting into the 6 figures and the site seems to be going fine.

I do a lot of pruning, regularly. I also regularly optimize the database. So far the database is 88MB in size (with 183,141 records). We don't keep forum logs for long. Also, the IPB code seems pretty decent when it comes to resources.
I rent the server, so don't have limitations on what I can do.

Some server details (for the forums only):
Site Storage Size: 518.04MB
Site Bandwidth Usage: 11281.35MB (this month, so far)
 Screen shot 2009-12-23 at 22.42.11.png   103.65K   2 downloads

Anyway, my input on the Etomite logging is the following:

I think most of the logging problems is that it doesn't work right in all cases, not picking up browsers etc, or in some cases just not working and causing php errors.

Most of the problems are because the Etomite database goes from a a couple of megs up to a couple of hundred megs, in a pretty short time. Most hosts don't allow databases to be so large and most hosting packages run out of resources pretty quickly.

Personally, I don't see the point in the Etomite logging - Google Analytics (which is free) can't really be beaten, and takes up no space. Also, most hosts have built in logging of stats (see image above).

[mikef]

Its a long time since I looked at the logging but I think the problems were:

• used innodb tables with no good reason, which damaged performance
  
• logged absolutely everything, and didn't (couldn't) distinguish bots from users. (I wrote some crude patches to sort of address this, but don't know if they are still in the codebase - I abandoned the etomite logging shortly afterwards)
  
• etomite never optimised the database, so with continuous changes the efficiency of the DBMS declined, and particularly so on tables that had to be read and updated on every page access.

I use Google Analytics mostly, plus statcounter on some sites, since its easy to give others access to statcounter's limited data. (actually, its probably as easy now to send Analytics summaries, but it wasn't in the early days)