bug found while using caching and its log

[kevinwen]

I found some bugs when turning on cache_queries and cache_queries_log:

When cache_queries=1, at line 66 in xataface/Dataface/DB.php:
$this->_fcache_base_path = DATAFACE_PATH.'/templates_c/query_results';
It should be:
$this->_fcache_base_path = DATAFACE_PATH.'/Dataface/templates_c/query_results';
since we never had a directory templates_c under xataface. For line 64, should we have Dataface directory under our application so it looks the same as xataface? Now it is:

$this->_fcache_base_path = DATAFACE_SITE_PATH.'/templates_c/query_results';
Should we change to:
$this->_fcache_base_path = DATAFACE_SITE_PATH.'/Dataface/templates_c/query_results';
??

When cache_queries_log=1, at line 316, 324, 337, 655 in DB.php and line 404 in OutputCache.php, the path to querylog.log and dump.sql have been hard-coded, which make it not work on Windows. Should we use a path under the DATAFACE_PATH so it's compatible for all platforms?

[shannah]

Code: Select all

$this->_fcache_base_path = DATAFACE_SITE_PATH.'/Dataface/templates_c/query_results';


No. Your first change was correct as in Xataface the templates_c directory is inside the Dataface directory. But in the context of the site path, the templates_c directory is meant to be under the main app directory (not a subdirectory). Hence this line should stay unchanged.

When cache_queries_log=1, at line 316, 324, 337, 655 in DB.php and line 404 in OutputCache.php, the path to querylog.log and dump.sql have been hard-coded, which make it not work on Windows. Should we use a path under the DATAFACE_PATH so it's compatible for all platforms?

Good thought. I think it is probably best to use the sys_get_temp_dir() function, but this isn't available in PHP less than 5.2.1, so it would be necessary to implement it for older installs.

[kevinwen]

Good thought. I think it is probably best to use the sys_get_temp_dir() function, but this isn't available in PHP less than 5.2.1, so it would be necessary to implement it for older installs.

Well, the log file is specific to the xataface and all of the log files may be grouped together for better organized. Not everyone likes to put log files into a temp directory. So saving it under DATAFACE_PATH is better. On linux (even in Mac), the log file may be a symbolic link to the actually log file under a centralized log directory.

[shannah]

good point.

Perhaps a subdirectory of templates_c e.g. templates_c/logs would be the best solution.
I haven't really made it a point to tell people to cut off access to the templates_c directory in their Xataface installation. If they are storing log files in there, it would become especially important to do this though. (e.g. an .htaccess rule)

[kevinwen]

Could we create a directory just under DATAFACE_PATH like DATAFACE_PATH . '/logs'? templates_c is a place for the cached content and can be deleted without preventing the app from running. If we put logs under templates_c, it may cause problem when this directory is emptied.

[shannah]

I suppose this is the better solution. The only drawback is the increased amount of setup required, as the logs directory would need to be writable by the webserver. Currently templates_c is the only directory that needs to be writable. We would just need to make sure that Xataface just ignores it if the directory isn't there or isn't writable.

[kevinwen]

Can we add a variable at the beginning of conf.ini, like cache_log="/usr/logs/xataface", so the developer is able to decide where to store the cache log?

[shannah]

Along the same lines, a more general purpose setting for a logs directory to contain all types of logs that could occur.
Default location:
DATAFACE_SITE_PATH/logs

If the "logs" directive is set in the conf.ini file, the location could be overridden with another location.

[kevinwen]

I made some changes in DB.php and make the caching work. However, when I added a new record, the cached results come out, instead of the newer version of results as mentioned in this article:

Table Modification Times and Dependencies
In order to determine whether a result cache is current, Xataface checks the table modification times of the tables that are used in a query and compares them to the cached result modification time. If the cache is newer, then it uses the cache.

http://xataface.blogspot.com/2009/06/using-query-caching-in-xataface.html

Can somebody tell me what would go wrong? Thanks.

[shannah]

INNODB tables and Views don't store their table modification times, so caching has not worked well in those situations. in SVN (and in the latest 1.2.3b2 release) I have added support for storing table modification times by adding a table to the database that stores these - but this solution isn't perfect as the table only gets updated if records are inserted via xataface.

[kevinwen]

I see. I have an idea but not sure if it would work:

Create a db table(or in memcache) store the freshness of the cache, which will have 4 fields: Primary key, cache_id, involved_tables.

The SQL Parser should do the following for Insert/delte/update: search for this table and grep all cache_ids that have the corresponding tables updated, then delete those caches before caching the new results. This way, all the caches should be the latest. Would that work?

[shannah]

This is one possible solution... and it should be considered. If you are planning on hacking the code/making changes to Xataface, I suggest that you work with the latest in SVN. Are you familiar with subversion?

The solution that I have implemented (and should be available in SVN) is just to store the update time in a special table, then it compares the update time with the cache at the time that it serves the cache. It should achieve the same results - it just does the checking at different times. The table name is dataface__mtimes. You may want to check and see if this table has been created. This will indicate whether your version includes this fix. If the table does exist, and the cache still isn't updating, then it is possible there is a bug and we'll need to look at it more closely.

[kevinwen]

Here's what I got from MySQL site: http://dev.mysql.com/doc/refman/5.1/en/show-table-status.html
Update_time

When the data file was last updated. For some storage engines, this value is NULL. For example, InnoDB stores multiple tables in its tablespace and the data file timestamp does not apply. For MyISAM, the data file timestamp is used; however, on Windows the timestamp is not updated by updates so the value is inaccurate.


My app is running on Windows. It may be the cause. Is that possible to consider the solution I came up with? There is no time checking at all, but just flush all caches that are related to the tables being involved in the current insert/update/delete statement.

[shannah]

Can you confirm whether your database has a dataface__mtimes table?

[kevinwen]

Yes, dataface__mtimes tables exists, and it look like working on Windows, but I did some modification to specify the DATAFACE_QUERY_LOG in config.inc.php and use DATAFACE_QUERY_LOG in DB.php, so I can turn on the cache_query_log. Did you make that change as well? I need to grep the latest one. Thanks.