Parser modifications and tests

Over the weekend and today I worked on parser rules. I started it from the scratch having the modified grammar and existing grammar implementations as references. I was able to get rid from most of shift/reduce and reduce/reduce conflicts. But I still get a single shift/reduce conflict. I don't know how to find it easily.
Commit the changes of the parser[1]. Then I started with tests, first I go through with the documentations[2][3][4].

[1] svn//svn.zope.org/repos/main/Sandbox/adamg/ocql/branches/qo-compiler - implementation
[2] http://pypi.python.org/pypi/zope.testing/3.5.1
[3] http://wiki.zope.org/zope3/Zope3Book
[4] http://wiki.zope.org/zope3/unittests.html

Week progress report from 21st July to 25th July

I was able to complete the rewriter component implementation during this week[1] and was able to do this with out many difficulties because I discuss the work well before Adam's holiday.

Next, I have two tasks is to complete - adding tests and parser modifications. I hope to interleave both tasks in next weeks[2].

[1] svn//svn.zope.org/repos/main/Sandbox/adamg/ocql/branches/qo-compiler - implementation

[2] http://spreadsheets.google.com/ccc?key=pqiTfcdeaKaYf-C6HpM32VQ&hl=en - Time plan

Query Object Adaptaion

Today I started with query object adaptation. I use the previous branch for query object compiler [1]. Add few changes identified previously, but not in this branch. Refer the algebra adaptation previously I did.
[1] svn://svn.zope.org/repos/main/Sandbox/adamg/ocql/branches/qo-compiler

Week progress report from 14th July to 18th July

Main task of this week is to implement the algebra optimization. To do that it needs to re-organize the algebra tree, such that new tree leads for a optimize search. For that, need to find a specific pattern in the tree and replace it by simple algebra. So I started to find the target algebra object pattern in the tree and later moved to more generic algorithm to find a general pattern in the tree.

By the time I had to spend time to set up the code coverage, svn version problems and google mid term survey. I was previously using eclipse 3.2 and it worked well with subclipse and svn. But using eclipse 3.4 leads to touch the svn files as version 1.5 but the actual client is version 1.4.6. I tried the instructions given in [1] to solve the problem. Since it didn't work for me I moved to the previous eclipse which is simplest to continue on work.

I discuss the things to be done for the next two weeks [2] and break the tasks in to smaller sub tasks. For the next week I will continue with the rewriter module.

[1] http://durin42.blogspot.com/2008/03/setuptools-and-subversion-15.html

[2] http://spreadsheets.google.com/ccc?key=pqiTfcdeaKaYf-C6HpM32VQ&hl=en - Time plan

[3] svn//svn.zope.org/repos/main/Sandbox/adamg/ocql/branches/optimize-with-index - implementation

Algebra optimization

After a enjoyable weekend I continued with the project. I complete google mid term survey and continued with the algebra optimization. I face with a problem where I was unable to run commands from the terminal to run the tests. It was because I used latest eclipse version and update the subclipse. I am using 1.4.6 of svn but new eclipse seems to be touched with 1.5 svn. We can look in to the first character in the .svn/entries to check the current version of svn using. I tried to use a patch given to overcome this. But since it doesn't work for me I downgrade the eclipse version to 3.2, since it is too simple. The patch is given in here[1].

Then I spent some amount to add a recursion tree to the algebra optimizer script. Since I have used children variable in class level, my recursion part loop forever. It was because mutables like list, we can only work with the only instance. So the recursion loop execute for long iterations until it reach maximum. I got mentor's help to identify the problem. So it can be solved by adding setting the variable in the object initiation[2].

[1] http://durin42.blogspot.com/2008/03/setuptools-and-subversion-15.html
[2] svn//svn.zope.org/repos/main/Sandbox/adamg/ocql/branches/optimize-with-index/src/ocql/rewriter/algebra.py

Week progress report from 04th July to 11th July

Early in this week I was able to complete the two tasks assigned to me for the algebra compiler[1]. It was about do a proper interface declarations for algebra objects and to implement location in the algebra object tree, by adding parent and name properties to the algebra objects and adding a 'head' element to the tree. All the implementations related to algebra compiler is in a separate branch[2].


Then I moved to the queyobject compiler, which has pretty much the same kind of work to be done as I did in the algebra compiler. I did the interface declarations and implement location in the query object tree. But leave the adaptation part to do later. Implementations for this part goes here [3].

After doing that I moved for a algebra optimization part. Identified optimization is related to transform the algebra tree if it contains a particular pattern. That means look for a pattern in the tree and modify the tree to get the results. For a example say we have a condition,
class.property operation value,
so if there is an index available for the class.property we can use that for the fast searching. Currently, engine gets all objects of an interface and then it uses some loop to filter what it wants. So I include methods to find out whether it can decide the class.property has an index and retrieve data for particular class.property. Repository branch for this work linked here.[4]


[1] http://spreadsheets.google.com/ccc?key=pqiTfcdeaKabBUy48qR0JTw&hl=en_GB - OCQL - TODO list
[2] svn://svn.zope.org/repos/main/Sandbox/adamg/ocql/branches/alg-compiler - algebra compiler
[3] svn://svn.zope.org/repos/main/Sandbox/adamg/ocql/branches/qo-compiler - query object compiler
[4] svn://svn.zope.org/repos/main/Sandbox/adamg/ocql/branches/optimize-with-index

Query compiler and identified optimization

I completed creating the query object tree and interface declarations. Implementation was reviewed by my mentor and made few corrections in it. Then, I moved to a identified optimization part of the algebra. For a example say we have a condition,
class.property operation value,
so if there is an index available for the class.property we can use that for the fast searching. Identified the method need to be added to the class and introduced a index for attributes(property). They still gave me some errors, need to correct them tomorrow, hopefully. :)

Algebra Compiler and Queryobject Compiler

I was able to complete the two tasks of algebra compiler. To implement the location in the algebra tree, I create a separate class to hold the algebra object tree, which is the 'head' considered to be the top of the tree. Again use subclipse to see the modification done by the mentor by comparing the consecutive revisions.
Then we moved to the queyobject compiler, which has pretty much the same kind of work to be done related to the algebra compiler. Check out the code from the related branch for the proceedings.

Project design

I went back through the implementations and other project related things I did in the past few months. I am happy with the things I have done up to now, since this is my first open source and Python project. Its interesting to work on a open source project where I got many experience with different areas, including coding styles, discussions by mailing lists, chats, etc. According to my view point since there are plenty of people looking at what we are doing, is a plus point because we can know our mistakes up front. And having people known the system can always got the chance to ask help when we are stuck in some point.

I looked in to the whole project design and implementation. I was able to identify the behaviour of the implementation using debugging. Did some modifications and looked at the deviations. Since I am still learning things related to Python language, referred to some Python books and clear ambiguities.

Week progress report from 27th June to 04th July

Previous week I did DB metadata and data provider, to provide class/object declaration to the engine. In early this week my implementations were reviewed and made the required adjustments. One important thing I learn was that to test all parts of the application after modification of one part. From my new implementation there were two failures in other parts where I didn't notice.

Then I moved to the algebra compiler as it seems more important over the other parts of the DB data and metadata provider. I need to complete two tasks (number 4 and 6) in OCQL TODO list [1].

I started with the sub-task 6, where I introduce interfaces to algebra operation classes and adapt them. All the implementation related to algebra compiler goes here[2]. During the implementation I needed to look the changes done by the mentor. I installed subclipse[3], to look in to revision histories and compare them. Then regarding the sub-task 4, I referred and used Zope.location to introduce parent, child relationship in the algebra object tree. I used remote debug option quite a lot to identify the execution and behaviour.

Finally, its again a interesting week with learning every bits and pieces and hope to continue on the same section in the next week too.

[1] http://spreadsheets.google.com/ccc?key=pqiTfcdeaKabBUy48qR0JTw&hl=en_GB - OCQL - TODO list

[2] svn://svn.zope.org/repos/main/Sandbox/adamg/ocql/branches/alg-compiler - SVN repository

[3] http://subclipse.tigris.org/

Algebra Compiler

Review the DB provider code with mentor and did adjustments in the code. Then we decided to move on to the algebra compiler, mainly because we need to identify the difficulties and we still don't know what we exactly wanted.
In algebra compiler I need to do interface declarations to the algebra operation classes. So I just create interfaces and implement them in the appropriate classes.
I declare attribute and method definition in interfaces and use identified types using zope.schema module, such as Text(), Int(), etc.
This interface declaration is targeted on using the compile method using an adapter. Latest code managed in here[1] and dbprovider code is merged in the trunk.
[1] svn://svn.zope.org/repos/main/Sandbox/adamg/ocql/branches/alg-compiler