Dataface Configurator

[Aoirthoir]

The following is a comment I emailed to Mr. Steve regarding Dataface. It is a bit long winded however I hope it is useful here for us to all start some conversations regarding the possibilities of dataface. More comments will follow:

Quote (myself):

Mr. Steve,

First, I wish to commend you on your Dataface product. It is very similar to what I have desired in a database development system for years. I have used other programming languages and systems like Cobol, Clarion, MagicPC(ugh..). All back in the DOS days. Most modern programming systems seem to focus on GUI. Certainly OOP programming has made some things easier than in the past. Yet, most programs seem to ignore the fundamental needs of a programming system, data. VB and other like tools concentrate on how to place a button, what the button does, what the properties of the button are. To interact with data the programmer is left on his own, or forced to purchase addons. Worse, most systems that generate code for the programmer as a starting point, have few (if any) code embed points. Thus the programmer must change the code directly. Yet if he returns to the Rad/Case tool or framework, to make a change, and regenerates the code, he must keep track of all the individual changes he made and reinsert them into his code.

Clarion is a system which has thousands of embed points so code regeneration offers no difficulty for a programmer. Also, the entire ARAD of clarion is template based. Which means that if the programmer wishes to change the generated code, he can change the template. And still he has embeds throughout. This also means that Clarion can be used to create programs in other languages. For instance Softvelocity (the maker of Clarion) currently offers templates in not only the Clarion programming language, but also ASP and PHP. Further, like your system Clarion is Data Centric. Since it has been in development for years, it has a myriad of features. For instance in the data dictionary one can define Data Verification constraints, picture elements (how the field is displayed) and so forth. Thus any program that uses a specific data dictionary will automatically have all of the features on every window for a specific table.

There are though many issues I have with Clarion. Firstly, it is STILL a 16bit system. Further, I have emailed their sales people, for questions about the PHP templates, and the Version 7 upgrade (coming this year). And have yet to receive a response. In addition, while the data dictionary is a big help to the programmer, it is not stored in a central repository. Each data dictionary for each project is in a flat? file. (Not text based though). Which means that if I need a particular table layout in a project, I have to import it from another data dictionary. Or simply use the other data dictionary, which means having references to tables that I do not need in my system (though not my generated code...clarion does not generate code for variables and files that are not used in a project).

Now I bring up Clarion for a reason. It is the closest thing on the market now that I am aware of to the type of system I have wanted, until your system came along. Now true enough you are in the beginning stages of your project. When I read your plans for version .6 I could see though that you are right along the same path as I am.

For instance having the settings for a table stored in a table itself. Yet still having the ability to use ini files if one so chose. In addition, you write EXCELLENT documentation. I cannot over-emphasize this. I have yet (until I read your docs) to read the documentation for a programming system of any sort, wherein the projects designers did not obfuscate even the most simple actions in their system. Often explanations of ways to perform things are left unsaid.

In fact just this weekend I had downloaded a program called dbfConvert. It is a php program for converting DBF files to MySQL tables. I ran it several times with his sample dbf files. I had to experiment several times over and over before I finally realized the dbf files had to be in subdirectories of the dbfConvert program. The config file, (his form of documentation) did not make this requirement clear. Yet with your system I was able to set up some tables the first night. I had an excel spreadsheet from one of our clients with a list of persons we are required to contact for them. I imported it into a mysql spreadsheet, ran your system against it, and had a running program. Thus, as I told you in one post already, my boss was impressed.

Currently I work in the shipping department of my company. In a few weeks, they will be hiring a replacement, so that I can concentrate on programming full time. We have an entire system I will be programming as a replacement for the current programs they are using. Eventually it will be a full accounts receivable, accounts payable, payroll, order entry, sales contact manager, system. It will have many options and modules.

I could take several paths. One hand write the code. Two (the one my boss's brother wants me to try), I could download another system as a starting point. Then modify their code to meet our needs. Well you commented very well on the difficulties of doing that. Fortunately my boss and his tech guy (who is a windows networking fella not a programmer) agree with me that this option is the worst. Third I could use a programming system like Clarion, and be disatisfied along the way with the way they SHOULD have done things. Finally, and my choice,is to work with Dataface. Not only using dataface to program our modules, but also working with what I hope will become an entire community of dataface developers as the dataface system grows along the lines that you have spoken.


With all of this in mind I wish to toss out several ideas to you. This is all merely brainstorming on my part. A lot of it is inspired by your Dataface system. Some of it I have thought about for years.

As you know most systems, can be datacentric. Heck even images can be stored in databases. Most programming systems add data as an afterthought. Thus we have had to cludge our way through often. Most data can be accessed the same way. We do not need different routines to read this table than we need to read that table. Thus the reason your system works so well.

Further, the large chore of programming is the repetition. When we have a data spec, we have to enter it somehow. This means creating tables, then adding fields, adding all of the requirements for data validation, data integrity, relationships and so forth. I have noticed that very often these too, are the same. Table specs for contact information rarely change. In our case we have patients to deal with, insurance companies, venders, referrers (who refer patients to us), sales persons etc. Right there are five tables. These five tables most likely follow nearly the same format. Name, Address, State etc. Each table may have a few additional fields which differ from the other tables.

Since this information is always the same, why not place it too in a database? For that matter field information is often the same. Name fields are usually text, often 20, 30 characters long. With fewer data validation requirements than say a phone number. If there is a field information table, then when one crafts a table, the field specs can be pulled from the field information table. If information on the type of field you want to create is not existing yet, then it is added at that time, stored, and ready for retrieval next time. Then as you create a table spec, its Specs are stored in a Table Information table. Thus the table information table, relates to the field information table. (Technically the table information table would have a table information item table for the fields and that table is what actually relates to the field information table).

Since Data relationships are often the same, they too could be stored in a Relationships table. So if I use a particular table, or series of tables, their relationships are already set up.

All of these specs could be uploaded, at the programmers option, to a central, or any number of central, data repositories. Thus if I need to create an order entry system, I could go online, and see what order entry tables and systems are already created. Download those into my data repository and v—ila, in a matter of minutes the base of the system that I need, if not the ENTIRE program, is already written. Then I can add modules as I define later. Or I can download more, modify them, adding fields more tables, relationships etc, to polish my system. Then if I chose, I could return my changes to the repository I got them from, or another repository.

This data centric repository programming system does not have to be related only to tables, fields, relationships etc. It could also be related to programming logic, business logic, business rules, and so forth. We could use it to create a report writer, a query system, data conversion and so forth. Naturally some of these options would be a while down the road. But by having everything stored in a data system, just like any other data, the speed at which we could create any specific project would be drastically increased. As your dataface has already increased it for many already.

I hope these ideas are clear and that you find them interesting. I would like to get your feedback when you have time. I know you are very busy. I would also like put them on the forum, but I wanted your permission first. If you would like to discuss these things on the phone please let me know. I can call any time.

Thank you for your consideration in these matters.

Joseph James Frantz

Personal Contact Information:
(216) 453-0684
boscagarda-dataface6031@yahoo.com

====End Quote====

[Aoirthoir]

Mr. Steve responded with some excellent ideas. In particular that we could create another program called "The Dataface Configurator".. "Dataface meets code generation". We all know that he has been very busy with school, consulting, dataface and so on. After I had originally emailed him I realized it should not be too difficult to create some databased system that would generate the needed Dataface INI files on our own. When he responded about the Dataface Configurator, that solidified my own perceptions. Basically, as he pointed out, we could move all configuration data to a database. Further, it would not have to be just the data that Dataface uses. In the configuration database could be information specific to MySQL, for instance in the Field Spec table would be a check box, "Allow Nulls?". It would also have any other field specific info that MySQL saves.

With that in mind, I would like to start tossing these ideas around with those of you that are using Dataface. I believe that a good starting point would be several tables for Field and Table specific settings. So we would have to gradually examine what information do we regularly need to store about fields. Then add that information to a table for field specific data. The same with Tables.

Later on this evening (I hope) I will try to explain a little further. If anyone has any comments about what I have written thus far, I am certainly interested in them. If anything I have said is not very clear, I will try to clarify.

Holler back yall.

[Aoirthoir]

In addition, it would be Dataface itself that the Dataface Configurator would be written in. The tool would become the tool for programming itself. Once we have an understanding of what Tables and Sub-Tables we need to configure dataface, and what our future desires are, even if it is something dataface has yet to support, we could begin to design the database, and then use Dataface to access it. I would be perfectly willing to host the entire system on my webspace.

[Aoirthoir]

Now by storing all of the SQL information for any table in a database, we could use the database to generate the SQL statements to create or modify the database or table specs. This would be useful for other systems besides MySQL. dBase? Clarion DAT files? Access? Excel?

Of course with some of these, some features will not work. Eventually though, as in the Allow Nulls example, if we output a file that technically did not manage those features it would not matter, because the programs themselves we output would handle the validation. Which is what we want to do even in the case of MySQL...we certainly do not want to have the user seeing a php/mysql generated error, instead of our own clean error msg designed in the theme of our website.

That is why I said we can begin thinking now about what specs we want to be able to set now, at the field, table and database level, because some of those specs will be database specific (my sql prevents no nulls whereas excel does not), and some of the specs will be enforced by programming on our part.

By creating such specs, and using a code generation system to enforce it, we would not have to search through entire blocks of code to find what we need, we would just go to the spec we were looking to modify, and change a few lines of code right there.

In addition some other examples of things we would be able to do, is state that such and such a field is a TOTAL field. For an invoice header as an example. Then in our spec we would just have to have a link (similar to normal data integrity constraints) to which field it was totalling. In theory we would not have to over and over again write the code to do totallying. Do it once, done.

We could also have code embed points for each field such as:

Before entering field,
After entering field,
after leaving field,
after incorrect data entry on field,
and so on. Some of those embed points could be direct code. Even SQL statements. But they could also be simple statements like "Totals invoice.item.price"

A little simplistic but hopefully yall see what I mean.

[Aoirthoir]

In addition, another thought just came to me. If we have databases for the configuration data, then it does not matter what database format any specific person uses. One person could create an excel spreadsheet, another could use access, another mysql etc. From there it would only be a requirement to convert it into whatever format the Dataface Code Generator used. And of course for that matter if someone preffered some other system, access for instance, then they could write their own code generator in that system. Even frankly a report writer.

Though this wont be as powerful as the template based code generator clarion uses, it is always an option in the beginning.

A lot of this is just me musing right now yall so please bear with.

[njw]

I'm sorry because I am in a rush right now, but I feel a couple of inital coments might be helpful.

First, I am always cautious about automating too much as that makes the software complex and complexity always brings bugs with it. The effort needs to be in the area of automating the bits that are repetitive and don't need complexity - which Dataface does well now - but being very careful how far down the path we go.

Second, whilst template data definitions are a nice idea at one level, reality usually means that for wide access the data definitions will either need to be amended every time for the precise application or have a large amount of redundancy and/or complexity built into the definitions. Either way, I suspect I'll find it quicker to develop my own. We already have phpAdmin for manipulating MySQL and I never, ever, like to re-invent the wheel!

Putting the data definitions into a table already happens - it's called MySQL - and to be honest, I don't think anyone should use Excel for any kind of database application.

So, I think this is a very interesting set of posts and well worth consideration, but let's make sure we keep Dataface as a simple to use tool - that's its real power in my view!

[Aoirthoir]

Ok been a while since I had a chance to really reply to this. As to the Dataface configurator, it would be an entirely seperate program from Dataface. Essentially it would be just a bunch of tables in MySQL. Then of course I can use Dataface to display and edit those tables. In those tables would be stored the configuration information for any project I was working on. In the beginning, a program would have to be written to read those tables and then output the needed INI files for dataface to access for any specific project. The same types of information that are now stored in the INI files would be stored in the MySQL tables.

So for instance one table lists basic information about a specific project. Such as the MySQL server, the Database, the User and Password. Just like in the Config INI. Also another table would be a list of tables in this project. It would have a field for the table name and the description. Thus, as in the config.ini file you can say contacts_table = "Contact Information" well the same would occur in that case with two fields at least being designated. Table_Name Table_Description. If there was more information we wanted to store about a table, be it comments, or further dataface config info, we could have fields for that also.

Then there would be tables for the relationships.ini, fields.ini and so forth. So essentially any .ini file that dataface uses, would have a table. Thus storing all of the data related to an application in one central place. Taking it a step further, just as orders are related to a customer in a standard database, you could have a project header which then by means of relationships, ties together all the settings for that project in the MySQL tables.

Ok so that is the basics. Write out the required tables. Set up the required ini files in dataface. Then we would use dataface to manage our settings. (Again currently with a need for some method to write out the ini files in the proper directories). Now if in the future dataface itself actually is written to use MySQL to store and access project info, even better. But this is not required.

As to a lot of the other things, such as tables of field information which can be copied to table structures and so on, that kind of thing would be done entirely seperately. But, it would still be done in dataface.

Hope this starts to make more sense. A lot of the stuff above is brain storming to see what the possibilities are.

[Aoirthoir]

Regarding a couple of things:


========================
Second, whilst template data definitions are a nice idea at one level, reality usually means that for wide access the data definitions will either need to be amended every time for the precise application or have a large amount of redundancy and/or complexity built into the definitions. Either way, I suspect I'll find it quicker to develop my own.
========================

Under normal circumstances I find this to be true. For instance if I attempt to pick up another application, say an accounting program, to which I have the code, I often find that it is easier just to start with scratch than to use their code because of the complexities. However, with Dataface, an application is now much simpler. What are the fields, what are the relationships. Perhaps in the future, what are the totals fields, and so on... A lot of times a lot of tables are going to be very similar if not nearly identical. For instance contact tables usually have a lot of similarity to them. Name, Address phone number. Now there will be additional fields or variations in the fields used in some cases. (Say a medical contact database or a european address and phone).

So with Dataface we have the ability to instantly have an application running. Then we tweak it here and there to fit more of our needs. But the basics..how the record is read, how it is displayed and so on, really does not change from table to table. So too with the basics. If I have in my own personal Dataface Configurator a list of Contact Tables I can than choose the one that most meets my needs and tweak it, adding a field or two, or adjusting one...

Now if there is a central repository (either in house or publically accessible and shared) then work that others have done might benefit me. It might not. I take what I can get.

=============================

We already have phpAdmin for manipulating MySQL and I never, ever, like to re-invent the wheel!

=========================

Yes I agree I dig phpMyAdmin. However, I have found often it is tedious to click over and over a lot of the boxes that it has. On my computer I have to do a horizontal scroll to change fields or if I am creating a new table, to add fields. As I learn SQL I am more and more just doing copy and pastes to generate my SQL tables or changes to them or the data. Now for other things I find phpMyAdmin excellent..for instance to copy a table structure or the data. Now what if phpMyAdmin could not only copy the table structure but also the Dataface configs? The relationships, how you want this field displayed in this app and so on...Well that's essentially what the Dataface Configurator add ons would do. So if I already have a setup, I dont have to go into phpMyADmin and redo all my work (including my Dataface config work...) So in a way it is like you said, not reinventing the wheel...but in this case I mean not redoing work I have already done.

[njw]

I've had a few problems recently too - work and PC related.

My comments were really based on where I see Dataface's biggest community - people like me who understand a little php and MySQL, but will never become experts. Dataface has solved a problem for me by allowing me to produce a multi-user web based front end for an application easily. v0.6 looks to produce a set of features that will be helpful, but whatever is done needs to keep the interface simple. I can see that putting the ini files into MySQL might make life simpler, but I am still to be convinced!

On phpMyAdmin, I suppose I am looking at the number of times I create or modify a database (rarely).

I accept that I may not be a typical user!

[Aoirthoir]

With phpMyAdmin, we have been creating a lot of databases and tables lately. Mostly what I do now is, create the first table with as many matching fields as possible (for tables that are very similar) in phpMyAdmin, then I do a SHOW CREATE TABLE [TABLENAME] to get the layout. I copy that into a text editor and copy it down making changes as needed. Then I paste the code for the other tables back into phpMyAdmin and I then have all of my tables created auotmatically.

If I were only creating one or two tables every few months phpMyAdmin would definitely meet my needs. But this project we are working on will eventually be sold to others with customizations. In addition the base tools (dataface, dataface configurator, our MySQL stored procedures and triggers and so forth) will be used in the future for other projects. So we will be creating a lot of tables. phpMyAdmin would work well if I could see all of the fields for creating a table without horizontal scrolling but click, scroll, tab - screen shift, doesnt work so well for repetitive work. As an update comes out, and those things are changed or if in CSS we can change them, then my mind about that will change.

As far as the simplicity of dataface being in a mysql table instead of ini files, consider this...You use dataface to put data into a table and display it. How much more complicated would that data be if it were stored all in text files that the users had to open. Now, if you are also using table relationships in dataface, consider how even more complicated it would be to make a user go to one text file for a customer record, then to another text file for the order header and then yet another text file for order items, but still another text file for inventory items. Even if the text files are well sorted this creates some possibilities for mistakes to creep in.

This happened to me when I was working on the fields.ini for one of my tables. I have quite a few settings and I needed to add another parameter for a field. I forgot that I had already set parameters for that particular field. But I scrolled through, eyeballing it just in case, and could not see it. So I went to the top and added the field. (Yes it should have been in alpha order). So that parameter I set, did not take, because later on dataface hit the other parameters for that field and used them.

Now, consider using dataface itself to configure your tables. So perhaps you have a main project tab. In here you have the stuff that would be stored in conf.ini. Also perhaps the path. Whatever things are needed...in addition with the new permissions setup you would have the settings that are currently in projectpath/conf/ApplicationDelegate.php or any other application specific information.

So then on a project based on the conf settings, you might have a tab for tables. This gives you a dataface LIST for each table. When you click on that the specifics of the dataface instructions appear. (Probably not much in a view or edit other than table name, perhaps permissions etc..)

However on the view or edit for a particular table you have other tabs for that specific table such as your Fields, Relationships, Valuelists and other tabs. These would be the same setup as in those ini files. Each of these then is a table in itself with fields that you would normally place in its ini file such as for fields:

widget:type = (this would be a select with all the available types)
widget:attrs:size = (a text value which is actually set to allow only numbers)

etc..

Now it is true that as dataface evolves more fields would be needed. However that is the beauty of dataface. Simply add a field to the table in mysql and it is added to dataface. So we would add a field to the fields_setup_table (whatever we call it) and then that particular portion of the field can now be changed by dataface instead of in a text file.

Now a lot of what I wrote up there in the beginning is just some brain storming that may or may not ever be useful. For instance I use the field prefix ID_ to tell me that this field manages something about the table and is internal, not for users at all. So ID_RECORD is the record number of a record. I really never want users to see it. I always want it to be mediumint (except in the most rare cases where there is the possibility for millions of records. I will handle that when it occurs). I also, for those times it is displayed in reports by admin, want it zero padded. I want it unsigned etc....this will always be the case.

Now some of those are mysql parameters some of them are dataface parameters. In the beginning we start out with just the simple dataface parameters for the dataface configurator. And assuming dataface continues to read ini files instead of the mysql tables of the dataface configurator, well that is ok too because the configurator (or an additional module) could be designed to write out those ini files. Later on maybe we add mysql parameters to the configurator. Thus in one place you have your mysql and your dataface.

(As a side note...I am attempting as much as possible to put information about the database or tables into the table itself...and will use stored procedures and triggers of mysql for everything I can. This centralizes my control even more and like with html and css it seperates content from layout...or in this case data logic (in mysql) from data presentation (in dataface). And if the time comes that dataface can make use of things like mysql views and mysql relationships then the seperation will become even deeper. The purpose for this seperation is to insure that even if I am adding records on an admin level in mysql itself, the table constraints remain...(See my history table posts for an example)

So the simple dataface configurator would just store dataface settings for a project (and all your projects) in a series of tables which is then accessible from a central location (dataface itself). Then it would write out the ini files when you wish it. This automation can of course create errors (bugs) for instance if it writes out widget:typp = "whatever". But most of those are very simply solved. There really isnt a lot of logic in the program. Just storing data and writing it out (or it being accessed by dataface in mysql directly if it ever should be).

Now hopefully eventually dataface will grow to have the ability to copy records natively. It actually can be done now with triggers and delegate classes, which you could use to make calls to mysql stored procedures or do the sql yourself in dataface __sql__ = . And so it can used it to copy one field setup to another. Or one table setup to another. Then one simply enters the tabs for those and makes the changes or additions you want. If there is nothing that would match what you need, you just start from scratch. As I said above there are a lot of times certain fields will always have certain specs (ID_RECORD, ID_CHANGED, ID_CREATED, ID_USER, PHONE, STREET_ADDRESS, STATE, ZIP). So I dont see the point of doing them over and over when I could just copy.

Currently in dataface I can copy and even mysql. But it requires many steps. Copy this value from that fields.ini to that fields.ini, and that value from relationships.ini to another (or several) relationships.ini. Whereas with this method I could just copy a record in mysql. Even doing it directly in mysql without an interface would be faster than by hand. Also it would insure fewer human mistakes. And thus I personally would have as I said above a field layout table. Then I could just pick fields from it and be done. But this is not part and parcel to what would be required in the configurator. It's just one of those ideas. If it becomes useful cool

Hope that explains better my thinking.

[njw]

I think we are totally different users - you are now pretty much a professional programmer, whereas I am a very part-time programmer. I need something that is really simple because I haven't got the time to go up steep learning curves. Years ago, different story!

Have you considered using DbDesigner? It's open source and allows you to design databases in a WYSIWYG fashion. It will then create the SQL statements to create the MySQL database, tables etc. The database designs can be copied and then amended pretty easily - and it gives you a pretty print option so you can show the relationships. Doesn't answer all the problems, but it helps with the phpAdmin stuff. I have dabbled with it, but hit a couple of issues with it creating indexes I didn't want - then I ran out of time and had to do my real job!

[Aoirthoir]

I will have to check it out. Thanks for the info.

Also for me the entire point of a configurator..is to eliminate learning curves and so on. Basically figure as much as possible pre-done.

You know how like with dataface currently..it works because most of the time your basic programming stuff is going to be the same...How you access a table how you display it and so on. While from table to table there might be little differences...in cosmetics..field types and so on..but the basic read, list, write routines are always going to be the same..so why do them each time?

Well my feeling is kind of the same with a lot of these other things..a phone number is always going to be the same. Why should I click add field, type in the field name, drop the list for the type of field it is (char, int whatever) then type in the number of characters or digits, then drop the null/not dull field and so on all the way down to the comments field... since all my phone numbers are going to be speced the same, i might ass well have a repository for it and just select, copy..done

thats kinda one part of this idea.