Re: Multiple specification of constraints

A good discussion - first my response to Dawn's original thoughts:

Dawn wrote:
> Where should constraints be specified?

In one place, the best place, where they can be specified declaratively, and which won't allow violations of them. If you require violations, your consraints are wrong.

In addition, the constraints must be easy to extract, so that boilerplate code can be generated from them. I think that's key, and while it's done in ad hoc ways by many different companies, this type of generation must be elevated. It is logic in action; the implementation technologies are irrelevant.

> Take a services framework that includes an interface for "web services"
such
> as SOAP-wrapped XML that comes in from some place external to our system
and
> also a user interface for people doing data entry. This is an appropriate
> design for Orders, for example.
>
> In this case, we could break down the high level services as follows:
>
> Web Service "entry service"
> GUI
>
> (those two being the analogous interfaces to what used to be called
"batch"
> and "interactive" computing)
> These services might then both be clients to the same
>
> Data Validation Service
>
> This would apply all business rules and constraints related to the data
> entered through one means or the other.

This means that your constraints have to be applied for ALL data entry. (Just repeating because it's critical.)

> Ignoring workflow types of services
> for now, we would add in at least
>
> Data CRUD Services to Create, Read, Update, and/or Delete data stored in a
> database
>
> When we use DBMS's, the Data Validation Service and CRUD Services are
> tightly coupled. This gives an advantage of a "lock box" (as Al Gore
might
> say) for our data. It is not only impossible for a user to write data with
> an improper type, for example, it is also impossible for an IT
professional
> without DBA access to do so.

Any file system does CRUD. No one must be allowed to update data independent of the "data validation service." Given that we're trying to abstract above the level of a file system, I see no good reason to differentiate the "CRUD services" - that's what the DBMS vendor does.

> However, if you look at software applications, it is common, even
required,
> to have a GUI employ such business rules, for example, in giving users a
> list of valid codes from which to choose, or even sometimes in finishing
the
> data entry based on prior entries (as with Excel) or based on valid
entries.
> Additionally, once a set of data is passed to the Validation Service,
prior
> to heading to the CRUD Services (whether Validation and CRUD are tightly
or
> loosely coupled), these business rules are employed again.

The GUI data "validation" isn't really validation in the same sense, though its rules are derived directly from constraints. I'd say the GUI should be "declarative" - rather than validating, give the user no choice to enter bad data. A drop-down list doesn't validate - it's the sole input mechanism for that field.

> Isn't it rather common, for example, for there to be a drop down box
locking
> in the user to a valid set of choices, with the specification for this
drop
> down box in one place/language and then these choices again validated when
> changes are made to the database? We are then taking our industry's data
> (specs/code) and duplicating it, thus making it difficult to maintain
while
> keeping integrity in this data.

Agreed - that's the trouble with multi-tier systems, and it's the reason code generation is so popular these days. Declared constraints give a much, MUCH stronger basis on which to generate code, boilerplate, data access mechanisms, etc. They are for more than just validation - they are are the meaning of your application.

> My point is that with the way that software is written today, quite often
> data constraints are coded in multiple places, even multiple languages.

With multi-tier systems, they need to be in multiple places, in slightly different forms. But that code doesn't need to be written and maintained separately.

> There are apparently many people as well as exceptional conditions that
> might prompt multiple locks on a door, but this need not be the norm, I
> would think.

Don't think of it as multiple locks on a door - think of it as channeling everyone going to a secure room past the cameras, through the patrolled hallway, etc.

If your data has much value to the organization, then it will be accessed by a wide variety of programmers. Declaring constraints does more than just defend against corruption (though that's extremely important). It also communicates the meaning of the data and the system.

> If the front-end services are written in Java, for example, then I would
> prefer that the GUI and the Validation Services make use of the exact same
> constraint specs and logic to validate the data against the constraints.

Agreed, the specs should be the same. Their implications within a tier are not. Only the GUI needs a drop-down list, for example, but the contents of that drop-down are derived using constraints. If you have an HTML and a Swing interface to your data, you'll need different code for the same drop-down, but it's still the same constraints which define its essence.

Separating essence (the constraints) from accident (the implementation language) is critical, and I hope it's the "next big thing" in software development.

> Even if we tap into the specifications that are placed in a DBMS, taking
the
> logic of the DBMS that is used to apply the constraints and using that up
> front in the GUI isn't happening in current software applications, in
> general, for at least a few reasons I can think of:
>
> 1) The DBMS's logic is applied when CRUD services are launched, while the
> logic in the GUI is not ready to launch such services, so there is not an
> easy means (or perhaps any means) for the application software developer
to
> tap into the DBMS's logic.

There can and should be. The system catalog should tell the GUI most of what it needs to know.

> This is true for type constraints as well as ref
> int constraints where the GUI might need to come back to a user and ask
the
> equivalent of "Are you sure you want to delete this folder and all of its
> contents?" when a user requests that a parent be removed, for example.

That's derivable.

> 2) The application developer cannot easily see the specs and logic applied
> by the DBMS to know whether that is the same as what they believe they
need
> for this application. So they do not even make the decision on whether
the
> constraints they are applying are tighter in this case than what the
> database spec permits. It is easier for the application developer to
treat
> the DBMS constraints as a black box where they try not to bump into the
> limits, but they don't think of them as related.

Code generation (or direct higher-level language execution) based on constraints doesn't give them as much of a choice in the matter.

> 3) Reading a database to get constraints when trying to respond quickly to
> the end-user is sometimes considered by the application developer to be
out
> of the question compared to employing perhaps byte code that runs in an
app
> server JVM.

The constraints can be read up-front; JSPs and ASPs are dynamically-generated GUIs, and can read whatever they like on the server, before the HTML is streamed to the browser. The app build process can use the DBMS to generate much, so the code won't run slower than something with hard-coded duplicated constraints. The structure of your database is going to change far, far less than its contents.

> One solid design could be to have all of the constraints specified and the
> validations applied in a Validation Service that is able to reuse the same
> constraint specifications that the GUI uses and that the CRUD routines
use,
> all specified outside of the DBMS.

I still haven't seen a compelling reason to be outside the DBMS. You're DESCRIBING a DBMS!. It's more than just a file system!

> We could code and run all such services
> in a JVM on an app server (rather than the database server).

And you were worried about speed before? :-)

> In this case,
> employing a file system that might not be considered a DBMS would be
> acceptable provided that we still have the door locked.

And feel like doing a lot more work.

> This can be done by
> permitting CRUD services on the stored data only via the designated
> services.

Could be, but why?

> The constraint specs and logic need to be moved right up to the front of
the
> application -- to the GUI.

Depends on what you mean by "moved." Certainly they need to be manifest there.

> I know some have advocated for the database
> language to deliver the applications, but that tends to tightly couple way
> too much of the code, rather than employing a good design pattern, such as
a
> services pattern to the application.

I'm unclear as to why a services pattern is "good," and what the nature of the coupling you mention is. I don't see it.

> I'm just thinking out loud and I'm sure something I wrote was not
brilliant,
> but help me out here -- isn't this a real issue in application software
> development written today?

It's the very heart of the matter, and I thank you for raising this discussion.

> I'm suggesting something rather radical --
> decouple the constraint specification and validation logic from the
> database. Does anyone agree?

Not I. A database is defined by constraints and validation - decouple them, and you have what we already have: file systems. You have what DBMSs were created to abstract above! But again, a good topic.

• Eric