HyperText Structured Query Language

HyperText Structured Query Language (HTSQL) is a mechanism for accessing industry standard SQL-92 data sources over HTTP . This document specifies a URL scheme , HTTP methods and extensions which enable database access from standard web browsers. The principle advantage of HTSQL is the expression of queries as URIs instead of a traditional SQL syntax: for common database tasks, path-based URLs are both simple and intuitive. A secondary advantage of HTSQL is the integrated use of the HTTP protocol to provide authentication, data caching, encryption, content negotiation, and numerous other network operations. This approach to SQL-over-HTTP puts forth a reusable, application independent, and testable middleware layer which translates HTTP requests into SQL statements, returning the execution results to the user in a format their user-agent can handle. While database access over HTTP is a very common feature of today's systems, most implementations are application specific; or fail to take advantage of HTTP/URIs to the fullest extent possible. Application specific protocols are often limited by immediate needs of the development, and hence do not enable more generative usage. Other protocols that send SQL directly fail to make their URLs human-readable and usable without detailed knowlege of SQL. HTSQL is intended to be both application independent and human-readable -- a thoughtful approach taking into account the potential synergy of HTTP and SQL.

The target audience for HyperText Structured Query Language is not career programmers, nor is it casual users. HTSQL is designed for technical "power" users including screen designers, database administrators, medical researchers, and other accidental programmers. HTSQL advances a human-friendly URL based query syntax over a traditional SQL queries, and HTTP over a more typical database access protocol. URLs are instantly familiar to users who have been using the web for many years; human-readable URLs provide direct control over database information which is often lacking in traditional systems. A database accessable via a web-browser with persistent URLs allows query results to be bookmarked and emailed to collaborators; such an interface also enables easy information navigation using web-browser controls. HTSQL offers great flexibility than a purly graphical user interface which necessarly limits the kinds of the kind of retrievals that can be specified. With even moderate exposure, an advanced user can learn to modify a URL to achieve results beyond what a graphical interface may provide. HTSQL builds upon existing standards. By using standard HTTP, the database access protocol need not be burdened with encryption, signatures, cache control, content types, authentication, detailed audit-trails, or other network related issues. HTSQL provides increased security; an HTTP interface enables broad database access, while at the same time providing a clear proxy for database activity monitoring. Implicit in this approach is a compromise. While 90% of common database tasks can be expressed in human-readable URL format, the other 10% will necessarly be elusive. Rather than find equivalents for all SQL syntax expressions, this interface will drop-down into a "raw" database query syntax when any advantage that comes with a URL based approach is negligable.

The overarching goal of HTSQL is to provide a standardized, human-readable mechanism to access SQL databases over HTTP. For all but the most complicated database interactions, the corresponding URLs in HTSQL must be easy to read and understand. Conversely, while it may not be simple to perform, all activities possible with SQL-92 should be possible, dropping down to raw SQL-92 when necessary. Since it is meant for HTSQL be used by casual programmers, error messages must be informative and layered so that casual users are not frightened and expert users are given the details they need to resolve an issue. HTSQL requests must use, whenever possible, standard URI path-based syntax and query formats -- relying upon SQL syntax only when there is not a well-accepted URI format. HTSQL must use HTTP/1.1 features for well-known operations such as authentication, caching, range requests and content negotiation; HTSQL should extending or augment HTTP/1.1 only when necessary. HTSQL must allow for fine-grained access permissions as allowed by SQL-92, mapping remote users onto specific database users. HTSQL must support the standard FORM element for common database operations over HTTP/1.0; requiring full HTTP/1.1 only when necessary. HTSQL should support a wide-variety of query result formats, including JavaScript Object Notation and the eXtensible Markup Language for standard Javascript and DOM enabled web-browsers, as well as Comma Separated Variable for spreadsheets and data analysis tools. HTSQL should minimize configuration, using information available in the database catalogue to reduce deployment burden; basic functionality should not require additional database tables. HTSQL should minimize server side-state; following as much as possible the principles of represenatational state transfer, . HTSQL must support and use style dates. The HTSQL specification may require SQL-99 constructs only when they provide significant value and are easily emulated using SQL-92; this specifcation should not prevent future refinements to fully enable SQL-99 or SQL-2003 support.

As a prelude to the formal specification, we provide a taste of HTSQL by presenting a limited set of URLs; associating each URL with an equivalent SQL expression and a description of the corresponding query result. For these examples, we use a "Research Exchange Database" schema, representing participants tracked for multi-study medical research facility.

The first set of examples assume that the user-agent is a standard web-browser using the HTML FORM element and the the GET HTTP method. In this case, the HTSQL operation is specified at the very end of the URI; ugly but necessary for this particular use case. GET /red/individual/select

This selects all columns from the `red.individual` table and returns the results in a format determined by HTTP content negotiation. The corresponding SQL executed in the database would be: SELECT * FROM red.individual ORDER BY collection, individual collection | individual | family | mother | father | gender ----------------------------------------------------------- AGRE | 001 | 001 | | | male AGRE | 002 | 001 | | | female AGRE | 003 | 001 | 002 | 001 | female AGRE | 004 | 001 | 002 | 001 | female ... GET /red/individual/select?gender=male

This selects all male individuals in the research database. This query syntax corresponds to the URL generated by a HTML <form/> element, with an <input name="gender" />. SELECT * FROM red.individual WHERE gender = 'male' ORDER BY collection, individual collection | individual | family | mother | father | gender ----------------------------------------------------------- AGRE | 001 | 001 | | | male ... GET /red/identity/select?birth-date<=1970-01-01

This rows from `red.identity` with a `birth-date` less than or equal to January 1st, 1970. This example could be encoded in an HTML form using input element having the `name` `birth-date<` (note the < after the field name). SELECT * FROM red.identity WHERE "birth-date" <= '1970-01-01' ORDER BY collection, individual collection | individual | family-name | given-name | birth-date --------------------------------------------------------------- AGRE | 001 | Flintstone | Fred | 1953-03-11 ... GET /red/sample/select?kind=blood,cell-line GET /red/sample/select?kind=blood&kind=cell-line

It is possible to match on more than one value by either using a comma separator in the query argument or by providing the argument more than once. The latter usage reflects the output of an HTML form using a set of check-boxes having the same name. SELECT * FROM red.sample WHERE kind IN ('blood','cell-line') ORDER BY collection, individual, sample collection | individual | sample | type | derived-from ----------------------------------------------------------- AGRE | 001 | B1 | blood | AGRE | 001 | C1 | cell-line | B1 ... GET /red/collection/select?description~=genetic&name!=Clinic

This query selects rows from `red.collection` with a `description` containing the word `genetic` with a `name` that is not equal to `Clinic`. SELECT * FROM red.collection WHERE LOWER(description) LIKE '%genetic%' AND name != 'Clinic' ORDER BY collection collection | name | description ----------------------------------------------------------- AGRE | Autism Genetic Resource Exchange | ... ... With a bit of exposure to these sorts of URLs and seeing the corresponding results; the occasional programmers should be able to combine these operators in a generative manner without requiring assistance. The above queries are rather straight-forward mapping of URIs onto SQL, and thus are not particularly complicated. The next few sets take more advantage of the SQL92's `INFORMATION_SCHEMA` to produce far more complicated SQL expressions with simple URLs.

It is often useful to visualize data stored in a relational database as being organized into hierarchies, where child tables contain as part of a primary or unique key information uniquely identifying a category or parent table. HTSQL uses the hierarchical URL notation to accomplish implicit joins of these relationships transparently. GET /red/individual;gender=male/select

This query is identical to a query in the previous section, only that the filter criteria, `gender=male` is included in a parameter rather than in the query arguments. Parameters are a useful feature of URIs allowing relative URI references to include filters implicitly. SELECT * FROM red.individual WHERE gender = 'male' ORDER BY collection, individual GET /red/individual;gender=male/sample/select

This query returns all `red.sample` rows for male individuals. In HTSQL, if two tables mentioned in a URI are related via a foreign key reference, they are automatically joined. SELECT s.* FROM red.sample AS s JOIN red.individual AS i ON (s.collection = i.collection AND s.individual = i.individual) WHERE i.gender = 'male' ORDER BY s.collection, s.individual, s.sample GET /red/family;collection=AGRE&family=002/sample/select

This query returns all tissue samples for family `002` in the `AGRE` (Autism Genetic Research Exchange) collection. It is perfectly acceptable to skip intermediate levels as long as they can be joined by an intermediate. While the query below isn't ideal (it could be automatically optimized), it is kept simple to better illustrate the concept. SELECT s.* FROM red.sample AS s JOIN red.individual AS i ON (s.collection = i.collection AND s.individual = i.individual) JOIN red.family AS f ON (i.collection = f.collection AND i.family = f.family) WHERE f.collection = 'AGRE' AND f.family = '002' ORDER BY s.collection, s.individual, s.sample With this set of examples, the divergence between HTSQL's human-readable URLs, and the corresponding SQL starts to emerge. The "path" based metaphor of URLs, together with the parameter syntax allows relatively common joins to be easily specified.

HTSQL provides explicit support for selecting a particular row using primary key columns of the given table. When using this syntax, each primary key column is called a label, and a dotted sequence of labels is called an identifier. Labels are treated different from normal column comparisons in that they ignore case and are independent of leading zeros. GET /red/collection!agre/select

This query returns the 'AGRE' collection using the identifier syntax -- comparison is case insensitive. It is an error if more than one row is returned. SELECT * FROM red.collection WHERE htsql_normalize(collection) in ('AGRE') ORDER BY collection GET /red/individual!Agre.3/select

This query returns the individual identified by 'AGRE.3'; since leading zero's are stripped, this will match an individual with label '003'. Allowing for leading zeros in a code is common practice when dealing with programs that use fixed-width identifiers, hence their explicit support within HTSQL. SELECT i.* FROM red.individual AS i JOIN red.collection AS c ON (c.collection = i.collection) WHERE htsql_normalize(c.collection) in ('AGRE') AND htsql_normalize(i.individual) in ('3') ORDER BY i.collection, i.individual GET /red/individual!agre.3,agre.0/sample/select?kind=blood

This query returns blood samples for two individuals: 'agre.3' and 'agre.0'. Just as the previous queries must return exactly 1 row, this returns exactly 2 rows with the corresponding identifiers -- implicit in this syntax is the assertion that a particular row exists. SELECT s.* FROM red.sample AS s JOIN red.individual AS i ON (s.collection = i.collection AND s.individual = i.individual) JOIN red.collection AS c ON (c.collection = i.collection) WHERE s.kind = 'blood' AND (htsql_normalize(c.collection) in ('AGRE') AND htsql_normalize(i.individual) in ('3','0')) ORDER BY s.collection, s.individual, s.sample Identifiers provide a handy notation for resource location; it satisfies a role similar to the Internet's domain name system. This short-hand notation can be used to label physical artifacts, providing a direct correspondence with the database representation. For example, `AUT.3.B1` written on a laboratory vial provides human-readable and informative label, yet one that gives all of the information necessary to retrieve the corresponding information from the laboratory database. Although the notion of identifiers are not part of the SQL standard; they can be easily supported with data available in SQL92's `INFORMATION_SCHEMA`.

Besides general control over hierarchy and filtering, HTSQL provides a support for column grouping (for sparse data handling), column selecting, and result ordering. When a set of columns is often accessed together, they should be moved into a facet table which has the same primary-key as its base table. In our example schema, the `identity` table is a facet of `individual`: for each row in the individual table there is at most one corresponding row in the identity table. GET /red/individual(gender,collection,individual)/select

This locator finds all rows from the individual table, but only returning the selected columns. By default, sorting is performed on the first three columns returned. SELECT gender, collection, individual FROM red.individual ORDER BY gender, collection, individual GET /red/individual(gender,*)/select

The wildcard specifier `*` can be used to return all columns of a given table; duplicates eliminated. This is useful for providing a specific sort order, but otherwise returning all columns. SELECT gender, collection, individual, family, mother, father FROM red.individual ORDER BY gender, collection, individual GET /red/individual(identity.birth-date,*)/select

Any facet table can be mentioned in a column selector to include rows from that column-set, if they exist. Note in this case that an outer join is used, since some individuals may not have a corresponding identity. SELECT f."birth-date" AS "identity.birth-date", i.* FROM red.individual AS i LEFT OUTER JOIN red.identity AS f ON (f.collection = i.collection AND f.individual = i.individual) ORDER BY f."birth-date", i.collection, i.individual GET /red/individual/select?identity.birth-date>=1970-01-01

It is also possible to filter content based on a facet; this example returns all individuals that are born after 1969. SELECT i.* FROM red.individual AS i JOIN red.identity AS f ON (f.collection = i.collection AND f.individual = i.individual) ) WHERE f."birth-date" >= '1970-01-01' ORDER BY i.collection, i.individual GET /red/individual(collection,individual,count(sample))/select

Finally, aggregate functions are permitted in a column selector (or filter). This example returns a listing of individuals including the count of sample rows for each one. SELECT i.collection, i.individual, count(s.*) AS "count(sample)" FROM red.individual AS i, LEFT OUTER JOIN red.sample AS s ON (s.collection = i.collection AND s.individual = i.individual) ) GROUP BY i.collection, i.individual

Columns that act in a foreign key to another table can be used in a way that creates an automatic join to the referenced table. This mechanism permits common table join criteria to be used by occasional programmers without having to concern themselves with the details. GET /red/individual(collection,individual,family.contact)/select

This returns one row for each individual, but including in the results the contact person for their family. Generating SQL is possible since the `family` column in the individual table participates in a foreign-key relationship. SELECT i.collection, i.individual, f.contact AS "family.contact" FROM red.individual i LEFT OUTER JOIN red.family as f ON (f.collection = i.collection AND f.family = i.family) ORDER BY i.collection, i.individual GET /red/individual/select?mother.gender=male

One would hope that this query returns zero rows, assuming a strict definition of genetic motherhood. Since the column `mother` participates in a foreign key back to the same table this query is possible. SELECT i.* FROM red.individual i JOIN red.individual as mother ON (mother.collection = i.collection AND mother.individual = i.mother) WHERE mother.gender = 'male' ORDER BY i.collection, i.individual GET /red/sample/select?individual.family.contact~=fred

It is possible to chain specifiers to specify a multi-table join. For example, the above query returns sample rows, but filtered by going up to `individual` and then over to `family` to do a LIKE filter on the `contact` column. SELECT s.* FROM red.sample s JOIN red.individual as i ON (i.collection = s.collection AND i.individual = s.individual) JOIN red.family as f ON (f.collection = i.collection AND f.family = i.family) WHERE LOWER(f.contact) LIKE '%fred%' ORDER BY s.collection, s.individual, s.sample GET /red/individual/select? mother.identity.birth-date==father.identity.birth-date

The `==` evaluation equality operator can be used to do comparisons where the right-hand side is a specifier rather than a literal constant. In this case, we are looking for individuals who's mother and father are born on the same day. SELECT i.* FROM red.individual i JOIN red.individual as mother ON (mother.collection = i.collection AND mother.individual = i.mother) JOIN red.identity as mident ON (mident.collection = mother.collection AND mident.collection = mother.individual) JOIN red.individual as father ON (father.collection = i.collection AND father.individual = i.father) JOIN red.identity as fident ON (fident.collection = father.collection AND fident.collection = father.individual) WHERE mident."birth-date" == fident."birth-date" ORDER BY i.collection, i.individual The overall effect of HTSQL's specifier mechanism is to enable fault-free construction of complicated join criteria. When combined with the other features of HTSQL, it provides very robust filtering and selection with a human-friendly URL. This facility is also possible with the SQL 1999's `REF` data type as introduced in section 4.10 of ISO/IEC 9075-2:1999.

So far the examples have only shown SELECT queries; HTSQL supports INSERT, UPDATE, DELETE, and the MERGE operation from SQL 1999. In HTSQL, data manipulation operators use the HTTP `POST` method, and thus these examples detail both the URL and the POST parameters. POST /red/collection/insert

collection: AGRE name: Autism Genetic Resource Exchange description: This is an data-set of autistic individuals with genetic samples for order.

This inserts a row into the `collection` table; the POST body accepts standard HTML content types allowing a simple HTML form to put data into the database. INSERT INTO red.collection (collection, name, description) VALUES ('AGRE','Autism Genetic Resource Exchange', ('This is an data-set of autistic individuals ' || 'with genetic samples for order.')) POST /red/collection!agre/update

description: This is an data-set of autistic individuals with genetic samples for order; no identifying information is available for this collection.

The `update` operation acts in a manner similar to a `select` allowing a full query to be specified in the URL, providing the replacement column values in the message body. Unless specified otherwords, updates assert that exactly one row is affected, or the transaction is rolled back. UPDATE red.collection SET description = ( 'This is an data-set of autistic individuals ' || 'with genetic samples for order; no identifying ' || 'information is available for this collection.') WHERE htsql_normalize(collection) in ('AGRE') POST /red/individual/delete?identity.birth-date=<1900-01-01

The `delete` operation acts in a manner similar to a `select` allowing a full query to be specified in the URL. Depending upon the CASCADE rules of the database, deleteing a parent object may succeed removing all children; or it may be prevented unless all children are deleted first. Like the `update` operation, unless specified otherwise, a delete can affect only one row to prevent unexpected data loss. DELETE FROM red.individual WHERE (collection, individual) IN SELECT i.collection, i.individual FROM red.individual i JOIN red.identity y ON (y.collection = i.collection AND y.individual = i.individual) WHERE y."birth-date" <= '1900-01-01'

The normalize function reflects expectations of how codes should work in the web, case and leading '0's are not significant. The space, underscore and dash are treated as synonyms. Leading and trailing spaces are not significant and a null value is treated as the empty string. Either normalize is expanded in the query, or implemented as a database function. COALESCE(NULLIF( TRANSLATE(LOWER( TRIM(LEADING '0' FROM TRIM(BOTH ' ' FROM CAST(COALESCE( <column-expression>, '') AS TEXT) ))), '- ','__'), ''),'0') TODO: show DELETE, MERGE, affecting more than one row, and updating facets in one operation and transations.

HTSQL relies upon meta-data regarding an SQL schema in order to provide a mapping of URLs onto SQL statements. In particular, the INFORMATION_SCHEMA as defined by SQL-92 and implemented by most modern databases is mostly sufficient for this purpose. Lacking an compliant information schema, a particular implementation could, of course, provide a replacement mechanism.

A reasonably simple HTTP URI access mechanism requires a restrictive data access and representation model. This chapter outlines the conceptual model which HTSQL supports. While this model can be implemented with an SQL database, not every SQL construct will have a corollary in this model.

A workspace refers to a coarse level of information storage and authorization. In a relational database, a workspace typically corresponds to a database instance or schema. While it is possible for two workspaces to have the same structure, all workspaces in a system need not be structured identically. Besides identifying the data repository, the workspace can be used for access control, limiting particular users to given workspaces.

A class refers to a collection of data items (or entities) having the same structure. An entity is a single instance of a class, such as a particular person or a given object. In a relational database, a class is often associated with a table and each entity in the class is stored as a row. Each class belongs to exactly one workspace and has a name that is unique within its workspace. By convention, the name of a class is singular, e.g., 'person' not 'persons' or 'people'.

A field refers to descriptive detail of a class, such as a person's name or gender. Each entity in a class may have at most one value for each of its fields. If a value is not provided, the field is said to be null. In a relational database, a field is usually associated with a column of the class's table and a particular entity's value is often called a cell. There are three sorts of fields, scalars, references, and special. A scalar field is value represented as a sequence of characters. The text field type represents a Unicode string, each text field has a specified length and is typically shown on the screen with a text box. The memo field type represents a multi-line Unicode string, without a set length. It is usually shown on the screen as a text area. The date field type represents a date, which is typically shown in , YYYY-MM-DD format. A date field is usually displayed as a single text box, but this can vary depending upon the application. The integer field type can hold 32 bit integer values. The sort is a numeric field that is used primarly for overriding the sort in a list. The boolean field type can hold a true or false value and is usually rendered with a checkbox. The code field type is used for unique keys. In HTSQL, codes are alpha-numeric, admitting the dash. Codes are case-insensitive, but case-preserving. Codes also ignore leading 0's. A reference field is used to record links from one entity to another. Implementation details of this mechanism are not specified, but HTSQL uri's use reference fields extensively to associate entities. There are a few special field types supported by HTSQL which are neither scalar values nor references. The file field type is used to store files which have been attached to a given entity. The choice field type represents a list of values, at most one of which may be selected. This field is usually represented on the screen using a single-select drop-down list. The specific value for each choice is a code. Choice items are associated with a title, note, and sort. When shown in a list box, the items are sorted according to sort, and the title is shown in the list box. The actual code value is what is stored and shown in tabular output.

Besides the system provided handle, HTSQL requires that each entity in the system have a unique, human-readable identifier. These identifiers are constructed using a dotted-style concatenation of the relevant code fields.

TODO: this section needs to be complete re-written.

This specification uses the Augmented Backus-Naur Form (ABNF) notation of , including the following core ABNF syntax rules defined by that specification: ALPHA (letters), CR (carriage return), DIGIT (decimal digits), DQUOTE (double quote), HEXDIG (hexadecimal digits), LF (line feed), and SP (space). The complete URI syntax is collected in . The HTSQL specification is a URI scheme intended for use over the HTTP protocol . Hence, the productions in the HTSQL syntax follow the generic syntax for uniform resource identifiers. Besides strict compliance with RFC 3986, the design of HTSQL syntax is influenced by several important considerations: the need to represent Unicode characters to permit usage in locales where ASCII is insufficient compatibility with HTML forms and modern browsers, with particular attention to ``form-urlencoded`` enctype as defined by RFC 1866 ability of HTSQL URIs to be embedded in host languages in a natural way, especially a higher level transaction syntax when not conflicting with other goals follow the precedent of SQL-92 syntax and other modern programming languages

Characters not allowed in a particular context can be expressed using an escape sequence consisting of the percent character "%" followed by two hexadecimal digits representing the octet code. For example, "%20" is the escaped encoding for the US-ASCII space character. For the purposes of this specification, we casually assume all hexadecimal digits are upper-case although lower-case or mixed-case is permitted. escaped = "%" HEXDIG HEXDIG

For various reasons (see ), certain characters have been excluded from direct use within a URI. Strictly speaking, these characters must be percent-encoded. However, if the host environment (say, an embedded programming language) allows it may be acceptable to not encode spaces or other key characters. delims = "<" | ">" | "#" | "%" | '"' unwise = "{" | "}" | "|" | "\" | "^" | "`" control = <US-ASCII coded characters 00-1F and 7F hexadecimal> space = <US-ASCII coded character 20 hexadecimal>

To allow entity names from languages not supported by the ASCII character set, HTSQL permits unicode characters using a UTF-8 representation. As traditional to XML and similar serializaiton languages, the actual character ranges allowed are limited to: unicode-xtra = [#xA0-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] unicode-char = unicode-xtra | [#x20-#x9F] | #x9 | #xA | #xD Strictly speaking, one must percent-encode ``unicode-extra`` characters since octets in the range of 80-FF are not included in either the ``reserved`` or ``unreserved`` character set. However, all HTSQL servers should support octets in this range so that percent-encoding is not necessary.

RFC 3986 defines reserved characters as a combination of general delimiters together with scheme specific delimiters. HTSQL obviously builds upon and complies with these productions. reserved = gen-delims / sub-delims gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@" sub-delims = "!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" / "," / ";" / "=" In previous versions of this specification (2396), the right and left square brackets were in the ``unwise`` character set. In the most recent specification, they are only used for IPv6 addressing in the host part of the URI and are not used in the general path or query portions of the URI. One would hope that they could be added to the pchar production in future HTSQL revisions. In HTSQL we use these characters to delimit an array (following SQL-99 syntax) -- although strictly speaking these two characters must be percent-encoded. Another character of interest is the colon, ":", which is permitted in any path segment so long as it isn't the first relative segment. This character is useful in-combination with equality symbol, following the Pascal language's notation for assignment.

An express goal of HTSQL is to be compatible with HTML form submission, in particular the ``form-urlencoded`` enctype first defined in RFC 1866. Query strings constructed with this mechanism percent-encode all characters which are not explicitly unreserved, using "+" to represent the space character (%20). Field submission is then an ordered list of ``name "=" value`` triplets, separated by "&". Hence, the following ``sub-delims`` have specific meaning within HTSQL: reserved-html = "=" / ";" / "&" / "+"

In some cases the distinction between these characters or their percent-encoded variant are not significant, in those cases the following productions are used: char-equals = "%3D" / "=" char-amp = "%26" / "&" char-plus = "%2B" / "+" char-semi = "%3B" / ";"

Besides ``reserved-html`` characters, HTSQL reserves three other ``sub-delims`` with specific meaning. The dollar-sign, "$", is used for variable substitution purposes. The parenthesis "(" and ")", are used to differentiate between literal expressions and computed expressions. reserved-expr = "$" / "(" / ")" / ","

Note that the usage of these three characters is somewhat problematic, in RFC 2396, they are explicitly included in the ``mark`` set of unreserved characters. Thankfully, most browsers do not go out of their way to percent-encode these characters when they are included in a form's ``action`` attribute. char-dollar = "%25" / "$" char-lpar = "%28" / "(" char-rpar = "%29" / ")" char-comma = "%2C" / "," In the cases where the distinction between the literal character and its percent-encoded variant is not important, we use the productions above.

RFC 3986 defines unreserved characters as alpha-numeric plus a few common word connectors. As recommended, the percent-encoded versions of these characters are normalized before processing begins. unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~"

HTSQL goes beyond this set and additionally treats three more characters in the ``sub-delims`` set as unreserved. In many browsers the ``*`` is almost never percent-encoded when it should be, so this simply cannot be used reliably. Furthermore, there doesn't seem to be a pressing need to reserve the exclamation mark. Finally, some browsers tend to always encode the single-quote no matter what the context; rendering its use as a reserved character impratical. unreserved-htsql = unreserved / "'" / "*" / "!" Putting the single-quote in this unreserved class deserves further discussion since it is used to delimit literal strings, and an escape mechanism is needed for single-quotes used within those strings. However, the double-quote, which is used in an analogous way for table and column names has the same problem, and it is "unwise" so it must always be percent-encoded. Following SQL-92's syntax, these delimited strings encode the quote delimiter with two adjacent quote marks. While it may be inconvenient, it is safe and consistent. Please note that while these characters are ``unreserved`` in the sense of RFC 3896, they cannot be used willy-nilly. In most contexts, each of these characters has very specific meanings, regardless of their status of percent-encoded or otherwise.

There are several characters which are either ``unwise``, ``delims``, or ``gen-delims`` sets but are otherwise used by HTSQL. In practice these characters may often be used in a web-environment without percent-encoding, and hence, an HTSQL processor must be able to handle these characters appearing without being percent-encoded. expressions. char-quot = "%22" / '"' char-colon = "%3A" / ":" char-commat = "%40" / "@" char-verbar = "%7C" / "|" char-lt = "%3C" / "<" char-gt = "%3E" / ">" char-div = "%2F" / "/" char-mod = "%25" / "%"

In HTSQL it is not necessary to use all of the ``sub-delims`` which RFC 3986 makes available. In particular, the single-quote, asterix, and exclamation are treated as unreserved characters. reserved-htsql = gen-delims / reserved-html / reserved-expr

Hence, the following productions are used in HTSQL to indicate that non-reserved characters have been decoded (including UTF-8 content): normal-char = unreserved-char | reserved-htsql unreserved-char = (unicode-char - reserved-htsql - "%") | pct-encoded The ``normal-char`` production signals cases where chararacters not in the reserved set are decoded. Hence, "%20" is normalized to " ", however, "%3D" is not normalized to "=" since it is reserved and may be used to delimit components of an HTSQL request. Most productions in this specification are based on normal-char.

The syntatic building blocks for HTSQL expresions are sequences of characters, joined by periods to indicate hierarchy or path navigation. This notation is borrowed directly from RFC 1035's "domain names": the components of a domain name are called labels. In HTSQL, we have two sorts of labels: names and codes. Whereas names refer only to database entities such as tables and columns; codes more generally refer to any value used within a unique or primary key column. Other than having different quoting rules and slightly different syntax constraints, they behave in a similar manner. Broadly speaking, each kind of label (name or code), has both a regular and a delimited form. The regular variant is not only unquoted, but brings with it case-insensitive comparison rules motivated by SQL92's ``regular identifier``. In contrast, the delimited form requires that the value be quoted and further implies literal comparison.

For label comparisons, let us define a normal form with the following operations: - leading and trailing spaces are stripped - the content is converted to a lower-case form - space and dash characters are converted to an underscores - leading 0's are trimmed from all but the last character When comparing labels, if either label was constructed using the "regular" unquoted syntax, then equivalence is defined as equality on their normalized representations. Otherwise, comparison is defined as equality of literal values. Unlike SQL, the space, dash and underscore are collapsed for "regular" comparison purposes: this allows for the common usage of spaces in a mixed-case table-name, yet allowing these tables to be referred to without quoting. In HTSQL, special consideration is given to numeric values which have leading zero's used for padding. Often times the input and output of database are text-based programs which require fixed-width strings; this treatment provides the greatest compatibility with those tools at quite a minimal cost in processing complexity.

A ``specifier`` is a dotted sequence of names which uniquely determine a path in the database structure. The names refer to either tables or columns; the dots represent joins between tables. Each component of a specifier is mandatory; although the last component may be '*' to indicate "all" columns within the given structure. specifier = spec-wild | name ("." name)* ("." spec-wild)? spec-wild = "*"

The components within a specifier are called names; these are labels for tables, columns, and other database entities. There are three syntax forms for an HTSQL name: name = regular-name / delimited-name / special-name The "regular" form uses case-insensitive comparison with a limited character-set, while the "delimited" variant is universal but requires special quoting. Special names are meant for HTSQL and proprietary extensions which are not necessarly handled via the normal interpretation rules.

The ``regular-name`` production reflects ``regular identifier`` in the SQL-92 specification. This syntax requires that the identifier start with a letter, provides the underscore for a word separator, and defines case-insensitive comparison. The HTSQL definition extends this notion by additionally allowing Unicode characters: regular-name = regular-name-start *regular-name-char regular-name-start = ALPHA / unicode-extra regular-name-char = ALPHA / DIGIT / "_" / unicode-extra Note that this production is awfully close to the ``unreserved`` production of RFC 3968, less the dash "-", period ".", and tilde "~". This is intentional as these other characters, while they are not reserved, are given specific meanings in HTSQL. If a database object name must include one of these characters, the delimted form is needed.

The ``delimited-name`` production is a catch-all mechanism for names that don't fit into the prior category; it is motivated by the ``delimited-identifier`` production in the SQL-92 specification, using the double-quote for a delimiter. Including a double-quote character in a delimited name is done with two adjacent double-quotes (and not with a C-style \" escape). delimited-name = DQUOTE delim-name-char+ DQUOTE delim-name-char = unreserved-char - DQUOTE | DQUOTE DQUOTE | "+" Within a delimited name, all ``reserved`` characters are forbidden, with the exception of the ``+`` symbol, which is treated as a space character (%20). The use of the double-quote in this manner requires comment. This character is not not a RFC 3986 "reserved" character, neither is it "unreserved". In prior URI specifications it is listed as "unwise". Depending on the context in which an HTSQL URI is used, the double-quote character may need to be percent-encoded (%22). This character was chosen since it directly maps onto SQL's syntax, and since the single-quote (a reserved character) is needed to delimit literal scalar values. Since the usage of this form should be relatively uncommon, this should not be a huge complication.

The ``special-name`` production is meant to provide an extension mechanism for HTSQL URIs. Following the example of the Python programming language, where a leading underscore indicates that a name is given special treatment: special-name = metadata-name / reserved-name / extension-name metadata-name = "_" extension-name = "_" regular-name reserved-name = "__" regular-name As in the ``regular-name`` production above, special names are compared in a case-insensitive manner. Names beginning with a double underscore are reserved for HTSQL's usage, while those starting with a single underscore are encouraged for extension modules. The single underscore is allocated for a top-level entity like SQL92's ``INFORMATION_SCHEMA``. It is recommended that organizations that wish to augment HTSQL with proprietary extensions should use an extension-name with a reverse domain name style package name, ``_com_clarkevans_myextension", to avoid naming conflicts.

An ``identifier`` is a dotted sequence of components, with a correspondence to the columns in the table's primary key. Each component is either a ``code`` or, recursively, an ``identifier``. identifier = ident-regular | ident-paired ident-paired = (identifier '.') + code ident-regular = ident-component ('.' ident-component)* ident-component = '(' identifier ')' | code

Codes are specific labels intended to match against unique or primary key column values; there are three syntax forms for an HTSQL code: code = regular-code / delimited-code / array-code Like the ``name`` productions, the "regular" form uses case-insensitive comparison with a limited character-set, while the "delimited" variant requires quoting.

The ``regular-code`` reflects the ``label`` production of 1035, but extends the notation to permit numbers and allowing the underscore synonymously with the dash as a word separator; like the ``name`` production, Unicode characters are also granted. regular-code-char = ALPHA / DIGIT / unicode-extra / "-" / "_" regular-code = 1*regular-code-char This range explicitly excludes all unprintable and "reserved" characters from RFC 3986 ("Uniform Resource Identifiers"). Furthermore, this range does not include the period "." or tilde "~" as these characters have explicit meaning in HTSQL and are particularly problematic since they cannot be escaped via percent-encoding. Note that any value matching the ``regular-name`` production also matches ``regular-code``

The ``delimited-code`` reflects the ``string literal`` production in the SQL92 specification and is a catch-all for codes which do not fit into the prior category. It is identical to the HTSQL ``literal`` production using the single-quote character for its delimiter. delimted-code = literal

The ``array-code`` production meets the requirement of SQL99's ARRAY data-type, which could possibly be used as foreign or unique key column. It's definition is straight-forward, permitting recursion and insignificant whitespace: array-xtra = iwsp "," iwsp code array-code = "[" iwsp code *array-xtra "]" The use of the right and left brackets requires commentary. These two characters are neither "reserved" nor "unreserved" according to RFC 3986; and are listed as "unwise" in prior specifications. Depending on the context in which the HTSQL URI is used, these characters might need to be percent encoded, %5B for the left bracket and %5D for the right bracket. In practice, this is not usually a problem.

The construction of an identifier is non-trivial, but intuitive once you get the hang of it. Primary key column values (aka codes) that are not used to refer to another row in the system are included directly in the identifier. If columns in a table's primary key participate in a foreign key such that the target columns are exactly a unique key of the target table, then those columns collectively are represented by a single ``identifier`` as defined by the referenced table. Note that the actual column values need not be included in the referenced identifier; for example, a sequence number may be a unique key of a parent table, referenced in the primary key of the child table, yet, not included in the parent table's primary key. The position of this identifier is the position of the first column appearing in the foreign key constraint. If the column is a SQL99 ``REF`` data-type or the HTSQL SQL92 compatible equivalent, then the column is represented by an identifier of the referenced table. The actual values of these ``REF`` data types are internal and should never be accessable via the HTSQL interface. If these columns are selected, then the corresponding identifier of the referenced object should be returned as the column value. When an ``identifier`` occures as a component in another identifier, it is serialized using parenthesis to indicate boundary. Parenthesis may be omitted in one of two cases: when the referenced table has an identifier with exactly one code, and when the current table's identifier is composed of exactly one identifier (from a parent table) and a code. Segments are given the same order in the identifier as the corresponding columns are ordered in the table's primary key. The canonical form of an identifier always uses parenthesis and delimited codes. When used in matching contexts, each component of an identifier may be replaced with a wild-card component, ``*``, which matches any corresponding value within the database. In all other cases, the ``*`` value is forbidden since it is not a valid ``code``.

This section contains other low-level productions for variables, string literals, etc.

The variable construct in HTSQL can be used to break complicated logical steps into more managable components. Variable substution is done in a the HTSQL processor and is not passed on to the underlying database.

Variable declaration is done using the ":=" assignment operator. A variant of this operator, ":@=" is used when the right-hand-side should be treated as an evaluated expression. Since both ":" and "@" are reserved characters, they may be, but are not required to be percent-encoded. To permit variable declarations within a HTML form, the dollar sign may also be percent-coded as required. var-decl = char-dollar regular-name (var-decl-norm / var-decl-eval) var-decl-norm = char-colon "=" expr-norm var-decl-eval = char-colon char-commat "=" expr-eval Note that ``expr-eval`` is terminated by the end of request, a reserved character such as the semi-colon ";", parenthesis ")" or amperstand "&".

Variables can be used as expressions directly delimited by a space character. They can also be used *within* a any quoted value, given that the dollar-sign may be percent-encoded if this is not the intended outcome; however, in this case, parenthesis are use to futher delimit the scope of the variable. var-expr = char-dollar regular-name var-subst = "$(" regular-name ")" Note that in the latter case, only the reserved form of the character and surrounding parenthesis is admitted. Variables can be declared in either the current HTSQL request, or by the enclosing environment. Unresolved variables are an error.

In HTSQL numeric values can be expressed without any special presentation. However, regular character literals must either be quoted or put into a context bounded by reserved characters. expr-literal = quoted-literal / numeric-literal

Numbers, including the decimal point and sign, may be included without escaping in all HTSQL contexts. This is your standard definition, including the exponent. numeric-literal = DIGIT+ numeric-fraction? numeric-exponent? numeric-exponent = ( "e" / "E" ) numeric-sign? DIGIT+ numeric-fraction = "." DIGIT+ numeric-sign = char-plus / "-"

These sorts of literals are found on the right-hand-side of a comparison expression, they are percent-encoded values that end at the first reserved character (excepting "+" and "$" as detailed below). plain-literal = literal-char* literal-char = unreserved-char | "+" | var-replace Within a character literal, a plus sign is treated as a space character (the actual plus sign is percent-encoded). Furthermore, variable substutiton is performed with character sequences such as ``$(variable)``.

In some circumstances when a plain-literal is not permitted a quoted literal form is allowed. The syntax of a quoted literal followes the precedent of SQL-92 where literal values are delimited by a single-quote character (%27). Within a quoted literal, a single-quote can be represented using two adjacent single-quotes in a manner similar to SQL-92. Note that the single-quote is treated as an unreserved character in HTSQL even though it is included in the ``sub-delims`` production. quoted-literal-char = literal-char - "'" | "'" "'" quoted-literal = "'" quoted-literal-char* "'" In a manner similar to plain literals, the "+" symbol is treated as a space, and variable substutiton is done with sequences like ``$(variable)``.

By comparison, we mean expressions which take arbitrarily typed values and return a boolean result. Comparison expressions in HTSQL substantially differ from their corresponding SQL-92 syntax, there are ``evaluated`` and ``normal`` comparison forms: comparison = expression iwsp (comp-eval / comp-norm)

The intent of the evaluated comparison form is to permit HTSQL syntax to be used in an HTML query form. The value provided to the HTSQL server is percent-encoded, and it is the expectation that once this is decoded, the result matches the ``expression`` production for the given context. comp-eval = char-commat comp-oper eval(plain-literal) For example, ``?ship_date@=purchase_date`` would result in a comparison of the "ship_date" column with the "purchase_date" column. An example with function invocation might be ``?ship_date@=today%28%29``, which represents a user typing in ``today()`` in a HTML field with name ``ship_date@``. If the resulting expression is invalid, it is an error. It should be noted that the at-sign, "@", is a ``gen-delims`` reserved character so it may or may not be percent-encoded, either form should work for this production. Note that the un-encoded form is safe to use in any place where this production would appear in an HTSQL URI. The reason this is reserved character is to delimit user-name in the authority component of the HTTP request.

While the above comparison form gives the most flexibility, users more commonly enter data they expect to be treated literally. Further, we wish to allow function calls on the URI line without requring them to be percent-encoded. This is handled with the following production: comp-norm = comp-oper comp-rhs ("," comp-rhs)* comp-rhs = expr-???? iwsp / plain-literal The problem with the ``comp-rhs`` production is that it is ambiguous; however, we mean to choose the first alternative ``expr-????`` first, if it fails, then use ``plain-literal`` which always succeeds.

Comparisons are complicated by a historical fact: HTTP queries via the GET method are already quite well defined. When submitting a form, the right-hand-side of an equality operator is percent-encoded, substituting a "+" in lieu of "%20" for readability. This syntax is both well suported and intutive: so regardless of its divergence from SQL, HTSQL supports it directly. For example, a query fragment such as ``?department=math`` treats the left-hand-side as a specifier for the column "department", and the right-hand-side as a literal value, 'math'. This behavior differs subtantially from SQL, where both sides would name columns. While this may seem problematic at first, it actually fits the common and generally accepted usage pattern. Most of the time when columns are being compared in SQL, they are part of a join clause and these sorts of comparisons are automated by HTSQL. The remaining cases when one wishes to compare one column to another are far less common. While the SQL syntax is more general, this historical pattern is more common.

In order to facilitate a more full comparison mechanism we must signal that the right-hand-side is not a ``plain-literal``. Luckily, this is possible in HTSQL since the parenthesis are reserved ``sub-delims`` and thus they will be percent-encoded if they appear in a plain literal. Further, since the ``expression`` production allows any expression to be enclosed in parenthesis, we have a clean mechanism for distinguishing between these two cases. For example, the query fragment ``?purchase_date=(ship_date)`` will treat the right-hand-side as a specifier for the "ship_date" column. The use of a function will also trigger this behavior, for example, ``?name=lower(name)`` will return all rows where the "name" column is lower case. Similarly, since the dollar-sign is reserved, variable references can be used, as well as expressions based on variables. While the balance here is somewhat awkward, it supports the common cases with the least amount of characters and it makes the less common cases possible, if not straight-forward.

For comparison operators, the right-hand-side can optionally be a list of values. This makes checking for inclusion a handy operation. For example, ``?department=math,english`` will result in an SQL fragment similar to ``"department" IN ('math','english')``. The actual meaning of the inclusion list depends upon the particular operator; but it is intended to be an alternation where the left-hand-side need match only one of the options on the right.

Comparison operators are restricted to three cases: (a) where the right-most character is reserved, or where (b) the current character is a "modifier" and hence we know the operator is not finished. An exhaustive list of supported comparison operators: comp-oper = ">=" / "<=" / "==" / "~=" / "~" / "!>=" / "!<=" / "!=" / "!~=" / "!~" / "<" / ">" / "=" comp-norm = comp-oper comp-rhs ("," comp-rhs)* comp-rhs = plain-literal / iwsp expression iwsp In the above listing, "==" means equality, while "!=" means inequality. The greater and less than symbols have their usual meaning, as well as ">=" and "<=" which include the end-points in the comparison. The "!<=" operator is equivalent to ">", only that it can be used in an HTML form. For example, "?column>value" can be encoded using an input name of "column!<". During form submission, the equal-sign is appended giving "column!<=value". While it is ugly, it works well and the user only sees the input box and what ever label is attached to it. The tilde "~" indicates a POSIX regular expression. If the underlying database does not support regular expressions using the POSIX syntax, it should convert the value (if possible) to an equivalent LIKE or SIMILAR expression. Direct support for the LIKE operator isn't that helpful given the standardization and power of regular expressions.

The plain equal-sign "=" all by itself has special meaning in HTSQL, the expression on the left-hand-side and right-hand-side are compared after the following transformation: COALESCE(NULLIF( TRANSLATE(LOWER( TRIM(LEADING '0' FROM TRIM(BOTH ' ' FROM CAST(expression AS TEXT) ))), '- ','__'), ''),'0') While this pattern is a bit esoteric, it matches typical expectations of a case-insensitive match which ignores leading 0's and treats dashes, underscores and spaces as equivalent. If the left-hand-side is a literal, it can be prepared by the HTSQL translator; if searches on a particular column become common, the result of this transform can be indexed to speed up performance. This particular transform is quite useful for comparison of "codes", an extremely common use case.

In HTSQL, expressions support functions, together with unary and binary operators. Expressions are recursively defined: expression = function / expr-literal / specifier / expr-paren / / expression expr-continue expr-continue = expr-cont-numeric expr-compound = expression expr-comp-numeric / expression = expr-paren / expr-predicate / expr-numeric / specifier / quoted-literal / function / expr-comp expr-comp = "(" iwsp ( expression iwsp / comparison ) / ")" expr-paren = "(" iwsp expression iwsp ")" Expostssions are complicated by comparisons which terminate only with an unreserved character and hence must be enclosed with parenthesis in its common form.

HTSQL uses not only symbolic operators common in programming languages, but also text versions of the same operators. To use the text versions they must be followed by and/or preceded with either whitespace or parenthesized expressions. expr-rhs = wspc+ expression / iwsp expr-paren / iwsp expr-comp expr-lhs = expression wspc+ / expr-paren iwsp / expr-comp iwsp

Logical expressions are comparisons and other expressions that are joined logical conjunction, disjunction, and negation. Both symbolic and named version of these operators are provided. expr-predicate = expr-negation / expr-alternation / expr-conjunction expr-negation = "!" iwsp expression | "not" iwsp expr-rhs expr-conjunction = comparison char-amp iwsp expression | expression iwsp char-amp iwsp expression | comparison char-amp comparison EOS | expression iwsp char-amp comparison EOS | expr-lhs "and" expr-rhs expr-alternation = expression iwsp char-amp iwsp expression | expr-lhs "or" expr-rhs Note that the conjunction operator permits a unparenthesized comparison on its left-hand-side; this is permitted since "&" is a reserved character and thus does not introduce ambiguity. The unparenthesized comparison is also permitted on the right-hand-side via the end of stream or segment EOS, which matches but does not consume "/" or ";".

Conjunction of numeric values by an operator follows the SQL-92 syntax modified for URLs. Since the plus "+" character is a ``sub-delims`` reserved character, we allow for its literal inclusion and also for percent-encoding. Since the soldius "/" character is a ``gen-delims`` it must either be percent-encoded or, "div" may be used in its place. expr-numeric = numeric-literal / expr-negation / expr-expr-addition / expr-subtraction / expr-division / expr-multiplication / expr-modulo expr-negation = "-" expression expr-subtraction = expression iwsp "-" iwsp expression | expr-lhs "less" expr-rhs expr-addition = expression iwsp "+" iwsp expression | expr-lhs "plus" expr-rhs expr-multiplication = expression iwsp "*" iwsp expression | expr-lhs "times" expr-rhs expr-division = expression iwsp "%2F" iwsp expression | expr-lhs "div" expr-rhs expr-modulo = expression iwsp "%25" iwsp expression | expr-lhs "mod" expr-rhs Even though the SQL-92 specification does not include the modulo operator, it is provided here since many database implementations provide for this functionality. It should be used only when needed since an underlying database may not support its usage. Operator precedence, of course, follows SQL-92; left-to-right evaluation with multiplication and division binding more tightly than addition and subtraction. These constructs are directly translated into SQL, there is no special handling done by HTSQL.

Functions in HTSQL do not follow SQL-92 syntax, which have optional parameters and a sort-of keyword argument mechanism. Instead, HTSQL functions are modeled more after Python and other languages which have by-position and keyword arguments. The value arguments of a function call is a normal expressions - that is, they are treated as specifiers and not as literal values unless single quoted. function = regular-name iwsp "(" iwsp func-args? iwsp ")" func-pair = regular-name iwsp "=" expression func-item = func-pair / expression func-args = func-item ( iwsp "," iwsp func-item )* Note that the equal sign, parenthesis, and comma are reserved ``sub-delims`` charaters and that the percent-encoded variants are not delimiters in these productions (and indeed must be percent-encoded if they occur in a value expression). In some cases, a particular function may optionally treat particular ``specifier`` arguments as values in a pre-defined enumeration rather than as a column specifier. For example, the ``trim`` function may use an argument value of ``both`` not as a column specifier, but rather as a flag to trim spaces on both sides of the given value.

HTSQL requests are a specialization of RFC 3986's ``hier-part`` and ``query`` assuming that relative resolution has already been performed.

At the top level, a request can either be a user defined resource, or an HTSQL action. In an HTSQL context, either the slash "/" or the semi-colon ";" may be used to separate path segments. An HTSQL system treats them as synonymously. request = "/" context? ( user-resource / action query? ) context = schema-name (separator segment)* separator separator = "/" / ";" schema-name = name The ``schema-name`` production is simply a ``name`` which refers to a schema in the underlying database system.

While HTSQL provides access to a database, many static or semi-static resources are requried for screens, and other purposes. Anywhere within an HTSQL request, a path segment that begins with the tilde "~" indicates a resource the meaning of which is out-of-scope of this specification.

The user resource production must exactly follow RFC 3986; in particular, the ``path-rootless`` and ``query`` productions are used here. user-resource = "~" rfc3986-path-rootless ("?" rfc3986-query)? Note that a user resource may be contextualized, that is, they can be prefixed with a schema, tables, and other parameters. It is intended that user-resources may be dependent upon not only preceding content but the meta-data associated with this information.

The query fragment in HTSQL are expressions with a boolean value, alternatively permitting variable declarations. Note that query fragments are either terminated with a reserved character, or the end of the request. query-part = comparison / expr-predicate / var-decl query = query-part (";" query-part)* Query components are resolved with respect to the left-most table mentioned in the request's path segments.

Path segments are either query components included within the actual request path, or they are table names and column specifiers. segment = query / table-name selector? picker? selector = iwsp "(" iwsp expression (iwsp "," iwsp expression)* ")" picker = "!" identifier

This chapter describes the translations between HTSQL and the resulting SQL queries. The result of an HTSQL query is a flat list, having at most one row from each member of the right-most context. This result set is filtered in several ways. A set of links from the driving class outward, as part of the filter is used to restrict values.

...

/-/schema/select.csv This item should list all of the schemas in the system, returned as a comma separated variable file. In the result set the first row is always a unique identifier (via dotted notation) for the given row. /-/table(schema,name,major,minor)/select.csv This item returns all of the tablees that exist. Note that parent, minor, and code all refer to columns which are role type references. While there is duplication between the identifier (first column) this will help make things easier; also, by default data is sorted according to the identifier. /-/table()/select.csv?title=ick&~title=ick The columns selected can be indicated by listing them within parenthesis. By default, if the selector is missing, or if it just contains '*', all of the colums are returned. In this case, the selector is empty, so only the identifier is returned. /-/table(title,memo)/select.csv?title=&~title= In this instance, the 'title' and 'memo' columns are selected, along with the default indicator column for each row. /-/table(schema.title,major.title)/select.csv It is possible, using a dotted notation, to ask for columns from other tablees referenced. The only columns returned are ones that are joined to; note that a row is returned even if one of the selected values does not correspond to an entity. In this case, the schema title, 'Administrative' is returned for each row; along with the name and title of the parent table. Since HTSQL allows for two parents, they are named parent and minor; where the minor parent comes second. /-/table/select.csv?name=resource This item filters the table list looking for a name with the value 'resource'. By default, the filter applies to the right most table, that is, it will not filter for a schema called 'resource'. /-/table/select.csv?name=resource&~name=&title=&~title= This item filters the table list looking for a name with the value 'resource'. By default, the filter applies to the right most table, that is, it will not filter for a schema called 'resource'. /-/table;name=resource/select.csv The same query can be encoded using a path argument, as shown above. The reasoning for path arguments is that they are survive relative URL cross links and query arguments. /-/schema;name=-/table/name=resource/select.csv While the previous query returns only a single row now, if another schema was added, and a resource table created within that schema, it would also be returned. To limit the result to only the meta-data schema, another path segment is required. /-/schema!-/table/!resource/select.csv Since 'name' is the code column for both the 'table' and 'schema' tablees, the short-cut above can be used. This allows for smaller urls, but has no other effect, in fact, the query plan after the parse is identical to the one above. /-/table/select.csv?title~=C A regular expression can be provided in a query argument using the tilde (~). For example, this query returns all tablees matching the regular expression 'C'. /-/table/select.csv?name=cHoice+ Since the 'name' column of the table 'table' is a name, it is normalized using the name processing rules. In particular, it is made into a small case variant before comparision. Also, note that he extra (non-compliant) space character was stripped. This is done to make things user friendly in forms. /-/table/select.csv?name=choice&name=resource If a query argument is mentioned twice, it is treated as an alternation; thus, the query above returns both choice and view. /-/table/select.csv?name=choice,resource The alternation can, in many cases by providing comma separated list. This should work with codes, names, choose, and numbers. /-/table/select.csv?name!=choice,grant,group,schema,user-security, It is possible to negate the equal operator, by using '!' immediately before the equal sign. Furthermore, you can have a trailing comma. /-/table/select.csv?name=choice,resource&title!~=C When predicates do not share the same code/operator, the result is the intersection of the two result sets. /-/column(kind,lock,memo,title)/select.csv It is possible to only return particular columns. In the above query, all columns are returned, but only the ones mentioned. In particular the name and table columns are omitted since they are largely redundant with the @identifier column. The first column is the dotted identifier for each row. /-/column(column-text.*)/select.csv?column-text.length!= When a column is compared != blank, then it is checked to see that the value is provided. /-/column(title,column-display.sort)/select.csv?kind=code It is also possible to select columns from a main together with columns from a particular facet. Here the 'sort' column is returned along with the title. To keep the results short, we only show 'code' columns. /-/column(kind)/select.csv?column-list!= This returns all columns where there exists a column-list facet; as expected this is a list of 'list' kind columns. /-/table!-.column/column(kind)/select.csv?column-list= This returns all columns where there does not exist a column-length facet. /-/table!-.column/column(title)/select.csv?column-text.length= Return all columns where the column-text.length is blank, in this case, the 'title' column is omitted. /-/column(column-link.target.title)/select.csv This returns all columns that reference the 'column' table, and returns the title of that table, along with the title of the given column. The identifier is also returned in the first column. /-/column(*,column-link.*)!-.table.major/select.yml This returns all of the fields for the column, and its associated column link. It only returns one row '-.table.major' and pivots the result for YAML printing. /-/column(*,column-link.*)!-.table.major/select.xml?*strip=t This is an equivalent query, only returning the results in a XML format. The XML version has several additional columns which are there for XSLT, since that language is brain dead when doing string manipulations. /-/column(*,column-link.*)!-.table.major/select.xls?*strip=t This is similar, only that it returns a brain-dead HTML version with an Excel MIME type, to fool Excel into thinking it is an Excel spreadsheet. /-/column(*,column-link.*)!-.table.major/select.js?*strip=t This is an equivalent query, only returning the results in a format that you can 'eval' in Javascript. /-/column(*,column-link.target.*)!-.table.major/select.yml?*strip=t This this is similar to the above query, only that goes further and returns the columns fo the column-link target. The strip directive leaves out columns which are blank.

/-/group!root/grant/select.csv This retruns all grants cross-linked to the 'root' group. /-/schema!-/grouplink(schema)/select.csv This is a cross link filter, basically, only those group links related to the '_' schema are returned. /-/list!-.kind/column-list(column,list)/select.csv Also a cross link filter, in this case, only column-list items that use a particular list are returned. An open question is if this should work across facets, that is, if /column/ were selected, if it should return only those which have a facet anyway linked to the particular object? I think so, but it is not implemented. /-/column(title)/select.csv?column-link.target=-.column This returns the title of all columns in the '_' schema that reference the 'column' table. Note that the 'target' argument is actually a reference to an object, therefore, an identifier is required. Furthermore, this column is found in the column-link facet, so some extra leg-work is required. /-/schema!-/column(title)/select.csv?column-link.target=column It is possible to provide a relative identifier, in this case the 'column' table requires a schema parent; in this case, the particular schema, '_' is obtained from the schema context in the path. The result is identical. /-/schema!-/column(title);column-link.target=column/select.csv Of course, this mechanism is not just for query arguments, but can also be used in filters. /-/column-link(target)/select.csv?target=-.column&target=-.table For select operations, more than one value for a given field can be provided. In this case, the logic is considered to be an alternation. /-/column-link(target)/select.csv?target=-.column,-.table This same result can be obtained by using a comma to separate a list of identifiers.

/-/schema/insert?name=tst&title=Test+Database This request creates the 'tst' schema with the title of 'Test Database'; once the schema is created, it redirects to the newly constructed reference. /-/schema!tst/table!organization/insert?minor=code This request creates a 'organization' table in the 'tst' schema using a default title of 'Organization'. It specifies a primary code named 'code', since a parent is not provided, it is assumed that this is a root table type. /-/schema!tst/table!organization/column!title/insert This creates a column for the title of the organization. By default, the kind of a column is 'text' and it takes the default 60 characters wide, which is the usual CRT width. /-/schema!tst/table!organization/column!ein/insert This creates a column for the government employement identifier for the organization. This column is not the default, instead it is 20 characters. This is also a 'text' (default) column. /-/table!tst.organization/column!memo/insert This creates a memo column for the organization, by default the name 'memo' implies a kind of 'memo'. On the screen, memos are presented as scrollable text areas. /-/schema!tst/column(kind,title,column-text.length)/select.csv Note that the default field 'kind' is assumed based on the name of the field if not provided. /-/column!tst.organization.ein/delete It is possible to delete columns that are not needed. In the back-end system, the column is actually just renamed "deleted" with a timestamp so that data is preserved. /-/schema!tst/column-display(sort)/select.csv In a similar way, the default sort order for the column display is set according to the kind, if it is not provided. Note that the 'ein' record for the column-sort is also deleted. /tst/organization/insert?title=Willy+Wanka+Chocolate+Factory This inserts the first organization into the list. Since the code is missing, it is auto-numbered as '1'. /tst/organization/insert?code=magic&title=The+Thomas+Magic+Shop This organization has a specific code, 'magic', that is used, in addition to the given title. /tst/organization/insert?code=J-h-D&title=Jasper+House+Of+Debauchery Codes are case preserving, but otherwise insensitive. Illegal characters in a code are converted into an underscore. The dash is perfectly OK for a code. /tst/organization!1/insert?title=This+Is+A+Duplicate+Error Duplicate rows are detected, and 409, aka 'Conflict' is returned when an attempt to add a duplicate is made. /tst/organization!1/insert?title=Willy+Wanka+Chocolate+Factory It is possible to re-use a code once it is deleted. /tst/organization!1/insert?title=This+Is+A+Duplicate+Error But, just beacuse something was deleted, and then re-inserted, does not mean that duplicates are allowed. /tst/organization!J-h-D/update?code=J-H One can rename an object at any time simply by assigning it a new code. /tst/organization!J-H/update?code=A123456789012345678901234567890XX Codes are limited to 32 characters /tst/organization!J-H/update?code=A123456789012345678901234567890X This one just barely fits. /tst/organization/insert?code=0 Since codes are trimmed of leading 0s, a code with just 0s is an error. /tst/organization!A123456789012345678901234567890XXX/update?code=Jh Codes are not truncated when doing comparisons, the check just fails.

/-/schema!tst/table!staff/insert?minor=username This request creates a 'staff' table in the 'tst' schema using a default title of 'Staff'. It specifies a primary code named 'username', since a parent is not provided, it is assumed that this is a root table type. /-/schema!tst/table!staff/column!given/insert?title=First+Name This creates a a column for the given (first) name of the staff, by default the kind of columns are text and they are 60 characters wide, which is the usual CRT width. /-/schema!tst/table!staff/column/insert This creates a a column for the surname (last name) of an individual, note that the sort is set at 4 rather than the default of 5. This should make this column show up earlier in screens and grids. In this example, we use a args code in the test file so that the arguments can be more clearly seen without mudging. /-/schema!tst/table!staff/column/insert?name=memo This creates a 'memo' column for the staff table. Note that the name of the column is a well known kind, then this is set automatically. This works nicely for 'sort'. /-/schema!tst/table!staff/column/insert?name=sort If there is a 'sort' column, then this becomes an override default sort (typically the code column is the default sort). /-/schema!tst/table!private/insert?major=staff A facet table is one where there is only a primary code that points to its parent. In this case, the 'private' table is created to hold restricted information that should not be publicly available. /-/schema!tst/table!private/column!rate/insert?kind=float Create a floating point value containing the hourly rate for a given individual. /-/schema!tst/table!private/column/insert Create a social security column for the 'private' facet of the 'staff' table. This is a code (length=13). /-/schema!tst/column(kind,title,column-display.sort)/select.csv Note that the default field 'kind' is assumed based on the name of the field if not provided, and that the sort is also provided when not specified. /tst/staff/insert This inserts an object into the system, but it does not provide information for the facet (even though one exists). The facet should not be created for this specific object. /tst/staff/insert This tests an insert using facets, in this case social security and payrate. /tst/staff/insert This individual will get a facet row, but not yet. This also tests the redirect mechansim, where !! is replaced with the current identifier. /tst/private/select.csv Since there is only one individual with a private facet, the private table has exactly one item. /tst/staff!gre/update?private.rate=7.95&surname=Barns This update changes several fields, and inserts a new row into the private facet. /tst/staff!znt/private/delete Delete the private information about the staff 'znt'. /tst/staff!hw/delete When deleting an entire row, all associated facets are dropped.

/-/schema!tst/table!project/insert?major=organization&minor=code This creates a project table, which has organization as its parent.

/-/schema!tst/table!member/insert This creates a table recording a subset of the cross-product between project and staff.

/-/column!tst.project.manager/insert?column-link.target=member This creates a role link from a project one of its members. /-/schema!tst/table!address/insert?major=organization&minor=handle This creates a 'address' subordinate table which is used to hold address, phone and other information which varys per location. For example, a person may have an office, home, and perhaps a laboratory; each with different address details. The primary code for the address is called 'handle'. /-/column!tst.organization.address/insert This creates another 'role' link, this time between an top table and a child table. This is the primary address for the organization.

/-/schema!tst/table!staff/column!report-to/insert This creates a 'report-to' column which happens to be a reference to another staff in the system. If a target is provided, it must name a table, and the default kind of column is 'peer'.

/-/schema!tst/table!timeslip/insert?major=staff&minor=num For this project membership, there could be several timesips, recording a particular amount of work done on the project. /-/schema!tst/list!billable/insert This creates a drop-down list of billable items, in particular -- Yes, Unsure, No. /-/table!tst.timeslip/column!billable/insert This request creates a drop-down choice list for a timeslip. /-/table!tst.staff/column!type/insert?column-list.list=stafftype This request creates a drop-down choice list for a timeslip. /-/table!tst.timeslip/column!reviewer/insert?column-link.target=staff This is an example cross-reference where a reviewer can be assigned to a particular timeslip item, as a signal that they have approved the work.

/-/schema!tst/table()/select.csv Ok. Let us verify that everything is there.

/-/choice!tst.stafftype.coNtractor/update?name=conTract Change the code for a drop down list and verify that it propagages.