Cross-pollination #1: Speccing data

Clojure

A functional dialect of Lisp

Compiles to JVM bytecode

...and JavaScript (via the ClojureScript variant)

EDN

Namespaced keywords

{:person/id 1
 :person/first-name "John"
 :person/last-name "Doe"
 :person/favourite-meetups #{"WarsawJS"}
 :person/passport {:passport/number "PI 3141592"
                   :passport/issued-at #inst "2022-03-01"
                   :passport/valid-until #inst "2032-03-01"}}
        

spec·​i·​fy

transitive verb

to name or state explicitly or in detail

spec·​i·​fi·​ca·​tion

noun

a detailed precise presentation of something or of a plan or proposal for something

spec = specification = schema

What should the data look like?

Why write schemas?

• Validation
  Is the data what it should be?
• Error reporting
  If it isn’t, what exactly is wrong with it?
• Conformance (≈ parsing)
  Given a spec and data, construct the domain object(s) representing the data
• Data generation
  Given this spec, what sample data conforms to it?
• Documentation
  As a human reader, what can I learn about these data?

Type systems as schemas

type Passport = {
    number: string;
    issuedAt: Date;
    validUntil: Date;
};

type Person = {
    id: number;
    firstName: string;
    lastName: string;
    favouriteMeetups?: [string];
    passport: Passport;
};
            

Data as documents: Schemas for XML

DTD, XML Schema, RELAX-NG…

<!DOCTYPE person [
   <!ELEMENT person (id, firstName, lastName,
                     favouriteMeetups?, passport)>
   <!ELEMENT id (#PCDATA)>
   <!ELEMENT firstName (#PCDATA)>
   <!ELEMENT lastName (#PCDATA)>
   <!ELEMENT favouriteMeetups (meetup*)>
   <!ELEMENT meetup (#PCDATA)>
   <!ELEMENT passport (number, issuedAt, validUntil)>
   <!ELEMENT number (#PCDATA)>
   <!ELEMENT issuedAt (#PCDATA)>
   <!ELEMENT validUntil (#PCDATA)>
]>
        

JSON Schema

{
  "title": "Person",
  "type": "object",
  "properties": {
    "id": {
      "type": "integer"
    },
    "first_name": {
      "type": "string"
    },
    "last_name": {
      "type": "string"
    },
    "favourite_meetups": {
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "passport": {
      "$ref": "#/$defs/passport"
    }
  },
  "$defs": {
    "passport": {
      "type": "object",
      "properties": {
        "number": {
          "type": "string"
        },
        "issued_at": {
          "type": "string",
          "format": "date"
        },
        "valid_until": {
          "type": "string",
          "format": "date"
        }
      }
    }
  }
}
        

• Somewhat baroque
• Itself a JSON
• Multiple implementations
• Public repository of schemas: schemastore.org
• Somewhat extensible
• Still can't add custom constraints

clojure.spec

Composable primitives

(require '[clojure.spec.alpha :as s])

(s/def :person/id int?)
(s/def :person/first-name string?)
(s/def :person/last-name string?)
(s/def :meetup/name string?)
(s/def :person/favourite-meetups
  (s/coll-of :meetup/name))
(s/def :passport/number string?)
(s/def :passport/issued-at inst?)
(s/def :passport/valid-until inst?)
            

clojure.spec

Maps as open collections of keys

(defn valid-dates?
  [{:passport/keys [issued-at valid-until]}]
  (neg? (compare issued-at valid-until)))

(s/def :warsawjs/person
  (s/keys
   :req [:person/id
         :person/first-name
         :person/last-name
         :person/passport]
   :opt [:person/favourite-meetups]))

(s/def :person/passport
  (s/and
   (s/keys
    :req [:passport/number
          :passport/issued-at
          :passport/valid-until])
   valid-dates?))
            

Using specs: Validation

Is the data what it should be?

(def sample-person
  {:person/id "1"
   :person/first-name "John"
   :person/last-name "Doe"
   :person/favourite-meetups #{"WarsawJS"}
   :person/passport
   {:passport/number "PI 3141592"
    :passport/issued-at #inst "2022-03-01"
    :passport/valid-until #inst "2021-03-01"}})

(s/valid? :warsawjs/person sample-person)
;=> false
        

Using specs: Error reporting

If it isn’t, what exactly is wrong with it?

(s/explain :warsawjs/person sample-person)

;; prints:
"1" - failed: int?
          in: [:person/id]
          at: [:person/id]
        spec: :person/id
sample-person - failed: valid-dates?
                    in: [:person/passport]
                    at: [:person/passport]
                  spec: :person/passport
        

Using specs: Data generation

Given this spec, what sample data conforms to it?

(require '[clojure.spec.gen.alpha :as gen])

(gen/sample (s/gen :warsawjs/person) 1)

;; prints:
({:person/favourite-meetups ["" "" "" "" ""],
  :person/id 0,
  :person/first-name "",
  :person/last-name "",
  :person/passport
  {:passport/number "84g",
   :passport/issued-at #inst "1969-12-31T23:59:59.995-00:00",
   :passport/valid-until #inst "1969-12-31T23:59:59.999-00:00"}})
        

Applications

• Annotate inputs/outputs to functions
• Instrument functions with spec checking
• Generate tests
• Validate data at (sub)systems boundaries
• Validate API request/response payloads
• Validate Redux state after every action
• Validate React component properties (like prop-types)
• spec-provider: infer a spec based on real-world data samples
• ...and many more!

See also

Other libraries

• js.spec (JavaScript)
• @exodus/schemasafe (JavaScript; JSON Schema)
• Plumatic Schema (Clojure)
• Malli (Clojure)
• DataSpec (Python)

Talks

• Rich Hickey: Spec-ulation

Q&A

Go forth and specify

Thank you!