System
Light
DarkAstonishing Serializations & Schemas of Hyperborea
Go back to blogOriginally published at techblog.babyl.ca.
For the last two years I am part of a band of intrepid adventurers joining forces every Thursday night via the teleporting magic of Discords, semi-arguably doing our best not to die horrible deaths in the raucously unforgiving world of Astonishing Swordsmen and Sorcerers of Hyperborea, a pulpy cousin of Dungeons & Dragons. The game is orchestrated by evil dungeon mastermind Gizmo Mathboy, and it's a massive amount of fun.
But the world of Hyperborea is not only besieged by monsters. Oh no. It is also a realm filled with rules, and statistics, and all manners of fate-defining dice rolls. And of the nexus capturing a lot of those arcane laws is -- unsurprising to all savvy to the genre -- the character sheet.
Being good little nerds, we usually do a good job of keeping the character sheets up-to-date. But we're all fallible creatures; mistakes creep in. Which made me think... surely there are ways to automate some validations on those character sheets. In fact, we already keep our sheets as YAML documents. JSON Schemas can totally be used to define document schemas... surely it could twisted a little bit more to accommodate the exotic logic of a game?
The answer is that, of course, everything can be twisted provided the spell is dark enough. This blog entry and its associated project repository, while not an exhaustive solution (yet), is intended to the goodies that JSON Schema could bring to the table, as well as the tools of the ecosystem.
So... Interested, fellow adventurers? Then gird those loins, sheath those blades, and follow me: into the JSON Schema jungle we go!
Preparing the ground
First, let's introduce the core tools I'll be using for this project.
For all things JSON Schema, we'll be using ajv (and ajv-cli for cli interactions). It's a fast, robust implementation of the JSON Schema specs with a lot of bonus features, and to ice the cake it provides an easy mechanism to add custom validation keywords, something we'll abuse before long.
And since we'll do a lot of command-line stuff, I'll bring in Task,
a YAML-based task runner -- basically Makefile with the insane
whitespace-based syntax replaced by, uh, a different insane
whitespace-based syntax I'm comfortable with.
Incidentally, the final form of all the code I'm going to discuss in this article is in this repo.
JSON is the worst
Okay, that's overly mean. JSON is a great serialization format, but it's
a soulless drag to edit manually. But that's not much of a problem, as JSON Schema
is kind of a misnomer: both the target documents and the schemas themselves
are ultimately just plain old data structure -- JSON just happens to be
the typical serialization for it. Well, typical be damned, we'll go with
YAML as our source. And for convenience for the other pieces to come,
we'll convert those YAML documents to JSON via [transerialize][].
1# in Taskfile.yml
2tasks:
3 schemas: fd -e yml -p ./schemas-yaml -x task schema SCHEMA='{}'
4
5 schema:
6 vars:
7 DEST:
8 sh: echo {{.SCHEMA}} | perl -pe's/ya?ml/json/g'
9 sources: ["{{.SCHEMA}}"]
10 generates: ["{{.DEST}}"]
11 cmds: transerialize {{.SCHEMA}} {{.DEST}}Oh yeah, task is unfortunately janky where loops
are concerned, so I'm using fd and re-entries to
deal with all the individual schema conversions.
Setting up the validation train
Before we go hog-wild on the schema itself, we need to figure out how we'll invoke things. And to do that, let's seed our schema and sample document in the most boring, minimalistic manner possible.
1# file: schemas-yaml/character.yml
2$id: https://hyperboria.babyl.ca/character.json
3title: Hyperboria character sheet
4type: object1# file: samples/verg.yml
2
3# Verg is my character, and always ready to face danger,
4# so it makes sense that he'd be volunteering there
5name: Verg-LaWe have a schema, we have a document, and the straightforward way to validate it is to do the following.
1⥼ ajv validate -s schemas-yaml/character.yml -d samples/verg.yml
2samples/verg.yml validSweet. Now we just need to formalize it a little bit in Taskfile and
we're ready to roll.
1# file: Taskfile.yml
2# in the tasks
3validate:
4 silent: true
5 cmds:
6 - |
7 ajv validate \\
8 --all-errors \\
9 --errors=json \\
10 --verbose \\
11 -s schemas-yaml/character.yml \\
12 -d {{.CLI_ARGS}}Starting on the schema
To warm ourselves up, let's begin with some easy fields. A character has obviously a name and a player.
1# file: schemas-yaml/character.json
2$id: https://hyperboria.babyl.ca/character.json
3title: Hyperboria character sheet
4type: object
5additionalProperties: false
6required:
7 - name
8 - player
9properties:
10 name: &string
11 type: string
12 player: *stringNothing special there, except for the YAML anchor and alias, because I'm a lazy bugger.
1⥼ task validate -- samples/verg.yml
2samples/verg.yml invalid
3[
4 ...
5 "message": "must have required property 'player'",
6 ...
7]Woo! Validation is screaming at us! The output is abbreged here because I configured it to be extra-verbose in the taskfile. But the gist is clear: we're supposed to have a player name and we don't. So let's add it.
1# file: samples/verg.yml
2name: Verg-La
3player: YanickAnd with the player name added, all is well again.
1⥼ task validate -- samples/verg.yml
2samples/verg.yml validAdding stats and definitions
Next thing, core statistics! All statistics are following the same rules (numbers between 1 and 20). Copying and pasting the schema for all stats would be uncouth. Using anchors as in the previous section is an option, but in this case it's better to use the a schema definition, to make things a little more formal.
1# file: schemas-yaml/character.yml
2# only showing deltas
3required:
4 # ...
5 - statistics
6properties:
7 # ...
8 statistics:
9 type: object
10 allRequired: true
11 properties:
12 strength: &stat
13 $ref: "#/$defs/statistic"
14 dexterity: *stat
15 constitution: *stat
16 intelligence: *stat
17 wisdom: *stat
18 charisma: *stat
19$defs:
20 statistic:
21 type: number
22 minimum: 1
23 maximum: 20Note that the allRequired is a custom keyword made available by
ajv-keywords, and to use it we have to amend our call to
ajv validate in the the taskfile:
1# file: Taskfile.yml
2validate:
3 silent: true
4 cmds:
5 - |
6 ajv validate \\
7 --all-errors \\
8 --errors=json \\
9 --verbose \\
10 -c ajv-keywords \\
11 -s schemas-yaml/character.yml \\
12 -d {{.CLI_ARGS}}To conform to the schema, we add the stats to our sample character too:
1# file: samples/verg.yml
2statistics:
3 strength: 11
4 dexterity: 13
5 constitution: 10
6 intelligence: 18
7 wisdom: 15
8 charisma: 11And we check and, yup, our sheet is still valid.
1⥼ task validate -- samples/verg.yml
2samples/verg.yml validOne sample doesn't serious testing make
So far, we've used Verg as our test subject. We tweak the schema, run it against the sheet, tweak the sheet, rince, lather, repeat. But as the schema is getting more complex, we probably want to add a real test suite to our little project.
One way would be to use ajv test, which has the appeal that no additional
code is required.
1⥼ ajv test -c ajv-keywords \\
2 -s schemas-yaml/character.yml \\
3 -d samples/verg.yml \\
4 --valid
5samples/verg.yml passed test
6# bad-verg.yml is like verg.yml, but missing the player name
7⥼ ajv test -c ajv-keywords \\
8 -s schemas-yaml/character.yml \\
9 -d samples/bad-verg.yml \\
10 --invalid
11samples/bad-verg.yml passed testBut what it has in simplicity, it lacks in modularity. Those schemas are going to get a little more involved, and targeting pieces of them would be good. So instead we'll go with good old unit test, via [vitest][].
For example, let's test statistics.
1// file: src/statistics.test.js
2import { test, expect } from "vitest";
3
4import Ajv from "ajv";
5
6import characterSchema from "../schemas-json/character.json";
7
8const ajv = new Ajv();
9// we just care about the statistic schema here, so that's what
10// we take
11const validate = ajv.compile(characterSchema.$defs.statistic);
12
13test("good statistic", () => {
14 expect(validate(12)).toBeTruthy();
15 expect(validate.errors).toBeNull();
16});
17
18test("bad statistic", () => {
19 expect(validate(21)).toBeFalsy();
20 expect(validate.errors[0]).toMatchObject({
21 message: "must be <= 20",
22 });
23});We add a test task to our taskfile:
1# file: Taskfile.yml
2test:
3 deps: [schemas]
4 cmds:
5 - vitest runAnd just like that, we have tests.
1⥼ task test
2task: [schemas] fd -e yml -p ./schemas-yaml -x task schema SCHEMA='{}'
3task: [schema] transerialize schemas-yaml/test.yml schemas-json/test.json
4task: [schema] transerialize schemas-yaml/character.yml schemas-json/character.json
5task: [test] vitest run
6
7 RUN v0.10.0 /home/yanick/work/javascript/hyperboria-character-sheet
8
9 √ src/statistics.test.js (2)
10
11Test Files 1 passed (1)
12 Tests 2 passed (2)
13 Time 1.41s (in thread 5ms, 28114.49%)More schemas!
Next step: the character class. While we could just slam an enum
in the main schema and call it done, it's a list that might be re-used
somewhere else, so it might pay off to define it in its own schema, and
refer to it in the character sheet schema.
Addititional challenge! In Hyperborea you can have a generic class, or a class and sub-class. Which can be schematized explicitly, like this:
1oneOf:
2 - enum: [ magician, figher ]
3 - type: object
4 properties:
5 generic: { const: fighter }
6 subclass: { enum: [ barbarian, warlock, ... ] }
7 ...But that's a lot of repetitive typing. Instead, it'd be nice to have the source be more compact, if a little less JSON Schemy. Say, something like this:
1$id: https://hyperboria.babyl.ca/classes.json
2title: Classes of characters for Hyperborea
3$defs:
4 fighter:
5 - barbarian
6 - berserker
7 - cataphract
8 - hunstman
9 - paladin
10 - ranger
11 - warlock
12 magician: [cryomancer, illusionist, necromancer, pyromancer, witch]And then have a little script massage the data as we turn the YAML into
JSON. Fortunately (what a lucky break!), transerialize does allow
for a transformation script to be wedged in the process. So we can change
our taskfile schema task to be:
1schema:
2 vars:
3 TRANSFORM:
4 sh: |
5 echo {{.SCHEMA}} | \\
6 perl -lnE's/yml$/pl/; s/^/.\//; say if -f $_'
7 DEST:
8 sh: echo {{.SCHEMA}} | perl -pe's/ya?ml/json/g'
9 cmds:
10 - transerialize {{.SCHEMA}} {{.TRANSFORM}} {{.DEST}}And then we slip in a transform script that looks like this:
1# file: schemas-yaml/classes.pl
2sub {
3 my $schema = $_->{oneOf} = [];
4
5 push @$schema, { enum => [ keys $_->{'$defs'}->%* ] };
6
7 for my $generic ( keys $_->{'$defs'}->%* ) {
8 push @$schema, {
9 type => 'object',
10 properties => {
11 generic => { const => $generic },
12 subclass => { enum => $_->{'$defs'}{$generic} }
13 }
14 }
15 }
16
17 return $_;
18}With that, the output schema is inflated to what we want. We're having our concise cake eating the big fluffy one too. Nice!
So what is left is to link the schemas together. We refer to the classes schema from the character schema:
1# file: schemas-yaml/character.yml
2required:
3 # ...
4 - class
5properties:
6 # ...
7 class: { $ref: "/classes.json" }We also need to tell ajv of the existence of that new schema:
1validate:
2 silent: true
3 cmds:
4 - |
5 ajv validate \\
6 --all-errors \\
7 --errors=json \\
8 --verbose \\
9 -c ajv-keywords \\
10 -r schemas-json/classes.json \\
11 -s schemas-json/character.json \\
12 -d {{.CLI_ARGS}}Finally, we add Verg's class to his sheet:
1# file: samples/verg.yml
2class:
3 generic: magician
4 subclass: cryomancerAnd just like that, Verg (and our character schema) is all classy and stuff.
Referencing other parts of the schema
So far we can set up our character sheet schema to ensure that we have the fields that we want, with the types and values that we want. But something else we want to do is to validate the relations between properties.
For example, characters have a health statistic. Each time the character levels up, the player rolls a dice and increases the health accordingly. As you image, forgetting to get that bonus can prove to be a lethal mistake, so it'd be nice to ensure that never happens.
We'll do it through the magic of JSON Pointers and avj's $data, like so:
1# file: schemas-yaml/character.yml
2level: { type: number, minimum: 1 }
3health:
4 type: object
5 required: [ max ]
6 properties:
7 max: { type: number }
8 current: { type: number }
9 log:
10 type: array
11 description: history of health rolls
12 items: { type: number }
13 minItems: { $data: /level }
14 maxItems: { $data: /level }Basically (and once we add a --data flag to ajv to tell it to enable
that feature), any mention of { $data: '/path/to/another/value/in/the/schema' } will be replaced by the value for which that JSON pointer resolves to
in the document being validated. That's something that is not part of JSON
Schema proper, but it's a mightily useful way to interconnect the schema
and the document being validated.
Word of caution, though: I say 'any mention of $data', but that's
overselling it. There are a few cases where $data fields won't be resolved.
If you are to use that feature, make sure to reserve a few minutes to read
the AJV docs about it. Trust me, it'll save you a few "what the everlasting heck?" moments.
Custom keywords
In the previous section, we checked that the number of rolls for health is equal to the level of the character. That's already something. But the logical next step is to ensure that the sum of those rolls are equal to the max health points we have. We'd need something like:
1# file: schemas-yaml/character.yml
2health:
3 type: object
4 properties:
5 max:
6 type: number
7 sumOf: { list: { $data: 1/log } }
8 log:
9 type: array
10 items: { type: number }That's where custom keywords enter the picture. AJV allows us to augment the JSON Schema vocabulary with new keywords.
There is a few ways to define that custom keyword. The one I opted for is defining it as a JavaScript function (here made a little more complex because we're dealing internally with JSON pointers):
1// file: src/sumOf.cjs
2
3const _ = require("lodash");
4const ptr = require("json-pointer");
5
6function resolvePointer(data, rootPath, relativePath) {
7 if (relativePath[0] === "/") return ptr.get(data, relativePath);
8
9 const m = relativePath.match(/^(\d+)(.*)/);
10 relativePath = m[2];
11 for (let i = 0; i < parseInt(m[1]); i++) {
12 rootPath = rootPath.replace(/\/[^\/]+$/, "");
13 }
14
15 return ptr.get(data, rootPath + relativePath);
16}
17
18module.exports = (ajv) =>
19 ajv.addKeyword({
20 keyword: "sumOf",
21 $data: true,
22 errors: true,
23 validate: function validate(
24 { list, map },
25 total,
26 _parent,
27 { rootData, instancePath }
28 ) {
29 if (list.$data)
30 list = resolvePointer(rootData, instancePath, list.$data);
31
32 if (map) data = _.map(data, map);
33
34 if (_.sum(list) === total) return true;
35
36 validate.errors = [
37 {
38 keyword: "sumOf",
39 message: "should add up to sum total",
40 params: {
41 list,
42 },
43 },
44 ];
45
46 return false;
47 },
48 });As usual we have to tell ajv to include that new bit of code via
-c ./src/sumOf.cjs. But beside that, congrats, we have a new keyword!
More of the same
By now we have most of the tools we want, all that is left to do is to turn the crank.
Experience points? Much of the same logic as for the health points:
1# file: schemas-yaml/character.yml
2experience:
3 type: object
4 properties:
5 total:
6 type: number
7 sumOf:
8 list: { $data: '1/log' }
9 map: amount
10 log:
11 type: array
12 items:
13 type: object
14 properties:
15 date: *string
16 amount: *number
17 notes: *stringThe other basic attributes are trivial:
1# file: schemas-yaml/character.yml
2gender: *string
3age: *number
4height: *string
5appearance: *string
6alignment: *stringFields based on lists? Been there, done that:
1# file: schemas-yaml/character.yml
2 race: { $ref: /races.json }
3 languages:
4 type: array
5 minItems: 1
6 items:
7 $ref: /languages.jsonSpells are only for magicians? Not a problem.
1# file: schemas-yaml/character.yml
2type: object
3properties:
4 # ...
5 spells:
6 type: array
7 items: { $ref: /spells.json }
8 maxSpells:
9 class: { $data: /class }
10 level: { $data: /level }With the new keyword maxSpells:
1// file: src/maxSpells.cjs
2
3const _ = require("lodash");
4const resolvePointer = require('./resolvePointer.cjs');
5
6module.exports = (ajv) =>
7 ajv.addKeyword({
8 keyword: "maxSpells",
9 validate: function validate(
10 schema,
11 data,
12 _parent,
13 { rootData, instancePath }
14 ) {
15 if (schema.class.$data) {
16 schema.class = resolvePointer(
17 rootData, instancePath, schema.class.$data
18 );
19 }
20
21 if( schema.class !== 'magician'
22 && schema.class?.generic !== 'magician'
23 && data.length ) {
24 validate.errors = [
25 {
26 message: "non-magician can't have spells",
27 },
28 ];
29 return false;
30 }
31
32 return true;
33 },
34 $data: true,
35 errors: true,
36 });Gears? Pfah! Sure.
1# file: schemas-yaml/character.yml
2properties:
3 # ...
4 gear: { $ref: '#/$defs/gear' }
5$defs:
6 gear:
7 type: array
8 items:
9 oneOf:
10 - *string
11 - type: object
12 properties:
13 desc:
14 type: string
15 description: description of the equipment
16 qty:
17 type: number
18 description: |
19 quantity of the item in the
20 character's possession
21 required: [ desc ]
22 additionalProperties: false
23 examples:
24 - { desc: 'lamp oil', qty: 2 }By now you get the point. A lot of constraints can be expressed via vanilla JSON Schema keywords. For the weirder things, new keywords can be added. And for anything that is onerous to type of, we have to remember that underneath it's all JSON, and we know darn well how to munge that.