And it might also be worth waiting for the common test suite guys to make their format tests use valid json (not just raw strings) first... (As mentioned in #148 )

I have a branch that uses webmock to pass the refRemote specs, but haven't had a chance to send a PR for it. I'll submit it later today.

I'll look into passing the disallow tests.

For the format tests, we aren't far off from full draft4 support if we implement basic hostname and email validations. I've suggested just dropping them completely, but maybe an alternative is to do a minimal form of validation (say, an email must match .*@.* or something silly like that), and advise users in the README that more sophisticated email validation is up to them. Does the custom format validation support overriding our default validators?

draft3 defines more formats, but the test suite only exercises (in addition to email and hostname):

1. color: we'd have to maintain a list of defined CSS colors.
2. regexp: "A regular expression, following the regular expression specification from ECMA 262/Perl 5." This is exercised both by the optional/format test and the optional/jsregex test, so we couldn't validate it by just using ruby's Regexp class AFAIK.

I dunno how I feel about half-baked format validations just to pass the optional tests, tbh. Is it worth the effort / maintenance?

I think that, rather than putting in validations that we know are awful, we’d do better to continue skipping tests for formats that we don’t want to support. We’d need to update the common test suite test to have more fine-grained filtering of tests (so we can skip individual tests within a test file, rather than just the whole file)=

I agree with @iainbeeston - many of the formats just don't make any sense (or are ambigious) from a validation perspective.

@iainbeeston, what about changing the exclusions list for tests to something like this:

IGNORED_TESTS = {
  "draft3/disallow.json" => :all,
  "draft3/optional/format.json" => [
    "test_something",
    "test_something_else"
  ],
  ...
}

Then you can exclude a whole file, or just individual tests within a file. Alternately, if you planned on never ignoring an entire file permanently, I can drop the :all support, and force people to list all the tests (which is clearer, and makes it easier for people who come later to fix a couple of the tests, but it'll be far more verbose).

If that looks reasonable, I can whip up a PR for that pretty quickly.

My gut feel is go for whichever is easier to implement (I'm guessing the version without all?). It's not something that will change often, and even then only to add or remove one or two tests=

Does the custom format validation support overriding our default validators?

Yes, that is possible and by unregistering a custom validator the default will be restored. Removing a default validator is not possible.

@iainbeeston, PR submitted. Let me know if you think the structure of IGNORED_TESTS could be cleaner.

The two required tests are fixed by #164 and #165; once someone merges those, I'll send another PR to only disable the format tests we are actually failing right now.

This has been resolved since 31f6faf; we now only skip the format tests we are explicitly opting out of.