Description
If you guys are interested in this, you have some of the better tests in existence, and test262 would benefit from them greatly. Then hopefully stupid things like Safari's non-compliant implementations would not happen, because everybody would be running your tests :)
I'd love to help in any way I can to make this process more smooth, if you are up for it.
I am trying to blaze the trail with my Array.prototype.contains reference implementation: notably the package.json which configures the test runner, and the tests themselves. One big thing to notice is that test262 favors lots of small little files---more or less one for each Mocha it
.
I imagine in the process you will find test262 to be pretty anemic compared to Mocha/Chai. In particular we currently do if (!condition) { $ERROR(msg); }
instead of assert(condition, message)
. I think (@bterlson can correct me if I'm wrong) that putting together a basic assert library, containing e.g. assert
, assertEqual
, assertOwnPropertiesEqual
or a few other things, would probably be OK. Maybe a good first step would be to figure out the minimum set of assertions you'd need, and if we agree on them, get them into test262 as helpers.
Also tagging in @smikes who has been doing great test262 work and can probably help advise.
Activity
ljharb commentedon Aug 4, 2014
I've actually filed #48 to switch from Mocha/Chai to tape - I want tests that are simple, and make it easy to do
node
and cross-browser testing (ietestling
), and annpm test
command that returns proper unix exit codes.That said, I'd love to consolidate tests in any way that we can.
domenic commentedon Aug 4, 2014
Yeah, I guess I was hoping to preempt that move by suggesting you move to the test262 format instead, so that you can share better with browser vendors. The test262-harness project is nascent and we can fix it any way that you want (e.g., not sure if it does proper error codes yet, or how browserifiable it is to make testling or Zuul simple).
ljharb commentedon Aug 4, 2014
I need a) a console command that returns a zero or nonzero exit code, b) TAP-compliant test output, c) if it needs a harness, it needs to work with
testling
. I'm open to using test262's format if we can achieve those.domenic commentedon Aug 4, 2014
OK, well a) is easy if we don't already have it, b) it already does, and c) might be tricky but might be easy, we'll see.
BTW I don't recommend testling; it's too flaky. I'd go with Sauce Labs (probably via Zuul) instead.
cscott commentedon Aug 4, 2014
Probably what would be worthwhile in the sort-term is a mocha-test262 bridge, to allow mocha to run test262 test cases. That would let us gradually convert test cases over, instead of having to do it all in one big lump.
smikes commentedon Aug 5, 2014
Yes, I am hoping to build something like that because the python console runner we are currently using for Test262 is pretty limited.
ljharb commentedon Aug 5, 2014
Awesome, because a python test runner in a JS project is basically a non-starter :-)
Yaffle commentedon Aug 13, 2014
This is a good topic, because, i think, es5-shim, es6-shim, test262 and other probject should use same tests.
Is it a big difference between mocha or tape or something for ecmascript tests?
Are you serious?
I do not think so, i think testharness should be simple as possible.
Those small files slow down work really.
Also, it is really hard to change something in test262, as this project is very conservative.
mathiasbynens commentedon Aug 13, 2014
I agree that the many small little files are not very practical to work with as a test author. But I can see why this is unlikely to change.
What’s worse IMHO is that each file needs to contain a copyright header, so over 50% of the test262 repository is just comments with license information. Can’t we just enforce a default license for all new contributions so that this is no longer needed?
smikes commentedon Aug 13, 2014
New tests can have the simple copyright header:
I can't comment on ECMA's legal concerns with respect to existing code. It may be possible to bulk-replace the licenses for Sputnik and Microsoft-sourced tests with something more compact, but that's a decision well above my pay grade. Maybe @bterlson can comment on that?
(Edited in light of @bterlson's comments, below. I mistakenly thought I was assigning to ECMA, but unfortunately nothing is that simple.)
bterlson commentedon Aug 13, 2014
Just some thoughts...
@Yaffle
Yes, they do :( The rationale, if you're curious: each assert raises an error and stops test execution. More asserts per file mean more hidden bugs when something fails, which means more and slower work for implementers diagnosing failures, and test pass rates are less representative in the presence of failures.
I keep a buffer with the copyright header and a stub frontmatter block and I can crank tests out pretty quickly. I hope tooling will help address some of this too!
Test262 has different goals and assumptions. Among others, we cannot assume that tests are running on a fully functioning implementation! Test262 needs to remain valuable for implementers bringing up new implementations from nothing. This drastically limits what we are able to do in the test environment (and libraries similar to Chai are likely never to make it into the project despite being very handy).
That said, please file requests you have as issues on the official project! Some may be possible, and others we could figure out something comparable that works. Won't know until you try!
@mathiasbynens There are two pieces of information we need - who owns the copyright to the test, and what the license is. LICENSE.txt contains the license, but tests need to retain information about who owns the copyright. Granted, this information is already contained in the git commit metadata most likely, so I'll investigate whether we can get away with removing the copyright header.
@smikes Contributors retain copyright of all collateral contributed under tc39/test262-cla so the copyright header should reflect that. Historically, Microsoft has used a separate agreement that assigns copyright to ECMA, hence the number of tests with copyright ECMA.
Copyright header normalization is totally cool (I already did a lot of it with the previous normalization pass, removing most instances of copyright headers with the entire license text pasted in). If there's more you want to do, by all means open an issue!
Yaffle commentedon Aug 14, 2014
So the script, which will split tests into multiple files can be used, right? Although contributors will have to run this script anyway...
bterlson commentedon Aug 14, 2014
@Yaffle I'm not sure I understand your question but if you're asking if you can use a script to convert many-tests-per-file tests into one-test-per-file tests, the answer is yes that's fine...
8 remaining items