A modular, composable, strongly typed and lightweight implementation of the SCALE Codec
Using npm:
npm install --save scale-ts
Using Deno, it can be imported from:
https://deno.land/x/scale_ts/index.ts
import { bool, _void, str, u32, Enum, Struct, Vector } from "scale-ts"
const myCodec = Struct({
id: u32,
name: str,
friendIds: Vector(u32),
event: Enum({
_void,
one: str,
many: Vector(str),
allOrNothing: bool,
}),
})Something really cool about this library is that by having composable codecs
with really good typings, then the inferred types of the custom codecs are
also really good. For instance, the inferred types of the myCodec defined
above are:
type MyCodec = Codec<{
id: number;
name: string;
friendIds: number[];
event:
| { tag: _void; value?: undefined };
| { tag: one; value: string; }
| { tag: many; value: string[]; }
| { tag: allOrNothing; value: boolean; };
}>That's very useful, because on the one hand we will get a TS error if we try to
pass an invalid input to the encoder. For instance, in the following example TS
will because the property event.value is invalid for the provided tag:
myCodec.enc({
event: { tag: "one", value: 5 },
name: "Some name",
id: 100,
friendIds: [1, 2, 3],
})On the other hand, the result of the decoded value also has that same interface, which is extremely useful.
An example on how to encoded/decode a valid value:
myCodec.enc({
id: 100,
name: "Some name",
friendIds: [1, 2, 3],
event: { tag: "allOrNothing" as const, value: true },
})
// => 0x6400000024536f6d65206e616d650c0100000002000000030000000301
const decodedData = myCodec.dec(
"0x6400000024536f6d65206e616d650c0100000002000000030000000301",
)
console.log(JSON.stringify(decodedData, null, 2))
// =>
//{
// "id": 100,
// "name": "Some name",
// "friendIds": [
// 1,
// 2,
// 3
// ],
// "event": {
// "tag": "allOrNothing",
// "value": true
// }
//}In this library you won't find common codec definitions like AccountId.
However, since the codecs of this library are just composable functions, it is
very easy to create new custom codecs.
As an example, the internal implementation of the bool codec looks like this:
import { enhanceCodec, u8, Codec } from "../"
const booleanToNumber = (value: boolean) => (value ? 1 : 0)
const numberToBoolean = Boolean
export const bool: Codec<boolean> = enhanceCodec(
u8,
booleanToNumber,
numberToBoolean,
)Similarly, you could implement codecs based on other codecs. For instance, a
possible implementation of an AccountId codec could be:
import { enhanceCodec, Bytes } from "scale-ts"
import { decodeAddress, encodeAddress } from "@polkadot/util-crypto"
export const AccountId = enhanceCodec(Bytes(32), decodeAddress, encodeAddress)An Encoder is a function with the following signature:
type Encoder<T> = (value: T) => Uint8ArrayA Decoder is a function with the following signature:
type Decoder<T> = (value: Uint8Array | ArrayBuffer | string) => TA Codec is an interface that contains two functions: an Encoder and a
Decoder.
Also, for convenience, the codecs from scale-ts allow you to access
then encoder and the decoder in 2 different ways:
- Destructuring a tuple
[Encoder<T>], Decoder<T>]. E.g:
const [numberListEncoder, numberListDecoder] = Vector(u16)
const encodedList = numberListEncoder([4, 8, 15, 16, 23, 42])
// => 0x18040008000f00100017002a00
const decodedList = numberListDecoder(encodedList)
// => [4, 8, 15, 16, 23, 42]- Through the
encanddecproperties of the codec. E.g:
const numberListCodec = Vector(u16)
const encodedList = numberListCodec.enc([4, 8, 15, 16, 23, 42])
// => 0x18040008000f00100017002a00
const decodedList = numberListCodec.dec(encodedList)
// => [4, 8, 15, 16, 23, 42]Therefore, the type definition of Codec is as follows:
type Codec<T> = [Encoder<T>, Decoder<T>] & {
enc: Encoder<T>
dec: Decoder<T>
}A codec-creator is a function that takes one or many codecs through its argument(s) and returns a new codec.
For instance: Tuple, Vector, Struct, Enum, etc
A convention of this library is that codec-creators are capitalized, to differentiate them from codecs which are lowercase.
In the past this library used to refer to codec-creators as higher order codecs, which is (maybe?) a more accurate term. However, many developers find that terminology confusing, so from now on we will refer to them as "codec-creators", or "codec-creator functions", which is a much more descriptive name.
Supported codecs are: u8, u16, u32, u64, u128, u256, i8, i16, i32, i64, i128, i256
i128.enc(-18676936063680574795862633153229949450n)
// => 0xf6f5f4f3f2f1f0f9f8f7f6f5f4f3f2f1
i128.dec("0xf6f5f4f3f2f1f0f9f8f7f6f5f4f3f2f1")
// => -18676936063680574795862633153229949450ncompact.enc(65535)
// => 0xfeff0300
compact.dec("0xfeff0300")
// => 65535bool.enc(false)
// => 0x00
bool.dec("0x01")
// => trueNormal cases:
cosnt optionalCompact = Option(compact)
optionalCompact.enc()
// => 0x00
optionalCompact.enc(undefined)
// => 0x00
optionalCompact.enc(1)
// => 0x0104Exceptionally, if the input is bool, then it always returns one byte:
cosnt optionalBool = Option(bool)
optionalBool.enc()
// => 0x00
optionalBool.enc(true)
// => 0x01
optionalBool.enc(false)
// => 0x02const resultCodec = Result(u8, bool)
resultCodec.enc({ success: true, value: 42 })
// => 0x002a
resultCodec.enc({ success: false, value: false })
// => 0x0100Dynamic, for when the size is known at run time:
const numbers = Vector(u16)
numbers.enc([4, 8, 15, 16, 23, 42])
// => 0x18040008000f00100017002a00Fixed, for when the size is known at compile time:
const fiveNumbers = Vector(u16, 5)
numbers.enc([4, 8, 15, 16, 23])
// => 0x040008000f0010001700str.enc("a$¢ह€한𐍈😃")
// => 0x546124c2a2e0a4b9e282aced959cf0908d88f09f9883const compactAndBool = Tuple(compact, bool)
compactAndBool.enc([3, false])
// => 0x0c00const myCodec = Struct({
id: u32,
name: str,
friendIds: Vector(u32),
event: Enum({
_void,
one: str,
many: Vector(str),
allOrNothing: bool,
}),
})
myCodec.enc({
id: 100,
name: "Some name",
friendIds: [1, 2, 3],
event: { tag: "allOrNothing" as const, value: true },
})
// => 0x6400000024536f6d65206e616d650c0100000002000000030000000301const { enc, dec } = Enum({
nothingHere: _void,
someNumber: u8,
trueOrFalse: bool,
optionalBool: Option(bool),
optVoid: Option(_void),
})
enc({ tag: "nothingHere" })
// => 0x00
dec("0x012a")
// => { tag: "someNumber", value: 42 }Sometimes, mainly when creating your custom codecs, it's usefull to have a
codec that simply reads/writes a certain amount of bytes. For example, see the
example above for creating a custom AccountId codec.
const [encode, decode] = Bytes(3)
encode(new Uint8Array([0, 15, 255]))
// => 0x000fff
decode("0x000fff00")
// => 0x000fffThis is a special codec that it's mostly useful in combination with
Enum, its type is Codec<void>, and as you can imagine calling
_void.enc() returns an empty Uint8Array, while calling _void.dec
always returns undefined.
TODO: document them
A very important remark is that in this library you will only find the basic primitives that can be used for building more complex codecs. That being said, this library provides a set of utils to facilitate that.
Probably the easiest way to explain this is by solving a couple of examples, so let's get to it.
Let's say that you want to have a MapCodec function that works like this:
const myMap: Codec<Map<number, string>> = MapCodec(u8, str)How could we create that MapCodec with scale-ts?
Basically, what we want to do is to transform the result of a
Vector(Tuple(keyCodec, valueCodec)) to a Map instance, and viceversa.
So, let's first create the encoder function, using enhanceEncoder:
const MapEncoder = <K, V>(key: Encoder<K>, value: Encoder<V>) =>
enhanceEncoder(Vector.enc(Tuple.enc(key, value)), (input: Map<K, V>) =>
Array.from(input.entries()),
)Now, let's create its decoder counterpart, using enhanceDecoder:
const MapDecoder = <K, V>(key: Decoder<K>, value: Decoder<V>) =>
enhanceDecoder(
Vector.dec(Tuple.dec(key, value)),
(entries) => new Map(entries),
)Finally, lets create the MapCodec function:
export const MapCodec = <K, V>(
key: Codec<K>,
value: Codec<V>,
): Codec<Map<K, V>> =>
createCodec(MapEncoder(key.enc, value.enc), MapDecoder(key.dec, value.dec))
MapCodec.enc = MapEncoder
MapCodec.dec = MapDecoderThat's it 🎉!
Now, let's see how we can create a more complex function, like something
for encoding and decoding the instances of our classes, even if those instances
are more than mere setters/getters. Let's say that we want to create a
ClassCodec function that can be used like this:
class RepeatedString {
constructor(item: string, nTimes: number) {
this.repetition = Array(nTimes).fill(item)
}
}
// It's not necessary to have the `: Codec<RepeatedString>` notation
// b/c it's being inferred. It's been added just to make it easier to understand
const repeatedStrCodec: Codec<RepeatedString> = ClassCodec(
RepeatedString,
[str, compact],
(value: RepeatedString) => [value.repetition[0], value.repetition.length],
)How can we implement ClassCodec with scale-ts?
Basically, what we want to do is:
- For decoding: we want to instantiate the class using the result of a
Tuple - For encoding: with the help of a function that maps the instance of the
class back to the argumetns of the constructor, encode the
Tuple.
It goes without saying that this function could have other signatures, or more overloads, of course. In fact, it's probably not that useful in real life, but it's helpful for teaching purposes.
The only difficult thing about creating a codec-creator like this is to get the types right, but let's not shy away from it.
First, let's write the function for encoding:
const ClassEncoder =
<
A extends Array<Encoder<any>>,
OT extends { [K in keyof A]: A[K] extends Encoder<infer D> ? D : unknown },
Constructor extends new (...args: OT) => any,
>(
mapper: (instance: InstanceType<Constructor>) => OT,
): Encoder<InstanceType<Constructor>> =>
(instance) => {
return Tuple.enc(...mapper(instance)) as any
}So, leaving aside the complex types for inferring the arguments, the actual JS code is pretty straight-forward.
Then, let's create the function for creating the Decoder:
const ClassDecoder = <
A extends Array<Decoder<any>>,
OT extends { [K in keyof A]: A[K] extends Decoder<infer D> ? D : unknown },
Constructor extends new (...args: OT) => any,
>(
classType: Constructor,
...decoders: A
): Decoder<InstanceType<Constructor>> =>
enhanceDecoder(
Tuple.dec(...decoders),
(args) => new classType(...(args as any)),
)Same deal: complex types because we care about our users, but aside from that, the actual JS code is pretty simple.
And now we are ready to put everything together:
const ClassCodec = <
A extends Array<Codec<any>>,
OT extends { [K in keyof A]: A[K] extends Codec<infer D> ? D : unknown },
Constructor extends new (...args: OT) => any,
>(
classType: Constructor,
codecs: A,
mapper: (instance: InstanceType<Constructor>) => OT,
) =>
createCodec(
ClassEncoder(mapper),
ClassDecoder(classType, ...codecs.map((c) => c.dec)),
)
ClassCodec.enc = ClassEncoder
ClassCodec.dec = ClassDecoderHopefully, these 2 examples showcase the main goal of the library: to provide good and lean building blocks so that we can build complex things with them.
In the past this library used to have some "sugar" (Hex, MapCodec,
SetCodec, date32, etc).
However, all that "sugar" has been removed and it won't be coming back.
The main reason is that all those codecs (and codec-creators) can be easily implemented in userland, and if we start adding sugar, then this library could easily become a chaotic directory with all sorts of Codecs.
It's precisely because we want to enable the creation of any thinkable codec or codec-creator, that it's very important that the building blocks that we provide are as minimalist and ergonomic as they can be.