Port FlatBuffers to Rust #4898

rw · 2018-08-29T03:20:57Z

This is a port of FlatBuffers to native Rust. It provides code generation and a runtime library derived from the C++ implementation. It utilizes the Rust type system to provide safe and fast traversal of FlatBuffers data.

There are 188 tests, including many fuzz tests of roundtrips for various serialization scenarios. Initial benchmarks indicate that the canonical example payload can be written in ~700ns, and traversed in ~100ns.

Rustaceans may be interested in the Follow,Push, and SafeSliceAccess traits. These traits lift traversals, reads, writes, and slice accesses into the type system, providing abstraction with no runtime penalty.

ry · 2018-08-29T03:45:33Z

Awesome work @rw - thank you! We're already using your patch in our project and it's working great.

phayes · 2018-08-30T13:24:26Z

Should the fie tests/rust_usage_test/test_bench_output.txt have been included in this PR?

rw · 2018-08-30T17:42:20Z

@phayes I included it in the PR for people who don't want to run the code but still want to see the output of tests passing.

aardappel

WOW! This is some amazing effort, the most comprehensive language port yet!

I'm missing Rust specific changes to Tutorial.md, and a Rust sample in samples. Here is a sample checklist to see if you've touched all typical language port files: 4898809

CMakeLists.txt

aardappel · 2018-08-30T20:55:58Z

docs/source/RustUsage.md

+The generated accessor functions access fields over offsets, which is
+very quick. These offsets are not verified at run-time, so a malformed
+buffer could cause a program to crash by accessing random memory. (We try to
+prevent this in Rust by using safe slice accesses, instead of unsafe pointer


this phrase is a bit weird.. what exactly happens if an offset in the buffer that points outside of the buffer is turned into a slice, and then accessed? How is that different from a raw pointer?

aardappel · 2018-08-30T20:58:15Z

docs/source/RustUsage.md

+data from the network that can potentially have been modified by an
+attacker, this is undesirable.
+
+The C++ port provides a buffer verifier, but, at this time, Rust does not.


"Rust may provide this in a future version" ?

aardappel · 2018-08-30T21:02:36Z

rust/flatbuffers/src/builder.rs

+    /// the default, then this is a no-op.
+    #[inline]
+    pub fn push_slot<X: Push + PartialEq>(&mut self, slotoff: VOffsetT, x: X, default: X) {
+        self.assert_nested("push_slot must be called after start_table");


Given that users of the generated API likely aren't calling push_slot themselves, this message may be confusing? In fact, it may be easier to just have one message for nesting violations.

aardappel · 2018-08-30T21:05:10Z

rust/flatbuffers/src/builder.rs

+    }
+
+    #[inline]
+    fn fill(&mut self, zero_pad_bytes: usize) {


function doesn't appear to do anything?

aardappel · 2018-08-30T21:26:50Z

tests/generate_code.sh

@@ -14,8 +14,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-../flatc --cpp --java --csharp --dart --go --binary --lobster --lua --python --js --ts --php --grpc --gen-mutable --reflect-names --gen-object-api --no-includes --cpp-ptr-type flatbuffers::unique_ptr  --no-fb-import -I include_test monster_test.fbs monsterdata_test.json
-../flatc --cpp --java --csharp --dart --go --binary --lobster --lua --python --js --ts --php --gen-mutable --reflect-names --no-fb-import --cpp-ptr-type flatbuffers::unique_ptr  -o namespace_test namespace_test/namespace_test1.fbs namespace_test/namespace_test2.fbs
+../flatc --cpp --java --csharp --dart --go --binary --lobster --lua --python --js --ts --php --rust --grpc --gen-mutable --reflect-names --gen-object-api --no-includes --cpp-ptr-type flatbuffers::unique_ptr  --no-fb-import -I include_test monster_test.fbs monsterdata_test.json


please change .bat as well

aardappel · 2018-08-30T21:28:09Z

tests/monster_test_generated.rs

+impl<'a> flatbuffers::Follow<'a> for Monster<'a> {
+    type Inner = Monster<'a>;
+    #[inline]
+    fn follow(buf: &'a [u8], loc: usize) -> Self::Inner {


should this be _follow if there is a possibility of a clash with generated fields?

Good eye! This is actually okay because it's a trait method, so unless users are importing Push, they cannot access it. Furthermore, I added the trait methods we use to the reserved keywords list, so a field called follow would be generated as follow_.

aardappel · 2018-08-30T21:32:44Z

tests/rust_usage_test/test_bench_output.txt

@@ -0,0 +1,184 @@
+running 170 tests


seems weird to include this in the repo?

aardappel · 2018-08-30T21:34:37Z

tests/rust_usage_test/tests/integration_test.rs

+        let test4 = builder.create_vector_direct(&[my_game::example::Test::new(10, 20),
+                                                   my_game::example::Test::new(30, 40)]);
+        let pos = my_game::example::Vec3::new(1.0, 2.0, 3.0, 3.0, my_game::example::Color::Green, &my_game::example::Test::new(5i16, 6i8));
+        let args = my_game::example::MonsterArgs{


inline in the create call?

Added a comment explaining why we can't do this (the Args struct takes a reference to a Vec3 so we have to make sure it lives long enough).

aardappel · 2018-08-30T21:35:38Z

tests/rust_usage_test/tests/integration_test.rs

+            None => { return Err("bad m.testarrayofstring"); }
+            Some(x) => { x }
+        };
+        if testarrayofstring.len() != 2 { return Err("bad monster.testarrayofstring len"); }


should bad fieldname be generated with a macro?

or just use assert_eq ?

Wrote a macro for this.

rw · 2018-09-02T21:12:14Z

@aardappel Thanks for the feedback, I incorporated it all. Also, I resolved an alignment issue that I didn't catch before, and made the Push trait more useful.

aardappel · 2018-09-03T00:14:54Z

Thanks! Amazing work!

This is a port of FlatBuffers to Rust. It provides code generation and a runtime library derived from the C++ implementation. It utilizes the Rust type system to provide safe and fast traversal of FlatBuffers data. There are 188 tests, including many fuzz tests of roundtrips for various serialization scenarios. Initial benchmarks indicate that the canonical example payload can be written in ~700ns, and traversed in ~100ns. Rustaceans may be interested in the Follow,Push, and SafeSliceAccess traits. These traits lift traversals, reads, writes, and slice accesses into the type system, providing abstraction with no runtime penalty.

This is a port of FlatBuffers to Rust. It provides code generation and a runtime library derived from the C++ implementation. It utilizes the Rust type system to provide safe and fast traversal of FlatBuffers data. There are 188 tests, including many fuzz tests of roundtrips for various serialization scenarios. Initial benchmarks indicate that the canonical example payload can be written in ~700ns, and traversed in ~100ns. Rustaceans may be interested in the Follow, Push, and SafeSliceAccess traits. These traits lift traversals, reads, writes, and slice accesses into the type system, providing abstraction with no runtime penalty.

binary132 · 2018-09-03T01:41:23Z

👏 👏 👏

This is a port of FlatBuffers to Rust. It provides code generation and a runtime library derived from the C++ implementation. It utilizes the Rust type system to provide safe and fast traversal of FlatBuffers data. There are 188 tests, including many fuzz tests of roundtrips for various serialization scenarios. Initial benchmarks indicate that the canonical example payload can be written in ~700ns, and traversed in ~100ns. Rustaceans may be interested in the Follow, Push, and SafeSliceAccess traits. These traits lift traversals, reads, writes, and slice accesses into the type system, providing abstraction with no runtime penalty.

rw mentioned this pull request Aug 29, 2018

Support for Rust programming language #3894

Closed

rw force-pushed the 2018-08-28--rw-flatbuffers-rust branch from c855cd4 to e92a41e Compare August 29, 2018 03:39

rw force-pushed the 2018-08-28--rw-flatbuffers-rust branch from e92a41e to 4722510 Compare August 29, 2018 03:48

aardappel self-requested a review August 29, 2018 03:54

rw force-pushed the 2018-08-28--rw-flatbuffers-rust branch 9 times, most recently from dff1ac4 to de40027 Compare August 29, 2018 18:56

rw self-assigned this Aug 29, 2018

rw force-pushed the 2018-08-28--rw-flatbuffers-rust branch 10 times, most recently from 5c1afc2 to 21fefe9 Compare August 30, 2018 11:01

Port FlatBuffers to Rust: generator/runtime/tests.

93f9162

rw force-pushed the 2018-08-28--rw-flatbuffers-rust branch from 21fefe9 to 93f9162 Compare August 30, 2018 11:11

aardappel reviewed Aug 30, 2018

View reviewed changes

rw added 19 commits August 30, 2018 21:17

comment to explain why we do not inline a struct creation

b5e5096

refactor some tests with macros

77fb225

add support column for rust

47ebb61

ergonomics/lifetimes tweaks in builder

ee51cc0

no more need for ZeroTerminatedByteSlice

a78edd0

wip for making Push::size a static method

67004f3

simplifying push some more

894fe3e

delete old test

fb28cad

inlining, push

bd33f75

Push::size is static

a3ef1de

start_vector requires push

87cef86

more push/vector typing

bd2a5ed

no more phantomdata

36de894

good sample and docs

3c03f30

add a todo to rust lib

0de4628

regenerate rust test code

7972b79

more alignment/size tests

f070f53

remove dead code

8081c5a

doc tweak

acf48d0

revert old typescript file to master

4d005c1

aardappel merged commit 0758c1e into google:master Sep 3, 2018

rw mentioned this pull request Sep 3, 2018

Official Rust FlatBuffers is now available apache/arrow#2509

Closed

rw mentioned this pull request Jul 3, 2019

Contributing Golang and Rust implementations zarr-developers/community#5

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port FlatBuffers to Rust #4898

Port FlatBuffers to Rust #4898

rw commented Aug 29, 2018 •

edited

Loading

ry commented Aug 29, 2018

phayes commented Aug 30, 2018

rw commented Aug 30, 2018

aardappel left a comment

aardappel Aug 30, 2018

aardappel Aug 30, 2018

aardappel Aug 30, 2018

aardappel Aug 30, 2018

aardappel Aug 30, 2018

rw Sep 2, 2018

aardappel Aug 30, 2018

rw Sep 2, 2018

aardappel Aug 30, 2018

rw Sep 2, 2018

aardappel Aug 30, 2018

rw Sep 2, 2018

aardappel Aug 30, 2018

aardappel Aug 30, 2018

rw Sep 2, 2018

rw commented Sep 2, 2018

aardappel commented Sep 3, 2018

binary132 commented Sep 3, 2018

Port FlatBuffers to Rust #4898

Port FlatBuffers to Rust #4898

Conversation

rw commented Aug 29, 2018 • edited Loading

ry commented Aug 29, 2018

phayes commented Aug 30, 2018

rw commented Aug 30, 2018

aardappel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rw commented Sep 2, 2018

aardappel commented Sep 3, 2018

binary132 commented Sep 3, 2018

rw commented Aug 29, 2018 •

edited

Loading