Skip to content

Transaction retries corrupt subdocument arrays #14340

Closed
@juona

Description

Prerequisites

  • I have written a descriptive issue title
  • I have searched existing issues to ensure the bug has not already been reported

Mongoose version

8.1.1

Node.js version

18.18.2

MongoDB server version

5

Typescript version (if applicable)

No response

Description

Hello, me again!

  1. A document has a subdocument array.
  2. The array is modified (namely, push, pull or splice is used).
  3. The document is saved inside a transaction.
  4. The transaction is retried at least one (I mean the whole transaction as opposed to just the commit).

Result: the whole array is saved incorrectly and can even become corrupted, in turn ruining the whole document.

I noticed that in more recent versions document arrays are actually proxy objects, and my guess is that this might be causing problems, especially when using .pull()/.splice().

Steps to Reproduce

import mongoose from "mongoose";
import { MongoServerError, MongoErrorLabel } from "mongodb";

console.log("connecting...");

// Connecting to a *ReplicaSet* specifically to make sure that transactions work correctly.
// Nevertheless, whether or not an actual ReplicaSet is used has no effect on the issue.
await mongoose.connect(
  "mongodb://mongo-node1:27017,mongo-node2:27018,mongo-node3:27019/xxx?replicaSet=eof"
);

console.log("connected!");

// Define some schemas...
const subItemSchema = new mongoose.Schema(
  {
    name: { type: String, required: true },
  },
  { _id: false }
);

const itemSchema = new mongoose.Schema(
  {
    name: { type: String, required: true },
    subItems: { type: [subItemSchema], required: true },
  },
  { _id: false }
);

const schema = new mongoose.Schema({
  items: { type: [itemSchema], required: true },
});

// ...and a model
const Model = mongoose.model("MyModel", schema);

// Clear the collection...
await Model.deleteMany({});

// ...and create one document
await Model.create({
  items: [
    { name: "test1", subItems: [{ name: "x1" }] },
    { name: "test2", subItems: [{ name: "x2" }] },
  ],
});

const doc = await Model.findOne();

// Array modification - choose one...
// Works okay, but I guess that's because it's an idempotent operation
doc.items.addToSet({ name: "test3", subItems: [{ name: "x3" }] });
// Adds unexpected extra entries
doc.items.push({ name: "test3", subItems: [{ name: "x3" }] });
// Corrupts the document. It seems that moving this inside the `.transaction()` callback solves the problem
doc.items.pull(doc.items[0]);
// Corrupts the document
doc.items.splice(0, 1);

let attempt = 0;

await mongoose.connection.transaction(async (session) => {
  console.log(`Attempt: ${attempt}`);

  await doc.save({ session });

  // This is the important bit. Uncomment this to trigger a transaction retry.
  // if (attempt === 0) {
  //   attempt += 1;
  //   throw new MongoServerError({
  //     message: "Test transient transaction failures & retries",
  //     errorLabels: [MongoErrorLabel.TransientTransactionError],
  //   });
  // }
});

const updatedDoc = await Model.findOne();

// See the structure here. Depending on the array operation, the structure may be different,
// but it's always wrong.
console.log(updatedDoc.items);

// Bye bye!
await mongoose.disconnect();

Expected Behavior

If .push() is used

The total number of documents added to the array grows exponentially with every retry - it is 2^n, where n is the number of retries (forgive me, can't go without a lol here :)).

Expectation: only one document is added.

If .pull() or .splice() is used

The document that is saved in the database looks like this:

{
  "_id": {
    "$oid": "65c40f6c0975fbad509f0aa8"
  },
  "items": {
    "0": {
      "name": "test2",
      "subItems": [
        {
          "name": "x2"
        }
      ]
    }
  },
  "__v": 0
}

As you can see, the array got converted into an object, and that's why I'm throwing suspicious looks at the Proxy object used for document array.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    confirmed-bugWe've confirmed this is a bug in Mongoose and will fix it.

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions