Edit: I didn't realize this was a graph process. If you're using the full version and a single job fails, you can retry that job look for the orchestration cookbook in the docs.

Given that this is a JSON file, it might make sense for you to upload entries as chunks rather than a single document. The embedding requests are sent in batches with exponential back off, though, so I suspect that eventually this will succeed. If you're using the full version, and it fails, you can always retry the job which is especially helpful when you've broken the file up or have many files.

Even using Hatchet with smaller chunks it can technically run into same rate limit issue, no?
Is there any configuration to apply this rate limiting in hatchet queues?

I think what you're looking for then is the batch_size parameter in the configuration file. The default is 256. Changing this would only impact future graphs, though.

Unfortunately lowering batch_size did not help and instead it overloaded the container on cpu with tons of attempts to retry and fail. There should be a better way rather than brute forcing and hoping it eventually processes everything.