Performance of using splice(2) for TCP to File case

tokers

unread,

Dec 4, 2024, 3:34:02 AM12/4/24

to golang-nuts

Hi,

I noticed (*os.File).ReadFrom has supported splice(2) since go/1.21, and I'm trying to introduce it to my project. But the performance is not as expected, which is slower than the normal io.Copy (buffer size = 1MB). Does anyone who knows the reason?

For using splice(2), I tweaked the parameters:

echo 655360 > /proc/sys/fs/pipe-user-pages-soft

Some snippets for strace for splice(2):

[pid 2663045] splice(316, NULL, 48, NULL, 38920, SPLICE_F_NONBLOCK <unfinished ...>
[pid 2663045] <... splice resumed> ) = 38920
[pid 2663045] splice(222, NULL, 317, NULL, 199008, SPLICE_F_NONBLOCK <unfinished ...>
[pid 2663045] <... splice resumed> ) = 80620
[pid 2663045] splice(316, NULL, 48, NULL, 80620, SPLICE_F_NONBLOCK <unfinished ...>
[pid 2663045] <... splice resumed> ) = 80620
[pid 2663045] splice(222, NULL, 317, NULL, 118388, SPLICE_F_NONBLOCK <unfinished ...>
[pid 2663045] <... splice resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
[pid 2663045] splice(222, NULL, 317, NULL, 118388, SPLICE_F_NONBLOCK <unfinished ...>
[pid 2663045] <... splice resumed> ) = 14080
[pid 2663045] splice(316, NULL, 48, NULL, 14080, SPLICE_F_NONBLOCK <unfinished ...>
[pid 2663045] <... splice resumed> ) = 14080
[pid 2663045] splice(222, NULL, 317, NULL, 104308, SPLICE_F_NONBLOCK <unfinished ...>
[pid 2663045] <... splice resumed> ) = 26230
[pid 2663045] splice(316, NULL, 48, NULL, 26230, SPLICE_F_NONBLOCK <unfinished ...>
[pid 2663045] <... splice resumed> ) = 26230
[pid 2663045] splice(222, NULL, 317, NULL, 78078, SPLICE_F_NONBLOCK <unfinished ...>
[pid 2663045] <... splice resumed> ) = 19750
[pid 2663045] splice(316, NULL, 48, NULL, 19750, SPLICE_F_NONBLOCK <unfinished ...>
[pid 2663045] <... splice resumed> ) = 19750
[pid 2663045] splice(222, NULL, 317, NULL, 58328, SPLICE_F_NONBLOCK <unfinished ...>
[pid 2663045] <... splice resumed> ) = 58328
[pid 2663045] splice(316, NULL, 48, NULL, 58328, SPLICE_F_NONBLOCK <unfinished ...>
[pid 2663045] <... splice resumed> ) = 58328

Kurtis Rader

unread,

Dec 4, 2024, 5:57:35 PM12/4/24

to tokers, golang-nuts

On Wed, Dec 4, 2024 at 12:33 AM tokers <zcha...@gmail.com> wrote:

I noticed (*os.File).ReadFrom has supported splice(2) since go/1.21, and I'm trying to introduce it to my project. But the performance is not as expected, which is slower than the normal io.Copy (buffer size = 1MB). Does anyone who knows the reason?

For using splice(2), I tweaked the parameters:

echo 655360 > /proc/sys/fs/pipe-user-pages-soft

That sets the upper limit on how much data a kernel pipe can buffer to 2.5 GiB. That is not reasonable. It is also an upper limit, not the default size. Also, that kernel parameter only existed, AFAIK, in Linux 2.6.34 which was released in 2010. Are you really using a Linux kernel that old?

--

Kurtis Rader

Caretaker of the exceptional canines Junior and Hank

Chao Zhang

unread,

Dec 4, 2024, 9:29:07 PM12/4/24

to Kurtis Rader, golang-nuts

Hi,

> That sets the upper limit on how much data a kernel pipe can buffer to 2.5 GiB. That is not reasonable. It is also an upper limit, not the default size. Also, that kernel parameter only existed, AFAIK, in Linux 2.6.34 which was released in 2010. Are you really using a Linux kernel that old?

My kernel version is 5.4 and besides this parameter, the pipe-max-size
is 1MB, and I know that the pipe created by the golang runtime has a
hard-coded size, which is also 1MB.

Best regards
Chao Zhang

https://github.com/tokers

Robert Engels

unread,

Dec 4, 2024, 9:41:15 PM12/4/24

to Chao Zhang, Kurtis Rader, golang-nuts

Normally the pipe size doesn’t need to be very large if the producer and consumer “costs” are roughly the same. If the producer is costlier than the consumer than a large buffer does nothing. If the consumer is costlier than a large buffer only allow the producer to complete sooner. Otherwise, you only need to make the buffer large enough to overcome the cost of the system call.

Eg if the minimal read cost is 6 usec, you need to have the buffer be large enough that the processing costs 600 usecs - to make the IO less than 1%. It is very hard to “process” 2.5gb in 600 usecs.

> On Dec 4, 2024, at 8:28 PM, Chao Zhang <zcha...@gmail.com> wrote:
>
> Hi,

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/CAPr95Bih_tnPc8a3cUiJNE6%3DiVbg61WYh50x5vGqr5JgYqbLPA%40mail.gmail.com.

tokers

unread,

Dec 5, 2024, 12:51:56 AM12/5/24

to golang-nuts

> Normally the pipe size doesn’t need to be very large if the producer and consumer “costs” are roughly the same. If the producer is costlier than the consumer than a large buffer does nothing. If the consumer is costlier than a large buffer only allow the producer to complete sooner. Otherwise, you only need to make the buffer large enough to overcome > the cost of the system call.

> Eg if the minimal read cost is 6 usec, you need to have the buffer be large enough that the processing costs 600 usecs - to make the IO less than 1%. It is very hard to “process” 2.5gb in 600 usecs.

pipe-user-pages-soft is just a limitation, it doesn't means in my case there will have such number of pages allocated. Actually, I just use a file which size 2MB in the benchmark (downloading from another service, and write to the local disk with splice).

Reply all

Reply to author

Forward