Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't create a diskpool #3807

Closed
mattlqx opened this issue Nov 18, 2024 · 7 comments
Closed

Can't create a diskpool #3807

mattlqx opened this issue Nov 18, 2024 · 7 comments

Comments

@mattlqx
Copy link

mattlqx commented Nov 18, 2024

Description

I've installed OpenEBS via helm chart with mayastor enabled. I'm attempting to create a DiskPool (actually many across 3 hosts but can't get a single one working so let's just start there). I've tried specifying with blk_size and with aio:// nothing seems to matter much. The disk does have max activity for a minute or so, so it appears like it's hitting a request timeout but the operation continues until its done and then deletes and tries again.

Expected Behavior

DiskPool to go through Creating to Online.

Current Behavior

DiskPool always shows as Creating.

Log of io-engine on the node is showing gRPC timeouts, successes and deletions in a loop.

[2024-11-18T01:09:48.384827920+00:00  INFO io_engine::lvs::lvs_store:lvs_store.rs:508] Creating or importing lvs 'macro-1' from 'aio:///dev/disk/by-id/scsi-35000cca25905dbd0'...     
[2024-11-18T01:09:48.384878553+00:00 ERROR mayastor::spdk:bdev_aio.c:754] Specified block size 512 is smaller than auto-detected block size 4096                                      
[2024-11-18T01:09:48.384902779+00:00 ERROR io_engine::bdev::aio:aio.rs:118] Aio '/dev/disk/by-id/scsi-35000cca25905dbd0' error: Failed to create a BDEV '/dev/disk/by-id/scsi-35000cca
25905dbd0': EINVAL: Invalid argument                                                                                                                                                  
[2024-11-18T01:09:48.384908045+00:00 ERROR io_engine::lvs::lvs_store:lvs_store.rs:541] Failed to create pool bdev: CreateBdevFailed { source: EINVAL, name: "/dev/disk/by-id/scsi-3500
0cca25905dbd0" }                                                                                                                                                                      
[2024-11-18T01:09:48.384914360+00:00 ERROR io_engine::lvs::lvs_store:lvs_store.rs:504] error=InvalidBdev: Failed to create a BDEV '/dev/disk/by-id/scsi-35000cca25905dbd0'            
[2024-11-18T01:10:08.882759729+00:00  INFO io_engine::grpc::v1::pool:pool.rs:508] CreatePoolRequest { name: "macro-1", uuid: None, disks: ["/dev/disk/by-id/scsi-35000cca25905dbd0"], 
pooltype: Lvs, cluster_size: None }                                                                                                                                                   
[2024-11-18T01:10:08.882835235+00:00  INFO io_engine::lvs::lvs_store:lvs_store.rs:508] Creating or importing lvs 'macro-1' from 'aio:///dev/disk/by-id/scsi-35000cca25905dbd0'...     
[2024-11-18T01:10:08.882885725+00:00 ERROR mayastor::spdk:bdev_aio.c:754] Specified block size 512 is smaller than auto-detected block size 4096                                      
[2024-11-18T01:10:08.882909425+00:00 ERROR io_engine::bdev::aio:aio.rs:118] Aio '/dev/disk/by-id/scsi-35000cca25905dbd0' error: Failed to create a BDEV '/dev/disk/by-id/scsi-35000cca
25905dbd0': EINVAL: Invalid argument                                                                                                                                                  
[2024-11-18T01:10:08.882916090+00:00 ERROR io_engine::lvs::lvs_store:lvs_store.rs:541] Failed to create pool bdev: CreateBdevFailed { source: EINVAL, name: "/dev/disk/by-id/scsi-3500
0cca25905dbd0" }                                                                                                                                                                      
[2024-11-18T01:10:08.882922420+00:00 ERROR io_engine::lvs::lvs_store:lvs_store.rs:504] error=InvalidBdev: Failed to create a BDEV '/dev/disk/by-id/scsi-35000cca25905dbd0'            
[2024-11-18T01:10:29.397039762+00:00  INFO io_engine::grpc::v1::pool:pool.rs:508] CreatePoolRequest { name: "macro-1", uuid: None, disks: ["/dev/disk/by-id/scsi-35000cca25905dbd0"], 
pooltype: Lvs, cluster_size: None }                                                                                                                                                   
[2024-11-18T01:10:29.397114002+00:00  INFO io_engine::lvs::lvs_store:lvs_store.rs:508] Creating or importing lvs 'macro-1' from 'aio:///dev/disk/by-id/scsi-35000cca25905dbd0'...     
[2024-11-18T01:10:29.397164114+00:00 ERROR mayastor::spdk:bdev_aio.c:754] Specified block size 512 is smaller than auto-detected block size 4096                                      
[2024-11-18T01:10:29.397191588+00:00 ERROR io_engine::bdev::aio:aio.rs:118] Aio '/dev/disk/by-id/scsi-35000cca25905dbd0' error: Failed to create a BDEV '/dev/disk/by-id/scsi-35000cca
25905dbd0': EINVAL: Invalid argument                                                                                                                                                  
[2024-11-18T01:10:29.397198484+00:00 ERROR io_engine::lvs::lvs_store:lvs_store.rs:541] Failed to create pool bdev: CreateBdevFailed { source: EINVAL, name: "/dev/disk/by-id/scsi-3500
0cca25905dbd0" }                                                                                                                                                                      
[2024-11-18T01:10:29.397205252+00:00 ERROR io_engine::lvs::lvs_store:lvs_store.rs:504] error=InvalidBdev: Failed to create a BDEV '/dev/disk/by-id/scsi-35000cca25905dbd0'            
[2024-11-18T01:10:49.863861623+00:00  INFO io_engine::grpc::v1::pool:pool.rs:508] CreatePoolRequest { name: "macro-1", uuid: None, disks: ["/dev/disk/by-id/scsi-35000cca25905dbd0"], 
pooltype: Lvs, cluster_size: None }                                                                                                                                                   
[2024-11-18T01:10:49.863936658+00:00  INFO io_engine::lvs::lvs_store:lvs_store.rs:508] Creating or importing lvs 'macro-1' from 'aio:///dev/disk/by-id/scsi-35000cca25905dbd0'...     
[2024-11-18T01:10:49.863985101+00:00 ERROR mayastor::spdk:bdev_aio.c:754] Specified block size 512 is smaller than auto-detected block size 4096                                      
[2024-11-18T01:10:49.864009172+00:00 ERROR io_engine::bdev::aio:aio.rs:118] Aio '/dev/disk/by-id/scsi-35000cca25905dbd0' error: Failed to create a BDEV '/dev/disk/by-id/scsi-35000cca
25905dbd0': EINVAL: Invalid argument                                                                                                                                                  
[2024-11-18T01:10:49.864015127+00:00 ERROR io_engine::lvs::lvs_store:lvs_store.rs:541] Failed to create pool bdev: CreateBdevFailed { source: EINVAL, name: "/dev/disk/by-id/scsi-3500
0cca25905dbd0" }                                                                                                                                                                      
[2024-11-18T01:10:49.864021253+00:00 ERROR io_engine::lvs::lvs_store:lvs_store.rs:504] error=InvalidBdev: Failed to create a BDEV '/dev/disk/by-id/scsi-35000cca25905dbd0'

with this manifest:

---
apiVersion: "openebs.io/v1beta2"
kind: DiskPool
metadata:
  name: macro-1
  namespace: openebs
spec:
  node: macro
  disks:
    - /dev/disk/by-id/scsi-35000cca25905dbd0

Events on the Diskpool:

2m30s       Critical   Failure   DiskPool/macro-1   Unable to create or import pool error in response: status code '400 Bad Request', content: 'RestJsonError { details: "create_pool::status: InvalidArgument, message: \"Failed to create a BDEV '/dev/disk/by-id/scsi-35000cca25905dbd0': EINVAL: Invalid argument\", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"date\": \"Mon, 18 Nov 2024 01:19:02 GMT\", \"content-length\": \"0\"} }", message: "SvcError::GrpcRequestError", kind: InvalidArgument }'
2m10s       Critical   Failure   DiskPool/macro-1   Unable to create or import pool error in response: status code '400 Bad Request', content: 'RestJsonError { details: "create_pool::status: InvalidArgument, message: \"Failed to create a BDEV '/dev/disk/by-id/scsi-35000cca25905dbd0': EINVAL: Invalid argument\", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"date\": \"Mon, 18 Nov 2024 01:19:22 GMT\", \"content-length\": \"0\"} }", message: "SvcError::GrpcRequestError", kind: InvalidArgument }'
109s        Critical   Failure   DiskPool/macro-1   Unable to create or import pool error in response: status code '400 Bad Request', content: 'RestJsonError { details: "create_pool::status: InvalidArgument, message: \"Failed to create a BDEV '/dev/disk/by-id/scsi-35000cca25905dbd0': EINVAL: Invalid argument\", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"date\": \"Mon, 18 Nov 2024 01:19:43 GMT\", \"content-length\": \"0\"} }", message: "SvcError::GrpcRequestError", kind: InvalidArgument }'
89s         Critical   Failure   DiskPool/macro-1   Unable to create or import pool error in response: status code '400 Bad Request', content: 'RestJsonError { details: "create_pool::status: InvalidArgument, message: \"Failed to create a BDEV '/dev/disk/by-id/scsi-35000cca25905dbd0': EINVAL: Invalid argument\", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"date\": \"Mon, 18 Nov 2024 01:20:03 GMT\", \"content-length\": \"0\"} }", message: "SvcError::GrpcRequestError", kind: InvalidArgument }'
68s         Critical   Failure   DiskPool/macro-1   Unable to create or import pool error in response: status code '400 Bad Request', content: 'RestJsonError { details: "create_pool::status: InvalidArgument, message: \"Failed to create a BDEV '/dev/disk/by-id/scsi-35000cca25905dbd0': EINVAL: Invalid argument\", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"date\": \"Mon, 18 Nov 2024 01:20:24 GMT\", \"content-length\": \"0\"} }", message: "SvcError::GrpcRequestError", kind: InvalidArgument }'
47s         Critical   Failure   DiskPool/macro-1   Unable to create or import pool error in response: status code '400 Bad Request', content: 'RestJsonError { details: "create_pool::status: InvalidArgument, message: \"Failed to create a BDEV '/dev/disk/by-id/scsi-35000cca25905dbd0': EINVAL: Invalid argument\", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"date\": \"Mon, 18 Nov 2024 01:20:44 GMT\", \"content-length\": \"0\"} }", message: "SvcError::GrpcRequestError", kind: InvalidArgument }'
27s         Critical   Failure   DiskPool/macro-1   Unable to create or import pool error in response: status code '400 Bad Request', content: 'RestJsonError { details: "create_pool::status: InvalidArgument, message: \"Failed to create a BDEV '/dev/disk/by-id/scsi-35000cca25905dbd0': EINVAL: Invalid argument\", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"date\": \"Mon, 18 Nov 2024 01:21:05 GMT\", \"content-length\": \"0\"} }", message: "SvcError::GrpcRequestError", kind: InvalidArgument }'
6s          Critical   Failure   DiskPool/macro-1   Unable to create or import pool error in response: status code '400 Bad Request', content: 'RestJsonError { details: "create_pool::status: InvalidArgument, message: \"Failed to create a BDEV '/dev/disk/by-id/scsi-35000cca25905dbd0': EINVAL: Invalid argument\", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"date\": \"Mon, 18 Nov 2024 01:21:25 GMT\", \"content-length\": \"0\"} }", message: "SvcError::GrpcRequestError", kind: InvalidArgument }'

Your Environment
@tiagolobocastro
Copy link
Contributor

IIRC @dsharma-dc did you fix this recently?
@mattlqx you mentioned you tweaked the blk_size, did you delete the pool CR or modified the spec? Currently we don't support modifying the spec. (also as another improvement maybe we should allow modifying the spec as long as the pool is not successfully created)

@dsharma-dc
Copy link

dsharma-dc commented Nov 18, 2024 •
edited
Loading

@mattlqx Which Mayastor version is this?
This should work after deleting the CR and with blk_size specified as 4096 to create again.

@mattlqx
Copy link
Author

mattlqx commented Nov 18, 2024

Yeah, I've been deleting the DiskPool every time. I've also wipefs'd but the drive is already clean.

Running openebs/mayastor-csi-node:v2.7.1.

@dsharma-dc
Copy link

Yeah, I've been deleting the DiskPool every time. I've also wipefs'd but the drive is already clean.

Running openebs/mayastor-csi-node:v2.7.1.

Can you share the io-engine logs from when you set blk_size to 4096 and pool doesn't create?

@mattlqx
Copy link
Author

mattlqx commented Nov 18, 2024 •
edited
Loading

Sure...

[2024-11-18T17:41:28.046422421+00:00  INFO io_engine::grpc::v1::pool:pool.rs:508] CreatePoolRequest { name: "macro-1", uuid: None, disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"], pooltype: Lvs, cluster_size: None }
[2024-11-18T17:41:28.046529943+00:00  INFO io_engine::lvs::lvs_store:lvs_store.rs:508] Creating or importing lvs 'macro-1' from 'aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096'...
[2024-11-18T17:41:28.050477195+00:00 ERROR io_engine::lvs::lvs_store:lvs_store.rs:335] error=: cannot import LVS, failed to import pool macro-1
[2024-11-18T17:41:48.048931807+00:00  WARN io_engine::grpc::v1::pool:pool.rs:88] create_pool: gRPC method timed out, args: CreatePoolRequest { name: "macro-1", uuid: None, disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"], pooltype: Lvs, cluster_size: None }
[2024-11-18T17:41:51.439242173+00:00  INFO io_engine::grpc::v1::pool:pool.rs:531] DestroyPoolRequest { name: "macro-1", uuid: None }
[2024-11-18T17:41:51.441790546+00:00  INFO io_engine::grpc::v1::pool:pool.rs:571] ImportPoolRequest { name: "macro-1", uuid: None, disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"], pooltype: Lvs }
[2024-11-18T17:42:01.456944927+00:00  INFO io_engine::grpc::v1::pool:pool.rs:531] DestroyPoolRequest { name: "macro-1", uuid: None }
[2024-11-18T17:42:01.458370329+00:00  INFO io_engine::grpc::v1::pool:pool.rs:571] ImportPoolRequest { name: "macro-1", uuid: None, disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"], pooltype: Lvs }
[2024-11-18T17:42:11.474681845+00:00  INFO io_engine::grpc::v1::pool:pool.rs:531] DestroyPoolRequest { name: "macro-1", uuid: None }
[2024-11-18T17:42:11.476143100+00:00  INFO io_engine::grpc::v1::pool:pool.rs:571] ImportPoolRequest { name: "macro-1", uuid: None, disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"], pooltype: Lvs }
[2024-11-18T17:42:19.298072829+00:00  INFO io_engine::lvs::lvs_store:lvs_store.rs:493] Lvs 'macro-1' [/dev/disk/by-id/scsi-35000cca25905dbd0/3e20ed1e-0ffd-45dd-a2f6-937291e30e85] (12.72 TiB/12.72 TiB): new lvs created successfully
[2024-11-18T17:42:19.298233916+00:00 ERROR io_engine::core::reactor:reactor.rs:605] Failed to send response future result Ok(Pool { uuid: "1ad5a0b5-96b6-48de-a0fb-882c942e79e6", name: "macro-1", disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096&uuid=3e20ed1e-0ffd-45dd-a2f6-937291e30e85"], state: PoolOnline, capacity: 13986842017792, used: 0, pooltype: Lvs, committed: 0, cluster_size: 4194304 })
[2024-11-18T17:42:21.493372545+00:00  INFO io_engine::grpc::v1::pool:pool.rs:531] DestroyPoolRequest { name: "macro-1", uuid: None }
[2024-11-18T17:42:21.493446081+00:00  INFO io_engine::lvs::lvs_store:lvs_store.rs:681] Lvs 'macro-1' [/dev/disk/by-id/scsi-35000cca25905dbd0/3e20ed1e-0ffd-45dd-a2f6-937291e30e85] (12.72 TiB/12.72 TiB): destroying lvs...
[2024-11-18T17:42:21.494431329+00:00  INFO io_engine::lvs::lvs_store:lvs_store.rs:709] Lvs 'macro-1' [/dev/disk/by-id/scsi-35000cca25905dbd0/3e20ed1e-0ffd-45dd-a2f6-937291e30e85] (12.72 TiB/12.72 TiB): lvs destroyed successfully
[2024-11-18T17:42:21.494494557+00:00  INFO io_engine::bdev_api:bdev_api.rs:111] destroy uri="aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"
[2024-11-18T17:42:34.449373565+00:00  INFO io_engine::grpc::v1::pool:pool.rs:508] CreatePoolRequest { name: "macro-1", uuid: None, disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"], pooltype: Lvs, cluster_size: None }
[2024-11-18T17:42:34.449460893+00:00  INFO io_engine::lvs::lvs_store:lvs_store.rs:508] Creating or importing lvs 'macro-1' from 'aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096'...
[2024-11-18T17:42:34.450754992+00:00 ERROR io_engine::lvs::lvs_store:lvs_store.rs:335] error=: cannot import LVS, failed to import pool macro-1
[2024-11-18T17:42:54.451594713+00:00  WARN io_engine::grpc::v1::pool:pool.rs:88] create_pool: gRPC method timed out, args: CreatePoolRequest { name: "macro-1", uuid: None, disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"], pooltype: Lvs, cluster_size: None }
[2024-11-18T17:43:21.533543543+00:00  INFO io_engine::grpc::v1::pool:pool.rs:531] DestroyPoolRequest { name: "macro-1", uuid: None }
[2024-11-18T17:43:21.534781064+00:00  INFO io_engine::grpc::v1::pool:pool.rs:571] ImportPoolRequest { name: "macro-1", uuid: None, disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"], pooltype: Lvs }
[2024-11-18T17:43:25.849085121+00:00  INFO io_engine::lvs::lvs_store:lvs_store.rs:493] Lvs 'macro-1' [/dev/disk/by-id/scsi-35000cca25905dbd0/6e43414d-d4f3-444e-ba9a-ccd4e365ab21] (12.72 TiB/12.72 TiB): new lvs created successfully
[2024-11-18T17:43:25.849235670+00:00 ERROR io_engine::core::reactor:reactor.rs:605] Failed to send response future result Ok(Pool { uuid: "e25a188f-39c1-469d-bd59-2e5ffc4a6af5", name: "macro-1", disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096&uuid=6e43414d-d4f3-444e-ba9a-ccd4e365ab21"], state: PoolOnline, capacity: 13986842017792, used: 0, pooltype: Lvs, committed: 0, cluster_size: 4194304 })
[2024-11-18T17:43:31.548680611+00:00  INFO io_engine::grpc::v1::pool:pool.rs:531] DestroyPoolRequest { name: "macro-1", uuid: None }
[2024-11-18T17:43:31.548761085+00:00  INFO io_engine::lvs::lvs_store:lvs_store.rs:681] Lvs 'macro-1' [/dev/disk/by-id/scsi-35000cca25905dbd0/6e43414d-d4f3-444e-ba9a-ccd4e365ab21] (12.72 TiB/12.72 TiB): destroying lvs...
[2024-11-18T17:43:31.549746057+00:00  INFO io_engine::lvs::lvs_store:lvs_store.rs:709] Lvs 'macro-1' [/dev/disk/by-id/scsi-35000cca25905dbd0/6e43414d-d4f3-444e-ba9a-ccd4e365ab21] (12.72 TiB/12.72 TiB): lvs destroyed successfully
[2024-11-18T17:43:31.549820090+00:00  INFO io_engine::bdev_api:bdev_api.rs:111] destroy uri="aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"
[2024-11-18T17:43:40.967077394+00:00  INFO io_engine::grpc::v1::pool:pool.rs:508] CreatePoolRequest { name: "macro-1", uuid: None, disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"], pooltype: Lvs, cluster_size: None }
[2024-11-18T17:43:40.967165190+00:00  INFO io_engine::lvs::lvs_store:lvs_store.rs:508] Creating or importing lvs 'macro-1' from 'aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096'...
[2024-11-18T17:43:40.968419021+00:00 ERROR io_engine::lvs::lvs_store:lvs_store.rs:335] error=: cannot import LVS, failed to import pool macro-1
[2024-11-18T17:44:00.968572760+00:00  WARN io_engine::grpc::v1::pool:pool.rs:88] create_pool: gRPC method timed out, args: CreatePoolRequest { name: "macro-1", uuid: None, disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"], pooltype: Lvs, cluster_size: None }
[2024-11-18T17:44:01.601720257+00:00  INFO io_engine::grpc::v1::pool:pool.rs:531] DestroyPoolRequest { name: "macro-1", uuid: None }
[2024-11-18T17:44:01.603190293+00:00  INFO io_engine::grpc::v1::pool:pool.rs:571] ImportPoolRequest { name: "macro-1", uuid: None, disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"], pooltype: Lvs }
[2024-11-18T17:44:11.618359544+00:00  INFO io_engine::grpc::v1::pool:pool.rs:531] DestroyPoolRequest { name: "macro-1", uuid: None }
[2024-11-18T17:44:11.619805892+00:00  INFO io_engine::grpc::v1::pool:pool.rs:571] ImportPoolRequest { name: "macro-1", uuid: None, disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"], pooltype: Lvs }
[2024-11-18T17:44:21.638333009+00:00  INFO io_engine::grpc::v1::pool:pool.rs:531] DestroyPoolRequest { name: "macro-1", uuid: None }
[2024-11-18T17:44:21.639936735+00:00  INFO io_engine::grpc::v1::pool:pool.rs:571] ImportPoolRequest { name: "macro-1", uuid: None, disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"], pooltype: Lvs }
[2024-11-18T17:44:31.654369933+00:00  INFO io_engine::grpc::v1::pool:pool.rs:531] DestroyPoolRequest { name: "macro-1", uuid: None }
[2024-11-18T17:44:31.655801830+00:00  INFO io_engine::grpc::v1::pool:pool.rs:571] ImportPoolRequest { name: "macro-1", uuid: None, disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"], pooltype: Lvs }
[2024-11-18T17:44:32.250092603+00:00  INFO io_engine::lvs::lvs_store:lvs_store.rs:493] Lvs 'macro-1' [/dev/disk/by-id/scsi-35000cca25905dbd0/7dc69ffa-79e5-4653-bd99-d00c841d9a26] (12.72 TiB/12.72 TiB): new lvs created successfully
[2024-11-18T17:44:32.250218449+00:00 ERROR io_engine::core::reactor:reactor.rs:605] Failed to send response future result Ok(Pool { uuid: "d5c2e103-3575-4ac1-ab1f-e79646c6336c", name: "macro-1", disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096&uuid=7dc69ffa-79e5-4653-bd99-d00c841d9a26"], state: PoolOnline, capacity: 13986842017792, used: 0, pooltype: Lvs, committed: 0, cluster_size: 4194304 })
[2024-11-18T17:44:41.671967426+00:00  INFO io_engine::grpc::v1::pool:pool.rs:531] DestroyPoolRequest { name: "macro-1", uuid: None }
[2024-11-18T17:44:41.672047707+00:00  INFO io_engine::lvs::lvs_store:lvs_store.rs:681] Lvs 'macro-1' [/dev/disk/by-id/scsi-35000cca25905dbd0/7dc69ffa-79e5-4653-bd99-d00c841d9a26] (12.72 TiB/12.72 TiB): destroying lvs...
[2024-11-18T17:44:41.673065418+00:00  INFO io_engine::lvs::lvs_store:lvs_store.rs:709] Lvs 'macro-1' [/dev/disk/by-id/scsi-35000cca25905dbd0/7dc69ffa-79e5-4653-bd99-d00c841d9a26] (12.72 TiB/12.72 TiB): lvs destroyed successfully
[2024-11-18T17:44:41.673138173+00:00  INFO io_engine::bdev_api:bdev_api.rs:111] destroy uri="aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"
[2024-11-18T17:44:47.443166540+00:00  INFO io_engine::grpc::v1::pool:pool.rs:508] CreatePoolRequest { name: "macro-1", uuid: None, disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"], pooltype: Lvs, cluster_size: None }
[2024-11-18T17:44:47.443285870+00:00  INFO io_engine::lvs::lvs_store:lvs_store.rs:508] Creating or importing lvs 'macro-1' from 'aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096'...
[2024-11-18T17:44:47.444473826+00:00 ERROR io_engine::lvs::lvs_store:lvs_store.rs:335] error=: cannot import LVS, failed to import pool macro-1
[2024-11-18T17:45:07.444735619+00:00  WARN io_engine::grpc::v1::pool:pool.rs:88] create_pool: gRPC method timed out, args: CreatePoolRequest { name: "macro-1", uuid: None, disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"], pooltype: Lvs, cluster_size: None }
[2024-11-18T17:45:11.711919593+00:00  INFO io_engine::grpc::v1::pool:pool.rs:531] DestroyPoolRequest { name: "macro-1", uuid: None }
[2024-11-18T17:45:11.713381755+00:00  INFO io_engine::grpc::v1::pool:pool.rs:571] ImportPoolRequest { name: "macro-1", uuid: None, disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"], pooltype: Lvs }
[2024-11-18T17:45:21.732765239+00:00  INFO io_engine::grpc::v1::pool:pool.rs:531] DestroyPoolRequest { name: "macro-1", uuid: None }
[2024-11-18T17:45:21.734381410+00:00  INFO io_engine::grpc::v1::pool:pool.rs:571] ImportPoolRequest { name: "macro-1", uuid: None, disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"], pooltype: Lvs }
[2024-11-18T17:45:31.749474447+00:00  INFO io_engine::grpc::v1::pool:pool.rs:531] DestroyPoolRequest { name: "macro-1", uuid: None }
[2024-11-18T17:45:31.750900009+00:00  INFO io_engine::grpc::v1::pool:pool.rs:571] ImportPoolRequest { name: "macro-1", uuid: None, disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"], pooltype: Lvs }
[2024-11-18T17:45:38.651090177+00:00  INFO io_engine::lvs::lvs_store:lvs_store.rs:493] Lvs 'macro-1' [/dev/disk/by-id/scsi-35000cca25905dbd0/790734b5-f42f-4c3d-8d7c-b8a8230bdea8] (12.72 TiB/12.72 TiB): new lvs created successfully
[2024-11-18T17:45:38.651243677+00:00 ERROR io_engine::core::reactor:reactor.rs:605] Failed to send response future result Ok(Pool { uuid: "4d93f9b1-1ff2-4a4e-8f59-2b5fee96c8fe", name: "macro-1", disks: ["aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096&uuid=790734b5-f42f-4c3d-8d7c-b8a8230bdea8"], state: PoolOnline, capacity: 13986842017792, used: 0, pooltype: Lvs, committed: 0, cluster_size: 4194304 })
[2024-11-18T17:45:41.772408776+00:00  INFO io_engine::grpc::v1::pool:pool.rs:531] DestroyPoolRequest { name: "macro-1", uuid: None }
[2024-11-18T17:45:41.772494293+00:00  INFO io_engine::lvs::lvs_store:lvs_store.rs:681] Lvs 'macro-1' [/dev/disk/by-id/scsi-35000cca25905dbd0/790734b5-f42f-4c3d-8d7c-b8a8230bdea8] (12.72 TiB/12.72 TiB): destroying lvs...
[2024-11-18T17:45:41.773495212+00:00  INFO io_engine::lvs::lvs_store:lvs_store.rs:709] Lvs 'macro-1' [/dev/disk/by-id/scsi-35000cca25905dbd0/790734b5-f42f-4c3d-8d7c-b8a8230bdea8] (12.72 TiB/12.72 TiB): lvs destroyed successfully
[2024-11-18T17:45:41.773553354+00:00  INFO io_engine::bdev_api:bdev_api.rs:111] destroy uri="aio:///dev/disk/by-id/scsi-35000cca25905dbd0?blk_size=4096"

@dsharma-dc
Copy link

ok, it's not the issue with blk_size anymore. This looks like a case of pool taking longer to create and meanwhile the requests times out. This could happen sometimes when the pool size to be created is quite large, which in this case is pretty big.
Could you try adding these parameters to core-agent deployment and that should resolve issue by allowing a longer timeout for request.

- --no-min-timeouts
- --request-timeout=60s

We have an existing ticket for fixing this: #3805

@mattlqx
Copy link
Author

mattlqx commented Nov 19, 2024

Thank you, that did indeed fix it.

@mattlqx mattlqx closed this as completed Nov 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mattlqx @tiagolobocastro @dsharma-dc