Dear all,
I am writing again to submit this new problem to you.. As you can read from previous post I've got an MSA P2050 (Fibre Channel) system. This system initially had 8 x 2TB drives. The elements in the system were A disk group in radi 6, a pool A and one volume Vol0001. There were no Global Spare Disks.
I recently added 4 more 2TB disks, created a disk group in raid 6 and then extended the volume Vol0001 by about 4000GB.
The current configuration is as follows:
# show volumes
Pool Name Total Size Alloc Size Type Large Virtual Extents Health Reason Action
-------------------------------------------------------------------------------------
A Vol0001 15.3TB 6660.7GB base Disabled OK
-------------------------------------------------------------------------------------
Success: Command completed successfully. (2022-11-14 15:40:22)
# show pools
Name Serial Number Blocksize Total Size Avail Snap Size OverCommit Disk Groups Volumes Low Thresh Mid Thresh High Thresh Sec Fmt Health Reason Action
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
A ***removed*** 512 15.9TB 9.3TB 0B Enabled 2 1 50.00 % 75.00 % 98.65 % 512n OK
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Success: Command completed successfully. (2022-11-14 15:40:24)
# show disk-groups
Name Size Free Pool Tier % of Pool Own RAID Disks Status Current Job Job% Sec Fmt Health Reason Action
----------------------------------------------------------------------------------------------------------------------------------
dgA01 11.9TB 8657.0GB A Archive 75 A RAID6 8 FTOL 512n OK
dgA02 3994.3GB 663.9GB A Archive 24 A RAID6 4 FTOL VRSC 87% 512n OK
----------------------------------------------------------------------------------------------------------------------------------
Success: Command completed successfully. (2022-11-14 15:40:27)
# show disks
Location Serial Number Vendor Rev Description Usage Jobs Speed (kr/min) Size Sec Fmt Disk Group Pool Tier Health
------------------------------------------------------------------------------------------------------------------------------------------------------
1.1 ***removed*** SEAGATE N003 SAS MDL VIRTUAL POOL 7 2000.3GB 512n dgA01 A Archive OK
1.2 ***removed*** SEAGATE N003 SAS MDL VIRTUAL POOL 7 2000.3GB 512n dgA01 A Archive OK
1.3 ***removed*** SEAGATE N003 SAS MDL VIRTUAL POOL 7 2000.3GB 512n dgA01 A Archive OK
1.4 ***removed*** SEAGATE N003 SAS MDL VIRTUAL POOL 7 2000.3GB 512n dgA01 A Archive OK
1.5 ***removed*** SEAGATE N003 SAS MDL VIRTUAL POOL 7 2000.3GB 512n dgA01 A Archive OK
1.6 ***removed*** SEAGATE N003 SAS MDL VIRTUAL POOL 7 2000.3GB 512n dgA01 A Archive OK
1.7 ***removed*** HGST CD05 SAS MDL VIRTUAL POOL 7 2000.3GB 512n dgA01 A Archive OK
1.8 ***removed*** HGST CD05 SAS MDL VIRTUAL POOL 7 2000.3GB 512n dgA01 A Archive OK
1.9 ***removed*** HGST C907 SAS MDL VIRTUAL POOL VRSC 7 2000.3GB 512n dgA02 A Archive OK
1.10 ***removed*** HGST C907 SAS MDL VIRTUAL POOL VRSC 7 2000.3GB 512n dgA02 A Archive OK
1.11 ***removed*** HGST C907 SAS MDL VIRTUAL POOL VRSC 7 2000.3GB 512n dgA02 A Archive OK
1.12 ***removed*** HGST C907 SAS MDL VIRTUAL POOL VRSC 7 2000.3GB 512n dgA02 A Archive OK
------------------------------------------------------------------------------------------------------------------------------------------------------
Info: * Rates may vary. This is normal behavior. (2022-11-14 15:40:31)
The virtualization is made by oVirt. Hypervisor are some Proliant 580 with HBA 4/8Gbps and Fibre Channel SAN is made by two Brodace 8/20q.
Everythings works fine but when a virtual machine make I/O intensive activities oVirt display this message:
Storage domain DATA_FC_2050 experienced a high latency of 6.12239 seconds from host infn-vm05.management. This may cause performance and functional issues. Please consult your Storage Administrator.
For debugging purpose I show you some informations about P2050 activies.
These are some commands execute before I/O activities and "high latency" error.
# show host-port-statistics
Durable ID Bps IOPS Reads Writes Data Read Data Written Queue Depth I/O Resp Time Read Resp Time Write Resp Time
Reset Time
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
hostport_A1 1024B 0 128917 0 528.8MB 0B 0 206 206 0
2022-11-11 10:04:39
hostport_A2 1024B 0 128914 0 528.8MB 0B 0 217 217 0
2022-11-11 10:04:39
hostport_A3 0B 0 0 0 0B 0B 0 0 0 0
2022-11-11 10:04:39
hostport_A4 0B 0 0 0 0B 0B 0 0 0 0
2022-11-11 10:04:39
hostport_B1 184.3KB 1 515195 294675 93.4GB 194.9GB 0 1249 1414 609
2022-11-09 07:51:06
hostport_B2 6254.0KB 250 40162085 61901574 5937.1GB 1296.4GB 0 6494 13711 2281
2022-11-09 07:51:06
hostport_B3 0B 0 0 0 0B 0B 0 0 0 0
2022-11-09 07:51:06
hostport_B4 0B 0 0 0 0B 0B 0 0 0 0
2022-11-09 07:51:06
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Success: Command completed successfully. (2022-11-16 15:09:07)
# show volume-statistics
Name Serial Number Bps IOPS Reads Writes Data Read Data Written Allocated Pages % Performance % Standard
% Archive % RC Reset Time
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Vol0001 00c0ff3c465b00006510f85b01000000 2818.0KB 180 27966087 44377845 4691.6GB 1036.9GB 1588586 0 0
100 0 2022-11-11 10:04:39
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Success: Command completed successfully. (2022-11-16 15:07:48)
# show pool-statistics
Pool Pages Allocated per Min Pages Deallocated per Min Pages Unmapped per Minute
------------------------------------------------------------------------------------
A 0 0 0
Time Since Reset Reads Writes Data Read Data Written Bps IOPS I/O Resp Time Read Resp Time Write Resp Time
-----------------------------------------------------------------------------------------------------------------------
449961 20982921 12752790 5277.9GB 670.4GB 3847.1KB 134 63548 16493 217507
------------------------------------------------------------------------------------
Success: Command completed successfully. (2022-11-16 15:08:04)
# show disk-group-statistics
Name Time Since Reset Reads Writes Data Read Data Written Bps IOPS I/O Resp Time Read Resp Time Write Resp Time
------------------------------------------------------------------------------------------------------------------------------
dgA01 449961 2462399 1644910 1311.8GB 255.2GB 601.0KB 4 42433 42433 43705
Pages Allocated per Min Pages Deallocated per Min Pages Reclaimed
--------------------------------------------------------------------------------------------
0 0 0 0
dgA02 449961 18520522 11107880 3966.1GB 415.2GB 3246.0KB 130 84664 84664 391309
Pages Allocated per Min Pages Deallocated per Min Pages Reclaimed
--------------------------------------------------------------------------------------------
0 0 0 0
------------------------------------------------------------------------------------------------------------------------------
Success: Command completed successfully. (2022-11-16 15:08:24)
# show disk-statistics
Location Serial Number Pwr Hrs Bps IOPS Reads Writes Data Read Data Written Lifetime Read Lifetime Written Reset Time
----------------------------------------------------------------------------------------------------------------------------------------------------------
1.1 **removed** 35146 90.6KB 2 28674501 1253093 8177.8GB 52.1GB 0B 0B 2022-11-11 10:05:36
1.2 **removed** 35146 106.4KB 2 28822308 1311278 8183.4GB 57.7GB 0B 0B 2022-11-11 10:05:36
1.3 **removed** 35146 109.5KB 3 28782928 1339386 8183.5GB 57.9GB 0B 0B 2022-11-11 10:05:36
1.4 **removed** 35146 96.2KB 2 28727591 1235478 8177.7GB 52.2GB 0B 0B 2022-11-11 10:05:36
1.5 **removed** 35146 105.4KB 3 28782936 1326287 8183.5GB 57.8GB 0B 0B 2022-11-11 10:05:36
1.6 **removed** 35146 107.5KB 3 28785890 1336310 8183.9GB 58.2GB 0B 0B 2022-11-11 10:05:36
1.7 **removed** 34901 97.2KB 2 28739946 1218626 8177.7GB 52.1GB 0B 0B 2022-11-11 10:05:36
1.8 **removed** 34900 93.6KB 2 28689920 1243157 8177.7GB 52.1GB 0B 0B 2022-11-11 10:05:36
1.9 **removed** 1377 2973.6KB 84 26954337 8733922 9.1TB 260.6GB 0B 0B 2022-11-11 10:05:36
1.10 **removed** 996 3022.3KB 84 27056401 8759442 9.1TB 260.5GB 0B 0B 2022-11-11 10:05:36
1.11 **removed** 996 3036.6KB 82 26530303 8328735 9.1TB 258.7GB 0B 0B 2022-11-11 10:05:36
1.12 **removed** 317 3032.5KB 84 26937305 8716340 9.1TB 260.0GB 0B 0B 2022-11-11 10:05:36
----------------------------------------------------------------------------------------------------------------------------------------------------------
Success: Command completed successfully. (2022-11-16 15:08:49)
These are the other messages during I/O and "high latency" message:
# show host-port-statistics
Durable ID Bps IOPS Reads Writes Data Read Data Written Queue Depth I/O Resp Time Read Resp Time Write Resp Time
Reset Time
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
hostport_A1 1024B 0 129459 0 531.0MB 0B 0 157 157 0
2022-11-11 10:04:39
hostport_A2 1024B 0 129454 0 531.0MB 0B 0 253 253 0
2022-11-11 10:04:39
hostport_A3 0B 0 0 0 0B 0B 0 0 0 0
2022-11-11 10:04:39
hostport_A4 0B 0 0 0 0B 0B 0 0 0 0
2022-11-11 10:04:39
hostport_B1 169.6MB 256 516967 386051 93.8GB 256.8GB 0 206586 44157 207207
2022-11-09 07:51:06
hostport_B2 28.2MB 195 40307549 62133283 5950.5GB 1314.3GB 64 298826 156129 347158
2022-11-09 07:51:06
hostport_B3 0B 0 0 0 0B 0B 0 0 0 0
2022-11-09 07:51:06
hostport_B4 0B 0 0 0 0B 0B 0 0 0 0
2022-11-09 07:51:06
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Success: Command completed successfully. (2022-11-16 15:40:37)
# show volume-statistics
Name Serial Number Bps IOPS Reads Writes Data Read Data Written Allocated Pages % Performance % Standard
% Archive % RC Reset Time
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Vol0001 00c0ff3c465b00006510f85b01000000 91.8MB 302 28123750 44714117 4705.8GB 1117.1GB 1588586 0 0
100 0 2022-11-11 10:04:39
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Success: Command completed successfully. (2022-11-16 15:40:40)
# show pool-statistics
Pool Pages Allocated per Min Pages Deallocated per Min Pages Unmapped per Minute
------------------------------------------------------------------------------------
A 0 0 0
Time Since Reset Reads Writes Data Read Data Written Bps IOPS I/O Resp Time Read Resp Time Write Resp Time
---------------------------------------------------------------------------------------------------------------------
451920 21110214 12872383 5294.9GB 696.5GB 81.9MB 167 611071 40536 1043177
------------------------------------------------------------------------------------
Success: Command completed successfully. (2022-11-16 15:41:05)
# show disk-group-statistics
Name Time Since Reset Reads Writes Data Read Data Written Bps IOPS I/O Resp Time Read Resp Time Write Resp Time
----------------------------------------------------------------------------------------------------------------------------
dgA01 451890 2465240 1654643 1312.2GB 261.6GB 37.1MB 24 73652 73652 74042
Pages Allocated per Min Pages Deallocated per Min Pages Reclaimed
--------------------------------------------------------------------------------------------
0 0 0 0
dgA02 451890 18642591 11214435 3982.5GB 431.8GB 65.0MB 126 1280134 1280134 1821221
Pages Allocated per Min Pages Deallocated per Min Pages Reclaimed
--------------------------------------------------------------------------------------------
0 0 0 0
----------------------------------------------------------------------------------------------------------------------------
Success: Command completed successfully. (2022-11-16 15:40:11)
# show disk-statistics
Location Serial Number Pwr Hrs Bps IOPS Reads Writes Data Read Data Written Lifetime Read Lifetime Written Reset Time
----------------------------------------------------------------------------------------------------------------------------------------------------------
1.1 **removed** 35147 7130.1KB 114 28682336 1275624 8178.1GB 53.5GB 0B 0B 2022-11-11 10:05:36
1.2 **removed** 35147 8613.3KB 134 28832819 1336110 8183.9GB 59.2GB 0B 0B 2022-11-11 10:05:36
1.3 **removed** 35147 8595.4KB 135 28793852 1364809 8184.0GB 59.5GB 0B 0B 2022-11-11 10:05:36
1.4 **removed** 35147 7045.6KB 112 28735170 1257788 8178.0GB 53.6GB 0B 0B 2022-11-11 10:05:36
1.5 **removed** 35147 8681.4KB 136 28793715 1351443 8184.0GB 59.4GB 0B 0B 2022-11-11 10:05:36
1.6 **removed** 35147 8745.9KB 138 28797059 1361798 8184.4GB 59.8GB 0B 0B 2022-11-11 10:05:36
1.7 **removed** 34901 7166.9KB 116 28747777 1241330 8178.0GB 53.4GB 0B 0B 2022-11-11 10:05:36
1.8 **removed** 34901 7159.8KB 113 28697687 1265747 8178.1GB 53.5GB 0B 0B 2022-11-11 10:05:36
1.9 **removed** 1377 35.5MB 187 27072767 8829172 9.1TB 270.8GB 0B 0B 2022-11-11 10:05:36
1.10 **removed** 996 35.4MB 189 27175925 8855164 9.1TB 270.7GB 0B 0B 2022-11-11 10:05:36
1.11 **removed** 996 35.0MB 183 26644278 8420356 9.1TB 268.8GB 0B 0B 2022-11-11 10:05:36
1.12 **removed** 318 35.5MB 188 27056039 8811311 9.1TB 270.2GB 0B 0B 2022-11-11 10:05:36
----------------------------------------------------------------------------------------------------------------------------------------------------------
Success: Command completed successfully. (2022-11-16 15:40:45)
With "show events" there aren't any messages.
Any suggestions ?
Thanks again
Best Regards
Willy
...
..