I have an MSA 2040 SAN with dual controller and a second enclosure. They have worked fine since I installed them in late 2014. Tuesday, the SAN crashed, and the primary enclosure showed a fault light. Trying to access the web interface said "Initialization failed". I rebooted the system, and it came up normal and ran fine. That is until today. Same exact thing happened. I have made no hardware or software changes in the last few weeks.
Below are my error logs from Enclosure A. I can't really decipher what is going on. It appears one of my servers is constantly trying to log into the SAN and failing. I can't think of why it would be, and not sure if it is related. I am only attaching up to the end of day on Oct 3, because today's logs are basically EXACTLY the same.
FIRMWARE:
Bundle Version: GL105P001-03
Storage Controller: GLS105R04-01
Storage Controller Loader: 27.015
Management Controller: GLM105R009-01
Management Controller Loader: 6.04.16617
Expander: 3117
CPLD: 42
Sep 28 11:22:36 (none) auth.info sshd[10081]: Failed password for root from 192.168.30.91 port 12452 ssh2
Sep 28 11:22:39 (none) auth.err sshd[10081]: error: PAM: Permission denied for root from 192.168.30.91
Sep 28 11:22:39 (none) auth.info sshd[10081]: Connection closed by 192.168.30.91 [preauth]
Sep 28 15:16:49 (none) auth.info sshd[14885]: Did not receive identification string from 192.168.30.91
Sep 28 15:27:50 (none) auth.info sshd[15114]: Did not receive identification string from 192.168.30.91
Sep 28 15:27:53 (none) auth.info sshd[15117]: WARNING: /home/pablo/sandbox/heartbleed/GL105/platform/build.Gallium.GPM105P01-01.output/opt/etc/moduli does not exist, using fixed modulus
Sep 28 15:27:56 (none) auth.info sshd[15117]: Failed password for root from 192.168.30.91 port 24622 ssh2
Sep 28 15:27:58 (none) auth.err sshd[15117]: error: PAM: Permission denied for root from 192.168.30.91
Sep 28 15:27:58 (none) auth.info sshd[15117]: Connection closed by 192.168.30.91 [preauth]
Sep 28 19:16:09 (none) auth.info sshd[19801]: Did not receive identification string from 192.168.30.91
Sep 28 23:16:17 (none) auth.info sshd[24725]: Did not receive identification string from 192.168.30.91
Sep 29 03:16:49 (none) auth.info sshd[29637]: Did not receive identification string from 192.168.30.91
Sep 29 07:16:54 (none) auth.info sshd[2104]: Did not receive identification string from 192.168.30.91
Sep 29 07:22:22 (none) auth.info sshd[2260]: Did not receive identification string from 192.168.30.91
Sep 29 07:22:26 (none) auth.info sshd[2263]: WARNING: /home/pablo/sandbox/heartbleed/GL105/platform/build.Gallium.GPM105P01-01.output/opt/etc/moduli does not exist, using fixed modulus
Sep 29 07:22:30 (none) auth.info sshd[2263]: Failed password for root from 192.168.30.91 port 14036 ssh2
Sep 29 07:22:34 (none) auth.err sshd[2263]: error: PAM: Permission denied for root from 192.168.30.91
Sep 29 07:22:34 (none) auth.info sshd[2263]: Connection closed by 192.168.30.91 [preauth]
Sep 29 11:16:50 (none) auth.info sshd[7142]: Did not receive identification string from 192.168.30.91
Sep 29 15:16:54 (none) auth.info sshd[12035]: Did not receive identification string from 192.168.30.91
Sep 29 19:16:07 (none) auth.info sshd[16928]: Did not receive identification string from 192.168.30.91
Sep 29 23:16:23 (none) auth.info sshd[21845]: Did not receive identification string from 192.168.30.91
Sep 30 03:16:49 (none) auth.info sshd[26769]: Did not receive identification string from 192.168.30.91
Sep 30 03:22:06 (none) auth.info sshd[26883]: Did not receive identification string from 192.168.30.91
Sep 30 03:22:10 (none) auth.info sshd[26886]: WARNING: /home/pablo/sandbox/heartbleed/GL105/platform/build.Gallium.GPM105P01-01.output/opt/etc/moduli does not exist, using fixed modulus
Sep 30 03:22:13 (none) auth.info sshd[26886]: Failed password for root from 192.168.30.91 port 15958 ssh2
Sep 30 03:22:17 (none) auth.err sshd[26886]: error: PAM: Permission denied for root from 192.168.30.91
Sep 30 03:22:17 (none) auth.info sshd[26886]: Connection closed by 192.168.30.91 [preauth]
Sep 30 07:15:32 (none) auth.info sshd[31672]: Did not receive identification string from 192.168.30.91
Sep 30 11:16:47 (none) auth.info sshd[4305]: Did not receive identification string from 192.168.30.91
Sep 30 11:22:00 (none) auth.info sshd[4410]: Did not receive identification string from 192.168.30.91
Sep 30 11:22:04 (none) auth.info sshd[4418]: WARNING: /home/pablo/sandbox/heartbleed/GL105/platform/build.Gallium.GPM105P01-01.output/opt/etc/moduli does not exist, using fixed modulus
Sep 30 11:22:07 (none) auth.info sshd[4418]: Failed password for root from 192.168.30.91 port 30993 ssh2
Sep 30 11:22:10 (none) auth.err sshd[4418]: error: PAM: Permission denied for root from 192.168.30.91
Sep 30 11:22:10 (none) auth.info sshd[4418]: Connection closed by 192.168.30.91 [preauth]
Sep 30 15:16:36 (none) auth.info sshd[9230]: Did not receive identification string from 192.168.30.91
Sep 30 19:16:10 (none) auth.info sshd[14139]: Did not receive identification string from 192.168.30.91
Sep 30 23:16:27 (none) auth.info sshd[19071]: Did not receive identification string from 192.168.30.91
Oct 1 03:16:44 (none) auth.info sshd[23985]: Did not receive identification string from 192.168.30.91
Oct 1 07:15:43 (none) auth.info sshd[28878]: Did not receive identification string from 192.168.30.91
Oct 1 11:16:56 (none) auth.info sshd[1336]: Did not receive identification string from 192.168.30.91
Oct 1 15:16:45 (none) auth.info sshd[6384]: Did not receive identification string from 192.168.30.91
Oct 1 19:16:07 (none) auth.info sshd[11289]: Did not receive identification string from 192.168.30.91
Oct 1 23:15:18 (none) auth.info sshd[16192]: Did not receive identification string from 192.168.30.91
Oct 1 23:19:21 (none) auth.info sshd[16279]: Did not receive identification string from 192.168.30.91
Oct 1 23:19:24 (none) auth.info sshd[16282]: WARNING: /home/pablo/sandbox/heartbleed/GL105/platform/build.Gallium.GPM105P01-01.output/opt/etc/moduli does not exist, using fixed modulus
Oct 1 23:19:28 (none) auth.info sshd[16282]: Failed password for root from 192.168.30.91 port 60828 ssh2
Oct 1 23:19:31 (none) auth.err sshd[16282]: error: PAM: Permission denied for root from 192.168.30.91
Oct 1 23:19:31 (none) auth.info sshd[16282]: Connection closed by 192.168.30.91 [preauth]
Oct 2 03:16:47 (none) auth.info sshd[21152]: Did not receive identification string from 192.168.30.91
Oct 2 07:15:43 (none) auth.info sshd[26046]: Did not receive identification string from 192.168.30.91
Oct 2 11:16:52 (none) auth.info sshd[30993]: Did not receive identification string from 192.168.30.91
Oct 2 15:16:14 (none) auth.info sshd[3574]: Did not receive identification string from 192.168.30.91
Oct 2 15:20:26 (none) auth.info sshd[3660]: Did not receive identification string from 192.168.30.91
Oct 2 15:20:30 (none) auth.info sshd[3663]: WARNING: /home/pablo/sandbox/heartbleed/GL105/platform/build.Gallium.GPM105P01-01.output/opt/etc/moduli does not exist, using fixed modulus
Oct 2 15:20:34 (none) auth.info sshd[3663]: Failed password for root from 192.168.30.91 port 15754 ssh2
Oct 2 15:20:37 (none) auth.err sshd[3663]: error: PAM: Permission denied for root from 192.168.30.91
Oct 2 15:20:37 (none) auth.info sshd[3663]: Connection closed by 192.168.30.91 [preauth]
Oct 2 19:16:09 (none) auth.info sshd[8502]: Did not receive identification string from 192.168.30.91
Oct 2 23:16:11 (none) auth.info sshd[13431]: Did not receive identification string from 192.168.30.91
Oct 3 03:16:20 (none) auth.info sshd[18345]: Did not receive identification string from 192.168.30.91
Oct 3 07:15:14 (none) auth.info sshd[23234]: Did not receive identification string from 192.168.30.91
Oct 3 11:16:21 (none) auth.info sshd[28174]: Did not receive identification string from 192.168.30.91
Oct 3 11:20:37 (none) auth.info sshd[28266]: Did not receive identification string from 192.168.30.91
Oct 3 11:20:41 (none) auth.info sshd[28269]: WARNING: /home/pablo/sandbox/heartbleed/GL105/platform/build.Gallium.GPM105P01-01.output/opt/etc/moduli does not exist, using fixed modulus
Oct 3 11:20:44 (none) auth.info sshd[28269]: Failed password for root from 192.168.30.91 port 46661 ssh2
Oct 3 11:20:48 (none) auth.err sshd[28269]: error: PAM: Permission denied for root from 192.168.30.91
Oct 3 11:20:48 (none) auth.info sshd[28269]: Connection closed by 192.168.30.91 [preauth]
Oct 3 12:05:00 (none) user.warn kernel: sched: RT throttling activated
Oct 3 12:05:00 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:05:21 (none) user.warn kernel: ------------[ cut here ]------------
Oct 3 12:05:21 (none) user.warn kernel: WARNING: at net/sched/sch_generic.c:256 dev_watchdog+0x15c/0x24c()
Oct 3 12:05:21 (none) user.info kernel: NETDEV WATCHDOG: mcmc (): transmit queue 0 timed out
Oct 3 12:05:21 (none) user.warn kernel: Modules linked in: mcmclink g_serial ocores_udc mcfulink mooseproc mcscbridge msgdrv
Oct 3 12:05:21 (none) user.warn kernel: [<c0014428>] (unwind_backtrace+0x0/0xec) from [<c02a6c50>] (dump_stack+0x20/0x24)
Oct 3 12:05:21 (none) user.warn kernel: [<c02a6c50>] (dump_stack+0x20/0x24) from [<c001bf60>] (warn_slowpath_common+0x5c/0x74)
Oct 3 12:05:21 (none) user.warn kernel: [<c001bf60>] (warn_slowpath_common+0x5c/0x74) from [<c001c034>] (warn_slowpath_fmt+0x40/0x48)
Oct 3 12:05:21 (none) user.warn kernel: [<c001c034>] (warn_slowpath_fmt+0x40/0x48) from [<c022a1e4>] (dev_watchdog+0x15c/0x24c)
Oct 3 12:05:21 (none) user.warn kernel: [<c022a1e4>] (dev_watchdog+0x15c/0x24c) from [<c00281b8>] (run_timer_softirq+0x1d0/0x2dc)
Oct 3 12:05:21 (none) user.warn kernel: [<c00281b8>] (run_timer_softirq+0x1d0/0x2dc) from [<c0021cfc>] (__do_softirq+0xd8/0x1c0)
Oct 3 12:05:21 (none) user.warn kernel: [<c0021cfc>] (__do_softirq+0xd8/0x1c0) from [<c002219c>] (irq_exit+0x50/0x5c)
Oct 3 12:05:21 (none) user.warn kernel: [<c002219c>] (irq_exit+0x50/0x5c) from [<c000f750>] (handle_IRQ+0x84/0xa4)
Oct 3 12:05:21 (none) user.warn kernel: [<c000f750>] (handle_IRQ+0x84/0xa4) from [<c00086b8>] (asm_do_IRQ+0x18/0x1c)
Oct 3 12:05:21 (none) user.warn kernel: [<c00086b8>] (asm_do_IRQ+0x18/0x1c) from [<c000e55c>] (__irq_usr+0x3c/0x80)
Oct 3 12:05:21 (none) user.warn kernel: Exception stack(0xd5e5bfb0 to 0xd5e5bff8)
Oct 3 12:05:21 (none) user.warn kernel: bfa0: 001cc5f8 80000163 00000008 4390de30
Oct 3 12:05:21 (none) user.warn kernel: bfc0: 43aa46c0 00271970 41a2e8b0 b60fe65c 3bf37a15 43a7d83c 00000000 b60fe09c
Oct 3 12:05:21 (none) user.warn kernel: bfe0: 43946d98 b60fe088 41e66154 4390de30 20000010 ffffffff
Oct 3 12:05:21 (none) user.warn kernel: ---[ end trace 3867a107f846c95a ]---
Oct 3 12:05:21 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:05:21 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:05:21 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:05:30 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:05:31 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:05:33 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:05:39 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:05:40 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:05:53 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:05:55 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:05:59 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:06:00 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:06:04 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:06:05 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:06:06 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:06:11 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:06:13 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:06:17 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:06:19 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:06:23 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:06:24 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:06:29 (none) user.warn kernel: MCMC: error status (0xda) - Unexpected failure. (Inter MC link message(0x17))
Oct 3 12:06:42 (none) user.warn kernel: MCMC: error status (0x5f) - Message timed out. (Inter MC link message(0x17))
Oct 3 12:06:53 (none) user.warn kernel: MCMC: error status (0x5f) - Message timed out. (Inter MC link message(0x17))
Oct 3 12:06:53 (none) user.warn kernel: MCMC: error status (0xdc) - Memory write failed. (Inter MC link message(0x17))
Oct 3 12:06:53 (none) user.warn kernel: MCMC: error status (0xdc) - Memory write failed. (Inter MC link message(0x17))
Oct 3 12:06:53 (none) user.warn kernel: MCMC: error status (0xdc) - Memory write failed. (Inter MC link message(0x17))
Oct 3 12:06:53 (none) user.warn kernel: MCMC: error status (0xdc) - Memory write failed. (Inter MC link message(0x17))
Oct 3 12:06:53 (none) user.warn kernel: MCMC: error status (0xdc) - Memory write failed. (Inter MC link message(0x17))
Oct 3 12:06:53 (none) user.warn kernel: MCMC: error status (0xdc) - Memory write failed. (Inter MC link message(0x17))
Oct 3 12:06:54 (none) user.warn kernel: MCMC: error status (0xdc) - Memory write failed. (Inter MC link message(0x17))
Oct 3 12:06:59 (none) user.warn kernel: MCMC: error status (0xdc) - Memory write failed. (Inter MC link message(0x17))
Oct 3 12:07:00 (none) user.warn kernel: MCMC: error status (0xdc) - Memory write failed. (Inter MC link message(0x17))