The mpi API on beowulf at Hanford

Legend:
green ball Normal status or debugging message
yellow ball Notable condition which may be a non-fatal error
orange ball Error condition not fatal to job
red ball Error condition fatal to job
blue ball Notable condition which is not an error
purple ball Currently undefined
email Condition requires email notification of the responsible administrator of this API
telephone Condition requires phone notification of the responsible administrator of this API

Link: API Status Page for Hanford

01/31/07-17:34:53 PST 02/01/07-01:34:53 GMT 854328907 STARTUP archiveLog file "/ldas_outgoing/logs/LDASmpi.log.html" already closed. (archived as /ldas_outgoing/logs/archive/mpiAPI/LDASmpi.854328292)
01/31/07-17:34:53 PST 02/01/07-01:34:53 GMT 854328907 STARTUP closeListenSock no cid registered for service 'data'
01/31/07-17:34:53 PST 02/01/07-01:34:53 GMT 854328907 STARTUP mpi::init unused data port 10018 closed
01/31/07-17:34:53 PST 02/01/07-01:34:53 GMT 854328907 STARTUP mpi::init port 10018 (jobstate) opened on beowulf as sock7
01/31/07-17:34:53 PST 02/01/07-01:34:53 GMT 854328907 STARTUP bgLoop Looping process watchlogs started
01/31/07-17:34:53 PST 02/01/07-01:34:53 GMT 854328907 STARTUP openListenSock port 10016 (operator) opened on beowulf as sock8
01/31/07-17:34:53 PST 02/01/07-01:34:53 GMT 854328907 STARTUP openListenSock port 10017 (emergency) opened on beowulf as sock9
01/31/07-17:34:53 PST 02/01/07-01:34:53 GMT 854328907 STARTUP leakLogger inital size of mpi API: 21036 kB
01/31/07-17:34:53 PST 02/01/07-01:34:53 GMT 854328907 STARTUP bgLoop Looping process etchosts started
01/31/07-17:34:53 PST 02/01/07-01:34:53 GMT 854328907 IDLE bgLoop Looping process statpagefile started
01/31/07-17:34:53 PST 02/01/07-01:34:53 GMT 854328907 IDLE bgLoop Looping process killedjobreaper started
01/31/07-17:34:53 PST 02/01/07-01:34:53 GMT 854328907 IDLE bgLoop Looping process logrotate started
01/31/07-17:34:54 PST 02/01/07-01:34:54 GMT 854328908 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://198.129.208.245') (::FTPDIR '') (::HTTPURL 'http://198.129.208.245/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/export/grid/ldas') (::GRIDFTPDIR '/export/grid/ldas') (::LDAS_GATEWAY 'ldas 198.129.208.245') (::LDAS_SYSTEM 'ldas-wa') (::RUNCODE 'LDAS-WA')
01/31/07-17:34:58 PST 02/01/07-01:34:58 GMT 854328912 STARTUP mpi::killAllMpirun cleaning up for user ldas
01/31/07-17:34:59 PST 02/01/07-01:34:59 GMT 854328913 STARTUP mpi::killAllMpirun ran kill 10 times in 1.058 seconds
01/31/07-17:34:59 PST 02/01/07-01:34:59 GMT 854328913 STARTUP mpi::prestartLamds running lamboot for user search01
01/31/07-17:35:00 PST 02/01/07-01:35:00 GMT 854328914 STARTUP mpi::prestartLamds running lamboot for user search02
01/31/07-17:35:01 PST 02/01/07-01:35:01 GMT 854328915 STARTUP mpi::prestartLamds running lamboot for user search03
01/31/07-17:35:03 PST 02/01/07-01:35:03 GMT 854328917 STARTUP mpi::prestartLamds running lamboot for user search04
01/31/07-17:35:04 PST 02/01/07-01:35:04 GMT 854328918 STARTUP mpi::prestartLamds running lamboot for user search05
01/31/07-17:35:06 PST 02/01/07-01:35:06 GMT 854328920 STARTUP mpi::prestartLamds running lamboot for user search06
01/31/07-17:35:07 PST 02/01/07-01:35:07 GMT 854328921 STARTUP mpi::prestartLamds running lamboot for user search07
01/31/07-17:35:08 PST 02/01/07-01:35:08 GMT 854328922 STARTUP mpi::prestartLamds running lamboot for user search08
01/31/07-17:35:09 PST 02/01/07-01:35:09 GMT 854328923 STARTUP mpi::prestartLamds running lamboot for user search09
01/31/07-17:35:09 PST 02/01/07-01:35:09 GMT 854328923 STARTUP mpi::prestartLamds running lamboot for user search10
01/31/07-17:35:10 PST 02/01/07-01:35:10 GMT 854328924 STARTUP mpi::prestartLamds running lamboot for user search11
01/31/07-17:35:11 PST 02/01/07-01:35:11 GMT 854328925 STARTUP mpi::prestartLamds running lamboot for user search12
01/31/07-17:35:12 PST 02/01/07-01:35:12 GMT 854328926 STARTUP mpi::prestartLamds running lamboot for user search13
01/31/07-17:35:13 PST 02/01/07-01:35:13 GMT 854328927 STARTUP mpi::prestartLamds running lamboot for user search14
01/31/07-17:35:13 PST 02/01/07-01:35:13 GMT 854328927 STARTUP mpi::prestartLamds running lamboot for user search15
01/31/07-17:35:14 PST 02/01/07-01:35:14 GMT 854328928 STARTUP mpi::prestartLamds running lamboot for user search16
01/31/07-17:35:15 PST 02/01/07-01:35:15 GMT 854328929 IDLE mpi::updateCmonNodelist updated ::beowulfNodes in cntlmonAPI to 'beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf'
01/31/07-17:35:15 PST 02/01/07-01:35:15 GMT 854328929 STARTUP mpi::prestartLamds STARTUP search01 beowulf ok!
01/31/07-17:35:15 PST 02/01/07-01:35:15 GMT 854328929 STARTUP mpi::prestartLamds STARTUP search02 beowulf ok!
01/31/07-17:35:15 PST 02/01/07-01:35:15 GMT 854328929 STARTUP mpi::prestartLamds STARTUP search03 beowulf ok!
01/31/07-17:35:15 PST 02/01/07-01:35:15 GMT 854328929 STARTUP mpi::prestartLamds STARTUP search04 beowulf ok!
01/31/07-17:35:15 PST 02/01/07-01:35:15 GMT 854328929 STARTUP mpi::prestartLamds STARTUP search05 beowulf ok!
01/31/07-17:35:15 PST 02/01/07-01:35:15 GMT 854328929 STARTUP mpi::prestartLamds STARTUP search06 beowulf ok!
01/31/07-17:35:15 PST 02/01/07-01:35:15 GMT 854328929 STARTUP mpi::prestartLamds STARTUP search07 beowulf ok!
01/31/07-17:35:15 PST 02/01/07-01:35:15 GMT 854328929 STARTUP mpi::prestartLamds STARTUP search08 beowulf ok!
01/31/07-17:35:15 PST 02/01/07-01:35:15 GMT 854328929 STARTUP mpi::prestartLamds STARTUP search09 beowulf ok!
01/31/07-17:35:15 PST 02/01/07-01:35:15 GMT 854328929 STARTUP mpi::prestartLamds STARTUP search10 beowulf ok!
01/31/07-17:35:15 PST 02/01/07-01:35:15 GMT 854328929 STARTUP mpi::prestartLamds STARTUP search11 beowulf ok!
01/31/07-17:35:15 PST 02/01/07-01:35:15 GMT 854328929 STARTUP mpi::prestartLamds STARTUP search12 beowulf ok!
01/31/07-17:35:16 PST 02/01/07-01:35:16 GMT 854328930 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://198.129.208.245') (::FTPDIR '') (::HTTPURL 'http://198.129.208.245/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/export/grid/ldas') (::GRIDFTPDIR '/export/grid/ldas') (::LDAS_GATEWAY 'ldas 198.129.208.245') (::LDAS_SYSTEM 'ldas-wa') (::RUNCODE 'LDAS-WA')
01/31/07-17:35:16 PST 02/01/07-01:35:16 GMT 854328930 STARTUP mpi::prestartLamds STARTUP search13 beowulf ok!
01/31/07-17:35:16 PST 02/01/07-01:35:16 GMT 854328930 STARTUP mpi::prestartLamds STARTUP search14 beowulf ok!
01/31/07-17:35:17 PST 02/01/07-01:35:17 GMT 854328931 STARTUP mpi::prestartLamds STARTUP search15 beowulf ok!
01/31/07-17:35:18 PST 02/01/07-01:35:18 GMT 854328932 STARTUP mpi::prestartLamds STARTUP search16 beowulf ok!
01/31/07-17:35:18 PST 02/01/07-01:35:18 GMT 854328932 STARTUP mpi::killAllMpirun {ldas@beowulf:mpirun: child process exited abnormally} {ldas@beowulf:wrapperAPI: child process exited abnormally}
01/31/07-17:35:20 PST 02/01/07-01:35:20 GMT 854328934 IDLE mpi::updateCmonNodelist updated ::beowulfNodes in cntlmonAPI to 'beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf'
02/27/07-08:44:44 PST 02/27/07-16:44:44 GMT 856629898 SHUTDOWN closeListenSock port 10016 (sock8) (operator) closed on beowulf
pehrens@ligo.caltech.edu, gmendell@ligo-wa.caltech.edu, bjohnson@ligo-wa.caltech.edu 856629898 SHUTDOWN mpi::sHuTdOwN Subject: LDAS Hanford mpi shutdown at 856629898 ( 02/27/07 08:44:44 PST ); Body: mpi shutting down NOW
02/27/07-08:44:44 PST 02/27/07-16:44:44 GMT 856629898 search12 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10014 on datacon. {connection refused}
02/27/07-08:44:45 PST 02/27/07-16:44:45 GMT 856629899 search04 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10014 on datacon. {connection refused}
02/27/07-08:44:45 PST 02/27/07-16:44:45 GMT 856629899 search14 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10014 on datacon. {connection refused}
02/27/07-08:44:45 PST 02/27/07-16:44:45 GMT 856629899 search06 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10014 on datacon. {connection refused}
02/27/07-08:44:46 PST 02/27/07-16:44:46 GMT 856629900 search16 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10014 on datacon. {connection refused}
02/27/07-08:44:46 PST 02/27/07-16:44:46 GMT 856629900 search08 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10014 on datacon. {connection refused}
02/27/07-08:44:46 PST 02/27/07-16:44:46 GMT 856629900 search01 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10014 on datacon. {connection refused}
02/27/07-08:44:46 PST 02/27/07-16:44:46 GMT 856629900 search11 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10014 on datacon. {connection refused}
02/27/07-08:44:47 PST 02/27/07-16:44:47 GMT 856629901 search03 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10014 on datacon. {connection refused}
02/27/07-08:44:47 PST 02/27/07-16:44:47 GMT 856629901 search13 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10014 on datacon. {connection refused}
02/27/07-08:44:47 PST 02/27/07-16:44:47 GMT 856629901 search05 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10014 on datacon. {connection refused}
02/27/07-08:44:47 PST 02/27/07-16:44:47 GMT 856629901 search15 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10014 on datacon. {connection refused}
02/27/07-08:44:48 PST 02/27/07-16:44:48 GMT 856629902 search07 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10014 on datacon. {connection refused}
02/27/07-08:44:48 PST 02/27/07-16:44:48 GMT 856629902 search10 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10014 on datacon. {connection refused}
02/27/07-08:44:48 PST 02/27/07-16:44:48 GMT 856629902 search09 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10014 on datacon. {connection refused}
02/27/07-08:44:48 PST 02/27/07-16:44:48 GMT 856629902 search02 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10014 on datacon. {connection refused}
02/27/07-08:44:48 PST 02/27/07-16:44:48 GMT 856629902 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search01'
02/27/07-08:44:49 PST 02/27/07-16:44:49 GMT 856629903 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search02'
02/27/07-08:44:49 PST 02/27/07-16:44:49 GMT 856629903 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search03'
02/27/07-08:44:49 PST 02/27/07-16:44:49 GMT 856629903 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search04'
02/27/07-08:44:50 PST 02/27/07-16:44:50 GMT 856629904 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search05'
02/27/07-08:44:50 PST 02/27/07-16:44:50 GMT 856629904 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search06'
02/27/07-08:44:50 PST 02/27/07-16:44:50 GMT 856629904 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search07'
02/27/07-08:44:51 PST 02/27/07-16:44:51 GMT 856629905 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search08'
02/27/07-08:44:51 PST 02/27/07-16:44:51 GMT 856629905 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search09'
02/27/07-08:44:51 PST 02/27/07-16:44:51 GMT 856629905 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search10'
02/27/07-08:44:52 PST 02/27/07-16:44:52 GMT 856629906 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search11'
02/27/07-08:44:52 PST 02/27/07-16:44:52 GMT 856629906 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search12'
02/27/07-08:44:52 PST 02/27/07-16:44:52 GMT 856629906 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search13'
02/27/07-08:44:53 PST 02/27/07-16:44:53 GMT 856629907 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search14'
02/27/07-08:44:53 PST 02/27/07-16:44:53 GMT 856629907 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search15'
02/27/07-08:44:53 PST 02/27/07-16:44:53 GMT 856629907 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search16'
02/27/07-08:44:54 PST 02/27/07-16:44:54 GMT 856629908 SHUTDOWN closeListenSock port 10017 (sock9) (emergency) closed on beowulf
02/27/07-08:44:54 PST 02/27/07-16:44:54 GMT 856629908 SHUTDOWN closeListenSock no cid registered for service 'data'
02/27/07-08:44:54 PST 02/27/07-16:44:54 GMT 856629908 SHUTDOWN closeLog /ldas_outgoing/logs/LDASmpi.log.html (file5) closed