2017/02/16 (Thursday)
| Setup: DIB1/3/4 → 3.2 MHz filters 3/1/2 → cable 3/1/2 →gender changer 3/1/2 → DCRC55(1)/33(2)/50(3) (on/on/off) | |
| 0900 | Tower = 34.5 mK, Main bath = 68.9% |
| 0905 | Fill LN2 cold trap bath |
| 1135 | Updated ODB for DCRC3 (see yesterday) so that it no longer is connecting LED voltage to phonon output, and so that test signal is no longer connected to pA and pB. Then turn off DAQ programs and set up to include DCRC3. Power up all three DCRCs and start programs - all three now working. But, after a few minutes, DCRC3 hangs/flatlines. |
| 1150 | Power cycle DCRC50. Then change ODB identities of “DCRC1” and “DCRC3” by swapping IP addresses. Also change the master/slave relationship in ODB to reflect the cabling, which still has DCRC55 as master. Turn DAQ programs back on. Initially all three DCRCs working. But after a few minutes DCRC50 hangs again. |
| 1200 | Repower DCRC50(3) but leave DAQ programs off. Check whether the DCRCs are working or hanging by issuing rd 0 commands through telnet. All three are working (rd 0 returns different values indicating that the readout is advancing). Try issuing a few commands to DCRC50 through telnet. After about 10 minutes it continues to function OK. |
| 1210 | Set ODB to enable only DCRC50 and using the ODB DIB1 settings. Then start DAQ programs. Telnet session shows DCRC50 still working. Start pulse viewer and look at channels, change some settings… DCRC50 still works for about 11 minutes but then hangs. |
| 1225 | Power cycle DCRC50 but leave DAQ programs off. Again telnet says that DCRC50 is working. Will leave it for a while to see whether it ever hangs. |
| 1230 | Start DAQ programs with only DCRC55(1) and DCRC33(2) enabled. DCRC50 continues to function according to telnet. Occasionally changing settings on DCRC50 through telnet. Still functioning. 40 minutes later DCRC50 continues to function with the MIDAS DAQ not communicating with it. Occasional telnet commands to it don't seem to be disturbing it. |
| 1315 | Stop DAQ programs, configure ODB to communicate with all three DCRCs, then start programs again. But don't do anything else through MIDAS - don't change any settings or view any traces. (A MIDAS run is going however.) Will continue to monitor DCRC50 via telnet. After 20 minutes it is still working. |
| 1335 | Start pulse viewer to view traces on DCRC50(3). Looking at one channel, not adjusting anything. At less than 19 minutes (I had stopped watching) it goes flatline. |
| 1406 | After powercycling DCRC50 start programs again with all three DCRCs enabled and a run going. Don't issue any commands or look at pulse viewer. Monitor DCRC50 status via telnet. (This is a repeat of the 13:15 test.) After 37 minutes DCRC50 is still working. |
| 1444 | Start pulse viewer to view one channel on DCRC50(3), a repeat of the 13:35 test. Within 4 minutes it goes flatline. |
Here's an email I wrote to Anthony and Vuk which sums up some of today's observations:
Hi,
we've been testing the DCRC hanging problem today. There's more info since what I reported at the meeting, and Anthony your input on the symptoms would be appreciated.
First, the problem that we'd mentioned that had to do with ODB settings seems to have been a separate issue that is apparently fixed now. That problem was different in that 1) it affected both the “good” board (DCRC55) and the “bad” board (DCRC50) and 2) the DCRC would immediately hang when the MIDAS dcrc_driver program was started. As I said we seem to have gotten rid of that problem by restoring the stray settings we found but I didn't do a deep investigation of this.
The main problem is still continuing, that DCRC50 is hanging after a few minutes of operation. But we've found some further details.
First, DCRC50 will hang even if it's using the exact settings that we use for DCRC55 and even if it's the only DCRC enabled by MIDAS. This rules out any interference with other DCRCs and any settings issue.
Second, DCRC50 does not seem to ever hang if MIDAS isn't communicating with it. We can tell through a telnet session whether it has hung, and it seems to continue without hanging for over an hour (as opposed to less than 5 minutes).
Third, DCRC50 also seems to continue to work after starting the MIDAS programs with DCRC50 enabled, IF we don't do anything else, such as viewing pulses with the pulse viewer. We start the programs with a run still going in MIDAS, but if we don't start the pulse viewer it continues to work without hanging for over 30 minutes at least.
Correction, see below. It turns out that we were starting the MIDAS programs but we did not have a run going. The run was therefore being started by the pulse viewer. It turns out that just starting a MIDAS run causes DCRC50 to hang after a few minutes.
Fourth, in contrast to the above, after it has been running a while with MIDAS on, if we then turn on the pulse viewer and look at a DCRC50 trace (without adjusting any other settings), it hangs after a few minutes.
So, the third and fourth tests could sorta make sense if you assume the DCRC can no longer handle the constant requests for data for very long. On the other hand, Nick's vague recollection was that MIDAS should actually be constantly requesting data even before we turn on the viewer - simply having a MIDAS run going should be just as much communication with the DCRC whether the viewer is being used or not.
Anthony, do you have a better sense of how much communication MIDAS is doing with the DCRC under different circumstances?
Matt
| 1515 | Tower/MC/Coldplate = 34.6/31.1/197 mK, Still = 1.3 K / 0.120 mbar, G1/G2 = 220/215 mbar, 1K Pot = 1.688 K / 28 mbar, Main bath = 59.4%. Boil off rate: close to slow mode. |
|---|---|
| 1520 | Turn off MIDAS programs and powercycle DCRC50. Start programs again with all three DCRCs enabled. |
| 1525 | Just noticed that I did not have a MIDAS run going when I started the programs in the tests earlier today. Starting the pulse viewer is what started the runs. So we haven't tested yet whether simply starting a MIDAS run causes DCRC50 to hang. Will try that now… at 15:26, press Start button on MIDAS front page. Monitor DCRC50 via telnet. It hangs after about 5 minutes. |
| 1717 | Fill LN2 cold trap |