UMN CDMS Wiki archive
start › cdms › k100 › run_diary › 20171006

2017/10/06 (Friday)

Previous Day

Status: warm. All mixture in dump. LN2 traps warm and pumped out. Contaminants from fridge were pumped out at 220 K. Cryostat vacuum jacket pumped out. Tower removed.
0840 DCRC1 is powered and connected to the mini-BOB and a MIDAS run is in progress. But no triggering is set. rd 0 and rd 14 are both updating, so it has run all night without getting stuck. rd 2 and rd 3 are also updating, so I guess MIDAS continues to send queries of some sort even though it isn't triggering (?) (see below: test-signal triggering was on)
  1. DCRC1 connected with MIDAS run with no triggers, doesn't get stuck. rd 0 and rd 14 updating, rd 2 and rd 3 updating.
  2. Stop MIDAS run. Now rd 2 and rd 3 do not update.
  3. On ODB set random trigger rate to 5 Hz. ChargeWaveFormLength 151384 PhononWaveFormLength 512 as yesterday
  4. Start MIDAS run. Gets stuck after 30-45 seconds.
  5. Stop MIDAS run and turn off MIDAS programs. rd 2 = 00DA, rd 3 = 52B8
  6. Issue RESET command via telnet. Disconnects the telnet session.
    from HELP: “RESET: Hardware Reset of Ethernet, FPGA, then soft restart”
  7. Reconnect telnet. rd 0 is updating, DCRC is unstuck.
  8. Start MIDAS run, 5 Hz triggering. Gets stuck after 15-30 seconds.
  9. stop MIDAS run. rd 2 = 00DF, rd 3 = CCD8
  10. RESET again, then try another MIDAS run with 5 Hz triggering. Gets stuck after 18 seconds.
  11. RESET again, then try a MIDAS run with 0.5 Hz triggering. Gets stuck between 4.5-5.0 minutes
  12. rd 2 = 00D3, rd 3 = 25F8
  13. RESET again, then try another MIDAS run with 0.5 Hz triggering. Gets stuck between 2.5-3.0 minutes
  14. RESET again, then try a MIDAS run with 20 Hz triggering. Gets stuck after about 10 seconds
  15. RESET again, then try a MIDAS run with 0.5 Hz triggering. Gets stuck after about 1.5 minutes
  16. RESET again, then try a MIDAS run with 0.5 Hz triggering. Gets stuck after about 1.0 minutes
  17. issue MO 1 command, temperature is 36.0 C
  18. unplug DCRC1 to let it cool down
0935 Taped lab thermometer probe to big copper-colored box on DCRC1. Temperature reads 31 C (lab is 22.5 C). Cooling with tau ~ 12.5 minutes.
1000 DCRC1 = 23.7 C according to lab thermometer
  1. Power on DCRC1. MO 1 reads 21.0 C. Couple minutes later it's up to 23.0 C
  2. Start MIDAS run with 0.5 Hz triggering. Gets stuck at 0.8 minutes.
  3. DCRC1 reads 28.0 C, temperature probe reads 24.8 C
  4. RESET again, then try MIDAS run with 0.1 Hz triggering. Gets stuck at 6.2 minutes.
  5. DCRC1 reads 34.0 C, temperature probe reads 29.0 C
  6. RESET
1050 DCRC1 = 31.0 C according to lab thermometer. Via MO 1 telnet command, DCRC = 34.0 C
  1. Start MIDAS run with 0.1 Hz triggering. Gets stuck at 0.4 minutes.
    Note: this was after a test of Anthony's code that repeatedly reads the trigger. Did this affect the hold time?
  2. RESET again, then try MIDAS run with 0.1 Hz triggering. Gets stuck at 2.0 minutes.
  3. RESET again, then try MIDAS run with 0.1 Hz triggering. Gets stuck at 0.0 minutes (immediately, less than 5 seconds)
  4. RESET again, then try MIDAS run with 0.1 Hz triggering. Gets stuck at 0.8 minutes.
  5. RESET
  6. On ODB set ChargeWaveFormLength 128 PhononWaveFormLength 64, much shorter
  7. Start MIDAS run with 0.1 Hz triggering and short traces as shown above. Lasts at least 30 minutes without getting stuck.
1135 Note: DCRC1 has been set to trigger on test signal since last night. Test signal = 10 Hz. That means that during the overnight run, which never got stuck, MIDAS was getting events continuously! And during today's tests which show that the hold time depends on the randoms rate, the fact that there was also test-signal triggering was not relevant!
  1. Stop MIDAS run. Then set randoms rate to 0
  2. rt is updating, meaning that the DCRC is issuing triggers
  3. on ODB turn off test signal triggering (in the General window of DCRC1). Now rt does not update.
  4. On ODB set ChargeWaveFormLength 16384 PhononWaveFormLength 8192. These are the standard settings for 3.2768 ms traces.
  5. Start MIDAS run with 1 Hz triggering and above trace lengths.
  6. still ok after 6 minutes - increase randoms rate to 10 Hz… wait another 4.75 minutes, still not stuck - that's 100,000,000 samples, longer than any previous test today
  7. at 12 minutes turn test signal triggering back on… still not getting stuck… lasts at least 5.5 minutes more…
  8. at 18.5 minutes set chargewaveformlength back up to 151384.
  9. Screwed up the tests above - I was doing rd 2 instead of rd 1 or rd 0. Need rd 1 or rd 0 or rd 14 to see if it's stuck. Currently it is stuck.
  1. RESET DCRC1
  2. On ODB: Test signal triggering off, ChargeWaveFormLength 16384 PhononWaveFormLength 8192 (standard values)
  3. Start MIDAS run with 5 Hz triggering and above trace lengths. At 10 minutes it's still not stuck, 100,000,000 samples total…
  4. at 10 minutes turn on test signal triggering on DCRC1. Wait another 5 minutes, still not stuck, nearly 150,000,000 samples
  5. Stop MIDAS run
  6. Set chargewaveformlength back up to 151384. That's ~ 30 ms - this is where we had left it at the end of the run.
  7. Start MIDAS run with 5 Hz triggering and above trace lengths. Gets stuck almost immediately, < 5 seconds.
  8. RESET DCRC1
1230 These recent tests indicate that the DCRC does not get stuck with standard or short charge trace length, only with the long charge trace length. Also the stuckness with the long charge trace length seems to occur with random triggerings but test-signal triggering is not relevant.
  1. On ODB: Test signal triggering off, ChargeWaveFormLength 16384 PhononWaveFormLength 8192 (standard values)
  2. Start MIDAS run with 20 Hz triggering and above trace lengths. At 30 minutes it's still not stuck. Stop run.
  3. On ODB: change ChargeWaveFormLength to 16386 (just 2 longer than before)
  4. Start MIDAS run with 20 Hz triggering and above trace lengths. At 5 minutes it's still not stuck. Stop run.
  5. On ODB: change ChargeWaveFormLength to 32768 (2x standard)
  6. Start MIDAS run with 20 Hz triggering and above trace lengths. At 6 minutes it's still not stuck. Stop run.
  7. On ODB: change ChargeWaveFormLength to 65536 (4x standard, about 13 ms)
  8. Start MIDAS run with 20 Hz triggering and above trace lengths. At 5 minutes it's still not stuck. Stop run.
  9. On ODB: change ChargeWaveFormLength to 131072 (8x standard, about 26 ms)
  10. Start MIDAS run with 20 Hz triggering and above trace lengths. At 6 minutes it's still not stuck. Stop run.
  11. On ODB: change ChargeWaveFormLength to 151384 (the setting from earlier today, around 30 ms)
  12. Start MIDAS run with 20 Hz triggering and above trace lengths. At about 8 seconds it gets stuck.
  13. RESET DCRC1
  14. On ODB: change ChargeWaveFormLength to 262144 (16x standard, about 52 ms)
  15. Start MIDAS run with 20 Hz triggering and above trace lengths. At 5 minutes it's still not stuck. Stop run.
  16. On ODB: change ChargeWaveFormLength to 151384 (the setting from earlier today, around 30 ms)
  17. Start MIDAS run with 20 Hz triggering and above trace lengths. At about 35 seconds it gets stuck.
1430 OK, DCRC1 does not seem to get stuck at any trace lengths attempted except charge waveform length = 151384 (30.2768 ms)
1430 Start experimenting with DCRC 3 Note: later realized that DCRC3 had QO threshold trigger enabled during these tests.
  1. Power on DCRC3. Check on telnet, behaving normally, rd 0 is updating
  2. On ODB change the DCRC1 IP address to that for DCRC3, so we know that we are not changing any other settings
  3. Set ODB to standard lengths, ChargeWaveFormLength 16384 PhononWaveFormLength 8192
  4. test signal triggering is off. DCRC is set as master.
  5. Start MIDAS run with 20 Hz triggering and standard trace lengths. At 10 minutes it's still not stuck. Stop run.
    Note that on Sep 25 when we stopped trying to use this DCRC, it lasted longer than this, so it's not a definitive test.
    Note: looking at ODB records, it looks like DCRC3's charge length may have been set to the bad value before it stopped working…
  6. Stop MIDAS programs then reset ODB so that IP addresses are normal. Enable DCRC3, disable others. Set DCRC3 settings as above (master, standard trace lengths, no test signal trigger)
  7. Start MIDAS run with 20 Hz triggering and standard trace lengths. At 10 minutes it's still not stuck. Stop run.
  8. On ODB: change ChargeWaveFormLength to 151384 (the “bad” setting)
  9. Start MIDAS run with 20 Hz triggering and above trace lengths. At about 5 seconds it gets stuck.
  10. RESET DCRC3
  11. On ODB: change ChargeWaveFormLength to 16384 (standard setting)
  12. Start MIDAS run with 20 Hz triggering and standard trace lengths (start time 15:15). At 47 minutes it's still not stuck.
1530 Well this is all very strange, but encouraging. As far as can be seen so far, DCRC1 and DCRC3 get stuck only if the charge length is set to a “bad” value. This is only checking the case of random triggering. It seems to be independent of test-signal triggering. We haven't checked threshold triggering. But what is clear is that with random triggering, we can reliably make them get stuck with a “bad” trace length and reliably keep them unstuck with a “good” trace length.
1545 DCRC52, RevC.1, arrives from Berkeley. Nick sets IP address, designating this as DCRC6. See DCRC procedures page.
1600 A few more DCRC3 tests:
  1. Stop MIDAS run
  2. rdb 14 16384. This is a command to read out samples, 16384 bytes worth (the standard charge trace length)
  3. rdb 14 151384. Same with the “bad” charge trace length. Repeat several times.
  4. wr C 0000, wr D 0000, rdb 14 151384. The first two commands should shift to the beginning of the qInner portion of the buffer.
  5. wr E 0000, wr F 0000, rdb 14 151384. The first two commands should shift to the beginning of the qOuter portion of the buffer.
  6. throughout these tests see if the DCRC is stuck. Doesn't get stuck. But maybe you need to do hundreds of these.
1635 We have shown that powers-of-two charge lengths are OK, and in one case a non-power-of-two was OK. The bad value is not a power of two. Breaking it into binary, it has 9 “1”s. Compare with powers-of-two, only 1 “1”, and the one good non-power-of-two, which only has 2 “1”s. Maybe certain bits in the trace length value are “bad” on the DCRC. The bits that are on in the “bad” length that aren't present in any of the “good” lengths are 3, 4, 6, 8, 9, 10, 11. If we construct a trace length from just these bits we get 3928. Maybe try that…
  1. Power on DCRC3. Check on telnet, behaving normally, rd 0 is updating
  2. Start MIDAS run at 20 Hz with standard trace lengths. After a couple of minutes it is still not stuck.
  3. Stop run, then set ChargeWaveFormLength = 3928
  4. Start MIDAS run at 20 Hz with above trace lengths. At less than 10 seconds, DCRC gets stuck
  5. RESET DCRC3
  6. Start MIDAS run at 20 Hz with standard trace lengths. After 2.5 of minutes it is still not stuck. Stop run.
  7. set ChargeWaveFormLength = 2048 = 2^11
  8. Start MIDAS run at 20 Hz with above trace lengths. After 1 minute still not stuck. Stop run.
  9. set ChargeWaveFormLength = 3072 = 2^11+2^10
  10. Start MIDAS run at 20 Hz with above trace lengths. After 1 minute still not stuck. Stop run.
  11. set ChargeWaveFormLength = 3584 = 2^11+2^10+2^9
  12. Start MIDAS run at 20 Hz with above trace lengths. After 1 minute still not stuck. Stop run.
  13. set ChargeWaveFormLength = 3840 = 2^11+2^10+2^9+2^8
  14. Start MIDAS run at 20 Hz with above trace lengths. After 1 minute still not stuck. Stop run.
  15. set ChargeWaveFormLength = 3904 = 2^11+2^10+2^9+2^8+2^6
  16. Start MIDAS run at 20 Hz with above trace lengths. After 1 minute still not stuck. Stop run.
  17. set ChargeWaveFormLength = 3920 = 2^11+2^10+2^9+2^8+2^6+2^4
  18. Start MIDAS run at 20 Hz with above trace lengths. After 1 minute still not stuck. Stop run.
  19. set ChargeWaveFormLength = 3928 = 2^11+2^10+2^9+2^8+2^6+2^4+2^3
  20. Start MIDAS run at 20 Hz with above trace lengths. At 25 seconds, DCRC gets stuck

so it seems that it could be that setting the 3rd bit = 1 is what causes the freeze-up. Or it could be that setting more than 6 bits to 1 causes the freeze-up.

  1. RESET DCRC3
  2. set ChargeWaveFormLength = 16384 = 2^14. This is back to the standard setting.
  3. Start MIDAS run at 20 Hz with above trace lengths. After 1 minute still not stuck. Stop run.
  4. set ChargeWaveFormLength = 16392 = 2^14+2^3
  5. Start MIDAS run at 20 Hz with above trace lengths. After 1 minute still not stuck. Stop run.
  6. set ChargeWaveFormLength = 16632 = 2^14+2^7+2^6+2^5+2^4+2^3. That's 6 “1”s
  7. Start MIDAS run at 20 Hz with above trace lengths. After 1 minute still not stuck. Stop run.
  8. set ChargeWaveFormLength = 16888 = 2^14+2^8+2^7+2^6+2^5+2^4+2^3. That's 7 “1”s
  9. Start MIDAS run at 20 Hz with above trace lengths. After 1 minute still not stuck. Stop run.
  10. set ChargeWaveFormLength = 3928 = 2^11+2^10+2^9+2^8+2^6+2^4+2^3
  11. Start MIDAS run at 20 Hz with above trace lengths. At 15 seconds, DCRC gets stuck

OK so we haven't quite found the pattern. 7 “1”s isn't always enough to make the trace length bad. But the two “bad” values we know about have a lot of binary digits in common: 

 151384 = 100100111101011000  bad
   3928 =       111101011000  bad
  16888 =    100000111111000  good (at least to >2x as long as bad ones)
1800 Stop MIDAS run, MIDAS programs. RESET DCRC3. On ODB set standard lengths for 3.2768 ms traces, ChargeWaveFormLength 16384 PhononWaveFormLength 8192.
1805 Start MIDAS run. We'll let it go a while, see if it ever gets stuck.
1820 DCRC3 still not stuck.

Summary of observations about DCRCs getting stuck readouts

  • DCRC1 started getting a stuck readout shortly after starting a MIDAS run
  • requires power cycle or just a “RESET” to get it unstuck
  • Setting only randoms can make it stuck. (but it seems that setting it to trigger on the test signal with no randoms does not cause it to get stuck, this is still confusing)
  • Although the hold time before getting stuck varies, there's a clear correlation with randoms rate: high rate → short hold time
  • When we experimented with different trace lengths, it turned out not to get stuck at most trace length settings
  • We found two “bad” trace lengths for the charge channels, which have similar patterns when written in binary
  • DCRC3 behaves similarly: it gets stuck on “bad” trace lengths and is ok on “good” ones
  • (not all of this has been tested exhaustively but the patterns seem pretty clear)
1830 DCRC3 still not stuck.
1850 DCRC3 still not stuck. Quit the telnet session on jiajin, this may be necessary to check it remotely (?)
2010 DCRC3 still not stuck. Checked remotely.
2230 DCRC3 still not stuck. Checked remotely.

Next Day