2013/03/27

Plugged DCRC33 & DCRC32, one-at-a-time, into the mini-BOB with copper shield on power supply side removed to allow for visual check of LEDs on circuit board. Simplified circuit to computer by connecting ethernet cable from PC to Phihong PoE supply. The DCRCs were then plugged into the Phihong PoE supply.

DCRC33 was tested first. The 15 V power supply LEDs turned on without issue. DCRC32 was then tested, and no matter how many times the power was cycled, the 15 V power supply LEDs did not turn on. Turning on the FET heater and checking it on the mini-BOB appeared to indicate that it had not switched on. Additionally, checking Qi & Qo biases to ground yielded potentials of around 0.7 V. Interestingly, the board would repeatedly appear to lose connection at seemingly random intervals, closing the PuTTY session.

DCRC33 was then retested, and the 15 V power supply LEDs turned on again without issue. However, the power supply cut out at some point during tests with the mini-BOB, and repeatedly cycling the power had no effect.

Another point of interest is that it seemed to take 20-40 seconds for each board to connect to the computer. It is uncertain if this is normal behaviour, as it is known that the boards do take some time to connect when initially plugged in. However, the time taken to establish a connection with the computer seemed a little longer than usual.

It is difficult to believe that both boards have developed the same fault at the same time. What is strange is that DCRC33 appears to work on occasion. It is possible that DCRC33's power supply requires less power than DCRC32's, and this could explain the difference. There are a number of possibilities:

1. Both DCRCs may have developed the same fault (at the same time). This is very unlikely, but not beyond the realms of possibility. One way to check this is to simply test DCRC31. It would be very hard to believe 3 boards had developed the same fault at the same time. Having said that, DCRC33 and DCRC32 have seen a lot of use and DCRC32 is known to have a problem with its connection to the computer. Perhaps their connections have just degraded futher and DCRC33 is now behaving like DCRC32 used to, ie. it's close to complete failure.
2. The Phihong PoE supply could be at fault. There is another one that can be used instead.
3. The ethernet cables could be dodgy. There are plenty of ethernet cables to test this hypothesis. However, the cable connected to the computer cannot be removed from the ethernet card, ie. it's stuck in its socket. Therefore, it does not appear to be possible to check this cable.
4. The ethernet card may be faulty. As mentioned above, unfortunately one cannot check the cable connected to the ethernet card as it is stuck in its socket. However, the card connected to the physics network could be unplugged temporarily and diverted to the DCRCs. It is important to bear in mind that if this solved the problem, it would not necessarily mean that the ethernet card was the source of the problem as the cable has also been changed.

DCRC31's 15 V power supply turns on without issue once command is sent to board via PuTTY session - nothing unusual about that as this board does not turn on the power supply automatically once plugged in. Cycled the power a couple of times on the board and turned 15 V power supply and FET heater on and off in different combinations in an attempt to test it. Only once did the power cut out when turning off the power supply but leaving the FET heater on, but this was not repeatable. This could have been due to a simple write error, so may not be anything to worry about.

DCRC31's 15 V power supply was left on for around 20 minutes at a time to see if connection to the board was lost. However, the connection appeared to be stable. At the end of this period, the 15 V power supply and FET heater were also turned on and off in different combinations in an attempt to test it, but it passed with flying colours.

It appears as though DCRC31 has no power issues, which lends further evidence to an issue with DCRC32 & DCRC33, no matter how unlikely that may seem. To check this hypothesis, DCRC32 & DCRC33 were plugged back into the mini-BOB and the same PoE supply.

Cycled the power a few times on DCRC32, but to no avail: 15 V power supply remained off and LEDs did not light up. Turning the FET heater and 15 V power supply on and off in different combinations yielded no dividend.

DCRC33's behaviour is stranger. Upon initial connection, the 15 V power supply LEDs turned on. However, checking Qi & Qo biases to ground yielded a potential difference of 13 V. Reading back registers 5A (Qi bias) & 59 (Qo bias) both with the DetectorControl vi and directly from the board via the PuTTY session returned 20A3 & 2093 respectively, which is consistent with calibrated 0 V. Turning on the FET heater, leaving the 15 V power supply on, resulted in the loss of power. No combination of turning on and off both the FET heater and 15 V power supply resulted in the restoration of power. One wonders what biases the board delivered when it was plugged into DIB 2 during Run 36, and whether this may offer an explanation for the lack of statistics on Side 1. It is difficult to ascertain whether this may have had an effect without a simulation of the field due to the compexity of the interleaved design.

The aforementioned effect is not witnessed again when cycling the power again on DCRC33. In fact, the 15 V power supply and the corresponding LEDs remain off even after attempting to turn it off and on manually via the PuTTY session. The FET heater does not appear to turn on. Another attempt at cycling the power yields the same result.

As a sanity check, testing DCRC31 again with the same PoE supply exhibits no evidence of any power problems. This is very strong evidence that the improbable appears to be true and both DCRC32 and DCRC33 have developed similar problems at the same time.

Switched over to second Phihong PoE supply and started with DCRC32. Tested DCRC32 the standard three times, cycling the power in between. As observed above, both the 15 V power supply and FET heater did not turn on, and refused to turn on no matter what instructions were sent to the board via the PuTTY session. The 15 V power supply LEDs did not turn on either.

DCRC33 continues to exhibit weird behaviour. Upon plugging it in, the 15 V power supply LEDs turn on. However, checking the FET heater with the breakout box shows that it is on when it should actually be off (register 30 reads 0800, which means that the 15 V power supply is on, but the FET heater is off; it should read 0900 if both were on). Additionally, even though Qi bias to ground appears to be consistent with 0 V (0.02 V to be precise, but there is an error associated with the calibration and this bias is perfectly reasonable), Qo bias to ground yields -0.26 V, which is clearly incorrect. Reading register 59 returns the expected value of 2098. Writing 3000 to register 59 yields 6.37 V between Qo bias and ground, which looks reasonable. Writing 2098 to register 59 does and measuring Qo bias to ground results in a potential difference consistent with 0 V.

Writing 0900 to register 30 to turn on both the FET heater and 15 V power supply seems to work, ie. no difference is found. However, writing 0800 to register 30 now turns off the FET heater, but leaves the 15 V power supply on as it should. When initially plugged in, the board does turn on the FET heater and 15 V power supply automatically, and this result seems to indicate that it does not automatically turn it off, even though reading register 30 indicates that it has.

The above result may disprove the theory that the field may not have been as expected during Run 36, as registers 5A & 59 (Qi bias & Qo bias respectively) are written to at the start of every run. However, the fact that the FET heater doesn't appear to turn off automatically may have given birth to a new hypothesis: the increased level noise witnessed on this board may be due to the FET heater being left on while data is acquired. Zapping SQUIDs may turn off the FET heater as the instruction shares the same register. However, during ionisation-only runs, register 30 is never touched after the board has been turned on.

Upon cycling the power on DCRC33, the 15 V power supply LEDs do not turn on. A quick check with the breakout box confirms that the 15 V power supply is indeed off, as is the FET heater. No amount of writing to register 30 changes this fact.

The power was cycled on DCRC33 three more times before the 15 V power supply turned on again. This time round, the power the FET heater was off, and Qi & Qo biases to ground yielded potential differences of -0.24 V & -0.26 V respectively. Again, these values are not consistent with 0 V. Turning on the FET heater and 15 V power supply at the same time does nothing, ie. the 15 V power supply remains on but the FET heater remains off. Even turning off the power supply but leaving the FET heater on makes no difference. Additionally, writing to register 59 does nothing, implying that there is a communication problem. Finally, the 15 V power supply and FET heater were switched off in the PuTTY session, which did result in the 15 V power supply turning off. However, it was impossible to resurrect the power supply. Judging from the fact that anything to the registers appeared to be ignored by the board, the fact that the 15 V power supply did turn off could simply be a coincidence.

Tested DCRC31 three times, cycling the power in between tests. No problems witnessed: the board did exactly what it was asked to, when it was asked to.

Switched back to first Phihong PoE supply and replaced ethernet cable connecting DCRC to PoE supply. Tested DCRC32 first, three times cycling power in between tests. Again, 15 V power supply and FET heater did not turn on even when values were written to regsiter 30 manually through the PuTTY session.

DCRC33 continues its unpredictable behaviour. During the first test with this board, the 15 V power supply did not turn on. However, the FET heater did. Additionally, the FET remained on even when it was turned off by writing to register 30 via the PuTTY session. The board did take about a minute before connecting to the PC. Finally, the PuTTY session quit without prompting during the test, indicating that connection was lost. Connection was reestablished automatically, but same problem with FET heater and writing to register 30 was witnessed.

After cycling the power, the second test with DCRC33 also resulted in the FET heater staying on after it should have been switched off automatically, and the 15 V power supply did not turn on. However, this time it was possible to turn off the FET heater by writing to register 30. Yet, it was not possible to turn the FET heater back on, neither was it possible to switch on the 15 V power supply. It is unknown if the FET heater turned off because it was instructed to or if this was a coincidence.

Neither the FET heater nor the 15 V power supply turned on during the third test with DCRC33, again after cycling the power. The board remained unresponsive to anything written to register 30.

Testing of DCRC31 was as expected: three tests cycling power in between; no problems.

The results of the above tests can be summarised as follows:

1. DCRC31
   • No problems with power supplied or communication to board, thereby implying that the problems witnessed with DCRC32 & DCRC33 lie exclusively with these boards rather than the PoE supplies or cables.
   • Not tested here, but charge (Qi & Qo) baseline has large offset of the order of +2 V. Will require further testing to understand the reasons for this.
   • Connection to board stable.
2. DCRC32
   • Both 15 V power supply and FET heater refuse to turn on no matter how many attempts are made to cycle the power or write directly to the boards. This implies that there is a major problem with the power supplied to the board.
   • Unsure if there is a communication problem. Perhaps further testing is required, although a problem with the power supplied may hide any evidence of this.
   • Connection to board is sometimes lost.
3. DCRC33
   • Very unpredictable behaviour witnessed. Sometimes FET heater will turn on upon startup, sometimes 15 V power supply will turn on, sometimes both will fail to turn on, etc. There is almost certainly a problem with the power supplied to the board, although the board does occasionally work.
   • As the board is occasionally operational, this has revealed a problem communicating with it. Although it appears as though values are written to registers, no changes are made to the board. For example, instructing the board to turn on the FET heater may not result in the FET heater turning on, yet reading register 30 will yield 0100 implying that it has indeed been turned on.
   • Connection to board is sometimes lost.
   • Occasionally, upon connecting the board to the computer, measuring Qi & Qo biases to ground yields values that are not consistent with the board's default values upon startup. The default values are calibrated such that Qi & Qo biases to ground should read 0 V when checked with a multimeter.

Begin with leak checker on and open up to vent with a background rate of 4.9E-9 mbar*l/s. Currently valves 11A, 11B, 2, 7, 1, 2A, and 5A open, all others closed. With He4 pump also on, opening the vent speedivalve raises the leak rate to 7.4 (+/-0.2)E-6 mbar*l/s. Closing 2A and 5A to isolate He4 pump lowers the leak rate to 3.42E-7 mbar*l/s. The following tests were performed:

1. Use the syringe to apply He to a small area. Tested all flanges on back of IGH and cold trap 1 and 2 flanges. No significant raise in leak rate.
2. Remove syringe cover in order to apply heavy flow of helium over large area. Test flanges from cold traps 1, 2 and He3 return. Initial rate was at 3.51E-7 mbar*l/s. During/after helium was applied the leak rate increased with a maximum leak rate of 4.36E-7 mbar*l/s. Without the syringe, it is difficult to pinpoint which of these flanges is leaky, and we can't discount any neighboring flanges as the source of the leak.
3. A small gap leading into the IGH from the upper back corner was noted. He tube was stuck a few inches into the IGH. After a few seconds no increase was noted ans so the tube was removed. Immediately after, as the tube was testing the condenser flange, the leak rate rose to a maximum of 4.87E-7mbar*l/s. This might have been due to He in the IGH or entering the condenser flange.
4. After the leak rate settled He was again applied to the condenser flange, which resulted in a maximum leak rate of 4.91E-7mbar*l/s. As the leak rate began decreasing the He was again applied to the inside of the IGH through the small hole in the upper left corner. The leak rate reached a lower value of 4.83E-7mbar*l/s before increasing to a maximum value of 4.98E-7mbar*l/s.
5. Apply He to a matching hole on the upper right corner of the back of the IGH which is further from neighboring flanges than the left corner hole. Leak rate was still decreasing as He was applied and reached a minimum value of 4.63E-7mbar*l/s before increasing to a maximum value of 4.70E-7mbar*l/s. Once again, the He was then applied to the hole on the left corner. The leak rate level was at 4.52E-7mbar*l/s and dropping as the tube was inserted. Leak rate reached a minimum of 4.47E-7mbar*l/s before increasing to a maximum value of 4.95E-7mbar*l/s. It's apparent that there is a leak either in the IGH or in the flanges near the entry to the IGH. However, with this method it is difficult to find a more specific source.
6. Starting with a background leak rate of 3.65E-7mbar*l/s the lines leading to the He cold trap as well as the plastic flange leading to the He cold trap were tested. No significant change in leak rate was detected.

The next step is to open the side of the IGH and use the syringe valve to test the inner portion of the IGH.