Request for Marlin/Arduino Expertise

[Josh]

Oh MakerGear community, we request your assistance to identify the source of this elusive issue:

We've received reports and also experienced infrequent, intermittent X/Y skips while printing. Most of the skipping issues that are reported are related to the specific model being printed or improper machine assembly. However, there is also an elusive scenario that we've been trying to track down.

During our in-house investigation, we've found an underlying issue with the M2 firmware (Marlin) - during certain moves at certain speeds (most commonly, from the front left to rear right corner, at speeds above default "travel" moves in S3D) there will be a glitch in the stepper STEP signal. If the axes are moving at high enough speed during the glitch, the sudden "attempt" at a full stop (no signals being sent = no movements being commanded = no motion) can lead to a skip/layer shift in X, Y or both.

The rate of occurrence of this glitch depends on the version of the Arduino IDE used to compile and upload the firmware, the direction and possibly distance of the move, and the speed of the move. Arduino IDE 1.5.7 and above will cause glitches to occur when moving from 0,0 to 220,245 at F18000 reliably, while 1.5.6 and below will not; both versions see skipping when commanding a move from 0,0 to 220,245 at F90000 (which is intentionally faster than the machine can support - the firmware should be clamping this move to 500mm/s [F30000] depending on firmware max feedrate settings, and even lower to ~450mm/s [~F27000] due to the "max step frequency" and steps/mm values in the firmware). I was able to further isolate the change to the compiler version change from 1.5.6 to 1.5.7 - when forcing IDE 1.6.3 to use the compiler from 1.5.5, the glitch incidence rate matched that of 1.5.6 and below (which is to say, "normal").

This post on the Marlin github issue tracker (https://github.com/MarlinFirmware/Marlin/issues/2370 ) contains most of my testing protocol and results, oscilloscope shots of the glitch and timing, and discussion between other Marlin users/developers on this issue. If you have any information or questions, feel free to join the discussion here, on the issue tracker, or by emailing me directly (josh @ the place I work dot com)

Thank you!

[innkeeper]

What version(s) of marlin

[Josh]

The issue post has all that information, but - both our old stable version (~3 years old at this point) and one downloaded from the main Marlin release ~a month ago.

[sshwarts]

One question Josh, when you turn on compiler warnings and compare the two compiles, any difference?

Scott

[lem]

If the Rambo board has enough memory try turning compiler/linker optimizations off and see if the problem persists.

If code space on the Rambo is tight, someone may have taken advantage of a compiler quirk that reduced instruction count, but is a characteristic that is not in the newer compiler.

[PcS]

All of my firmware uploads have been with the most current adruino. 1.6.1 is it ? I have not seen the shift at all in 500 hrs of printing. Although I do not think I am running my machine fast enough to cause it.

[jsc]

An hypothesis: at high step rates, the length of time it takes to run the bulk of the interrupt handler is overrunning the compare value (OCR1A) set towards the end of the ISR in stepper.cpp. When this happens, it has to go one through one whole overflow and back again before it triggers the next interrupt. Taking just a little longer to run than OCR1A would explain the timing of the gaps you are seeing, and also why both axes are cutting out at the same time (if you were to probe the extruder, I bet that would cut out as well). To see if this is happening, you can check the value of TCNT1 after OCR1A is set to see if it is >= the value it is getting set to, and then do something to indicate that that happened (set a flag, have the main loop report it; halt and catch fire; etc.) It would also explain why different compiler versions would have an effect, perhaps the code is slightly slower on one release.

To fix it, I would try compiling with different compiler flags to see if it can get any faster (-O3 instead of -Os, maybe; remove -g because who needs it when you can't run a debugger). You can also reset TCNT1 manually at the end of the interrupt if it has overrun to something slightly less than OCR1A to make sure the next interrupt happens as soon as possible.

I dont know what's happening with issue 2, but my guess is something is underflowing an unsigned value and it's going from "move real slow" to "move real fast". Looking into it a little, the first thing that catches my eye is line 728 of stepper.cpp: "step_rate = acc_step_rate - step_rate;" That is checked against a lower limit a few lines later, but if step_rate > acc_step_rate, it will have underflowed already. I don't know if that's the issue, but it's plausible.

[innkeeper]

Is this repeatable if you run chose directly off the sd card with nothing connected to the sub ... In other words initiated from the lcd. Also you mentioned s3d ... Is it repeatable using other tools

[Josh]

Thank you all for the rapid feedback X)

sshwarts: A bunch of "only initialized variables can be placed into program memory area" errors in 1.5.5, but I need to go through and make sure I have all warnings enabled in both compilers before I can make a definitive statement on which are present.

lem: That certainly could be part of it, though in all of these tests used memory space is ~20%.

PcS: Even with 1.6.3, it's not a constant event; I did find that at F18000 (300mm/s) it's pretty common, but if none of your speeds hit that, you may not ever actually see it; I've also done all of my testing on only a few different boards (several different RAMBos and an Arduino Mega), so there's a chance it could be hardware-exacerbated as well.

jsc: Rick and I have been thinking of some sort of race condition all along, though yours is the first clear concept on where it is and exactly what is going on. Compiling with -O3 doesn't seem to have changed anything; I'm going to need to do some research and experimentation (both deeper in the firmware and on compiler usage) to run the other tests.

innkeeper: SD only is actually one I have not tested yet, as one of my first tests was disabling SD and other non-necessary functions... That will be next on my list. As for S3D - I was simply using that as an example of the feedrate source of 300mm/s, which, as I said, is the default non-printing move speed in S3D; most of my testing has been run from Pronterface, either hand-coded .gcode files streamed over USB, macros (the main reason for using Pronterface, as I can easily have more than three at a time), or direct entry/send.

[Josh]

innkeeper: SD card doesn't help. Compiled in both 1.5.5 and 1.6.3, running an auto0.g file from the SD card (so no LCD to add complications), still see the glitching, when powered from both USB and PSU.