Lowering output latency to min

Hello,

I hope this is the correct forum, if not please just let me know.
I am an electrical engineer and not a guru programmer,so i hope you will be a little understanding, if i make obvious mistakes or don't mention important details right away.

tl;dr: I am using a HifiBerry DAC+ ADC on a Rasp4B 2GB with Portaudio on Rasbian.
I can't get a latency of smaller than 4ms (bout 2k Samples in given config) between writing a signal to the outputs and receiving it on the inputs, when they are connected directly.
The config i use atm is:

fs =44.1kHz
64 samples
2 in-/2 outputs

I tried enableing realtime in Portaudio ("PaAlsa_EnableRealtimeScheduling(stream,1)") and raising the priority of my program, but that doesn't change a thing. Lowering the buffersize or raising the sampling frequency didn't help either. I tried to get into ElkOS, because it was repeatedly stated, that it's faster, but that's a deep rabbit hole for me.

So my questions are:
Is it simply not possible to go faster? (if yes forget all other questions)
Do i need to get a realtime capable OS?
Are there some configs i didn't see? (Maybe the output collects multiple buffers before playing them)
Do i need to go deeper and write my code in ALSA?



Long Version/purpose:
I try to do an active noise canceling set-up and the delay between my first and second microphone is in the range of 1.5ms, so i hope to get the latency down to at least 1ms (ideally less).

Best regards
Jerry