I think if the officer did not give you a ticket then the rejection of your road test is illegal, as it's prior to the test and RMV does not own the plaza you took the practice.
I think your choice of not arguing is better so that you can save time and energy. Alternatively, you can call and ask if you can avoid this officer as he would have gave you prejudge and may treat you differently.