It looks like it wasn’t actually using STARTTLS on port 25. I just assumed it was, because port 465 is the only standard(ish) SMTP port that assumes TLS. I captured some network traffic however, and the app starts spewing incomprehensible binary junk right away, when the checkbox is checked.
I guess my Exim4 server was just smart enough to think “hmm, this doesn’t look like a plain text command, maybe I should try interpreting it as an encrypted connection… hey, that works!”
There’s some good news though. I looked through the documentation for the library that the app is using to actually do the mailing; even if you do not check the box, STARTTLS will be used as long as it’s advertised by the server: https://nodemailer.com/smtp/#tls-options
I verified this by capturing the handshake part of the conversation:
< 220 AM4PR07CA0003.outlook.office365.com Microsoft ESMTP MAIL Service ready at Wed, 26 Feb 2020 21:11:20 +000
> EHLO [127.0.0.1]
< 250-AM4PR07CA0003.outlook.office365.com Hello [xxx.xxx.xxx.xxx]
< 250-SIZE 157286400
< 250 SMTPUTF8
< 220 2.0.0 SMTP server ready..
The only downside is that it’s a bit too voluntary for my liking. If for some reason, O365 would decide to not advertise STARTTLS, the app would happily send stuff “in the clear”.
Assuming someone could intercept your network traffic, it’s reasonable to assume they could do a MitM attack and just forget about that little line advertising STARTTLS, and presto.
If someone were to do this, you might get lucky and the app might not find a supported method to authenticate, as simple username/password authentication schemes are refused over unencrypted connections and several other standard authentication schemes also seem to be refused by O365.
Major bummer I ran into though: even though STARTTLS will be used, there doesn’t seem to be much certificate checking going on. I intentionally used a different host name than my mail server certificate is for, the the app doesn’t care. Sends its stuff anyway. So: no protection from MitM attacks whatsoever, only from casual readers with access to your local network that you could possibly deter.